Thursday, April 27, 2006

Missing NUnit Fixture Errors In CC.Net

I've noticed a few times that when we have an error in our test fixture teardowns at work that the CruiseControl.Net build report page didn't say what the cause was.

It correctly identifies an error return code from nunit-console, however it is not listed as a "error" on the build report or on the NUnit Details page since it isn't actually a specific test that failed. Rather confusing at the time...

I've tweaked the unittests.xsl stylesheet for CruiseControl.Net to fix this problem, and now get the test-suite failure message listed on the front build report page. If you are interested you can download it from here. I've also submitted it to the CC.Net developers group so hopefully it may be included in a future release.

[Update 14-Oct-06 - A new version of this stylesheet for CruiseControl.Net 1.1 is available from here (rename it to unittests.xsl). There is no real difference I saw - the CC.Net developer refactored their stylesheet to clean it up so I just reapplied the changes over the top. It is also listed in JIRA]

Friday, April 07, 2006

Illegal Characters & NCover Versions

Some people are reporting problems with NCoverExplorer which are caused by an older version of NCover - that is NCover version 1.4.6. The problem occurs when they try to drill down into the source code within the GUI.

Please do NOT use NCover 1.4.6. The problem is within the coverage.xml file created - the filename attributes end up with extraneous characters appended (something like '�'). I may try to strip this out myself in a future NCoverExplorer version, however as Peter does not even list this version for download nowadays I would rather people were forced to use a more "supported" NCover download.

So to clarify - which version of NCover should you use?

If you do not have the .Net 2.0 framework installed on your machine then you should use NCover 1.3.3. This is exactly what we use at work every day without any problems.

If you want to profile a .Net 2.0 application, then you must use one of the latest NCover 1.5.x betas. The very latest 1.5.4 beta does have some known issues (see the NCover forums - in particular here, here and here). Peter has had more important things to think about recently (congrats by the way mate!) but he has indicated the next release with some bug fixes will be coming out soon.

If you want to profile a .Net 1.1 application but happen to have .Net 2.0 installed then you can use either version of NCover. I'm sure Peter would much rather people were using 1.5.x than 1.3.3 so once his next beta release comes out that would be my preference, particularly for the coverage exclusion attribute support.

All supported NCover versions can be downloaded from NCover.org.

One final point - I see additional confusion coming from the two NCover projects out there. The NCover we want you all to be using is from NCover.org - unfortunately all too easily confused with another "inactive" project at SourceForge. Irritating I'm sure for all concerned that the two projects have the same name - just remember NCover.org and you can't go wrong.

Tuesday, April 04, 2006

NCoverExplorer... v1.3.3

Another stack of new features in the latest generation of NCoverExplorer. I've added summary html/xml reporting, filtering capabilities, basic code metrics and very significantly a console version for automated build usage.

Html reporting as I posted about here has been requested by a number of users, including myself! For this release I have implemented summary reports since from the feedback I received they were of most interest. From either the GUI or the the console application you can now generate one of three reports:
  • Module Summary
  • Namespace Summary
  • Module and Namespace Summary
As you can see from this dialog you have a choice of output either directly to html or to xml (the latter having an associated stylesheet for rendering html).


An example of the module summary report is shown here. Note the inclusion of some basic statistics at the top right. You can also see that our "satisfactory coverage threshold" is displayed at the top (90%) and that any graph bars exceeding this threshold (but not 100%) have the uncovered code shown in yellow. So a glance at the report for the red bars will tell you which modules or namespaces are possibly in need of investigation.

Statistics are also available within the NCoverExplorer GUI. Note that these are derived from the NCover coverage report, not the source code. The totals deliberately ignore any "excluded" code.


Filtering offers a way of simplifying down the coverage tree without affecting the coverage totals (unlike "excluding" which recalculates the totals). There are two filters available - hiding all 100% covered nodes (for when you want to focus on the todo list), and hiding all 0%/unvisited nodes (for when unit testing isolated functionality).


My thanks to Lutz Roeder for the public donation of his excellent CommandBar code. You can see the impact in this in the new look to the menus with images and the new Toolbar shown below.


Including nodes that were marked as "excluded" is also now possible. You can think of this as an "undo" feature to override either manual exclusions or those defined in the options dialog. Including a node will also recalculate the related coverage statistics.


NCoverExplorer.Console came about from two features on my list. One of these from Matt Hargett of imeem.com was to enable failing a build if coverage does not meet a particular threshold.

The second item was the ability to produce xml and html coverage summary reports in an automated fashion. As anyone who has used NCover on large projects will testify, coverage.xml files can be very large. Rendering these using xsl while being an elegant and flexible solution is just way too slow, particularly when the coverage file is many megabytes. Merging the coverage.xml file(s) into your CruiseControl.Net buildlog.xml files also slows down the rendering of every single page for that build on the CCNet web server.

My solution to this is based on the assumption that I do not want to see line level coverage information on the build server. I just want to see a summary of the project, modules and perhaps down to namespace level of coverage. By adding NCoverExplorer.Console.exe to the build pipeline we can transform the NCover results very quickly into a tiny summary xml file. With the appropriate stylesheets we can now render coverage summary reports in CCNet in a fraction of the time previously taken.


All of the options available in the GUI in terms of exclusions, thresholds, project name, output as xml and/or html are available.

I have also developed an additional stylesheet for use on the Build Report page of CCNet as shown below:

Since these are just stylesheets transforming the NCoverExplorer xml file they can be customised to your own tastes if required.

The final piece of the puzzle is to automate through NAnt and MSBuild the creation of the xml/html reports. You can of course just use an <exec> task, however I have put together some custom tasks for both tools which offer a bit more flexibility. These are available for download in the NCoverExplorer.Extras zip file at the bottom of this post along with some included examples and the necessary stylesheets for CruiseControl.Net.


If anyone requires it I may put up a blog entry walking through such configuration or perhaps see about getting it documented on the CruiseControl.Net wiki. It is fairly straightforward though:

  1. Copy the NCoverExplorer.xsl and NCoverExplorerSummary.xsl stylsheets from the NCoverExplorer.Extras zip file to your CruiseControl.Net\webdashboard\xsl folder
  2. Edit your CruiseControl.Net\webdashboard\dashboard.config file to point to these stylesheets. I have included an example dashboard.config in the .zip
  3. Modify your NAnt or MSBuild script. As stated you can either execute the console application directly with command line arguments or use my custom tasks. For instance to use the custom NAnt task copy NAnt.NCoverExplorer.Tasks.dll to your NAnt\bin\Tasks\net folder and follow the example syntax contained within "example.build".
  4. Modify your CruiseControl.Net project file (ccnet.config) to use the merge publisher to include the xml output from NCoverExplorer.

For my own sanity, other than any bug-fixes I am intending to put a hold on further development for a little while. This has consumed every evening and weekend for the last couple of months so it's time for a break. Any donations could of course help rekindle my interest ;-)

As always I welcome any feedback. Enjoy!

Download TestDriven.Net 2.0.1578 combined install

Download the latest NCoverExplorer versions from here.

Release Notes
FAQ