Thursday, October 05, 2006

CCStatistics for CruiseControl.Net 1.1

A few days ago CruiseControl.Net 1.1 was officially released. Thanks to all those responsible - it has become an entrenched daily tool for many of us. I've upgraded a couple of build servers so far - well worth doing for the performance improvements and statistics features alone. This also gave cause for me to knock up a supporting tool called "CCStatistics" - more on this below.

When upgrading there are a few things you need to change in your CC.Net project files that are not all listed in the release notes...
  1. Remove any <publishExceptions> elements.
  2. Change the format of your <weburl> links to jump to the latest report in CCTray. They should now be in the format below, unlike the query-string approach used previously:
    http://localhost/ccnet/server/local/project/MYPROJECTNAME/ViewLatestBuildReport.aspx
  3. If you want to use the new statistics functionality you need to add a <statistics /> node to your <publishers> section. Without it when you click on the "View Statistics" page you get a nasty exception being thrown.
Speaking of statistics, if you hadn't guessed already this is one of the more interesting additions for me. In my opinion it is a "0.9" version in terms of readiness for release - there are a number of bugs I found I have added to Jira and the documentation is incomplete or misleading in a couple of instances at the time I write this.

That said it has the potential to be a stonkingly good feature. Who doesn't want to see the most common reasons their build fails (when you have ClearCase performance issues like we do) or see their unit tests count, code coverage etc over time all on a single web page? No more endless drilling down into build log hyperlinks...

To use this feature you add a <statistics> node to your <publishers> section in the project file. I recommend you put this at the end of your publishers (after any <merge> element) so that way you can get additional custom statistics from files merged into the build log. When CC.Net runs the project it will then generate in your artifacts folder:
  • report.xml (not statistics.xml as the documentation says). Contains all the statistics for builds to date.
  • reports.bmp (which is actually a png - JIRA). A graphic illustrating the "TestCount" statistic across the build log cycle.
  • statistics.csv (JIRA). Intended to be an exportable alternative to report.xml - but it has garbage output in it currently due to a few bugs.
When you click on the "View Statistics" link from a build project page you will see the standard statistics on a page like the following:


The report is fairly basic in appearance but that should be easily changed by editing the statistics.xsl file. That is next on my list and once done I will contribute it for anyone interested (or feel free to beat me to it!). Strangely the reports.bmp file is not referenced in this stylesheet by default - so the graph produced is not actually being used yet as far as I can tell.

A minor limitation in the current release is that you cannot "replace" statistics with your own ones without editing the xsl. Not the end of the world but again I've posted a feature request on JIRA on ideas to make that more flexible. You can however add new statistics as mentioned here. So for instance if we want to add the code coverage totals for the build as a result of running NCoverExplorer we can do this:

<statistics>
<statisticList>
<firstMatch name="Coverage" xpath="//coverageReport/project/@coverage" />
</statisticList>
</statistics>

This will result in an additional column being added to the statistics page on the right-hand side. You could of course add other NCoverExplorer statistics as well such as lines of code etc.

There is a catch to all this - statistics will only be generated for future builds, not for your legacy ones. That is where my CCStatistics tool comes in. This will parse all your build log files for a project and produce an updated reports.xml, reports.bmp and statistics.csv file as though you had been running CruiseControl.Net 1.1 "forever".

There is another benefit to this tool I can see. Undoubtedly you will over time decide on additional statistics you would like to measure. Provided they are sourced from data that existed in historical build logs, you can regenerate all your statistics to date with the new data included to give you a much better picture of the progress of your project. If the data isn't present (say you just started merging NCoverExplorer code coverage) then the values will be blank.


It should be pretty trivial to use - drop the executable into your CruiseControl.Net\server folder, click on the Load button to list your CC.Net projects, choose which to recalculate statistics for and away it goes...

I've made all the source code available - perhaps the CruiseControl.Net developers might be interested in adding it to their solution and maintaining it? There is also a compiled executable against CC.Net build 1.1.0.2498 (aka CC.Net release 1.1.1). To use the source code just unzip into the same \project source code folder as the other CC.Net projects and add to ccnet.sln.

This was a very quick hack from a few hours work and has the usual "it works on my machine" proviso. There is no care or attention over internationalization and I have only ever used the 1.x versions of CruiseControl.Net. So use at your own "risk" - although as the only file access it does is reading logs and writing the couple of statistics files I don't think there's much to go badly wrong. The code itself has a few "hacks" so as to work without touching the original CC.Net source code - these may be cleaned up at a later date if the dependent objects are enhanced. Note also that this code results in the same "useless" .csv file - that bug is in CC.Net source and even if I did work around it obviously your future builds would still have incorrect lines being published.

If you notice any problems or have suggestions feel free to let me know. Note that it will backup your previous statistics files with a .old prefix so if you notice a problem after running it once you can revert.

Also thanks to Jamie Cansdale for his help today with a sticky issue I had (the NetReflector component only "works" if the assembly containing the type it is trying to instantiate is loaded in memory - just referencing the .dll containing it is not enough).

Download CCStatistics executable (Depends on CCNet 1.1.0.2498)
Download CCStatistics source code

[Update 14-Oct-06: I've pushed up a new version which includes progress bars and timings. When processing a big directory of large files CCStatistics may become unresponsive - don't panic as it will finish eventually. I have also updated the unittests.xsl stylesheet to fix the still outstanding issue in JIRA of test suite failures not showing up that I blogged about previously.]

[Update 22-Sep-07: Damon Carr has significantly updated this tool to work with the latest CC.Net (1.3) which you can can download here as mentioned in the comments below. Thanks also to Drew Noakes for his previous work for compatibility with CC.Net 1.2. Having "served its purpose" at the time for my needs I did not have the time to maintain it myself but am glad that others have found it useful enough to do so. Well done guys...]

15 Comments:

At October 05, 2006 11:03 PM, Anonymous Jeroen van Menen said...

Grant,

Thanks for this great tool. Works like a charm. Now I've got all the historic data in the statistics (great new CC.Net feature by the way)

Regards,
Jeroen

 
At October 07, 2006 11:50 AM, Blogger kiwidude said...

Thanks for the feedback Jeroen, glad to hear it did the trick and was of use to you as well.

 
At October 08, 2006 11:35 AM, Anonymous Moshe Hajaj said...

Thank you again, Grant.
I think this tool will be very yseful for us too.
btw, I noticed that reports.bmp always shows one metric (TestCount), did you find how to graph it will all the data (coverage, etc.)? it would be great if it's possible, and we'll find how to display it on the dashboard.

 
At October 08, 2006 1:39 PM, Blogger kiwidude said...

Moshe,

The statistics are hardcoded in the the StatisticsChartGenerator.cs constructor ("TestCount" and "Duration") so there is nothing that can be done at this point without changing the source.

As I put in my blog entry I'm sure this is just the first cut of the whole feature - hopefully someone will enhance it further in future releases. It would be nice to produce multiple images for the different types of statistics as you suggest.

 
At December 19, 2006 2:35 PM, Anonymous Koob said...

Thanks for sharing this tool!

 
At April 01, 2007 9:46 PM, Anonymous Anonymous said...

THanks kiwidude. I downloaded the source, set the references to point to CruiseControl 1.2. and got it to work fairly easiely.

I was wondering if it would be possible to execute a custom ms build task AFTER the statistics file is generated. I'd like to create a build task that generates a custom report that includes things from the statics file, among other things. Of course, I'd want to generate the report after the statistics file was generated. Thanks!

 
At April 01, 2007 10:59 PM, Blogger kiwidude said...

Hi mate,

Have you tried placing an msbuild task in your publishers section just after the statistics node in the publishers section? As ccnet executes the tasks in order I would have thought that would do what you want...

 
At July 11, 2007 6:25 PM, Blogger drewnoakes said...

Hi Grant,

Thanks for this tool. I'm trying to make it work here... Unfortunately I'm getting an error:


System.BadImageFormatException: The format of the file 'ccnet.vsts.plugin.dll' is invalid.

File name: "ccnet.vsts.plugin.dll"
at System.Reflection.Assembly.nLoad(AssemblyName fileName, String codeBase, Boolean isStringized, Evidence assemblySecurity, Boolean throwOnFileNotFound, Assembly locationHint, StackCrawlMark& stackMark)
at System.Reflection.Assembly.InternalLoad(AssemblyName assemblyRef, Boolean stringized, Evidence assemblySecurity, StackCrawlMark& stackMark)
at System.Reflection.Assembly.LoadFrom(String assemblyFile, Evidence securityEvidence, Byte[] hashValue, AssemblyHashAlgorithm hashAlgorithm)
at Exortech.NetReflector.NetReflectorTypeTable.Add(String path, String searchPattern)
at ThoughtWorks.CruiseControl.Core.Config.NetReflectorConfigurationReader..ctor()
at ThoughtWorks.CruiseControl.Core.Config.DefaultConfigurationFileLoader..ctor()
at ThoughtWorks.CruiseControl.CCStatistics.MainForm._OnLoadCCNetConfigButtonClick(Object sender, EventArgs e)

[snip]

=== Pre-bind state information ===

LOG: Where-ref bind. Location = \\\\devapp02\c$\Program Files\CruiseControl.NET\server\ccnet.vsts.plugin.dll

LOG: Appbase = \\devapp02\c$\Program Files\CruiseControl.NET\server\

LOG: Initial PrivatePath = NULL

Calling assembly : (Unknown).

===

LOG: Policy not being applied to reference at this time (private, custom, partial, or location-based assembly bind).

LOG: Attempting download of new URL file://devapp02/c$/Program Files/CruiseControl.NET/server/ccnet.vsts.plugin.dll.


Has anyone else encountered this?

It appears that NetReflector is loading the VSTS (Team Server?) plugin DLL "ccnet.vsts.plugin.dll" and failing. If I remove the file, it fails when looking for it.

Thanks again,

Drew.

 
At July 11, 2007 7:02 PM, Blogger kiwidude said...

Hi Drew,

I havent used this for a long time - I suspect it is all to do with how CC.Net 1.3 runs under .NET 2.0 now (likely that plugin dll is compiled under 2.0) and the CCStatistics executable is running under 1.1. Try forcing CCStatistics to run under 2.0 with a .exe.config (or recompile it) and that should sort your issues...

 
At July 12, 2007 10:38 AM, Blogger drewnoakes said...

You're right Grant. I recompiled the app for CCNet 1.2.1.7 in .NET 2 and made it available at my website. You're welcome to put the binary on your site -- probably more useful there. For now, others can download it from my site rather than build it themselves.

 
At July 27, 2007 8:35 PM, Blogger Damon Carr said...

Hi there... I've updated the source to run on the new 1.3 release (this requireed a number of code changes).

Also, I am almost done with a new feature allowing a UI driven specification of the items you would like graphed. I plan on also integrating this with the 'Dojo Graphs' : http://www.ridgway.co.za/archive/2007/04/22/Dojo-Based-CruiseControl-Statistics-Graphs.aspx


If anyone wants the source or just a binary let me know. Otherwise as far as I know your stuck if running 1.3 due to API changes.

Thanks,
Damon Carr
http://damon.agilefactor.com

 
At August 05, 2007 9:16 PM, Blogger carel said...

Hi Damon

I would love to have the updated 1.3 version of the CCStatistics. If possible can you mail it to me at carel dot lotz at sanlam dot co dot za.

Thanks
Carel Lotz

 
At September 22, 2007 5:27 AM, Blogger Damon Carr said...

All,

This is now available from the this link:

http://damon.agilefactor.com/2007_07_01_archive.html



Thanks,
Damon Carr

 
At September 22, 2007 8:51 AM, Blogger kiwidude said...

Great effort Damon (and Drew as well), and thanks for taking on the effort. I have updated the blog article to link to your posting. I've been spreading myself a bit thin this year and with no personal need for this tool since it was first hacked together I'm afraid it dropped down the priority list.

I'm sure the time will come when I have use of it again so now we know where to go looking...

Regards,
Grant.

 
At February 12, 2012 4:03 AM, Blogger VC said...

Damon's link is broken; Could you please share a working link?

 

Post a Comment

Links to this post:

Create a Link

<< Home