Thursday, April 19, 2007

NCover Problems / Fixes - Part 2

In Part 1 of my "everything you wanted to know about NCover but were afraid to ask" series I covered the basics of how NCover works and profiler registration issues. Let's hit another of the most common problems:

Empty Coverage.xml File

This is quite an interesting one, as there are a number of possible causes. You perform your coverage run and don't see any errors in the console output. Yet when you open your coverage.xml file you see something like this from NCover 1.5.7.
<?xml version="1.0" encoding="utf-8"?>
<?xml-stylesheet href="coverage.xsl" type="text/xsl"?>
<!-- saved from url=(0022)http://www.ncover.org/ -->
<coverage profilerVersion="1.5.7 Beta" driverVersion="1.5.7.0"
startTime
="2007-04-19T18:38:47+01:00" measureTime="2007-04-19T18:38:47.921875+01:00" />
If using an earlier version of NCover the xml will be even less as there will be no attributes on the <coverage> node. So what has gone wrong?

Fundamentally it comes down to one thing - there were no debug symbol files found for assemblies that NCover thought it should be profiling. There are multiple reasons as to why this might be.

Debug Symbol Files

Ok, so not everyone knows what a .pdb file is and what purpose it serves. A program database (PDB) file holds debugging and project state information (Source: MSDN), such as line numbers, source code file information and sequence points. This is what NCover needs to produce in the coverage.xml file the line, column and filename information for each sequence point profiled in each method.

If you are compiling a windows or console application with a "Debug" configuration in VS.Net then by default a .pdb file will be produced for each of your assemblies. You can control this in the project properties dialog. Note that by default for "Release" builds no pdb file is produced - another common thing people forget.

For those of not using VS.Net, you can also produce these files using the command line compilers using the /debug option.

When NCover runs, it looks in the current directory of the loaded assembly for a matching debug file. If it doesn't find the .pdb for an assembly, you will see a "Failed to load debug symbols" message in the coverage.log file (more on the log in a future post). NCover then adds that assembly to it's internal list of ones to "skip" and will not bother attempting to profile it.

You can use this to your advantage. The less assemblies being profiled, the smaller the resulting coverage.xml file and the faster it will run. So rather than using the //a argument for NCover to specify assemblies (see below) you could simply delete the .pdb files from your testing bin folder prior to the coverage run for assemblies you do not want profiled.

Having ensured that you are correctly generating .pdb files in your output folder with the assemblies, what circumstances can lead to NCover saying it cannot find them?

- Your assembly was installed in the Global Assembly Cache (GAC)
- You are shadow copying assemblies such as when running nunit.

Profiling GAC Assemblies

When an assembly is installed into the GAC using gacutil it is physically copied to a physical folder location in:

%windir%\Assemblies\GAC\<simple name>\<version>_<culture or blank>_<public key token>

However what isn't so immediately obvious is that the .pdb file is not installed into that same folder by default. So when NCover runs and your assembly is loaded from the GAC, then NCover will not find the .pdb and will skip profiling for it. For the same reason you will have problems trying to use a debugger with that assembly as well.

So how can you get around this? Obviously uninstalling from the GAC is one option and certainly the easiest. I appreciate this isn't always the most convenient though.

Plan B is to move the mountain to Mohammed - and copy the .pdb files into the physical GAC folder. Unfortunately Microsoft didnt make that very easy for us but in typical fashion a number of people out there found a way. You can figure out the location using the pattern above - using the command line of course. Patrick Cauldwell has a blog entry and script for automating it with Powershell (Monad) here. Also Scott Colestock has a NAnt task for doing this here.

NUnit Shadow Copying

This scenario is one I only figured out today in response to an issue posted on the NCover forums here.

By default NUnit will shadow copy your assemblies to a shadow cache folder for running it's unit tests - no shock/horror hopefully in that statement. What isn't so obvious however is the behaviour if you compile your assemblies and .pdb files on one machine and then test them on another.

It appears that .NET does not copy the .pdb files to the shadow cache when they were compiled on another machine - only the assemblies.

So your unit tests still happily pass - but you get no coverage information. Yikes!

Why does it happily copy them on the same machine but not on a different one? I haven't had time to figure that out yet. My guess is some sort of security/signing type of issue but that is just pure speculation. If you know please tell me - I've put a thread on the nunit forums here.

What is the workaround? Either profile your assemblies on the same machine you compiled them on, or run nunit-console with the /noshadow argument. Works a treat.

Choosing Assemblies To Profile

NCover will by default profile all assemblies loaded into memory for which it finds .pdb files. If that path includes a lot of debug files for other assemblies that you aren't interested in getting coverage information for, then this will both harm your profiling performance and give you massive coverage files.

You may also intentionally want to exclude layers of your application - perhaps the presentation or unit test assemblies are deemed irrelevant to your coverage results. I exclude test assemblies from our projects as I feel they unfairly distort the overall coverage percentage. I appreciate others like to have them so they can spot how much of their unit test code was not run.

I mentioned above that a "quick and dirty" trick is to delete the .pdb files for assemblies you do not want coverage for. A more gentlemanly approach is to use the //a argument with NCover. However there are a couple of tricks to it.

Firstly, note that this is the AssemblyName, not the physical assembly file name! No path information, no .dll suffixes etc. A common mistake is to do something like this:

ncover.console.exe nunit-console.exe foo.tests.dll //a foo.dll

If you do this, then you will get an empty coverage file - as NCover did not find a matching AssemblyName. Almost certainly the correct command line for the above is:

ncover.console.exe nunit-console.exe foo.tests.dll //a foo

If you have multiple assemblies you want to report on, then they are separated by semi-colons.

ncover.console.exe nunit-console.exe foo.tests.dll //a foo;bar

Building this list up in a script dynamically can be a minor hassle - thankfully my NAnt and MSBuild tasks for NCover will take care of this for you. In fact, they allow you to use a physical file matching pattern to specify the assemblies to be profiled - the tasks then internally strip all the extraneous information and pass just what NCover expects.

Filed in:

Wednesday, April 18, 2007

NCover Problems / Fixes - Part 1

From time to time I try to chip in with support on the NCover forums, not only with NCoverExplorer issues but NCover as well since they are so dependent. There are a bunch of issues which come up repeatedly, particularly for people new to using NCover.

In a perfect world (future release?) NCover itself could offer some more helpful error messages, but where is the fun in that? For now at least it is more a case of deciphering the clues from the console output and coverage.log files.

Hopefully this series of posts may help a few people and reduce the times I end up repeating myself on the forums, so win-win if it works for you. My examples will use NCover 1.5.7. You should also take a look at the NCover FAQ which is included in your installation. I've got a copy here too if that helps...


Profiled process terminated. Profiler connection not established.

Easily the most common problem people stumble on. As shown below your application starts (in this case unit tests), but after a delay of 60 seconds or so you get this message.

>E:\apps\NCover\ncover.console.exe E:\apps\nunit\nunit-console.exe MyTests.dll
NCover.Console v1.5.7 - Code Coverage Analysis for .NET - http://ncover.org
Copyright (c) 2004-2006 Peter Waldschmidt

Command: E:\apps\nunit\nunit-console.exe
Command Args: "MyTests.dll"
Working Directory:
Assemblies:
Coverage Xml: Coverage.Xml
Coverage Log: Coverage.Log

Waiting for profiled application to connect...
NUnit version 2.2.8

Copyright (C) 2002-2003 James W. Newkirk, Michael C. Two, Alexei A. Vorontsov, Charlie Poole.
Copyright (C) 2000-2003 Philip Craig.
All Rights Reserved.

OS Version: Microsoft Windows NT 5.1.2600 Service Pack 2 .NET Version: 2.0.50727.42
.
Tests run: 1, Failures: 0, Not run: 0, Time: 0.016 seconds

Connected
Profiled process terminated. Profiler connection not established.


Most likely this is due to one of two reasons I know of:

(1) You are profiling a windows service or IIS - this feature has been reported as currently broken.

(2) You are profiling an executable - but Coverlib.dll was not COM registered

Unfortunately the windows service/IIS testing fell down the cracks for 1.5.7 for those of you trying to use this feature. You can monitor some of the threads on it here and here. Personally for my applications I remove the need to profile services at all. By making the windows service a super lightweight wrapper and moving all the implementation out into classes that are more easily tested there is no need to unit test what remains.

I appreciate however there are scenarios for people where they do want coverage of services for integration testing etc - all I can say for now is that Peter is on the case. Let's now focus on the more common scenario for people trying to profile executables and offer some background info.

NCover Basics

So what's that registration bit about then - surely NCover is a .NET application? Sure, some of it is - but the really clever bit of NCover is the C++ implementation of ICorProfiler in Coverlib.dll, which is a callback class for use with the .NET Profiler API. The requirements of the API are that the callback class has to be COM registered and it cannot be written in managed code. By using some environment variables set within NCover before launching your process, the .NET profiler is able to be "switched on" to make the API callbacks while your process is running.

This is the heart of NCover - it monitors the loading of AppDomains and modules, intercepts the JIT execution of methods and injects it's own instructions to monitor the execution counts. This data is then pumped back to the "managed code" part of NCover, which munges the data together to produce the coverage.xml output you know and love for NCoverExplorer to manipulate.

So the "profiler connection not established" message indicates that there was some sort of problem with communication between NCover and the .NET profiler. As illustrated with the windows service problem above this message doesn't always mean that it was caused by a registry problem, but many times it is and it's worth double-checking first.

NCover Installation / XCopy Deployment

If you installed NCover using the MSI installer, then it will have automatically COM registered CoverLib.dll for you in "C:\Program Files\NCover\". If however, you are like many people intending to run NCover using an xcopy style deployment mechanism as with other tools such as NUnit, then you must remember about this requirement.

So let's say you do want to relocate NCover to a tools subfolder in a source controlled directory - what's the idiot-proof way to do it?

Firstly, make sure you copy all the NCover files from the installation folder above. Make sure you don't skip the Microsoft.VC80.CRT.manifest, MSVCP80.dll or MSVCR80.dll files. If you do skip them, then an attempt to register Coverlib.dll will fail on machines without these .dlls already in a path somewhere. You may find it works on your development machine with VS.Net installed, but then fails on your build server.

Secondly, make sure you run the NCover uninstall which will clean up the registry as well as the installed files. So now your machine should be back in "pristine" condition.

Options For Registering NCover

So having identified that when running NCover.Console.exe you need to ensure that Coverlib.dll is registered - what are your options?

1. If you stuck with the default NCover installation, have no need for locating it elsewhere and need only one version of NCover for all your applications. Congratulations - you shouldnt have seen the problem above! If you did, you should manually re-register it using regsvr32 or try running the installer again.

2. The NAnt/MSBuild tasks I provide automatically register and unregister Coverlib.dll on the fly under HKCU. So if using those you are already sorted.

3. If using the command line, NCover 1.5.7 now offers a //reg option. This will do the same as the tasks above of registering and unregistering on the fly.

4. You could invoke regsvr32 on it as part of your script. I suggest you unregister it as well afterwards in case you have multiple NCover versions on your machine to prevent any later confusion.

NCover With TestDriven.NET

I got myself caught out with this little gotcha recently and I'm sure I won't be the last!

As you likely know the excellent TestDriven.Net add-in by Jamie Cansdale includes a copy of (currently) NCover 1.3.3 and NCover 1.5.7 bundled with it. What you probably don't know is under which scenarios it runs which version of NCover and where this can go wrong.

The first thing TestDriven.Net does is check the registry to see if you have installed NCover yourself using the MSI installer. If there isn't then it knows to use one of the versions it has installed. If you have .NET 2.0 installed on your machine it will use the 1.5.x version, otherwise it will use NCover 1.3.3.

If however you do have NCover installed from the MSI, then TestDriven.NET assumes that you want to use that version. Jamie's rationale for this is simple - if you ended up with later versions of NCover on your machine than that supplied with TestDriven.NET then it should use that over it's own.

Where this can go wrong as it did for me is if for some reason you have unregistered NCover in that folder. TestDriven.NET makes the reasonable assumption that if you installed it and left it there then you have it registered there too. However as I dabble with testing and supporting multiple NCover versions all over my drives I had manually unregistered it without remembering to uninstall it. That mistake will then lead to error messages from TestDriven.NET about "Unable to create a coverage report". Either fully uninstall that NCover installation or register CoverLib.dll again in that folder to fix this. My thanks to Jamie for his excellent support!

TypeMock And NCover NAnt/MSBuild Tasks

TypeMock is a very powerful mock objects framework which no doubt some of you are using. Assuming you have purchased the commercial version then you will likely want to be running your tests with code coverage using NCover.

The one gotcha with this is that a requirement of the .NET profiling API is that there can only be one profiler callback registered on a machine when it runs. TypeMock takes care of this by managing the registry itself. However if you use my NAnt/MSBuild tasks then by default they want to "undo" the work of TypeMock by registering NCover, causing a problem.

The solution is available from the 1.3.6.11 build of my tasks in NCoverExplorer.Extras, the latest of which is available here. By adding a registerProfiler="false" attribute to your NAnt task or equivalent for MSBuild the default behaviour of the tasks to register in HKCU is turned off and control is able to be maintained by TypeMock.

That's enough for today. Part 2 will cover the mysterious blank coverage.xml file. Feedback/corrections welcomed - if you have an NCover problem your best bet is to head to the NCover forums.

Filed in:

Friday, April 06, 2007

NCoverExplorer... v1.3.6

Some of you may already have noticed the long awaited release of NCover 1.5.7 yesterday - hoorah! That now being out of the way and in conjunction with some easing of my day job pressure, has made it a good time for me to finally rollout an NCoverExplorer update.

Note - if you have downloaded NCoverExplorer or TestDriven.Net over the last week or so you may want to do so again. I have been sneaking in fixes for issues right up until this blog posting. Check the downloads page to make sure you have the very latest build number.

This is predominantly a bug-fix release, addressing a range of issues that have accumulated over the last six months. Of utmost importance was improving the coverage result accuracy from either the merging of multiple coverage.xml files or multiple AppDomains within a single file. Some of you have experienced issues with the "optimisations" of removing noop sequence points that NCover 1.5.4, 1.5.5 and 1.5.7 apply in different ways. This resulted in duplicated nodes in the treeview, or lower than expected coverage with brackets being shown as not hit etc. My thanks to everyone who contributed to the testing of this with their coverage files/repro projects and patience while I got the kinks out.

On the new feature front, the biggest is a "Find Type" dialog (ctrl+F) allowing you to navigate to types by entering their name (supports * for a wildcard).


A change to previous behaviour is to the fail on minimum coverage option for NCoverExplorer.Console. Based on feedback this has now been changed so that any assembly not reaching the threshold will cause the build to fail, rather than total coverage.

All the other tweaks and fixes can be found in the release notes.

I have updated the NCoverExplorer Extras download containing the NAnt and MSBuild tasks. The <ncover> task now uses a reference counting style approach to the registration of coverlib.dll, which should get around issues reported on build servers of simultaneous build issues if you got unlucky with the timings.

The task now includes a registerProfiler optional attribute for those of you using TypeMock - you can set this to "false" to ensure the NCover task does not try to register CoverLib.dll itself.

There is an enhancement to the <nunitproject> task to support the appbase property for people using multiple testing folders (thanks to Garth Williams).

Another addition is a new <ndoc2> NAnt task in there for anyone using NDoc2 Alpha as I blogged about in this post. Documentation is available via the links on this page but anyone familiar with the <ndoc> task should be happy. I've used it for generating CHM documentation without any problems.

For any issues with the release or further enhancement suggestions, please post to the NCoverExplorer forums or send me an e-mail (Help -> Send Feedback).

A quick further mention on NCover 1.5.7. This is a mandatory upgrade for every NCover user as far as I am concerned. Peter Waldschmidt's major rewrite of the profiler core and the addition of an integration test suite sets up a great platform for future releases. I've been fortunate to be able to chip in here and there on this release, although arguably more nuisance than help at times to Peter I'm sure!

I am highly optimistic however that this is the most solid NCover beta to date and was worth the wait to get it right. Top stuff Peter and many thanks for all the effort!

Last but certainly not least, Jamie Cansdale has made available a new build of his excellent TestDriven.Net add-in. Amongst other goodies this now includes the latest build of NCoverExplorer, NCover 1.5.7 and Visual Studio Express SKU support. Right-clicky clicky goodness...

Downloads
Release Notes