Continuous Testing In .NET

Since the article was published, there have been several new continuous testing tools enter the market. I recommend checking out NCrunch, which is in my opinion the best continuous testing tool available for .NET – Tim 26/10/2012

The Test-Driven Development cycle of red-green-refactor gets you into a rhythm of writing a failing test, making it pass and then refactoring. It is important that we get feedback early if something is broken so we can keep this rhythm going. Stopping to run tests and wait for the result can impact this rhythm and distract our focus on the next step.

The concept of continuous testing came from research carried out by the Program Analysis Group at MIT. They found that continuously running tests increased developer productivity and reduced waste. You can find more information about continuous testing in this blog post by Ben Rady.

Tools For Continuous Testing

There are a number of tools available that support continuous testing. Ruby has AutoSpec, a command-line continuous testing tool. Java has Infinitest, a plug-in for Ecplise and IntelliJ. In .NET, James Avery has been working on AutoTest.NET and I was involved in a project to write a Visual Studio add-in called QuickTest. The main problem I have found with using these tools for .NET development is the complexity of real-world project structures. Whenever I’d try to use QuickTest on real projects, I found I was dealing with very different project structures, test configurations and naming conventions. It was really difficult to factor all these configuration scenarios into a tool that is designed to “discover” which unit tests to continuously run.

That gave me the idea for AutoBuild.

Introducing AutoBuild

AutoBuild is a continuous testing tool for .NET that runs a NAnt script whenever a file is saved. You simply tell AutoBuild to watch a particular folder and give it a simple NAnt script to run whenever a file changes. The advantage of this approach is that all the complexity in customising a continuous testing tool for your project is contained within the NAnt script. The NAnt script is likely to be a small subset of an actual build script. For example, you might not want tests that hit the database to be run on every save, so you could add a “DatabaseTest” attribute to your test and configure the NAnt script to ignore all tests with that attribute.

If you aren’t familiar with NAnt, don’t worry. A default build file is provided and you only need to modify the solution path and unit test assembly path properties.

Currently, the test failure output in AutoBuild only supports the NUnit task. I plan to add support for other unit test frameworks in the near future. I am also interested in adding support for Rake build scripts.

Using AutoBuild

You can find the source code for AutoBuild here:
http://code.google.com/p/autobuildtool/

To get started, get the source and run the default.build.cmd file. This will run the build and create an output folder that contains the AutoBuild assemblies, along with a sample build script and command file. You can copy these to your own project and customise the build file for your solution.

To run the “dog food” sample, simply run the autobuild.cmd file in the root AutoBuild folder (not the one in the output folder). This will open AutoBuild and start watching the source directory. Then open the AutoBuild solution and modify any .cs file. This should kick-off the build and run the tests. Mess around with the code to watch the build fail. Note that this autobuild.build script is configured to only run tests not marked with “integration”.

AutoBuild will produce an output with build errors and failing tests in red and successful builds and test runs in green. You can change the output colours by modifying the log4net settings in the AutoBuild.Console.exe.config file. You can also switch on debug output by changing the log level to debug.

autobuild

AutoBuild is ideally suited to a dual-monitor set up, with Visual Studio in one monitor and AutoBuild in the other. This allows you to focus on writing code and check the status of the build at a glance. I would like to integrate the console into Visual Studio for cases when you only have one monitor (e.g. working on a laptop). This might involve writing a Visual Studio add-in that allows you to load the console into the IDE.

Please submit any issues to the project on Google Code. If you have any comments or suggestions, then please feel free to contact me.

12 Responses to “Continuous Testing In .NET”


  1. 1 Tobin Harris August 6, 2009 at 8:29 am

    I miss not having AutoTest in .NET, so I’m looking forward to trying this, nice work!

    It looks cool, and peeking at the code I’m assuming that this could be wired up to .exe if you don’t use NAnt?

    Finally, you could tie in Growl for windows into this to have it automatically display when tests fail: http://www.growlforwindows.com/gfw/

  2. 2 timross August 6, 2009 at 11:16 am

    Hi Tobin, thanks for the link to Growl. That looks like it will do the job nicely! Yes, AutoBuild is just using Nant.exe, so you could reference another .exe or create an implementation of IBuildRunner to support another build tool. As I mentioned in my post, I plan to add support for running Rake scripts.

  3. 3 Tobin Harris August 6, 2009 at 12:03 pm

    Yeah, Growl has a fairly simple .NET API, enjoy :)

    Rake support would be ace.

    We’re working on Golem – a project automation tool like Rake for .NET

    http://github.com/tobinharris/golem/tree/master

    We’re using it here at Engine Room, but it’s far from perfect. I’d be interested to create a Golem task for running your auto-tests.

  4. 4 nappisite August 6, 2009 at 1:11 pm

    What niche does Autobuild fill that isn’t already covered by CruiseControl.NET or TeamCity, among others?

  5. 5 timross August 6, 2009 at 1:34 pm

    Hi nappisite, thanks for your question. The purpose of AutoBuild is to provide a flexibile way to support continuous testing. This is achieved by building and running your tests whenever you save a file. Unlike CruiseControl.NET and TeamCity, etc, AutoBuild is not meant to provide continuous integration across a team. AutoBuild runs on your local machine and is designed to give you instant feedback on whether tests are passing or failing, as you code. I hope this answers your quesiton.

  6. 6 timross August 6, 2009 at 1:41 pm

    Hi Tobin, Golem looks great, nice work! Where do you see Golem integrating with AutoBuild? Would you have a Golem task that runs AutoBuild, or would AutoBuild support running Golem tasks? I would definitely be interested in supporting some kind of integration with Golem.

  7. 7 Tobin Harris August 6, 2009 at 4:51 pm

    Right now I’d be thinking of

    c:\Code\Blah> golem test:auto

    AutoTest is a go!…

    That’s mainly because I use golem for loads of stuff and it provides a single entry point to many dev-related tasks (preparing demo data, NDepend report generation, db drop/create, project-specific utility methods etc).

  8. 8 Paul Harrington September 23, 2009 at 4:09 pm

    Hey Tim, still no joy with Autobuild on my machine – have you seen this kind of thing before?

    2009-09-23 17:07:55,792 [1] NAnt executable path, C:\Projects\Trunk\Insurance\Se
    rvice\Tools\output\bin\NAnt\NAnt.exe
    2009-09-23 17:07:55,803 [1] Watching path C:\Projects\Trunk\Insurance\Service
    2009-09-23 17:08:00,494 [3] File Changed Domain\Requests\Nodes\CDLHomeInsurance\
    CDLHomeInsuranceRequestBuilder.cs
    2009-09-23 17:08:06,644 [1] Build started
    2009-09-23 17:08:06,655 [1] Buildfile: file:///C:/Projects/Trunk/Insurance/Servi
    ce/Tools/autobuild.build
    2009-09-23 17:08:06,656 [1] Target framework: Microsoft .NET Framework 3.5
    2009-09-23 17:08:06,656 [1] Base Directory: C:\Projects\Trunk\Insurance\Service\
    Tools.
    2009-09-23 17:08:06,656 [1] Target(s) specified: run
    2009-09-23 17:08:06,656 [1]
    2009-09-23 17:08:06,665 [1] Build failed

  9. 9 Tim Ross September 23, 2009 at 4:29 pm

    Hi Paul, what happens when you run the autobuild.build file directly using NAnt?

  10. 10 GenMX April 20, 2011 at 3:49 pm

    You always post a good and interesting article. I am going to save your page for the next time.


  1. 1 Reflective Perspective - Chris Alcock » The Morning Brew #406 Trackback on August 6, 2009 at 6:59 am
  2. 2 Blogs From The Geeks » Blog Archive » Continuous TDD with Autobuild .Net - Daily insightful nuggets Trackback on September 2, 2009 at 1:56 pm

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s





Follow

Get every new post delivered to your Inbox.

%d bloggers like this: