Since the article was published, there have been several new continuous testing tools enter the market. I recommend checking out NCrunch, which is in my opinion the best continuous testing tool available for .NET – Tim 26/10/2012
The Test-Driven Development cycle of red-green-refactor gets you into a rhythm of writing a failing test, making it pass and then refactoring. It is important that we get feedback early if something is broken so we can keep this rhythm going. Stopping to run tests and wait for the result can impact this rhythm and distract our focus on the next step.
The concept of continuous testing came from research carried out by the Program Analysis Group at MIT. They found that continuously running tests increased developer productivity and reduced waste. You can find more information about continuous testing in this blog post by Ben Rady.
Tools For Continuous Testing
There are a number of tools available that support continuous testing. Ruby has AutoSpec, a command-line continuous testing tool. Java has Infinitest, a plug-in for Ecplise and IntelliJ. In .NET, James Avery has been working on AutoTest.NET and I was involved in a project to write a Visual Studio add-in called QuickTest. The main problem I have found with using these tools for .NET development is the complexity of real-world project structures. Whenever I’d try to use QuickTest on real projects, I found I was dealing with very different project structures, test configurations and naming conventions. It was really difficult to factor all these configuration scenarios into a tool that is designed to “discover” which unit tests to continuously run.
That gave me the idea for AutoBuild.
AutoBuild is a continuous testing tool for .NET that runs a NAnt script whenever a file is saved. You simply tell AutoBuild to watch a particular folder and give it a simple NAnt script to run whenever a file changes. The advantage of this approach is that all the complexity in customising a continuous testing tool for your project is contained within the NAnt script. The NAnt script is likely to be a small subset of an actual build script. For example, you might not want tests that hit the database to be run on every save, so you could add a “DatabaseTest” attribute to your test and configure the NAnt script to ignore all tests with that attribute.
If you aren’t familiar with NAnt, don’t worry. A default build file is provided and you only need to modify the solution path and unit test assembly path properties.
Currently, the test failure output in AutoBuild only supports the NUnit task. I plan to add support for other unit test frameworks in the near future. I am also interested in adding support for Rake build scripts.
You can find the source code for AutoBuild here:
To get started, get the source and run the default.build.cmd file. This will run the build and create an output folder that contains the AutoBuild assemblies, along with a sample build script and command file. You can copy these to your own project and customise the build file for your solution.
To run the “dog food” sample, simply run the autobuild.cmd file in the root AutoBuild folder (not the one in the output folder). This will open AutoBuild and start watching the source directory. Then open the AutoBuild solution and modify any .cs file. This should kick-off the build and run the tests. Mess around with the code to watch the build fail. Note that this autobuild.build script is configured to only run tests not marked with “integration”.
AutoBuild will produce an output with build errors and failing tests in red and successful builds and test runs in green. You can change the output colours by modifying the log4net settings in the AutoBuild.Console.exe.config file. You can also switch on debug output by changing the log level to debug.
AutoBuild is ideally suited to a dual-monitor set up, with Visual Studio in one monitor and AutoBuild in the other. This allows you to focus on writing code and check the status of the build at a glance. I would like to integrate the console into Visual Studio for cases when you only have one monitor (e.g. working on a laptop). This might involve writing a Visual Studio add-in that allows you to load the console into the IDE.