Continuous Integration with Hudson, ColdFusion, and MXUnit: Failing builds for failing tests

Saturday, August 29, 2009

Randy posted this question to the MXUnit group, and I wanted to expand a bit on it with a full code sample. His question:
"My question now is about the proper way to fail a unit test. Before I was just using haltonerror attribute to stop my unit tests. But now with Hudson I found I can't do that because I won't get the error logs. So, my question is, what is the 'best' way to fail the build when you are using it as a depends for a commit and still get the unit report that Hudson is looking for?"
By default, if you don't set MXUnit task's "haltonerror" property to true, then all tests will run, and if you pass JUnit XML to Hudson, Hudson will read it, see that tests failed (but the build didn't fail), and it'll record the build as a "warning". Here's what a warning looks like in Hudson. Note the link to “Test Result”, which you can drill down into to get all the details.
But if you use haltonerror, the MXUnit task essentially does a cfthrow, and tells the build to stop right now; consequently the JUnit task won’t run, so you get no useful messages about why the build failed. What Randy wants is to retain the ability to “fail” the build, but also to get the JUnit report messages. Here’s what happens if haltonerror is set to true. Note no link for Test Results. With a build like this, all you know is that “it failed”, but you do not know why. This is useless.

It’s time to talk about this notion of “failing” the build.  What Randy wants to do is to stop other things from happening if his tests fail. For example, maybe he wants to stop the process that FTPs the bits to the prod server; or perhaps he wants to not package the app for deployment if the tests fail. In these cases, haltonerror is usually a fine way to go. But since he’s doing continuous integration, and he wants proper recording of all that happens, he can’t do that. So what’s a boy to do?
Simple. But first, let’s look at a real-world task that runs tests, runs the JUnit report, and then emails stuff. I’ll use MXUnit’s own build file as an example:
<target name="runTests" depends="init,update">

<!-- load the current version of the package-summary file; this way, we can put it at the end of the email we send so that
comparing test the aggregate test results is easy; makes it simpler to see if any test failures crept in -->
<loadfile property="currentresultshtml" srcFile="${junit.out.dir.html}/mxunit/tests/package-summary.html" failonerror="false" />

<delete dir="${junit.out.dir.html}" />
<delete dir="${output.dir}" />
<mkdir dir="${junit.out.dir.html}" />
<mkdir dir="${output.dir}" />

<!-- get the server, port, and webroot properties from the antrunner file; this way
each developer can have their own version of the properties file with different
values depending on their setup -->
<property file="${antrunner.file}" />

<mxunittask server="${server}" port="${port}" defaultrunner="/mxunit/runner/HttpAntRunner.cfc" outputdir="${output.dir}" verbose="true" testResultsSummary="my.summary">
<directory remoteMethod="run" path="${webroot}\mxunit\tests\framework" packageName="mxunit.tests.framework" componentPath="mxunit.tests.framework" recurse="false" includes="*.cfc" excludes="" />
<directory remoteMethod="run" path="${webroot}\mxunit\tests\runner" packageName="mxunit.tests.runner" componentPath="mxunit.tests.runner" recurse="false" includes="*.cfc" excludes="" />

<!-- create nice pretty report -->
<junitreport todir="${junit.out.dir.html}">
<fileset dir="${output.dir}">
<include name="*.xml" />
<report format="frames" todir="${junit.out.dir.html}" styledir="${style.dir}" />

<!-- Read the properties files we generated 
<property file="${output.dir}/my.summary" />
<echoproperties />

<!-- now load the version we just created; this will be the first set of results in the email we send -->
<loadfile property="resultshtml" srcFile="${junit.out.dir.html}/mxunit/tests/package-summary.html" />
<loadfile property="resultscss" srcFile="${junit.out.dir.html}/stylesheet.css" />

<!—other stuff omitted for brevity -->

The important thing here is that no matter what, all the steps after the MXUnit task will run since the haltonerror attribute isn’t set. So how do I get all that stuff to run, but then fail the build  if tests fail (or error)?

1) You set the failureproperty attribute on the MXUnit task

2) You then call the ant built-in <fail> task, with a conditional.

So, in my example above, here’s what I’d do:

<mxunittask …[omitted for brevity] testResultsSummary="my.summary" failureproperty="testsfailed">




<fail if="testsfailed" message="Failing the build because tests failed"/>

If tests fail (or error), the MXUnit ant task will set the property named “testsfailed”. Then, the ant fail task will run because if=”testsfailed” will be true, since the “if” statement of an ant task runs if the property is set, not whether the property itself evaluates to true. Thus, with these two small changes to the build file, we get all the reporting we want via JUnit report but also a failure that prevents further processing.

Here’s the “overview” result in Hudson. Note that it has the data for failed tests, which we can then drill down into to get the goods.



Sean Corfield said...

Philosophically how do you feel about a build that is just unstable due to failed MXUnit tests vs a build that fails due to those tests?

I'm working on a project where we have several known failures that are in things that wouldn't stop the deployment. Is that good or bad?

Marc Esher said...

Sean, at work, I don't have the builds failing due to test failures. I have 5 projects set up, and all of them simply go to Unstable when tests fail.

One project is in a constant state of instability. It's just one of those "yeah, we know" kind of things.

I think this is definitely a case of "do what's right for your project".

For MXUnit, we have a "deliverable", and in that case I think it's probably not appropriate to do a push unless all tests pass; consequently, for that project, I think it's correct to fail the build if the tests fail. But at work, our deployment process is more complicated and perhaps more "nuanced" than that. For better or worse, there's a lot more flux and churn in our work projects and more "partial" deployments. If a test is failing but isn't part of a given deployment (which is entirely a human decision), then I don't want the build system to get in my way.

I would say that in my head, in my work projects, "Unstable" is just as bad as "Failed", but the difference is that it doesn't prevent further things from happening (deployables being built, etc).

What do you think?

Sean Corfield said...

Thanx for the elaboration. That makes a lot of sense. Right now I think we're in the "yeah, some tests fail... yawn!" phase as we're still developing and don't have a specific deployment artifact created. Later on in the project I believe that will change and we'll want test failures to fail the build.

Sean Corfield said...

I wanted to follow up on this (now that we're further along in the project). We ran into a problem (today) where the build succeeding causes developers to miss the fact the tests failed. However, we like the "unstable build" approach of Hudson. So we changed our targets so that Hudson runs a low-level target to run the tests but manually we run a higher-level target that fails the build if the tests fail. That way we get the best of both worlds!

Marc Esher said...

Sean, does that mean you don't send notifications on "unstable build", but only on build failure?

Sean Corfield said...

Yes, we do send emails on unstable builds but what was happening was developers would run the build locally with all the tests and not realize they'd introduced bugs because the build seemed to succeed. So we want the build to fail locally if any test fails but pass (unstable) on Hudson.

ciaranarcher said...

Thanks Marc - was looking for this step!