Comments on MXUnit Blog: Continuous Integration with Hudson, ColdFusion, an...

Yes, we do send emails on unstable builds but what...

2009-12-11T00:15:13.589-05:00

Yes, we do send emails on unstable builds but what was happening was developers would run the build locally with all the tests and not realize they'd introduced bugs because the build seemed to succeed. So we want the build to fail locally if any test fails but pass (unstable) on Hudson.

Sean, does that mean you don't send notificati...

2009-12-10T16:27:50.620-05:00

Sean, does that mean you don't send notifications on "unstable build", but only on build failure?

I wanted to follow up on this (now that we're ...

2009-12-08T21:54:51.093-05:00

I wanted to follow up on this (now that we're further along in the project). We ran into a problem (today) where the build succeeding causes developers to miss the fact the tests failed. However, we like the "unstable build" approach of Hudson. So we changed our targets so that Hudson runs a low-level target to run the tests but manually we run a higher-level target that fails the build if the tests fail. That way we get the best of both worlds!

Thanx for the elaboration. That makes a lot of sen...

2009-08-31T21:29:28.808-04:00

Thanx for the elaboration. That makes a lot of sense. Right now I think we're in the "yeah, some tests fail... yawn!" phase as we're still developing and don't have a specific deployment artifact created. Later on in the project I believe that will change and we'll want test failures to fail the build.

Sean, at work, I don't have the builds failing...

2009-08-31T07:37:57.883-04:00

Sean, at work, I don't have the builds failing due to test failures. I have 5 projects set up, and all of them simply go to Unstable when tests fail.

One project is in a constant state of instability. It's just one of those "yeah, we know" kind of things.

I think this is definitely a case of "do what's right for your project".

For MXUnit, we have a "deliverable", and in that case I think it's probably not appropriate to do a push unless all tests pass; consequently, for that project, I think it's correct to fail the build if the tests fail. But at work, our deployment process is more complicated and perhaps more "nuanced" than that. For better or worse, there's a lot more flux and churn in our work projects and more "partial" deployments. If a test is failing but isn't part of a given deployment (which is entirely a human decision), then I don't want the build system to get in my way.

I would say that in my head, in my work projects, "Unstable" is just as bad as "Failed", but the difference is that it doesn't prevent further things from happening (deployables being built, etc).

What do you think?

Philosophically how do you feel about a build that...

2009-08-29T18:17:43.782-04:00

Philosophically how do you feel about a build that is just unstable due to failed MXUnit tests vs a build that fails due to those tests?

I'm working on a project where we have several known failures that are in things that wouldn't stop the deployment. Is that good or bad?