Continuous Integration Best practices: #4 Never go home on a broken build

Manuel de la Peña

July 19, 2013

A Minute Read

I love that picture! Imagine yourself any Friday, at the end of your day-work. You look at the CI server and, unluckily, it is broken. You have only three options:

Resign yourself that you will be leaving late, because you'll try to fix it.
Revert you changes and retry next week.
Leave now and leave the build broken.

Of course, the best choices are to choose number 1 or number 2, never number 3. In the above picture, Iron-man decided that he doesn't mind what will happend after that "bomb". But why is it a bomb to leave the build broken?

It's a bomb, because any co-worker that pulls from your master branch will get dirty code. And what happens with this? Please look back to my first post about don't check-in on a broken build.

On the other hand, if you try to solve, or even revert your changes before leaving, you will keep the build green, and other developers will be happy to pull safe code from SCM repository.

Some good practices to avoid potential problems are:

Check in frecuently and early enough to give yourself time to deal with problems should they occur
Experienced developers often save checks in for the next day

Well, at this point you could say: "Ok, I follow similar practices but my team is distributed and we have problems working in different time zones".

In Liferay we actually have this "problem":

As you can see, we work in different time zones: China, Europe and America, following the sun.

In case of problems:

If China breaks the build... then Europe day's work is dramatically affected
If Europe goes home on a broken build... America would be screaming and crying

How have we solved it? Well, at this point the figure of the Build Master appears:

This role not only mantain the build but also policed it, ensuring that whoever broke the build was working to fix it. If not, the build engineer would revert that check-in, so it's mandatory that the build engineer has write access to the master branch in your SCM, or prioritize his commits, otherwise.

The build master is a controversial role, because nobody wants to see his/her commits rolled back. But the whole team should accept that this is not a personal offense, it is another effort to improve the quality of the product, never criticizing the developer.

For that, we all should open our mind and accept that is not bad to revert someone's commits: they are still present in the history of the project, so we could restore them whenever we need them. And after all, those commit were breaking something, weren't they?

I will talk about reverting in the next episode, so let's see then!

3 Comments

Please sign in to comment.

AFAIK the "normal" CI process works the way that you describe (and also has the issues that you describe :-).
But I have a question (which I hope that makes sense): why not simply create a "pre" master branch where all check-ins will be done, and this "pre" master branch will be automagically synced with the "real" master branch *only* when the build is green?
So, all developers would only checkout clean code from the "real" master branch and would never be able to check-in dirty code into the "real" master branch. This way you would avoid many of the issues that you describe.

Please sign in to reply.

Hi Remis, thanks for your comment!

Unfortunately our scenario is more complex as we have different production versions to support. Not to mention that we have several products now which needs to be compatible with different versions of each other. So having a pre master branch for each version could be hell for us at this moment emoticon

But we are trying to do our best to improve that.

Cheers!

Please sign in to reply.

Absolutely! We called him the 'Build Barbarian' (he had a little hat with horns), but the same function.

A corollary to this rule is not to commit your stuff to master/trunk if you're going to go home before you see the build of your change go green! If you have to go now, commit your stuff to a branch, then on Monday morning merge it to master and watch it go emoticon

Please sign in to reply.