Thinking about levels of Risk and Releases

One aspect of releases in the real world that seems to get little mention is the reality of working on more than one of them at a time. Many of us work in shops with a main branch and then multiple named releases hanging from various places off of the main branch. This is standard.

So, the inconvenient truth is that when I make a fix in release 2.3 it probably needs to also go in release 2.3.1, release 2.4 release 3.0 and "main" (whatever you call your root branch). I tend to think of those as pass through integrations. Someone made a fix in a past release, likely for a real customer (so it was done quickly) and then wanted to make sure it got propagated to all of the other releases that might correspond to any existing or future customers.

Now, the awkward question is: do you think the developer ran a full regression test on every release they checked the code into? Officially we all probably say yes, but in reality the answer is "I made sure it compiles" if you're lucky.

Depending on your branching strategy and the level of automation of your regression tests you may not need all builds deployable at all times or you may simply not be able to afford it even if you want it. If your tests are not automated it may simply be too expensive for each checkin to cost a developer-day of regression testing.

So, the two key questions are: how much risk do you have and how much can you afford?

A proposed approach to dealing with these questions is as follows. Each time a developer makes a check in to a branch the risk that the branch has destabilized has increased.by a quantum. Each time QA runs the regression suite the risk is reduced. Make the invalid but simplifying assumptions that all checkins add a constant risk and the regression suite reduces the risk to zero.

Imagine a graphical tool that shows a graph of the accumulated risk associated with each branch. This would allow you to answer the question of how much risk you have with each branch. Put another way, it lets you know how much deferred work would be required should any particular branch need to be released.

The Atlassian tool suite might already do this but I suspect that any development shop with a code management system and decent script writer could create a system for building such graphs.

Once you have the ability to measure your risk you can make an informed decision about how much to accept. You might decide that some branches should be regression tested everytime they’ve accumulated x checkins, while other branches only need testing every y checkins. The point is that you get to decide.