Why is testing so often a bottleneck?  Silly question really – it’s that way because we make it so!

In my previous post, I described a situation in which the testing bottleneck revealed by our Kanban board and CFD was more imagined than real: the work was getting done but completion criteria were inadequate; some clarity and some WIP limits soon fixed things.

Addressing mistakes like that early on in a project is easy, but here are a just a few of the ways in which we have contrived to make life on our projects difficult for ourselves over the years:

1. “Hey guys, we want to do a release soon.  Can everyone please do some testing?”
2. Analysis (and some design), development, testing and rollout in syncronised, pipelined phases
3. An impressively large automated test suite with a hopeless failure rate

Let’s look at these from a lead time perspective.

1. “Can everyone please do some testing?”, or sequential phases

Glossing over the apparent cluelessness, there are problems here common to any approach that has the same set of developers doing a large amount of development (over several months in this case), then downing their IDEs to start testing.  Features developed at the start of the development phase (including some of the most important ones) are already several months old before testing begins in earnest.  This adds a significant context switch overhead even where the original developer is now doing the testing (not something that can be guaranteed over these timeframes), added to which is the worry that the foundations on which that feature was built have since changed.  And those foundations aren’t merely technical – the business context in which the feature was conceived and development will have moved on too.  In other words, the wastefulness of long lead times is compounded by change, both planned and unplanned.

So if sequential phases are a problem, how about doing them in parallel?

2. Syncronised, pipelined phases

This is an approach that may seem attractive to teams that have the luxury of dedicated analysts, testers and release managers.  Why not let each team keep busy on what they do best and allow analysis, development, testing and rollout to happen concurrently?  So while we’re rolling out release 1.1 (on top of release 1.0), we can be testing 1.2, developing 1.3 and readying ourselves for 1.4.  Efficient, right?

Efficient it might be, but from a lead time perspective it’s dangerous.  What happens when one of these activities slips?  The others either stall or take on extra work to fill the slack period.  Taking the latter, “efficient” route adds to the workload of subsequent phases, thereby making the longer cycle the new normal and a descent to low performance begins.

I should mention that I have seen this pattern twice now: the first time it was entered into quite deliberately and lived with for a long time; the second time it was the consequence of allowing testing activities to spill over from one short development sprint to the next.  Those downward descents are so much easier to enter than to leave!

3. Automation (hopeless or otherwise)

However good its coverage, a test suite with an 85% pass rate is a burden, not a help; any new failure is lost in the noise.  Taking that figure to 99.something% was a challenging but worthwhile exercise, but I now take a hard line and insist on 100%, all the time.  “All the time” means that a failure becomes (in Lean-speak) a “stop the line” event.  Furthermore, the depth of the test suite needs to be such that the focus of the team’s manual testing activities can move from repetitive regression testing to focussed learning about the system, warts-and-all (“exploratory testing” in testing-speak).

Pulling it together

Recap: long, sequential phases are bad, whether syncronised or not.  Short cycles are better but aren’t immune to problems, especially if testing can’t be completed right away.  Automation is a help, except when it isn’t.

So: test as soon as features are developed, both by adding to the automated suite and via serious exploratory testing.  Make the regression test a non-event by running it all the time; stop the line when there’s a problem.  In short: practice Continuous Integration (CI) seriously, make learning part of the process, go for flow!