What is your test ROI

When we write code we sometimes get carried away and forget to think about if we are spending our time right. However more common I see people (including myself) forgetting to think of cost vs benefit of the tests we write.

In the past I have been writing a lot of unit tests resulting in almost 100% code coverage. While code coverage itself is not a guarantee for quality it gives you an idea of all the cases I needed to cover to get there. Anyway, most of the time it wasn't even hard to get to 100%. But I started to notice that a lot of the tests covered simple error cases like null arguments. A null argument test takes only a few seconds to write, but is it really worth it?

Since I always use StyleCop and FxCop I already get warnings when I compile if I don't check my arguments for null so in practice the tests are not adding any value. And all those null tests together takes up a few minutes of my time so I actually started to be more critical about the tests I wrote. Rather than going for the easy things that give me high coverage with little or no value I started to spend that time on writing more tests covering important behaviors of my code. All in all I spend more time on my tests today than a year ago but the biggest difference is the quality of the tests.

When I reflected on this I realized how easy it is to get a sense of completion when you achieve something measurable like high code coverage. And it doesn't matter how well you know that that measure is not really worth anything because you still achieved something.

Interestingly enough this is something I often see when people write functional tests. I've often seen a tendency to favor lots of automated functional tests rather than good and stable functional tests. The result being the team spending a lot of time investigating tests that are failing randomly. I think there is more value in having a smaller set of reliable tests rather than having more tests for the sake of having tests.

So bottom line; don't just write tests the way you always have. Always question yourself if the test you are about to write really is worth writing or if there is another test you should be spending your time on.