A couple of weeks ago I had just read Dan North's article about how he saw six things he thought he would never see because they were impossible. I had no idea I would myself see the seventh impossible thing just a few days later.
Since I started to work at Microsoft I have several times tried to convince the testers on my team that it would be a good idea to write some unit tests for the functional and integration tests they have been writing. I have always been met with an argument that writing tests for tests is a stupid idea. Yet my experience have been that these untested functional and integration tests have been failing then easily 95% of the time it has been a bug in the test and not the code it is testing. So from my perspective writing unit tests for the tests would reduce the time spent on debugging tests...
You can imagine my surprise when I saw unit tests in a code review for some functional tests! I did a little happy dance and then got into the details. The unit tests were covering a number of helper classes used in the functional tests. In my opinion this is a no brainer. If you have a bunch of helper classes which complicated logic it makes sense to have some unit tests covering it. And my tester colleague confirmed that writing unit tests found some bugs...
Since I finally had somebody on the hook I sat down and talked to him a little about what he thought about adding unit tests for the actual functional tests too. We decided that while it sounds a little crazy it actually makes sense. Testing the logic of the functional tests with fakes ensures that the test does not have a bug meaning that functional test failures actually are functional failures! I am looking forward to the future!
There's an infinite stack of turtles outside that wants a word with you.
ReplyDeleteSeriously, though, if the test fails, you know *something* needs to be fixed, most often the SUT, but failing that, the test..
Hmm, though, what about the tests that *pass*? Are they passing incorrectly? Better write a... OW. Turtles bite!
To be clear; If you change the code and suddenly a test fails, yes it is typically the SUT that is to blame. And I don't think there is a problem with unit tests typically since unit tests by definition should be passing or failing consistently. However once you start doing large end to end tests there are so many things that can go wrong and it is this set of tests where I've seen occasionally failing (without a code change) and most of the time there is a problem in the test and not the SUT.
ReplyDelete