Monday, 28 March 2011

Acceptance tests make big claims

If we have good acceptance tests, why do we need unit tests?

Acceptance tests tell us a lot. When they pass, they tell us things like:
  • A feature works from an end-user perspective.
  • All the components required to serve user requests have been wired up correctly. 
  • The supporting infrastructure has been installed and configured in the test environment.
  • The user interface has not significantly changed since the acceptance tests were first written.
  • An intermittant network outage did not occur.
But when they fail, they tell us very little. They make such ambitious claims about the state of the software that failures could indicate a large number of disparate underlying issues.

Unit tests, on the other hand, claim very little. If a unit test fails, it is immediately obvious that there is a problem with a very specific section of code.

In general, the easier it is for a test to fail, the less it tells us when it does. Failures from tests that intermittently break for no reason at all aren't very informative. But if there's a little red dot next to that test that "couldn't possibly fail" then you've just discovered something startlingly new about your system.

Relying on acceptance tests makes sense if the code-base is relatively static, because failures will be relatively rare. But software that must change to remain valuable (i.e. almost all software) also needs fine-grained unit tests to give specific and actionable feedback on failures.

    No comments:

    Post a Comment