I have been bit more than once when developing an application where all of my tests ran green yet the application still failed. This situation is inherent in basic Code Unit Test First methodology when you gloss over the details. I have met many developers that will recite “Write a unit test for the desired new capability. Write the code for the functionality. Run the test and if it succeeds then you’re done.” but will miss the very important “run the test first before the functionality code is written and make sure that it fails”. (Never Writea Line Of Code Withouta Failing Test)
But ensuring that the test fails without the desired functionality does not ensure that the test will fail when the desired functionality is broken in some other manner. Let’s look at an example:
When a particular piece of functionality is missing an exception is thrown. A unit test is written to explicitly check for this thrown exception and fail when it is found. When the new functionality is added the exception is not thrown and the test succeeds.
Is this particular unit test meaningful or useful? If a common failure mode is known to be that the functionality is not present (say, that the functionality is loaded dynamically at runtime) then the test may be deemed useful. If the functionality’s presence can be determined by other means such as compilation errors, then the test is not particularly useful.
What is important to note in this case is that other failure modes of the functionality must be exploited. Enter negative testing. Negative testing (based on the methodology that you follow) is “showing an error when not supposed to and not showing an error when supposed to” or “showing that software will fail and that the failure is handled in a specified manner”. Rather than having a lengthly
posting about negative testing, the paper entitled A Positive View of Negative Testing is a great read. Understanding the subtleties of positive and negative testing is important to ensure that your unit tests are providing the sturdy safety net that you’re relying on.