Software Quality

February 4, 2010

Code Contracts may reduce unit test isolation

Filed under: Uncategorized — David Allen @ 10:33 pm

If you use comprehensive automated testing and code contracts, you may have noticed something: sometimes, when a contract fails, it causes large numbers of tests to fail. This can seem overwhelming at first. And if you are accustomed to the ideal of unit test isolation, it may even seem to be a bad smell.   This ripple effect of a failure in one component is not unique to the use of contracts.  And ways to handle it are described well in several places. such as 

http://c2.com/cgi/wiki?UnitTestIsolation

http://martinfowler.com/articles/mocksArentStubs.html#TestIsolation

To summarize some key points from these articles

First, the fact that a single contract failure can trigger many test failures is merely an indication of the breadth of impact of a change.  It simply gives immediate and wide visibility to what your change has broken. As I said earlier, this ripple effect is not unique to the use of code contracts. But I mention it because code contracts sometimes magnify the effect because they catch failures IMMEDIATELY, even if you are not “testing” for them. This is caused by the fact that contract assertions are like built-in tests.

Second, diagnosing and repairing is not as bad as it seems. You may have many test failures. But if  you look at most of the failures, they will exhibit identical exception messages, indicating the contract failure, and where it is located. Once you find it, and fix it, the tests will pass.

Finally, as mentioned in the articles above, if you are making changes in small increments, you know where the breaking change is. It is the one that you just made since your tests last passed. And as a last resort, merely undo your last change, verify your tests pass, then start reapplying the last set of changes very slowly until you identify what you did that broke the system.

So don’t blame contracts because they reveal the full impact of what you broke. Just test often, and checkin often so you can revert if needed.

Leave a Comment »

No comments yet.

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Blog at WordPress.com.

%d bloggers like this: