Software Quality

October 22, 2011

Software Quality – What should you know today? What should you learn for the future?

Filed under: Code Contracts, Practices, Testing — David Allen @ 8:11 pm

How can we make good software? What are current best practices and what are some promising areas for future development? In this post, I share a couple of ideas that you can use now, and a few ideas to keep an eye on for the future.

Our current practices should include

  • automated unit testing
  • design by contract
  • Test-Driven Development (TDD)

Emerging ideas to watch include

  • automated test case generation
  • model-based verification

Automated Unit Testing

Automated unit testing (e.g. NUnit, JUnit, and Microsoft Test) has now gained enough popularity that not only can I recommend the practice, but I can expect that you have some idea what I’m talking about. Automated testing not only helps to produce better designs initially, but it detects regression as software evolves. See if you want further information.

Test-driven development (TDD)

Test-driven development (TDD) is the practice of writing tests before you write the code. Test driven development offers several benefits such as

  • resulting in improved API design
  • ensuring automated regression tests are created and not overlooked
  • uncovering missing or ambiguous requirements early

If you want to learn more, the following article by Scott ambler is a good explanation of TDD:

Since this practice is not nearly as well understood, I want to emphasize one aspect of test driven development that is often overlooked. Test driven development is most beneficial when you do it as a social activity. Let me describe my own style of development using TDD to illustrate. When I first receive an assignment, I request a meeting with the business analyst, a customer, and a professional tester. During this meeting, the business analyst and customer will give me the preliminary requirements. I will ask questions in an effort to clarify their goals. I may ask several questions that are slightly beyond the boundary of my immediate task to ensure that I have a broader context for the problem. I may also engage in a dialogue about possible design solutions at this time. But let me return to that point about clarifying the requirements. My most powerful tool is the following question: “When I’m finished with this code, how are you going to test it?” I do not consider the requirements good enough to start on unless I have several test cases that have been articulated and validated by all of the different specialists mentioned above. In an agile process like Scrum, these tests are called “Acceptance Tests.” But regardless of your process, the idea is the same.

Each of the different specialists I mentioned play an important role in ensuring the quality of the requirements. The customer is essential because they are the authority on what they want. The absence of a customer from the table is a cause for serious concern. It is a symptom of lack of commitment by the customer. This lack of commitment can easily jeopardize the entire project. It is naïve and arrogant of us as technical professionals to think that we can develop software that is good for the customer unless we have their sustained collaboration to keep us on track at every step.

The business analyst is essential because customers are not universally good at designing new software systems. They often express requirements that are vague or ambiguous. A skilled business analyst can recognize this and help ensure the requirements are complete enough, consistent enough, and clear enough to be understandable.

The professional tester is invaluable because they are skilled in thinking about boundary conditions as well as the practical matters of where we will get our sample test data and how we will approach the testing process. As a programmer who is getting ready to sit down at his computer in the next 30 min. and write a series of automated tests, I am extremely grateful for the participation of the professional tester.

With all these people at the table helping me to plan the test, we produce much better test plans. Depending on your process and approach, you may or may not try to get all of the test cases up front. But you definitely need enough test cases to give you an understanding of the real problem being addressed. Notice that my description of test driven development has a heavy emphasis on cross functional collaboration. It is about people helping people. It provides an opportunity for people with different skills to teach one another and learn from each other. The synergy that arises from this cross functional collaboration is indescribably powerful. And it is an awful lot of fun to.

Code Contracts

While automated unit testing is now widely known (if not widely used), most programmers do not understand software contracts. The idea behind “Design by Contract” was articulated by Dr. Bertrand Meyer as long ago as the 1980s. The idea has been available over 20 years, yet most of the programmers I work with have never heard of the concepts. The concept is important enough that Microsoft introduced a concrete implementation of the approach as a technology called Code Contracts; and they placed it into the 4.0 version of the .Net framework in 2010. If you are unfamiliar with “Design by contract” please see If you happen to be a Microsoft programmer and you want to learn more about code contracts, see

These two practices, automated unit testing, and the use of software contracts, are both well understood, and well-documented. But there are some emerging technologies and ideas you should be aware of for the future: 1) automated test case generation and 2) model-based development.

Emerging Practices

Automatic Test Case Generation

AutoTest is an automated test case generator that can also execute the automated tests it produces. This test case generator should not be confused with automated unit testing that merely exercises the code to test cases that are designed explicitly by the programmer. This tool and the practice of designing contract into the software are both discussed in a recent blog posting by Dr. Bertrand Meyer. You can read the post (“All Bugs Large and Small”) started by Dr. Meyer in the Eiffel technology community group on LinkedIn. See

Another automated test case generator is Pex, a product out of Microsoft Research. You can learn more about Pex at

I call AutoTest and Pex future ideas because they are not widely known yet, even though programmers are using both products successfully on commercial projects to improve the quality of their software today.

Automatic Model Generation and Verification

Another emerging technology we can use to improve code quality is called model-based development. Dr. Mats Heimdahl, the Director of the Software Engineering Center at the University of Minnesota, gave a talk on this at a seminar this morning, October 22, 2011. In his talk, Dr. Heimdahl talks about the many ways we know to ensure safety in safety critical systems. Dr. Heimdahl and his students have applied these techniques successfully in the aerospace industry and other safety critical areas.

That concludes my brief summary of some current best practices as well as some promising techniques that are emerging. If you are a software developer who does not routinely practiced automated unit testing and design by contract, I strongly encourage you to learn about these practices and incorporate them into your daily work of building software.

If you already follow these practices, and you are hungry to take your skills and our profession to the next level, and I can recommend the automated testing and verification techniques mentioned earlier as areas that have shown serious promise. These advanced techniques are currently being used by customers in real commercial and industrial setting, backed by commercial tool vendors and academic researchers.

Leave a Comment »

No comments yet.

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Blog at

%d bloggers like this: