Software Quality

September 2, 2010

Why do we write acceptance tests before we code?

Filed under: Case Study, Practices, Testing — David Allen @ 6:18 am

Topic: Acceptance tests as requirements

Thesis: We should write acceptance tests before we accept a story for development. The story is not ready for coding until acceptance tests have been developed (or at least started). 

Assumptions:

You are using an agile method like Scrum or XP which has stories, which express potential business value requested by users. Stories are placeholders to start a conversation with users about their requirements.

By “acceptance tests” I mean tests written in collaboration with business users, in the language of business users, understandable by business users.  These are the tests, that if they pass, the feature is considered “done.”

Unit tests are tools for programming design and verification. They are focussed on components to be tested. If the components are modelled after the business, some of them may indeed be related to acceptance tests at a lower level.  They are a different topic.

Why do we want to write acceptance test before we consider a story ready for development? Consider two perspectives:

  • Good things that happen if we do.
  • Bad things that happen if we don’t:

Case Study of the bad things that happen

We got our requirements for a data conversion. They said “Take the data from the old file and put it in the new file.” That was it. A data architect interviewed a lot of people and came up with an enormous data mapping document (the requirements) which were written down in a giant spreadsheet. They called her work “done” so her contract was ended. We coded the data conversion. Six month later, users were trying to plan acceptance testing. So they test the results for the first time ever. A number of defects are immediately reported.  These are the requirements we were missing in the original assignment. We spent the next several weeks doing Defect Driven Development, which is NOT a method I am selling. It is my tongue-in-cheek take on what happens when you fail to provide decent requirements up front.  So acceptance testing was delayed because the code had to be rewritten. Effort was wasted writing the code originally to incomplete requirements.  But how do you know what decent requirements look like?  They look like acceptance tests. If they don’t include acceptance tests, they are incomplete requirements. I believe requirements can include many things, that vary with the story.  But to me, acceptance tests are an essential artifact. And collaboration with users to develop and understand them is essential.

Case study of good things that can happen

So, we learned our lesson. We coders started demanding acceptance tests, approved by the users, before we would start coding.  This required project management to involve users in the way they must be invovled.  If they could not produce a user to help write the acceptance tests, we would not code the story. We started using the number and complexity of the acceptance tests to inform our backlog estimates and later our sprint estimates.  The writing of these acceptance tests uncovered numerous ambiguities in requirements.  In some cases, we even delayed or cancelled stories because the users could not agree on acceptance tests.  It would have been a waste to code to ambiguous requirements, and make assumptions,  only to find at the end that the users were confused and unhappy.  Better to find that out up front and invest our scarce development on things our users KNOW they want. We wrote automated tests, expressing the acceptance tests.   This uncovered more surprises which we resolved.  When we turned over the code to testers, they loved it. They understood it. And we lived happily ever after.

Summary of Benefits of Having Acceptance Tests Before Coding

  • Users are forced to clarify what they want.
  • Coders understand what the goal is.
  • Writing acceptance tests first uncovers scope and complexity of feature, allowing a more realistic estimate.
  • Automating acceptance tests further uncovers ambiguities and technical challenges sooner, allowing quicker resolution and less rework.

Recommendations

  1. Acceptance tests MUST be established with users and QA testers BEFORE development starts. If they are not available, do NOT make assumptions about what is needed. Demand to collaborate with users and testers to obtain the acceptance tests and the understanding that goes with that collaboration.
  2. Acceptance tests should be used to inform estimation. More complex tests suggest more complex coding. After all, we WILL be coding these tests somehow, won’t we?

Questions

For the questions below, be clear that acceptance tests are NOT necessarily the same thing as unit tests.

  1. Do YOU require acceptance tests before you will start coding?
  2. Do you INSIST on participation by users in the development and validation of acceptance tests?
  3. Do you write automated tests of some sort, to express acceptance tests?

1 Comment »

  1. Hi David,

    Awesome post. I completely agree on the point where you mention that no assumptions should be made at all. I think on (of many) reasons for a project to fail is simply bad communication between the IT staff and clients.

    Cleary, acceptance tests help to reduce this misunderstanding and improve the chances of developing a common culture between IT people and clients.

    To your questions:
    1. Yes, most of the times we do. Excluded are small projects (like tools)
    2. Yes. Insist, but in a friendly way. If the customer is not available, then we wait until the next meeting.
    3. Most of the times: Yes. However, sometimes, the time pressure does not allow for a 100% coverage.

    Best regards,
    Martin

    Comment by angler — September 2, 2010 @ 9:43 am


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Blog at WordPress.com.

%d bloggers like this: