【推荐原文】 How Much Is Enough?

namisang

作者： James Bach
推荐理由：测试人员应当看看的故事
原文地址：http://www.satisfice.com/articles/how_much.shtml

A classic question asked about test strategy is "How much testing is enough?" If you're doing testing strictly from pre-scripted procedures, the answer may seem obvious: You've done enough testing when you've performed all the test procedures. But that answer is not worthy of a thoughtful tester. A thoughtful tester needs a way to answer the question that addresses the mission of testing, not merely its tactics. All the test procedures that currently exist might not be enough to satisfy the mission…or they may be more than needed.

Our mission is not to run a certain set of tests. For most of us, our mission is to learn enough important information about the product so that our clients (developers and managers, mainly) can make informed decisions about it.

Testing as Storytelling
When you test, you are doing something much like composing an investigative news story. It's a story about what you know about the product, why you think you know it, and what implications that knowledge has for the project. Everything you do in testing either comprises the story or helps you discover the story. You've done enough testing when you can tell a compelling story about your product that addresses the things that matter to your clients. Since your compelling story amounts to a prediction about how the product will be valued by its users, another way of saying this is that your testing is finished when you believe you can make a test report that will hold true over time—so try to write a classic.

For instance, I recently tested the install process of a complex product. My mission was to assess and catalog all the changes that this product made to systems on which it is installed. So my first step was to analyze the install process. Then I diagrammed it, chose test cases corresponding to important parts of that process, and found ways to perform those test cases under controlled and repeatable conditions. I came to a conclusion about this product that flowed logically from the tests, and then I checked the conclusion to be sure that each aspect of it was indeed corroborated and supported by the tests I performed. I needed this to be a good, compelling story, so I tried to anticipate how it could be criticized by “the reader.” Where is my story weak? How might my story turn out to be false? I ran additional tests to rule out other sources of system change. I ran tests multiple times to improve my confidence that the results I was seeing were related to the processes and variables I was controlling, and not coincidental events.

When I exhausted the concerns of my internal critic (and external critics I asked to review my work), I decided it was good enough.

A Short Story Can Be Just as Complete as a Novel
Testing is potentially an infinite process. If complete testing means you have to run all possible tests, you will never finish. But you can say you're done when you have a testing story with all the major plot points, and you can make the case that additional tests will not significantly change your story. Here's the thing: Although you never know for sure if you have reached that point of diminishing returns, you don't need to know for sure! All that's required, all that anyone can expect of you, is that you have a compelling story for why a thoughtful and responsible tester like you might come to the judgment that you know enough about the product under test. In some situations, that will be months of testing; in other situations, only hours.

Plot Points for a Testing Story
A complete testing story answers the questions: What was tested (coverage)? In what ways was it tested (techniques)? If problems occurred, how were they detected (evaluation)?

Whether my test strategy is sturdily scripted, or flowingly exploratory, the plot for my test reports has the same basic structure. It goes like this: Tester meets product. Tester learns product. Product wanders and changes; tester follows it. Tester is worried that product might fail. Tester checks product for possibility of anticipated failures using risk-oriented tests. Tester checks product for unanticipated failures using diversified and coverage-oriented tests. Tester has questions and notices problems and issues. Then, one day, the product settles down, no questions remain, issues are resolved, all known risks have been examined, and all important aspects of the product have been examined. Known problems are either fixed or accepted. Tester has a reasonably clear idea how users will react to the product. Tester lives happily ever after (and so, hopefully, does the product).

Writing a good testing story takes skill and care. You'll get it wrong, sometimes. Don't worry too much about being wrong. The main thing is that your story is thoughtful and well researched. Oh, and if you can make it a taut thriller, that won't hurt.

xiao小尾巴

慢慢体会....