James Bach回答测友问题之四（How Many Times Should You Run a Test?）

skinapi

原始链接请见：
http://www.satisfice.com/blog/archives/67

Kevin asks: What is the best or industry standard for how many times a test case should be run?

There are questions that should not be answered. For instance, “What size unicorn do you wear?” or “How many cars should I own?” Sure, I could answer them, but the answers are worthless. My answers are A) I don’t wear unicorns and B) 2. In these cases, the more helpful reply is to question the question. For the first question, perhaps you said “uniform” and I misheard you. For the second question, perhaps you own a railroad and you were talking about train cars of different kinds, whereas I assumed you’re a small family and you were asking about automobiles.

I can tell you this for sure: No one I respect in the testing field will give you a direct answer to the general question of how many times a test should be run (except maybe as a joke).

Imagine if the answer was 100,000. Would you believe it? What if the answer was 7? Wouldn’t you wonder what was wrong with 6? I can imagine 7 being the right answer, but only for a very specific hypothetical case, not as any sort of general principle.

The first potentially useful answer I have is to tell you that this question would not even occur to you if you knew how to test, therefore, what you really need to do is start learning how to test. I mean if someone was re-wiring your house, and during that process he asked you what “voltage” is, wouldn’t you get someone else to wire your house? Like electrical work, plumbing, computer programming, or welding, good testing is a skilled activity.

I rarely give that answer, though, because I worry I will just leave people feeling discouraged.

The closest thing to a direct answer I can give you is this:

There exist no testing industry standards that are universally binding or even, in my opinion, more than negligibly helpful. Yes, there are documents that purport to be standards. If you are bound by them then you already know that. You aren’t subject to standards unless one has been imposed upon you by a regulating authority or by contract. Therefore, considering that testing costs money and time, I suggest that you don’t run any tests unless there is a reason to do so. In general, don’t do the same work a second time if you have already done it once. Certainly, if your clients would benefit from you running a test again, go for it. Otherwise, you are just indulging in compulsive/obsessive behavior, and you need help of a different kind than I offer.

A problem with this answer is that it begs the question of how you know when to run a test again. Fortunately, I wrote an essay on possible reasons to repeat tests. I can think of ten good reasons that you may want to repeat any given test (along with one big reason not to).

That’s a pretty good answer, but I think I can offer a little more:

Your job is probably to discover if there are terrible as-yet-unknown problems in your very complex product that you have little time to test. To do that job really well requires that you design and perform many tests, more tests than you probably have time to run. Therefore, when you run a test a second time, you are spending precious time and resources (even if it’s automated, though possibly less so) on something other than running a test you have not yet run that may find one of those big bugs you haven’t yet found. Get it?

So, how about having a small set of very basic tests that touch upon a lot features of the product. You may even want to automate these. It should take ten minutes to run these tests, ideally. Perhaps as long as an hour. Repeat those for every build. Their purpose is to quickly detect huge obvious things that may be wrong. Call that the smoke test suite. For everything else, make a test coverage outline that lists every significant element of the product and every significant element of data. Visit the items on that list and test each one according to its importance and potential for failure. Whenever any part of the product changes, try to figure out what could have been affected, and retest that area– but using different tests; perhaps variations on what you’ve already done.

By the way, the more you learn about testing, the less you will find advice like the preceding paragraph useful, because you will carry within you the ability to design your own test strategy that fits your specific purposes and contexts.

AlexanderIII

唔,学到东西了