标题: （续）Theme Two: Planning the Testing Effort 典型测试错误（二）——计划测试工作 [打印本页] 作者: pele 时间: 2006-8-14 22:56 标题: （续）Theme Two: Planning the Testing Effort 典型测试错误（二）——计划测试工作 How about avoiding testing altogether?
不做测试会怎么样？
At a conference last year, I met (separately) two depressed testers who told me their management was of the opinion that the World Wide Web could reduce testing costs. "Look at [wildly successful internet company]. They distribute betas over the network and get their customers to do the testing for free!" The Windows 95 beta program is also cited in similar ways.
Beware of an overreliance on beta testing. Beta testing seems to give you test cases representative of customer use - because the test cases are customer use. Also, bugs reported by customers are by definition those important to customers. However, there are several problems:
1. The customers probably aren't that representative. In the common high-tech marketing model, beta users, especially those of the "put it on your web site and they will download" sort, are the early adopters, those who like to tinker with new technologies. They are not the pragmatists, those who want to wait until the technology is proven and safe to adopt. The usage patterns of these two groups are different, as are the kinds of bugs they consider important. In particular, early adopters have a high tolerance for bugs with workarounds and for bugs that "just go away" when they reload the program. Pragmatists, who are much less tolerant, make up the large majority of the market.
2. Even of those beta users who actually use the product, most will not use it seriously. They will give it the equivalent of a quick test drive, rather than taking the whole family for a two week vacation. As any car buyer knows, the test drive often leaves unpleasant features undiscovered.
3. Beta users - just like customers in general - don't report usability problems unless prompted. They simply silently decide they won't buy the final version.
β用户象客户一样，除非特别要求，一般不会报告可用性错误。他们只是暗自决定不去购买最终产品。
4. Beta users - just like customers in general - often won't report a bug, especially if they're not sure what they did to cause it, or if they think it is obvious enough that someone else must have already reported it.
5. When beta users report a bug, the bug report is often unusable. It costs much more time and effort to handle a user bug report than one generated internally.
Beta programs can be useful, but they require careful planning and monitoring if they are to do more than give a warm fuzzy feeling that at least some customers have used the product before it's inflicted on all of them. See [Kaner93] for a brief description.
The one situation in which beta programs are unequivocally useful is in configuration testing. For any possible screwy configuration, you can find a beta user who has it. You can do much more configuration testing than would be possible in an in-house lab (or even perhaps an outsourced testing agency). Beta users won't do as thorough a job as a trained tester, but they'll catch gross errors of the "BackupBuster doesn't work on this brand of 'compatible' floppy tape drive" sort.
Beta programs are also useful for building word of mouth advertising, getting "first glance" reviews in magazines, supporting third-party vendors who will build their product on top of yours, and so on. Those are properly marketing activities, not testing.
Planning and replanning in support of the role of testing
计划和重新计划测试的支持作用
Each of the types of testing described above, including functional testing, reduces uncertainty about a particular aspect of the product. When done, you have confidence that some functional areas are less buggy, others more. The product either usually works on new configurations, or it doesn't.
There's a natural tendency toward finishing one testing task before moving on to the next, but that may lead you to discover bad news too late. It's better to know something about all areas than everything about a few. When you've discovered where the problem areas lie, you can test them to greater depth as a way of helping the developers raise the quality by finding the important bugs.
Strictly, I've been over-simplistic in describing testing's role as reducing uncertainty. It would be better to say "risk-weighted uncertainty". Some areas in the product are riskier than others, perhaps because they're used by more customers or because failures in that area would be particularly severe. Riskier areas require more certainty. Failing to correctly identify risky areas is a common mistake, and it leads to misallocated testing effort. There are two sound approaches for identifying risky areas:
1. Ask everyone you can for their opinion. Gather data from developers, marketers, technical writers, customer support people, and whatever customer representatives you can find. See [Kaner96a] for a good description of this kind of collaborative test planning.
2. Use historical data. Analyzing bug reports from past products (especially those from customers, but also internal bug reports) helps tell you what areas to explore in this project.
"So, winter's early this year. We're still going to invade Russia."
“今年冬天来得很早。但我们还是要入侵俄国。”
Good testers are systematic and organized, yet they are exposed to all the chaos and twists and turns and changes of plan typical of a software development project. In fact, the chaos is magnified by the time it gets to testers, because of their position at the end of the food chain and typically low status. One unfortunate reaction is sticking stubbornly to the test plan. Emotionally, this can be very satisfying: "They can flail around however they like, but I'm going to hunker down and do my job." The problem is that your job is not to write tests. It's to find the bugs that matter in the areas of greatest uncertainty and risk, and ignoring changes in the reality of the product and project can mean that your testing becomes irrelevant.
That's not to say that testers should jump to readjust all their plans whenever there's a shift in the wind, but my experience is that more testers let their plans fossilize than overreact to project change.