51Testing软件测试论坛

 找回密码
 (注-册)加入51Testing

QQ登录

只需一步,快速开始

微信登录,快人一步

手机号码,快捷登录

查看: 2304|回复: 1
打印 上一主题 下一主题

When Should a Test Be Automated?

[复制链接]

该用户从未签到

跳转到指定楼层
1#
发表于 2005-12-8 10:55:24 | 只看该作者 回帖奖励 |倒序浏览 |阅读模式
I want to automate as many tests as I can. I’m not comfortable running a test only once. What if a programmer then changes the code and introduces a bug? What if I don’t catch that bug because I didn’t rerun the test after the change? Wouldn’t I feel horrible?

Well, yes, but I’m not paid to feel comfortable rather than horrible. I’m paid to be costeffective. It took me a long time, but I finally realized that I was over-automating, that only some of the tests I created should be automated. Some of the tests I was automating not only did not find bugs when they were rerun, they had no significant prospect of doing so. Automating them was not a rational decision.

The question, then, is how to make a rational decision. When I take a job as a contract tester, I typically design a series of tests for some product feature. For each of them, I need to decide whether that particular test should be automated. This paper describes how I think about the tradeoffs.

Scenarios

In order for my argument to be clear, I must avoid trying to describe all possible testing scenarios at once. You as a reader are better served if I pick one realistic and useful scenario, describe it well, and then leave you to apply the argument to your specific situation. Here’s my scenario:

1. You have a fixed level of automation support. That is, automation tools are available. You know how to use them, though you may not be an expert. Support libraries have been written. I assume you’ll work with what you’ve got, not decide to acquire new tools, add more than simple features to a tool support library, or learn more about test automation. The question is: given what you have now, is automating this test justified? The decision about what to provide you was made earlier, and you live with it.


In other scenarios, you might argue for increased automation support later in the project. This paper does not directly address when that’s a good argument, but it provides context by detailing what it means to reduce the cost or increase the value of automation.

2. There are only two possibilities: a completely automated test that can run entirely unattended, and a "one-shot" manual test that is run once and then thrown away. These are extremes on a continuum. You might have tests that automate only cumbersome setup, but leave the rest to be done manually. Or you might have a manual test that’s carefully enough documented that it can readily be run again. Once you understand the factors that push a test to one extreme or the other, you’ll know better where the optimal point on the continuum lies for a particular test.


3. Both automation and manual testing are plausible. That’s not always the case. For example, load testing often requires the creation of heavy user workloads. Even if it were possible to arrange for 300 testers to use the product simultaneously, it’s surely not cost-effective. Load tests need to be automated.

4. Testing is done through an external interface ("black box testing"). The same analysis applies to testing at the code level - and a brief example is given toward the end of the paper - but I will not describe all the details.

5. There is no mandate to automate. Management accepts the notion that some of your tests will be automated and some will be manual.

6. You first design the test and then decide whether it should be automated. In reality, it’s common for the needs of automation to influence the design. Sadly, that sometimes means tests are weakened to make them automatable. But - if you understand where the true value of automation lies - it can also mean harmless adjustments or even improvements.

7. You have a certain amount of time to finish your testing. You should do the best testing possible in that time. The argument also applies in the less common situation of deciding on the tests first, then on how much time is required.
分享到:  QQ好友和群QQ好友和群 QQ空间QQ空间 腾讯微博腾讯微博 腾讯朋友腾讯朋友
收藏收藏
回复

使用道具 举报

该用户从未签到

2#
发表于 2005-12-8 11:47:04 | 只看该作者
when your boss want it to
回复 支持 反对

使用道具 举报

本版积分规则

关闭

站长推荐上一条 /1 下一条

小黑屋|手机版|Archiver|51Testing软件测试网 ( 沪ICP备05003035号 关于我们

GMT+8, 2024-11-23 08:06 , Processed in 0.068990 second(s), 25 queries .

Powered by Discuz! X3.2

© 2001-2024 Comsenz Inc.

快速回复 返回顶部 返回列表