When Should a Test Be Automated? 07

angelcc216

Brian Marick

Testing Foundations
marick@testing.com


Suppose your environment is very congenial to automation, and an automated test is only 10% more expensive than a manual test. (I would say this is rare.) That still means that,once you’ve automated ten tests, there’s one manual test - one unique execution of the product - that is never exercised until a customer tries it. If automation is more expensive,those ten automated tests might prevent ten or twenty or even more manual tests from ever being run. What bugs might those tests have found?

So the first test automation question is this:

If I automate this test, what manual tests will I lose? How many bugs might I lose with them? What will be their severity?

The answers will vary widely, depending on your project. Suppose you’re a tester on a
telecom system, one where quality is very important and the testing budget is dequate.

Your answer might be "If I automate this test, I’ll probably lose three manual tests. But I’ve done a pretty complete job of test design, and I really think those additional tests would only be trivial variations of existing tests. Strictly speaking, they’d be different executions, but I really doubt they’d find serious new bugs." For you, the cost of automation is low.