Can automated testing make a good development team great?
Submitted by koch.geoff on Fri, 03/02/2006 - 04:12.

Thanks, Mike, for being willing to host this discussion on your blog. And a caveat before I get started that I'm not a programmer, so some of my questions may be inelegant or downright dumb.

I guess my first question is: can automation make up for talent gaps on a team? I get the sense from perusing various forums so far that any widespread use of automated testing requires a relatively savvy, sophisticated development team to begin with. Can a good team switch from manual to automated testing and catch a great team that uses all manual testing?

I've got lots of other questions but maybe I'll just stop at this point and see where we go from here.

Thanks,
Geoff

Re: Can automated testing make a good development team great?
Submitted by Mike Kelly on Sat, 04/02/2006 - 15:30.

First, I'm not sure if you are referring to automated testing done by developers or testers. Your question seems to indicate you are referring to test automation done by developers in the test-driven development style. However, you also mention switching from manual to automated testing. It may be that you are referring to switching from a traditional development team (developers write code and testers test) to a more agile development team (developers write code, developers write test code, and testers test).

While I'm familiar with developer testing, my experience is with tester testing. Either way, I think the answer is no -- I don't think automation can make up for talent gaps on a team. Someone who can perform test automation understands how to write code, use a simulator or probes, or can perform activity recording. It does not mean they know how to test or how to use those tools in any sophisticated way.

example or two of when automated testing didn't work out?
Submitted by koch.geoff on Sun, 05/02/2006 - 14:20.

Thanks, Mike. I guess I meant development teams in the broadest, most global sense -- from gathering requirements and specs to coding to testing, integrating and performing other QA functions and so forth. But you did answer the question. Whether it's agile or traditional waterfall development, you do have to be fairly sophisticated to get lots of mileage of automation.

One other quick follow-up to that: for savvy teams that embrace automated testing, have you seen cases where it's possible to reduce the number of developers and/or testers on a project? Automation in other industries seems to shrink the number of people required to do the job. Is that the case in software testing, too?

I'm also curious if you're willing to share an example or two, either from your own experience or something you've witnessed firsthand, of when automated testing hasn't worked out? A case where testers had some of the skills you list -- knew how to write code, use a simulator or probes and perform activity recording -- but didn't know how to use the tools in a sophisticated way, and thus ran into problems.

Thanks.

Re: reduce the number of developers and/or testers on a project?
Submitted by Mike Kelly on Mon, 06/02/2006 - 04:26.
I’ve never seen test automation reduce the number of developers or testers on a project. Test automation is powerful because it enhances our ability to test for certain kinds of bugs. By using test automation, we can many times find those bugs faster then we could without. That frees us to do other tasks that can’t be automated. For testers, that’s richer, more complex testing. For developers, I would imagine it’s design related tasks like architecture and refactoring.

Cem Kaner has an excellent paper on the impossibility of complete testing which can be found here: http://www.kaner.com/pdfs/imposs.pdf. As Cem points out, complete testing is impossible for several reasons:

We can’t test all the inputs to the program.
We can’t test all the combinations of inputs to the program.
We can’t test all the paths through the program.
We can’t test for all of the other potential failures, such as those caused by user interface design errors or incomplete requirements analyses.
It’s important to understand these issues, because many times this is where automation becomes helpful. Many times automation can be leveraged to help with combinations of inputs and paths thought the program, but it’s difficult to automate the identification of inputs or all of the other potential failures. That means there will always be more work for any given tester then automation can solve; and thus, always more work once they have something automated.

Re: when automated testing hasn't worked out
Submitted by Mike Kelly on Mon, 06/02/2006 - 04:32.

I’ll give what I consider the two “classic” examples of automation failure. I’ve been a party to both…

Example One: Large scale automated regression test projects
My first software job was as an intern as a member of a test automation team. I was part of a team of interns, responsible for creating hundreds of automated regression tests for a rather sophisticated application. It was a lot of fun. We were all CS students, so we knew to modularize our code, implement coding standards, try not to use record and playback as much as possible, etc…. I knew nothing about testing and less about good test automation. I remember the first time someone asked me if I was familiar with Kaner’s work on the maintainability of test automation and I said, “Cem who?”

I specifically remember, after about a year and several successes I was asked to audit an automation implementation by a large consulting firm. This was daunting, I was still a $12 an hour intern and these were $150 consultants. What I found was less then encouraging. With the combined effort of three people for four months (if I remember correctly) they had created over 1,500 scripts. The first time I ran them, I ran them on a release that had supposedly just been “verified” by the same scripts. I saw a 60% failure rate. I investigated further and I couldn’t find any documentation of successful testing. After trying to debug 1,500 record and playback scripts, I finally convinced management to let me throw them away and rewrite them in a maintainable way. It took me about three weeks to get what I think was the same coverage. (I don’t really know if it was the same because their work was so horrible I couldn’t tell.)

They knew the tools. They didn’t have any sophistication to their use of the tools. Their automation was a complete waste of money by the company.

Example Two: Performance testing
It is very easy to create performance test scripts. I’ve yet to work with a performance test tool where I haven’t been able to generate a script and get it running within about two hours (and that’s at the high end). The problem with that, is that many people new to performance testing (myself included when I was new), think that it’s relatively easy to create and run a meaningful performance test. Add to that the confusion about what people mean when they say “performance test” and you have a scenario where bad test automation can do more harm then good by providing an false sense of confidence in the performance of the application under test.

On one of my first projects where I did performance testing, I didn’t understand that modeling what your users were actually doing with the application was important. So I developed a test case with what turned out to be a trivial task and ran my tests using that one script (under various loads). Needless to say, the application performed great. It wasn’t until much later in the project, when it was a lot more costly to fix due to resource constraints, that we found the blatant error in my testing and were able to get someone to help with the performance testing so we could isolate problems and fix them.

The two best resources I’ve seen that address the complexity and the sophistication required by performance testers are two series by Scott Barber. They can be found on the PerfTestPlus website, but just to illustrate the point I’ve included titles for both series below:

User Experience, not Metrics Series:

Introduction
Modeling Individual User Delays
Modeling Individual User Patterns
Modeling Groups of Users
What should I time and where do I put my timers?
What is an outlier and how do I account for one?
Consolidating Test Results
Choosing Tests and Reporting Results to Meet Stakeholders Needs
Summarizing Across Multiple Tests
Creating a Degradation Curve
Handling Authentication and Session Tracking
Scripting Conditional User Path Navigation
Working with Unrecognized Protocols
Beyond Performance Testing Series:

Introduction
A Performance Engineering Strategy
How Fast is Fast Enough?
Accounting for User Abandonment
Determine the Root Cause of Script Failures
Interpreting Scatter Charts
Identifying the Critical Bottleneck
Modifying Tests to Focus on Failure/Bottleneck Resolution
Pinpointing the Architectural Tier of the Failure/Bottleneck
Creating a Test to Exploit the Failure/Bottleneck
Collaborative Tuning
Testing and Tuning on Common Tiers
Testing and Tuning Load Balancers and Networks
Testing and Tuning Security
These articles cover what I believe to be the basics for a performance tester. In fact, these articles make up the bulk of my curriculum when I offer training courses on performance testing.

last question
Submitted by koch.geoff on Wed, 08/02/2006 - 03:32.

Got it, Mike, and thanks. You've given me more than enough for the column. I'm just curious... is there a particularly hot topic or trend in test automation that you think isn't getting talked about enough these days? (Just want to make sure I'm not missing some big, untold story.) Also, how do you prefer that I refer to you in the magazine if I quote you or attribute comments to you? Thanks again.

I had to check on automation's relation to the size a programming team. I recently toured the Ford River Rouge Plant in Dearborn, Mich., where they make the F150, among other trucks. The factoid I heard while at the plant was that number of employees over the years has dwindled from a peak of 100,000 (many decades ago) to just 7,000 today. Obviously, there are lots of reasons for the drastic reduction. But automation was mentioned several times as one of the driving factors. I suspect, however, that comparing manufacturing cars to writing code is apples to oranges. (Unfortunately, Michiganians frame everything in terms of the auto industry.) Thanks again for your help.

Anyone else?
Submitted by Mike Kelly on Wed, 08/02/2006 - 04:19.

This is where I would invite the silent readers to join in and add a comment if they have one. I know there are some people who read this blog who do a lot more automation then I do (in terms of test-driven development, performance, and scripting — now days).

For me, I think it’s more of an over told story then an untold story. Many of the companies I work with look at automation as a panacea or as something to be feared. Either it can solve everything or solve nothing. Keep in mind, I’m a huge proponent for automation.

There is a balance to automation. Somewhere where it empowers the tester (or developer) with more visibility, reach, or confidence then they would have without it, but yet still does not slow them down due to it’s inertia. Automation is a means to an ends:

a test that can’t be executed any other way (performance)
a test that we can run faster with the help of the computer (a quick script to test N x N x N x N combinations)
the ability to have a change detection agent (unit level tests)
even something as simple as a way to see something otherwise difficult to see (probes and log parsers for example)
While I think automation skills (that’s some combination of plain old technical skills and coding skills) are important, they are not as important as one’s ability to develop good tests in the first place. If I’m a bad tester, but a good automater, all I’ll ever automate is bad tests.

My stories: good and bad
Submitted by Ainars Galvans on Thu, 09/02/2006 - 13:34.

OK, I'm going to add a little bit although I don't believe this is anything "new". I'll start with good news - when you say tests impossible otherwise (performance, 24 hour tests, etc) they are not impossible - you could ask 100 or 1000 testers press the buttons, it's just the place where automation is really really worth it. Also those N*N*N*N tests – the same you could theoretically have N*N*N*N testers with the same number of environments all pressing the same buttons, but do you want to wast their time so badly? Let's now talk about those typical functional testing. There are 3 types - test-before-code, data-driven testing and automate for regression reasons (regression functions that passed manual or semi-manual tests). Each have their own issues: the first and last one don't aim at to find bugs, instead – to demonstrate that there are no bugs, and I have to admit quite effective. Data-driven-testing is the best at finding bugs but require a lot of skills, testing experience and efforts. This answers to question –

“can automation make up for talent gaps on a team?“

NO, instead – a good automation need even more talent. But let me also mention there are one more dimension of options - either you automate at API level or at UI level. API level have rarely regression defects, so even if you are able to announce a great test coverage, those tests ... somehow ... just don't cache the bugs which are introduced. Still this level is great for test-first approach and this is where it should be used as much as possible,m I believe. UI-regression tests also works somewhat unreliable. They mostly fail due to changes not due to bugs. Even if your automation tool could recognize UI elements by name, UI don't typically change placement of buttons or their names, they change the buttons: e.g. instead of one OK there are now Submit and Send Manually.... One more issue is that UI automation tests are slow. They are typically only able to work twice as fast as expert human (or even slower depending on tool). It means and it is typically happening in my projects that if you have large enough set of automated tests once you have passed some 100 of them and fail on the 101, the defect is already discovered by human tester.

Also - Automated tests improves schedule reliability
Submitted by Ainars Galvans on Mon, 13/02/2006 - 09:20.

Wanted also to mention – although those test-first and regression test automation are weak at finding defects, they are strong at supporting schedule, exactly because they will not find more and more defects as human tester will typically do. Is it good or bad...? So I update my answer to

“can automation make up for talent gaps on a team?“

It can't make up testing talent gaps, only management (or probably business analyze, communication, etc.) talents.

看了一下，有些东西讲的不错，如果有朋友在自动化测试道路上碰到了难题，或者在思考自动化测试是否适合自己的时候，可以看看这些讨论。推荐！

收藏下来