[E文资料共享]组合测试时如何减少测试用例数

skinapi

Pairwise testing成对测试
http://blogs.msdn.com/micahel/archive/2004/04/28/122702.aspx

The test cases go marching two by two, hurrah, hurrah
One of the banes of my existence is test matrices.  Everywhere I look, it seems, another matrix is lurking, impatiently waiting to lob another set of mind-numbinginly boring test cases at me that are guaranteed to suck up two more weeks' worth of the extra time I don't have.  Matrices, matrices, matrices galore, all focused on me and mine.  No sooner do I vanquish one but two more pop their squirmy heads out of a trans-dimensional rift and splat themselves all over my office.

Since my current product requires Longhorn, I escape (this release, anyway) the dreaded Platform Matrix.  This is perhaps the most well-known of the species, dreaded and hated by testers everywhere.  Let me give you a typical example.  I'll include only Microsoft products just to keep things simple (ha ha):  

OS
Windows 2000 Gold (the original, un-SPd version)
Windows 2000 SP1
Windows 2000 SP2
Windows 2000 SP3
Windows 2000 SP4
Windows Server 2003
Windows XP Gold
Windows XP SP1
Windows XP SP1a
Browser
IE 5.01 Gold
IE 5.01 SP1
IE 5.01 SP2
IE 5.01 SP3
IE 5.01 SP4
IE 5.5 Gold
IE 5.5 SP1
IE 5.5 SP2
IE 6.0 Gold
IE 6.0 SP1
.Net Framework
1.0 Gold
1.0 SP1
1.0 SP2
1.1 Gold
Office Suite
Office 2000 Gold
Office 2000 SP1
Office 2000 SP2
Office 2000 SP3
Office 2000 Developer Gold
Office 2000 Developer SP1
Office 2000 Developer SP2
Office XP Gold
Office XP SP1
Office XP SP2
Office XP SP3
Office 2003 Gold
Your app here
Current version
Previous version
Version before that
To well and truly test this matrix, you have to test every single combination.  Let's see...9 OS x 10 Browser x 4 .Net Framework x 12 Office x 3 versions of your app...that's 12,960 different configurations you need to run all of your tests on!  If you can get through 36 configurations every day, that's still a year's worth of working weekends and holidays (and don't even think about vacation).

Of course, some of these combinations aren't valid -- you can't get IE5 on Windows XP, for example.  Even with these removed, however, there's still a huge number of configurations to cover.  It's so huge, in fact, that most teams (at Microsoft and elsewhere) don't even bother trying.  Instead they'll take a (vertical, horizontal, and/or diagonal) slice or two through the matrix, add in a few additional configurations that seem likely to be popular, and leave it at that.  This is pretty much voodoo testing, however, and your chances of missing a particularly gnarly bug because you chose to skip some particular configuration is pretty high.

Even if you happen to have the luxury of testing on just a single OS with a single web browser and a single version of Office and your app, you still aren't safe.  Test matrices are a resilient breed that can live off just about anything.  If your app supports richly formatted text, then you have a matrix of font, font size, the various font styles (e.g., bold, small caps), and so on.  If you have a method in your API with more than one parameter, you have a matrix of the various test values for each parameter.  Matrices are everywhere!

All hope is not lost, however.  Starting with Siddharta Dalal and company in 1997, people started to realize that many bugs are caused by either a) individual incorrect values, or b) particular pairs of values.  A Google search on "pairwise combinatorial test -gene" (the "-gene" weeds out a bunch of bio-results) will tell you more than you probably want to know about the gory details of how pairwise testing works.  The short version is that pairwise reins in test matrices like a champion calf roper.

Here's how it works:  since most bugs are a result of a particular pair of values, you don't need to cover the full test matrix but rather need only hit every individual pair.  If you have more than two axes in your matrix, you can cover multiple pairs in each test case.  As you can imagine, this makes for a rather large reduction in the number of test cases (the more axes you have the bigger the reduction you'll see).  Taking the above Platform Matrix as an example, pairwise whittles the 12,960 configurations a straight multiplicative combination gives you down to just 126 combinations.  Here's the first five my tool spat out:

Windows XP SP1a, IE 5.01 SP3, .Net Framework 1.0 Gold, Office 2000 SP3, Previous version of your app
Windows XP Gold, IE 5.5 SP1, .Net Framework 1.1 Gold, Office XP SP3, Current version of your app
Windows 2000 Gold, IE 5.5 Gold, .Net Framework 1.0 SP1, Office 2000 Gold, Version of your app before the previous one
Windows 2000 SP2, IE 6.0 Gold, .Net Framework 1.0 SP2, Office 2000 Gold, Previous version of your app
Windows 2000 SP1, IE 5.01 SP3, .Net Framework 1.0 SP2, Office 2000 Developer SP1, Current version of your app
Combining disallowing invalid combinations with a few judicious decisions not to support particular configurations reduces this number even further.  

It's very important to note that you *must* test every combination the pairwise tool spits out.  If you decide to skip a particular configuration for some reason (it contains an invalid combination, for example), you're skipping not only that invalid combination but several other pairs of values as well.  Live dangerously if you want, but you're much better off adding that combination to your matrix definition's "Don't do this" restrictions and generating a new set of combinations.

It's not too hard to write your own basic pairwise combinatorial tool, but to generate really efficient results requires some fancy math.  Luckily for those of us who aren't math geniuses, pairwise tools abound on the Web.  I found the following without trying very hard; I've only glanced at them and so can't vouch for how well they work, but they should get you started:

If you have smallish matrices, check out the precalculated pairwise matrices at http://www.freequality.org/beta% ... files/tamatrix.htm.
If you can wrangle a Perl script into doing what you want, give AllPairs (http://www.satisfice.com/testmethod.shtml) a try.
If Java's your game, try http://www.site.uottawa.ca/~awilliam/.
If all points lead toward the C, give http://burtleburtle.net/bob/math/jenny.html a look-see.
If you have loads of money, Telcordia Technologies (http://aetgweb.argreenhouse.com/) will generate "robust and efficient test cases faster and more comprehensively than ever before" for a mere $6000 per year for two users.  (They do have a free two-week trial.)
*** Comments, questions, feedback?   Do test matrices excite you?  Or do you just want a fun job on a great team? </g>  Send two coding samples and an explanation of why you chose them, and of course your resume, to me at michhu at microsoft dot com. I need testers, and my team needs a data binding developer, program managers, and a product manager. Great coding skills required for all positions.