This month I’m battling (and want to sweet out my feelings) in my company with main stakeholders regarding the role of test cases and run-logs. They do believe:
1)Test Case content has the only reason to extend specifications with details in form of input data and expected results for each particular input. Approved by customers they define more exact development scope.
2)Test Case execution having main purpose to collect pass/fail statistics to be used as quality evaluation or shop/no-ship decision.
I’ve just red STP august 2006 column “Bringing logic into play“. The several pages long story pretend to be about deductive reasoning. Yuri Cherkan talks about direct issues of this approach. No offence to Yuri and his article, but I’m afraid he demonstrated inattentional blindness – ignoring the actual problem. I wonder how this issue is alive for years (and accepted as common wisdom) despite the fact that people like Cam Kaner are trying to fight them for decades. I could suggest to learn basics: Black Box Software Testing: By Cem Kaner & James Bach about Oracle problem, heuristic (instead of final and strict) reasoning, bug advocacy, the phenomenon of inattentional blindness and other problems of relying on test scripts.
Test Case content audience
I’ll address test case execution issue in next blog entry. I will not try to offend “commonly accepted wisdom” about test case content, which I started already in Why do we write Test Cases?.
In the first of two described cases we suppose customers or stakeholders to be the main audience. We demonstrate them what the best things we have done to make sure quality is there in the product.
In the second case our main audience is engineering. We demonstrate developers why do we believe the ticket reported is a bug.
I practice wiring test cases where the main audience is testers. I know this may be weird as most of testers, when there are no request from others to write tests cases, don’t bother themselves with writing. However there are at least two good reasons to do so:
a) Pre-planned exploratory testing. Before go frenzy into exploring the software product functionality try to understand the scope of testing by analyzing whatever documents you have and talking whoever willing to share any bit of information. Note what should be tested, what are the risks, etc. Maybe test case is too sound name for those notes as they may not include validation steps, but be simply notes like “try what happens when this...” or “pay more attention to that ...”. And still those are items that you could mark passed, failed, blocked or not tested – once your tests are done. You could make one more testing session to complete those not tested. You could plan more tests to test failed and blocked. And so on.
b) Regression testing is another case. The most misused case. Do you use for regression testing purpose subset of test cases created to do initial testing? Or do you develop specific regression test cases? In the second case you need to dynamically change them based on current project context, as regression risks tend to change. Why don’t you simply make test cases notes for one who will be doing regression tests. Test cases may be too sound again. Notes may not include validation steps or even execution steps. There could be notes like: “try to input from both keyboard and mouse”, “try all 4 cases: skip data entry, leave default, change values and input empty”, “check data in database”, “back button should not reset values already entered”. Those notes have a goal to help you to do regression as fast as possible. It may include data to be imported (saving your time to input it manually). Again you could pass, fail, block or do not test each of them.
You can’t however in both cases use them as reasoning for defect advocating and you can’t really use them as test evidence. You should rely on testers skills. It is up to tester how to interpret things like “check data in database”.