50 Specific Ways to Improve Your Testing - 03
Item 3: Design Test Procedures As Soon As Requirements Are AvailableJust as software engineers produce design documents based on requirements, it is necessary for the testing team to design test procedures based on these requirements as well. In some organizations, the development of test procedures is pushed off until after a build of the software is delivered to the testing team, due either to lack of time or lack of properly specified requirements suitable for test-procedure design. This approach has inherent problems, including the possibility of requirement omissions or errors being discovered late in the cycle; software implementation issues, such as failure to satisfy a requirement; nontestability; and the development of incomplete test procedures.
Moving the test procedure development effort closer to the requirements phase of the process, rather than waiting until the software-development phase, allows for test procedures to provide benefits to the requirement-specification activity. During the course of developing a test procedure, certain oversights, omissions, incorrect flows, and other errors may be discovered in the requirements document, as testers attempt to walk through an interaction with the system at a very specific level, using sets of test data as input. This process obliges the requirement to account for variations in scenarios, as well as to specify a clear path through the interaction in all cases.
If a problem is uncovered in the requirement, that requirement will need to be reworked to account for this discovery. The earlier in the process such corrections are incorporated, the less likely it is that the corrections will affect software design or implementation.
As mentioned in Item 1, early detection equates to lower cost. If a requirement defect is discovered in later phases of the process, all stakeholders must change the requirement, design, and code, which will affect budgets, schedules, and possibly morale. However, if the defect is discovered during the requirements phase, repairing it is simply a matter of changing and reviewing the requirement text.
The process of identifying errors or omissions in a requirement through test-procedure definition is referred to as verifying the requirement's testability. If not enough information exists, or the information provided in the specification is too ambiguous to create a complete test procedure with its related test cases for relevant paths, the specification is not considered to be testable, and may not be suitable for software development. Whether a test can be developed for a requirement is a valuable check and should be considered part of the process of approving a requirement as complete. There are exceptions, where a requirement cannot immediately be verified programmatically or manually by executing a test. Such exceptions need to be explicitly stated. For example, fulfillment of a requirement that "all data files need to be stored for record-keeping for three years" cannot be immediately verified. However, it does need to be approved, adhered to, and tracked.
If a requirement cannot be verified, there is no guarantee that it will be implemented correctly. Being able to develop a test procedure that includes data inputs, steps to verify the requirement, and known expected outputs for each related requirement can assure requirement completeness by confirming that important requirement information is not missing, making the requirement difficult or even impossible to implement correctly and untestable. Developing test procedures for requirements early on allows for early discovery of nonverifiability issues.
Developing test procedures after a software build has been delivered to the testing team also risks incomplete test-procedure development because of intensive time pressure to complete the product's testing cycle. This can manifest in various ways: For example, the test procedure might be missing entirely; or it may not be thoroughly defined, omitting certain paths or data elements that may make a difference in the test outcome. As a result, defects might be missed. Or, the requirement may be incomplete, as described earlier, and not support the definition of the necessary test procedures, or even proper software development. Incomplete requirements often result in incomplete implementation.
Early evaluation of the testability of an application's requirements can be the basis for defining a testing strategy. While reviewing the testability of the requirements, testers might determine, for example, that using a capture/playback tool would be ideal, allowing execution of some of the tests in an automated fashion. Determining this early allows enough lead time to evaluate and implement automated testing tools.
To offer another example: During an early evaluation phase, it could be determined that some requirements relating to complex and diversified calculations may be more suitable tested with a custom test harness (see Item 37) or specialized scripts. Test harness development and other such test-preparation activities will require additional lead time before testing can begin.
Moving test procedures closer to the requirements-definition phase of an iteration carries some additional responsibilities, however, including prioritizing test procedures based on requirements, assigning adequate personnel, and understanding the testing strategy. It is often a luxury, if not impossible, to develop all test procedures immediately for each requirement, because of time, budget, and personnel constraints. Ideally, the requirements and subject-matter expert testing teams are both responsible for creating example test scenarios as part of the requirements definition, including scenario outcomes (the expected results).
An iteration, used in an iterative development process, includes the activities of requirement analysis, design, implementation and testing. There are many iterations in an iterative development process. A single iteration for the whole project would be known as the waterfall model.
Test-procedure development must be prioritized based on an iterative implementation plan. If time constraints exist, test developers should start by developing test procedures for the requirements to be implemented first. They can then develop "draft" test procedures for all requirements to be completed later.
Requirements are often refined through review and analysis in an iterative fashion. It is very common that new requirement details and scenario clarifications surface during the design and development phase. Purists will say that all requirement details should be ironed out during the requirements phase. However, the reality is that deadline pressures require development to start as soon as possible; the luxury of having complete requirements up-front is rare. If requirements are refined later in the process, the associated test procedures also need to be refined. These also must be kept up-to-date with respect to any changes: They should be treated as "living" documents.
Effectively managing such evolving requirements and test procedures requires a well-defined process in which test designers are also stakeholders in the requirement process. See Item 4 for more on the importance of communicating requirement changes to all stakeholders.
页:
[1]
