50 Specific Ways to Improve Your Testing - 02

angelcc216

Item 2: Verify the Requirements
In his work on specifying the requirements for buildings, Christopher Alexander[1] describes setting up a quality measure for each requirement: "The idea is for each requirement to have a quality measure that makes it possible to divide all solutions to the requirement into two classes: those for which we agree that they fit the requirement and those for which we agree that they do not fit the requirement." In other words, if a quality measure is specified for a requirement, any solution that meets this measure will be acceptable, and any solution that does not meet the measure will not be acceptable. Quality measures are used to test the new system against the requirements.

[1] Christopher Alexander, Notes On the Synthesis of Form (Cambridge, Mass.: Harvard University Press, 1964).

Attempting to define the quality measure for a requirement helps to rationalize fuzzy requirements. For example, everyone would agree with a statement like "the system must provide good value," but each person may have a different interpretation of "good value." In devising the scale that must be used to measure "good value," it will become necessary to identify what that term means. Sometimes requiring the stakeholders to think about a requirement in this way will lead to defining an agreed-upon quality measure. In other cases, there may be no agreement on a quality measure. One solution would be to replace one vague requirement with several unambiguous requirements, each with its own quality measure.[2]

[2] Tom Gilb has developed a notation, called Planguage (for Planning Language), to specify such quality requirements. His forthcoming book Competitive Engineering describes Planguage.

It is important that guidelines for requirement development and documentation be defined at the outset of the project. In all but the smallest programs, careful analysis is required to ensure that the system is developed properly. Use cases are one way to document functional requirements, and can lead to more thorough system designs and test procedures. (In most of this book, the broad term requirement will be used to denote any type of specification, whether a use case or another type of description of functional aspects of the system.)

In addition to functional requirements, it is also important to consider nonfunctional requirements, such as performance and security, early in the process: They can determine the technology choices and areas of risk. Nonfunctional requirements do not endow the system with any specific functions, but rather constrain or further define how the system will perform any given function. Functional requirements should be specified along with their associated nonfunctional requirements. (Chapter 9 discusses nonfunctional requirements.)

Following is a checklist that can be used by testers during the requirements phase to verify the quality of the requirements.[3],[4] Using this checklist is a first step toward trapping requirements-related defects as early as possible, so they don't propagate to subsequent phases, where they would be more difficult and expensive to find and correct. All stakeholders responsible for requirements should verify that requirements possess the following attributes.

[3] Suzanne Robertson, "An Early Start To Testing: How To Test Requirements," paper presented at EuroSTAR 96, Amsterdam, December 2?, 1996. Copyright 1996 The Atlantic Systems Guild Ltd. Used by permission of the author.

[4] Karl Wiegers, Software Requirements (Redmond, Wash.: Microsoft Press, Sept. 1999).

Correctness of a requirement is judged based on what the user wants. For example, are the rules and regulations stated correctly? Does the requirement exactly reflect the user's request? It is imperative that the end user, or a suitable representative, be involved during the requirements phase. Correctness can also be judged based on standards. Are the standards being followed?

Completeness ensures that no necessary elements are missing from the requirement. The goal is to avoid omitting requirements simply because no one has asked the right questions or examined all of the pertinent source documents.

Testers should insist that associated nonfunctional requirements, such as performance, security, usability, compatibility, and accessibility,[5] are described along with each functional requirement. Nonfunctional requirements are usually documented in two steps:

[5] Elfriede Dustin et al., "Nonfunctional Requirements," in Quality Web Systems: Performance, Security, and Usability (Boston, Mass.: Addison-Wesley, 2002), Sec. 2.5.

A system-wide specification is created that defines the nonfunctional requirements that apply to the system. For example, "The user interface of the Web system must be compatible with Netscape Navigator 4.x or higher and Microsoft Internet Explorer 4.x or higher."

Each requirement description should contain a section titled "Nonfunctional Requirements" documenting any specific nonfunctional needs of that particular requirement that deviate from the system-wide nonfunctional specification.

Consistency verifies that there are no internal or external contradictions among the elements within the work products, or between work products. By asking the question, "Does the specification define every essential subject-matter term used within the specification?" we can determine whether the elements used in the requirement are clear and precise. For example, a requirements specification that uses the term "viewer" in many places, with different meanings depending on context, will cause problems during design or implementation. Without clear and consistent definitions, determining whether a requirement is correct becomes a matter of opinion.

Testability (or verifiability) of the requirement confirms that it is possible to create a test for the requirement, and that an expected result is known and can be programmatically or visually verified. If a requirement cannot be tested or otherwise verified, this fact and its associated risks must be stated, and the requirement must be adjusted if possible so that it can be tested.

Feasibility of a requirement ensures that it can can be implemented given the budget, schedules, technology, and other resources available.

Necessity verifies that every requirement in the specification is relevant to the system. To test for relevance or necessity, the tester checks the requirement against the stated goals for the system. Does this requirement contribute to those goals? Would excluding this requirement prevent the system from meeting those goals? Are any other requirements dependent on this requirement? Some irrelevant requirements are not really requirements, but proposed solutions.

Prioritization allows everyone to understand the relative value to stakeholders of the requirement. Pardee[6] suggests that a scale from 1 to 5 be used to specify the level of reward for good performance and penalty for bad performance on a requirement. If a requirement is absolutely vital to the success of the system, then it has a penalty of 5 and a reward of 5. A requirement that would be nice to have but is not really vital might have a penalty of 1 and a reward of 3. The overall value or importance stakeholders place on a requirement is the sum of its penalties and rewards梚n the first case, 10, and in the second, 4. This knowledge can be used to make prioritization and trade-off decisions when the time comes to design the system. This approach needs to balance the perspective of the user (one kind of stakeholder) against the cost and technical risk associated with a proposed requirement (the perspective of the developer, another kind of stakeholder).[7]

[6] William J. Pardee, To Satisfy and Delight Your Customer: How to Manage for Customer Value (New York, N.Y.: Dorset House, 1996).

[7] For more information, see Karl Wiegers, Software Requirements, Ch. 13.

Unambiguousness ensures that requirements are stated in a precise and measurable way. The following is an example of an ambiguous requirement: "The system must respond quickly to customer inquiries." "Quickly" is innately ambiguous and subjective, and therefore renders the requirement untestable. A customer might think "quickly" means within 5 seconds, while a developer may think it means within 3 minutes. Conversely, a developer might think it means within 2 seconds and over-engineer a system to meet unnecessary performance goals.

Traceablity ensures that each requirement is identified in such a way that it can be associated with all parts of the system where it is used. For any change to requirements, is it possible to identify all parts of the system where this change has an effect?

To this point, each requirement has been considered as a separately identifiable, measurable entity. It is also necessary to consider the connections among requirements梩o understand the effect of one requirement on others. There must be a way of dealing with a large number of requirements and the complex connections among them. Suzanne Robertson[8] suggests that rather than trying to tackle everything simultaneously, it is better to divide requirements into manageable groups. This could be a matter of allocating requirements to subsystems, or to sequential releases based on priority. Once that is done, the connections can be considered in two phases: first the internal connections among the requirements in each group, then the connections among the groups. If the requirements are grouped in a way that minimizes the connections between groups, the complexity of tracing connections among requirements will be minimized.

[8] Suzanne Robertson, "An Early Start to Testing," op. cit.

Traceability also allows collection of information about individual requirements and other parts of the system that could be affected if a requirement changes, such as designs, code, tests, help screens, and so on. When informed of requirement changes, testers can make sure that all affected areas are adjusted accordingly.

As soon as a single requirement is available for review, it is possible to start testing that requirement for the aforementioned characteristics. Trapping requirements-related defects as early as they can be identified will prevent incorrect requirements from being incorporated in the design and implementation, where they will be more difficult and expensive to find and correct.[9]

[9] T. Capers Jones, Assessment and Control of Software Risks (Upper Saddle River, N.J.: Prentice Hall PTR, 1994).

After following these steps, the feature set of the application under development is now outlined and quantified, which allows for better organization, planning, tracking, and testing of each feature.