Explore the World of Gray Box Testing?

connie

Explore the World of Gray Box Testing?

If you ask most "Testers" a question about gray box testing, they likely will give you a strange look - especially if all they have known is black box testing. However, this should not stop you from exploring this powerful testing technique.

Software development shops in general see testing teams as a necessary evil to completing projects. There is an increase in the number of sophisticated testing teams being utilized to fulfill project demands. Testers are growing up and their skill levels are maturing. We are witnessing a shift from black box to gray box testing.

As we all well know, system defects have costs:
Cost to prevent the defect
Cost to find the defect
Cost to report the defect
Cost to fix the defect
Cost to distribute the fix
For most applications that now are under development, white and black box testing was defined long ago and does not cover all the possible scenarios that must be tested. For example, how can a tester design a test case where a process on a web server is down or that may use numerous global variables that require concurrent input from multiple sources?

By implementing gray box testing, we simply reduce the overall cost of system defects - and prevent more from passing the testing stage. Our intention is to help you explore the different testing techniques. We do not expect or hope that you implement all of the different possible gray box testing methods. Use common sense, and take on what you can handle and successfully employ.

Black box testing and traditional waterfall methodologies are being replaced to meet the growing demand on our testing teams. CIOs are asking "ROI" questions and questioning the value of our testing teams - low skilled black box testing teams are not the wave of the future. We can improve the quality of our testing efforts through better planning. Doing things right the first time and asking ourselves key questions. Where is the value in what I am doing? Is there a better way towards building quality products? And if so, how do I get there?

First, you need to figure out where you are- yes, rate yourself. How do you plan to get anywhere unless you know where you are, you just might be there! The Juran Theoretical Quality Model is a great place to start and will help you determine if you need to make changes.

Juran Theoretical Quality Model

Software Black Box Testing Black box testing also known as functional testing is a software testing technique whereby the tester does not know the internal workings of the software.

Black box testing can best be described as placing your code in a box and creating your test cases independent of the code. This is the most common method for testers, as most of us don’t read code! One of our struggles with black box testing is the lack of documentation, which prevents us from creating effective test cases. We often get “code” dumped in our laps and are subsequently told to test.

The following are well known black box requirements- based test case selection techniques:

Equivalence Classing
Partition Analysis
Cause-Effect Graphing
Factor/Action matrix (FAM) Technique
Heuristic Test Case Selection
As previously mentioned, black box testing is a part of the testing process as a whole. It is a complementary method and excels in determining certain specific errors that can be defined in the following categories:

Incorrect or missing functions
Interface errors
Errors in data structures or files
Performance errors
Initialization or termination problems
In addition, black box testing occurs toward the end of the coding process, and is therefore used to test the following:

The functional validity of the software.
The classes of errors applicable to the specific software.
The boundaries of error classes
Requirements.
Output.
Software Black Box Testing is not a bad thing. This software testing technique works and has advantages and disadvantages.

Advantages


Tests are unbiased because the designer and the tester are independent of each other
Tester does not need programming knowledge
Tests are done from the point of view of the client, not the designer/coder
Test cases can be designed as soon as the specifications are complete

Disadvantages


Tests can be redundant if the coder has already run the test cases
Test cases are difficult to design - very difficult when coding is done ahead of specifications/design documentation
Testing every possible input stream is unrealistic because it would take a inordinate amount of time; therefore, many program paths will go untested

Software White Box Testing (synonym glass-box).

Testing is done under a structural testing strategy and requires complete access to the object's structure, that is, the source code. [B. Beizer, 1995 p8], The following are well known white box, code-based test case selection techniques:

Statement/Path Coverage
Data Flow Coverage
Domain Testing

Software white box testing technique is most commonly used by individuals who can read and write code. White box testing has many advantages such as increased testing coverage and reduced duplicate testing. The down side is the loss of focus of the testing team. For example, is the software user friendly and does it meet the business requirements? White box testers are so focussed on the code that they lose their objectivity. The other disadvantage is the cost of white box testing which tends to be higher due to the increased skill levels.

Why We Need Gray Box Testing

Have you heard the expression "It doesn’t matter whether a cat is black or white so long as it catches mice"? The same holds true for testing. gray box testing will increase our testing coverage by allowing you to focus on all of the layers of any complex system through the combination of all existing white and black box techniques.

Software Gray Box Testing

Cem Kanner defines gray box testing as involving inputs and outputs, but test design is educated by information about the code or the program operation of a kind that would normally be out of view of the tester. Gray box testing can be seen as the blending of structural and functional testing methods throughout the entire testing procedure.

Gray-box testing examines the activity of back-end components during test case execution. There are two types of problems that can be encountered during gray-box testing. The first is when a component encounters a failure of some kind, causing the operation to be aborted. For example, an edit check to allow dollars does not accept dollar amounts, i.e. "AAA". The second is when the test executes in full, but the content of the results is incorrect. Example: calculations - produces a number but it is incorrect.


We will introduce a sample of gray box testing techniques below in order to provide you with the general overview.
Matrix Testing
Strategy for Gray box Regression testing
Pattern Testing
Orthogonal Array Testing

Matrix Testing

The idea of beginning your testing activities with a list of variables used in the software is not new. You may have heard the term CRUD (created, read, update and deleted) method. Basically it starts with developers defining all the variables that exist in their programs. Each variable will have an inherent technical risk, business risk and can be used with different frequency during its’ life cycle. All of this information is summarized in tables 1 and 2, which then drives the design of test cases as illustrated in the following example.

Example: Client name and address input-recall screen

Client name and address input-recall table

Further information is extracted to complete the testing matrix and is summarized in the table 2 for the example in table 1.

Risk Matrix

From the chart above, a testing analyst can quickly ascertain that the business and technical aspect of the code, deleting and saving records requires testing. It would not be uncommon for a black box tester to concentrate his or her efforts solely on updating record functionality; hence, little return from their effort - more system defects/higher cost down the road.


Software Regression Testing

Regression testing -testing that is performed after making a functional improvement or repair to the program. Its purpose is to determine if the change has regressed other aspects of the program [Glenford J.Myers, 1979]. This can be accomplished by executing the following testing strategies.

Retest all: Rerun all existing test cases
Retest Risky Use Cases: Choose baseline tests to rerun by risk
Retest By Profile: Choose baseline tests to rerun by allocating time in proportion to operational profile
Retest Changed Segment: Choose baseline tests to rerun by comparing code changes. (White box strategy)
Retest within Firewall: Choose baseline tests to rerun by analyzing dependencies. (White box strategy)

As a gray box tester, you can mix all the above strategies. You may want to use the following percentage breakdown for regression testing.
Running existing test cases- 80% of allowed time (automated testing really helps here). Do not forget to create matrix tables as above to choose priority of test cases during execution. Exploratory Testing - 20% of allowed time

Pattern Testing

Pattern testing is best accomplished when historical data of previous system defects are analyzed. Your analysis will include specific reasons for the defect, which will require system analysis. Unlike black box testing where you are tracking the type of failures that are occurring, gray box testing digs within the code and determines why the failure happened. This information is extremely valuable, as future design of test cases will be proactive in finding the other failures before they hit production. Remember, having coding structure in place influences gray box test case design. Therefore, take advantage of powerful technique and design test cases that your developers and black box testers would not cover.

Analysis Template will include:

The problem addressed
Applicable situation
The problem that can be discovered
Related problems
Solution for developers that can be implemented
Generic test cases

New test cases will include:

Security related test cases
Business related test cases
GUI related test cases
Language related test cases
Architecture related test cases
Database related test cases
Browser related test cases
Operating System related test cases

Orthogonal Array Testing: Taguchi Method

Orthogonal Array Testing is a statistical testing technique implemented by Taguchi. This method is extremely valuable for testing complex applications and e-comm products. The e-comm world presents interesting challenges for test case design and testing coverage. The black box testing technique will not adequately provide sufficient testing coverage. The underlining infrastructure connections between servers and legacy systems will not be understood by the black box testing team. A gray box testing team will have the necessary knowledge and combined with the power of statistical testing, an elaborate testing net can be set-up and implemented.

The theory -Orthogonal Array Testing (OAT) can be used to reduce the number of combinations and provide maximum coverage with a minimum number of test cases. OAT is an array of values in which each column represents a variable - factor that can take a certain set of values called levels. Each row represents a test case. In OAT, the factors are combined pair-wise rather than representing all possible combinations of factors and levels.

A simple example: Three Factors and Three Levels Array is shown in table 3.

Three Levels Array

Now given the example above, a tester would have to create 3³ or 27 test combinations as shown in table 4.

Number of test cases table

By assigning values for each factor and then extrapolating for combined pairing, our total number of test cases drops to nine from 27. This simple but effective technique maximizes the required testing coverage.

Summary

We have covered a number of different testing techniques that can be successfully deployed. Our intent is to broaden your scope of thinking and provide you with a starting point to move forward. Check out the experts in the biography and talk to people who share your vision of a well-balanced and highly skilled testing team. Reduce the cost of system defects and thereby increase the quality of your software.

Bio:
Alex Samurin is a QA Analyst in Toronto, Canada. Mr. Samurin volunteers for The Quality Assurance Institute (QAI) as well as the Toronto Association of Systems & Software Quality.

Joseph (Joe) Larizza, is a Project and QA Manager, Banking and Investment Industry at CGI. Mr. Larizza is a member of the Board of Directors for the Toronto Association of Systems & Software Quality and volunteers for theQuality Assurance Institute. He is a Certified System Quality Analyst and holds a Bachelor of Arts Degree in Economics.

Bibliography

1- Nguyen, Hung Q. Testing Applications on the Web. Canada: Robert Ipsen, 2001.
2- McGregor, John D. and Sykes, David A. Practical Guide to Testing Object-Oriented Software. Canada: Addison-Wesley, 2001.
3- Poston, Robert M. Automating Specification-Based Software Testing. USA: IEEE Computer Society Press, 1996.
4- Cem Kaner, James Bach, Bret Pettichord. Lessons Learned in Software Testing. John Wiley & Sons 2002.
5- Kaner, Fank, and NguyenTesting Computer Software. USA: International Thomson Computer Press, 1998.
6- Boris Beizer. Black-Box Testing : Techniques for Functional Testing of Software and Systems
7- Robert V. Binder. Testing Object-Oriented Systems: Models, Patterns, and Tools (The Addison-Wesley Object Technology Series). October, 1999 Addison-Wesley Pub Co

Published by Toronto Association of Systems and Software Quality - July 2003