The software tester can be inundated with numerous testing strategies … how do you choose?

First – determine what types of defects are likely to lie in your code. Then, choose a strategy which is likely to uncover those types of defects.

Functional testing is performed on all projects, to at least a minimum level. It is used to demonstrate that the software performs the functions for which it was developed. It assumes no knowledge of the software’s implementation. For this reason it is often referred to as “Black-Box” testing. Test procedures are developed that specify the expected program outputs for a particular set of program inputs.  The tests are run and the actual outputs are compared to the expected outputs to detect problems. Depending on how well the tests are developed and executed, Functional testing can be effective at finding un-implemented requirements, interface problems, performance issues, and errors in the most-used features of the program.

The other commonly used type of testing is referred to as Structural or White-Box testing. Unlike Black-Box testing, it requires an unobstructed view of the inner workings of the software (hence the name “White-Box”).  Structural testing looks at the details of how the software is built. It may examine every conditional expression, every mathematical manipulation, every input, and every output. This is a significant amount of detail, so the software is tested one unit at a time, typically a single function or class. Since Structural testing takes such a different approach to testing than Functional testing, you might surmise that it would find different types of errors. That has been shown to be the case: Structural testing is effective at finding errors in the logic, control flow, calculations, and data in the code.

Code Inspection also uses this same detailed knowledge of the implementation to find software defects which could lead to hazards. It has proven to be a very effective method of finding software errors. Like Structural testing, Inspection is generally performed on one unit of the software at a time. The focus and intense scrutiny required to perform an effective Inspection precludes covering more than a few pages of code at a time. Because of this similar focus, Inspection tends to turn up many of the same types of defects as Structural testing. However, because it’s performed earlier in the development process, it finds them sooner. Herein is the chief benefit of Inspection: early detection of software problems. The “quality” of an Inspection is very dependent on the “Inspector’s” expertise in the language, the system application, and the amount of time spent in preparation for the Inspection.

That these techniques, Functional and Structural testing and Inspection, are so common, speaks to their effectiveness. However, the very nature of each method that makes it successful also prevents it from finding other types of errors. Both Inspection and White-Box testing focus on only a very small portion of the code at a time, ignoring the rest of the system. Although they are successful in finding errors local to the unit under test, they do not find problems involving interaction with other areas of the code. Black-Box testing generally exercises the system as a whole, but ignores the implementation details.  It is difficult to even run through every path in the code by means of Functional testing, let alone exercise it thoroughly.

Lastly you must consider those classes of defects that cannot be detected with these “traditional” methods. These problems are uncovered by focusing on the implementation details as they interact within the entire system. If the software does not preclude the existence of these types of defects by design, then we must assume that these errors are latent within the application. Detecting these errors requires additional techniques: an examination of the software system in its entirety, looking for the particular causes of specific problems. Since an exhaustive analysis of every detail in a program and it’s interaction with all other items in the code is impossible, analysis must be focused on specific areas of the software which are known to cause problems, such as Deadlock, Race Conditions, Stack Overflows, and Timing Problems. (See Critical Analyses.)

To summarize, all software must have some Black-Box Functional Testing performed to insure that the major requirements have been met. Depending on the criticality of your application, White-Box Structural Testing may also be required – especially if important features cannot be proven with Functional Tests. Inspection is a part of most well-developed Software Development Plans because it’s so much less expensive to fix an error when it’s found early in the development process. Lastly, if your software has any of the following:

• hard real-time requirements
• uses interrupts
• uses a multi-tasking OS or scheduler of some sort
• shares resources (data, hardware peripherals) between tasks or interrupts and the “main code”

… you’ll need to perform some Critical Analyses to insure you uncover the problems which Testing can’t find.