IN THIS CHAPTER
Static White-Box Testing: Examining the Design and Code
Formal Reviews
Coding Standards and Guidelines
Generic Code Review Checklist
Software testing isn't limited to treating the specification or the program like a black box as described in Chapters 4, “Examining the Specification,” and 5, “Testing the Software with Blinders On.” If you have some programming experience, even if it's just a little, you can also perform testing on the software's architecture and code.
In some industries, such verification isn't as common as black-box testing. However, if you're testing military, financial, factory automation, or medical software, or if you're lucky enough to be working in a highly disciplined development model, it may be routine to verify the product at this level. If you're testing software for security issues, it's imperative.
This chapter introduces you to the basics of performing verification on the design and code. As a new software tester, it may not be your first task, but it's one that you can eventually move into if your interests lie in programming.
Highlights from this chapter include
The benefits of static white-box testing
The different types of static white-box reviews
Coding guidelines and standards
How to generically review code for errors
Remember the definitions of static testing and white-box testing from Chapter 4? Static testing refers to testing something that isn't running—examining and reviewing it.
White-box (or clear-box) testing implies having access to the code, being able to see it and review it.
Static white-box testing is the process of carefully and methodically reviewing the software design, architecture, or code for bugs without executing it. It's sometimes referred to as structural analysis.
The obvious reason to perform static white-box testing is to find bugs early and to find bugs that would be difficult to uncover or isolate with dynamic black-box testing. Having a team of testers concentrate their efforts on the design of the software at this early stage of development is highly cost effective.
A side benefit of performing static white-box testing is that it gives the team's black-box testers ideas for test cases to apply when they receive the software for testing. They may not necessarily understand the details of the code, but by listening to the review comments they can identify feature areas that sound troublesome or bug-prone.
NOTE
Development teams vary in who has the responsibility for static white-box testing. In some teams the programmers are the ones who organize and run the reviews, inviting the software testers as independent observers. In other teams the software testers are the ones who perform this task, asking the programmer who wrote the code and a couple of his peers to assist in the reviews. Ultimately, either approach can work. It's up to the development team to choose what works best for them.
The unfortunate thing about static white-box testing is that it's not always done. Many teams have the misconception that it's too time-consuming, too costly, or not productive. All of these are untrue—compared to the alternative of testing, finding, and even not finding bugs at the back end of the project. The problem lies in the perception that a programmer's job is to write lines of code and that any task that takes away from his efficiency of churning out those lines is slowing down the process.
Fortunately, the tide is changing. Many companies are realizing the benefits of testing early and are hiring and training their programmers and testers to perform white-box testing. It's not rocket science (unless you're designing rockets), but getting started requires knowing a few basic techniques. If you're interested in taking it further, the opportunities are huge.
A formal review is the process under which static white-box testing is performed. A formal review can range from a simple meeting between two programmers to a detailed, rigorous inspection of the software's design or its code.
There are four essential elements to a formal review:
Identify Problems. The goal of the review is to find problems with the software—not just items that are wrong, but missing items as well. All criticism should be directed at the design or code, not the person who created it. Participants shouldn't take any criticism personally. Leave your egos, emotions, and sensitive feelings at the door.
Follow Rules. A fixed set of rules should be followed. They may set the amount of code to be reviewed (usually a couple hundred lines), how much time will be spent (a couple hours), what can be commented on, and so on. This is important so that the participants know what their roles are and what they should expect. It helps the review run more smoothly.
Prepare. Each participant is expected to prepare for and contribute to the review. Depending on the type of review, participants may have different roles. They need to know what their duties and responsibilities are and be ready to actively fulfill them at the review. Most of the problems found through the review process are found during preparation, not at the actual review.
Write a Report. The review group must produce a written report summarizing the results of the review and make that report available to the rest of the product development team. It's imperative that others are told the results of the meeting—how many problems were found, where they were found, and so on.
What makes formal reviews work is following an established process. Haphazardly “getting together to go over some code” isn't sufficient and may actually be detrimental. If a process is run in an ad-hoc fashion, bugs will be missed and the participants will likely feel that the effort was a waste of time.
If the reviews are run properly, they can prove to be a great way to find bugs early. Think of them as one of the initial nets (see Figure 6.1) that catches the big bugs at the beginning of the process. Sure, smaller bugs will still get through, but they'll be caught in the next testing phases with the smaller nets with the tighter weave.
In addition to finding problems, holding formal reviews has a few indirect results:
Communications. Information not contained in the formal report is communicated. For example, the black-box testers can get insight into where problems may lie. Inexperienced programmers may learn new techniques from more experienced programmers. Management may get a better feel for how the project is tracking its schedule.
Quality. A programmer's code that is being gone over in detail, function by function, line by line, often results in the programmer being more careful. That's not to say that he would otherwise be sloppy—just that if he knows that his work is being carefully reviewed by his peers, he might make an extra effort to triple-check it to make sure that it's right.
Team Camaraderie. If a review is run properly, it can be a good place for testers and programmers to build respect for each other's skills and to better understand each other's jobs and job needs.
Solutions. Solutions may be found for tough problems, although whether they are discussed depends on the rules for the review. It may be more effective to discuss solutions outside the review.
These indirect benefits shouldn't be relied on, but they do happen. On many teams, for whatever reasons, the members end up working in isolation. Formal reviews are a great way to get them in the same room, all discussing the same project problems.
The easiest way to get team members together and doing their first formal reviews of the software is through peer reviews, the least formal method. Sometimes called buddy reviews, this method is really more of an “I'll show you mine if you show me yours” type discussion.
Peer reviews are often held with just the programmer who designed the architecture or wrote the code and one or two other programmers or testers acting as reviewers. That small group simply reviews the code together and looks for problems and oversights. To assure that the review is highly effective (and doesn't turn into a coffee break) all the participants need to make sure that the four key elements of a formal review are in place: Look for problems, follow rules, prepare for the review, and write a report. Because peer reviews are informal, these elements are often scaled back. Still, just getting together to discuss the code can find bugs.
Walkthroughs are the next step up in formality from peer reviews. In a walkthrough, the programmer who wrote the code formally presents (walks through) it to a small group of five or so other programmers and testers. The reviewers should receive copies of the software in advance of the review so they can examine it and write comments and questions that they want to ask at the review. Having at least one senior programmer as a reviewer is very important.
The presenter reads through the code line by line, or function by function, explaining what the code does and why. The reviewers listen and question anything that looks suspicious. Because of the larger number of participants involved in a walkthrough compared to a peer review, it's much more important for them to prepare for the review and to follow the rules. It's also very important that after the review the presenter write a report telling what was found and how he plans to address any bugs discovered.
Inspections are the most formal type of reviews. They are highly structured and require training for each participant. Inspections are different from peer reviews and walkthroughs in that the person who presents the code, the presenter or reader, isn't the original programmer. This forces someone else to learn and understand the material being presented, potentially giving a different slant and interpretation at the inspection meeting.
The other participants are called inspectors. Each is tasked with reviewing the code from a different perspective, such as a user, a tester, or a product support person. This helps bring different views of the product under review and very often identifies different bugs. One inspector is even tasked with reviewing the code backward—that is, from the end to the beginning—to make sure that the material is covered evenly and completely.
Some inspectors are also assigned tasks such as moderator and recorder to assure that the rules are followed and that the review is run effectively.
After the inspection meeting is held, the inspectors might meet again to discuss the defects they found and to work with the moderator to prepare a written report that identifies the rework necessary to address the problems. The programmer then makes the changes and the moderator verifies that they were properly made. Depending on the scope and magnitude of the changes and on how critical the software is, a reinspection may be needed to locate any remaining bugs.
Inspections have proven to be very effective in finding bugs in any software deliverable, especially design documents and code, and are gaining popularity as companies and product development teams discover their benefits.
In formal reviews, the inspectors are looking for problems and omissions in the code. There are the classic bugs where something just won't work as written. These are best found by careful analysis of the code—senior programmers and testers are great at this.
There are also problems where the code may operate properly but may not be written to meet a specific standard or guideline. It's equivalent to writing words that can be understood and get a point across but don't meet the grammatical and syntactical rules of the English language. Standards are the established, fixed, have-to-follow-them rules—the do's and don'ts. Guidelines are the suggested best practices, the recommendations, the preferred way of doing things. Standards have no exceptions, short of a structured waiver process. Guidelines can be a bit loose.
It may sound strange that some piece of software may work, may even be tested and shown to be very stable, but still be incorrect because it doesn't meet some criteria. It's important, though, and there are three reasons for adherence to a standard or guideline:
Reliability. It's been shown that code written to a specific standard or guideline is more reliable and secure than code that isn't.
Readability/Maintainability. Code that follows set standards and guidelines is easier to read, understand, and maintain.
Portability. Code often has to run on different hardware or be compiled with different compilers. If it follows a set standard, it will likely be easier—or even completely painless—to move it to a different platform.
The requirements for your project may range from strict adherence to national or international standards to loose following of internal team guidelines. What's important is that your team has some standards or guidelines for programming and that these are verified in a formal review.
Figure 6.2 shows an example of a programming standard that deals with the use of the C language goto, while, and if-else statements. Improper use of these statements often results in buggy code, and most programming standards explicitly set rules for using them.
Figure 6.2. A sample coding standard explains how several language control structures should be used. (Adapted from C++ Programming Guidelines by Thomas Plum and Dan Saks. Copyright 1991, Plum Hall, Inc.)
The standard has four main parts:
Title describes what topic the standard covers.
Standard (or guideline) describes the standard or guideline explaining exactly what's allowed and not allowed.
Justification gives the reasoning behind the standard so that the programmer understands why it's good programming practice.
Example shows simple programming samples of how to use the standard. This isn't always necessary.
Figure 6.3 is an example of a guideline dealing with C language features used in C++. Note how the language is a bit different. In this case, it starts out with “Try to avoid.” Guidelines aren't as strict as standards, so there is some room for flexibility if the situation calls for it.
If your project, because of its nature, must follow a set of programming standards, or if you're just interested in examining your software's code to see how well it meets a published standard or guideline, several sources are available for you to reference.
National and international standards for most computer languages and information technology can be obtained from:
American National Standards Institute (ANSI), www.ansi.org
International Engineering Consortium (IEC), www.iec.org
International Organization for Standardization (ISO), www.iso.ch
National Committee for Information Technology Standards (NCITS), www.ncits.org
There are also documents that demonstrate programming guidelines and best practices available from professional organizations such as
Association for Computing Machinery (ACM), www.acm.org
Institute of Electrical and Electronics Engineers, Inc (IEEE), www.ieee.org
You may also obtain information from the software vendor where you purchased your programming software. They often have published standards and guidelines available for free or for a small fee.
The rest of this chapter on static white-box testing covers some problems you should look for when verifying software for a formal code review. These checklists[1] are in addition to comparing the code against a standard or a guideline and to making sure that the code meets the project's design requirements.
To really understand and apply these checks, you should have some programming experience. If you haven't done much programming, you might find it useful to read an introductory book such as Sams Teach Yourself Beginning Programming in 24 Hours by Sams Publishing before you attempt to review program code in detail.
Data reference errors are bugs caused by using a variable, constant, array, string, or record that hasn't been properly declared or initialized for how it's being used and referenced.
Is an uninitialized variable referenced? Looking for omissions is just as important as looking for errors.
Are array and string subscripts integer values and are they always within the bounds of the array's or string's dimension?
Are there any potential “off by one” errors in indexing operations or subscript references to arrays? Remember the code in Listing 5.1 from Chapter 5.
Is a variable used where a constant would actually work better—for example, when checking the boundary of an array?
Is a variable ever assigned a value that's of a different type than the variable? For example, does the code accidentally assign a floating-point number to an integer variable?
Is memory allocated for referenced pointers?
If a data structure is referenced in multiple functions or subroutines, is the structure defined identically in each one?
Data declaration bugs are caused by improperly declaring or using variables or constants.
Are all the variables assigned the correct length, type, and storage class? For example, should a variable be declared as a string instead of an array of characters?
If a variable is initialized at the same time as it's declared, is it properly initialized and consistent with its type?
Are there any variables with similar names? This isn't necessarily a bug, but it could be a sign that the names have been confused with those from somewhere else in the program.
Are any variables declared that are never referenced or are referenced only once?
Are all the variables explicitly declared within their specific module? If not, is it understood that the variable is shared with the next higher module?
Computational or calculation errors are essentially bad math. The calculations don't result in the expected result.
Do any calculations that use variables have different data types, such as adding an integer to a floating-point number?
Do any calculations that use variables have the same data type but are different lengths—adding a byte to a word, for example?
Are the compiler's conversion rules for variables of inconsistent type or length understood and taken into account in any calculations?
Is the target variable of an assignment smaller than the right-hand expression?
Is overflow or underflow in the middle of a numeric calculation possible?
Is it ever possible for a divisor/modulus to be zero?
For cases of integer arithmetic, does the code handle that some calculations, particularly division, will result in loss of precision?
Can a variable's value go outside its meaningful range? For example, could the result of a probability be less than 0% or greater than 100%?
For expressions containing multiple operators, is there any confusion about the order of evaluation and is operator precedence correct? Are parentheses needed for clarification?
Less than, greater than, equal, not equal, true, false. Comparison and decision errors are very susceptible to boundary condition problems.
Are the comparisons correct? It may sound pretty simple, but there's always confusion over whether a comparison should be less than or less than or equal to.
Are there comparisons between fractional or floating-point values? If so, will any precision problems affect their comparison? Is 1.00000001 close enough to 1.00000002 to be equal?
Does each Boolean expression state what it should state? Does the Boolean calculation work as expected? Is there any doubt about the order of evaluation?
Are the operands of a Boolean operator Boolean? For example, is an integer variable containing integer values being used in a Boolean calculation?
Control flow errors are the result of loops and other control constructs in the language not behaving as expected. They are usually caused, directly or indirectly, by computational or comparison errors.
If the language contains statement groups such as
begin...endanddo...while, are theends explicit and do they match their appropriate groups?Will the program, module, subroutine, or loop eventually terminate? If it won't, is that acceptable?
Is there a possibility of premature loop exit?
Is it possible that a loop never executes? Is it acceptable if it doesn't?
If the program contains a multiway branch such as a
switch...casestatement, can the index variable ever exceed the number of branch possibilities? If it does, is this case handled properly?Are there any “off by one” errors that would cause unexpected flow through the loop?
Subroutine parameter errors are due to incorrect passing of data to and from software subroutines.
Do the types and sizes of parameters received by a subroutine match those sent by the calling code? Is the order correct?
If a subroutine has multiple entry points (yuck), is a parameter ever referenced that isn't associated with the current point of entry?
If constants are ever passed as arguments, are they accidentally changed in the subroutine?
Does a subroutine alter a parameter that's intended only as an input value?
Do the units of each parameter match the units of each corresponding argument—English versus metric, for example?
If global variables are present, do they have similar definitions and attributes in all referencing subroutines?
These errors include anything related to reading from a file, accepting input from a keyboard or mouse, and writing to an output device such as a printer or screen. The items presented here are very simplified and generic. You should adapt and add to them to properly cover the software you're testing.
Does the software strictly adhere to the specified format of the data being read or written by the external device?
If the file or peripheral isn't present or ready, is that error condition handled?
Does the software handle the situation of the external device being disconnected, not available, or full during a read or write?
Are all conceivable errors handled by the software in an expected way?
Have all error messages been checked for correctness, appropriateness, grammar, and spelling?
This best-of-the-rest list defines a few items that didn't fit well in the other categories. It's not by any means complete, but should give you ideas for specific items that should be added to a list tailored for your software project.
Will the software work with languages other than English? Does it handle extended ASCII characters? Does it need to use Unicode instead of ASCII?
If the software is intended to be portable to other compilers and CPUs, have allowances been made for this? Portability, if required, can be a huge issue if not planned and tested for.
Has compatibility been considered so that the software will operate with different amounts of available memory, different internal hardware such as graphics and sound cards, and different peripherals such as printers and modems?
Does compilation of the program produce any “warning” or “informational” messages? They usually indicate that something questionable is being done. Purists would argue that any warning message is unacceptable.
Examining the code—static white-box testing—has proven to be an effective means for finding bugs early. It's a task that requires a great deal of preparation to make it a productive exercise, but many studies have shown that the time spent is well worth the benefits gained. To make it even more attractive, commercial software products, known as static analyzers, are available to automate a great deal of the work. The software reads in a program's source files and checks them against published standards and your own customizable guidelines. Compilers have also improved to the point that if you enable all their levels of error checking, they will catch many of the problems listed previously in the generic code review checklist. Some will even disallow use of functions with known security issues. These tools don't eliminate the tasks of code reviews or inspections—they just make it easier to accomplish and give testers more time to look even deeper for bugs.
If your team currently isn't doing testing at this level and you have some experience at programming, you might try suggesting it as a process to investigate. Programmers and managers may be apprehensive at first, not knowing if the benefits are that great—it's hard to claim, for example, that finding a bug during an inspection saved your project five days over finding it months later during black-box testing. But, static white-box testing is gaining momentum, and in some circles, projects can't ship reliable software without it.
These quiz questions are provided for your further understanding. See Appendix A, “Answers to Quiz Questions,” for the answers—but don't peek!
[1] These checklist items were adapted from Software Testing in the Real World: Improving the Process, pp. 198-201. Copyright 1995 by Edward Kit. Used by permission of Pearson Education Limited, London. All rights reserved.