IN THIS CHAPTER
An Overview of Configuration Testing
Approaching the Task
Obtaining the Hardware
Identifying Hardware Standards
Configuration Testing Other Hardware
Life could be so simple. All computer hardware could be identical. All software could be written by the same company. There wouldn't be confusing option buttons to click or check boxes to check. Everything would interface perfectly the first time, every time. How boring.
In the real world, 50,000-square-foot computer superstores are offering PCs, printers, monitors, network cards, modems, scanners, digital cameras, peripherals, net-cams, and hundreds of other computer doodads from thousands of companies—all able to connect to your PC!
If you're just getting started at software testing, one of your first tasks may be configuration testing. You'll be making sure that your software works with as many different hardware combinations as possible. If you're not testing software for a popular hardware platform such as a PC or a Mac—that is, if you're testing some specialized proprietary system—you will still need to consider configuration issues. You can easily tailor what you learn in this chapter to your situation.
The first part of this chapter deals with the generalities of PC configuration testing and then moves into the specifics of testing printers, display adapters (video cards), and sound cards for a PC. Although the examples are based on desktop computers, you can extrapolate the methods to just about any type of configuration test problem. New and different devices are released every day, and it will be your job to figure out how to test them.
Highlights of this chapter include
Why configuration testing is necessary
Why configuration testing can be a huge job
A basic approach to configuration testing
How to find the hardware you need to test with
What to do if you're not testing software for a desktop computer
The next time you're in one of those computer superstores, look at a few software boxes and read over the system requirements. You'll see things such as PC with a Pentium 4 processor, 1024x768 32-bit color monitor, 32-bit audio card, game port, and so on. Configuration testing is the process of checking the operation of the software you're testing with all these various types of hardware. Consider the different configuration possibilities for a standard Windows-based PC used in homes and businesses:
The PC. There are several well-known computer manufacturers, such as Dell, Gateway, Hewlett Packard, and others. Each one builds PCs using components designed themselves or obtained from other manufacturers. Many hobbyists even build their own PCs using off-the-shelf components available at computer superstores.
Components. Most PCs are modular and built up from various system boards, component cards, and other internal devices such as disk drives, CD-ROM drives, DVD burners, video, sound, fax modem, and network cards (see Figure 8.1). There are TV tuner cards and specialized cards for video capture and home automation. There are even input/output cards that can give a PC the ability to control a small factory! These internal devices are built by hundreds of different manufacturers.
Peripherals. Peripherals, shown in Figure 8.2, are the printers, scanners, mice, keyboards, monitors, cameras, joysticks, and other devices that plug into your system and operate externally to the PC.
Interfaces. The components and peripherals plug into your PC through various types of interface connectors (see Figure 8.3). These interfaces can be internal or external to the PC. Typical names for them are ISA, PCI, USB, PS/2, RS/232, RJ-11, RJ-45, and Firewire. There are so many different possibilities that hardware manufacturers will often create the same peripheral with different interfaces. It's possible to buy the exact same mouse in three different configurations!
Options and memory. Many components and peripherals can be purchased with different hardware options and memory sizes. Printers can be upgraded to support extra fonts or accept more memory to speed up printing. Graphics cards with more memory can support additional colors and higher resolutions. The system board can have different versions of BIOS (its Basic Input Output System) and, of course, various amounts of memory.
Device Drivers. All components and peripherals communicate with the operating system and the software applications through low-level software called device drivers. These drivers are often provided by the hardware device manufacturer and are installed when you set up the hardware. Although technically they are software, for testing purposes they are considered part of the hardware configuration.
If you're a tester gearing up to start configuration testing on a piece of software, you need to consider which of these configuration areas would be most closely tied to the program. A highly graphical computer game will require lots of attention to the video and sound areas. A greeting card program will be especially vulnerable to printer issues. A fax or communications program will need to be tested with numerous modems and network configurations.
You may be wondering why this is all necessary. After all, there are standards to meet for building hardware, whether it's for an off-the-shelf PC or a specialized computer in a hospital. You would expect that if everyone designed their hardware to a set of standards, software would just work with it without any problems. In an ideal world, that would happen, but unfortunately, standards aren't always followed. Sometimes, the standards are fairly loose—call them guidelines. Card and peripheral manufacturers are always in tight competition with one another and frequently bend the rules to squeeze in an extra feature or to get in a last little bit of performance gain. Often the device drivers are rushed and packed into the box as the hardware goes out the door. The result is software that doesn't work correctly with certain hardware configurations.
Those configuration bugs can bite hard. Remember the Disney Lion King bug described in Chapter 1, “Software Testing Background”? That was a configuration problem. The software's sound didn't work only on a few, but very popular, hardware configurations. If you've ever been playing a game or using a graphics program and the colors suddenly go crazy or pieces of windows get left behind as you drag them, you've probably discovered a display adapter configuration bug. If you've ever spent hours (or days!) trying to get an old program to work with your new printer, it's probably a configuration bug.
NOTE
The sure way to tell if a bug is a configuration problem and not just a bug that would occur in any configuration is to perform the exact same operation that caused the problem, step by step, on another computer with a completely different hardware setup. If the bug doesn't occur, it's very likely a specific configuration problem that's revealed by the unique hardware used in the test.
Assume that you test your software on a unique configuration and discover a problem. Who should fix the bug—your team or the hardware manufacturer? That could turn out to be a million-dollar question.
First you need to figure out where the problem lies. This is usually a dynamic white-box testing and programmer-debugging effort. A configuration problem can occur for several reasons, all requiring someone to carefully examine the code while running the software under different configurations to find the bug:
Your software may have a bug that appears under a broad class of configurations. An example is if your greeting card program works fine with laser printers but not with inkjet printers.
Your software may have a bug specific only to one particular configuration—it doesn't work on the OkeeDoKee Model BR549 InkJet Deluxe printer.
The hardware device or its device drivers may have a bug that only your software reveals. Maybe your software is the only one that uses a unique display card setting. When your software is run with a specific video card, the PC crashes.
The hardware device or its device drivers may have a bug that can be seen with lots of other software—although it may be particularly obvious with yours. An example would be if a specific printer driver always defaulted to draft mode and your photo printing software had to set it to high-quality every time it printed.
In the first two cases, it seems fairly straightforward that your project team is responsible for fixing the bug. It's your problem. You should fix it.
In the last two cases, things get blurry. Say the bug is in a printer and that printer is the most popular in the world, with tens of millions in use. Your software obviously needs to work with that printer. It may take the printer vendor months to fix the problem (if it does at all) so your team will need to make changes to your software, even though the software is doing everything right, to work around the bug.
In the end, it's your team's responsibility to address the problem, no matter where it lies. Your customers don't care why or how the bug is happening, they just want the new software they purchased to work on their system's configuration.
The job of configuration testing can be a huge undertaking. Suppose that you're testing a new software game that runs under Microsoft Windows. The game is very graphical, has lots of sound effects, allows multiple players to compete against each other over the phone lines, and can print out game details for strategy planning.
At the least, you'll need to consider configuration testing with different graphics cards, sound cards, modems, and printers. The Windows Add New Hardware Wizard (see Figure 8.4) allows you to select hardware in each of these categories—and 25 others.
Figure 8.4. The Microsoft Windows Add New Hardware Wizard dialog box allows you to add new hardware to your PC's current configuration.
Under each hardware category are the different manufacturers and models (see Figure 8.5). Keep in mind, these are only the models with support built into Windows. Many other models provide their own setup disks with their hardware.
If you decided to perform a full, comprehensive configuration test, checking every possible make and model combination, you'd have a huge job ahead of you.
Say there are approximately 336 possible display cards, 210 sound cards, 1500 modems, and 1200 printers. The number of test combinations is 336 × 210 × 1500 × 1200, for a total in the billions—way too many to consider!
If you limited your testing to exclude combinations, just testing each card individually at about 30 minutes per configuration, you'd be at it for about a year. Keep in mind that's just one pass through the configurations. It's not uncommon with bug fixes to run two or three configuration test passes before a product is released.
The answer to this mess, as you've hopefully deduced, is equivalence partitioning. You need to figure out a way to reduce the huge set of possible configurations to the ones that matter the most. You'll assume some risk by not testing everything, but that's what software testing is all about.
The decision-making process that goes into deciding what devices to test with and how they should be tested is a fairly straightforward equivalence partition project.
What's important, and what makes the effort a success or not, is the information you use to make the decisions. If you're not experienced with the hardware that your software runs on, you should learn as much as you can and bring in other experienced testers or programmers to help you. Ask a lot of questions and make sure you get your plan approved.
The following sections show the general process that you should use when planning your configuration testing.
Does your application print? If so, you'll need to test printers. If it has sound, you'll need to test sound cards. If it's a photo or graphics program, you'll likely need scanners and digital cameras. Look closely at your software feature set to make sure that you cover everything. Put your software disk on a table and ask yourself what hardware pieces you need to put together to make it work.
If you're putting out a cutting-edge graphics program, you probably don't need to test that it prints well on a 1987 black-and-white dot-matrix printer. Work with your sales and marketing people to create a list of hardware to test with. If they can't or won't help, check out the recent equipment reviews from PC Magazine or Mac World to get an idea of what hardware is available and what is (and was) popular. Both magazines, as well as others, have annual reviews of printers, sound cards, display adapters and other peripherals.
Do some research to see if some of the devices are clones of each other and therefore equivalent—falling under the same equivalence partition. For example, a printer manufacturer may license their printer to another company that puts a different cover and label on it. From your standpoint, it's the same printer.
Decide what device drivers you're going to test with. Your options are usually the drivers included with the operating system, the drivers included with the device, or the latest drivers available on the hardware or operating system company's website. Usually, all three are different. Ask yourself what customers have or what they can get.
Color printers can print in black and white or color, they can print in different quality modes, and can have settings for printing photos or text. Display cards, as shown in Figure 8.6, can have different color settings and screen resolutions.
Figure 8.6. The display properties of number of colors and screen area are possible configurations for a display card.
Every device has options, and your software may not need to support all of them. A good example of this is computer games. Many require a minimum number of display colors and resolution. If the configuration has less than that, they simply won't run.
Given that you don't have the time or budget to test everything, you need to reduce the thousands of potential configurations into the ones that matter—the ones you're going to test.
One way to do this is to put all the configuration information into a spreadsheet with columns for the manufacturer, model, driver versions, and options. Figure 8.7 shows an example of a table that identifies various printer configurations. You and your team can review the chart and decide which configuration you want to test.
Notice that Figure 8.7 also has columns for information about the device's popularity, its type, and its age. When creating your equivalence partitions, you might decide that you want to test only the most popular printers or ones that are less than five years old. With the type information—in this case, laser or inkjet—you could decide to test 75 percent lasers and 25 percent inkjets.
NOTE
Ultimately, the decision-making process that you use to equivalence partition the configurations into smaller sets is up to you and your team. There is no right formula. Every software project is different and will have different selection criteria. Just make sure that everyone on the project team, especially your project manager, is aware of what configurations are being tested (and not tested) and what variables went into selecting them.
The key word here is unique. You don't want to, nor do you need to, completely test your software on each configuration. You need to test only those features that are different from each other (different equivalence partitions) that interact with the hardware.
For example, if you're testing a word processor such as WordPad (see Figure 8.8), you don't need to test the file save and load feature in each configuration. File saving and loading has nothing to do with printing. A good test would be to create a document that contains different (selected by equivalence partitioning, of course) fonts, point sizes, colors, embedded pictures, and so on. You would then attempt to print this document on each chosen printer configuration.
Figure 8.8. You can use a sample document made up of different fonts and styles to configuration test a printer.
Selecting the unique features to try isn't as easy as it sounds. You should first make a black-box pass by looking at your product and pulling out the obvious ones. Then talk with others on your team, especially the programmers, to get a white-box view. You may be surprised at what features are even slightly tied to the configuration.
You'll learn the details of writing test cases in Chapter 18, “Writing and Tracking Test Cases,” but, for now, consider that you'll need to write down the steps required to test each configuration. This can be as simple as
Select and set up the next test configuration from the list.
Start the software.
Load in the file
configtest.doc.Confirm that the file is displayed correctly.
Print the document.
Confirm that there are no error messages and that the printed document matches the standard.
Log any discrepancies as a bug.
In reality, the steps would be much more involved, including more detail and specifics on exactly what to do and what to look for. The goal is to create steps that anyone can run. You'll learn more about writing test cases in Chapter 18.
You need to run the test cases and carefully log and report your results (see Chapter 19, “Reporting What You Find”) to your team and to the hardware manufacturers if necessary. As described earlier in this chapter, it's often difficult and time-consuming to identify the specific source of configuration problems. You'll need to work closely with the programmers and white-box testers to isolate the cause and decide if the bugs you find are due to your software or to the hardware.
If the bug is specific to the hardware, consult the manufacturer's website for information on reporting problems to them. Be sure to identify yourself as a software tester and what company you work for. Many companies have separate staff set up to assist software companies writing software to work with their hardware. They may ask you to send copies of your test software, your test cases, and supporting details to help them isolate the problem.
It's not uncommon for configuration testing to run the entire course of a project. Initially a few configurations might be tried, then a full test pass, then smaller and smaller sets to confirm bug fixes. Eventually you will get to a point where there are no known bugs or to where the bugs that still exist are in uncommon or unlikely test configurations. At that point, you can call your configuration testing complete.
One thing that hasn't been mentioned so far is where you obtain all this hardware. Even if you take great pains, and risk, to equivalence partition your configurations to the barest minimum, you still could require dozens of different hardware setups. It would be an expensive proposition to go out and buy everything at retail, especially if you will use the hardware only once for the one test pass. Here are a few ideas for overcoming this problem:
Buy only the configurations that you can or will use most often. A great plan is for every tester on the team to have different hardware. This may drive your purchasing department and the group that maintains your company's PCs crazy (they like everyone to have exactly the same configuration) but it's a very efficient means of always having different configurations available to test on. Even if your test team is very small, three or four people having just a few configurations would be a great help.
Contact the hardware manufacturers and ask if they will lend or even give you the hardware. If you explain that you're testing new software and you want to assure that it works on their hardware, many will do this for you. They have an interest in the outcome, too, so tell them that you'll furnish them with the results of the tests and, if you can, a copy of the finished software. It's good to build up these relationships, especially if you find a bug and need a contact person at the hardware company to report it to.
Send a memo or email to everyone in your company asking what hardware they have in their office or even at home—and if they would allow you to run a few tests on it. To perform the configuration testing, you may need to drive around town, but it's a whole lot cheaper than attempting to buy all the hardware.
If you have the budget, work with your project manager to contract out your test work to a professional configuration and compatibility test lab. These companies do nothing but configuration testing and have every piece of PC hardware known to man. Okay, maybe not that much, but they do have a lot.
These labs can help you, based on their experience, select the correct hardware to test. Then, they will allow you to come in and use their equipment, or they will provide a complete turn-key service. You provide the software, the step-by-step test process, and the expected results. They'll take it from there, running the tests and reporting what passed and what failed. Of course this can be costly, but much less so than buying the hardware yourself or worse, not testing and having customers find the problems.
If you're interested in performing a little static black-box analysis—that is, reviewing the specifications that the hardware companies use to create their products—you can look in a couple of places. Knowing some details of the hardware specifications can help you make more informed equivalence partition decisions.
For Apple hardware, visit the Apple Hardware website at developer.apple.com/hardware. There you'll find information and links about developing and testing hardware and device drivers for Apple computers. Another Apple link, developer.apple.com/testing, points you to specific testing information, including links to test labs that perform configuration testing.
For PCs, the best link is www.microsoft.com/whdc/system/platform. This site provides technical implementation guidelines, tips, and tools for developers and testers developing hardware for use with Windows.
Microsoft also publishes a set of standards for software and hardware to receive the Windows logo. That information is at msdn.microsoft.com/certification and www.microsoft.com/whdc/whql.
So, what if you're not testing software that runs on a PC or a Mac? Was this chapter a waste of your time? No way! Everything you learned can be applied to testing generic or proprietary systems, too. It doesn't matter what the hardware is or what it's connected to; if it can have variations such as memory size, CPU speed, etc, or, if it connects to another piece of hardware, software configuration issues need to be tested.
If you're testing software for an industrial controller, a network, medical devices, or a phone system, ask yourself the same questions that you would if you were testing software for a desktop computer:
What external hardware will operate with this software?
What models and versions of that hardware are available?
What features or options does that hardware support?
Create equivalence partitions of the hardware based on input from the people who work with the equipment, your project manager, or your sales people. Develop test cases, collect the selected hardware, and run the tests. Configuration testing follows the same testing techniques that you've already learned.
This chapter got you thinking about how to approach configuration testing. It's a job that new software testers are frequently assigned because it is easily defined, is a good introduction to basic organization skills and equivalence partitioning, is a task that will get you working with lots of other project team members, and is one for which your manager can readily verify the results. The downside is that it can become overwhelming.
If you're assigned to perform configuration testing for your project, take a deep breath, reread this chapter, carefully plan your work, and take your time. When you're done, your boss will have another job for you: compatibility testing, the subject of the next chapter.
These quiz questions are provided for your further understanding. See Appendix A, “Answers to Quiz Questions,” for the answers—but don't peek!