IN THIS CHAPTER
Compatibility Testing Overview
Platform and Application Versions
Standards and Guidelines
Data Sharing Compatibility
In Chapter 8, “Configuration Testing,” you learned about hardware configuration testing and how to assure that software works properly with the hardware it was designed to run on and connect with. This chapter deals with a similar area of interaction testing—checking that your software operates correctly with other software.
Testing whether one program plays well with others has become increasingly important as consumers demand the ability to share data among programs of different types and from different vendors and take advantage of the ability to run multiple programs at once.
It used to be that a program could be developed as a standalone application. It would be run in a known, understood, benign environment, isolated from anything that could corrupt it. Today, that program likely needs to import and export data to other programs, run with different operating systems and Web browsers, and interoperate with other software being run simultaneously on the same hardware. The job of software compatibility testing is to make sure that this interaction works as users would expect.
The highlights of this chapter include
What it means for software to be compatible
How standards define compatibility
What platforms are and what they mean for compatibility
Why being able to transfer data among software applications is the key to compatibility
Software compatibility testing means checking that your software interacts with and shares information correctly with other software. This interaction could occur between two programs simultaneously running on the same computer or even on different computers connected through the Internet thousands of miles apart. The interaction could also be as simple as saving data to a floppy disk and hand-carrying it to another computer across the room.
Examples of compatible software are
Cutting text from a web page and pasting it into a document opened in your word processor
Saving accounting data from one spreadsheet program and then loading it into a completely different spreadsheet program
Having photograph touchup software work correctly on different versions of the same operating system
Having your word processor load in the names and addresses from your contact management program and print out personalized invitations and envelopes
Upgrading to a new database program and having all your existing databases load in and work just as they did with the old program
What compatibility means for your software depends on what your team decides to specify and what levels of compatibility are required by the system that your software will run on. Software for a standalone medical device that runs its own operating system, stores its data on its own memory cartridges, and doesn't connect to any other device would have no compatibility considerations. However, the fifth version of a word processor (see Figure 9.1) that reads and writes different files from other word processors, allows multiuser editing over the Internet, and supports inclusion of embedded pictures and spreadsheets from various applications has a multitude of compatibility considerations.
Figure 9.1. Compatibility across different software applications can quickly become very complicated.
If you're assigned the task of performing software compatibility testing on a new piece of software, you'll need to get the answers to a few questions:
What other platforms (operating system, web browser, or other operating environment) and other application software is your software designed to be compatible with? If the software you're testing is a platform, what applications are designed to run under it?
What compatibility standards or guidelines should be followed that define how your software should interact with other software?
What types of data will your software use to interact and share information with other platforms and software?
Gaining the answers to these questions is basic static testing—both black-box and white-box. It involves thoroughly analyzing the specification for the product and any supporting specifications. It could also entail discussions with the programmers and possibly close review of the code to assure that all links to and from your software are identified. The rest of this chapter discusses these questions in more detail.
Selecting the target platforms or the compatible applications is really a program management or a marketing task. Someone who's very familiar with the customer base will decide whether your software is to be designed for a specific operating system, web browser, or some other platform. They'll also identify the version or versions that the software needs to be compatible with. For example, you've probably seen notices such as these on software packages or startup screens:
Works best with AOL 9.0
Requires Windows XP or greater
For use with Linux 2.6.10
This information is part of the specification and tells the development and test teams what they're aiming for. Each platform has its own development criteria and it's important, from a project management standpoint, to make this platform list as small as possible but still fill the customer's needs.
Two terms you'll hear regarding compatibility testing are backward compatible and forward compatible. If something is backward compatible, it will work with previous versions of the software. If something is forward compatible, it will work with future versions of the software.
The simplest demonstration of backward and forward compatibility is with a .txt or text file. As shown in Figure 9.2, a text file created using Notepad 98 running under Windows 98 is backward compatible all the way back to MS-DOS 1.0. It's also forward compatible to Windows XP service pack 2 and likely will be beyond that.
Testing that multiple versions of platforms and software applications work properly with each other can be a huge task. Consider the situation of having to compatibility test a new version of a popular operating system. The programmers have made numerous bug fixes and performance improvements and have added many new features to the code. There could be tens or hundreds of thousands of existing programs for the current versions of the OS. The project's goal is to be 100 percent compatible with them. See Figure 9.3.
Figure 9.3. If you compatibility test a new platform, you must check that existing software applications work correctly with it.
This is a big job, but it's just another example of how equivalence partitioning can be applied to reduce the amount of work.
NOTE
To begin the task of compatibility testing, you need to equivalence partition all the possible software combinations into the smallest, effective set that verifies that your software interacts properly with other software.
In short, you can't test all the thousands of software programs on your operating system, so you need to decide which ones are the most important to test. The key word is important. The criteria that might go into deciding what programs to choose could be
Popularity. Use sales data to select the top 100 or 1,000 most popular programs.
Age. You might want to select programs and versions that are less than three years old.
Type. Break the software world into types such as painting, writing, accounting, databases, communications, and so on. Select software from each category for testing.
Manufacturer. Another criteria would be to pick software based on the company that created it.
Just as in hardware configuration testing, there is no right “textbook” answer. You and your team will need to decide what matters most and then use that criteria to create equivalence partitions of the software you need to test with.
The previous example dealt with compatibility testing a new operating system platform. The same issues apply to testing a new application (see Figure 9.4). You need to decide what platform versions you should test your software on and what other software applications you should test your software with.
So far in this chapter you've learned about selecting the software that you'll compatibility test with your program. Now, it's time to look at how you'll approach the actual testing. Your first stop should be researching the existing standards and guidelines that might apply to your software or the platform.
There are really two levels of these requirements: high-level and low-level. It may be a misnomer to refer to them as high and low, but in a sense, that's really what they are. High-level standards are the ones that guide your product's general operation, its look and feel, its supported features, and so on. Low-level standards are the nitty-gritty details, such as the file formats and the network communications protocols. Both are important and both need to be tested to assure compatibility.
Will your software run under Windows, Mac, or Linux operating systems? Is it a web application? If so, what browsers will it run on? Each of these is considered a platform and most have their own set of standards and guidelines that must be followed if an application is to claim that it's compatible with the platform.
An example of this is the Certified for Microsoft Windows logo (see Figure 9.5). To be awarded this logo, your software must undergo and pass compatibility testing by an independent testing laboratory. The goal is to assure that the software runs stably and reliably on the operating system.
Figure 9.5. The Certified for Microsoft Windows logo signifies that the software meets all the criteria defined by the guidelines.
A few examples of the logo requirements are that the software
Supports mice with more than three buttons
Supports installation on disk drives other than C: and D:
Supports filenames longer than the DOS 8.3 format
Doesn't read, write, or otherwise use the old system files
win.ini,system.ini,autoexec.bat, orconfig.sys
These may sound like simple, matter-of-fact requirements, but they're only four items out of a 100+ page document. It's a great deal of work to assure that your software complies with all the logo requirements, but it makes for much more compatible software.
NOTE
The details of the Windows logo can be obtained at msdn.microsoft.com/certification. Details for using the Apple Mac logo are at developer.apple.com/testing.
Low-level standards are, in a sense, more important than the high-level standards. You could create a program that would run on Windows that didn't have the look and feel of other Windows software. It wouldn't be granted the Certified for Microsoft Windows logo. Users might not be thrilled with the differences from other applications, but they could use the product.
If, however, your software is a graphics program that saves its files to disk as .pict files (a standard Macintosh file format for graphics) but the program doesn't follow the standard for .pict files, your users won't be able to view the files in any other program. Your software wouldn't be compatible with the standard and would likely be a short-lived product.
Similarly, communications protocols, programming language syntax, and any means that programs use to share information must adhere to published standards and guidelines.
These low-level standards are often taken for granted, but from a tester's perspective must be tested. You should treat low-level compatibility standards as an extension of the software's specification. If the software spec states, “The software will save and load its graphics files as .bmp, .jpg, and .gif formats,” you need to find the standards for these formats and design tests to confirm that the software does indeed adhere to them.
NOTE
Don't necessarily trust your team's interpretation of the standards or guidelines. Look them up yourself and develop your tests directly from the source. Remember the difference between precision and accuracy? You don't want your product's compatibility code to be perfectly precise, but totally inaccurate.
The sharing of data among applications is what really gives software its power. A well-written program that supports and adheres to published standards and allows users to easily transfer data to and from other software is a great compatible product.
The most familiar means of transferring data from one program to another is saving and loading disk files. As discussed in the previous section, adhering to the low-level standards for the disk and file formats is what makes this sharing possible. Other means, though, are sometimes taken for granted but still need to be tested for compatibility. Here are a few examples:
File save and file load are the data-sharing methods that everyone is aware of. You save your data to a floppy disk (or some other means of magnetic or optical storage) and then hand carry it over to another computer running different software and load it in. The data format of the files needs to meet standards for it to be compatible on both computers.
File export and file import are the means that many programs use to be compatible with older versions of themselves and with other programs. Figure 9.6 shows the Microsoft Word File Open dialog box and some of the 23 different file formats that can be imported into the word processor.
To test the file import feature, you would need to create test documents in each compatible file format—probably using the original software that wrote that format. Those documents would need to have equivalence partitioned samples of the possible text and formatting to check that the importing code properly converts it to the new format.
Cut, copy, and paste are the most familiar methods for sharing data among programs without transferring the data to a disk. In this case, the transfer happens in memory through an intermediate program called the Clipboard. Figure 9.7 shows how this transfer occurs.
Figure 9.7. The System Clipboard is a temporary holding place for different types of data that's being copied from one application to another.
The Clipboard is designed to hold several different data types. Common ones in Windows are text, pictures, and sounds. These data types can also be different formats—for example, the text can be plain old text, HTML, or rich text. Pictures can be bitmaps, metafiles, or
.tifs.Whenever a user performs a cut or copy, the data that's chosen is placed in the Clipboard. When he does a paste, it's copied from the Clipboard to the destination software. Some applications may only accept certain data types or formats being pasted into them—for example, a painting program may accept pictures, but not text.
If you're compatibility testing a program, you need to make sure that its data can be properly copied in and out of the Clipboard to other programs. This feature is so transparent and so frequently used, people forget that there's a lot of code behind making sure that it works and is compatible across lots of different software.
DDE (pronounced D-D-E), COM (for Component Object Model), and OLE (pronounced oh-lay) are the methods in Windows of transferring data between two applications. DDE stands for Dynamic Data Exchange and OLE stands for Object Linking and Embedding. Other platforms support similar methods.
There's no need to get into the gory details of these technologies in this book, but the primary difference between these two methods and the Clipboard is that with DDE and OLE data can flow from one application to the other in real time. Cutting and copying is a manual operation. With DDE and OLE, the transfer can happen automatically.
An example of how these might be used could be a written report done in a word processor that has a pie-chart created by a spreadsheet program. If the report's author copied and pasted the chart into the report, it would be a snapshot in time of the data. If, however, the author linked the pie chart into the report as an object, when the underlying numbers for the chart change, the new graphics will automatically appear in the report.
This is all pretty fancy, yes, but it's also a testing challenge to make sure that all the object linking, embedding, and data exchanging happens correctly.
This chapter introduced you to the basics of compatibility testing. In reality, an entire book could be written on the subject, and a single chapter doesn't do the topic justice. Every platform and every application is unique, and the compatibility issues on one system can be totally different than on another.
As a new software tester, you may be assigned a task of compatibility testing your software. That may seem strange, given that it's potentially such a large and complex task, but you'll likely be assigned just a piece of the entire job. If your project is a new operating system, you may be asked to compatibility test just word processors or graphics programs. If your project is an applications program, you may be asked to compatibility test it on several different platforms.
Each is a manageable task that you can easily handle if you approach your testing with these three things in mind:
Equivalence partition all the possible choices of compatible software into a manageable set. Of course, your project manager should agree with your list and understand the risk involved in not testing everything.
Research the high-level and low-level standards and guidelines that might apply to your software. Use these as extensions of your product's specification.
Test the different ways that data can flow between the software programs you're testing. This data exchange is what makes one program compatible with another.
These quiz questions are provided for your further understanding. See Appendix A, “Answers to Quiz Questions,” for the answers—but don't peek!