IN THIS CHAPTER
Types of Software Documentation
The Importance of Documentation Testing
What to Look for When Reviewing Documentation
The Realities of Documentation Testing
In Chapter 2, “The Software Development Process,” you learned that there's a great deal of work and a great number of non-software pieces that make up a software product. Much of that non-software is its documentation.
In simpler days, software documentation was at most a readme file copied onto the software's floppy disk or a short 1-page insert put into the box. Now it's much, much more, sometimes requiring more time and effort to produce than the software itself.
As a software tester, you typically aren't constrained to just testing the software. Your responsibility will likely cover all the parts that make up the entire software product. Assuring that the documentation is correct is your job, too.
In this chapter you'll learn about testing software documentation and how to include it in your overall software test effort. Highlights of this chapter include
The different types of software documentation
Why documentation testing is important
What to look for when testing documentation
If your software's documentation consisted of nothing but a simple readme file, testing it wouldn't be a big deal. You'd make sure that it included all the material that it was supposed to, that everything was technically accurate, and (for good measure) you might run a spell check and a virus scan on the disk. That would be it. But, the days of documentation consisting of just a readme file are gone.
Today, software documentation can make up a huge portion of the overall product. Sometimes, it can seem as if the product is nothing but documentation with a little bit of software thrown in.
Here's a list of software components that can be classified as documentation. Obviously, not all software will have all the components, but it's possible:
Packaging text and graphics. This includes the box, carton, wrapping, and so on. The documentation might contain screen shots from the software, lists of features, system requirements, and copyright information.
Marketing material, ads, and other inserts. These are all the pieces of paper you usually throw away, but they are important tools used to promote the sale of related software, add-on content, service contracts, and so on. The information for them must be correct for a customer to take them seriously.
Warranty/registration. This is the card that the customer fills out and sends in to register the software. It can also be part of the software, being displayed onscreen for the user to read, acknowledge, and complete online.
EULA. Pronounced “you-la,” it stands for End User License Agreement. This is the legal document that the customer agrees to that says, among other things, that he won't copy the software nor sue the manufacturer if he's harmed by a bug. The EULA is sometimes printed on the envelope containing the media—the floppy or CD. It also may pop up onscreen during the software's installation. An example is shown in Figure 12.1.
Labels and stickers. These may appear on the media, on the box, or on the printed material. There may also be serial number stickers and labels that seal the EULA envelope. Figure 12.2 shows an example of a disk label and all the information that needs to be checked.
Installation and setup instructions. Sometimes this information is printed directly on the discs, but it also can be included on the CD sleeve or as a CD jewel box insert. If it's complex software, there could be an entire installation manual.
User's manual. The usefulness and flexibility of online manuals has made printed manuals much less common than they once were. Most software now comes with a small, concise “getting started”–type manual with the detailed information moved to online format. The online manuals can be distributed on the software's media, on a website, or a combination of both.
Online help. Online help often gets intertwined with the user's manual, sometimes even replacing it. Online help is indexed and searchable, making it much easier for users to find the information they're looking for. Many online help systems allow natural language queries so users can type
Tell me how to copy text from one program to anotherand receive an appropriate response.Tutorials, wizards, and CBT (Computer Based Training). These tools blend programming code and written documentation. They're often a mixture of both content and high-level, macro-like programming and are often tied in with the online help system. A user can ask a question and the software then guides him through the steps to complete the task. Microsoft's Office Assistant, sometimes referred to as the “paper clip guy” (see Figure 12.3), is an example of such a system.
Samples, examples, and templates. An example of these would be a word processor with forms or samples that a user can simply fill in to quickly create professional-looking results. A compiler could have snippets of code that demonstrate how to use certain aspects of the language.
Error messages. These have already been discussed a couple times in this book as an often neglected area, but they ultimately fall under the category of documentation.
Software users consider all these individual non-software components parts of the overall software product. They don't care whether the pieces were created by a programmer, a writer, or a graphic artist. What they care about is the quality of the entire package.
NOTE
If the installation instructions are wrong or if an incorrect error message leads them astray, users will view those as bugs with the software—ones that should have been found by a software tester.
Good software documentation contributes to the product's overall quality in three ways:
It improves usability. Remember from Chapter 11, “Usability Testing,” all the issues related to a product's usability? Much of that usability is related to the software documentation.
It improves reliability. Reliability is how stable and consistent the software is. Does it do what the user expects and when he expects it? If the user reads the documentation, uses the software, and gets unexpected results, that's poor reliability. As you'll see in the rest of this chapter, testing the software and the documentation against each other is a good way to find bugs in both of them.
It lowers support costs. In Chapter 2 you learned that problems found by a customer can cost 10 to 100 times as much as if they were found and fixed early in the product's development. The reason is that users who are confused or run into unexpected problems will call the company for help, which is expensive. Good documentation can prevent these calls by adequately explaining and leading users through difficult areas.
NOTE
As a software tester, you should treat the software's documentation with the same level of attention and give it the same level of effort that you do the code. They are one in the same to the user. If you're not asked to test the documentation, be sure to raise this as an issue and work to have it included in your overall testing plan.
Testing the documentation can occur on two different levels. If it's non-code, such as a printed user's manual or the packaging, testing is a static process much like what's described in Chapters 4, “Examining the Specification,” and 6, “Examining the Code.” Think of it as technical editing or technical proofreading. If the documentation and code are more closely tied, such as with a hyperlinked online manual or with a helpful paper clip guy, it becomes a dynamic test effort that should be checked with the techniques you learned in Chapters 5, “Testing the Software with Blinders On,” and 7, “Testing the Software with X-Ray Glasses.” In this situation, you really are testing software.
NOTE
Whether or not the documentation is code, a very effective approach to testing it is to treat it just like a user would. Read it carefully, follow every step, examine every figure, and try every example. If there is sample code, type it in and make sure it works as described. With this simple real-world approach, you'll find bugs both in the software and the documentation.
Table 12.1 is a simple checklist to use as a basis for building your documentation test cases.
Table 12.1. A Documentation Testing Checklist
Finally, if the documentation is software driven, test it as you would the rest of the software. Check that the index list is complete, that searching finds the correct results, and that the hyperlinks and hotspots jump to the correct pages. Use equivalence partition techniques to decide what test cases to try.
To close this chapter, it's important for you to learn a few things that make documentation development and testing a bit different from software development. Chapter 3 was titled “The Realities of Software Testing.” You might call these issues the realities of documentation testing:
Documentation often gets the least attention, budget, and resources. There seems to be the mentality that it's a software project first and foremost and all the other stuff is less important. In reality, it's a software product that people are buying and all that other stuff is at least as important as the bits and bytes. If you're responsible for testing an area of the software, make sure that you budget time to test the documentation that goes along with that code. Give it the same attention that you do the software and if it has bugs, report them.
It's possible that the people writing the documentation aren't experts in what the software does. Just as you don't have to be an accounting expert to test a spreadsheet program, the writer doesn't have to be an expert in the software's features to write its documentation. As a result, you can't rely on the person creating the content to make sense out of poorly written specs or complex or unclear product features. Work closely with writers to make sure they have the information they need and that they're up-to-date with the product's design. Most importantly, tell them about difficult-to-use or difficult-to-understand areas of the code that you discover so they can better explain those areas in the documentation.
Printed documentation takes time to produce, sometimes weeks or even months. Because of this time difference, a software product's documentation may need to be finalized—locked down—before the software is completed. If the software functionality changes or bugs are discovered during this critical period, the documentation can't be changed to reflect them. That's why the
readmefile was invented. It's how those last-minute changes are often communicated to users. The solution to this problem is to have a good development model, follow it, hold your documentation release to the last possible minute, and release as much documentation as possible, in electronic format, with the software.
Hopefully this chapter opened your eyes to how much more there can be to a software product than the code the programmers write. The software's documentation, in all its forms, created by writers, illustrators, indexers, and so on, can easily take more effort to develop and test than the actual software.
From the user's standpoint, it's all the same product. An online help index that's missing an important term, an incorrect step in the installation instructions, or a blatant misspelling are bugs just like any other software failure. If you properly test the documentation, you'll find the bugs before your users do.
In the next chapter you'll learn about applying your testing skills to an area that's in the news almost daily—software security. It's an area that will require particular attention in every task you perform as a software tester from early code and specification reviews to testing the documentation.
These quiz questions are provided for your further understanding. See Appendix A, “Answers to Quiz Questions,” for the answers—but don't peek!
