IN THIS CHAPTER
Having Other People Test Your Software
Test Sharing
Beta Testing
Outsourcing Your Testing
In Chapter 15, “Automated Testing and Test Tools,” you learned how technology in the form of tools and automation can help make your testing more efficient. Using software to test software is a great way to speed up your work and to help find bugs that might otherwise be missed.
Another means to be a more effective tester is to leverage other people. If you could get more people, even if they aren't professional testers, looking at your software before it's released, they may be able to find bugs that you and your fellow testers failed to see.
Highlights of this chapter include
Why it's important to have other people test
How you can get others looking at your software
What beta testing is and how testers are involved
How to effectively outsource your test work
Figure 16.1 shows two almost identical views of the same scene. Get out your egg timer, set it for one minute, and carefully examine both pictures looking for differences between the two. Keep a list of the differences you find in the order that you discover them.
Figure 16.1. In one minute, try to find as many differences between the two scenes as you can. Figure courtesy of www.cartoonworks.com.
After you finish looking, have several friends do the same search and compare your lists. What you'll find is that everyone has very different results. The number of differences found, the order that they were found, even which ones were found will be different. Hopefully, if you combine all the lists and throw out the duplicates, you'll have a complete list of all the differences—but even then, there still may be a few that were missed.
Software testing works exactly the same way. You're likely under a tight schedule, you find as many bugs as possible in the time you have, but someone else can come in, test the same code, and find additional bugs. It can be discouraging to see this happen. After all your hard work, you'll think, “How could I have missed such an obvious bug?” Don't worry, it's normal, and there are several reasons and solutions for it:
Having another set of eyes look at the software helps break the pesticide paradox. As Figure 16.1 demonstrates, people notice different things. Bugs that were always present that built up immunity to your view can be readily seen by a new person on the project. It's the “Emperor Has No Clothes” dilemma.
Similarly, people don't just see differently from each other, they go about their testing differently, too. Despite your best efforts in reviewing the software's specification and deciding on your test cases, a new person can come in and find a bug by trying something that you never even considered—hitting a different key, clicking the mouse faster, starting a function in a different way, and so on. It's that pesticide paradox again.
Having someone assist you in your testing helps eliminate boredom. It can get pretty monotonous running the same tests over and over, using the same software features again and again. The boredom will also cause your attention to wane and you might start missing obvious bugs.
Watching how someone else approaches a problem is a great way to learn new testing techniques. There are always new and different approaches to testing that you can add to your bag of testing tricks.
It's easy to fall into the trap of wanting to be solely responsible for testing your own piece of the software, but don't do it. There's too much to gain by having others help you out.
The odds are, unless your project is very small, there will be at least several testers testing the software. Even if there are just a few, there are things you can do to get more than your eyes looking for bugs.
One common approach is to simply swap test responsibilities with another tester for a few hours or a few days. Think of it as “You run my tests and I'll run yours.” You'll both gain an independent look at the software while still having the basic testing tasks completed. Each of you will also learn about an area of the software that you might not be familiar with—which could lead to you thinking up additional test cases to try. At a minimum, get someone else to spend time reviewing your equivalence classes and test cases. Based on their experience, they may be able to offer ideas of new and different areas to test.
A fun way to share the testing tasks is to schedule a bug bash. A bug bash is a period of time (usually a couple hours) in which the entire test team stops their normally assigned testing tasks to participate in the bash. In a bug bash, a specific area of the software is selected and all the testers concentrate their testing on that one area or set of features. The selection might be an area that's been especially buggy to see if there are still more problems lurking about. Or, it might be an area that's suspiciously bug free. A bug bash could determine if the bugs have been missed by normal testing or if it's just well written code. There are lots of potential criteria for choosing the area, but ultimately a bug bash gets many different people looking at one particular area of the software for bugs.
One of your greatest allies in your quest to find bugs is your product support or customer service team—the people who will talk with customers when they call or email with questions and problems. These people are obviously very sensitive to bugs and are a great resource to leverage for helping you test. Find out who will be supporting your product once it's released and ask them to participate in your test sharing activities. You'll be amazed at the bugs they'll find for you.
NOTE
Probably the most common class of calls that product support people take is in the area of usability problems. Many of the calls are from people simply trying to figure out how to use the software. For this reason, it's a good idea to get your product support team helping you test the product early in the design cycle to help identify and fix usability bugs.
The test sharing ideas presented so far have been internal methods—that is, the people that would help you share the testing are either from your test team or the project development team. Another common method for having others verify and validate the software is a process called beta testing.
Beta testing is the term used to describe the external testing process in which the software is sent out to a select group of potential customers who use it in a real-world environment. Beta testing usually occurs toward the end of the product development cycle and ideally should just be a validation that the software is ready to release to real customers.
The goals of a beta test can vary considerably from getting the press to write early reviews of the software to user interface validation to a last-ditch effort in finding bugs. As a tester, you need to make it known to the person managing the beta testing what, if anything, you want to achieve from it.
From a test standpoint, there are several things to think about when planning for or relying on a beta test:
Who are the beta testers? Since a beta test can have different goals, it's important to understand who the beta participants are. For example, you may be interested in identifying any remaining usability bugs left in the software, but the beta testers may all be experienced techies who are more concerned with the low-level operation and not usability. If your area of the software is to be beta tested, make sure that you define what types of beta testers you need in the program so that you can receive the most benefit out of it.
Similarly, how will you know if the beta testers even use the software? If 1,000 beta testers had the software for a month and reported no problems, would that mean there were no bugs, that they saw bugs but didn't report them, or that the disks were lost in the mail? It's not uncommon for beta testers to let the software sit for days or weeks before they try to use it, and when they do, only use it for a limited time and a limited set of features. Make sure that you or someone running the beta program follows up with the participants to make sure that they're using the software and meeting the plan's goals.
Beta tests can be a good way to find compatibility and configuration bugs. As you learned in Chapters 8, “Configuration Testing,” and 9, “Compatibility Testing,” it's difficult to identify and test a representative sample of all the real-world hardware and software setups. If your beta test participants have been wisely chosen to represent your target customers, they should do a good job finding configuration and compatibility problems for you.
Usability testing is another area that beta testing can contribute to if the participants are well chosen—a good mix of experienced and inexperienced users. They'll be seeing the software for the first time and will readily find anything that's confusing or difficult to use.
Besides configuration, compatibility, and usability, beta tests are surprisingly poor ways to find bugs. The participants often don't have a lot of time to use the software, so they won't find much more than superficial, obvious problems—ones that you likely already know about. And, because beta testing usually occurs near the end of the development cycle, there's not much time to fix bugs that are found.
A beta test program can take up a lot of a tester's time. A common job for a new tester is to work with the beta customers to help solve their problems, answer their questions, and confirm the bugs they find. If you're assigned this task, you'll also need to work with your fellow testers to understand how the bugs slipped through to the beta testers and how to improve the test cases so that the bugs are found internally in the future. All this can be a full-time job, leaving little room to do any real testing yourself.
If you and your team plan on holding a beta test program, make arrangements in advance, preferably when the product's schedule is being defined. Make sure that the beta test's goals mesh with goals that you and your testing team want out of it and work closely with the person (or team) managing the beta program to keep testing's voice heard.
Beta testing can prove to be a valuable method for getting solid, independent test data back on your software, but to be effective it must be properly defined and managed—you could almost say it needs to be tested.
A common practice in many corporations is to outsource or subcontract a portion of the test work to other companies that specialize in various aspects of software testing. Although this may sound more cumbersome and more expensive than having the work done by testers on the product team, it can be an effective way to share the testing if done properly.
Configuration and compatibility testing are typically good choices for outsourcing. They usually require a large test lab containing many different hardware and software combinations and a staff of several people to manage it. Most small software companies can't afford the overhead and expense for maintaining these test labs, so it makes more sense for them to outsource this testing to companies who make it their business to perform configuration and compatibility tests.
Localization testing is another example that often lends itself to outsourcing. Unless you have a very large test team, it would be impossible to staff testers that speak all the different languages that your product supports. It would be beneficial to have a couple of foreign language speaking testers on your team to look for fundamental localization problems, but it's probably more efficient to outsource the testing of the specific languages. A company that specializes in localization testing would have testers on staff that speak many different languages who are also experienced testers.
As a new software tester, you likely won't be asked to make decisions on what testing tasks will be outsourced, but you may need to work with an outsourcing company if it's testing areas of the software that you're responsible for. The success, or failure, of the outsourcing job may well depend on you. Here's a list of things to consider and to discuss with your test manager or project manager to help make the job run more smoothly:
What exactly are the testing tasks that the testing company is to perform? Who will define them? Who will approve them?
What schedule will they follow? Who will set the schedule? What happens if the deadline is missed?
What deliverables are you to provide to the testing company? The software's specification, periodic software updates, and test cases are some examples.
What deliverables are they to provide to you? A list of the bugs they find would be the minimum.
How will you communicate with them? Phone, email, Internet, central database, daily visit? Who are the points of contact at both ends?
How will you know if the testing company is meeting your expectations? How will they know if they're meeting your expectations?
These aren't rocket science issues, but they're unfortunately often overlooked in the rush to outsource a testing task. Throwing software over a wall and telling a company to “just test it” is ripe for disaster. But, spending some time up-front planning the testing can make outsourcing a very effective means to perform tests that you otherwise couldn't handle because of limited resources.
What you should take away from this chapter and from Chapter 15 is that you should leverage whatever means possible to be an effective tester. One situation might dictate using technology, another might require extra people, another might need just plain old brute force manual testing. Every software testing problem will be unique, and you'll learn something new with each one. Experiment, try different approaches, watch what others do, but always strive to find the best way to make your testing more efficient and more likely to locate those bugs.
This chapter wraps up the book's topics on how to perform software testing. It's been fun. You've learned about the software development process, the basic techniques for software testing, how to apply your skills, and how to supplement what you can do. In Part V, “Working with Test Documentation,” you'll see how to pull together everything you've learned so far: how to plan and organize your testing tasks, how to properly record and track the bugs you find, and how to make sure those bugs get fixed.
These quiz questions are provided for your further understanding. See Appendix A, “Answers to Quiz Questions,” for the answers—but don't peek!