There’s an expression routinely vouchsafed in advertisements posted on Google’s restroom walls that reads “Debugging sucks. Testing rocks.” This mantra is powerful. Debugging requires you to deconstruct and disassemble your software until you get to the point where you find the problem. That’s effectively moving backward. Moving backward is the opposite of shipping. In addition to helping you feel confident that you’re doing the right thing, test-driven development helps your team survive in complicated systems environments, because as soon as another engineer or team breaks an interface you depend on, the tests will fail.

Test-driven development is covered extensively in other references (see Appendix C), but here’s an overview of the process:

Pretty straightforward, right? It is. It takes some discipline, though. What’s extra-great about test-driven development is that regressions are easier to spot because each build runs tests automatically. Look into software like JUnit (for Java-based unit testing) to automate your build and verification procedures.

No matter how good your engineering team is and regardless of how many unit tests they have written, you will have bugs. Your best plan of action to find these bugs is to hire or appoint someone to be the test guru. This test lead will be the primary owner of release quality and a critical partner to product management, engineering, and marketing leads. Test leads are responsible for making sure that the test cases are well written, cover the right areas, and are well executed. A great test lead will continuously train less experienced testers and help design great test automation architecture. If you have a really strong test lead, that individual will be sharp enough to push the engineering team to build more and better unit tests.

Another key reason why you want to start with a great test lead is that the test team culture is frequently unlike the engineering team culture. Your test team is trying to discover problems all day long. In a typical but poorly run engineering team, the test team generates complaints daily, and that can be hard to take. The processes, disciplines, and standards are a bit different than typical engineering teams, so having a solid test lead who can help manage the test team will help you immensely, even if you’ve done a great job of embedding testing with engineering.

Your test lead will also help you solve the unique problem of hiring testers. Brilliant test engineers are hard to come by because most folks who can write great software want to write their own software, not test the software your engineering team wrote. There are two ways to extract yourself from this dilemma: maintain a lower hiring bar and hire managers, or maintain a high hiring bar and hire contractors. As I’ll describe next, there are advantages to both approaches, but I favor hiring contractors.

Regardless of how you build your test team, you still need someone to write a test plan and you need to ensure the quality of the test cases, which means you need to review and approve the test plan.

A test plan is composed of many test cases and is derived from your product requirements document. It is therefore reasonable to expect that if your product requirements document stinks, your testing team is set up for failure. But if you did your job well, then your test lead can do a great job too.

The test plan is generally created in a spreadsheet so you can organize the test cases well. Check to see that your test cases have the following descriptive elements:

Figure 5-1 shows an example test case.

Figure 5-1. Test case spreadsheet

Because you’ve included the “severity” of each test, you can create quicker but less complete test passes by running only the high-priority tests. Test passes like these are good for verifying small changes. You can test only the small change and the high-severity tests in much less time than running all the tests. It’s important to still run the high-severity tests, even if you think the small change is well isolated from other features. You want to make sure that some fundamental feature wasn’t accidentally turned off. In a complicated software system with weak unit testing, it is easy to break major features—and the HSET preaches a “better safe than very sorry” approach.

The output of a full test pass is bugs and sometimes a sense of surprise. This is a great moment for you, as the team lead, to reinforce a “bad news is good news” mantra and greet the number of bugs found by the test team with loud applause. Think about it: you need the test team motivated to find failures. If your team ends up demoralized each time there’s a test pass, or if the relationship between test and development becomes acrimonious, you’re going to end up with fewer bugs and—wait for it—embarrassment!

Reviewing test cases can be incredibly boring. You need to do it nevertheless, if only to empathize with your test team. Here’s a trick: instead of slogging through all of your test cases, which is ideal and will get you a round of applause from anyone who notices, focus on the following three things:

If your test plan covers these major areas, you’re starting from a good place.

Remember how hard it is to hire a great test lead? One of the best work-arounds for this challenge is to find a test lead who’s willing to write test automation. If your test lead is able to craft testing systems that work independently from your production code, you’ve created a great project for a test engineer. What’s more, that software can run constantly and do the work of dozens of people.

You may be thinking, “Wait—that’s a lot of extra software to write!” Luckily, most test automation can be written in scripting languages, doesn’t need to scale particularly well, and can use established, preexisting frameworks. Test development can therefore be more efficient.

As a team lead you probably don’t need to own test automation, but you do need to make sure that it’s being built, because while you’ll never be able to afford enough testers, you will be able to afford enough computers on which to run the automation.

Microsoft pioneered the notion of “eating your own dogfood,” which means you should use the software you intend to ship within your company. Put another way, don’t feed your team peoplefood but give your customers dogfood. Forcing you and your team to suffer through customer pain is a great way to instill a sense of urgency, understand customer problems, and find defects. Amazon and Google dogfood religiously.

Dogfooding can be challenging when there’s a good alternative, such as the prior, less buggy version of your software. For example, Google wanted employees to dogfood Google Docs, and the best way to force that issue was to stop installing Microsoft Office on corporate computers by default. In addition to driving dogfood of Google Docs, the practice saved money!

In some cases, making it easier to taste the dogfood also works. For example, Amazon wanted employees to dogfood Amazon Prime, so they made it available to employees at a discount. I’ve seen teams offer awards (from t-shirts to iPads) for the most unique bugs reported. I even saw one engineering director offer a $5,000 bounty for a successful series of steps to reproduce a “heisenbug.” A heisenbug is a bug that spuriously appears and disappears, following the Heisenberg uncertainty principle;^[3] they’re a pain in the ass. However you do it, make dogfooding a key part of your team culture even if the other teams you work with don’t do it.

One fun part of the dogfood experience is that you’ll find your CEO always has some kind of awful experience that nobody else has seen. The first time this happens to you, you’ll be chagrined. This was precisely the person you wanted to impress, and you botched it! Forgive yourself now, because soon you’ll understand that every team lead has the same experience. Jeff Bezos always finds bizarre bugs. Things always break on Larry Page’s computer. There are many examples of these failures, and one pretty common reason for them: these execs are awesome dogfooders.

In Larry’s case, he has more alpha software on his computer and more experiments running on his account than you could possibly imagine, because team leads like you are desperate to convince Larry that they are making progress. It’s no surprise that there are bizarre interactions. Your best defense in Larry’s case is to figure out what else could interact with your product and plan for it.

In Jeff’s case, he brings a completely fresh perspective to your product because he’s never seen it before and has no idea what he’s supposed to do, so he breaks it. The best thing you can do in Jeff’s case is to try to think like Jeff. Put on your giant-alien-brain mask, get some coffee, clear your browser cache, reformat your hard drive, and try to forget everything you ever knew about your product. Then use it.

If you’ve decided to dogfood religiously, you’ll want to follow some best practices to get the most mileage from your dogfood experience:

A bug bash is an event where your team, or your whole company, takes a dedicated period of time—typically an hour—to find as many bugs in your dogfood product as possible. A good bug bash will almost certainly find a bunch of bugs that you’ll want to fix. You’ll want to do four things to encourage a good bug bash:

I frequently ask product management candidates, “How do you triage bugs?” when I perform phone screens. I’m always amazed at how incomplete the answers are! I think bugs are as simple to triage as 1, 2, 3!

The first of these three steps is bug grading. Your goal is to figure out which bugs you should fix, and that’s not as simple as fixing only the really bad ones, because some bugs are ugly and very easy to fix. So you need to look at three dimensions when you grade a bug:

After you and the team understand how you’re going to grade the bugs, you enter step 2, in which you have a daily bug triage meeting to decide which bugs you will fix. The PM, the dev lead, and the test lead should get together and go through the bugs. The gotcha in this process is that it can take forever if the three of you try to figure out what is going on in each bug. And bug triage can be really boring. You want to try to move through your triage meeting as fast as possible. In triage, try to do the following:

After some time, you’ll find that even though your bug count is going down nicely, new bugs keep popping up. This is the third step of triage: you have to keep moving the bug bar up, making it harder to declare that a bug is a launch blocker. As general guidance, this principle may seem counterintuitive. After all, you don’t want to be embarrassed, right? The reality is that you’re constantly writing new software, and that means you’re introducing new bugs. If you want to hit ZBB, you have to stop adding bugs to the list of launch blockers. Progressively, and carefully, raise the bug bar as you get closer to launch.

Trusted testers are users under NDA (nondisclosure agreement) who use the dogfood version of your product before it ships. They are using different computers than your team, have different expectations, and are generally much less technical than you. As a result, their feedback is immensely valuable.

At Amazon, I had a group of trusted Customer Reviews writers who could give us great feedback. I gave them my direct email address—they frequently found production issues faster than my engineering team. They also didn’t hesitate to email Jeff Bezos, and when they did, I got Jeff Mail. When you’re a team lead and you get Jeff Mail, you drop everything and address it!

At Google, we had hundreds of businesses in the trusted tester program for Google Talk. We turned on the same experiments we used internally and asked them to send us bugs. They gave us great feedback and helped us pinpoint quality issues.

To make the trusted tester system work—which in the case of Talk meant we had ~15% active participation—I followed these best practices:

It seems to me that it’s always the little things that get you. If you’re doing a good job dogfooding, you’re not going to be embarrassed by the majority of your product. But some of the most complicated parts of your product form the out-of-the-box experience (meaning, “I just opened the box; what’s inside?” not “Hmm, let’s think outside the box now!”). Specific things to look for are how you create an account and populate that account with data. As a dogfooder, you probably performed those tasks only once, and that was four months ago!

Pinterest.com is a great example of how to create a brilliant out-ofthe-box experience. It’s incredibly easy to sign up—you use your Facebook account—and your landing page is great because Pinterest suggests people for you to follow and fills your page with lovely images that you care about.

Here’s a tip to help ensure you experience what new users experience: when you hit feature complete and again when you hit code complete, make sure you delete all your data and accounts and start from scratch.