Chapter 5. Improving How Google Tests Software

Google’s testing process can be summed up quite simply: Put quality in the workflow of every engineer. When done earnestly and honestly, quality increases. Code freshly written is better. Early builds are better. Integrations are unnecessary. System testing can concentrate on real user-oriented issues. Keep the projects and all the engineers free of accumulated bug debt.

If your company has achieved this level of attention to quality, there is only one question remaining, “What’s next?”

Well, what’s next for Google is already beginning to take place. As we have perfected the role of testing in product development, we have created a process with more than a few obvious flaws. This chapter covers those flaws and talks about how testing is evolving, or devolving, at Google to address them. What has occurred is a decentralization of testing where the Engineering Productivity organization has been broken up and its pieces absorbed by the product teams. This, we believe, is a natural progression that occurs when a certain level of test maturity is achieved. Google is simply no longer best served by keeping development and testing separate.

Fatal Flaws in Google’s Process

Testing is often seen as a surrogate for quality and if you ask a developer what he is doing about quality, “testing” is often the answer. But testing is not about quality. Quality has to be built in, not bolted on, and as such, quality is a developer task. Period. This brings us to fatal flaw number 1: Testers have become a crutch for developers. The less we make them think about testing and the easier we make testing, the less testing they will do.

Every time we sit inside on our comfortable sofas watching television while someone else mows our lawn, we see this in action. People are capable of mowing their own lawn. Worse, we are often at home doing nothing while our lawns are being mowed! Lawn services have made it easy for us, enough so that we are happy to farm this work out without thinking about it. When testing becomes a service that enables developers to not think about it, then they will not think about it. Testing should involve some pain. It should involve some concern on the part of developers. To the extent that we have made testing too easy, we have made developers too lazy.

The fact that testing is a separate organization at Google exacerbates this problem. Quality is not only someone else’s problem; it is another organization’s problem. Like my lawn service, the responsible party is easy to identify and easy to blame when something goes wrong.

The second fatal flaw is also related to developers and testers separated by organizational boundaries.

Testers identify with their role and not their product.

Whenever the focus is not on the product, the product suffers. After all, the ultimate purpose of software development is to build a product, not to code a product, not to test a product, not to document a product. Every role an engineer performs is in service to the overall product. The role is secondary. A sign of a healthy organization is when people say, “I work on Chrome” not whey they say, “I am a tester.”

A few years ago, I saw a t-shirt at a testing conference with the phrase “I test, therefore I am” printed in Greek and English. I am sure whoever made it thought it devilishly clever, but it is a combative, anti-product slogan that puts the role of testing in an elevated light that it simply does not deserve. No role deserves it. Everyone on a team works on the product, not some part of the development process. The process itself is subservient to the product. Why else have any process except to build a better product? Users fall in love with products, not the processes that built them.

Google’s separation of development and test fosters a role-based association and makes it harder for testers to align themselves with their products.

The third fatal flaw is that testers often worship test artifacts over the software itself.

The value of testing is in the activity, not the artifacts.

Every artifact generated by testers is secondary to the source code. Test cases are less important; the test plan is less important. Bug reports are less important. The activities are what go into making these artifacts that actually provide value. Unfortunately, in the celebration of these artifacts (such as counting the number of bug reports submitted by a test engineer [TE] during an annual review), we forget about the software. All testing artifacts have value to the extent that they impact the source code and therefore, the product.

A separate team of testers has a tendency to focus on the construction and maintenance of test artifacts. The product would be better served if the activities involved in testing were aimed exclusively at the source code. Testers must put the product first.

The last fatal flaw is perhaps the most insightful. How often are products released only for users to find problems that escaped even the more voracious testing process? The answer: almost always. None of us have ever been associated with a product launch that didn’t suffer field issues the test team failed to identify (at Google or elsewhere). In fact, many of the best bugs in Google+, the product all three of us are working on at the time of this writing, were found by dogfooders—other Googlers outside the Google+ team trying to use the product. We were acting like users; the dogfooders were actual users!

It doesn’t matter who does the testing, only that the testing gets done.

Dogfood users, trusted testers, crowd-sourced testers, and early adopters are all in a better position to find bugs than test engineers. In fact, the less testing a TE does and the more enabling of others, the better.

So what do we do about all of this? How do we take everything that is right about Google testing and make it more product-focused and team-oriented? Now we’re getting into uncharted territory and the only thing we can do is speculate. However, there are some compelling trends that this book uncovers that, we believe, form the basis for predicting the future of testing at Google and elsewhere. In fact, the software engineer in test (SET) and TE roles themselves are already migrating toward this future.

These two roles are actually diverging at Google. The SET role is becoming more and more developer-like and the TE role is going in the exact opposite direction and becoming more user-like. This is happening organically as a natural progression of a mature software development organization. Partly, the trend is due to technological innovations such as compressed development cycles and continuous availability of the latest build to developers, testers, and users alike. This gives far more opportunity to include other nonengineer stakeholders into the development process. The other part of the equation is a maturation of the idea that quality is everyone’s job and not the sole domain of engineers with the word “test” in their title.

The Future of the SET

Simply put, we don’t believe there is a future SET. The SET is a developer. Period. Google pays them as developers, calibrates their performance reviews against developers, and calls both roles software engineers. So many similarities can only lead to one conclusion: They are the exact same role.

As doomed as we believe the role is, the work itself cannot go away. The magic in the Google formula is the work the SET performs. SETs provide features such as testability, reliability, debugging capability, and so on. If we treat these things as features in the same sense that we treat the UI and other functional components as features, then SETs are nothing more than developers who own these features. This is the evolution of the role we think will happen at Google and other mature software shops in the near future and what better way to make test development a first-class citizen than treating it just like any other feature?

Indeed, this is the part of the process that is flawed in its current state. Every user-facing feature is managed by product managers (PMs) and built by SWEs. Code for these features is tracked, managed, and maintained by a well-defined automated workflow. However, test code is managed by TEs and built by SETs. Why? This is a relic of the history of how the roles evolved. But the evolution has peaked and it’s time to treat test code as a first-class citizen: let it be managed by PMs and built by software engineers (SWEs).

Which SWEs will make the best SETs and can be trusted to own the quality features and take their work seriously? Well, at a company like Google that already has the SET role, we can simply convert all the SETs to SWEs and be done with it. But this is a subpar solution in our minds. Every SWE can benefit from having a tour of duty owning quality features;¹ however, it isn’t practical (nor Googley) to simply force people in this situation. Instead, ownership of testing features should fall to new team members, particularly the more junior ones.

Here’s our reasoning. Testing features cut across the entire footprint of a product. As such, the developers involved in building testing features are forced to learn the product from interfaces to APIs. What better way is there to take a deep dive into a product and learn its design and architecture quickly? Owning the test feature (either building it from scratch, modifying, or maintaining it) is the perfect starter project for any developer on any team. Arguably it is even the best starter project. As new members are on board, existing test developers move to feature development to make way for the new engineers. Everyone is fresh and engaged and over time, all developers are test savvy and take quality seriously.

Junior developers and hires fresh from college find test development the absolute best place to start. Not only do they get to learn about the entire project, but because much of the test code won’t ship, they avoid the pressure and potential embarrassment of writing a user-facing bug (at least until later in their career).

The fundamental difference between this single role model and Google’s existing model is that testing expertise is more evenly distributed among the SWE ranks as opposed to being concentrated in the SETs. This is an important distinction that powers greater productivity gains by removing the SET bottleneck. Additionally, because there is no difference in the titles of the engineers involved, there are no barriers or stigmas to moving from test to feature development and back: a single product, a single team, a single role.

The Future of the TE

The need for good test engineers has never been greater. However, we believe that need has peaked and is in rapid decline. The truth is that much of the test case creation, execution, and regression testing TEs traditionally perform is available in more complete and cost-effective formats.

Much of this bounty of opportunity is due to technological improvements in the way software is delivered. Back in the days of weekly or monthly builds and painstaking integrations, it was important to have testers who could find bugs and act like users as much as possible. Bugs had to be found before shipping a product to millions of users where the issues were hard to track and the product hard to update. Not any longer. Delivering software via the Web means the ability to deliver early to a few select users, respond to their feedback, and deliver updates quickly. The barriers to developer-user communication and collaboration are gone. The life span of a bug has declined from months to minutes. It’s a case of better software delivery and better user feedback so that we build, deliver (to dogfooders, trusted testers, early adopters, or actual users), fix, and redeliver faster than many users can even notice defects. Exactly where does a team of professional TEs fit into this equation? Users experience far less pain now than they did in the old days and it is time to adjust our testing resources appropriately.

Here’s the bottom line. Who would you rather have testing your software: highly paid, on-site exploratory testing experts doing their best to anticipate actual usage in the hopes they find important bugs or a large army of actual users who are incentivized to find and report real bugs? Access to real users willing to report issues in exchange for early access has never been easier and with daily or hourly updates, the actual risks to these users is minimal. In this new world order, the TE role needs a complete overhaul.

Test engineering, we believe, will morph into a test design role where a small number of test designers quickly map out the testing surface, risk heat map and tours for application (see Chapter 3). Then, when dogfooders, trusted testers, early adopters, or crowd testers submit feedback, the TE can assess coverage, calculate risk impact, ensure it trends down, and adjust the testing activities accordingly. These test designers can also identify places where expertise, such as security, privacy, performance, and exploratory testing, are required and then direct that work to specialists in the crowd. There is also the work of creating or acquiring tools to collect and analyze all this incoming data. There is no test creation, no test execution, no actual testing at all. Okay, perhaps “no” is too strong a word, but certainly, this aspect of the job is minimal. It’s a job that requires designing, organizing, and managing test resources that are mostly free.

TEs, we believe, will migrate to specialist roles such as security or they will become managers of testing activity that is performed by others. It is a challenging and senior role that requires a great deal of expertise. It may even be a role that pays a lot more than current TE positions, but it will be a role that requires far fewer people than it did in the past.

The Future of the Test Director and Manager

With all this change in the TE and SET roles, what does it mean for the future of test directors, managers, and even VPs at some companies? It means that there will be far fewer of them, and the more technical ones will move into more of a distinguished engineer, individual role. They will live on as thought leaders and coordinators across very loosely coupled TEs and quality-focused SWEs, but not ultimately accountable for quality or management of any specific project. Test activities should be accountable to the actual products people work on, not a centralized organization with tangential ties to shipping products and the rest of the engineering processes.

The Future of Test Infrastructure

The test infrastructure at Google is still surprisingly client-based. There are still a lot of Selenium and WebDriver tests written in Java or Python, checked into the source trees, built, and deployed to banks of dedicated virtual machines via shell scripts. These test drivers inject this Java-based test logic into the browser from code running as an application on the native operating system. This works, but our infrastructure needs an overhaul because this approach requires expensive people and machine infrastructure dedicated to test creation and execution. Testing infrastructure eventually moves entirely into the cloud. Test case repositories, test code editors, recorders, and execution will all work from within a website or from within a browser extension. Test authoring, execution, and debugging are most efficient when they occur in the same language and context as the application itself. For Google and many other projects these days, this is more often a web application. For projects that are still inherently non-Web such as native Android or iOS applications, they will be driven from the Web using adaptions of web-focused test frameworks. Native Driver² is a great early example of this move to Web first, native secondary.

As projects and testing needs appear and disappear more quickly in this “fail fast” environment, there will be fewer inhouse, custom test frameworks and dedicated test execution machines. Test developers will increasingly leverage and contribute to open-source projects, cobble them together, and execute them on shared cloud computing resources. Selenium and WebDriver set the model for community-maintained and corporate-sponsored infrastructure development. There will be more of these projects and tighter integration between these open test frameworks, open bug and issue-tracking systems, and source control.

Sharing test data, test cases, and test infrastructure will be worth giving up imagined advantages of keeping everything secret or proprietary. Secret and proprietary testing infrastructure just means it is expensive, slow-moving, and often not reusable even within the company across projects. The future testers will share as much code, test, and bug data as possible, as the returns from the community, new forms of crowd-based testing and test creation, and user good-will will outweigh the imagined benefits of keeping everything hidden.

This more open and cloud-based approach to testing will save money, provide test infrastructure developers more visibility, and most importantly, it will enable project-level test developers to focus on the test coverage and not the infrastructure, leading to higher quality products and faster release cycles.

In Conclusion

The end of testing as we know and love it is a hard message to hear. It will be much harder for those whose careers are defined by the status quo. But there is no arguing that the software development problem has been fundamentally altered by agile development, continuous builds, early user involvement, crowd-based testing, and online software delivery. Sticking to decades-old testing dogma is a recipe for irrelevance.

This transition is already well underway at Google, although not every-one here even knows it. The engineers, managers, and directors in centralized testing are dispersing into more project-focused teams and accountabilities. This transition is forcing them to move faster and focus less on the process of testing and focus more on the product itself. As Googlers, we are seeing this transition a bit earlier than many other companies. It’s likely that every tester’s world will adjust to these new realities very soon. Embrace and drive these changes to remain relevant as a tester.