Footnotes
Foreword by Alberto Savoia
1. Don’t get the Octomom reference? Google it!
Chapter 1
1. http://googletesting.blogspot.com/2011/01/how-google-tests-software.html.
2. GTAC is the Google Test Automation Conference (www.GTAc.biz).
3. http://googletesting.blogspot.com/2007/01/introducing-testing-on-toilet.html.
4. The term dogfood is used by most software companies in the U.S. to denote internal adoption of software that is not yet released. The phrase “eating your own dogfood” is meant to convey the idea that if you make a product to sell to someone else, you should be willing to use it yourself to find out if it is any good.
5. The original purpose of using small, medium, and large was to standardize terms that so many testers brought in from other employers where smoke tests, BVTs, integrations tests, and so on had multiple and conflicting meanings. Those terms had so much baggage that it was felt that new ones were needed.
6. Indeed, the concept of an enormous test is formalized and Google’s automation infrastructure uses these designations of small, medium, and so on to determine the execution sequence during automated runs. This is described in more detail in the chapter on SETs later in this book.
7. Google’s recording technology and automation-assisted manual testing are described in detail in subsequent chapters on the TE role.
Chapter 2
1 It is interesting to note that even with client software, Google attempts to make updating frequent and reliable with the development of an “auto update” feature, a necessary feature for any client application.
2 “The Life of” is a tip of the hat to a series of Google internal courses on how Search and Ads work. Google has courses for Nooglers (new Google employees) called Life of a Query that details the dynamics of how a search query is implemented and Life of a Dollar to show how Ads work.
3 Patrick Copeland covers the dawn of the SET in his preface to this book.
4 “Twenty percent time” is what Googlers call side projects. It’s more than a concept; it is an official structure that allows any Googler to take a day a week and work on something besides his day job. The idea is that you spend four days a week earning your salary and a day a week to experiment and innovate. It is entirely optional and some former Googlers have claimed the idea is a myth. In our experience, the concept is real and all three of us have been involved in 20 percent projects. In fact, many of the tools outlined in this book are the result of 20 percent efforts that eventually turned into real, funded products. But many Googlers choose to simply work on another product in their 20 percent time, and thus, the concept of a 20 percent contributor is something many products, particularly the new cool ones, enjoy.
5 The most common of which is Google’s often touted “peer bonus” benefit. Any engineer whose work is positively impacted by another engineer can serve a peer bonus as a thank you. Managers additionally have access to other bonus structures. The idea is that impactful community work be positively reinforced so it keeps happening! Of course, there is also the informal practice of quid pro quo.
6 The only local test labs Google maintains outside this common infrastructure are for Android and Chrome OS where various flavors of hardware must be kept on hand to exercise a new build.
7 Google’s protocol buffers are open sourced; see http://code.google.com/apis/protocolbuffers/.
8 An open-source version of Mondrian hosted on App Engine is publicly available at http://code.google.com/p/rietveld/.
9 Google’s C++ Style Guide is publicly available at http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml.
10 http://labs.google.com/papers/bigtable.html.
11 http://labs.google.com/papers/gfs.html.
12 http://code.google.com/apis/protocolbuffers/docs/overview.html.
13 http://www.sei.cmu.edu/cmmi/start/faq/related-faq.cfm.
14 OKRs are objectives and key results that individuals, teams, and even the company as a whole write each quarter and assess their progress against. Basically, they are the things an individual or team intends to accomplish.
15 ToTT is Testing on the Toilet, which has been mentioned before in this book and appears often on the Google Testing blog at googletesting.blogspot.com.
16 “Fixits” are another Google cultural event that bring people together to “fix” something that is deemed broken. A team might have a fixit to reduce bug debt. They might have one for security testing. They might have one to increase the usage of #include in C code or for refactoring. Fixits have been known to leap the technical divide and can be applied to improving the food in a café or making meetings run smoother. Any event that brings people together to solve a common problem is a fixit.
17 MapReduce is a form of distributed computing where the computation is broken into smaller pieces, categorized by key (mapped), and then rolled up (reduced) by key. See http://en.wikipedia.org/wiki/MapReduce.
18 Sharding is a form of database partitioning. Horizontal partitioning is a database design principle whereby rows of a database table are held separately, rather than splitting by columns. See http://en.wikipedia.org/wiki/Shard_(database_architecture).
19 Big O notation describes how long it takes some function to execute based on the size of the input data. See http://en.wikipedia.org/wiki/Big_O_notation.
20 The open-source version of Buganizer is called Issue Tracker and is available through the Chromium project at http://code.google.com/chromium/issues/list.
Chapter 3
1 This is a general view. Many TEs perform work more akin to the SET role and write a lot of code. Some TEs perform a role more closely aligned with a release engineer and write little code.
2 Clearly in the event a customer negotiates the development of a test plan or some government regulation requires it, the flexibility we talk about here disappears. There are some test plans that have to be written and kept up to date!
3 Leaving such variation to the tester is a good idea. It creates variety in how capabilities are interpreted and cast into actual test cases. This translates to better coverage.
4 The web page where people can download Chrome is http://www.google.com/chrome.
5 Tours are described in detail in James A. Whittaker, Exploratory Software Testing: Tips, Tricks, Tours, and Techniques to Guide Test Design (Addison Wesley, 2009).
6 Chrome’s cash-for-bugs program is discussed at http://blog.chromium.org/2010/01/encouraging-more-chromium-security.html.
7 Cukes are a behavioral test case definition. See http://cukes.info.
8 We interviewed Ted Mao in Chapter 2.
9 Like many bug-filing systems, we use PX, where X is an integer, to define how high a priority the bug is. P0 bugs are the worst, P1 less awful, and so on.
10 Triage is the process of reviewing incoming bugs and deciding what order and who should fix them. It is much like triage in a hospital emergency room.
11 An interview with Brad Green, engineering manager for Google Feedback appears in Chapter 4.
12 OKR stands for Objectives and Key Results. OKR is just a fancy way of saying list your goals and how you will measure your success on those goals. At Google, emphasis is placed on quantifiable metrics for success. Hitting 70 percent means you likely aimed high and worked hard. 100 percent means you likely weren’t ambitious enough.
13 The highest priority crawls execute on virtual machines from Skytap.com. Skytap provides a powerful virtual machine environment that lets developers connect directly to the virtual machine where the failure occurred and manage those debugging instances—all without leaving the browser. Developer focus and time is much more valuable than CPU cycles. Moving forward, Skytap enables bots to execute entirely on other users’ virtual machines and accounts, allowing access to their nonpublic staging servers.
14 XPaths are much like file paths, but work within web pages instead of file systems. They identify the parent-child relationships and other information that uniquely identifies an element within the DOM on a web page. See http://en.wikipedia.org/wiki/XPath.
15 Chrome builds many times per day.
16 URL to check in that caused regression is at http://trac.webkit.org/changeset/81691.
17 URL to WebKit BugZilla issue https://bugs.webkit.org/show_bug.cgi?id=56859. Tracking the issue in Chromium is http://code.google.com/p/chromium/issues/detail?id=77261.
18 Our friends at http://www.utest.com have been helpful in setting up these experiments. Their crowd of testers has been amazingly sharp and responsive. At times, the test results from their crowd have found more issues of higher quality than internal repetitive regression test runs.
19 The singularity is a term often used to describe the moment computers surpass human intelligence. It should be an interesting time and we are seeing glimpses of that today (http://en.wikipedia.org/wiki/Technological_singularity).
20 False positives are test failures that aren’t really failures of the product, but rather the testing software and can become expensive, aggravating to engineers, and quickly slow engineering productivity with fruitless investigations.
21 The DOM is the Document Object Model, the internal representation of all that HTML under the hood of a web page. It contains all the little objects that represent buttons, text fields, images, and so on.
22 Firefox was used as the benchmark, as it more closely adhered to the HTML standard, and many sites had IE-specific code, which wasn’t expected to render well in Chrome.
23 Just a hash of elements returned from getElementFromPoint(x,y) for a 800 × 1,000 section of the web pages. There are more efficient ways to do this, but it is simple and works for illustration.
24 Yes, one high-risk capability can be obscured by many other related, but very low-risk capabilities. This doesn’t happen very often, but the tool is designed to be simple, and is not a replacement for common sense and due diligence.
25 An example of other cloud companies looking at this approach is Salesforce. Phil Waligora of Salesforce.com is looking to integrate this into its internal tooling.
26 James Whittaker’s GTAC presentation on the future of testing can be found on You Tube at http://www.youtube.com/watch?v=Pug_5Tl2UxQ.
27 App Engine is a Google cloud service for hosing a website or service. Testers often leverage App Engine for tools and infrastructure today given its capability to get an application up and running very quickly and still having Google Scale for free. See http://appengine.google.com. It currently supports Java, Python, and the Go languages.
28 Ryan Hoppes and Thomas Flynn from Allion Test Labs were instrumental in helping us with hardware and network testing and certification.
29 You can find the AS3 player helper source code at http://code.google.com/p/youtube-as3-player-helper/source/checkout.
Chapter 4
1 At the time of this writing, Google employs six directors of test, each of whom has only a handful of TEMs reporting to them. Individual contributors generally report to a TEM, but senior engineers and technical leads often report to their director. It’s a flat structure meant to focus on working together and not heavy on the management side of the equations. Most, if not all, directors at Google also slice off time to perform as an individual contributor.
2 The concept of 20%, explained earlier, can help here. A common transition by an engineer from project A to project B is for that engineer to take a 20% on the new project B for a quarter and then flip the ratio to 80% on project B and 20% on their old project A for the next quarter.
3 You can get more information about PyAuto here: http://www.chromium.org/developers/testing/pyauto.
4 Information about Autotest can be found here: http://autotest.kernel.org/.
5 Not true as it turns out” http://en.wikipedia.org/wiki/Eskimo_words_for_snow.
Chapter 5
1 Google has such a program for Service Reliability Engineers (SREs) called Mission Control. SWEs who complete the six-month program in an SRE role get a significant cash reward and a leather jacket emblazoned with a Mission Control Google badge.
2 Google testing blog post on Native Driver: http://google-opensource.blogspot.com/2011/06/introducing-native-driver.html