You Don’t Have to Be Perfect

Shortly after agreeing to write this book, I was getting a haircut in New York City. My stylist was someone who had lived in Richmond, Virginia, at the same time I had. We never met while we lived there, but we quickly realized we both used to hang out at all the same places. It only took us about three minutes before we realized we also knew many of the same people. We hit it off immediately.

Molly is a great stylist, and I always had a good time catching up while getting my hair cut. The haircut relevant to this story, however, was the last one I’d get from her before she moved away from New York to open a coffee shop in Detroit. I had built so much trust in her, I didn’t find another stylist or get another haircut for four months.

During this final haircut, I told her that I had signed a book deal, and she asked me to tell her what it was about. So I laid it out in the same simple terms I do in the first chapter: you can’t be perfect, no one needs you to be perfect anyway, it’s too expensive to try to be perfect, and everyone is really happier at the end of the day if you accept those facts.

She responded with an anecdote of her own. When she first started cutting hair, she was so focused on making sure everything was absolutely perfect that it would sometimes take her an hour to do a simple men’s haircut that should have taken 30 minutes. Cosmetology school had ingrained in her that everything had to be as even and measured as possible. Trying to be perfect caused too many haircuts to run over in time, which upset the clients that had later appointments booked.

Eventually, as she progressed through her career, she realized that trying to be perfect didn’t really do much for anyone involved. It was more stressful for her, it was more time-consuming for everyone, and it didn’t actually improve anyone’s life. She learned that it was fine to only be a certain amount of perfect—a certain percentage of perfect, if you will. Spending less time on each customer meant that the haircuts weren’t always quite as symmetrical or even as they could have been. It meant that sometimes, technically, mistakes were made. But even if she had continued to try and be perfect, she knew that slight mistakes would have been made anyway. The important lesson is that none of her customers cared when she changed what her goal, her target, was.

This is because her haircuts were incredibly good. They got the job done—and more! I never had any complaints in the many years that she cut my hair. I always loved how I looked when I left, even if from her perspective she knew that she could have done just slightly better if she had concentrated just a little more or had spent a little more time.

She told me about becoming a little more lax in her standards as I explained this book to her. She didn’t make these changes in her approach because she was lazy. She changed how stringent she was because she realized it actually allowed her to provide a better experience for her customers—and that it was better for her own bottom line, as well. Her customers were happier with the shorter time the haircuts took, and she subsequently made more money in tips because everyone was in and out the door without having to wait too long. The owners of the shop were happier, too, since this new efficiency led to more positive reviews of the business. It turned out that every human involved was happier when perfection wasn’t being aimed for.

In a way, this story sums up what this book is about: it’s about how to make people happy by not trying to be perfect, and only making sure you’re being good enough.

Learn from Molly. The primary philosophies that are covered in this book are lessons she learned organically from her own business: nothing is perfect all the time, and it turns out that people don’t actually expect things to be. So, instead, think about what your customers—your users—actually need from you. It might be something entirely different than what you currently think it is, or what you were taught; however, everyone involved could end up much happier by accepting this fact.

For complex computer systems, there can be a lot that goes into this idea. You might have to perform very complicated math. You might have to architect and deploy entirely new services to provide you with the correct telemetry. You might have to conduct interviews with users, both internal and external. You might have to spend exhausting amounts of time convincing leadership that trying to be 100% reliable isn’t the right goal. There are many, many things you might have to do, and I’m not trying to say that all of this is necessarily easy. There is a reason why there is an entire book here for you to learn from.

People are happy when their haircut is good enough, it doesn’t cost too much, and they can get in and out the door in a reasonable amount of time. If you think about it, your computer services probably don’t have to be much different. SLOs, and everything that comes with them, give you a way of measuring exactly what your users need from you, provide you with powerful data to have better discussions and make better decisions, and ultimately can make everyone’s lives easier.

How to Read This Book

The most important thing that I want you to know when you start reading this book is that this is all only a model. SLO-based approaches to reliability are exactly that—they’re approaches, not some magic tincture that will immediately solve all of your problems. They’re a way to have better discussions and develop measurements that should allow for you to make better data-driven decisions. But there is no one-size-fits-all approach to this. Everything we talk about in this book is only a model. (For a great set of resources about how to think about things as being models, check out https://www.itsonlyamodel.com, maintained by John Allspaw.)

The book has been split into three parts.

Part I, SLO Development

This should be considered mandatory reading if you’re going to implement an SLO-based approach to reliability—it outlines the concepts, philosophies, and definitions you need to be familiar with to take your first steps. The chapters in this part explain what the various components of SLOs are, how to use them successfully, why they work, and how you can make them work for you.

Part II, SLO Implementation

Here you’ll start to hear from some of my brilliant friends from across the industry. While Part I helps you learn how things work, Part II is much more practical in nature. Together we’ll help you tackle some of the more complicated aspects of implementing an SLO-based approach, including the math you need to have meaningful service level indicators (SLIs) and accurate SLO targets, the most useful alerting for your team, as well as how to get buy-in company-wide.

Part III, SLO Culture

This part discusses how you can make sure that adoption of an SLO-based approach to reliability for your team, organization, or entire company can be engaged with in the most efficient manner. Too many people think of implementing SLOs as “something you do,” in the sense that it’s a singular goal you can check off a list. That isn’t how all of this actually works. SLO-based approaches encourage thinking about the reliability of your services in a different and more accurate way. The chapters in this section ensure that you can implement SLOs in the best possible way, and use the concepts behind them for better communications across teams.

Additionally, the final chapter, Chapter 17, Reliability Reporting, brings everything to a close and contains some of the most useful advice in the entire book. Make sure you don’t skip over that.

To make the most out of the book, I truly hope you read every chapter. The chapters in the first section should be read in order, but the last two sections can be read in whatever order best addresses the needs of your situation. You don’t need to read every chapter consecutively; I hope this book will be as much of a reference for you in the future as it is an educational tool to get you started. It was designed to be both a primer and a source for you to return to again and again.

No matter what your situation is, or what your goals are, I hope at the very least you can walk away from this text remembering the lessons Molly taught us. While this book is mostly about computer systems, the general philosophies are broadly applicable: you can’t be perfect, no one needs you to be perfect anyway, it’s too expensive to try to be perfect, and everyone is happier at the end of the day if you embrace this.

Conventions Used in This Book

The following typographical conventions are used in this book:

Italic

Indicates new terms, URLs, email addresses, filenames, and file extensions.

Constant width

Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords.

Constant width bold

Shows commands or other text that should be typed literally by the user.

Constant width italic

Shows text that should be replaced with user-supplied values or by values determined by context.

O’Reilly Online Learning

Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit http://oreilly.com.

How to Contact Us

Please address comments and questions concerning this book to the publisher:

• O’Reilly Media, Inc.
• 1005 Gravenstein Highway North
• Sebastopol, CA 95472
• 800-998-9938 (in the United States or Canada)
• 707-829-0515 (international or local)
• 707-829-0104 (fax)

We have a web page for this book, where we list errata and any additional information. You can access this page at https://oreil.ly/Implementing_SLOs.

Email [email protected] to comment or ask technical questions about this book.

For news and information about our books and courses, visit http://oreilly.com.

Find us on Facebook: http://facebook.com/oreilly

Follow us on Twitter: http://twitter.com/oreillymedia

Watch us on YouTube: http://www.youtube.com/oreillymedia

Acknowledgments

Wow. I wrote a book. I have an incredible number of people to thank for this. I’m going to miss someone who deserves to be in this section—please know that if I did, it was purely accidental.

First, I have to start with all of the contributing authors. It’s not so much that I wrote a book as that we wrote a book. When I first started thinking about this project, I had a few people in mind to cover certain topics. At first this was probably two or three chapters, and I figured I’d write the rest. But as word got around people started volunteering all over the place, and we ended up with eight chapters written by contributing authors. This is amazing, because there is almost nothing more important than having a bunch of voices speaking about the same topic.

Thank you Daria Barteneva, Blake Bisset, Toby Burress, Polina Giralt, Niall Murphy, Eva Parish, Dave Rensin, Ben Sigelman, Harold Treen, Salim Virji, and Jaime Woo. The readers of this book are much better off due to your contributions and to your telling this story with me. Please go read all about who they are and what they do in the Contributors section of this book.

But contributing to a book is about more than just writing the words.

I’d like to thank my acquisitions editor, John Devins, for believing in the very idea of this book, and my content editor, Corbin Collins, and my production editor Deborah Baker: without your support and patience along the way this book would never have happened. We did all of this during a global pandemic, after all.

Rachel Head, my copyeditor: you were amazing, insightful, and comprehensive. After months of putting this book together, it was brought to a whole new level once you were involved.

David Blank-Edelman: thank you for taking the time to answer a random email and getting me started on this entire journey. Without your response and your assistance this book could have ended up a forgotten pipe dream.

Gabe Ochoa: thank you for accidentally pushing me to do this. In the summer of 2019 I was lamenting that there wasn’t an entire book about this subject. I had gotten tired of repeating the same things to people over and over again. When I mentioned this to you, you said I should write the book. I started to protest, explaining that an expert should be the one to write it, and you responded with, “You are the expert.” Those four words are some of the most meaningful I’ve ever heard, and we can blame the entire existence of this book on them.

Eva Parish and John Turner: thank you for bowls of ramen and helping me find a home when I needed one the most.

To all of my reviewers and other helpers: thank you, thank you, thank you! Once chapters started getting sent off to O’Reilly, I got the following note back:

You guys are amazing. Hard to believe the pitiful pickiness I’ve been reduced to. Really good writing and careful quality control. Super refreshing.

Y’all kicked ass, and I thank every single one of you for that. Thank you to William Banfield, Abby Bangser, Betsy Beyer, Danyel Fisher, Liz Fong-Jones, Heinrich Hartmann, Marke Howell, Matt LeMay, Matthias Loibl, Christian Long, Ben Lutch, Charity Majors, Ben Picolo, Isobel Redelmeier, John Reese, Shelby Spees, Daniel Spoonhower, Emil Stolarsky, Tim Tischler, Amy Tobey, and Scott Wilson. Some of you I’ve known for a long time and some of us met just for this project, but I now consider every single one of you my friends.

And, I would be remiss if I didn’t call out a few of y’all specifically…

Jacob Scott, you reviewed more of the book than almost anyone and did so with your expert curiosity and skepticism. Thank you for making this book better, and I’m glad we don’t always agree.

Štěpán Davidovič, you showed up when we needed you most, and several chapters owe so much to your magnificent talent with your second language. You’re up next. You have things to say and you have a talent for saying them.

And, finally: Kristina Bennett. You were an absolute rockstar that stepped in at just the right moments and went way above and beyond with your reviews. I miss working with you, and this book would not exist in the state it does today without you. You helped start things off and you helped finish things, too. Thank you.

To Molly van Tassel: thanks for letting me share your story in the preface. I miss your haircuts.

To Denise Yu: thank you for the amazing wiener dog pictures in Chapter 12.

To Pizzachat: thank you for literally everything. Every day you help me feel connected, grounded, and loved.

To Prodigy Coffee: thank you for all of your pre-pandemic coffees, smiles, and daily inquiries into how the book was going. You have no idea how much your encouragement helped.

To The Gate and Skylark: a lot of this book was written within your walls and with one of your drinks by my side. It’s a shame I couldn’t finish it the way it started, but thank you for being there while you could. On the other side of this, thank you to Lucey’s Lounge for the encouragement and to-go margaritas I drank amongst lonely warehouses once the pandemic arrived.

To Jimmy Eat World: playing Clarity on repeat is how I’ve done all of my writing—big and small—for over two decades now. Thank you for recording my favorite album. It is seared into my soul.

Tanya Reilly, thank you for being there for me on my first day at Google, thank you for being there for me on my last day, thank you for always being the exemplar of both an SRE and a good human, and thank you for encouraging me to put more of my thoughts out into the wider world.

Niall Murphy, I cannot figure out how to say thank you enough. You were more important to the success of this project than anyone. No one could ask for a better mentor or friend. Thank you, thank you, thank you.

To my mother, brother, sister, and the rest of my family back home: thank you for indulging me as a child when I forced you to read my “books.” I always wanted to be an author. Turns out it happened.

To six-year-old me, I know it’s not a high fantasy tale filled with dragons and magic, but: We did it! We’re an author after all!

And, finally, and most importantly: thank you to my partner Jen and my rescue dog Taco. You are my soulmates; you are the reasons I do the things I do. I love you.

Black lives matter. We only have this one planet. Adopt, don’t shop. Take care of each other.