Preface

This report discusses how to train Site Reliability Engineers, or SREs. Before we go any further, we’d like to clarify the term “SRE.” “SRE” means a variety of things:

• Site Reliability Engineer or a Site Reliability Engineering team, based on the context (singular, SRE, or plural, SREs)

• Site Reliability Engineering concepts, discipline, or way of thinking (SRE)

• Belonging to an SRE individual, team, or way of thinking (SRE’s or SREs’)

Ben Treynor Sloss, the founder of Site Reliability Engineering at Google, describes SRE, or the Site Reliability Engineering discipline, as what happens when “you ask a software engineer to design an operations function.” The traditional systems administration model of software management in production requires an organization to scale the number of operators as the service increases in size and complexity.¹ SRE is able to scale humans sublinearly with the scale of the services they are supporting. This is done by applying proactive engineering solutions to eliminate repetitive, no-value-added tasks and toil.

We assume that you’re familiar with the concepts in the Site Reliability Engineering book. As we were growing our SRE department, we noticed how difficult it was to get new hires up to speed. You might be surprised to find that technical skills are not necessarily the most important skills to have. Without a doubt, troubleshooting is an important part of incident management, but we also show that growing the students’ confidence, explaining the importance of good relations, and encouraging clear communications with other SREs and dev teams are essential to bringing your students up to speed.

In this report, we share our experience ramping up new SREs, but we also look at other use cases. For example, we have talked with several smaller organizations that are successful in ramping people up to do SRE (or SRE-like) functions.

Training should be purposefully designed and not just thrown together, without any thought. Therefore, we also discuss the theory behind the training design. You need to know what your training needs are and who your audience is. Set clear learning objectives and build your training content based on that. We’ve seen that making SRE training hands-on is extremely important for building the confidence of the students who ultimately go on-call² for a production service.

Finally, when teaching how to “SRE,” we should implement the practices of SRE while administering the program: that is, “SRE” your SRE training program. In other words, monitor the results and be willing to adjust the training program if the monitoring shows it is necessary. We show that just like SRE has a hierarchy of needs, SRE training also has a hierarchy of needs, which follow SRE’s needs.

While much of this report focuses on the specific experience of Google SRE, we aim to present best practices and lessons learned over the past several years, which can be applied to organizations that are at varying points along the spectrum in terms of size and maturity.