Legacy orientation

For a decade, Google had no structured education for new SRE hires. One could say we were following the “sink or swim” model discussed earlier. Apart from haphazard, team-specific materials, and random classes taught at irregular intervals, there was no SRE-specific training for new people.

In 2015, Google SRE formed an education team called SRE EDU. To ramp itself up as fast as possible, first, an inventory was made of all available classes (and all the different versions available for those classes). This led to a curriculum that put 11 classes, all one to two hours long, in a week. The students were happy, but there were frequent remarks that learning would be more effective if there were more hands-on-oriented classes. We quickly and effectively launched a minimum viable training curriculum and then used survey feedback (a form of monitoring that we discuss in Chapter 5) to find opportunities to innovate and improve.

Current state of SRE orientation at Google

After onboarding at the local office, we have students travel to one of our three hubs for orientation, where they receive a week-long, SRE-oriented training. The training concentrates on SRE culture, tools and systems, and applications.

For the culture aspect, we have a number of classes sandwiched between the more technical classes. Following are the important factors in these cultural classes:

• Failure always happens—it is a logical consequence of Google’s size. Therefore, we must embrace failure and use redundancy and resilience to fight the effect of the failures that do occur.

• Toil is something we want to get rid of as much as possible. Our approaches are to automate as much as possible. Also, after an outage, investigating to find the root cause and fixing that lowers repeat outages and toil.

• Creating a good social fabric is important. This comprises having a good relationship with the developers, using blameless postmortems after outages, and only having actionable alerts. Also, having diverse opinions matter. This helps fight tunnel vision, which can prolong the duration of outages.

For the more technical parts of the curriculum, we have changed our approach drastically. Adult learners acquire new knowledge most effectively by doing and applying what they’ve just seen. Therefore, we went from a model in which we had a number of slide deck–based classes, to a model with fewer (and less-detailed) classes with specific, hands-on practice. In particular, we created a photo upload service called the Breakage Service, discussed in more detail in Chapter 4.

Wherever relevant, the technical classes refer to our photo service as an example, so the students see right away how that works in real life. They immediately use the tools they heard about and investigate how the parts of the service hang together. For example, when we teach about Remote Procedure Calls (RPCs) in Google, the students also look at diagnostic consoles with regard to the RPCs sent from the frontend servers to the backends.

We’ve found that a day and a half after orientation begins, the students are able to correctly triage, mitigate, and find a resolution for the first outage. Once, after the first breakage exercise, a student correctly gave a four-sentence summary of what had just happened. When asked what they would have thought if they’d heard those four sentences only a day before, their eyes went big, their jaw dropped and they said, “I would not have understood a word of what I just said.”

From surveys we held, we found that on a seven-point scale, from –3 (not very confident) to 0 (neutral) to +3 (very confident), the students, on average, rated their confidence nearly two points higher than they did before orientation. To further illustrate how SRE EDU orientation raises confidence, 89% of participants reported at least a one-point increase in confidence, with 29% of participants reporting at least a three-point increase in confidence. Figure 3-1 shows the shift in confidence in a histogram of survey responses.

Figure 3-1. Histogram of survey responses of self-reported confidence.

Takeaways

We found it beneficial to move from a class-only model to a model centered around real-life and hands-on troubleshooting. We saw much better student participation, happier students, and above all, a rise in student’s confidence that they’d be able to do their SRE jobs. We strongly suggest making your onboarding classes hands-on, with troubleshooting exercises that are as close to real world as possible. This might be more difficult to do when your organization is smaller (you would probably not build a complete application stack for your learning environment), but it’s still worth it. We look at how this is done in “SRE Training in Smaller Organizations”, later in this chapter.

Training of instructors

A training model that centers more around the experience than on actual technical knowledge requires that the instructors know what the philosophy behind the program is.

We’ve noticed in the past that giving students the full depth of information about the systems we have leads to information overload, and the students don’t remember the information afterward. However, to do the breakage exercises, only a minimum amount of detail is needed. Therefore, we have deliberately limited the depth of the class material to teach only what is necessary so that we can go through the breakage exercises. This shift in training method is sometimes difficult for instructors, who are often subject matter experts that volunteer to teach. The instructors often want to tell a lot of exciting details about their subject, and might even disagree with the amount of material that is left out, including specific details about the “missing” material. However, when the instructors see how students apply what they’ve been taught when doing the exercises, and resolving the deliberate breakages, they better understand why the curriculum is set up the way it is.

To help instructors see our reasoning behind the idea of less depth in the curriculum, we run regular “Train the Trainer” sessions. We have two different sessions: one for the class material, to give the background and reasoning behind the classes, and another for the story that ties them together.

The latter Train the Trainer class concentrates on how to be a facilitator (or teaching assistant) during the breakage exercises. We’ve found these sessions are valued and in high demand from our instructors. In the “breakage” part of Train the Trainer, we have the future facilitators be on the “receiving end” of the breakages, asking, before the exercise starts, to not just resolve the issue, but also observe how the Train the Trainer instructors behave. We have multiple breakage exercises, and in later exercises, the Train the Trainer instructor is more and more hands off. We discuss the tactics here and why it’s used (as the new SRE students gain more confidence, we keep more distance). After a number of these exercises, we have the Train the Trainer students practice running a breakage exercise. This sometimes poses a challenge, as the students have to act like they don’t know what is wrong, during the breakage exercise they actually solved earlier.

In the same way the entire curriculum is there to give the new SRE hires confidence, the Train the Trainer is also there to give facilitators and TAs the confidence they need to run a class. How we implemented the Train the Trainer program is described in detail in Chapter 4, Instructional Design Principles.

Having volunteers teach and facilitate is the only way we could scale our program to the size it is now, with a team of only seven people running 14 classes every month, in three different locations. To date, more than 3,000 students have gone through Google’s SRE EDU orientation.

Documentation

One source of information that new team members learn about the systems they will be responsible for comes from documentation. Documentation should describe the following:

• The systems the team is responsible for

• The procedures they follow; for example, for cluster turnups and turndowns

• A playbook/runbook that describes how to handle certain outages (see the section "Maintaining Playbooks" in Chapter 8 of Site Reliability Engineering Workbook).

This helps all new team members understand the systems the team is involved with. Some teams also have specific onboarding documents that are then referred to, from the onboarding checklist.

New team members should be encouraged to fix any inconsistencies they encounter in the material. If your documentation is checked into your versioning system after review, this is easy and safe to do. Having new team members verify the documentation helps keep the documentation up-to-date. New team members are in a unique position to conclude that the documentation no longer describes the setup of the system. Unfortunately, the state of documentation is often “slightly stale” more often than not.

In-person and recorded classes

Team members can teach classes about the systems, perhaps as impromptu one-offs or as deliberate ramp-up summits (if there are multiple new team members). If you record the classes delivered at the summits, they can also be used by new arrivals who join the team later. Even more than with documentation, the risk exists that the recordings are out of date because systems are continuously in flux. Although documentation can be updated, this is much more difficult to do for recorded videos.

Whiteboarding sessions

Having whiteboarding sessions is also useful. Here a team member who is the “owner” of a specific subsystem explains how that subsystem works, any weak spots it might have, and what the plans are for the future of the subsystem. Having these sessions not only helps other team members understand more about these subsystems, it also works as a forcing function to have people explain the subsystem in a clear and succinct way. Also, these sessions can be recorded for later consumption (with the same staleness risks).

Teach-back sessions

There is a special kind of whiteboarding session during which new team members are asked to prepare a session on a specific subsystem, and then use the experience from reading documentation, skimming through configuration, and looking at jobs running in production, to explain what they think the system does. At Google, we’ve found that usually, the new team members do a really good job, and any misunderstandings are quickly resolved. Many times, such teach-back sessions uncover new aspects of the subsystem that the non-experts of that subsystem did not know about yet. These sessions are of value not only to new team members, but to the team as a whole.

Mentoring

It’s important that the team assigns the new hire a team member as a mentor, as a first point of contact for questions related to the team’s systems and the company’s infrastructure in general. We also advise new hires to have a mentor outside the team, for specific issues not related to the team—someone who can give a point of view without the consequences of day-to-day work. For example, “What can I expect from my manager?” or “How does your team use tool XYZ?”

Classes

At Google, we offer newly hired SREs a set of classes after about six weeks, to prepare students for going on-call in the general sense. Of course, there are no team specifics in these classes. We have two classes that talk about the mechanics of being on-call (a little bit about the tools, and mostly about the procedures around incident management). As previously discussed, we also have two more “soft skills"–oriented classes: the first is about stress management during incidents. During incidents, the stakes can be very high, and keeping your cool in those situations is important. We want to teach this without scaring the students and having a negative influence on their confidence. This class was codesigned with an aviation consultant—stress management in the cockpit is vital!

The second soft-skills class is about proper handoffs and escalation—how to communicate when roping in help from other people. This teaches people the following:

• How to prevent misunderstanding by using very explicit communications

• How to behave, escalate, and ask questions during stressful situations, in such a way that the communication goes as smoothly as possible

Powerups

Some teams (most notably those with highly critical systems) use a mechanism called powerups. This works just like in video-games, in which a powerup gives a player certain abilities after having achieved an intermediate goal. In Google’s case, after a new team member has demonstrated that they’ve acquired the knowledge and skills needed for the “next level,” they are granted more permissions to manipulate the systems they are responsible for. There are usually multiple tiers. Having these powerups not only makes sure that people on the team with the permissions to administer the systems, have the skills for that, but also gives the new team members the confidence that they actually are “at the next level.”

Shadowing

An important part of ramping people up to be on-call is shadowing an experienced team member who is on-call. The engagement here varies from “looking over the shoulder” of the on-caller, to an almost reverse situation in which the new team member is driving the investigation, with the experienced team member only supervising and encouraging. Of course, it depends on the urgency of the outage, and how involved the shadower is.