SEV roles and responsibilities

Two key roles are critical in creating a successful high-severity incident-management program:

IM

Incident manager on call

TL

Technical lead on call

Roles and responsibilities of the IM

The primary role of the IM is to resolve SEVs in a safe and fast manner. At many companies, this person has the authority to take whatever steps are needed to resolve the incident. A key aspect is having one person to make the judgement calls needed when time is short.

IMs lead and coordinate the SEV team through the SEV life cycle. The SEV life cycle encompasses detection, diagnosis, mitigation, prevention, and closure. The IM role is also commonly referred to as the call leader.

Having only one person in the company responsible as the leader of an SEV results in faster resolution times. Having a chain of command during urgent events enables the entire company to react as quickly as possible. This is measured as MTTR.

Having an IM responsible for an SEV provides many benefits:

• The IM is focused on time to recovery (TTR) for the SEV.

• The IM aims to resolve SEV 0s within 15 minutes.

• Everyone in the company knows who to follow as a top priority.

• The IM uses their leadership skills to keep the team calm.

• Engineers working on SEV mitigation can remain focused on the technical work required to recover. They are required to provide only status updates to the IM.

• The IM keeps the entire company updated on the progress of the SEV.

The IM plays a critical role during these SEV 0s. The entire company is notified and the external status page is updated to notify customers. The IM aims to resolve SEV 0s within 15 minutes. The IM promotes the following principles:

• Staying calm

• Working as a team

• Following the lead of the IM

• Communicating clearly and concisely

• Prioritizing focus on the right things at the right time

The IM creates and facilitates any required chat channels, conference calls, video calls, or in-person SEV rooms. They use the most suitable communication methods that enable them to work effectively with everyone actively working on the SEV.

They keep everyone on the same page by creating and updating an SEV timeline during the SEV. The timeline includes what actions are happening and who is responsible. The IM makes sure to identify and raise anything that has changed during the SEV.

IMs have a wide knowledge of services and engineering teams. They have an understanding of all major changes that are happening across all services. They are aware of product launches and of migrations, and of changes to team services and structure. The IM stays calm and collected at all times. They have an ability to focus and drive the entire company toward mitigation and resolution.

The IM rotation team is responsible for supporting the IM. This is usually a team of fewer than 10 engineers across the company. The IM rotation team works together to proactively ensure that the entire company understands how SEVs are managed and categorized. The more effective the entire company is at categorizing and communicating SEVs, the quicker the IM can gauge the priority and impact of SEVs.

The IM rotation is a small rotation of engineering leaders, with one person on call in this role at any point in time. It is a 24/7 rotation with one primary IM and one secondary IM on call for a week. The secondary is also on call 24/7 and is there to pick up any pages that slip through past the primary. Secondaries are a backup for the primary and need to be available at any time so that the primary can escalate to them if needed. The secondary role is commonly activated in one of two ways. The first activation mode is when a page is not acknowledged by the primary within the set period of time (e.g., one minute), and the page is then automatically sent to the secondary. In the second mode of activation the primary pages the secondary and asks them to assist as a backup for an SEV.

Figure 1-1 shows an example of a five-person rotation.

Figure 1-1. Example of a five-person IM rotation structure

It is useful for the IM rotation to hold a monthly synchronization in which all IMs are allocated time to share feedback, and raise and review action items.

IM training is best conducted in a one-hour face-to-face training session with time for questions. Training involves shadowing and instruction sessions conducted to give IMs an understanding of the following:

• SEV levels

• The full life cycle of SEVs

• Examples of previous SEV 0s that have occurred within your company

• Access to an IM runbook with communications templates

• Gamedays

It is vital to train IMs before they do their first rotation because they will be the sole person responsible for leading the entire company toward resolution of the SEV 0. IMs will first shadow as “super primary” before being added as a primary or secondary. If they are added as a super primary, this means that they will be paged the same as the primary.

Roles and responsibilities of the TL

TLs are technical experts from different service areas charged with diagnosing, mitigating, and resolving SEVs as quickly and safely as possible. But they aren’t burdened with keeping engineers calm or keeping management in the loop—that’s the IM’s job. Rather, a TL settles in the trenches and stays laser-focused on technical problem solving, calling up to the IM for help—or to give status updates—only when necessary.

Other engineers respect the TL’s need to focus, but they are ready to jump in and help when called on. The TL works heroically, but not alone!

After an SEV, the TLs work with their service teams to determine its root cause and create action items; for example, fixing a bug or deprecating some legacy system. After any fixes, the TLs lead chaos experiments to ensure the SEV doesn’t recur. These experiments are like integration tests but for your entire application stack.

Over time, this post-SEV practice improves mean time between failures (MTBF) and mean time to prevention (MTTP).

TLs take turns being on call, of course, and this is known as a TL rotation. If your engineering team is small (e.g., five engineers), you’ll create a single rotation that covers all service areas. For a larger engineering team (e.g., 50 engineers), you’ll create one rotation for each service area. How those service areas break down depends on the size of your team.

Suppose that you have 10 engineers. That’s enough for two rotations, given that an ideal rotation has five TLs. (Any more engineers, and no person will be on call often enough to stay sharp; any fewer, and the TLs might burn out.) With two rotations, you need to break down your services into two buckets, as in the following example:

Rotation 1

Infrastructure engineering services: responsible for internal services such as MySQL, Memcache, Amazon S3, Kafka, monitoring, and self-healing software.

Rotation 2

Product engineering services: responsible for customer-facing services such as UI, billing, web apps, desktop apps, and mobile apps.

During any given week, each rotation designates a primary and a secondary TL. At any given moment, however, each rotation has only one acting TL. Letting a single engineer take charge keeps everything moving forward, which improves your MTTD and MTTR.

Figure 1-2 illustrates what your two rotations might look like.

Figure 1-2. Example on-call rotation structure

Notice that each TL serves first as secondary and then as primary the week after. This lets a likely rusty TL warm up as they return for duty.

As your engineering team grows, you’ll add more five-person rotations and redefine your service buckets in whatever way makes sense for your company. For example, if your mobile app is more complex and less stable than other parts of your stack, it might warrant its own TL rotation as your team grows to 15. A different company might give its web app its own rotation.

Because TLs are solely responsible for driving technical resolution of SEVs, new TLs must receive training before their first on-call rotation.

One or more experienced TLs should hold a one-hour, face-to-face training session that covers the following:

• The basics of SEV management programs

• The basics of chaos days (view the O’Reilly Velocity Keynote to learn more)

• Examples of previous SEV 0s at the company, including debugging techniques

• Mitigation techniques for SEV 0s

After training, add each new TL to the pager rotation for their service area—and test that they actually receive pages. Also test that pages roll over to secondary TLs when the primary doesn’t answer within one minute. TLs will first shadow as super primary before they are added as a primary or secondary. Being super primary means that they will be paged the same as the primary.

Finally, give each TL full access to any monitoring, reliability, networking, and performance tools and dashboards.

Determining the SEV level

After being paged, the IM will determine or confirm the SEV level. Ideally this stage takes a maximum of one minute. The SEV level will then do one of the following:

• Stay at the original level

• Be upgraded to a higher level; for example, from SEV 1 to SEV 0

• Be downgraded; for example, from SEV 0 to SEV 1

An SEV level could change due to the IM realizing that the impact is much worse than originally assessed, or that the SEV has been occurring for a long period of time, or is a false alarm.

Routing to a TL to accept responsibility for SEV resolution

The IM triages and routes to an engineer (TL) who can begin working on a resolution.

Key Indicators

The top section of the dashboard in Figure 1-5 is labeled “Key Indicators” to demonstrate the health or lack of health of the particular service in question.

Figure 1-5. Key Indicators dashboard

$Servicename$ Instruments

The next section down on the dashboard includes other metrics and instruments directly measuring the service. We just call this $Servicename$ Instruments; $Servicename$ being a placeholder name for each service. Figure 1-6 shows an example of a Proxy Instruments dashboard.

Figure 1-6. Proxy Instruments dashboard

System Services Health

Next, there is System Services Health. These are all the other services required for the infrastructure to be considered “healthy.” These services are: Secure Shell (SSH), Domain Name System (DNS), Puppet is running, Network Time Protocol (NTP), and Lightweight Directory Access Protocol (LDAP). Figure 1-7 shows an example of a System Services Health dashboard.

Figure 1-7. System Services Health dashboard

Host General Health

After that, you should ideally keep base metrics on every host regardless of which services are running on it. This is the Host General Health dashboard. The next few sections focus on a set of metrics we have identified and honed over the years to give us some clear metrics around how the host is performing outside of the operation of the specific services we care about. These measure things like uptime, CPU load, memory usage, I/O latency, and a few key others that we have identified. Recently, we added SSD wear; we found SSDs with wear measuring over a certain amount—as reported by smartctl (specifically, /usr/sbin/smartctl -l ssd $pathtodevice(s))—were prone to fail, so we preempt that by replacing drives when they reach a particular number. Figure 1-8 shows an example of a Host General Health dashboard.

Figure 1-8. Host General Health dashboard

Host Network Health

Next, there is Host Network Health, which comprises just the general network measurements, TX/RX rates, latency, interface errors, frame errors, and packet loss. Figure 1-9 shows an example of a Host Network Health dashboard.

Figure 1-9. Host Network Health dashboard

How to track SEV impact by service

During the detection phase of an SEV, it is critical to quickly detect on which services your team will need to perform mitigation actions.

Service triage

During service triage, you are working to identify the critical service teams that will need to work on the SEV.

You are also identifying impacted services. For example, a critical issue with the API might be causing a problematic impact for the mobile app, but the mobile app team only needs to be notified. It is unlikely that they will need to perform an action.

Service ownership

Before an incident happens, it is useful to have service ownership in place. This usually involves having a system to track services and the teams that own them. Service ownership is an advanced SRE practice. At a minimum, you should aim to at least be able to identify who owns the service and who the on-call is.

A simple example of a service ownership system is having an owners.yaml file in each code repository directory. A more advanced version is having a web application and command-line tool to view services and their owners.

Service ownership is often easier when there are fewer than 15 services. After that it becomes much more complicated; when there are thousands of services, it becomes very difficult to find service owners. Your service ownership system should answer these questions:

• Which team owns this service?

• Who is on call for this service?

• How do I escalate?

• How do I page this person?

• Where does this person sit? Are they remote or in the office?

• Can I call them on the phone during emergencies?

Building a Service Ownership System

Many companies have built their own Service Ownership Systems (SOS). To build one for your own company, follow these steps:

1. Define a list of services and the teams that own them.

2. Build a Responsible, Accountable, Consulted, and Informed (RACI) model that specifically relates to service impacts and changes, making clear the importance of manager accountability.

3. Automate service ownership.

4. As mentioned earlier, build a comprehensive System Services Health dashboard that shows the following SRE “Golden Signals” (as defined by Google and others) for each service:

   Rate

   Request rate, in number of requests/sec

   Errors

   Error rate, in number of errors/sec

   Latency

   Response time, including queue/wait time, in milliseconds

   Saturation

   How overloaded the resource or system is

   Utilization

   How busy the resource or system is

There is a convincing argument to be made that ownership of services in an application is the most important aspect of production readiness and creating a culture of high reliability.

SOS can take many forms, but generally speaking, a good SOS will contain the following data points:

• Service name

• Single sentence that describes what the service does (this can be the most difficult and time-consuming part of creating a service catalog)

• Brief technical description of how the service is consumed or accessed

• Brief description of the immediate impact of disruption or degradation of the service

• Team responsible for managing the service

• Manager accountable for the service

• RACI model for communicating changes to as well as disruptions/degradations of the service

Having centrally defined ownership is key to managing monitoring, self-healing, and escalation of the service. Open source tools like LinkedIn’s oncall allow for teams to define ownership and on-call teams. It is important to have this information defined before the service goes into production.

Services

When setting up services within your paging and alerting system it is useful to label them by the software name; for example, DynamoDB and/or cluster name. Here is a list of example services:

• MySQL

• MemcacheD

• S3

• Lambda

• Kafka

Service teams

Your service teams should be named as they are known internally within your company. For example, if you have a serverless team, you would add Serverless as a service team. Here is a list of example service teams:

• Traffic and Edge

• Databases

• Messaging

• Storage

Schedules

When you’re setting up schedules in your paging and alerting system, it’s useful to create rotations six months in advance. This enables engineers to know when they are on call, and assists with logistics such as switching on-call schedules during vacation. Here are some recommendations that you might want to follow:

• Create 24-hour, 7-day rotations during which one engineer is a primary and one engineer is a secondary.

• Start schedules on Wednesday at 10 a.m. to enable on-call handoff to occur at 10 a.m. Wednesday.

• Aim to keep rotations for services under 10 people; 5 or fewer is ideal.

Users

Every engineer who is on call will be added as a user to the paging and alerting platform. Here are some recommendations for doing this:

• Create an email alias for each user so that they can be paged via email; for example, [email protected].

• Create a Slack slash command so that each user can be paged via Slack, /page-tammy.

• Have each engineer responsible for running the on-call handoff for their scheduled rotation.

Service teams

Figure 1-10 shows an example of an on-call schedule for a Traffic and Edge team.

Figure 1-10. An on-call schedule for a Traffic and Edge team

Configuring alerting

In general terms, paging alerts should be critical alerts only, for which automated remediation is either not an option because of scale or because the complexity of the alert requires intervention by multiple teams.

An SRE who is paged should be able to begin taking action on the alert within a few minutes. You can sharpen up the granularity of your time scale here, but, in general, you don’t want to page someone and have them still researching the issue and triaging the root cause an hour later.

False positives, alert burden, and alert maintenance

The problem of false positives is a persistent one. SRE teams should commit to reviewing alert architecture at least once per quarter and examining in detail the top 10 most frequent alerts to determine whether there is action needed to mitigate the root cause or if alert thresholds need to be tuned.

SRE managers should intervene and mandate some sort of “time out” period on new infrastructure changes or code deployments if the alert volume for critical, pageable alerts exceeds a 1:1 ratio with SREs.

Alert maintenance is a necessary part of the operational management of a service. If alerts fire frequently (especially with little value to the person receiving it), it can create a phenomenon known as alert fatigue. Alert fatigue leads to desensitization, which can lead to longer response times or missed alarms.