Chapter 9. Managing Crises and Escalations

There is no instance of a country having benefitted from prolonged warfare.

—Sun Tzu

You’ve probably never seen the 404 (page not found) error on Etsy’s site (www.etsy.com). The error page is a cartoon that depicts a woman knitting a three-armed sweater (Figure 9.1). Etsy is the de facto marketplace for handmade or vintage items, and as such it seems only fitting that the company commissioned an actual three-armed sweater that it uses to reward the engineer who most spectacularly brings down the site. While this fun fact is indicative of the amazing culture that thrives in Etsy’s technology teams, what’s more interesting about this 404 page is the site incident that it caused a few years ago. John Allspaw, Senior VP of Technical Operations at Etsy and author of The Art of Capacity Planning (O’Reilly Media, 2008) and Web Operations: Keeping the Data on Time (O’Reilly, 2010), tells the story. It all started with the removal of a CSS file. This particular CSS file supported outdated versions of Internet Explorer browser (IE versions less than 9.0). Everything appeared normal for a few minutes after the removal of the file—but then the CPU cycles on all 80 Web servers spiked to 100%.

Figure 9.1 Etsy’s 404 Error Page

Allspaw and his team have been incredibly focused on monitoring; they monitor more than 1 million time-based metrics such as registrations and payment types. All of this data gives the operations and software developers incredible insight into the systems. In Allspaw’s story, the monitors almost immediately began alerting that something was amiss. A combination of tech ops and software developers gathered in a virtual IRC chat room as well as a physical conference room in Etsy’s Brooklyn office. The team formed hypotheses and quickly began vetting them. Anthony, the engineer who would eventually get awarded the three-armed sweater for this incident, determined that during the space of time between the rsync of the static assets and the execution of the code (i.e., the Smarty tag that checked asset version), an error was being thrown. The code in question was executed on every request, not just IE browsers, which consumed all the available CPU cycles on all the servers.

Now that Allspaw’s team had identified the problem, the fix should have been easy—put back the missing CSS file. But not so fast: With all the CPU being consumed by the infinite loop, there wasn’t even enough CPU to allow an SSH connect onto the servers. The administrators connected to the iLO (integrated-Lights Out) cards to reboot the servers, but as soon as they came back online, the CPU spiked. The team started pulling servers out of the load balancer and then rebooting them, which gave them time to SSH into the servers and replace the CSS file. The administrators and network engineers were able to do this so quickly because Allspaw’s team had practiced synchronization and sequencing exercises many times before this incident. Because they already had a communication plan in place, they were able to communicate with their customers during the entirety of this incident through both the site EtsyStatus (http://etsystatus.com/) and the Twitter account @etsystatus. These communications reassured customers that Etsy was working on the problem.

Etsy is one of the best-prepared and best-executing teams in the industry. But even this execution prowess and preparedness doesn’t keep the company from having issues and needing to award the three-armed sweater to an engineer. Etsy understands how critical it is to respond quickly and appropriately to incidents when they happen—and take our word for it, they will happen. In this chapter, we discuss how to handle major crises both during the incident and afterward.

How do you know if an incident has risen to the level sufficient to qualify it as a crisis? To answer this question, you need to know the business impact of an incident. Perhaps a 30- to 60-minute failure between 1 a.m. and 1:30 a.m. is not really a crisis situation for your company, whereas a 3-minute failure at noon is a major crisis. Your business may be such that you make 30% of your annual revenue during the three weeks surrounding Christmas. As such, downtime during this three-week period may be an order of magnitude more costly than downtime during the remainder of the year. In this case, a crisis situation for you may be any downtime between the first and third weeks of December, whereas at any other point during the year you are willing to tolerate 30-minute outages. Your business may rely upon a data warehouse that supports hundreds of analysts between the hours of 8 a.m. and 7 p.m., but has nearly no usage after 7 p.m. or anytime during weekends. A crisis for you in this case may be any outage during working hours that would idle your analysts.

Of course, not all crises are equal, and obviously not everything should be treated as a crisis. Certainly, a brownout of activity on your Web site for 3 minutes Monday through Friday during “prime time” (peak utilization) is more of a crisis than a single 30-minute event during relatively low user activity levels. Our point here is that you must determine the unique crisis threshold for your company. Everything that exceeds this threshold should be treated as a crisis. Losing a leg is absolutely worse than losing a finger, but both require immediate medical attention. The same is true with crises; after the predefined crisis threshold is passed, each crisis should be approached the same way.

Recall from Chapter 8, Managing Incidents and Problems, that recurring problems (those problems that occur more than once) rob you of time and destroy your ability to scale your services and scale your organization. Crises also ruin scale, because they steal inordinately large amounts of resources. Allowing the root cause of a crisis to surface more than once will not only waste vast resources and keep you from scaling your organization and services, but also carries the risk of destroying your business.

Although we believe that there is a point at which adding resources to a project has diminishing and even negative returns, in a crisis you are looking for the shortest possible time to resolution rather than the efficiency or return on those resources. A crisis is not the time to think about future product delivery, as such thoughts and their resulting actions will merely increase the duration of the crisis. Instead, you need to lead by example and be at the scene of the crisis for as long as it is humanly possible and eliminate all other distractions from your schedule. Every minute that the crisis continues is another minute that destroys shareholder value.

Your job is to stop the crisis from causing a negative trend within your business. If you can’t fix it quickly by getting enough people on the problem to ensure that you have appropriate coverage, three things will happen. First, the crisis will perpetuate itself. Events will happen again and again, and you will lose customers, revenue, and maybe your business. Second, in allowing the crisis to siphon precious time out of your organization over a prolonged period, you will eventually lose traction on other projects. The very thing you were trying to avoid by not putting “all hands on deck” will happen anyway and you will have allowed the problem to go on longer than necessary. Third, you will lose credibility.

Our job, of course, is to avert crises whenever possible given their cost to shareholders and employees. But when crises happen (and they will), we must adhere to the adage that “A crisis is a terrible thing to waste.” This adage indicates a strong desire to maximize the learning from a crisis through diligent postmortem assessments, and as such strengthen our people, processes, technologies, and architectures.

If left unmanaged, the collision of a large number of people from multiple organizations within a crisis situation will create chaos. This chaos will feed on itself, creating a vicious cycle that can actually prolong the crisis or—worse yet—aggravate the damage done in the crisis through someone taking an ill-advised action. Indeed, if you cannot effectively manage the force you throw at a crisis, you are better off using fewer people.

Your company may have a crisis management process that consists of both phone and chat (instant messaging or IRC) communications. If you listen on the phone or follow the chat session, you are very likely to see an unguided set of discussions and statements as different people and organizations go about troubleshooting. You may have questions asked that go unanswered or requests to try something that go without authorization. You might as well be witnessing a grade school recess, with different groups of children running around doing different things with absolutely no coordination of effort. But a crisis situation isn’t a recess—it’s a war, and in war such a lack of coordination results in an increased casualty rate through “friendly fire.” In a technology crisis, these friendly casualties are manifested as prolonged outages, lost data, and increased customer impact.

You must control the chaos. Rather than a grade school recess, you hope to see a high school football game. Ideally, you will witness a group of professionals being led with confidence to identify a path to restoration and subsequently a path to identification of root causes.

Different groups should have specific objectives and guidelines unique to their expertise. For all groups, there should be an expectation that they will report their progress clearly and succinctly in regular time intervals. Hypotheses should be generated, quickly debated, and either prioritized for analysis or eliminated as good initial candidates. These hypotheses should then be quickly restated as the tasks necessary to determine validity and handed out to the appropriate groups to work them, with times for results clearly communicated.

Someone on the call or in the crisis resolution meeting should be in charge, and that someone should be able to paint an accurate picture of the impact, the steps that have been tried, the best hypotheses being considered, and the timeline for completion of the current set of actions. Other members should be managers of the technical teams assembled to help solve the crisis and one of the experienced (described in organizations as senior, principal, or lead) technical people from each manager’s teams. We will now describe these roles and positions in greater detail. Other engineers should be gathered in organizational or cross-functional groups to deeply investigate domain areas or services within the platform experiencing a crisis.

The characteristics of the person who is capable of successfully managing chaotic environments are rather unique. As with leadership, some people are born with the ability to make order out of chaos, whereas others build this skill over time. An even larger group of folks have neither the desire nor the skills to lead in a time of crisis. Putting these people into crisis leadership positions can be disastrous. The person leading the crisis team absolutely needs to be technically literate, but does not necessarily need to be the most technical person in the room. This individual should be able to use his technical base to formulate questions and evaluate answers relevant to the crisis at hand. The problem manager does not need to be the chief problem solver, but rather needs to effectively manage the process followed by the chief problem solvers gathered within the crisis. This person also needs to be incredibly calm “inside,” yet persuasive “outside.” This might mean that the problem manager has the type of presence to which people are naturally attracted, or it might mean that he isn’t afraid to yell to get people’s attention within the room or on the conference call.

The problem manager needs to be able to speak and think in business terms. He needs to be sufficiently conversant with the business model to make decisions in the absence of higher guidance on when to force incident resolution over attempting to collect data that might be destroyed and would be useful in problem resolution (remember the differences in definitions from Chapter 8). The problem manager also needs to be able to create succinct, business-relevant summaries from the technical chaos that’s occurring to keep the remainder of the business informed about the crisis team’s progress.

In the absence of administrative help to document the event, the problem manager is responsible for ensuring that the actions and discussions are represented in a written form for future analysis. Thus, the problem manager will need to keep a history of the crisis and ensure that others are keeping their own histories, to be merged later. A shared chat room with timestamps enabled is an excellent choice for this type of documentation.

In terms of Star Trek characters and financial gurus, the ideal problem manager is one-third Scotty, one-third Captain Kirk, and one-third Warren Buffet. He is one-third engineer, one-third manager, and one-third business manager. He has a combat arms military background, an MBA, and an MS in some engineering discipline. Clearly, it will be difficult to find someone with the right blend of experience, charisma, and business acumen to perform such a function. To make the task even harder, when you find the person, he probably will not want the job, because it is a bottomless pool of stress. However, just as some people enjoy the thrill of being an emergency room physician, some operators enjoy leading teams through technical crises.

Although we flippantly suggested the MBA, MS, and military combat arms background, we were only half kidding. Such people actually do exist! As we mentioned earlier, the military has a role for people who manage its battles or what most of us would view as crises. The military combat arms branches attract many leaders and managers who thrive on chaos and are trained and have the personalities to handle such environments. Although not all former military officers have the right personalities, the percentage who do is significantly higher than in the rest of the general population. As a group, these leaders also tend to be highly educated, with many of them having at least one and sometimes multiple graduate degrees. Ideally, you would want someone who has proven himself both as an engineering leader and as a combat arms leader.

A team manager functioning solely in a management capacity is expected to manage her team through the crisis resolution process. The majority of this team will be located somewhere other than the crisis resolution (or “war”) room or on a call other than the crisis resolution call if a phone is being used. This means that the team manager must communicate and monitor the progress of the team as well as interact with the crisis manager. Although this may sound odd, the hierarchical structure with multiple communication channels is exactly what gives this process so much scale. This structured hierarchy affects scale in the following way: If every manager can communicate and control 10 or more subordinate managers or individual contributors, the capability in terms of human resources grows by orders of magnitude. The alternative is to have everyone communicate in a single room or through a single channel, which obviously doesn’t scale well. In such a scenario, communication becomes difficult and coordination of people becomes near impossible. People and teams quickly drown each other out in their debates, discussions, and chatter. Very little gets done in such a crowded environment.

Furthermore, this approach of having managers listen and communicate on two channels has proved very effective for many years in the military. Company commanders listen to and interact with their battalion commanders on one channel, and issue orders and respond to multiple platoon leaders on another channel (the company commander is in the upper-left portion of Figure 9.2). The platoon leaders then do the same with their platoons; each platoon leader speaks to multiple squads on a frequency dedicated to the platoon in question (see the center of Figure 9.2). Thus, although it might seem a bit awkward to have someone listen to two different calls or be in a room while issuing directions over the phone or in a chat room, this concept has worked well in the military since the advent of radio. It is not uncommon for military pilots to listen to four different radios at one time while flying the aircraft: two tactical channels and two air traffic control channels. In our consulting work, we have employed this approach successfully in several companies.

Figure 9.2 Military Communication

The voice channel should be used to issue commands, stop harmful activity, and gain the attention of the appropriate team. It is absolutely essential that someone from each of the teams listens on the crisis resolution voice channel and simultaneously controls his or her team. In many cases, two representatives—the manager and the senior (or lead) engineer—should be present from each team on such a call. They serve as the command and control channel in the absence of everyone being in the same room. All shots are called from here, and this channel serves as the temporary change control authority and system for the company. The authority to do anything other than perform nondestructive “read” activities like investigating logs is first approved within this voice channel or conference room to ensure that two activities do not compete with each other and either cause system damage or result in an inability to determine which action “fixed” the system.

The chat or IRC channel is used to document all conversations and easily pass around commands to be executed so that time isn’t wasted in communication. Commands that are passed around can be cut and pasted for accuracy. Additionally, the timestamps within the IRC or chat can be used in follow-up postmortem sessions. The crisis manager is responsible for ensuring not only that he puts his own notes in the chat room and writes his decisions in the chat room for clarification, but also that status updates, summaries, hypotheses, and associated actions are put into the chat room.

In our experience, it is absolutely essential that both the synchronous voice and asynchronous chat channels remain open and available during any crisis. The asynchronous nature of chat allows activities to go on without interruption and allows individuals to monitor overall group activities between the tasks within their own assigned duties. Through this asynchronous method, scale is achieved while the voice channel allows for immediate command and control of different groups for immediate activities. Should everyone be in one room, there is no need for a phone call or conference call other than to facilitate experts who might not be on site and to give updates to the business managers. But even when everyone is present in the same room, a chat room should be opened and shared by all parties. In the case where a command is misunderstood, it can be buddy checked by all other crisis participants and even “cut and pasted” into the shared chat room for validation. The chat room allows actual system or application results to be shared in real time with the remainder of the group, and an immediate log with timestamps is generated when such results are cut and pasted into the chat.

In the ideal case, the war room is equipped with phones, a large table space, terminals capable of accessing systems that might be involved in the crisis, plenty of work space, projectors capable of displaying key operating metrics or any person’s terminal, and lots of whiteboard space. Although the inclusion of a whiteboard might initially appear to be at odds with the need to log everything in a chat room, it actually supports chat activities by allowing graphics, symbols, and ideas best expressed in pictures to be drawn quickly and shared. These concepts can then be reduced to words and placed in chat, or a picture of the whiteboard can be taken and sent to the chat members. Many new whiteboards even have systems capable of reducing their contents to pictures immediately. Should you have an operations center, the war room should be close to that to allow easy access from one area to the next.

You might think that creating such a war room would be a very expensive proposition. “We can’t possibly afford to dedicate space to a crisis,” you might say. Our answer is that the war room need not be expensive or dedicated to crisis situations. It simply needs to be given a priority in any crisis. As such, any conference room equipped with at least one and preferably two lines or more will do. Moreover, the war room is useful for the “ride along” situation described in Chapter 6. If you want to make a good case for why you should invest in creating a scalable organization, scalable processes, and a scalable technology platform, invite some business executives into a well-run war room to witness the work necessary to fix scale problems that result in a crisis. One word of caution here: If you can’t run a crisis well and make order out of its chaos, do not invite people into the fray. Instead, focus your time on finding a leader and manager who can run such a crisis and then invite other executives when you feel more confident about your crisis management system.

The natural tendency for engineering teams is to believe that they can solve the problem without outside help or without help from their management teams. That may be true, but solving the problem isn’t enough—it needs to be resolved the quickest and most cost-effective way possible. Often, that will require more than the engineering team can muster on their own, especially if third-party providers are to blame for the incident. Moreover, communication throughout the company is important, because your systems are either supporting critical portions of the company or—in the case of Web companies—they are the company. Someone needs to communicate with shareholders, partners, customers, and maybe even the press. People who aren’t involved in fighting are the best options to handle that communication.

Think through your escalation policies and get buy-in from senior executives before you have a major crisis. It is the crisis manager’s job to adhere to those escalation policies and get the right people involved at the time defined in the policies, regardless of how quickly the problem is likely to be solved after the escalation.

A status message should include the start time, a general update of actions since the start time, and the expected resolution time if known. This resolution time is important for several reasons. Perhaps you are supporting a manufacturing center, and the manufacturing manager needs to know if she should send her hourly employees home. Or perhaps you provide sales or customer support software in SaaS fashion, and those companies need to figure out what to do with their sales and customer support staff.

Your crisis process should clearly define who is responsible for communicating to whom, but it is the crisis manager’s job to ensure that the timeline for communications is followed and that the appropriate communicators are properly informed. A sample status email is shown in Figure 9.3.

Figure 9.3 Status Communication

Absolutely everything should be evaluated. The very first crisis postmortem is referred to as the “master postmortem,” and its primary task is to identify subordinate postmortems. It is not intended to resolve or identify all of the issues leading to the incident, but rather to identify the areas that subordinate postmortems should address. For example, you might have postmortems focused on technology, process, and organization failures. You might have several postmortems on technology covering different aspects—one on your communication process, one on your crisis management process, and one on why certain organizations didn’t contribute appropriately early on in the postmortem.

Just as you had a communication plan during your crisis, so you must have a communication plan that remains in effect until all postmortems are complete and all problems are identified and solved. Keep all members of the RASCI chart updated, and allow them to update their organizations and constituents. This is a time to be completely transparent. Explain, in business terms, everything that went wrong and provide aggressive but achievable dates in your action plan to resolve all problems. Follow up with communication in your staff meeting, your boss’s staff meeting, and/or the company board meeting. Communicate with everyone else via email or whatever communication channel is appropriate for your company. For very large events where morale might be impacted, consider conducting a company all-hands meeting, to be followed by weekly updates via email or on a blog.

The technology team is charged with responding to, resolving, and handling the problem management aspects of a crisis. The roles on this team include the problem manager/crisis manager, engineering managers, senior engineers/lead engineers, and individual contributor engineers from each of the technology organizations.

Four types of communication are necessary in crisis resolution and closure: internal communications, escalations, and status reports both during and after the crisis. Handy tools for crisis resolution may also be employed, including conference bridges, chat rooms, and the war room concept.

Teams who either don’t see crises a lot or are new to the process should consider drilling in crisis management. Practice what each person would do so that when an actual crisis occurs—and it inevitably will—the team is amply prepared to manage it effectively.

• To resolve crises as quickly and cost-effectively as possible, you must contain the chaos with some measure of order.

• The leaders who are most effective in crises are calm on the inside but capable of forcing and maintaining order throughout the crisis management process. They must have business acumen and technical experience and be calm leaders under pressure.

• The crisis resolution team consists of the crisis manager, engineering managers, and senior engineers. In addition, teams of engineers reporting to the engineering managers are employed.

• The role of the crisis manager is to maintain order and follow the crisis resolution, escalation, and communication processes.

• The role of the engineering manager is to manage her team and provide status to the crisis resolution team.

• The role of the senior engineer from each engineering team is to help the crisis resolution team create and vet hypotheses regarding cause and to help determine the most appropriate rapid resolution approaches.

• The role of the individual contributor engineer is to participate in his team and identify rapid resolution approaches, create and evaluate hypotheses about the cause of the crisis, and provide status information to his manager on the crisis resolution team.

• Communication between crisis resolution team members should happen face to face in a crisis resolution or war room; when face-to-face communication isn’t possible, the team should use a conference bridge on a phone. A chat room should also be employed.

• War rooms, which are ideally sited adjacent to operations centers, should be developed to help resolve crisis situations.

• Escalations and status communications should be defined during a crisis. After a crisis, the crisis process should provide status updates at periodic intervals until all root causes are identified and fixed.

• Crisis postmortems should be vigorous examinations that identify and manage a series of follow-ups in the form of subordinate postmortems that thematically attack all issues identified in the master postmortem.