Chapter 39. The Maestros of Incident Response
Andrew Louis
We’ve all been there: the first time we’re the IMOC (Incident Manager On-Call, or Incident Commander, others might call it). My first IMOC page hit a year into the gig and, regardless of all the observing I did before, my handling of it paled compared to the performances before mine. It wasn’t my last fumble, but I began to build a high-level framework for incident management. With each fumble that followed, I added something new to it. The framework has remained valuable as a starting point, and I hope it will be helpful to you too.
There is more and better material dedicated to expanding how to manage an incident, but here are the primary principles that I keep at the forefront.
Stop the Bleeding
Keep the focus unrelentingly on prioritizing mitigation. Although the conversation might drift into deep root-cause investigations and discussions of longer-term solutions, the first impulse should be to keep the ongoing conversation focused solely on recovering the current situation.
What’s Everyone Doing?
At regular intervals (be wary of the cost this could impose on folks working on the problem), continue to raise the question of what everyone is doing. The goal here will be to keep track of the efforts, prevent overlapping work from going on, and get health checks from the parties involved.
Raising this question also gives you the opportunity to ask another—Do you need any help?—to gauge whether more resources should be leveraged.
As you work through the incident management process, it might not be obvious when to move to a next step. Perhaps you could be a bit more certain about which systems are affected if you spent an extra five minutes gathering some more data points. In this scenario, always optimize for speed over quality as you make your decisions, keeping the big-picture goal in mind: to recover your systems fast.
At this point, you might realize that a lot of working through incident response boils down to building the muscle memory and neural pathways that come from repeated experience, but that doesn’t mean you can’t prepare for the rotation.
Here’s a little starter preparedness checklist:
What are the organization’s key metrics? In an e-commerce organization, this may be checkout rate and volume, storefront availability, and so on.
Do you have a sense of how to index from services to owners?
Are you able to get a sense of the ongoing alerts going off across the organization?
Regardless, all the frameworks in the world won’t prepare you nearly enough for the first time you’ll lead an incident. My first hit around 4 a.m. in Toronto and my performance felt like the Chernobyl of incident management. In a seat formerly warmed by maestros, I was a clumsy, amateur conductor.
A lot of working through incident response boils down to building muscle memory and neural pathways that come from repeated experience. Over the many pages that followed, with patience and practice, I started to fumble less and less and soon started getting the orchestra to play some bops.