CHAPTER 2: INTO THE FIRE

They call them War Rooms for a reason. Ours was a commandeered windowless conference room. An old wrinkled sheet of copy paper with the phrase “RESERVED FOR WAR ROOM,” was taped askew to the door. The word “WAR” circled by an ancient coffee stain. Inside were a long rectangular table, and an enormous whiteboard on the wall at the far end. The room was overfilled with people, and everyone seemed to be talking at once. On the table, two speakerphones were hosting different conversations while people clustered around them. Those not clustered around the phones were sitting around the table, or against the wall, in chairs wheeled in from nearby offices for the occasion. Judging by the stains on the carpet, and the marks on the wall, the room hadn’t been cleaned in a long time.

Sean nudged me and whispered, “This is the part where the incident manager takes charge of the meeting.” He did an exaggerated look around the room and said, “Since I don’t see Sarah here, I guess that means you.”

This was my first War Room at the company. I knew Sean was right, and I needed to take charge now that Sarah was gone. I hesitated, because I wasn’t sure I could start the meeting, or control it, without guidance from someone like Sean who was more familiar with the people and workings of the company.

Sean’s phone rang and he stepped just outside the room to take the call. It didn’t seem like he realized I could hear his end of the conversation, as well as see him.

“Yes, Jessica,” said Sean. “Chris is here now. I made sure of that. Running the meeting is another story.” He paused and nodded.

Sean kept talking and gesturing, “Yes, I can imagine sales is pretty torqued about now. Can you hold them off for a little while, until I get things organized and we can give you an estimate?”

Sean nodded several times while listening. “Okay. I understand.” After a pause, he turned his back to me and whispered quietly, but I could still make out his words, “I’m worried Chris can’t handle this – not the right type for incident management. You sure you don’t want me to take over and get it under control?” After a few nods he said, “Okay,” and hung up.

I tried motioning to people to quiet down and sit, but almost no one paid any attention. I started asking people one by one, when Sean said to me, “Watch and learn.”

Sean slammed the whiteboard hard with the flat of his hand. Before the sound faded, he yelled, “Shut up and sit down. Now!”

The room quickly got silent, except for the clatter of keyboards, and the beeping of messages leaving and arriving on a variety of electronic devices.

Sean whispered to me, “Good thing I’m your friend. I had to go to bat for you with Jessica just now. She was ready to punt you out and have me take over. But I convinced her you needed some time to succeed.” He paused, “So you can owe me one for that. It’s your show, but you better get to it. Patience is not an inexhaustible resource.”

Before I could respond, my phone beeped as a series of text messages came rolling in. The first one was a hysterical all-caps message from Jason, the VP of sales and marketing, wanting to know when they could begin making sales again. The next one was from Jessica, the CIO. That one was less hysterical, but wanting a firm commitment when it would be fixed. And then one from Ramesh followed, asking for a detailed status on the cause and time to fix.

I decided to chance waiting until I knew more.

Mimicking Sean, I slammed my hand flat on the table and told the room, “Settle down. We need to get to work. I need an update on what we know about what happened to take the business off the network. Who’s got some updates?”

Nicola, leader of the firewall team, was chugging a can of energy drink and had two more on the table in front of him. I remembered that he’d been putting in a number of approved changes last night, so he probably hadn’t been to sleep yet. In his thick Ukrainian-accented English he said, “Where is Sarah and who are you?”

In the chaos of the moment, I started to say something, but remembered that Jessica and Ramesh had told me not to say anything about Sarah until there was a formal announcement that put the proper perspective on her, “transition”. Their instructions had probably been the right thing to say because of the way they seemed to identify incident response with Sarah. I was having enough trouble trying to figure out how to keep this group focused. Her rapid departure would probably have been a major distraction from getting the service restored.

Fortunately, Ramesh’s voice rose up on one of the speakerphone conversations. “This is Ramesh. I can answer that. Sarah is handling something else for Jessica right now. Chris will be leading restoration from … ”

Nicola didn’t let him finish. “Why are we changing? What is the reason? Does Chris know how do to Sarah’s job? I have never seen Chris here before. Do you realize this risks further impacting the business, if Chris cannot perform at Sarah’s level? This is a distraction that we do not need.”

Irritation coated Ramesh’s voice. “Nicola, Jessica has made this decision. We all have to have faith in her as our leader, and that she has made the right decision for the company as a whole. I know that she is counting on you, and the rest of the IT staff, to rely on Chris’ leadership to get this outage resolved, so we can better support our business partners. She expects each of you to give Chris your complete co-operation and support, so we can minimize the impact on the business of this service disruption. If you disagree with Jessica, I will be happy to arrange some one-on-one time between her and you later today. Any questions?”

Nicola tossed his empty energy drink can at the trash can, but missed. “Understood,” he snapped. “That won’t be necessary.”

That was my cue. I stepped up to the head of the table. “My name is Chris and I’ll be leading the restoration of this disruption. What are the … ”

Sean jumped over my words and began to write on the whiteboard. “Okay, let’s help Chris. You all know the drill. What do we know? Where did the alert come from? Who is impacted? How severe is it? Is anybody currently working on it?”

At first I was ticked at Sean for stealing my role, but as I watched him work, I was amazed at how, within a few minutes, Sean pulled and effectively organized on the whiteboard all the information the group had about the incident, added his own substantial knowledge, and assigned owners to restoration tasks.

“Good, so we are all agreed then?” said Sean as he filled in the last bit of information on the whiteboard. “The root cause was the print server crashing.”

Nicola stood up and began drawing lines on the whiteboard that looked like a cross between an organization chart and a sentence diagram.

“What’s that?” I asked.

Nicola stopped and stared at me like I had asked him why he was breathing. “Obviously you have no training in root cause.” He waved his hand at the board. This is an Ishikawa fishbone diagram, to determine root cause. We need to know how to prevent it from happening again.”

I shook my head, “We shouldn’t focus on the root cause right now. We know what broke and took the store printing offline. We need to make sure we can get that service working for the business again. That is job one. We can worry about cause later.”

“You must understand, Chris,” said Nicola. “The root cause is what we seek. By identifying it, we learn about what fails in our environment. That is the primary function of our incident team; to respond to weaknesses, then identify how to prevent them from happening again. Otherwise we risk fixing the wrong thing.”

Nicola turned back to the whiteboard and continued talking as he worked. “To ensure maximum efficiency, we have instituted a standard root cause methodology. Every incident has its root cause done the same way, using the same methodology. That ensures we use tools with which we are experienced, so we know we will have the correct cause for the incident, before we waste efforts on repairing the wrong thing.”

Nicola began filling out the fishbone and highlighting the conditions involved in the outage in red. When I asked if all of those items connected to the outage were really contributors to the outage, Nicola shook his head. “You see. You do not understand the proper way to conduct incident root cause. You will learn the importance of these things if you pay attention to how we work.”

In a few minutes, Nicola had finished filling out the chart and came to the same conclusion as Sean. The only difference was that Sean understood it instinctively, whereas Nicola was totally dependent on the methodology.

“Clearly, it is the Windows^® platform team that owns this root cause. Glad I’m not getting tagged for that outage,” said Nicola. “I’m already on the ‘Grid’ too many times.”

Before I could ask what the “Grid” was, Sean pointed to a worn page from a flipchart someone had taped onto the whiteboard. It was covered with handwritten squares, with people’s names across the top, and dates down the side. Some of the intersecting squares had numbers. Some were blank. A few of the names had lines through them, crossing them out. “It’s a rolling matrix by team and date, of the number of outages they own,” said Sean.

I pointed to the crossed out names.

“If you get too many, you get set free to extend your success elsewhere.” A few people in the room laughed. “This isn’t the official one. Only leadership gets to see that. This is our copy we keep, so that you can see when you’re getting in trouble.”

“How many marks do you have to have to be … set free?” I asked.

Sean shook his head. “Don’t know. No one has been able to figure out the algorithm leadership uses. You never really know how many mistakes are tolerated, and how many are too much. It seems to vary, depending on which VP is looking at it, and how loud the business is yelling.” He crossed out Sarah’s name and stepped back, like a painter checking his perspective. “Something less than what Sarah had, that’s for sure.”

I watched on my laptop as the rebooted Windows^® print servers came back up. As soon as they seemed stable, I tested them out, to make sure they worked. I thought that would be a nice little piece of proactive work for my first day.

Satisfied they were running again, I stepped into the hall, and returned the call from Jason, the VP of sales and marketing. I was 10 minutes later than I had promised in my text, but figured he wouldn’t mind, now that everything was working again.

“Hi, this is Chris from IT. I’m just following up to let you know that … ”

“To let me know that everything IT touches turns to crap. Where the hell have you been?” he screamed into the phone. “I don’t appreciate being dragged away every few hours just to tell IT how to do its job. Every night I get called because IT is incompetent and can’t keep their servers running, but it now appears that either none of you own watches’ or else you don’t believe in meeting your commitments. Which is it? Are you clueless or just plain irresponsible?”

I took two slow breaths and fought the impulse to respond immediately. As the urge passed, I responded, “I’m very sorry the service went down. I wish I could go back and prevent it from happening, but I cannot. What I can do, is promise you we will take steps to prevent it from happening again. I will be personally responsible for that to you.”

Jason interrupted me, “That’s what your predecessor said and look where it got her. But I’m going to hold you to your commitment,” he paused for a moment to let his words sink in. “And after you fail, I’ll hold your successor accountable, too. And if that doesn’t work, I’ll see to it that IT gets outsourced to a firm that knows what it’s doing.”

He paused for a moment, and then said something to someone else in the room with him. All I caught was, “ … be right there.”

Jason was a little more composed when he said, “You do understand that if I don’t meet my commitments, then IT doesn’t get money for its fancy toys, don’t you? And if you can’t meet even the most basic of tasks, and learn not to break anything, then perhaps we are giving IT too much money, and you’re all distracted by your techie toys. Think about how IT would do if we cut your funding by 20%, or if we started outsourcing your work. Consider that, the next time you get the urge to break something.”

He hung up without even saying goodbye. Actually, I was grateful. Besides, we’d gotten the service back up and running. It hadn’t been pretty, but at least it got done. I could hardly wait to tell Ramesh all the details. I knew he was going to be happy and very pleased with my performance.