CHAPTER 13: THE RIGHT THING THE WRONG WAY

“Who do you think you are?” snapped Brad, the data centre director, as he paced in front of Ramesh’s desk, just as he had done since arriving 10 minutes ago. His face was red, like bad sunburn. Eyes wide, his speech was jerky and punctuated with sharp jabbing arm gestures. With greying hair looking a little like smoke, he acted like glowing coals about to flashover into a full blown firestorm.

He ignored me, and vented his anger on my manager, Ramesh, seemingly unwilling to waste the energy on me because of a confidence my chastisement would come at the hands of my own manager after he was done.

At my insistence, Ramesh had called in Chester, the change manager. Chester had worked for the company almost his entire career. Chester had built a huge network of friends across the company over the years, and could always be found at every after-hour’s employee party buying drinks for the crowd. He only stumbled into the change manager role two years ago, when his old position was eliminated. His manager at the time had dropped him into that role to keep him from getting laid off.

Chester was not your typical change manager. Not only did he hate refusing change requests, he would go to almost any length to avoid it. He couldn’t stand confrontation, and then dealing with the requestor’s resulting frustration and objections. Fortunately, he was smart enough to assemble a change advisory board that complemented him well. They seemed to revel in confrontation and declining changes. That let him play the good cop, while they played the heavy. Most of the time it worked … except when changes in the process needed to be made. Then there was no one Chester could hide behind.

“Your team had no business interfering with the operations of my data centres,” said Brad, poking his finger at Ramesh. “They don’t belong in my data centre, and their presence there put critical business production operations in peril. Unlike some teams, we are essential to the day-to-day operations of the business. Without us, the company cannot conduct business. Without us, every person here would be out of a job.”

Ramesh scowled. His eyes narrowed as he pursed his lips before speaking. Brad was organizationally his peer, in that they both reported to Jessica, but Brad was actually several levels higher from an HR perspective, and he treated Ramesh accordingly.

“Brad, will you just relax for a minute,” said Ramesh. “Have a seat and let’s talk about this civilly. We need to get beyond emotion and into facts.”

“Facts! You want facts? Chris caused the damn outage! Now I’ve got to get on bended knee and beg Jessica’s forgiveness, because I let someone into my data centre that shouldn’t have been there. And I’ll probably have to do the same for the CEO after that. If I don’t get fired first.”

Brad jerked a chair away from its place in front of Ramesh’s desk and into the centre of the room. He didn’t sit in it, but rather used it as a prop, slapping it and shaking it when making his point.

“Do you realize that Chris’ stupidity took down key systems just as the new marketing program, Mountain Top, was about to be rolled out? Do you know how many millions of dollars had been invested in this … how many people had worked countless hours to make it happen? We’re on a tight schedule to meet the requirements the business lays down.”

Brad pointed a finger directly at Chester, who seemed to squirm under the attention. Brad was so upset, I could see his finger shaking from across the room. “Isn’t that right, Chester? Just look at the volume of changes we get from change management. We can’t slow down without impacting the rollout of all these new business programs.”

“That’s right,” said Chester. “We have more change requests than ever.”

Brad slapped the side of the chair and shook a finger straight at me. “And you screwed the whole thing up and nearly got yourself hurt in the process. It’s a good thing you don’t work for me, or you’d wish that rack had knocked your head in. And if I have my way, you’ll be out of a job by this afternoon. We cannot afford to have bumbling meddlers like you in this company.”

Ramesh glared at me and I knew that was the sign for me to sit quietly and just let Brad vent, but I wasn’t going to let Brad’s accusations stand. I hadn’t done anything wrong. In fact, I’d done the right thing, and I wasn’t going to let Brad intimidate me into being punished for doing the right thing.

I looked directly at Brad and calmly said, ”I didn’t cause the outage, Brad. There is plenty of blame to go around. It was a perfect storm of mistakes. Your team, the development team, and change management, each played a role in causing the incident. And the same chronic behaviors will continue to cause more outages unless something is done about them, and the process used for making changes in the data centre.”

For the first time since he’d walked in, Brad was totally silent. His eyes got wide and his mouth opened silently. He wasn’t used to underlings pushing back on him when he was ranting. After an awkward moment, he mumbled incredulously, ”What did you say?”

“I agree that I shouldn’t have been able to get into the data centre last night. But then again, neither should the development team. And the fact that changes can be made without going through change management, only makes things worse. Your team, Brad, is the only team that should be in the data centre itself, and then, only when there is an approved change.”

“It seems to me that you were the one we had to rescue from having a server rack nearly crush them on the floor,” said Brad with a grin. “I think you would at least be a little grateful.”

“I was in the data centre last night. Sure, that’s right. But so were members of the development team. And the change management process isn’t mature enough to ensure neither the development team, or I, was not there. Not to mention that your operations team should have thrown all of us out as soon as they spotted us. Cowboy installations that work outside the change management process should not be the accepted norm for your team.”

“But that’s not key here. It’s not about pointing fingers,” I said. “It’s about figuring out how it happened, and preventing it from happening again. It’s about root cause and remediation. It’s about applying some problem management best practices.”

Brad rolled his eyes and slowly shook his head, as he looked to the ceiling. “Please! Don’t give me that ITIL crap again. I had to sit through more of your presentations on that than I cared to, and all of them were a waste of time.”

“But Brad,” added Ramesh. “Jessica has made a commitment to applying ITIL best practices where possible, to improve delivery of our services to the business. You’ve seen the data on how it helped cut back on our incidents. Remember … ?”

“Both of you have been swallowing too much of that consultant bull,” snapped Brad. “I’ve been working in, and running data centres, for over 20 years. And I’ve done a damn good job at it. Hell, I’ve had this job for the last six years and outlived three CIOs so far. So I know I must be doing a good job, or they would have fired me by now.”

“No one is questioning your experience, or your capabilities, Brad,” said Ramesh, trying to defuse the emotion and get to the facts.

“They’d better not. Because I’ll tell you, all this ITIL stuff is nothing new. I’ve taken the class. I know how it works. But guess what people, and you’ve probably not been around long enough to realize this. I’ve seen it all before … different names, different schemas … but never was there anything in there that any worthwhile data centre manager hasn’t seen, or used before. ITIL just changed the name of everything, but there was nothing new in there that I didn’t already know. I don’t need it, and my data centres don’t need it, no matter how much these consultants come in here and get leadership all in a panic. I run a tight ship, and the only time something goes wrong is when interlopers like Chris stumble around inside and muck things up.”

Brad threw himself down in the chair. He seemed to have burned through some of his rage. His voice was calmer now. “What do I have to do to get some coffee around here? I always offer my guests coffee, no matter why they come to see me. It’s not like you have to pay for it yourself, Ramesh.”

Ramesh stepped outside for a moment and when he returned said, “Coffee is on its way.”

Brad nodded and turned to me. “Why did you go into my data centre and start messing with environment? Are you malicious or just plain ignorant?”

“Because I saw people racking and stacking equipment in the production area.”

Brad rolled his eyes. “You do realize that is what my people do, don’t you? It’s a data centre. We do adds, moves and changes in the facility. That’s our job.”

I shook my head. “These weren’t your people. These were developers. They were doing this during a freeze period, and without approval from the change board.”

I turned to Chester. “That’s right, isn’t it? They had no approved change request in the system.”

Chester nodded slightly, and I could tell he was looking for a way to soft-pedal the situation. “Perhaps it was an emergency, and they were planning to complete the emergency change request documentation today, after the change. We have to assume positive intent. I’m sure they would never do anything to disrupt the company’s objectives.”

“No,” I said. “It is okay to assume positive intent on the part of requestors, but that doesn’t mean we can let them do whatever they want … and it doesn’t mean we can let people make changes without going through the change management process. That’s what the change advisory board is for. That’s why we use leaders to be on that board; because they have a broader, and more integrated perspective, than the average technician. They are much better suited to know what will impact our business and what will not.”

There was a knock at the door, and an administrative assistant wheeled in a cart with coffee on it. Brad took a cup, and placing it to his lips, took a huge gulp. “Jeez, Ramesh. You call that coffee? That’s so weak, it makes green tea seem like a robust invigorating drink. If you want to know how to make real coffee, come down to my office and I’ll show you how we make it the data centre way.”

Before Ramesh could respond, the phone in the office rang. “This is Ramesh. Yes … yes … Brad and Chris and Chester. Okay, I’ll put you on speakerphone.”

Ramesh punched up the speakerphone. Jessica’s voice came through loud and clear. And she was ticked.

“I just got done having several new orifices chewed into my body by both Jason and Sully, our CEO … my boss. They’ve just educated me that, once again, IT broke something critical to the business, and did so at one of the worst possible times; that what should have been a simple change to the Mountain Top program, turned into a giant cluster. Not only didn’t it get done, but somehow, we apparently broke some portions of the program that were working just fine.”

Brad was the first to speak, “Jessica, this is Brad. It was because Chris … ”

Jessica cut him off. “Brad, I really don’t care about the specifics. All of you are at the centre of this, and if you cannot work together to make this kind of problem go away, then I will have my admin order a silver platter to hold all the heads I’m going to deliver to the business. I don’t want to see any fingers pointed among any of you. I don’t care what you do, or how you do it, but this nonsense has to stop. Am I clear?”

Everyone in the room nodded silently. “Everyone here is aligned with you, Jessica,” said Ramesh.

“Well I guess the message is clear,” said Brad. “Don’t do anything that may impact, or interfere, with the business needs. I know my team works that way, the question is; can the rest of the people here?”

I wasn’t going to let Brad not change anything his team did. If they couldn’t police their own space, there was no way we could prevent this type of issue going forward. “Will you at least tell your team to not let anyone into the data centre; much less do any work there. And that means anyone who is not part of your team?” I asked.

“So you want me to keep you out of the data centre? Is that it?” asked Brad with a grin, knowing that Jessica could not see him.

“Yes,” I said. “And the development team, and the platform team, and any other team besides yours.”

“Look, if I do that, then there is no way I can keep up with all of the requests coming through from the business,” said Brad. “I just don’t have the resources. The only way IT can keep meeting the business time lines is, if we use members of the development and platform teams to do some of the add/move/changes.”

“But they are key contributors to the change related incidents. Aren’t they Chester?”

“Well, there is only preliminary data,” said Chester.

He was lying. I had worked with him over the last two weeks going through all of the change related incidents. The data was conclusive. Some of the teams were the source of a disproportionate number of change related incidents, and many of those involved business requests.

“But remember, Chester. How we were able to identify the teams involved in a high proportion of the incidents?”

“Yes, but I would like the chance to revisit the information. I don’t want to point fingers at the wrong people. That would be very demotivating for our teams, and to tell the business that we can’t meet their expectations for delivery would go against IT’s basic charter as an organization.”

“That’s right, Chester,” said Brad. “We all seem to easily forget that our whole reason for being here is to support the business, and meet their expectations when it comes to delivering new and current services. If we can’t do that, then frankly they’d be better off outsourcing us.”

“But we don’t do them any favors if we simply agree to everything they ask for, even if we know it is not possible in a safe way,” I said. “That may make them happy for the moment, but when we don’t deliver, IT gets the reputation of being incompetent, unable to plan, and unable to deliver. The thing the business wants more than anything else is clarity and predictability. They may not like the answers we give them, but if we are honest and consistent, they may go back to trusting us as partners, instead of trying to out game us as suppliers, like they currently do. That honesty and transparency is what differentiates us from an outsourced situation. If we can’t partner with them in the true sense of the word, then we deserve to be outsourced, because they can’t do any worse.”

The room went silent for a moment. I hadn’t really planned those words. They had just popped out. I felt good about them. They were perhaps not the most politic thing to say after Jessica’s warning, but they needed to be said.

Jessica spoke first. “I think Chris has a point. Brad, I want you to lock down the data centre. Just so I am clear. You are responsible for whatever goes on there, whether your team does it or not. If there is a problem in the data centre, yours will be the first head on that platter. Do you understand?”

“Yes, ma’am,” said Brad, as he began sending text messages to his team. “Consider it done. But in exchange, you’ve got to get change management to vet these things, not just for their doability, but also for our capability and capacity to make it happen on the schedule the business wants. Because I can tell you right now, we will be doing some serious pushback on the business requests.” He paused, and took a long look at Chester. “As long as change management stands up to all the pressure and escalations from the business as well as the other IT teams, I’ll execute only what they approve.”

Jessica said, “Chester will need to step up and hold the line. That’s his role and he needs to execute.”

Chester looked like he wanted to disappear, until I said, “I’ll back him up in the meetings, and with the information he needs to ensure we keep changes under control.”

Ramesh looked at me and asked, “So you’re taking accountability for managing the push-back on change requests? You and Chester are going to manage all of that?”

I nodded, half wondering what I had just impulsively gotten myself into. “If Chester wants the help, I’ll partner with him.”

Chester quickly nodded yes.

“Since I don’t hear a ‘no’ from Chester, I’m assuming he’s okay with it,” said Jessica. “Okay Chris, I admire your initiative. Just remember that if anything gets through that should not, or people go around the process … well, let’s just say that I’ll need sufficient room on that silver platter for two heads. Am I clear, Chris?”

I nodded, and sat their silent for a moment, until I remembered Jessica could not see me. “Yes,” I said, as I wondered if I had gone too far this time.