It’s 11:30 p.m. on a Friday, when John, the IT operations manager, hears his phone ring. The ringtone is distinct, one that John has programmed so that he can instantly recognize a call from the office. He answers the phone, and on the other end is Valentina, one of the senior software developers at John’s office. There’s a problem in the production environment.
The last software release included additional functionality that changed how the application interacted with the database. But because of a lack of adequate hardware in the testing environments, the entire application couldn’t be tested prior to release. Around 10:30 this evening, a scheduled task that runs only quarterly began executing. The job was missed during the testing phase, and even if it wasn’t, there isn’t enough data in the test environment to create an accurate test. Valentina needs to stop the process, but she doesn’t have access to the production servers. She’s spent the last 45 minutes searching through the company intranet site to find John’s contact information. John is the only person Valentina knows who has the production access she needs.
Killing the scheduled task isn’t straightforward. The task usually runs overnight and wasn’t designed to be stopped midway through processing. Because Valentina doesn’t have production access, her only alternative is to dictate a series of cryptic commands to John over the phone. After a few missteps, John and Valentina finally manage to stop the task. The two plan to regroup on Monday to figure out what went wrong and how to fix it for the next quarter. Now both John and Valentina must stay on guard over the weekend in case the behavior repeats itself with another job.
Chances are this story feels familiar to you. Having production code that hasn’t been properly tested feels like a scenario that could have been avoided, especially when it interrupts a team member on their off-time. Why is the testing environment insufficient for the needs of the development group? Why wasn’t the scheduled task written in such a way to make stopping and restarting it straightforward? What’s the value of the interaction between John and Valentina if John is just going to blindly type what Valentina dictates? Not to mention the two probably skipped the organization’s change approval process. Nothing raises the safety of a change like five people approving something they don’t understand!
The questions raised here have become so commonplace that many organizations don’t even think to examine them in detail. The dysfunction detailed is often accepted as inescapable, due to the difference in roles between development and IT operations teams. Instead of addressing the core issues, organizations continue to heap more approvals, more processes, and tighter restrictions onto the problem. Leadership thinks that they’re trading agility for safety, but in reality, they’re getting neither. (When was the last time you said, “Thank goodness for change control”?) These negative and sometimes wasteful interactions between teams and processes is exactly what DevOps is attempting to solve.
1.1 What is DevOps?
These days, “What is DevOps?” feels like a question you should ask a philosopher more than an engineer. I’ll give you the story and the history of DevOps before presenting my definition. If you ever want to start a fight at a conference, though, you can ask the “What is DevOps?” question to a group of five people, and then walk away and watch the carnage. Luckily, you’re reading this and not talking to me in the hallway, so I don’t mind putting my definition out there and seeing what happens. But first, the story.
1.1.1 A little DevOps history
In 2007, a systems administrator by the name of Patrick Debois was consulting on a large data center migration project for the Belgium government. He was in charge of the testing for this migration, so he spent a fair amount of time working and coordinating with both the development and operations teams. Seeing the stark contrast between how development and operations teams functioned, Debois got frustrated and started thinking of solutions to this problem.
Fast-forward to 2008. Developer Andrew Clay Shafer, attending the Agile Conference in Toronto, proposes an ad hoc discussion session called “Agile Infrastructure.” He received such poor feedback on his proposal that he didn’t even attend the session himself. In fact, only a single attendee joined the session, Patrick Debois. But because Debois was so passionate about discussing this topic, he tracked Shafer down in the hallway, where they had an extensive discussion about their ideas and goals. Directly out of those conversations, they formed the Agile Systems Administrator Group.
In June 2009, Debois was back in Belgium, watching a live stream of the O’Reilly Velocity 09 conference. At this conference, two employees from Flickr, John Allspaw and Paul Hammond, gave a talk titled “10 Deploys per Day: Dev & Ops Cooperation at Flickr.” Debois, moved by the talk, was inspired to start his own conference in Ghent, Belgium. He invited developers and operations professionals to discuss various approaches to working together, managing infrastructure, and rethinking the way the teams worked together. Debois called this two-day conference DevOps Days. A lot of the conversations about the conference were happening on Twitter, which then limited the number of characters per message to 140. To save as many precious characters as possible, Debois shortened the conference’s Twitter hashtag from #devopsdays to just plain #devops, and with that, DevOps was born.
Definition DevOps is a set of software-development practices that combines a software development mentality with other functions in the organization. DevOps puts a heavy emphasis on shared responsibilities across all teams throughout the software development life cycle. The edges of job functions soften, as operations team members take on tasks that were traditionally more developer-focused, and development team members do the same. The term DevOps is most commonly associated with development (Dev) and IT operations (Ops), but the approach can be extended to other groups as well, including but not limited to security (DevSecOps), QA, database operations, and networking.
It’s been more than 10 years since that fateful meeting. Since then, DevOps has moved beyond small web startups and has begun to penetrate larger enterprises. The success of DevOps, however, has brought the most cantankerous enemy of any movement: market forces.
According to LinkedIn Talent Solutions, in 2018 the most recruited job overall, not just in tech, was DevOps engineer. Considering we’ve defined DevOps as a set of practices, it’s strange how a style of work quickly became a job title. You’ve never heard of an Agile engineer, because it just sounds silly. As transformational as DevOps is, it couldn’t escape market forces. With that much demand, the job title of DevOps has led to scores of candidates rebranding themselves as DevOps engineers.
Product marketers are looking to cash in on the DevOps craze. Simple products like metrics and monitoring get rebranded into “DevOps dashboards,” further diluting the meaning of the word. With the market pulling the term “DevOps” in different directions, it has splintered into different meanings for different people. I could spend an entire chapter arguing about what DevOps should and shouldn’t mean; instead, I’ll use the definition that I proposed previously. But if you ever see me at a conference and want to see me go on a tirade, ask me what it’s like being a “DevOps manager.”
1.1.2 What DevOps is not
Ironically it might be easier to define what DevOps is not rather than what it is. Thanks to market forces, these details will probably fall on deaf ears, but since this is my book, I figure I might as well go for it! For starters, it’s not about tools. If you purchased this book hoping to learn about Jenkins or Docker or Kubernetes or AWS, you’re going to be sorely disappointed. I don’t do refunds, but you can feel free to scream into the ether with your disdain.
DevOps isn’t about tools, but about how teams work together. Technology is definitely involved, but, honestly, the tools are less important than the people. You can install the latest version of Jenkins or sign up for CircleCI, but if you don’t have a solid test suite, it’s useless. If you don’t have a culture that considers automated testing valuable, the tool doesn’t provide value. DevOps is about people first, then process, then tools.
You need the people on-board and ready for change. Once the people are on-board, they need to be involved and engaged with creating the process. Once a process is created, you now have the necessary input to pick the right tool!
So many people focus on the tool first and try to work backward from there. This is probably one of the top DevOps follies. You can’t choose a tool and then tell the people that they have to change all their processes. Our brains are wired to immediately be hostile to that type of approach. When tools are launched like that, the tool feels like it’s happening to them, not through them. That approach differs significantly from the way people accept new ideas. You have to have buy-in.
In addition, when you get excited about a new tool, you begin applying it to problems you never had. When you buy a new table saw, suddenly everything in your home becomes a construction project. It’s the same thing with software tools.
All this is to say that the major focus of this book and DevOps is about people and their interactions. While I may reference specific tools here and there, the book avoids giving specific examples based on architecture. Instead, the examples focus on capabilities, regardless of which tool provides that capability. To highlight this approach, the DevOps philosophy is structured on top of the CAMS model, which aims to place people first when addressing problems.
1.2 CAMS, the pillars of DevOps
DevOps is structured around four pillars of attention and focus. Those pillars are culture, automation, metrics, and sharing: CAMS, as it’s called for short. As illustrated in figure 1.1, these pillars for DevOps are crucial to holding up the entire structure.
Figure 1.1 Culture, automation, metrics, and sharing are all necessary for a successful DevOps transformation.
The details of the four pillars are as follows:
-
Culture is about changing the norms by which your teams operate. These norms might be new communication patterns between teams or completely new team structures. Cultural changes are dictated by the type of cultural problems you have. I outline specific examples in this book, but you’ll also learn how to identify these problem areas yourself so that you can apply the problems beyond the examples highlighted here. Don’t underestimate the value and impact of a company’s culture on its technology outcomes. As you’ll find in this book, most problems are people problems, not technology problems.
-
Automation is not just about writing shell scripts. (I mean, that’s definitely part of it, but stick with me.) Automation is about freeing human capital from the mundane. It’s about empowering people to do their jobs safely and autonomously. Automation should be used as a cultural re-enforcer of the way work gets done within your organization. Simply saying, “Automated testing is a cultural value” is one thing, but embedding that cultural value into your processes through automated checks and merge requirements enforces that cultural norm. When implemented properly, it sets a new standard for how work is completed.
-
Metrics are the way you tell whether or not something is working. The absence of errors is not sufficient. Metrics should also be used as a cultural re-enforcer for the way we evaluate our systems. It’s not enough for order processing to not produce errors; we should be able to show successful orders flowing through the system as well.
-
Sharing is based on the idea that knowledge wants to be free! Humans often learn best when they’re teaching something to someone else. Sharing is about creating that--ready for it--cultural re-enforcer! Knowledge management is incredibly important in a world where we continue to build more and more complex systems.
While my focus on CAMS varies throughout the book, understand that these four pillars are the underpinning for everything in a DevOps transformation. If you think back to these four pillars, you can solve a lot of problems inside your organization.
1.3 Another DevOps book?
You’re probably looking at this book and asking yourself, “Do I need another book that’s going to tell me my company culture is toxic?” You probably don’t. In all my years in this industry, I’ve never “enlightened” someone by saying that their company has a bad culture. Instead, most employees fall short in the understanding of why their company has a bad culture.
It’s true that culture often comes from the top of the organizational chart, but it is equally true that various cultures exist within the same organization--some good, some bad. I can’t promise that I can help you transform your entire organization with a few sips from the DevOps Kool-Aid, but I can promise you that I can help you transform your corner of the organization.
My motivation for writing this was driven by the number of people I meet who feel that because their manager isn’t on board, they can’t do DevOps. To some extent, that’s true. For some areas, you need senior leadership buy-in. But there’s still a lot of change that an individual contributor or a team lead can push through to make your lives and processes better. A little bit of investment will also free up some of your time that’s currently spent on wasteful, low-value tasks and unleash that freed time on more productive work. This book takes a lot of the lessons that I’ve learned and distills them into a specific set of actions that you can take to bring the DevOps transformation out of the C-suite and down into the trenches.
Lastly, this book will be a little more prescriptive about the changes that are needed in your corner of the organization. Together we’ll dive deeper than just case studies about how things are handled at Company X. Instead, we will come up with a specific set of actions that you will implement, tweak, and iterate on within your organization. Notice that I said “you.” This isn’t a book that will paint you as a hapless victim of poor management. You have the power to make your situation better. You will be the change agent that your organization needs to start this DevOps journey. All you need is a plan. Let’s get started on that.
Summary
-
DevOps is more than just a new set of tools. DevOps is truly about redefining the way you look at work and the relationship between tasks across different teams. The changes that a DevOps transformation brings will go beyond pure technology and stretch into the fabric of how we look at the nature of our work.
-
To introduce change, you need a way to examine the problems within your organization and address these productivity killers. That’s the meat of this book.
-
Despite the need for culture, I recognize that most of you reading this will be engineers, biased toward action. I open up the first part of the book with examples of problems your organization is most likely experiencing and a few concrete approaches to solving them. Even in these scenarios, I highlight how company culture influences the current state of affairs. Let’s start with the reasons Valentina has to call John in the middle of the night.