DevOps Explained with an Example

Let us say that you are a project manager for an Internet banking product. The past weekend you deployed a change to update a critical component of the system after weeks of development and testing. The change was deployed successfully; however, during the postimplementation review, it threw out an error that forced you to roll back the change.

The rollback was successful, and all the artifacts pertaining to the release were brought to the table to examine and identify the root cause the following Monday. Now what happens? The root cause is identified, a developer is pressed into action to fix the bug, and the code goes through the scrutiny of various tests, including the tests that were not originally done that could have caught the bug in the functional testing stage rather than in production. All tests run OK; a new change is planned; it gets approved by the change advisory board; and the change gets implemented, tested, and is given all green lights.

These are the typical process activities that are undertaken when a deployment fails and has to be replanned. However, the moment things go south, what is the first thing that comes to your mind as the project manager? Is it what objective action you should take next, or do you start thinking of the developer who worked in this area, the person responsible for the bug in the first place? Or do you think about the tester who identified the scenarios, wrote the scripts, and performed exploratory testing? Yes, it is true that you start to think about the people responsible for the mess. Why? It is because of our culture. We live in a culture that blames people and tries to pass the buck.

I mentioned earlier about some respondents telling me that DevOps is about culture. So, what culture am I talking about within the context of this example? The example depicts a blameful culture where the project manager is trying to pin the blame on the people within his team directly responsible for the failure. He could be factually right in pinning the blame on people directly responsible, but I am focusing on the practice of blaming individuals.

How is this practice different from a DevOps culture? In DevOps, the responsibility of completing a task is not considered as an individual responsibility but rather a shared responsibility. Although an individual works on a task, if the person fails or succeeds, the entire team gets the carrot or the stick. Individuals are not made responsible when we look at the overall DevOps scheme of things, and we do not blame individuals. We follow a blameless culture. This culture of blamelessness arises from the fact that we all make mistakes because we are humans after all and far from being perfect. We make mistakes. So, what is the point in blaming people? In fact, we expect that people do make mistakes—not based on negligence but from the experimentation mindset.

This acceptance (of developers making mistakes) has led us to develop a system where the mistakes that are made get identified and rectified in the developmental stages, well before they reach production.

How is this system (to catch mistakes) built? To make it happen, we brought the development and operations teams together (to avoid disconnect); we developed processes that are far more effective and efficient than what was out there; and finally we took umbrage under automation to efficiently provide us with feedback on how we are doing (as speed is one of the main objectives we intend to achieve).

DevOps is a common phrase, and with its spread reaching far and wide, there are multiple definitions coming from various quarters. No two definitions will be alike, but they will have a common theme: culture. So, for me, DevOps is a cultural transformation that brings together people from across disciplines to work under a single umbrella to collaborate and work as one unit with an open mind and to remove inefficiencies.

Why DevOps

You might ask what gave rise to a new culture called DevOps. The answer is evolution. If you take a timeline view of software, from the 1960s up to the advent of the Internet in the 1990s, developing software was equivalent to a building project or launching a space shuttle. It required meticulous planning and activities that were planned to be executed sequentially. The waterfall project management methodology was thus born with five sequential steps, as indicated in Figure 2-1.

When the Internet boomed, the software was far more accessible than the earlier era, and this generated demand not seen earlier. When the software industry started to expand, the waterfall model’s limitations were exposed. The need to complete a detailed planning exercise and the sequential practice of flow did seem like an impediment to the advancement of the software industry.

Then in 2001 at a ski resort in Utah, the Agile Manifesto was born. Several prevalent Agile methodologies came together to form a common goal that would remove the cast-in-stone waterfall sequential activities.

Agile was more fluid because it did not conceive of all the requirements at the beginning and try to solve everything overnight. It was an approach that was based on iterations, where all the activities of project management just cycled repeatedly.

In between, if a requirement were to change, that’s OK because there were provisions to make changes that were neither bureaucratic nor tedious in nature. In fact, the Agile methodology puts the emphasis on the response to changes in requirements rather than a map to be followed.

The flexibility and dynamism that came about through Agile spread its wings across the software industry. Several software projects migrated to the Agile way of working, and to this day there are projects that are undergoing serious coaching during this transformational phase.

The Agile methodology is simple, where you keep things small enough to manage and large enough to be rendered meaningful. The time frames that defined iterations in Agile did not carry too much wriggle room. From an efficiency perspective, Agile was far better than the waterfall model. However, the demands from the market were out of sync with what Agile could provide. While the market shouted out for faster deliveries, the need to increase quality (reducing defect rate) was perennially being pursued. The Agile project management methodology needed something, like an elixir to run things faster. It needed automation. Enter DevOps!

Automation by itself is like giving a drone to a kid without really teaching him the process to make it fly. Generally speaking, technology by itself has no meaning if there are no underlying functional architecture, process, and embedded principles. DevOps, therefore, is not just automation but a whole lot more. You will find out the nitty-gritty details in the coming sections.

A Note on DevOps Scope

The word DevOps gives away the scope through its conjunction of two parts of a software life cycle. While Agile existed mainly to put an end to the rigidity brought forth by the waterfall model, it was said that the methodology can be used for operations as well. But, without an overarching process or a framework, using Agile for operations with the same rigor was not going to work. DevOps bridged this gap by bringing in the operational phases along with the developmental activities under a single umbrella, and applying common processes and principles to be employed across the board.

DevOps comes into play when you get started with the software development process, which is the requirement gathering phase. It ends when the software retires from service. DevOps spans the entire wavelength of a software life cycle, and if you read between the lines, you cannot just implement and execute DevOps until deployment and be done with it. It will continue to function until the software is used by its designated users. In other words, DevOps is here to stay, and stay for as long as services are delivered. So, in practice, the operational phase runs perpetually, and DevOps will deliver the required optimization and automation. The processes to run operations will be borrowed from the ITIL service management framework.

Benefits of Transforming into DevOps

Several software companies have been delivering applications for a number of years now. Why do we need DevOps to tell us how we must develop now?

Our services are being delivered to several customers including top banks and mines around the globe. I am running simply fine with my service management framework. Why DevOps?

People have lived for thousands of years now. They did just fine, reproducing and surviving. What has changed in the past 100 years? We have changed the modes of transport for better efficiency; we communicate faster today; and overall, our quality of life has gone up several notches. The fact that something is working is not a barrier for improvements to be brought about and to transform. DevOps introduces several enhancements in the areas of working culture, process, technology, and organization structure. The transformation has been rooted in practices that were developed by some like-minded organizations that were willing to experiment, and the results have vastly gone in favor of DevOps over other ancient methodologies that still exist today.

Amazon, Netflix, Etsy, and Facebook are some of the organizations that have taken their software deliveries to a whole new level, and they don’t compete anymore with the laggards. They have set new benchmarks that are impossible to meet with any of the other methodologies.

At the 2011 Velocity conference, Amazon’s director of platform analysis, Jon Jenkins, provided a brief insight into Amazon’s ways of working. He supported it with the following statistics.

During weekdays, Amazon can deploy every 11.6 seconds on average. Most organizations struggle to deploy weekly consistently, but Amazon does more than 1,000 deployments every hour (1,079 deployments to be precise). Further, 10,000 hosts receive deployments simultaneously on average, and the highest Amazon has been able to achieve is 30,000 hosts simultaneously receiving deployments. Wow! These numbers are really out of this world. And these are the statistics from May 2011. Imagine what they can do today!

Culture

There is a popular urban legend that the late Peter Drucker, known as the founder of modern management, famously said, “Culture eats strategy for breakfast.” If you want to make a massive mind-boggling earth-shaking change, start by changing the culture that can make it happen and adapt to the proposed new way of working. Culture is something that cannot be changed by a swift switching process. It is embedded into human behavior and requires an overhaul of people’s behavior.

Automation

Automation is a key component in the DevOps methodology. It is a massive enabler for faster delivery and also crucial for providing rapid feedback. Under the culture principle, I talked about a safety net with respect to experimentation. This safety net is made possible through automation.

The objective is to automate whatever possible in the software delivery life cycle. The kinds of activities that can be efficiently automated are those that are repetitive and those that don’t ask for human intelligence. For example, building infrastructure was a major task that involved hardware architects and administrators; and most importantly, building servers took a significant amount of time. This was time that was added to the overall software delivery. Thanks to the advancement of technology, we have cloud infrastructure today, and servers can be spun up through code. Additionally, we don’t need hardware administrators to do it. Developers can do it themselves. Wait, there’s more! Once the environment provisioning script is written, it can be used to automate spinning up servers as many times as necessary. Automation has really changed the way we see infrastructure.

Activities involving executing tasks such as running a build or running a test script can be automated. But, the activities that involve human cognizance are hard to automate today. The art of writing the code or test scripts requires the use of human intelligence, and the machines of today are not in a position to do it. Tomorrow, artificial intelligence can be a threat to the activities that are dependent on humans today .

Lean

DevOps has borrowed heavily from Lean methodology and the Toyota Production System (TPS). The thinking behind the Lean methodology is to keep things simple and not to overcomplicate them. It is natural that the advent of automation can decrease the complexity of architecture and simplify complicated workflows. The Lean principle aids in keeping us on the ground so we can continue working with things that are easy to comprehend and simple to work with.

There are two parts to the Lean principle. The primary one is not to bloat the logic or the way we do things; keep it straightforward and minimal. An example is the use of microservices, which support the cause by not overcomplicating the architecture. We are no longer looking to build monolithic architectures that are cumbersome when it comes to enhancements, maintenance, and upgrades. A microservice architecture solves all the problems that we faced yesterday with monolithic architectures; it is easy to upgrade, troubleshoot (maintain), and enhance.

The second part of the principle is to reduce the wastage arising from the methodology. Defects are one of the key wastes. Defects are a nuisance. They delay the overall delivery, and the amount of effort that goes into fixing them is just a sheer waste of time and money. The next type of waste focuses on convoluted processes. If something can be done by passing the ball from A to B, why does it have to bounce off C? There are many such wastes that can be addressed to make the software delivery more efficient and effective.

Measurement

If you seek to automate everything, then you probably need a system to provide feedback whenever something goes wrong. Feedback is possible if you know what the optimum results are and what aren’t. The only way you can find out whether the outcome is optimum or not is by measuring it. So, it is essential that you measure everything if you are going to automate everything!

Measurement principle provides direction on the measures to implement and keep tabs on the pulse of the overall software delivery. It is not a simple task to measure everything. Many times we do not even know what we should measure.

Even if we do it, the how part can be an obstacle. A good DevOps process architect can help solve this problem. For example, if you are running static analysis on your code, the extent of passable code must be predetermined. It is not a random number, but a scientific reasoning must be behind it. Several companies allow a unit test to pass even if it parses 90 percent of the code. We know that ideally it must be 100 percent, so why should anybody compromise for 90 percent? That is the kind of logic that must go behind measuring everything and enabling fast feedback, to be realistic about the kind of feedback that you want to receive.

In operations, monitoring applications, infrastructure, performance, and other parameters come under this principle. Measurements in monitoring will imply when an event will be categorized as a warning or an exception. With automation in place, it is extremely important that all the critical activities, and the infrastructure that supports them, be monitored and optimized for measurement.

There are other measurements as well that are attached to contracts and SLAs and are used for reporting on a regular basis. These measurements are important as well in the overall scheme of things.

Sharing

The final principle is sharing, which hinges on the need for collaboration and knowledge sharing between people. If we aim to significantly hasten the process of software delivery, it is only possible if people don’t work in silos anymore. The knowledge, experience, thoughts, and ideas must be put out into the open for others to join in the process of making them better, enhanced, and profound.

One of the key takeaways of this principle is to put everyone who works on a product or a service onto a single team and promote knowledge sharing. This will lead to collaboration rather than competition and skepticism.

There are a number of collaboration tools on the market today that help support the cause. People do not even have to be colocated to share and collaborate. Tools such as Microsoft Teams and Slack help in getting the information across not only to a single person but to all those who matter (such as the entire team). With information being transparent, there will be no reason for others to worry or be skeptical about the dependencies or the outcome of the process.

People

The word DevOps is derived from the conjunction of two words, development, and operations. I have already familiarized you with what DevOps is all about: a change of culture in the way we deliver and operate software. People are at the heart of this cultural transformation, and they are one of the three critical elements that enable the DevOps culture.

The development and operation teams are amalgamated to bring about a change in culture. The thinking behind it is quite straightforward. Let’s say that an application is developed, and it comes to the change advisory board (CAB) for approval. One of the parties on the CAB is the operational teams. They specifically ask questions around the testing that has been performed for this software, and even though the answer from development is yes for the success rate of all the tests, the operational teams tend to be critical. They don’t want to be blockers, yet they find themselves in a position where they have to support software that they haven’t been familiarized with yet. The bugs and defects that come with the software will become their problem after the warranty period (usually between 1 and 3 months). Most importantly, they only have the confirmation of the developers to go with when the quality of the software is put on the line.

In the same scenario, imagine if the operational teams were already part of the same team as development. Being on the same team will give them an opportunity to become familiar with the development process and the quality controls put in place. Instead of asking questions in the CAB, they can work progressively with the development teams in ensuring that the software is maintainable, and all possible operational aspects are undertaken beforehand. This is one such case study that showcases the benefit of having a single team.

In Figure 2-4, you can visualize the development team on the edge of one cliff, while the operations team is on the opposite. In between the two cliffs lies an area of uncertainty where activities that fall between the two teams have a knack for being unpredictable and they are usually sparred over, generally over ownership. In other words, you want things to be either with the development team or with the operations team. There is no bridge between the teams, meaning there can be a lot of confusion, miscommunication, and mistrust between the two opposing teams.

Let’s play out the priorities for both teams. The development team still has a job because there is a need to develop new features. That is their core area, and that is what they must do to remain relevant. The operations team’s big-ticket goal is to keep the environment stable, in the most basic sense. They need to ensure that even if something was to go wrong, they are tasked to bring it back to normal, in other words, maintain the status quo. So, here we have a development team intending to create new features and an operations team looking to keep the environment stable.

Does it have to be rocket science to have evolved into a methodology called DevOps that promises to shake the industry from its roots? Well, the environment is going to remain stable if there are no changes introduced to it. As long as it stays stagnant, nothing ever will bother its stability, and the operations team would have been awarded for a stellar job. But we have the development team waiting in the wings to develop new features. New features that are developed will be deployed in the production environment, and there is every chance that the deployment of new features could impact stability. So, stability is something that can never be achieved as long as new features are introduced, and no software will remain stagnant without enhancements and expansion.

A decent way to tackle this conundrum between the development and operation teams is to put them together and create channels of communication within the team members. Both the development and operations teams have a shared responsibility to ensure that development, testing, deployment, and other support activities happen smoothly and without glitches. Every team member takes responsibility for all the activities being carried out, which translates to the development and operation teams jointly working on the solution that begins with coding and ends with deployment. The operations teams will have no reason to mistrust the test results and can confidently deploy onto the production environment.

Process

Processes are a key component in ensuring the success of any project. However, we often find that most DevOps implementations focus more on automation and technology and give a backseat to processes that are supposed to be the basis for automation. They say that backseat driving is dangerous, so placing processes in this position and hoping that the destination would be reached in record time with no mishaps is a gamble that plays with unpredictability. Therefore, it is important that processes are defined first along with a functional DevOps architecture and then translated into tooling and automation. The process must always drive tools and never the other way around.

With DevOps combining different disciplines under a single banner, the processes too need to be rejigged to fit the new objectives. This section covers the processes pertaining to the development area.

Waterfall project management methodology (such as PMI-backed Project Management and PRINCE for projects in controlled environments) are not favored in the IT field anymore. There are various reasons going against this methodology, mainly stemming from the rigidity it brings into the project management structure.

Most IT projects are run on Agile project management methodologies because of the flexibility it offers in this ever-changing market. According to PMI’s “Pulse of Profession” 2017 publication (www.pmi.org/-/media/pmi/documents/public/pdf/learning/thought-leadership/pulse/pulse-of-the-profession-2017.pdf), 71 percent of organizations have been leveraging Agile. Another study by PricewaterhouseCoopers, named “Agile Project Delivery Confidence (July 2017)” (www.pwc.com/gx/en/actuarial-insurance-services/assets/agile-project-delivery-confidence.pdf), reported that Agile projects are 28 percent more successful than their waterfall counterparts. This is huge considering that Agile is still new and emerging and the waterfall methodology has existed since the 1960s.

When we talk about Agile project management, there are a number of methodologies to pick from. Scrum, Kanban, Scrumban, Extreme Programming (XP), Dynamic Systems Development Method (DSDM), Crystal, and Feature Driven Development (FDD) are some examples. However, all the methodologies are aligned by a manifesto that was formulated in a ski resort in Utah in 2001. And there are a set of 12 Agile principles that provide guidance in setting up the project management processes .

Technology

Technology is the third element of DevOps and is often regarded as the most important. It is true in a sense that without automation, we cannot possibly achieve the fast results that I have shared earlier through some statistics. It is also true that technology on its own, without the proper synchrony of people (roles) and processes, is like a spaceship in the hands of kindergarteners. It is a must that the people and process sides of DevOps are sorted out first before heading this way.

The number of tools that claim to support DevOps activities is enormous—too many to count.

Continuous Integration

Several developers work together on the same piece of code, which is referred to as the mainline in software development lingo. When multiple developers are at work, conflicts arising due to changes performed on pieces of code and the employed logic are quite common. Software developers generally integrate their pieces of code into the mainline once a day.

When conflicts arise, they discuss and sort it out. This process here of integrating the code manually at a defined time slows down the development. Conflicts at times can have drastic results, with hundreds of lines of code having to be rewritten. Imagine the time and effort lost due to manual integration. If I can integrate code in almost real time with the rest of the developers, the potential amount of rework can significantly be reduced. This is the concept of continuous integration.

To be more specific, continuous integration is a process whereby developers integrate their code into the source code repository (mainline) on a regular basis, say multiple times a day. When the code is integrated with the mainline, any conflicts will come out into the open as soon as it is integrated. The resolution of conflicts does not have to be an affair where all developers sit across the codebase and break their heads. Only those who have conflicts need to sort them out manually. By doing this conflict resolution multiple times a day, the extent of conflicts is drastically minimized .

Integrating the code with the mainline is just the beginning. Whenever the code is integrated, the entire mainline is built, and other quality checks such as unit testing and code-quality checks (static and dynamic analysis) also are carried out.

Let us say a particular project has three developers, and each developer integrates their code three times a day. Daily, this equates to nine integrations every day. As per Figure 2-5, code that is integrated gets unit tested first, followed by software build and code quality checks. All this happens automatically whenever code gets integrated.

With nine integrations on a daily basis, we are staring at a possibility of having nine unit tests, nine builds on the entire mainline, and nine code quality checks.

Suppose one of the builds or unit tests or code quality checks fail. The flow gets interrupted, and the developer gets down to work to fix the defect at the earliest. This ensures that the flow of code does not get hampered and other coders can continue coding and integrate their work onto the mainline.

Continuous integration allows for fast delivery of software, and any roadblocks are avoided or identified as early as possible, thanks to rapid feedback and automation. The objective of continuous integration is to hasten the coding process and to generate a binary without integration bugs.

Continuous Delivery

The next item in the SDLC is to test the generated binary from various angles, aspects, and perspectives. Yes, I am referring to the tests such as system tests, integration tests, regression tests, user acceptance tests, performance tests, and security tests; this list is quite endless. When we are done with the agreed number of tests, the binary is deemed to be of quality and deployable into production. The qualified binary can be deployed into production with a click of a button. The qualification of any of the binaries as releasable into production is termed as continuous delivery (see Figure 2-6). It is generally seen as a natural extension of the continuous integration process.

Figure 2-6 depicts a continuous delivery pipeline. After every successful cycle of continuous integration, the binary is automatically subjected to an integration test. When the integration test is successful, the same binary is automatically system tested. The cycle passes on up to the preproduction environment as long as the tests (regression and user acceptance testing [UAT] in this illustration) are successful. When the same binary is successfully deployed in the preproduction environment (in this illustration) or any other environment that comes before the production environment, the binary becomes qualified to be deployed on to production. The deployment into the production environment is not done automatically but requires a trigger to make it happen. The entire cycle starting from the code push into the source code repository up to the manual deployment into the production environment is continuous delivery.

In the illustration, I have shown three developers integrating their code and three deployable binaries. Continuous delivery does not dictate that all three binaries have to be deployed into production. The release management process can decide to deploy only the latest binary every week. Remember that the latest binary will consist of the code changes performed by all the developers up until that point in time.

The sequence of automation for the activities beginning in the continuous integration process up until the production environment is referred to as a pipeline, or continuous delivery pipeline in this case .

Continuous Deployment

Continuous deployment is one step beyond continuous delivery. In continuous delivery, the deployment to production is based on a manual trigger. However, in the continuous deployment process, the deployment to production happens automatically, as depicted in Figure 2-7.

As soon as all the tests are successful, the binary is deployed to the preproduction environment. When the deployment to preproduction goes as planned, the same binary is deployed into production directly. In continuous delivery (Figure 2-7), the binaries were qualified as deployable, and the release manager was in a position to not deploy every single qualified binary into production. On the contrary, in continuous deployment, every single qualified binary gets deployed onto the production instance.

You might think that this is far too risky. How can you deploy something into production without any checks and balances and approvals from all stakeholders? Well, every single test that is performed and all the quality checks executed are the checks that are qualifying binaries as deployables . It is all happening in an automated fashion. You would do the same set of things otherwise but manually. Instead of deploying multiple times a day, you might deploy once a week. All the benefits that you derive from going early into the market are missing from the manual processes.

Let us say that one of the deployments were to fail. No problem! There is an automated rollback mechanism built into the system that rolls back the deployment within seconds. And it is important to note that the changes that are being discussed here are tiny changes. So, the chances of these binaries bringing down a system are remote.

Continuous Delivery vs. Continuous Deployment

The difference lies in the final sequence, where the deployment to production instance is automatic in continuous deployment and has a manual trigger for continuous delivery, as depicted in Figure 2-8.

Any organization on a journey of implementing DevOps will implement the continuous delivery process and upon gaining sufficient maturity will move toward the pinnacle of DevOps maturity: the continuous deployment process.

Organizations that feel a need to keep total control of their production environment through a formal structure of approvals and visibility tend to opt for continuous delivery. Banking and other financial segments fall into this category.

There are other organizations that have scaled the DevOps maturity ladder and are quite confident that the automatic deployment doesn’t cause significant impact to their production environment. Even if something was to fail, the rollback will be rapid too, even before anybody can notice it. Companies like Amazon, Netflix, and Google have been in this space for a while now. I shared a statistic earlier about Amazon managing a deployment every 11.6 seconds. How is it even possible? Look no further than continuous deployment.