Chapter 12. DevOps
DevOps is a set of principles and practices intended to help development and operations collaborate and communicate more effectively. DevOps is truly taking the industry by storm and, in some circles, reaching almost mythical proportions. I hear folks suggesting that DevOps can help solve almost any issue, which given the versatility of its cross-functional approach, is a view that has some merit, but some groups are losing sight of what this methodology is all about and how it can really help us implement the ALM. Unfortunately, some folks are also using DevOps to justify some very bad practices, such as allowing developers access to production machines, in some cases in violation of federal regulatory requirements for a segregation of duties. We absolutely believe that you can enjoy the great benefits of DevOps and still achieve your audit and regulatory requirements. In fact, we find that there are tangible benefits in adhering to regulatory and audit requirements if we approach them with a goal of creating right-sized processes and procedures that minimize risk and help ensure quality. DevOps should not be interpreted as all things to all people, but it does have a fairly broad focus and a great deal of potential to positively drive the entire application lifecycle. Make sure that you consider the maturity of your DevOps framework while also focusing on the primary goal of DevOps, which is to provide a fast and reliable deployment framework.
12.1 Goals of DevOps
The goal of DevOps is to ensure that we can reliably deploy code as often as evolving business needs require. DevOps has a strong focus on ensuring that there is a completely automated way to deploy changes, either in terms of systems configuration or application upgrades. DevOps also has a strong goal that development and operations work together collaboratively as a cross-functional and highly effective team. DevOps, by design, brings us both speed and reliability.
12.2 Why Is DevOps Important?
DevOps is important because it helps prevent mistakes leading to systems glitches and outages that have adversely affected many organizations, often in high-profile incidents that have tarnished the reputation and profitability of the firm. When DevOps is implemented successfully, organizations enjoy a significant competitive advantage, delivering business functionality and fixing defects in a rapid and secure way. DevOps enables agile development while improving both quality and productivity.
12.3 Where Do I Start?
We often find ourselves called in to help with implementing DevOps right after bad things have happened—usually, a failed release or serious cybersecurity incident that could and should have been prevented, or at least detected, by DevOps best practices. As always, we like to focus on assessing current practices, especially with regard to existing communication patterns, and incidents that can be attributed, in whole or in part, to less-than-effective communication and collaboration.
We always ask teams to explain exactly how they would verify that the correct code is running in production and to explain their procedures to detect unauthorized changes.
The call for DevOps sometimes comes after a serious application outage, often as a result of a failed systems upgrade or configuration change.
After we discuss what DevOps is really all about, we are usually asked to describe what is involved with implementing improved application build, package, and deployment.
The assessment results in a document that explains the current practices (“as-is”) and also specifies a target improved state (“to-be”), usually based upon industry standards (e.g., ISO, IEEE) and frameworks (e.g., ITIL, Cobit, CMMI). The best step is to implement the process improvements.
12.4 How Do I Implement DevOps?
Implementing DevOps must include both a strong focus on communication and a complete review of the systems and applications deployment technology, tools, and procedures. As always, we believe that you must first comprehensively assess your current practices and then compare them to industry best practices and afterwards create a reasonable plan to help your team move from where you are today to exactly where you need to be to meet your current and future business requirements. Implementing DevOps is both a culture and a technology shift. The new focus will be on automating everything that you do and ensuring that mistakes are minimized while speeding up the rate of change. DevOps moves from the team big-bang deployments to smaller, well-rehearsed rapid deployments that are repeatable and generally a lot faster.
To really understand DevOps, you need to appreciate the inherent conflict between developers and operations.
12.5 Developers and Operations Conflict
Developers and operations professionals have inherently different views and perspectives. Providing new features and innovations that meet the demands of the business is the goal of not only the application developers, but also just about everyone engaged in the software development process. The operations group has a very different view, which is focused on ensuring that systems are completely reliable and always available. Developers move us forward with rapid speed, often exhibiting what can seem like a reduced focus on caution with regard to avoiding costly defects and challenges. Developers see the system as working. Our operations colleagues are often in the role of telling us to slow down, pay attention to the road ahead, and above all avoid any possible risks that might cause a systems incident or outage. Ops is focused on reliability and security, whereas Dev wants to constantly update the application with new features and capabilities.
Although the perspectives of developers and operations often clash, both groups possess remarkable capabilities, and when they collaborate, organizations benefit greatly from the synergies of their talents and accomplishments.
12.6 Developers and Operations Collaboration
As DevOps evangelists, we help developers and operations professionals share their skills, expertise, and unique approaches, which yields amazing results. We find that developers are excellent at learning new technologies and building prototypes, but must rely upon their operations counterparts to ensure that there are repeatable processes and that the systems themselves will be supportable in production usage. Developers typically spend many months learning new technologies and then make critical decisions on their technical approach. They write the code and understand the intimate details of how the system was constructed. In contrast, operations understands how the application will behave in production under real work conditions. Our focus is on promoting collaboration, resulting in a more holistic understanding of the system, which ultimately benefits development, operations, and the business and end users.
The analogy of mountain climbing can be useful in understanding the DevOps perspective.
Operations engineers are often excited to learn from developers, especially when it improves their technical skills. We sometimes see developers being surprised by how much their operations colleagues actually know and can contribute to the collaboration. Bob learned this lesson regarding wisdom from Ops while still in college at his part-time job doing research into blindness and disability.
We have seen many other circumstances where development and operations were highly effective once they were brought to the table and agreed to collaborate.
Many organizations find themselves in a very demanding business environment with competitive pressures and other challenges.
12.7 Need for Rapid Change
DevOps is driven by an insatiable thirst for constant change. Years ago, businesses were often satisfied with changes to systems every few months, but expectations today are much higher. Organizations that cannot adapt to the need for rapid change are simply outgunned by other companies that have managed to become more agile, and consequently, more capable of promptly responding to and satisfying shifting business needs. Bigger companies are finding themselves having to make a choice between adapting to the new world and seeing their market share erode as more nimble competitors siphon off their customers. Some organizations find themselves having to restructure to become more competitive—effectively transforming themselves into many smaller autonomous units, with each acting as if they are a separate company. We have heard many people speak of sizing a team in terms of pizza delivery.
Although the two-pizza theory sounds catchy, we believe that the underlying premise is the ability of the team to effectively communicate and collaborate. Some business problems cannot really be divided into such small chunks that they can be solved in a timely manner by such a small team—not to mention the inconvenient reality that coordinating across many small teams is not without its own problems of complexity. We saw one large organization enthusiastically take this approach, oblivious to the fact that their customers were unable to benefit from many small apps that were not well integrated. We conclude that the size of the team should have more to do with the complexity of the problem and its suitability to be deconstructed into smaller units versus how many pizzas you get delivered. Of much greater importance than team size is how you handle knowledge delivery.
12.8 Knowledge Management
The effective management and sharing of complex technical knowledge is a crucial aspect of DevOps. We find that the technology industry today is largely composed of specialists, who, although highly capable and accomplished, often have rather narrow areas of technical knowledge. DevOps can help discover, capture, and share all of the technical knowledge that exists throughout the organization. In our highly technical world, we find many areas of expertise are required to keep systems running smoothly; it takes considerable time to master languages, frameworks, and methodologies. Often, we find that silos develop across the organization to centralize specialized knowledge, but at the same time, it also leaves many technology professionals without the benefit of a wider “systems” view that is crucial for troubleshooting and understanding how the system behaves as a whole. Systems administrators often have their own view and even specialized lifecycles governing their work. Too often, DBAs and middleware administrators are equally as siloed, adversely affecting the creation of a comprehensive view of system architecture. DevOps seeks to enhance communication and collaboration across the organization to maximize sharing information, while still maintaining the existing organizational structures, which are often there for a good reason.
We would certainly be remiss if we did not acknowledge that some DevOps thought leaders actually want to eliminate organizational structures altogether, a practice that is more feasible when you are building your own Internet startup from your dorm room than dealing with today’s large firms.
The main building block for success is the cross-functional high-performance team.
12.9 The Cross-Functional Team
Cross functional teams are the very best aspect of taking a DevOps approach. With cross-functional teams, highly skilled professionals from different disciplines are temporarily assigned to work together as a team and collectively held responsible for the results. In practice, we find that these teams consist of developers, operations deployment engineers, testers, and usually a business subject matter expert. These folks still report into the organizations from which they come, but are temporarily assigned to the cross-functional team on either a full-time or part-time basis. The cross-functional team is able to avoid many dysfunctional behaviors that we often see when technology professionals from different parts of the organization interact.
To really understand and deal with disruptive organizational behaviors, we often find ourselves looking at how the managers of different groups interact.
DevOps is of great value to agile development, and, truthfully, the rapid nature of agile development makes DevOps an absolute necessity. However, this does not necessarily restrict DevOps to agile.
12.10 Is DevOps Agile?
The simple answer to the question of whether or not DevOps is strictly agile is no. We have employed DevOps successfully in many waterfall environments. Getting development and operations to collaborate and communicate effectively is imperative, regardless of whether the team is embracing agile, waterfall, or any other methodology. However, agile development, with its focus on rapid iterative development, cannot possibly survive without DevOps and its well-respected capabilities to streamline application build, package, and deployment.
DevOps does not depend upon agile, but agile depends heavily upon DevOps. The successful implementation of DevOps benefits greatly from taking an agile approach. When we implement DevOps, we do benefit greatly from agile principles, so, in that sense, it is true that the DevOps transformation is greatly enhanced by agile. But, rest assured that even if you are in a waterfall organization or have a project that must adhere to a waterfall methodology, DevOps will absolutely still provide great benefits. Also, please keep in mind that enlightened waterfall enthusiasts, echoing Winston Royce’s original description, also consider their methodology to be iterative. Those who portray waterfall as not being iterative are almost always simply looking to discredit this approach.
12.11 The DevOps Ecosystem
Every organization exists within an ecosystem, and successfully implementing DevOps depends largely upon practices being aligned successfully within the existing structures. When we consider how best to implement DevOps, we spend considerable time understanding the existing organizational culture and history. We dig deeply into the communications patterns and existing means of operating. We also look outside the organization to the competitive pressures, economic environment, and the regulatory forces that may affect the business. DevOps will only succeed if you take this holistic “systems” view of the organization.
A fundamental aspect of this effort is to understand how each group operates and where in the software and systems lifecycle they are involved, as well as its interactions with other groups. We customarily try to help each group get involved earlier in the lifecycle, a practice referred to as moving the process upstream.
12.12 Moving the Process Upstream
Moving the process upstream means that the team gets involved earlier in the lifecycle. Traditional IT projects focused on determining requirements and deciding what should be built in the beginning of the software or systems lifecycle. Having only some stakeholders involved in the beginning means that some decisions will be made that impact others, who will only learn of these decisions later in the lifecycle when things are pretty much locked in. On the other hand, obviously, you cannot reasonably involve everyone up-front. Pragmatic choices have to be made, but increasingly we find that including certain key stakeholders up-front results in higher quality and increased productivity.
QA and testing definitely emerge as one of the disciplines that should be involved from the very beginning, especially since it has long been established that you cannot add quality in later.
Moving upstream is sometimes called “left-shift.”
12.12.1 Left-Shift
Left-shift refers to bringing in stakeholders earlier in the lifecycle. For example, we always try to bring operations into the deployment process from the very beginning because it is an extremely effective way to implement robust deployment pipelines. The alternative is to allow development to handle their own deploys in the beginning of the software development lifecycle, in what are usually called the lower environments (e.g., development test and integration).
Recently, we came across a scenario that we did not expect whereby operations developed an approach without bringing in development early enough in the process. We have decided to call the requirement for Ops to include development as right-shift.
12.12.2 Right-Shift
As a general rule, operations groups build their own tools and processes to manage the production environment. Recently, we came across a scenario where the operations group developed an essential infrastructure tool that affected the way in which software would be handed off from development to operations. This approach made quite a few assumptions about development, some of which were valid, and others that turned out to be less than accurate. The Ops team had kept quiet about what they were doing until the work was completed. From our conversations, we realized that the operations managers were not confident in how much they would be able to accomplish and therefore decided not to say anything until the work was done rather than be judged poorly for not delivering what they had said they would.
We now refer to “right-shift” as the need for operations to involve development in their efforts to build well-designed infrastructure to transition software and systems from development to production operations.
There are two key lessons here. The first is that technology professionals need to feel safe when doing their work. As Deming reminds us, we need to “drive out fear.” The operations team kept their development counterparts in the dark because they were afraid of criticism if they didn’t measure up to expectations. The second is that stakeholders need an opportunity to give their input when systems that will affect them are being designed.
DevOps should always involve both development and operations (and often other stakeholders such as QA, testing, and information security as well). In practice, we see much DevOps-related activity going on exclusively in development.
12.13 DevOps in Dev
DevOps in Dev refers to efforts by the development team to create automated application build, package, and deployment without any involvement from their operations colleagues. This is a common approach, and we would be remiss if we failed to discuss this reality.
When DevOps is exclusively in Dev, developers are focused entirely on build, package, and deployment to development test environments. This is usually accomplished through the use of a continuous integration server and often fails to adequately consider the requirements for deployment to the upper environments, including user acceptance testing (UAT), staging, and production. Some development teams are capable of building automation that can be seamlessly run by the operations team, but, by and large, we view DevOps as requiring collaboration between development and operations.
The real danger of DevOps in development is that the operations team will never reach a level where they can be effective at application deployment.
Writing deployment automation is a key requirement for DevOps and should be treated as a full development effort.
12.14 DevOps as Development
Automating the deployment is often a full-scale development effort. Sometimes, teams consider this up-front and create applications that can be deployed effectively using automated procedures. But, more often than not, we find that we need to create automated procedures as part of a full development effort.
In these situations, we usually find it advisable to start by watching the developers doing the deployments to development test environments using manual procedures or perhaps automated scripts within the continuous integration servers. We almost always approach this effort as an agile iterative project whereby we start with the existing procedures, which may be less than ideal, and then iteratively work to improve the process by automating each successive step.
We call our approach the three-step process.
Although we like to be pragmatic in our approach, the end goal is certainly the fully automated deployment pipeline.
12.14.1 Deployment Pipeline
The deployment pipeline, an automated encapsulation of the application build, package, and deployment, is the most common goal of any DevOps transformation. Creating the deployment pipeline sounds easy, but actually involves quite a bit of complexity.
Automating each step is often not an easy task at all, and some steps may be very difficult to automate without any user intervention. Although having a single pushbutton deploy is a nice goal, ensuring that you cannot make mistakes is a far greater priority.
Deployment pipelines should always be used consistently across all environments, from development test through production. They also should provide traceability, usually in the form of logging. Most importantly, we like to see at least half of the code focused on testing and verifying each step. The deployment pipeline should enable the team to handle many deployments without any chance of human error, and if something does go wrong, your deployment pipeline should “fail fast” so that the problem is immediately identified and may be addressed in a timely manner.
The DevOps approach facilitates the creation of the deployment pipeline by ensuring collaboration and communication across all stakeholders whose expertise is needed to successfully accomplish this goal.
Deploying application upgrades and fixes is essential, but equally important is understanding and managing application dependencies.
12.15 Dependency Control
Understanding and managing dependencies in any complex software architecture has always been a tough challenge. We believe this to be an area where DevOps can truly add value by ensuring that the various owners of software components are engaged in identifying and documenting dependencies.
All too often, this ship has sailed, in that the developers who wrote the code may have already moved on to the next project. When this happens, the best that you can hope to do is capture this information as part of the testing processes, as well as through incident management when things unfortunately go wrong.
Some companies have gotten very creative with discovering and understanding dependencies through a form of testing where they randomly cause components to stop processing correctly and then observe the impact. This approach has been coined by at least one company as a “chaos monkey.”
Another approach is to try to encapsulate all dependencies inside of a deployable structure. Microservices has emerged as an approach in which systems are divided into completely self-contained deployable structures. We are currently working on automating the deployment of microservices and see some value in this approach. However, where the business problem itself has many inherent dependencies and great complexity, microservices may be less than ideal because integration challenges will be numerous. Similarly, we are working with container-based deployments, which also hold much promise.
Containers are lightweight virtualization structures that allow for a complete structure capable of encapsulating dependencies and thereby reducing complexity during deployments to be created programmatically. The developer works within the container, which is created through code, and then deploys the code to each environment as a fully contained structure. Although we view containers as being an innovative approach, it will be some time before we fully understand its advantages and complexities. We remain convinced that you need to fully understand your architectural dependencies and design deployment approaches that can handle these essential considerations.
Understanding dependencies is essential for your success, and this is certainly no easy task. Just as important is an approach to handle the complexities of configuration changes.
12.16 Configuration Control
Configuration control is a key discipline for any deployment framework. Once again, understanding the configuration dependencies is the first step and benefits greatly from the DevOps approach to communicating and collaborating with key stakeholders. We see many organizations that struggle greatly with configuration control, often because key stakeholders may have moved on to other projects and their expertise may no longer be available. We view documenting the technical configuration as being a fundamental requirement. Verification of all configuration dependencies should be part of your overall environment management strategy.
You also need to verify that the correct code has been deployed—a procedure known as a configuration audit.
12.17 Configuration Audits
We generally start our DevOps assessments by asking if the organization can prove that the correct code has been deployed and that it has not been tampered with. The physical configuration audit, described in almost any industry standard or framework, verifies that the right code is running in production, usually by checking the version ID and often through the use of cryptographic keys such as MAC SHA1 or MD5. The deployment pipeline must conduct a physical configuration audit to verify that the correct code has been deployed.
We have given many examples of the physical configuration audit, and closely related is the use of cryptography to ensure that unauthorized changes can be detected in what we refer to as the secure trusted application base.
Similarly, the functional configuration audit verifies that the code is doing what it should be doing and is closely aligned with QA and testing.
12.18 QA and DevOps
We noted earlier in this chapter that agile depends upon DevOps because of the need to support rapid application deployments that are crucial to the success of the agile development effort. DevOps itself depends heavily upon QA and testing. It is common for the DevOps deployment automation to quickly outpace the QA and testing function’s ability to certify that releases can be deployed to production. DevOps helps the QA effort by ensuring effective communication between development and the QA and testing function. There is much synergy in this approach because developers cannot usually test their own code effectively. However, the development organization does have a great deal of technical expertise that can help QA and testing better understand effective use of its limited resources. The QA and testing organization can have considerable synergies with the development organization in much the same way that operations also enjoys synergies with their development counterparts.
DevOps depends upon good testing and also helps facilitate excellent testing. We see a similar synergy between development and information security.
12.19 Information Security
Information security plays a key role in ensuring that the software and systems under development can withstand what has become a constant attempt to penetrate and adversely impact most production systems. We are confronted daily by stories of systems that have been harmed by hackers who often have malintent, including stealing personally identifiable information and using it for identify theft and other forms of fraud. Unfortunately, the information security group may be lagging behind in understanding the application and what must be done to safeguard corporate assets from those who would intend to steal them. Once again, we view a synergy between InfoSec and development in which development shares its technical expertise and benefits from learning security-related best practices.
One of the most important security competencies is the ability to build infrastructure as code.
12.20 Infrastructure as Code
Infrastructure as code means that you can programmatically build the systems environment from scratch using automated procedures. Most competent systems administrators automate every aspect of building and provisioning servers. DevOps itself was motivated by much of the thought leadership in the agile systems administration community. We have regularly worked to automate provisioning of operating systems to middleware and then, of course, the application deployments as well. Infrastructure as code has many advantages, including being able to quickly provision servers with identifiable and verifiable configurations on both bare-metal and cloud-based providers.
This is very important because when a security incident occurs, your best approach is to provision a completely new machine and quickly deploy your code. Infrastructure as code makes this possible. Without this core competency, you would be forced to try to identify the malware left on your machine and attempt to remove it without damaging the system. This is often impractical, so being able to completely reprovision your machines is a much better approach.
12.21 Taming Complexity
We have discussed many complicated topics in this chapter. DevOps places a strong value on taming complexity; it accomplished this by facilitating the cross-functional team to share knowledge and by providing a means to capture and store knowledge.
12.22 Automate Everything
Another key lesson of DevOps is how much power and control are gained by automating everything. Without automation, DevOps and the deployment pipeline would simply not be possible.
12.23 Disaster Recovery and Business Continuity
We view disaster recovery and business continuity as being key areas that also benefit greatly from the DevOps approach. The ability to rebuild servers using infrastructure as code, understand dependencies, and manage complex configuration changes are all essential aspects of managing significant interruptions in service that trigger disaster recovery and business continuity efforts.
12.24 Continuous Process Improvement
Implementing DevOps is no easy task. The most effective approach is to start small and continuously improve by approaching DevOps as an agile development effort. DevOps itself is an emerging industry capability, and thus it should come as no surprise that DevOps is also evolving and continuously improving itself.
12.25 Conclusion
The way that you approach DevOps in one particular organization may be quite different from how you implement it in another. The platforms and technologies may greatly affect your strategies. But the core DevOps principles do not change. We have discussed many of them throughout this chapter and the rest throughout this book. DevOps is a journey of discovery cooperation, which if embraced by stakeholders and implemented well, leads to enhanced communication and productivity throughout the organization.