Chapter 15: Defining the Appropriate DevOps Platform Strategy

Congratulations! If you've made it this far, which is the start of Section 4 of this book, you are about to enter the final set of topics that will be covered in this book. Specifically, we will now turn our attention to understanding the approaches we can use to implement DevOps capabilities, as well as how to use those capabilities to support digital business transformations.

There are four primary main topics that will be covered in this final section on applying DevOps to drive digital business transformations, as follows:

• Avoiding DevOps implementation pitfalls
• Deciding on an appropriate DevOps platform strategy
• Addressing the pitfalls of DevOps implementations
• Interviewing the experts
• Dealing with corporate implementation mandates
• Dealing with creative versus repeatable pipeline activities

We will cover the first three topics in this chapter before discussing how we can use VSM and DevOps tools to help affect digital business transformations in Chapter 16, Transforming Businesses with VSM and DevOps. In this chapter, you will learn about four basic DevOps implementation strategies, as well as the pros and cons of each. Then, we'll move on and discuss some of the pitfalls that can harm a DevOps implementation initiative. Finally, we'll review 18 strategies that can help improve your DevOps toolchain implementations.

Much of the information presented in this chapter comes from DevOps experts and the VSM and DevOps tools companies that were interviewed for this book. However, we've waited to recommend these strategies as they are relevant regardless of the DevOps tools and toolchains your organization may select for deployment.

We must address the implementation options and issues before we move on to the next chapter, because it won't matter how much benefit can be derived from VSM and DevOps tools if we fail to deploy them. Successfully deploying our VSM and DevOps methods and tools is necessary to support our digital transformations.

In the next chapter, you will discover that modern VSM tools and practices support aligning DevOps-based software deliveries to make other organization value stream improvements. In this sense, we will have come full circle to show how improved software delivery capabilities support the organization's value stream improvements, which are necessary to compete in our modern digital economy.

With these objectives in mind, let's begin by exploring the potential pitfalls of DevOps implementations.

Avoiding DevOps implementation pitfalls

While preparing to write this chapter, I interviewed several people whose opinions I value as expert practitioners in deploying DevOps tools and toolchains. They are Scott Ambler (Vice President and Chief Scientist for Disciplined Agile at the Project Management Institute), Al Wagner (HCL Software VSM and DevOps Evangelist), Helen Beal (Chief Ambassador at DevOps Institute and Chair of the VSMC), Pramod Malhotra (DevOps Services Executive), and Joel Kruger (Senior DevOps Engineer specialized in developing reusable software factories).

Helen Beal's credentials in the DevOps and DevOps communities are immense. She explains how DevOps implementations typically evolve to include multiple tools and the need for VSM tools to both improve and orchestrate software delivery workflows. Scott Ambler approaches DevOps from a Lean-Agile perspective and choosing your Way of Working (WoW) when implementing DevOps platforms.

Pramod Malhotra discusses his DevOps implementation experiences as a Federal Prime and commercial systems integration contractor. Al Wagner will represent the point of view of a VSM/DevOps platform vendor. Finally, Joel Kruger will discuss the benefits of creating downloadable CI/CD and DevOps configurations as reusable software factories.

With these initial introductions made, let's get going.

Interviewing the experts

In this section, we will understand the views of five industry experts and their take and ideas on DevOps.

Interview with Helen Beal

Helen Beal is a DevOps and Ways of Working coach, Chief Ambassador at DevOps Institute, Ambassador for the Continuous Delivery Foundation, and the Chair of the Value Stream Management Consortium (VSMC). She provides strategic advisory services to DevOps industry leaders and serves as an analyst at Accelerated Strategies Group.

I am privileged that Helen agreed to serve as a technical reviewer for this book and wanted to contribute her thoughts for this chapter. I've also had the opportunity to work with Helen as one of the advisors of the VSMC. So, with this introduction made, let's hear what Helen has to say.

Developing a DevOps mindset

Helen starts by stating that she does not believe that organizations should create DevOps teams. Instead, she views DevOps as a mindset and not so much an organizational structure. As she puts it, "instill the mindset that DevOps is a cultural movement for the whole organization and design value stream teams that practice DevOps."

Value streams are the fundamental organizational structure for delivering value in a Lean enterprise. DevOps is a collaboration and technical implementation strategy that supports value streams in IT. So, in this context, Helen's statement makes overwhelming sense.

DevOps began as a collaboration strategy to align the efforts of software development and operations teams to improve value-based software delivery capabilities. You also know now that mature DevOps pipelines collectively implement tool integrations, activity automation, and work and information orchestrations to improve value stream flows. DevOps requires a change in thinking and culture to align software delivery with value streams.

Empowering people

DevOps changes the way people work within and across the organization. For that reason, DevOps affects the organization's people and its culture. Helen goes on to note that we must empower people to participate – "people don't like having change done to them – they have to be given autonomy to find their own way forward." Helen also notes that it's hard to become empowered if you haven't been for a long time.

Leading the way

Helen makes the point that leaders lead. Therefore, they can't take a backseat role in overseeing the deployment of DevOps tools and toolchains, as well as the organizational changes that must come to utilize this new way of working effectively.

To be effective in their roles, the executives within the organization must learn the skills and principles of DevOps before they come out with any mandate for its use. Otherwise, they can't possibly make informed decisions on tools, related budgets, aligning resources, and training requirements.

Helen observes that many leaders think they are above learning. Instead, leaders need to retrain the organization, starting with themselves. DevOps moves the organization away from a traditional hierarchical and command and control structure. Therefore, its leaders – including the organization's executives, managers, and coaches – must have the skills and knowledge to encourage their teams to self-discover ways to improve and help remove the impediments identified by their teams.

Evolving over transforming

A trend among many analysts today is to speak about using Lean-Agile practices to support business transformations. In this book, you have heard me describe how we can use VSM and DevOps methods and tools to support business transformations to compete in our modern digital economy.

However, Helen is concerned that the notion of effecting business transformation is too big bang and that many people have change fatigue from previously failed transformations. Instead, Helen believes we are better served by aiming for evolution with continuous and incremental improvements. In this context, Helen believes it's better for the organization to steer its efforts toward achieving a culture of continuous evolution, and not a one-time and relatively short-term business transformation objective.

Making time to learn

Helen noted previously that the organization's leaders must gain knowledge to support an effective DevOps transformation. But that same learning requirement extends throughout the organization, and time and resources must be made available for their continued education.

These training requirements mean the organization's executives must make time for learning. In other words, continuing learning must be viewed as an ongoing job requirement and not a nice thing to have.

DevOps evolution requires all humans in an organization to unlearn long-held beliefs and practices (for example, PRINCE, project management, and so on) and learn new ways of thinking and working. Helen notes that humans have cognitive load limits, and it takes time to learn and practice new behaviors. Therefore, time must be allocated to learning outside of business as usual (BAU) and new feature development and resolution.

Resolving technical debt

Though not technically part of training, Helen made the same point about resolving technical debt. In other words, organizations need to build time into their busy schedules to refactor software code, as this helps address performance issues that accumulate when the teams are focused on conveniently delivering new features. Similarly, the development and support teams need to improve a product's architecture and design and implement new technology improvements from time to time.

Building the DevOps platform

Helen points out that new DevOps tools become available all the time and that their requirements change over time, too. Therefore, software delivery teams need to build an adaptable DevOps framework – think API first. Taking on a DevOps framework strategy helps ensure the platform remains extensible over time, allowing new capabilities and future technologies to be integrated to support the organization's evolving software delivery needs.

Another important consideration is to aim for traceability throughout the toolchain. Helen believes you should accept that your DevOps platform will become a heterogeneous toolchain, even if you start with a commercial DevOps as a Service (DaaS) platform. Supporting an evolutionary approach to make continuous improvements in your DevOps toolchains will drive that eventual outcome. On the positive side, your VSM platform will hook it all together for you.

Also, Helen believes you should aim to provision your DevOps toolchain/VSM platform as a service to your value stream teams. While teams can have different requirements and tools, you can separate the architecture or categories of your toolchain from the actual toolchain and switch tools out (for example, NUnit for .NET unit tests, JUnit for Java unit tests). We'll discuss this topic more in the Adopting software factory strategies and Building reusable software factories sections, both of which discuss how downloadable and self-service configurations can serve as reusable software factories.

Overcoming DevOps implementation challenges

Helen believes it's difficult to establish a common view of what DevOps is in an organization. Some people may think of it as just a piece of automation, such as implementing a CI/CD pipeline. Others recognize that the breadth of DevOps includes the entire value stream, connecting all parts of the business and incorporating Agile, Lean, site reliability engineering (SRE), DevSecOps, DataOps, or AIOps.

Regardless of how your organization elects to define DevOps, make sure you have evangelists and communities of practice define and socialize why and how DevOps supports your business.

Helen notes that the most common challenge when implementing DevOps is around culture. The tools and toolchains are the easiest things to see and implement. But it's changing people's thinking and ways of working that is much harder to accomplish in practice.

To affect cultural change toward positive outcomes, Helen believes the organization must actively build psychological safety, train leaders to distribute authority, invest heavily in continuous learning, and teach people to talk about their emotions, feelings, and behavior. In addition, she suggests organizations employ the latest findings in neuroscience in their efforts to help create real and lasting change.

Finally, Helen discussed the relevance of KPIs and OKRs to define the goals and objectives for your DevOps implementation activities. But she also notes that they should not be inflicted on teams. Instead, allow your DevOps teams to define and measure their objectives and metrics. And make sure your teams have the tools (that is, VSMPs) to monitor their results continually and in real time.

Interview with Scott Ambler

Scott is the co-creator, along with Mark Lines, of Project Management Institute (PMI)'s Disciplined Agile (DA) toolkit. Scott works also with Al Shalloway, the thought leader behind DA FLEX. Disciplined DevOps is one of the layers of the DA toolkit, and it is from this perspective that Scott discusses DevOps implementation pitfalls and platform strategies.

Scott has extensive IT experience and has shared his knowledge for years. He has authored or co-authored over 20 books in the IT and process space and has worked with organizations worldwide to help them improve their way of working. Scott is currently the Vice President and Chief Scientist for DA at PMI.

You can't buy DevOps

Scott believes that the greatest implementation pitfall, when it comes to DevOps, is when the decision-makers do not understand the scope of the challenge. Too many organizations want to install DevOps, thinking that they can buy their way out of their current mess. Or, they think they can transform to DevOps in a few short months. Nothing can be further from the truth.

Successful DevOps implementations require significant and long-term investment in people, processes, and technology. First, there is a significant people issue. Every firm Scott has worked with needed to develop new skills, implement training programs, and find or develop mentors and coaches to support the learning journey. DevOps also clearly requires new WoW, as you've seen throughout this book, plus investments in tools and technologies to support this new and evolving WoW.

DevOps is about more than Dev and Ops

The most important observation that Scott makes about DevOps, echoing what Pramod and Al share, is that it is about more than just merging Development and Operations. In disciplined DevOps, one of the four layers of the DA toolkit, PMI has melded together six key aspects of enterprise-class DevOps:

• Solution delivery: This is a fit-for-purpose, tactically scalable approach to software-based solution delivery. Called Disciplined Agile Delivery (DAD), it weaves all aspects of solution delivery to produce consumable solutions from beginning to end. Consumable means that something is functional (it works), usable (it works well), and desirable (people want to work with it). A solution potentially includes software, hardware, documentation, business process improvements, and organization structure improvements. Agile teams typically focus on producing working software, where DA teams focus on producing consumable solutions – a big difference.
• DevSecOps: Disciplined DevOps builds security, both information/cybersecurity and physical security, practices right into the toolkit for the reasons that Pramod and Al share. Security should never be an afterthought.

  In this chapter, we'll see the acronym DevSecOps used oftentimes in lieu of the more traditional DevOps acronym. Where DevOps evolved as a collaborative strategy to break down the silos between development and operations, DevSecOps brings security into the collaborations. In other words, security is involved in each phase of the software development cycle as a part of the team, and not a siloed function.

  The goal of DevSecOps is to avoid security issues cropping up in a deployed software solutions. To achieve this goal, the DevSecOps pipeline includes integrated, automated, and orchestrated threat modeling and security testing activities. The automation capabilities ensure each new code change is thoroughly tested with attending reports and alerts generated detailing any potential vulnerabilities.

  In a DevSecOps pipeline, security is not limited to software coding and testing activities. Once released, monitoring tools continuously scan for threats and vulnerabilities, generating incident reports when and as they are discovered.

• Data DevOps: Data is the lifeblood of your organization, yet it's often ignored or at least treated as a low priority in most DevOps implementations. What is the value of deploying changes to your software many times a day if you can't also deploy changes to your data at the same rate?
• Multi-solution support: The DevOps philosophy of you build it, you run it, and you support it is a valuable motivator for better collaboration and process improvement. But it doesn't scale. When your organization has hundreds, or even thousands, of systems in operations, your end users need a common and coherent strategy to get support.
• Common IT operations: Once again, if you have many solutions running in production or supporting many DevOps pipelines or both, you want to support some common operational infrastructure to streamline things. In several organizations, Scott helped them identify the common, shared infrastructure and what is unique to support the needs of individual teams. How you treat, support, and evolve shared infrastructure is different from how you treat application-specific functionality. Scott's goal is to help his clients learn how to work with and evolve the two types of infrastructures accordingly.
• Business operations: Time and again, Scott has found that you don't want to restrict Ops to IT operations. That's certainly important, but if your business operations aren't comparatively flexible, you're better off investing effort in improving that aspect of your value stream first.

Scott believes a critical danger in DevOps implementations comes from ignoring the fact that there is more to DevOps than merely telling Dev and Ops teams to start working together in collaboration. Disciplined DevOps shows how all these critical aspects fit together in a streamlined and evolvable manner. And it also shows how to improve in each of these areas in parallel, but still as a collaborative effort.

In the next six subsections, Scott will explain how DevOps is employed within the DA Toolkit.

We'll start with the first DevOps concept, Start with a mindset.

Starting with a mindset

Scott points out that the Agile Manifesto, which most people point to as a description of the Agile mindset, was written over 20 years ago to address the problems of that time. But times have changed, and we've learned a few things along the way. Scott stresses that the mindset that was captured in the Agile Manifesto is a great start, and that there are many great ideas around a DevOps mindset that you've learned in this book, but what we really need is a mindset for business agility. Although DevOps is an enabler of value streams and business agility in general, it doesn't exist in a vacuum. To be successful, you need to look at the bigger picture beyond DevOps.

This approach is what PMI has done with the DA toolkit, including its DA Mindset, which captures principles, promises, and guidelines for business agility. A common mindset for business agility provides a foundation from which people can collaborate and work together, supporting the creation of a shared culture among your disparate teams.

Unexpectedly, the DA community found that this still wasn't enough. People from different domains come to the table with their own unique experiences, skillsets, priorities, and ways of looking at things. For example, security professionals have their unique philosophies, as do data professionals, marketing professionals, product managers, and other critical stakeholders. So, not only do we need the foundational principles, promises, and guidelines of the DA mindset, but we must also extend them with unique philosophies for each of the groups, or tribes, within our organizations.

Each process blade within DA, what DA calls process areas – such as security, data management, enterprise architecture, IT operations, and many others – extend the DA mindset with several philosophies pertinent to that domain. This approach enables people to focus on those process areas to tackle the challenges they face with a fit-for-purpose mindset for their context, all while sharing a common culture with the rest of the organization.

To summarize, DA promotes the idea that we need a common mindset to interact well with others. And we must also respect the differences that people bring to the table, including their outlooks. Success in DevOps requires everyone to evolve their way of looking at the world and apply their unique strengths in new ways.

Your technical debt has come due

If Scott had to pick a single reason for the slow implementation of DevOps within organizations, it would have to be technical debt. Poor quality source code and a lack of automated regression tests have been the bane of anyone bringing DevOps into their organizations for years. But data-oriented technical debt and quality problems with your data sources have been a blind spot for many organizations for years. Although senior leadership often recognizes there's a problem, they've often given up on ever trying to address it. But unless your organization addresses all aspects of technical debt, you will struggle to implement DevOps successfully.

Belying the name, the primary causes of technical debt aren't technical in nature but rather people oriented. In fact, Scott's experience is that most technical debt stems from project management thinking and behavior. In particular, the desire to be on time and on budget tends to force teams to produce lower-quality solutions than they would prefer, thereby increasing technical debt. The we'll fix it later claim rarely materializes. And, of course, the poor levels of training in design and architecture concepts and techniques are also contributors to technical debt.

Over the years, every organization Scott has worked with – spanning software firms, financial organizations, manufacturers, and grocery chains – needed to invest in paying down technical debt. However, investing in software quality, automated tests, and improving data quality always proves to be the lion's share of your DevOps infrastructure investment, requiring years of hard work to resolve.

Evolving into a fit for purpose DevOps strategy

The DA toolkit takes a very different approach to providing process advice. Frameworks such as SAFe or LeSS provide a collection of best practices that prescribe what to do, which can be an excellent start. But it's only a start. Scott believes your organization is unique and facing a unique and evolving situation, so you need to go beyond Agile frameworks.

Instead, the DA toolkit tells you what to think about, provides options to address the challenges you face, and describes the trade-offs of those options. It puts you in a position to decide on what techniques to experiment with to develop your fit-for-purpose approach. It supercharges the "fail fast" improvement strategy by helping you make better decisions and thereby fail less often, leading to faster improvement.

Once again, Scott notes that a common implementation pitfall is thinking that you can install or quickly transform into a DevOps organization. You really can't. Instead, you need to do the hard work to evolve into DevOps. You want to have a fit-for-purpose DevOps strategy that reflects your organization, your people, and your goals, leading to your desired outcomes. You need to take responsibility for your WoW, and the DA toolkit helps you do exactly that.

You can buy some of your DevOps infrastructure

We've talked about DevOps platforms. Yes, you can buy DevOps tooling or adopt cloud-based DevOps as a Service (DASS) options and combinations thereof. So, the situation isn't entirely bleak, but you still need to install and configure the tooling. You still need to train people so that they know how to use. You still need to use the new infrastructure effectively. And as Scott pointed out earlier, you also need to invest in paying down technical debt. A vital aspect of that is infrastructural – such as developing automated tests. So, in short, you can buy some of your infrastructure, but you'll need to build a lot of it yourself.

Disciplined DevOps is table stakes for Agile enterprises

You can do the hard work of figuring out how development, operations, security, data, and support all fit together. And you can work through how it all supports your value streams and fits into your overall organization. Alternatively, you can look at the DA toolkit, which has already done all this heaving lifting, and use that as a starting point.

The DA toolkit shows how it all fits together, going far beyond DevOps to address how to effectively implement value streams and then support them at the enterprise level. Being choice-based, rather than prescriptive, teaches your teams how to evolve a fit-for-purpose WoW to be as effective as it can be, yet still fit into the overall value stream they're part of. And, of course, Lean governance strategies are woven through the toolkit; otherwise, chaos would ensue.

Evolving with the times

There are two critical and unfortunate observations that you must embrace if you're to be successful with DevOps:

• Your organization is unique: We've said this before; the implication is that you need to choose your WoW so that you have something that is fit for purpose.
• Your environment is fluid: Your WoW cannot be static; instead, it must evolve as your situation evolves. You must become a learning organization that knows how to improve continuously.

DA teaches you how to get better at getting better. It explicitly shows you that you have options and how to choose the best option for your current situation. It embeds improvement strategies at the team and organizational level, and also embraces a guided approach to experimentation beyond the fail fast mantra of mainstream Agile.

There isn't an easy answer

Scott couldn't say this enough – there are no shortcuts. You can't buy a DevOps solution, you can't install one, and you can't quickly transform it into DevOps. Instead, you need to do the hard work of evolving your culture, evolving your WoW, and improving your infrastructure. His experience is that DevOps is about people, processes, and technology. His final admonishment is ignore these issues at your peril.

Interview with Pramod Malhotra

Having direct and extensive operational experience, Pramod is an executive with Salient CRGT who serves as the company's DevSecOps and application modernization thought leader. In addition, he has overseen numerous DevOps implementations in large commercial businesses and federal government agencies. Having worked with Pramod directly over the past 5 years, it's been interesting to see how his views have evolved along with the industry. Here are some of his insights.

Getting executive support

First, Pramod believes that any efforts to implement DevOps tools and toolchains will fail without executive-level support. This concern has been a recurring theme throughout this book. But Pramod has observed firsthand how difficult it is to implement DevOps at scale without executive leadership.

First, DevOps requires a cultural change that cannot be driven as a bottoms-up strategy. It's natural for people to resist change and to continue what they have been doing. For that reason, DevOps cannot be mandated either. Instead, executives need to take a leadership role to establish relevant Objectives and Key Results (OKRs) and help drive the organization to obtain the desired results.

On the other hand, the benefits of DevOps as a software delivery improvement strategy are worth the effort. DevOps improves software development capabilities, and everyone throughout the enterprise is affected in terms of how they do their work. In other words, in the digital economy, the software delivers value as standalone products, enables digitally enhanced features in physical products, and supports value stream process improvements throughout the enterprise.

Bluntly, Pramod would suggest organizations should not waste their time attempting to implement DevOps tools and toolchains unless they have chief and line of business executives, support to fund the initiatives, can allocate sufficient resources, and can hold people responsible for achieving identifiable and measurable outcomes with timeframes, budgets, and ROIs that justify the efforts.

Implementing effective training programs

Next, training throughout the enterprise is critical. DevOps' tools and toolchains cannot be thrown over the wall with the expectation that the development and operations teams can effectively use them. Moreover, the value streams that can benefit from DevOps-based software delivery capabilities need to understand what's possible and how to engage effectively with the IT organization. This statement is also true for business owners who depend on competitive software delivery capabilities to achieve the enterprise's strategies, goals, and objectives.

This training can extend beyond the organization's employees to include third-party consultants and vendors. For example, most government agencies and large commercial enterprises leverage external software development organizations to build applications that support the business. If the executives choose to commit to DevOps, then their consultants must make similar commitments.

Finally, stakeholders also need training on DevOps. I personally define the term stakeholders so that it includes anyone and everyone who has an opinion that matters. Anyone who has managed software delivery projects over any period will have experienced situations where external stakeholders affected the outcomes of their projects.

Sometimes, it's because those external stakeholders are competing for the budgets and resources that have been allocated to your projects. But other times, the stakeholders may not see the value of the work the software delivery team is performing, and they may feel left out of decision making. While Agile practices help address some of those communication and collaboration issues, we can't forget that the concerns and needs of our stakeholders are every bit as important once the organization moves to DevOps-based software delivery.

Going all in

In this book, you have learned about the importance of using value stream mapping to assess the current state of operations and identify improvement opportunities that help achieve a desired future state. So, different from Agile retrospectives – which evaluate areas for immediate improvements on a limited scale – VSM initiatives tend to take a larger and longer-term view to make sure that the entire value stream, as a system, operates with greater efficiencies and less waste.

DevOps is very much a Lean-oriented strategy to streamlining software development, delivery, and support functions as an integrated, automated, and orchestrated value stream. Conceptually, DevOps toolchains operate as a streamlined software delivery pipeline.

VSM offers an approach to evaluating a series of improvement opportunities that increases the flow of value across our software delivery pipelines. For reasons of financial and resource constraints, we may need to prioritize our investments over an extended time. But this does not mean that we can afford to do the assessments and implementations in an ad hoc manner. It is a combination of such improvements that allows a DevOps pipeline to deliver software value efficiently and rapidly.

Therefore, the executives of an organization must not allow their IT organization to go about implementing DevOps tools and toolchains in a piecemeal fashion. Implementing DevOps tools and toolchains is a strategic initiative. The organization needs to plan and guide their DevOps investments, just as they would any other product and portfolio investments driven by OKRs.

We'll get into the subject of OKRs in more detail in Chapter 16, Transforming Businesses with VSM and DevOps. But for the time being, know that OKRs establish high-level expectations in terms of desired outcomes and measurable results. In other words, executives should establish OKRs as guidance for those involved in the DevOps implementation, in terms of what the expectations are and the metrics that indicate successful outcomes.

Establishing a DevSecOps Center of Excellence (COE)

So far, we have discussed the importance of having executive leadership, effective training programs, and an end-to-end DevOps implementation strategy. But the organization also needs a center of excellence to establish governance policies, ensure the construction of reusable toolchain configurations, and provide resources for training, mentoring, and coaching.

After overseeing multiple enterprise-scale DevOps implementations, Pramod has come to the opinion that organizations need to select one vendor or group of experts (COE) to establish the organization's overall DevSecOps and CI/CD platform solutions. That's not to say that multiple groups cannot participate in the DevOps platform and tool selections and governance policies. Still, it's unwise to have different teams or groups working independently. Otherwise, the organization will end up with a hodge-podge of tools to license, track, integrate, support, and sustain over the life cycle of their DevOps platforms.

The COE should help set up and guide the development of the initial DevSecOps platforms as a prototype. When the prototype platform is ready, the COE should help guide one or more product teams in their transition to the new platform. As the software product teams use the prototype DevSecOps platform, the COE and software teams should collaborate to improve and perfect the platform to support wider-scale use.

A series of rolled deployments will help prove and build out its capabilities for wider-scale deployments for larger organizations. The COE must be prepared to tweak the platform as necessary to support new needs that are identified with each new deployment. This tweaking will include integration, automation, and configurations that support additional DevSecOps activities, tools, and toolchains.

Continuous learning and enhancements keep improving the primary platform, taking it to the next level. However, suppose you don't get the fundamental platform right. In that case, the organization will get off to a bad start, which could reduce executive support and thereby reduce opportunities for future investments and deployments. In contrast, building the right basic DevSecOps platform allows the organization to use VSM to justify further investments. But, more importantly, the VSM initiatives guide the expanded use of the improved DevSecOps platform to support digitally enabled value stream improvements across the enterprise.

By having one vendor or a CoE leading the DevSecOps practice and applying VSM principles, we avoid every team or every vendor needing to be an expert at DevSecOps and VSM practices. Think of it this way: we hire software developers to build software products, not build their DevOps-based software factories.

In most cases, we would not redirect the employees building products in manufacturing firms to build the factories they work in – nor should we expect our software developers to create their software factories. Yes, many can learn the skills of a DevOps engineer – with enough time and practice. But that time and effort take away from their regular jobs to deliver value in the form of new and enhanced software products. So, don't do that – it's a non-value-added waste of their time!

It is appropriate to encourage a very open and collaborative culture of feedback, including the COE, software delivery team members, and your related vendors. It is those collaborations that raise visibility on issues and contribute to improving your DevSecOps platforms.

Defining the roles and responsibilities of the COE

The COE includes your experts on DevSecOps tools and platforms. But it must also include or be aligned with the organization's IT architecture group. The responsibilities of your DevSecOps platform COE and IT architecture group include the following:

• COE's don't make anything – they are responsible for governance and policies:

  a. Establish a list of dos and don'ts in the form of DevSecOps governance policies.

• Work with executives and portfolio management functions to establish DevSecOps platform improvement programs and budgets.

  Evaluate Software as a Service (SaaS) and DevSecOps vendor offerings:

  a. Single DevOps platforms such as AWS, Azure, GitLab, or HCL Software.

  B. Multitool platforms such as ConnectALL, Digital.ai, Plutora, ServiceNow, or Tasktop.

  c. Define the workflow to initiate VSM and DevSecOps tool and platform requests and approvals.

  d. Conduct Analysis of Alternatives (AoA).

  e. Work with legal, IT, and finance departments to negotiate licenses, terms and conditions, and service-level agreements (SLAs).

  f Establish and maintain the list of approved DevSecOps tools for use within the organization.

• Guide the work of any related VSM team and VSM initiative requiring software deliveries:

  a. Conduct IT Budget Review Boards or support portfolio management functions to establish software value stream/product development support priorities.

Deciding on the appropriate DevOps platform strategy

In Pramod's opinion, he believes the best strategy is to go with a DevSecOps as a Service (DaaS) vendor instead of an open source tools strategy. DaaS examples include Azure DevOps Services, GitLab, and AWS CodeDeploy. He cites several reasons for this view, as follows:

• It helps avoid FISMA and FedRamp compliance and approval issues associated with obtaining security approvals to use open source tools within federal government agencies.
• The organization does not need to implement, integrate, and maintain disparate tools.
• The IT organization does not need to write Infrastructure as Code (IaC) configurations.

  For example, developing custom DevSecOps platforms requires hiring experts in IaC tools such as Ansible, Terraform, and others.

• He's become a fan of Azure Kubernetes Service.

  Organizations don't need to install Kubernetes, maintain Kubernetes, and deal with the complexity of Kubernetes.

Pramod believes most organizations can do quite well with DaaS-based solutions. However, larger digital and high-tech organizations such as Netflix, Amazon, Google, Walmart, and perhaps very large federal agencies have exceptional needs for performance and large volume software deliveries that justify building custom DevSecOps and CI/CD pipelines.

But Pramod also recognizes that some IT folks feel they have better control over the operations when they can incorporate specific tools instead of a service. And, let's face it, DevSecOps tool selections are emotional issues for many developers who have developed their skills and competencies around a specific set of tools.

Interview with Yaniv Sayers

Yaniv Sayers is a Fellow and CTO in Micro Focus responsible for its Application Delivery Management and Software Factory program. With over 20 years in the IT and SW industry, Yaniv has experienced and led various transformations. He and I met virtually and discussed our experiences on DevOps and VSM. He contributes to this section in an interview I conducted with him. We started with a discussion of major pitfalls in DevOps implementations', such as ignoring the organization's DNA and challenges around Big Data and applying the Software Factory approach to overcome the pitfalls.

Acknowledging your Organization's DNA

Yaniv stresses that ignoring an organization's DNA is a common mistake that many organizations make. In the digital era, organizations that want to remain relevant look at how leading high-tech companies such as Facebook, Google, Amazon, Spotify, and Netflix leveraged DevOps to move at the speed of a bullet.

These success cases inspire executives and other critical stakeholders. They may feel pressure to become more like these other organizations and think they can find similar success by implementing the practices and tools used by these other organizations.

However, these popular references to DevOps transformations apply mainly to organizations born in the digital era. Their people are digital natives, their processes are agile from the get-go, and their technologies are cloud-native.

That is very different from IT organizations in financial services, pharmaceuticals, or government. Many enterprises still use traditional software development methodologies, such as Waterfall or Water-Scrum-Fall processes. They include a broad range of technologies that range from mainframe to client/server to cloud, plus they have people who learned their skills in a previous era. Many are not yet comfortable operating or competing in our modern digital economy.

For the traditional enterprise, the challenge and even the meaning of running "faster" can differ from the digital native experience. These organizations have a different DNA, and Yaniv believes that fact must be acknowledged. They can be inspired by what digital native organizations have achieved and how they operate, yet they cannot simply follow the digital native implementation, and they cannot ignore their past and current environment. Instead, they need to continue running their current environment while setting a transformation plan that fits their DNA.

Becoming outcome Oriented

Most organizations take a technology-centric approach. They start with the tools and equipment and work out to the users or consumers – yet most fail to achieve the desired outcome. For example, just implementing an agile planning tool does not make a project agile. The project may still operate with long delivery cycles, no fast feedback, or people finding it difficult to transition to an Agile mindset. It was Grady Booch who famously said, "A fool with a tool is still a fool." 

Instead, it would be best to start by taking an outside-in view and working with the customers and consumers to align on the outcomes that matter to them and the value streams that empower these outcomes. Then, based on the desired outcomes, you can identify and determine the people, process, and technology changes required. For example, it may require you to train your people on new ways of working, defining new roles and responsibilities, enhancing existing processes, and implementing new tools – all of which need to occur in concert as a collaborative effort to achieve the expected outcomes.

Discovering the Art of Balance

Speed, quality, cost, and happiness do not compete; they don't operate in a silo, nor should they be considered as tradeoffs - they are intertwined. For example, suppose you focus on speed and neglect quality. In that case, low quality ultimately leads to the user and employee dissatisfaction. Low quality also increases rework, which leads to higher costs and increased technical debt, which collectively leads to slower delivery.

On the other hand, higher-quality reduces failures and the need for rework. Thus, improving quality reduces costs, increases delivery speed, and promotes happiness [employee, customer, and end-user].

Stakeholders should be aware that perceived short-term tradeoffs eventually lead to the reverse outcome for the long term. Instead, be conscious that tradeoffs are interdependent and intertwined. In this context, IT organizations need to balance speed, quality, cost, and happiness continuously.

Making better decisions through Big-Data

Yaniv often encounters stakeholders making key decisions based on gut feel and guestimates, which end in sub-optimal outcomes for the lucky ones and big failures for others—for example, prioritizing investments based on their belief of what users and customers may need without obtaining customer input and feedback, deciding whether to move a release into production without a clear understanding of the release quality and security risks, changing interfaces and data models without realizing the impact on other services.

When decision-makers lack access to required data, this is typically due to a set of reasons, such as organizational and political barriers that prohibit data sharing, a variety of "languages" within the different domains that make it challenging to have a shared context, lack of integration between systems that equates to no access and no traceability between data points.

With DevOps, the data challenge becomes even more extreme. There are higher continuous delivery cycles and shift to automation, the amount of data increases by levels of magnitude, and decision-makers need to make conscious decisions faster and continuously. In practice, it becomes a big data challenge.

Organizations that want to be successful with DevOps at scale must address the big data challenge, and access to data traceable by decision-makers is the first step. In addition, having the objective to enable value-driven deliveries, conscious and continuous decision making, analytics, and machine learning should transform big data into actionable insights.

Taking a Software Factory Approach

Yaniv states that Enterprise organizations that need to transform by implementing DevOps should consider taking a software factory approach. A Software Factory aligns the organization's strategic planning with an integrated set of services, processes, tools, and data that enables the organization to plan, build, test, release, and operate and manage the software delivered to their customers.

Start by creating a baseline taking an outside-in view and working with stakeholders to align on the outcomes and value streams that empower these outcomes. Then, map the main activities, roles, systems, and data and their interaction.

Yaniv also notes that you can also leverage frameworks such as IT4IT that provide references for common value streams. For example, in IT, DevOps-oriented value streams include activities spanning the receipt of a requirement through its delivery to the user or a user experiencing a problem until resolution or paying down technical debt.

Next, analyze issues, such as missing information for stakeholder decision-making, duplicate systems, error-prone manual activities, wasted time. For each, map possible improvement areas such as consolidating to a standard agile planning service to enable dependency management and transparency, an integration service to provide data visibility, a performance testing service to enable shift left, and earlier problem detection.

After mapping the values streams and analyzing the issues, you have a common language and understanding of outcomes that matter, current problems, and opportunities – in other words, the valuable services. Then, aim for shared services wherever possible and differentiated when required due to a business need.

So what is the mechanics to deliver a service? Yaniv notes they are precisely the same as those you use to deliver digital products - by taking an agile-based development approach with continuous improvement.

Yaniv states that IT organizations should take an iterative, continuous improvement approach for the service delivery, start small, and evolve as they go. He briefly describes the lifecycle to make this practical:

• Select your critical value streams: After analyzing the value streams, identify 2-3 services to improve. A value stream can be a knowledge base service, a quality management service, a security testing service, or any other critical IT service.
• Create a Minimal Viable Product (MVP): The purpose of the MVP is to validate the value of the service and reduce unknowns and incubate it with a few consuming teams to get quick feedback. Then, learn and improve the service based on the feedback in short cycles.
• Create a standard model: The model serves as a blueprint that depicts the technology, people (roles and responsibilities), and the process needed to create a standard service system. Then instantiate the technology and the automation around it to ensure you can make it consumable, fulfillable and operational at a good enough level.
• Go live: Make your improved services globally available by publishing them to your consumers and placing the required infrastructures to scale them to meet demands. Then, measure and monitor your critical performance metrics and optimize accordingly. Finally, harvest learnings for the next iteration.
• Rinse and repeat: After completing your first improvement cycle, identify the next 2-3 services and cycle through this iterative, continuous improvement process again and again. Each cycle improves people, processes, and technology.

For enterprise organizations that need to transform by implementing DevOps, the Software Factory approach helps set a transformation plan to fit their DNA by starting small without over-engineering and allowing continuous improvement.

You can learn more about the Software Factory Approach recommended by Micro Focus at the following link: https://www.microfocus.com/en-us/digital-transformation/our-perspective/software-factory.

Interview with Allan Wagner

Allan "Al" Wagner is the Transformation Advisor/DevOps enthusiast within HCL Software's DevOps Advisory and Adoption group. He and I met virtually on several occasions and found an immediate affinity in our views on VSM and DevOps. He was kind to contribute to this section in an interview I conducted with him. We started by discussing the major pitfalls in DevOps implementations, which center on spending money in IT without seeing a return in quantifiable business value. Let's see what he has to say.

Spending money without verifiable results

When I asked Al what he viewed as pitfalls to DevOps implementations, he immediately came up with a specific and common use case. Specifically, as HCL's VSM and DevOps evangelist, he often hears their prospective clients note how their IT managers and executives are tired of spending money on IT, without showing value across the rest of the organization.

I submit that the issue is one I've addressed several times in this book. Without proper alignment, organizations can spend a lot of money, resources, and time implementing mature software delivery pipelines. Moreover, as Pramod noted in his interview, DevOps is not easy to learn and not easy to implement. Left to their own devices, many, if not most, software development teams, lack the support, funding, or even the skills to integrate, automate, and orchestrate software development pipelines.

Frankly, such work is not even their responsibility. Software developers produce value by delivering software products, not building software delivery platforms and pipelines.

Failing safely

Al noted that a major pitfall of implementing DevOps capabilities is that significant cultural and organizational change is required. He advocates that organizations should make incremental changes so that they can fail safely. IT organizations, and those that depend on their services, must understand that not everything will work out as planned. But to improve our chances for success, we should also spend some time and effort planning!

Managing through mandates

Al and I spoke about how DevOps programs often come down as mandates. The problems with such mandates are multifold. First, executive-level expectations may be out of line with reality. As a result, there is too often no pre-planning and inadequate funding for tools, platforms, and training. It's hard to properly identify budgets if there is no planning to determine the needs and requirements of the organization.

Also, without planning and building prototype DevSecOps platforms, we leave the work to every software development team to figure out – regardless of whether they have the time and skills to work on the issues. In the meantime, their software delivery work suffers.

And, even if the software development teams get past the platform development issues, there is likely no budget available to execute the implementations – certainly not on a large enterprise scale. The desired tools may not even be approved for use within the organization, affecting all security and risk-conscious organizations where compliance and governance are critical issues. Unplanned costs are also a factor. A mandate without an execution plan, a forecast budget, and the means to connect investments to value deliveries are useless. Everyone involved is set up to fail, which leads to frustration, more work, missed expectations, and, ultimately, mistrust.

In my previous book, Scaling Scrum Across Modern Enterprises, I told the story about my track coach, Al Cantello, at the US Naval Academy. Every day for 4 years, I heard him make the same statement over and over again: "All you young whippersnappers! All you want is instant gratification!" His point was that nothing worthwhile comes easily or quickly. As a two-time all-American javelin student-athlete and a former Olympian – who once held every national and international record in the javelin – he had every right to call us out when we needed to refocus our efforts.

My point in relating this story is that the same instructions apply to business. We need to understand our objectives, define and set measurable goals, plan how we will achieve our goals, and then work hard every day to achieve them. Mandates are useless without planning, prototyping (experimentation), budgets, and resources linked to specific objectives and key results. Moreover, the OKRs come first to guide our plans, experiments, budgets, allocation of resources, and work.

Getting it wrong

Here is the common scenario that Al Wagner cites. Over the years, organizations have been building out and funding full year or multiyear software development plans, only to be disappointed when the development team fails to deliver under the constraints of the project. At the start of the fiscal year, engineering teams may submit a budget estimate for each quarter or perhaps their full-year development plan. Through negotiation, an amount of funding would be approved by the business. But an even worse scenario is when the executives and managers dictate schedule, budget, and resources constraints solely on ROI considerations not informed by the realities of the work.

Still, engineering departments may gladly accept the constraints and work to develop the requested new feature or functionality, perhaps delivering updates once a quarter – or perhaps once over the annualized project budget. Eventually, stakeholders would have an opportunity to review and check out what engineering had built, only to realize that they didn't get what they wanted. Or, the demands of the business changed, and these changes were not communicated to development. Regardless of the outcome, the situation was always the same – a lack of business value and return on investment.

Agile practices emerged to align development efforts to support the evolving needs and priorities of the business and its customers. Specifically, Agile addressed the problems associated with the traditional project-based development model. Later, CI/CD and DevOps pipeline strategies evolved to deliver software products more rapidly and efficiently.

Agile development corrected the project-based software delivery model by having teams deliver earlier and continuously, embrace change, and deliver more frequently with closer alignment to the business. And to this day, those things still matter. What has changed or improved is an increased focus on delivering business value, with teams working to become more efficient and basing decisions on data, from ideation to implementation and beyond. This is the promise of CI/CD and DevOps-based software delivery pipelines.

Early on, as organizations began their DevOps journey, they may have empowered individual software development teams to select automated solutions to build and deploy their individual delivery pipelines. And while the individual teams probably loved the autonomy – it's an Agile thing – this approach created a bit of a mess, for all the reasons noted previously.

But there's another problem also at work. It is challenging for business leaders to make business decisions based on data when the engineering and product data resides in several different repositories and different formats.

Al notes that it is probably unrealistic to mandate that each product or project team use the same toolset across their delivery pipelines, since there are many factors that will determine the best tool or solution for the job. And, as noted previously in this chapter, we can't discount the impact that software developers and operations staff have on promoting their preferred methods and tools.

But organizations may be able to find a compromise by allowing teams to choose the best solution for the task at hand – provided there is integration with a solution (or solutions) that provides a holistic view or dashboard. This is what modern VSM with DevOps platforms offer – integrating tools across DevOps toolchains with a common and normalized data model, thus providing end-to-end and real-time visibility of software delivery pipeline flows.

Finding your sources of truth

Whether an organization chooses to construct its own dashboard or purchase a COTS-based VSM solution, there are several sources of truth that we need to access when hosting the data needed to populate a dashboard and make accurate business decisions. Al lists these sources of truth as follows:

• Enterprise planning: A shared enterprise planning tool that provides a view of all development activity across the organization.
• Version control: This implements a single version control system across the organization for storing all assets used in developing, building, testing, and deploying software.
• Artifact repository: A single location for all deployable artifacts.
• Release orchestration: A single source of data that orchestrates the automated deployment of software releases. It also offers an inventory view of the released versions and locations of deployed software.
• Quality management: A single source of data that provides details on the level of quality. Moreover, this capability supports traceability to link requirements to test cases, the executed tests, and the related test results.
• Event management: A single repository to holistically manage and coordinate the resolution of incidents across the enterprise.

Al notes that this approach to integrating disparate sources of truth still provides individual software delivery teams with flexibility around tool choice. But this strategy only works when there is an integration platform to link the shared data repositories that business leaders depend on to make informed business decisions.

Creating an acceptance strategy

These issues of lack of trust did not emerge overnight – they have been long in the making. We can't trivialize or shorten the process of breaking down the organizational barriers that formed over time. Moreover, organizations managed software product deliveries for too long as projects were constrained by uninformed budgets, schedules, and resources. This, too, sets up the IT department for failure.

Lean production processes that create continuous and predictable flows offer the best way to improve our way of delivering software value. But Lean also dramatically changes the way we work. It changes how we make decisions and measure progress. It also changes our tools and requires skills.

As Pramod noted in the previous section, we need our executive leaders to provide the resources that will help guide the organization through these changes. For example, the organization's leaders must find the resources and budgets to install the following:

• COE
• Coaches and mentors
• Thought leaders
• Change advocates and innovators
• Prototype engagements
• VSM and DevOps toolchain procurements

Without these resources and a determined effort, any enterprise-scale VSM and DevOps platform implementation exercise is doomed to fail. And, as Al puts it, "When bad things happen to good people, they will go back to what they know."

Moving forward

Al presents some common-sense steps for moving forward to a more desirable future state that's possible through VSM and DevOps methods and tools, as shown in the following list:

• Build time into weekly schedules for learning.
• Crawl/walk/run to make incremental steps toward success.

  Al states: "You may have to slow down to go faster." For example, you must allocate time, effort, and resources to address issues associated with technical debt.

• Don't start a VSM or DevOps platform implementation initiative without executive support.
• Disconnect from project and program management-based release schedules:

  a. PM-based schedules are driven by monetary and ROI considerations that are disconnected from the day-to-day realities of supporting evolving market changes, customer needs, and customer priorities.

  b. The PMO cannot be expected to understand the dev teams' capacities and constraints.

  c. The team lead needs to be involved in advance while developing the schedule and negotiate with product management on how they should proceed; for example:

  A. Do we need more people?

  B. Is everything a priority?

  C. What WIP limits should we establish in our pipelines?

  D. Evaluate throughput as a flow to establish burndown rates.

  E. Establish mechanisms to collaborate better within the team and with our internal or external customers.

This list provides a starting point for moving forward to a successful VSM and DevOps tools implementation. But none of this matters if we don't also recognize that the investments must drive value.

Adding value

A basic reality in business is that there are almost always more things to work on than the organization has the time, money, and resources to take on. So, fundamentally, what's more important is to understand our priorities in terms of deciding what adds the most value in terms of our customers and the organization's OKRs. So, focus on what delivers the most value for the investments, and then use flow to deliver the value incrementally.

Al puts it this way: "Focus on products and continuously delivering incremental change that delivers value to the business." He also notes that "Happier people are delivering higher quality software more frequently to the end user." Al also stated that happier working environments eliminate retention issues as people tend to prefer to work in highly productive development organizations.

We'll come back to the topic of adding value in Chapter 16, Transforming Businesses with VSM and DevOps. In fact, using VSM to drive software delivery value across all organizational value streams is the central theme of this book. For now, let's move on and look at four potential DevOps platform implementation strategies.

Jumping onto the cloud-native bandwagon

As a final point, Al Wagner made an important observation that many organizations seem to be jumping onto the cloud-native bandwagon. Cloud-native environments offer tremendous flexibility for leveraging the resources of Infrastructure as a Service (IaaS) providers as a fee-based and pay-per-use service that offers continuous integration, container engines, and cloud orchestration capabilities.

However, Al also makes this critical point: "The legacy systems being used by large enterprise corporations– built with years and years of unique capabilities – are not going away any time soon. The mainframe is not going away; it is not dying. The reason for this is because modernizing these legacy system and application capabilities has a considerable cost. [The organization must ask] is there an ROI to justify the rewrites and migrations?"

We'll discuss this issue again later in the Deciding on the appropriate DevOps platform strategy Subsection. Specifically, it may make sense to operate a hybrid-cloud environment over an extended period.

Interview with Joel Kruger

Joel Kruger is a DevSecOps engineer and AWS solutions architect with 10 years of experience building CI/CD pipelines in commercial and federal sectors. He is also an expert in employing container orchestration systems for automating computer application deployments at scale. In his current role, he is constructing reusable CI/CD pipeline configurations for a large federal agency with hundreds of software product teams. Joel is a proponent of building CI/CD pipeline configurations as downloadable and self-serve software factories. It is from this perspective that he discusses potential DevSecOps implementation pitfalls and platform implementation strategies.

Leveraging reusable configurations

Joel notes that configuration management (CM) is not a new concept and is a necessary set of practices, policies, and repositories to maintain a product across releases over its life cycle.

Joel notes the idea of CM comes from other institutions, such as the military, and that the software community borrowed those ideas and retrofitted them to protect our organization's IT assets. Joel also points out that CM was traditionally a purely manual task, usually completed by a system administrator or junior developer. But, in smaller projects, the CM task can fall to more senior developers.

Regardless, the CM role involved a lot of manual work to document the system's state carefully. But those days are rapidly disappearing, as the industry is moving to implement Configuration as Code. It's not that CM is going away; instead, we can reduce the amount of non-value-adding work involved to capture, maintain, and use certain information. These changes came from the need to accelerate configuration tasks across CI/CD and DevOps pipelines, support cloud-based computing environments, and implement new API-based automation tooling.

However, sometimes, CM is confused with version control. There are some similarities in that both activities track versions of products, product components, and information artifacts as they evolve. The primary difference is that version control identifies individual components as they change, irrespective of whether they are included in a production release. In contrast, CM processes track the versions of all the software and infrastructure components, as well as other information artifacts related to each release of a product across its life cycle.

We've already discussed how IaC is the practice of ensuring all provisioned infrastructure is automated through code. But a supplementary purpose of IaC is to establish a written record of the services that exist, where they are located, and under what circumstance they are deployed.

CM may seem like an overly bureaucratic and non-value-added task. However, that's not true because the organization must protect its IT assets over their respective life cycles. So, it's not that its CM that is non-value-added; it's the manual processes we have to employ to record, save, and relate code and other artifacts to software releases that are burdensome. Therefore, organizations may find it easier to document the use of code and other artifacts via IaC, to maintain a full record of all the technology assets a company owns and has deployed.

Joel points out that as a software product matures, not all its components and information artifacts change with each release. However, things can get out of sync very quickly from a versioning perspective. Therefore, CM is necessary to maintain the product's performance, functionality, and physical attributes associated with the unique set of requirements, design, and operational information specified for each release.

While CM is not a new concept, what is relatively new is automating the deployment of new releases through configurations implemented as code, in the form of Infrastructure as Code (IaC) and Configuration as Code (CaC). Both IaC and CaC fall under modern configuration management practices, and both use scripting languages to automate configurations across disparate environments. But the two terms have different contextual use meanings.

Let's take a look:

• IaC: This approach is used to define your IT infrastructure, such as networks, servers, load balancing, and security as a text file (script or definition file), that is checked into version control. The text files serve as the baseline source for creating or updating the specified environments. IaC provides an executable specification in machine-readable syntax and includes steps capable of producing virtualized infrastructure, can be versioned as a release, and tracked in SCM repositories.
• CaC: This defines how your software components interface with each other – by specifying parameters and settings for applications, server processing, and operating systems – that are also managed as configuration files in a repository. As a result, CaC makes it possible to build and test software code changes earlier in the pipeline, to discover and address issues sooner, and improve the quality of each release. As with IaC, CaC implements executable specifications in machine-readable syntax and includes steps to align application infrastructure configurations as a versioned release that's tracked in SCM repositories.

Now that you understand the basics behind IaC and CaC, let's learn what Joel has to say about how and why they are used.

Implementing IaC and CaC resources

Joel starts by noting that organizations should architect their business' IaC and CaC resources modularly via infrastructure functionality. For example, a classic web-based application includes a frontend user interface, backend business logic, and a database. Rather than placing all your configuration code in one file, it's best to break the code into separate files for each application layer. This modular structuring and decoupling strategy makes it easier to switch out components and systems over time.

This will help them better position their engineers to execute new software and infrastructure deployments swiftly, especially if any of the following events happen:

• Changing customer demands for specific tools and features.
• Needing to repurpose existing code snippets for other products or portfolios.
• Untenable increases in cost for previously implemented tools or third-party service providers.
• Rip and replace business mandates.

In practice, IT architectures are far more complex than the simple three-tier architectural model noted previously. Large organizations that contain hundreds of applications must support potentially hundreds or thousands of configuration options that can change over time.

At this point, Joel moved on to speak about using IaC and CaC concepts to improve interoperability.

Maintaining interoperability

Regarding IaC and CaC, Joel notes that you must construct your plans, playbooks, pipelines, and scripts with interoperability in mind. This strategy mitigates the amount of code refactoring and time that is needed to adopt new toolchains. It also minimizes the opportunity for significant disruptions to the rest of the operational infrastructure when those things occur.

Joel suggests the best approach is to encourage your developers to use input parameters, expressed as environment variables and boilerplate code configurations. This way, each component of your solution's portfolio can behave like a software function and be called on in any permutation of complimentary toolchain stacks.

Removing "secrets" from code

Application and systems security is a critical issue across all organizations, or at least it should be. However, Joel notes that he has seen too many developers take shortcuts in these issues by hardcoding passwords and system access inputs within their configuration files. His bottom-line admonishment is "Don't hard code secrets in source control!"

Instead, the much better option is to store them externally as encrypted data in a Privileged Access Management (PAM) tool such as HashiCorp Vault, Akeyless Vault, Thycotic Secret Server, BeyondTrust, or AWS Secrets Manager. Secrets Managers (or PAMs) typically inject sensitive information into container runtimes as environment variables or mount them as volumes. The bottom line is, developers must not make it easy for hackers to see an application's entry points, access codes, and other sensitive information.

Most Secrets Managers incorporate a client process that is intended to be run within microservice containers (Docker, Kubernetes) that can unlock secrets using an API key issued by the Secrets Manager server. In many cases, authorization can be configured at group or individual levels, per secret, or even dynamically for generating time-sensitive, single-use passwords. Joel notes that besides being more secure, parameterizing secrets is a critical step in ensuring that your software configurations are reusable, extensible, and scalable.

Items that IT organizations should specify as secret include usernames, passwords, tokens, API keys, SSH keys, PGP keys, TLS keys, TLS certificates, IP addresses, port numbers, domain names, secure strings, and sensitive files. Also, the IT organization should identify any other digital information that is considered secret or protected by your organization and include it in this list.

Avoiding configuration lockouts

Defining reusable configurations through code reduces the number of experts the organization must find and maintain to support its CI/CD and DevOps platforms. IaC and CaC also reduce the time and effort each software development team must expend to set up their environments, run their tests, and deploy their code. But all those benefits are lost if the configurations become unusable.

With this issue in mind, Joel strongly recommends that developers should not hard code key-value pairs that will prevent their IaC, CaC, or in-house code from becoming reusable. Instead, whenever possible, use inputs at the command line, API calls, or implement environment files that can be committed to source control and consumed by your automation flows.

Like parameterizing secrets, parameterizing crucial key/value pairs in your code is a critical step in ensuring that your software configurations are reusable, extensible, and scalable.

The following list contains key/value pairs items you should parameterize:

• Resource Values: CPU, memory, disk size, VM names
• Cloud Provider Attributes: Regions, availability zones, AMIs, VPCs
• Network Configurations: Subnets, IP addresses, domain names, DNS resolvers
• Application Configurations: App name, port numbers, versions, dependencies
• Container Orchestration: Container registries, image names, image tags

Now that we've covered how to hard code secrets while avoiding configuration lockouts, let's look at what Joel has to say about the need for frequent releases.

Encouraging frequent releases

Joel cited a frequently identified objective for CI/CD and DevOpsOps pipeline builds: "If the master code branch of a product can't be successfully deployed to a production environment within 20 minutes notice, you aren't doing DevOps correctly. Only stable deployments that have been thoroughly tested to meet customer specifications count here."

In other words, Joel encourages development teams to produce smaller units of improvement to the applications frequently. Joel recommends that product release processes be fully automated with push-button type implementations or as close to it as possible to accomplish this goal. He goes on to state that, with contemporary CI/CD tools, it is possible to build this capability, even in many legacy environments that still have mainframes kicking around.

Adhering to a rapid build and release philosophy ensures that your customers will experience a constant flow of desired improvements aligned with their needs. This strategy improves the customer experience, even when your products or services don't contain all the features that your customers desire yet.

The objective is to make sure each new release is incrementally more closely aligned with your customer's expectations, and never less. Just as importantly, this strategy takes tremendous pressure off your organization or business unit to deploy new features before they are ready. No IT organization wants to must roll back their releases because they weren't ready for production.

Instead, with fast build and deploy capabilities installed, customers are left with tangible evidence that their interests are constantly being considered before they lose interest in your products.

With frequent releases, often going out over different production environments, we need to make sure we build and deploy the right configurations with each release.

Configuring the right DevOps platform

Joel claims that there is no way around creating and maintaining scripts or configurations that utilize API-based infrastructure deployments. This statement is true regardless of whether your organization implements a SaaS or is operating on-premises or in the cloud DevOps platform.

Joel further notes that DevOps platform strategies fall into the following three broad categories:

• Declarative configurations
• Programmatic self-service SaaS/PaaS tools
• A combination of both

The benefit of using the declarative configurations strategy is that all IaC, CaC, and parameters can be stored and audited together in source control management. This strategy provides ease of change management and gives your organization a solid position in the event of a security audit.

Popular in the government and financial sectors, this strategy is often used when an enterprise forms a centralized BizDevOps team to author approved IaC, CaC, and automation templates that all development teams must standardize around. The benefit of this approach is that individual teams have more freedom to create and manage their operational infrastructure.

The downside is that automated pipelines won't scale as nicely as the self-service PaaS/SaaS approach. Many sprawling code repositories will likely be created, and each one will need to be tracked and maintained, along with their associated CI/CD pipelines and other artifacts.

On the other hand, Joel notes that the programmatic self-service SaaS/PaaS strategy is optimal if your organization prefers to standardize business processes around software kiosks, API-driven web services, ephemeral deployments, and third-party vendors. With this approach, infrastructure can be programmatically assembled, provisioned, and configured to suit various workloads on demand. Once each service's workload has been executed, any resulting artifacts and metadata get pushed to persistent storage, and the infrastructure that was previously utilized gets re-provisioned for use by others.

The benefit of this approach is that business processes can scale to demand and with less administrative overhead. The downside is that funneling activities into standardized processes can feel inflexible or restrictive to development teams. Another challenge with this strategy is tracing changes in governance and accountability.

Of course, Joel notes that it's never quite that cut and dry from a DevOps platform decision-making perspective. There are situations where governance and accountability take precedence over scalability. There may be other situations where transaction speed and velocity are more critical than maintaining extensive configurations for audit purposes in the same organization. So, large organizations should be prepared to maintain various DevOps platform solutions and configurations to support their disparate business needs.

Adopting software factory strategies

When federated effectively, an enterprise can decentralize to some extent by allowing individual CI/CD and DevOps pipeline development teams to build and maintain scalable services, that can then be consumed by other teams in the rest of the enterprise. This strategy is economical and leads to a culture of cross-functional collaboration, shared responsibility, and value stream productivity improvements. Joel notes that one very effective method for bringing these principles to life is developing software factories.

A software factory is a structured collection of related software assets that aid in producing computer software applications or software components according to specific, externally defined end user requirements through an assembly process. Leveraging IaC and CaC capabilities, the software factory concept enables rapid and automated infrastructure and application infrastructure deployments.

Joel's take is that organizations can extend these concepts to include business development and security operations applications. Essentially, a software factory applies traditional manufacturing techniques and principles to software development, infrastructure deployment, and business operations.

For those interested in exploring the technical details of implementing software factories and automation techniques, Joel provides more details on his website, dynamicVSM.com. This website serves as a convergence for DevOps and VSM technical tips and includes code snippets, cloneable demos, educational exercises, related information, and industry news.

Joel believes the adoption of software factories offers the most efficient solution to implementing Lean value stream fundamentals across your organization's DevOps platforms. The rapid software delivery capabilities of DevOps, when aligned to support business strategies and objectives, helps streamline all organizational value stream activities, which can significantly increase user productivity. Moreover, employing software factories with reusable configurations can help the IT development-oriented value streams establish standard and consistent user interfaces across all business applications, reducing the need for end user training.

Joel notes that businesses need to evolve to support emergent business, market, and customer needs. Likewise, the organization's business applications must evolve in lockstep to implement new process changes. The ease of deploying new and updated functionality, plus flexible user interfaces through reusable and extendable software factories, allows the organization's business applications to evolve to support emergent needs and assist end users in performing their tasks to implement the new business workflows.

We've reached the end of our interviews with DevOps implementation experts. In these interviews, you learned what the experts think regarding DevOps implementation pitfalls and platform deployment strategies. In the next section, you will learn about four common DevOps platform implementation strategies.

Deciding on an appropriate DevOps platform strategy

Now that you know some of the potential DevOps implementation pitfalls, we'll start learning about four optional DevOps implementation strategies. The platforms for deploying DevOps capabilities have evolved over time, and some are more common today than others.

In this section, you will learn about the four common DevOps implementation strategies, as well as their potential applications.

These four implementation strategies include the following:

• Building a custom DevOps toolchain
• Purchasing DevOps as a Service/DevSecOps as a Service (DaaS)
• VSM tools-based DevOps platform integration and orchestration solution
• Developing DevOps pipeline configurations as reusable software factories

Each option has its own set of pros and cons, which we'll explore in this chapter. We'll start our discussion by looking at building a custom DevOps platform solution option.

Building a custom DevOps platform

As this subsection's title implies, this approach involves internal DevOps engineering teams doing the work to integrate, automate, and orchestrate tools to form custom DevOps platforms. This was the only approach available in the early days of DevOps. You can liken this strategy, by analogy, to building a custom race car, plus the tooling to build it. Or, as another analogy, purchasing and building the equipment for your manufacturing facility and installing it. So, you end up owning both sets of problems – building the tools for your software delivery platform and building the platform.

There is no practical justification to do this today. There is no technical requirement that makes this an effective strategy. Instead, the sole motivations for going this route today are empire building and job security. Also, independent software development teams may go down this route on their own when they cannot secure corporate sponsorship and funding.

Al Wagner likes to refer to this strategy as the gift that keeps on taking. Organizations that go down this route fail to look at the downside of taking on the custom integrations, licensing, support, and maintenance issues. After a while, the DevOps engineering team spends more and more time maintaining the configurations and less time building software products. Moreover, if your DevOps engineering experts leave, the company is in a boatload of trouble!

Now, let's go to the other end of the spectrum to discuss the use of procuring a DevOps platform as a service, where all the tools to install a software delivery pipeline are made available and integrated.

Purchasing DaaS

This strategy involves purchasing an already integrated platform that includes an end-to-end toolchain to support software deliveries. Examples include Amazon Web Services (AWS), GitLab, Microsoft Azure DevOps, and a host of independent companies that integrate third-party DevOps tools and offer them on the cloud as a DaaS product.

Vendor lock-in is the biggest downside to this approach, and their platforms may not support the tools your organization needs. On the other hand, virtually all DaaS vendors allow you to integrate other tools to create custom solutions. But what if your organization goes all in with customizations? In that case, they own the problems associated with integrating, automating, and orchestrating the disparate tools employed to develop the DevOps capabilities that run your business.

Moreover, your data is living inside someone else's environment. We all need to ask ourselves what happens if their or your systems go down. Software runs our digital world. Everything we do in business has a dependency on software. So, not having control over your data and application development pipelines has the potential for catastrophic consequences if they are hacked or go down.

Also, what happens if your DaaS vendor's tools and platform offerings become prohibitively expensive? Are you locked in and bound by an agreement that is financially painful to break? A journey to the cloud is very expensive and a multi-year project. Do you have the ROI to justify a DaaS rollout? And can you afford to lose or make up the DevOps implementation investments if you must make a change to another DaaS vendor?

Here's another concern. Let's assume you have an organization that's already made investments in DevOps tools and platforms. If your company is considering a mandate to move to a DaaS solution provider, the leaders and COE need to evaluate the company's ability to make it through the migration.

What if what you're doing isn't broke? The organization's executives need to take a deep look at what they are attempting to fix. Frequently, the executives are hoping to avoid DevOps tool investments and toolchain integration costs. And, they may have concerns about long-term maintenance costs and supportability issues. They are not wrong to have those concerns. But the organization must conduct a thorough investigation/analysis of alternatives before jumping into a major change of this magnitude. Also, don't force a mandate overnight. Take the time to plan the migration, including prototyping several rollouts before mass migration.

By the way, it's not that employing a cloud-based DaaS solution is a wrong decision. But, in many cases, a large-scale or enterprise migration to a platform-based tool suite may cost time and be resource prohibitive. We also need to consider the impact of a DevOps migration on our customers. Can we sustain our deliveries without loss of service through the migrations? How will we make sure we can quickly roll back any failed migrations?

DevOps is not usually about solving intractable technology problems – it's about solving a business problem. So, we need to approach migration to a DaaS-based solution as a business decision.

Al Wagner made an important observation that's relevant to this section: "The mainframe is not dying; they are not going away. The legacy systems – built with years and years of unique capabilities – are not going away. Duplicating their capabilities is a huge cost. (The organization must ask) is there an ROI to justify the rewrites and migrations?"

So, we've viewed the two extreme positions: one is to build your own DevOps platform solution, while the other is to purchase a fully enabled DaaS-based solution. Now, let's look at a couple of alternatives that lie in the middle: employing VSM tools and building reusable software delivery factories. First, we'll start with the VSM-based DevOps platform solution option.

Employing VSM tools

We can use VSM tools to assist with the Integrate, Automate, and Orchestrate capabilities of disparate tools. This comes back to the issues of integrating disparate toolchains and importing data as your sources of truth for decision-making; that is, business decisions should be made based on data in the six sources of truth: enterprise planning, version control, artifact repository, release orchestration, quality management, and event management. Whatever tools you have deployed, you need to integrate data into your data repository.

Use the end-to-end data to make decisions. We don't really care what DevOps tools you use for any given particular task, but you need to access the data that's maintained in that tool. So, it would help to have the tools communicate back to a source of truth with a common and normalized data model for analysis and decision making. It's the integrations back to the primary data source that are doing all the hard work. The downside is that the VSM tool may not have all the necessary integration adaptors and connectors. So, you may need to own the integration problem (via custom APIs) to address those issues or leverage a DevOps tools vendor's plugin framework.

Building reusable software factories

Create internal teams or use an external software integration team to develop software factories that build self-service CAC that can be downloaded via Git or other SCM tools. The definitions for each of these three categories are as follows:

• Infrastructure as Code (IaC): Such as a terraform standup.
• Pipeline as Code (PaC): The whole, seamless end-to-end integration, automation, and orchestration, which is done through a script upon a commit. This triggers a holistic set of API commands through Jenkins or Ansible. The whole thing is code.
• Self Healing Pipeline: Any errors in execution (Jenkins or other automation tools) would trigger a machine-readable error and use utilities and automated runbooks to correct and rerun the failed step.

In other words, if the DevOps team sees an automation failure occur and fixes it, then the team triages the problem to determine its root causes and to build a fix. Next, they code the fix as an automated runbook that operates within the Jenkins file to resolve that type of error, should it ever happen again in the future. (This is what we mean by self-healing).

This subsection completes our discussion on how to decide which DevOps platform implementation strategy is best suited for your organization's needs. You've learned about four basic strategies here. Now, we are going to discuss what you might do if your organization's chief executives mandate the adoption of DevOps capabilities without proper planning, preparation, and budgeting.

Dealing with corporate implementation mandates

DevOps is such a critical business enabler that chief and line of business executives may push the transition via a corporate mandate without taking sufficient time to plan, prepare, and budget the initiative to support an enterprise-scale deployment. When this occurs, the best DevOps implementation strategy is to go with DevSecOps as a Service (DaaS). The IT department does not need to implement DevOps tools and maintain integrated toolchains to get started. And, when driven by corporate mandate, the decision-makers may not be aware of the technical implementation options, tools and toolchain alternatives, configuration and integration requirements, costs, and other issues.

DaaS is a multitenancy implementation concept. The term software multitenancy refers to a software architecture in which the software runs on at least one server, but more often on many virtualized servers and serves multiple tenants (customers). DaaS examples include Azure DevOps Services, GitLab, and AWS CodeDeploy. In addition, large organizations that need to deploy microservice-based applications across multiple environments might consider using Azure Kubernetes Service.

Critically, DaaS-based developers do not need to write IaC since the DaaS platforms already include an integrated toolchain. Most DaaS solutions are extensible, allowing software teams to add other tools as customizations to their essential platform services. If a delivery team wants to create a custom DevOps platform, they will need to have or hire experts in IaC tools such as Ansible and Terraform.

There are other considerations that can support rapidly deploying DevOps when mandated. For example, use a user-centered design (UCD) process to focus on the features and capabilities that will help the business. These tools allow customers and users to see what they want before you develop the code. As a result, the UCD process occurs before you write a single line of code!

Developers can use wireframe tools for UCD such as Adobe InDesign, AXURE, and Balsamic to create a visual display for business workflows. This is a great capability because it helps streamline the Fuzzy Frontend of development. UCD concepts also dramatically reduce design time and rework that results from defects in the code. Defects are misunderstood or previously unidentified requirements that your customers felt you should have known about and implemented.

Automated testing is a critical capability in CI/CD pipelines. Teams can use tools such as Leapwork (https://www.leapwork.com/) to implement a low code/no-code solution for automated testing.

VSM tools are even more vital to implement when executives mandate the implementation of DevOps capabilities. Everything tends to be rushed in such situations. After all, the mandate probably came down because the organization is failing to deliver sufficient value to its customers.

But how much worse will things be if the organization fails in its DevOps mandate? Modern VSM tools provide the metrics the software delivery teams need to improve their value-based deliveries and show the improvements they have made. In short, VSM tools help identify bottlenecks, orchestrate work, and improve pipeline efficiencies.

But there are still areas of the software delivery pipeline that are difficult to integrate, automate, and orchestrate. This is the topic of the next section.

Dealing with creative versus repeatable pipeline activities

One of the challenges with improving value stream deliveries is the challenges associated with automating or even estimating the scope of work involved in concept ideation, requirements definition, and analysis. In software development, we refer to this stage as the Fuzzy Frontend.

This term was originally coined by Peter Koen, a professor at the Stevens Institute of Technology. It refers to the value stream activities associated with creating new product ideas and concepts that need to be analyzed to determine customer fit and commercial viability.

Some of these activities are controlled by the product management or product owner function. This is because they alone have the responsibility of deciding what goes into a product and what does not. But the development teams are also involved as they must assess the difficulty and time required to implement each new requirement. In Agile, we refer to assessing requirements as Product Backlog Refinement.

The folks at HCL Software were kind enough to allow me to highlight their view of how the creative aspects of the Fuzzy Frontend fit in the overall software delivery pipeline, as shown in Figure 15.1.

An important feature of this software delivery pipeline diagram is highlighting the creative versus repeatable aspects of software development, which they break out as Agile versus DevOps practices.

Everything to the left of the vertical line is the fuzzy frontend, where creativity is a defining effort that is difficult to estimate. However, everything to the right is the commit and can be automated. So, everything to the left takes creativity, thought, design thinking, and so on, while everything to the right becomes part of an automated software delivery factory!

Figure 15.1 – Dealing with creative versus repeatable DevOps pipeline activities

Al Wagner articulates a DevOps implementation scenario as involving three phases, which he refers to as Day Zero, Day One, and Day Two. Let's understand each of them:

• Day Zero (to the left of code commit): Agile implementations of Agile practices governing Sprint planning, Sprint execution, releases, commits, and so on. This is the creative, Fuzzy Frontend side of development that is difficult to estimate and automate. Yet, these are the activities that keep our focus on the customer.
• Day One (to the right of code commit): This involves integrating, automating, and orchestrating DevOps pipeline activities. Here, the IID model is taken and made Lean through integration and automation toolchain capabilities.
• Day Two: This is the state we want to reach. That's the application of VSM – to focus on meeting business needs – which means we need to have a holistic view of both the fuzzy frontend (Agile) and the CI/CD automation processes. It involves constantly monitoring, looking across teams, and constantly improving. Use experimentation to find better ways of doing things. Maybe use AI to assess areas and methods for improvement. Always use lead and cycle times as the actual indicators of improvement.

We can also use VSM techniques across all organizational value streams to determine the high priority/highest value improvement activities that can benefit from DevOp's rapid and efficient software delivery capabilities. The idea is that we can spend a lot of time, money, and resources building our DevOps pipelines, only not to see any real impact on the business if those new capabilities are not applied to the organization's most impactful value stream improvement initiatives. For example, we can use OKRs and Weighted Shortest Job First (WSJF) (a method used in the Scaled Agile Framework (SAFe)) to help teams prioritize a list of initiatives.

At this point, you have heard differing views on the potential pitfalls of DevOps implementations and recommendations on how to address the issues you might face. You've also learned about four different DevOps platform implementation strategies and the pros and cons of each. You then learned about the best approach to take when the organization's executive mandates the use of DevOps, and you've also learned why it is easier to automate the backend of the software delivery pipeline but not the fuzzy frontend. Now, we are going to dive into how to address a host of potential DevOps platform implementation issues.

Addressing the pitfalls of DevOps

In this section, you will learn strategies for addressing 18 scenarios that can present a challenge to your DevOps implementation plans. The list is not meant to be an all-inclusive discussion on situations you may face, but it is reasonably comprehensive. As you read through these subsections, begin to think about how you can apply the VSM concepts, methods, and tools you've learned about to discover issues and their root causes, as well as how to improve them.

Let's begin by recognizing and avoiding the consequences of Conway's Law.

Avoiding the pitfalls of Conway's Law

In 1968, Melvin E. Conway wrote an article for the Harvard Business Review (HBR) titled How Do Committees Invent? Unfortunately, HBR rejected it on the grounds that he had not proved his thesis (http://www.melconway.com/Home/Conways_Law.html). However, Conway persisted and submitted the paper to Datamation, who published it in April 1968. His persistence paid off, as one of his observations in the papers was that organizations tend to make systems (intentionally, broadly defined by Conway) that mirror the communications structures within the organization.

Here is the quote, verbatim:

Conway's Law (http://www.melconway.com/Home/Conways_Law.html):

"Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure."

— Melvin E. Conway

Though Conway made his observation years ago, it is no less relevant today, including its application to systems that deliver software; even modern DevOps-based software delivery systems. So, let's use what we've learned about lean production systems to understand the consequences of Conway's observation.

Value stream management methods and tools help the organization streamline its value streams by eliminating waste and improving work and information flows. However, if the DevOps team is called to create an application that supports a critical business process, it would not make sense to construct its logic to mimic how it operates today. Instead, the business application needs to support the work and information flow improvements identified through a VSM initiative.

Conway's Law also applies directly to the software delivery team as a system. In fact, those are the most typical use cases when describing the impact of Conway's Law. Put most succinctly, any software application that's constructed will have as many layers or modules as there are discrete organizations participating in its build. In effect, the software will mirror how the teams separate their activities and communications.

AlanMacCormack, Carliss Baldwin, and John Rusnak discussed their findings relevant to Conway's Law in their 2012 HBR article titled Exploring the Duality Between Product and Organizational Architectures: A Test of the 'mirroring' Hypothesis. In their own words, they found the following correlation: "The mirroring hypothesis predicts that these different organizational forms will produce products with distinctly different architectures. Specifically, loosely coupled organizations will develop more modular designs than tightly coupled organizations."

Modern CI/CD and DevOps-based software delivery teams employ microservices, containers, and APIs, all of which support loosely coupled applications. Loosely coupled services are reusable and interchangeable and implemented without breaking existing interconnections.

Conway's Law describes the impacts organizational structures can have on software development. For example, large product teams that work as a disjointed collection of entities will create bottlenecks in the flow of their work, communications, and information flows. Looking at software delivery as a value stream, we know we need to eliminate these artificial barriers by aligning the teams horizontally around end-to-end value delivery flows. And we must also eliminate the waste that prevents continuous and synchronized flows.

Implementing a CALMS framework

Damon Edwards and John Willis initially coined the acronym CAMS, to stand for Culture, Automation, Measurement and Sharing. Later, Jez Humble added an L, to include Lean in the acronym, to make it CALMS. The CALMS Framework is a conceptual model for integrating DevOps teams, their activities, and the employment of their systems, tools, and toolchains. The elements within the CALMS acronym are further defined as follows:

• Culture: Requires a switch from command and control and hierarchical organizational business structures to collaborative work environments with shared responsibilities and organization around horizontal flows to support value-based deliveries.

  A DevOps culture welcomes change, uses feedback and information (metrics) to improve continuously, and is accountable for their work as a team. As with Lean, DevOps culture pushes decision-making as much as possible to those who perform or are responsible for the work.

• Automation: In DevOps, transformation includes automating manual tasks to the greatest extent possible. For example, recall the CI and test automation activities in the Automated Delivery Pipeline on the right of Figure 15.1.

  A DevOps-based software delivery pipeline eliminates repetitive and non-value-added manual work. IaC and CaC capabilities produce repeatable and high-quality processes and reliable systems. In fact, everything as code is the mindset behind DevOps automation, regardless of the tools you employ. Test automation (packaging builds and promoting them through test environments using automated deployments) implements CD capabilities.

• Lean: This involves making continuous improvements to streamline flows by eliminating waste that creates delays, waiting, and excess WIP. Lean seeks to achieve FLOW – smoothly transitioning work from one work center to the next – in the shortest amount of time, ideally with as few queues/buffers as possible. This is what this book is fundamentally about.
• Measurement: We can't fix something if we don't know what the problems are, their root causes, and what things most impact our ability to deliver value rapidly, frequently, and with high quality. For that, we need metrics and analytics. Identify your most critical Lean metrics, starting with the DORA Four: lead time (for change), change frequency, change failure rate, and mean time to repair (MTTR).
• Sharing: DevOps is fundamentally a collaboration strategy that aligns Dev and Ops activities. To be successful, Dev and Ops cannot be stovepipe silos. Instead, they must find common ground, and that common ground forms around delivering and sustaining software products customers will value. But collaboration is not enough; they must also share responsibilities for the success of the product across its life cycle.

Deciding on the appropriate DevOps platform strategy

Earlier in this chapter, you learned about four DevOps implementation options: build your own DevOps platform, purchase a DaaS license, use VSM tools, and build software factories. We do not need to cover the pros and cons of those four options again. However, we do need to address a few other issues, such as the use of containers and hybrid-cloud environments.

A container is a type of software that packages code and all its dependencies to deploy applications quickly and reliably, regardless of the target computing environment. However, if your organization is not operating at a very large scale, the use of container orchestration tools such as Kubernetes, Mesos, and others may be overkill. By a very large scale, we're talking about organizations the size of Google, Amazon, Netflix, most branches of government, and other businesses with huge volumes of constantly evolving internet-based services.

If your staff has no familiarity with the container orchestration tool, it can actually end up being a value-losing prospect. For example, even skilled developers find Kubernetes challenging to learn. Therefore, Elastic Container Services (ECS) or self-hosted Docker Swarm Mode are more appropriate for most organizations. Or even the use of traditional virtual machine (VM) environments may be sufficient to support your scaling needs. As a side note, even Kubernetes (also known as K8s) must run on a VM.

If you are committed to using containers, and you have a small shop that somehow managed to hire a couple of super K8s ninjas, they could make it work. But having that kind of talent on-hand in a small dev shop is rare and, likely, not cost-effective for the organization's application deployment needs.

If your team has plans to operate in one of the cloud service providers (AWS, Azure, GCP, and so on) but has no experience with those tools, it may make sense to take a hybrid-cloud approach initially. In other words, use the network and computing resources you have on-premises and slowly migrate to the cloud. This slow-roll strategy makes migration safer until your team formally decides they are ready to cut over to a 100% cloud architecture.

The hybrid-cloud approach does still require investing in new skills. For example, your developers need to learn the APIs and eccentricities of the selected cloud providers (that is, JSON or REST API syntax, Bash or Python for scripting in IaC applications, and automation servers such as Jenkins and Bamboo, which require specialized scripting knowledge).

Regardless of which DevOps platform implementation scenario you choose, the most effective or state-of-the-art tools may be too difficult to maintain if you don't keep a well-staffed team of experts on hand. Also, with more than 400 tools categorized as DevOps component solutions, which ones will you use, and why? Finally, what governance programs do you need to implement to evaluate the tools, handle licensing issues, and make informed budgeting decisions?

Finally, if you face a talent shortage in your workplace, your selection options will be constrained by your team's collective skills and experiences. So, how can you use the techniques of VSM to help your organization work through these issues? Once you have selected a DevOps implementation strategy, how can you use the metrics to make continued improvements to your software delivery flows?

Avoiding mandates

We spoke previously about DaaS being a viable option when the transition to implement DevOps-based software delivery capabilities is forced through executive mandates. The business drivers for such mandates may involve a burning platform situation or realizing that the organization is missing opportunities to compete effectively in our digital economy.

Regardless of the motivation, when supporting digital transformations, architect your DevSecOps solutions so that VSM roadmaps begin with your customer's current workflows and methods. Ripping away a development team's current practices and mandating a radically different one in short order will cause unexpected costs, reduce throughput, and lower team morale.

The bottom line is that people need to buy into change; it can't be mandated. However, suppose you encourage your Dev and Ops team members to participate in VSM value stream mapping and use a Kaizen-based approach to making lean-oriented improvements. In that case, those team members are more likely to buy into whatever change scenarios have been identified and help in the transitions.

Avoiding wasted time

Lean is fundamentally about eliminating waste, including those activities that take time away from our efforts to deliver customer-centric value. Some of the biggest time wasters are meetings instead of work. Yes, we must stay informed, but your organization's leaders must constantly ask whether specific meetings are essential. Are there better ways to keep people informed? And, if a meeting is necessary, they need to ask who really needs to attend.

Keep your daily standup meetings brief and laser-focused on identifying work that's been accomplished, specific issues or impediments, and determining action items. There's a reason they are called Standing meetings. It takes issues offline with only the appropriate people participating. In other words, you should conduct separate meetings for issue resolution.

Avoid long DevOps planning and implementation meetings. Big meetings have become popular, especially in SAFe with its large training classes and program increments (PIs). Large meetings work when they are relevant, informative, and engaging. But large meetings never work for problem solving activities unless you can logically break the group into small teams, and then bring the groups back together to share their findings.

Whatever we do, we must ask ourselves, does this meeting make our team Leaner and more Agile? If you cannot precisely define a customer-oriented deliverable output for the meeting, chances are the meeting is not necessary. A common question to ask is whether your customer would see value in paying for the meeting – because that is the end result of every meeting you hold. In other words, your customers ultimately pay for those meetings through the price of your products and services.

Always look at meetings from a value-added perspective, just like you now know how to do with any other value stream activity. Ultimately, your meetings serve one purpose – to improve organizational value stream deliveries. Otherwise, they become a form of waste from a Lean-oriented perspective.

Eliminating silos and increasing cross-functional team collaboration

We've addressed this subject multiple times throughout this book. But fundamentally, this is what DevOps is about. Instead of having development and operations teams working in silos, we bring them together to collaborate and improve software value delivery.

But the term DevOps is actually limiting from a value-added perspective. This book mentioned DevSecOps, which includes collaborating with the security department to make our networks and applications safe. However, it's becoming more challenging in a digital economy to separate the work that's performed in other organizational and development value streams from the software delivery value streams. So, I would encourage you to use the term BusDevSecOps when referring to VSM initiatives operating as strategic transformations to compete in our digital economy.

The BusDevSecOps approach encourages cross-functional team collaborations to break down the hierarchical silos in the functional organizational model. The goal of BusDevSecOps is to promote the development of horizontal work, material, and information flows to improve the delivery of business value.

Suppose there is one important lesson we've learned in Agile: the volume of people is not a primary indicator of future success and may instead be a source of failure due to the increasing interconnections that come with scale. Instead, people work most efficiently in small teams.

Scaling is accomplished through various team-of-team concepts but best achieved when the teams integrate value streams to support the flow of work. In the end, the diversity of skills, relevant experience, and cross-team collaborations are what contribute most to an organization's success.

Upskilling becomes critical

VSM employs methods, tools, and metrics to improve an organization's value delivery capabilities continuously. We spoke in the previous subsection about the need to have a diverse set of skills in small teams to accomplish all the work that's been assigned. Agile and Lean both employ the concepts of putting all the necessary skills into small, self-directed teams.

Ideally, individual team members have multiple skills to provide more flexibility and fault tolerance. Still, those teams cannot continuously improve the way they work unless they are also continuously learning. Therefore, time and resources need to be made available for this, and the executives must instill the notion of continuous learning as part of the organization's desired culture.

Implementing pre-production testing

Fundamentally, CI/CD is a sequential flow of work activities, from ideation through delivery to customers. Modern software engineering practices that instill CaC, IaC, and test automation make it practical to streamline these activities with minimal, if any, human intervention.

As a result, it's possible to accelerate software delivery cycles by running CI and CD practices straight into production. However, it's usually better to build the CI/CD automation sequence but implement a pre-production stage before release. This approach allows application releases to be monitored in production-like environments before allowing the product to go to the customer. The reason for this is that it's better to find configuration problems in a pre-production environment than to have customers find it and having to roll back the release.

Here, again, the data capture and analytics capabilities of modern VSM tools make it easier to analyze and improve application performance in the pre-production environment, before allowing it to be released into a production environment.

Separating DevOps engineering from DevOps as a practice

DevOps engineers have the responsibility of helping install the integration, automation, and orchestration capabilities across CI/CD and DevOps pipelines. However, within those pipelines, the roles of developers and operations staff persist.

Now, we know that we cannot allow the folks to operate as siloed functions. So, instead, we need Dev and Ops to collaborate to break down the barriers that would otherwise reduce the organization's ability to deliver high-quality products rapidly and frequently, and minimize the potential for failed or suboptimal production releases.

In this context, DevOps engineering helps build automated pipeline capabilities. Still, DevOps, as a practice, is a strategy that brings Dev and Ops functions together to solve product delivery issues, both before and after each product release. Both capabilities are necessary to accelerate and improve value delivery.

Allowing flexibility in DevOps policies and procedures

Large enterprises have many value streams and often many products with unique value propositions. Therefore, IT must offer flexibility to address their customer's needs and ensure they are in tune with the delivery capabilities that bring competitive advantage for each product line. Moreover, requirements, practices, and technologies change over time. What works well today may not tomorrow. And we can always improve.

In the previous chapter, we discussed the importance of establishing a COE to develop governance policies around tools and practices during the initial prototype engagements, oversee the maturity of an organization's DevOps practices, and provide guidance to new teams and existing DevOps teams. The learning curve to acquire these DevOps skills and capabilities is quite steep, and the CoE reduces the impact of the organization's transitions.

Still, the CoE cannot be directive in nature. They fulfill mentoring, coaching, and servant leadership roles. They can help DevOps teams situationally analyze alternatives when the standard practices fall short in meeting internal and external customer needs. They also operate within the CALMS framework to assist the DevOps teams through their transformations and ongoing improvement initiatives. The role of the CoE is not to implement a command and control oversight mechanism that gets in the way of making informed and experiential improvements. The purpose of the CoE is to provide leadership, guidance, and support, but not through directives. They are servant leaders.

Now, let's move on and talk about how quality can be improved even while increasing delivery velocity.

Improving velocity with quality

A common term used among DevOps-oriented Agilists is acceleration. The idea behind the term is that the Lean flows of DevOps help us accelerate the delivery of software value. But the term value in that statement is just as significant as the term accelerate. Increased software delivery speed without improvements to quality only means delivering the wrong product more quickly and efficiently. Still, it's the wrong product – which means we might be going toward bankruptcy more quickly.

This concept applies across the software delivery value stream. When we decide to accelerate flow without putting equal consideration into improving the quality of the delivery, we create an opportunity for increased waste and failures.

CI with automated testing is a critical enabler for improving software quality. But so too are the fuzzy frontend activities associated with gathering requirements, writing associated acceptance criteria, and defining the tests that confirm the completeness or Definition of Done for each constructed work item. We can't get the result right if we don't start right. Defects result from not getting the frontend right, which kills any velocity we might have had.

Building DevOps teams from within

We've noted that DevOps is a skillset that's not developed overnight. However, hiring dedicated resources to create a separate DevOps team is not the best way forward. Building dedicated DevOps teams from scratch will only create new siloes operating across your horizontal value stream.

Recall the issues of implementing horizontal value stream flows across the vertical functional silos in our discussions in Chapter 13, Introducing the VSM-DevOps Practice Leaders, in the Defining values streams in DA FLEX section. Refer to Figure 13.3. Instead, a better approach is to develop teams around value streams. This includes your DevOps engineers, plus existing quality assurance (QA), operations, and development team members. This approach brings your existing people and processes into the work of organizing around value. There's no need to replace them. Instead, we need to bring them along.

Start with the people who enjoy being innovators and early adopters. Help them be successful and build on their success to establish the core foundations of your enterprise DevOps tools, policies, and procedures. Then, use the experience and success of those teams to bring in the early majority and, ultimately, the laggards.

This process does not have to take forever. However, large organizations with multiple product lines and multiple Dev and Ops teams can expect a multi-year journey. Therefore, a 3-year timeline to implement an organizational transformation on this scale is not unrealistic.

Automating database builds

Say what!? When building CI/CD pipelines, we are used to thinking about automating the execution of software builds, the standup and configuration of infrastructures for testing, and then executing the tests. However, we don't often think about how our application databases fit into these scenarios.

In CI, we pull a batch of files from specific branches within our source code repository and then allow the configuration scripts to execute the build and integration test process. The problem is that database code (stateful, order-specific, additive database structures) does not lend itself to merging code across branches. Ideally, database changes get merged into your batches before they are retrieved for processing. Also, database changes must be handled serially, and the order matters.

Data snapshots created in separate Dev environments will drift from each other over time, making it challenging to sync data for each build. Therefore, someone needs to be responsible for automating the databases alongside the application automation configurations.

Maintaining incident handling procedures

No matter how much time and effort we devote to building our automation scripts, something will inevitably go wrong. The purpose of our Agile retrospectives is to blamelessly review issues that occur and identify ways to avoid those failures in the future. And, when the failures are not entirely avoidable, we need to make sure we improve and document our recovery procedures.

The DevOps team needs to maintain a rigorous incident handling procedure to document how the configuration, testing, and deployment failures are handled. The best place to put this information is in the runbook documentation, which may be maintained in a source code repository such as Git or GitHub.

Integrating security with DevOps

The integration of security with DevOps is so important that many organizations refer to their automated software delivery pipelines as DevSecOps. Unfortunately, security is another potential silo within the IT organization. And, just as is true with the operations team, security folks tend to be risk-averse and may therefore be viewed as a bottleneck in the acceleration of software delivery.

However, a bigger mistake is to ignore or go around the security function to avoid delays. In the past year, while this book was written, we have seen first-hand the negative consequences of not paying attention to security. For example, the ransomware hack of the Colonial Pipeline Company shut down 45% of the oil deliveries to the East Coast of the United States for nearly a week. Another well-publicized event with potentially catastrophic consequences is the malware attached by hackers to a software update from SolarWinds. The malware allowed the hackers to monitor the computer networks of nearly 18,000 SolarWinds customers in both the government and the private sector and over four months.

Gaining knowledge of DevOps

It should seem obvious that having knowledge of DevOps is a requirement before initiating a DevOps mandate. But, unfortunately, it's much easier to understand the potential benefits of DevOps than it is to understand the issues involved with implementing DevOps on an enterprise scale.

It goes back to that statement made by my former Navy track coach, Al Cantello, about our wanting instant gratification. By now, you should fully appreciate the difficulties in establishing DevOps at Scale to support a digital business transformation. But it's much easier to dream about a better future state and yet quite another thing to do the work to achieve those desired goals. So we have to put in the work.

And that work has to start with the organization's executives as only they have the authority to drive and fund the transformation on an enterprise initiative.

Getting fatigued while implementing DevOps

The IT industry undergoes constant evolution. It's hard enough for IT specialists to keep up, let alone the rest of the stakeholders in an organization affected by their activities. Moreover, the complexity of DevOps makes it especially frustrating to implement at an enterprise scale.

It's easy to look at a DaaS-based DevOps platform solution and think we can stand up a new DevOps platform virtually instantaneously. At a limited scale, that's true – so long as you have the trained staff in-house who can immediately go to work.

But, on a larger scale, a series of issues will quickly arise, spanning budgets, training, mentoring, coaching, preferred and emergent tool integrations, unique configurations to support legacy applications, security, compliance, licensing, long-term supportability and sustainability, and value stream performance improvements. At times, it may seem like the organization takes two steps forward, only to fall back a step or two while executives and other stakeholders reassess previous decisions.

This is where value stream management comes into play. It's impossible to execute a game plan if you don't have one at the start. Yes, even football coaches make halftime adjustments. But those adjustments are tweaks to the original plans, and the goals and objectives remain the same.

DevOps implementation fatigue is another reason why executive support is so vital to a successful outcome. The organization's chief and line of business executives must keep the momentum going through the hard times. Clear and continuous communication from the expected OKRs is one way they help drive the implementations. Other times, they may need to be cheerleaders and coaches. But, in any event, they need to be informed, involved, and committed.

Coding secrets in source control

Joel Kruger mentioned this topic in his interview. Security must be a critical concern in all software delivery organizations. For example, DevOps-based software delivery systems use IaC and CaC to automate the execution of creating or updating the specified environments and application infrastructure configurations. But these configurations can be a security hole when access information is hardcoded into the instructions. As Joel notes, "it's like posting your name and address and bank account information online."

Another issue with hardcoding secrets is that each variation forces a new and unique configuration file to be constructed. However, instead of maintaining multiple copies of configuration files that do precisely the same thing, developers can use parameterized, reusable scripts alongside environment variables to reduce the sprawl.

For example, developers might write Ansible scripts to configure a server, but then have 50 separate instances to deal with unique but minor variables. It's much better to make one configuration and parameterize it so that development teams can simply change the parameters that affect their configurations.

The parameters code concept applies to configurations, human input, machine-to-machine interactions, web apps, and more. Everything needs to be modularized (open-ended), including IP addresses, hostnames, application names, resource configurations (CPU and RAM), default configuration files, certificates, and tokens.

Note

The need to audit configurations, such as in government and highly secure applications, can lead to hardcoding parameters and integrations within source code and tool scripts. However, tools such as AWS Secrets Manager, for example, allow you to parameterize your variable inputs but securely manage them and still allow auditing by authorized personnel.

This section concludes our chapter on defining the appropriate DevOps platform strategy. The next chapter is the final chapter for this book, where we will take what we've learned to use VSM and DevOps methods and tools to help affect digital business transformations. But before we move on, let's summarize what you've learned in this chapter and check your understanding of the concepts presented.

Summary

In this chapter, you had the chance to hear from five industry experts on their views about potential DevOps implementation pitfalls and various DevOps platform implementation strategies. These five individuals were selected to provide as broad a base of hands-on experiences as possible.

Next, you learned about four fundamental approaches to implementing DevOps platforms. This included building a custom DevOps platform, purchasing a cloud-based DevOps as a Service (DaaS) solution offering, employing VSM tools to integrate and orchestrate your DevOps toolchains, and building software factories by constructing self-service CaC, downloadable via Git or another SCM tool.

Finally, you reviewed 18 implementation scenarios that can create issues for your DevOps platform implementation program. As you read through these scenarios, you were encouraged to identify how you can leverage what you've learned about VSM and DevOps methods, as well as the tools to overcome these issues.

With this information, you are now ready to learn how to employ what you've learned to use VSM and DevOps methods and tools to digitally transform your business, which is the subject of the next and last chapter in this book.

Questions

1. Helen Beal instructs us not to create DevOps teams. What does she think we should do instead?
2. Scott Ambler suggests we can't limit our thinking to just Dev and Ops. What are the six key aspects of enterprise-class DevOps defined in PMI's DA toolkit?
3. Pramod Malhotra suggests we shouldn't even consider initiating a DevOps initiative unless we have what?
4. Al Wagner hears one common complaint from his customers. What is that complaint?
5. Joel Kruger identified Configuration Management (CM) as the means to protect an organization's IT assets. While he notes CM is not a new concept, what is relatively new about CM practices in a CI/CD and DevOps pipeline context?
6. What are the four fundamental DevOps implementation approaches an organization might take?
7. When corporate executives mandate a rapid transition to implement DevOps, what is usually the best strategy?
8. Jez Humble created the CALMS framework as a conceptual model for integrating DevOps teams, their activities, and employing their systems, tools, and toolchains. What does the CALMS acronym stand for?
9. This book suggests that it's better to build your DevOps teams from within to support your organization's value streams. Why is that?
10. Who is ultimately responsible for addressing issues of DevOps implementation fatigue?

Further reading

MacCormack, Alan, Carliss Baldwin, and John Rusnak. 2012. Exploring the Duality Between Product and Organizational Architectures: A Test of the "mirroring" Hypothesis. Research Policy 41 (8) (October): 1309–1324. doi:10.1016/j.respol.2012.04.011. https://dash.harvard.edu/bitstream/handle/1/34403525/maccormack%2Cbaldwin%2Crusnak_exploring-the-duality.pdf. Accessed July 12, 2021.