7

The DevOps 360° Technological Ecosystem as a Service

In this chapter, we discuss the part of the DevOps 360° evolution that many in the industry find the most interesting to work with: the DevOps 360° technological ecosystem as a service. While interesting, the perception of this term, used by the technology and this book, is the source of most DevOps misconceptions. The chapter starts by discussing the value proposition of technology for the DevOps evolution, along with defining what is indicated by the term ecosystem and what the main counterparts of it are, in an incumbent’s context. Continuing, we focus on bringing clarity to the biggest DevOps misunderstanding in the industry, that is, equalizing DevOps and technology, while we provide our view on the real interrelation of the two. Moving on, we touch upon two initiatives that characterize the current DevOps technology approaches of incumbents – one of engineering transformation and one of technology standardization. Continuing, in the heart of the chapter, we look at the DevOps platform teams, which are the main mechanism that incumbents use to enable DevOps capability engineering through technological utilities. We cover their value proposition, operating and service models, product and service catalogs, social contracts, and other important elements. The chapter closes by outlining the five core DevOps platform teams’ domains from a representative sample of incumbents.

In this chapter, we’re going to cover the following main topics:

• The definition of the DevOps 360° technological ecosystem value proposition
• The misunderstood relationship between DevOps and technology
• The six categories of technological ecosystem counterparts
• The focus of incumbents in engineering transformations and technology standardization in enabling DevOps capability engineering
• The DevOps platform teams in terms of operating and service models
• Examples of the five core DevOps platform teams’ domains that incumbents focus on

The technology value proposition in the DevOps evolution

In the previous chapter, we discussed the importance of evolving and engineering the DevOps 360° SDLC while ensuring the fulfillment of the four 360° qualities. Without a doubt, the main mechanism to evolve and engineer your DevOps 360° SDLC is through technological means and advancements that enable capability engineering. What is the point in a cross-level test automation framework if you do not have the technological means to build it? Equally, what is the use of an open source library scanning policy if you do not have the necessary technological means to embed it? And how do you expect to improve your time to market without the necessary technological capabilities and automation across your SDLC? The role of technology is a fundamental necessity in materializing not only your DevOps 360° SDLC but also the broader DevOps 360° operating model, supporting your business to gain and maintain a competitive advantage. This multidimensional and foundational role of technology is becoming apparent also from a DevOps evolution enterprise Objective and Key Results (OKRs) (see Chapter 2, The DevOps Multi-Speed Context, Vision, Objectives, and Change Nature) perspective, as technology contributes to all of them and is not attributed to a specific one. Unfolding the rounded aspects of the technological value proposition for the DevOps evolution will be the focus of the upcoming sections of this chapter.

The DevOps 360° technological ecosystem as a service

The totality of technological enablers of your DevOps evolution is called the DevOps technological ecosystem in this book. And it is indeed, in my opinion, a spot-on term to use. We define an ecosystem as a geographic area where complex living organisms interact in a given landscape under certain conditions, working together toward enabling a cycle of life. A DevOps 360° technological ecosystem, therefore, consists of various technological teams in a certain organization that interact under the condition of the DevOps 360° evolution, enabling an infinite loop of technological capability engineering and consumption as a service. Before we move on and discover the core aspects of our technological ecosystem, let us first bust a DevOps myth to which we have also referred earlier in the book.

The misunderstood relationship between DevOps and technology

It is a widely known secret that the industry, to a large extent, has historically equalized DevOps with technology, and mostly with CI/CD pipelines and cloud capabilities, on several occasions, not even attempting to go further than those two, toward the broader technological ecosystem. The logic is simple: I have a CI/CD pipeline, I do DevOps or We have a Kubernetes cluster, we do DevOps. I have seen several DevOps strategies in my FSI DevOps career that were shaped with the main focus on the technological capabilities that enable DevOps. For all of us who have been part of DevOps adoptions at scale, we undoubtedly recognize the importance of technological enablement in DevOps. Though we also realize that while technology is the easiest (and most fun) of the DevOps enablers to be achieved, by itself it does not deliver the full DevOps potential. This industrial narrow dimension of the DevOps perception and the equalization of it with technology does more harm than good to DevOps evolutions. And even worse, this misconception has historically classified several DevOps adoptions as a failure, implying also that the concept itself is a fallacy. In many cases, that was simply because the implementation of an advanced CI/CD pipeline and the ability to spin up machines in the public cloud did not fundamentally transform the ability of certain organizations to deliver better value faster to end clients. I will get directly to the point now. Technology should not be equalized with DevOps in your organization, as it is just yet another enabler, like people skills, budgets, policies, culture, processes, WoWs, and your regulator are. At the end of the day, look at this book’s outline. Technology occupies only one chapter out of the several that are equal parts of the DevOps 360° evolution.

Why incumbents combine an engineering transformation with their DevOps evolution

A very common approach that I have seen with every incumbent I have worked with is either combining or running in parallel the DevOps evolution with an engineering transformation. Those two often are further combined with a business enterprise agility transformation, as we saw in Chapter 5, Business Enterprise Agility and DevOps Ways of Working Reconciliation. An engineering transformation will typically involve several engineering-related initiatives that are applied across the incumbent’s organization, including both the business’ Agile DevOps teams as well as the technological and infrastructure utility ones. What do those engineering transformations include?

• A mindset and a cultural shift toward lean products and service offerings in an agile context
• Shifting toward the enablement of technological utilities as a service
• Engineering the uplift of people skills
• The replacement of human-manual service models with self-service automation
• A shift left of reliability engineering patterns
• A great focus on the tactical automation of repetitive tasks
• Enhanced modernization and interoperability across the ecosystem
• Supporting technology industrialization and democratization
• Formally embedding the technological and infrastructure utility teams in the business enterprise agility model

Uplifting your engineering skills, WoWs, and capabilities is vital in executing your DevOps 360° evolution. My personal experience says that it is indeed best to combine your business enterprise agility, DevOps, and engineering transformations into one change, but I need to warn you that, in combination with your business-as-usual and unexpected activities, it can be too much to handle in parallel as an organization. If you nevertheless manage, you will get the best return on investment and long-term, sustained change.

Why technology standardization is the new black for incumbents

Standardization and, in more realistic terms, pragmatic standardization and where it makes sense has become the technological new black in the industry and it is included in the DevOps evolution agenda of every single incumbent. If you remember, standardization and simplification are two of the core strategic objectives and DevOps OKRs that we defined in Chapter 2, The DevOps Multi-Speed Context, Vision, Objectives, and Change Nature. As we have repeated several times in this book, your organization is most probably coming from a technological solutions proliferation and duplication context, which, depending on the conditions and circumstances, is of a larger or smaller scale. There is no better space to look at materializing standardization than technology. The benefits can be numerous and its methods balance between coercive (cost cutting, technical debt, and regulatory demand) and operational efficiencies (maintenance overhead and interoperability gaps). The following are some of the main standardization benefits that your organization can achieve:

• Freeing up Agile DevOps teams’ time by reducing the time spent on maintaining shadow IT
• Compliance out of the box in terms of the following:
  • Central teams take care of compliance adherence and evidence of the central platforms
  • Agile DevOps teams get the compliance policies and controls embedded into the tech
• Cost reduction in terms of licenses, servers, databases, and people
• Out-of-the-box full interoperability with the DevOps 360° technological ecosystem
• A clear governance mechanism through the central registration of technological solutions
• Common documentation built into the technology
• Inner sourcing opportunities
• Managing operational risk
• The ability to easily collect metrics across your portfolio
• Great solutions that have been built locally can be insourced and offered to the rest of the organization, minimizing the time and cost to innovate

True story – once upon a time on the trading floor

Once upon a time, I had a meeting with the markets CIO of an incumbent to discuss the DevOps vision for his organization. His office was on the trading floor and as I arrived a little bit early for the meeting, I was walking around. It is always entertaining to walk around a trading floor. The dynamic atmosphere and noise energize you. As I was walking, I noticed that the development teams were sitting quite close to the traders, grouped per asset class (equities, bonds, derivatives, and so on). All the development teams, who were sitting relatively close to each other, had big screens above their heads with any sort of DevOps technology you could imagine. Randomly, I came across an old colleague and I could not help asking (at the end of the day, it was for purely professional reasons, as I was to discuss the DevOps vision with their big boss), Morten, why so much variation in technologies? Do you really need Prometheus, AppDynamics, Nagios, and Jaeger? And why are some on OpenShift and others on native Kubernetes? He laughed and responded, It depends. Some do have valid business justifications, like the .NET guys in FX trading that use Azure DevOps and Azure cloud services. But in most cases, every developer has taken their own decision. It is crazy, I know.

Precautions and circumstances for standardization

Now, having outlined the potential benefits of standardization, I can already read your mind, But Spyridon, standardization this and standardization that...Does standardization really pay off? Let me clarify a couple of things, as the approach to standardization should be one of standardization pragmatism, per concept and context, while taking precautions, as the following sample cases describe:

• Let’s take the example that you make use of Jenkins, the CI/CD orchestration standard. This doesn’t mean that you will start building and deploying your applications the same way, nor that all the CI/CD steps and IT control thresholds are equally applicable to every Agile DevOps team.
• The degree of standardization will depend on the portfolio’s technological foundation of each Agile DevOps team. For instance, the fact that you are to create a common static code quality check policy does not mean that Agile DevOps team B cannot amend that based on the programming language they use.
• Standardization, in certain cases, might not be worth it financially. You need to create your mathematical formula of standardization due diligence. Simply put, in some cases, it is not worth it, so skip it. Some technological setups are too good and too big to be killed. Advanced areas might currently have much better setups than your standard offerings. It is irrational to ask them to move to the standard offerings. As I used to say to the employees at one of the banks, I have a Ferrari and you ask me to replace it with a Ford Focus. (No offense to the Ford Focus, as it was my first, beloved car.)
• There will be vendor cases where you are technologically locked in, with low standardization possibilities.
• There are special cases, such as big data warehouses where the implementation of CI/CD pipelines should cater to specialties. For example, a mainstream pipeline for the purposes of big data implementations will not be sufficient.
• In many cases, standardization activities will never be prioritized in the Agile DevOps teams’ backlogs by the product owners.
• You will face operational challenges as people will try to take advantage of standardization, not to migrate to standard offerings but to push to standard platform teams’ technologies that those teams do not have the bandwidth and knowledge to operate.
• Your common DevOps platform teams might deliver with a delay, or not have the capacity to absorb the number of tenants.
• Do not kill local innovations, as they will be mostly linked to your business lines’ specific context.

Standardization “clean cuts” is the most sustainable way

The most sustainable and pragmatic way toward standardization in my opinion is to define clean-cut dates. As in, providing a clear direction toward pragmatic standardization by giving an adaptation/planning period and then enforcing it for all new and modernized parts of your portfolio. For instance, from January 1, 2023, all newly built and modernized applications must use Azure DevOps as the CI/CD pipeline, OpenShift as their container platform, and provision infrastructure as code through a private cloud portal.

Standardization named “technology sustainability”

In identifying and registering duplicated solutions, a proven practice that I have seen is to have a specific and specialized team solely focused on that. The fanciest and most to-the-point naming of such a team I have heard is technology sustainability. There are plenty of license, code, binary, and IT asset scanning tools available out there that can reveal the level of duplicated solutions in your organization.

Countereffect of standardization

The tactic used to approach standardization is vital to its success. Be careful not to initiate a technological arms race across shadow IT, with Agile DevOps teams attempting to defend against standardization by growing their local setups to an extent that they are too big, complex, and expensive to kill anymore.

One last remark on standardization before we move on: your standardization tactic must be a very clear and targeted one, as you will have to include those activities in your enterprise portfolio planning, as we will see in Chapter 10, Tactical and Organic Enterprise Portfolio Planning and Adoption. Failure to provide a business justification and benefit realization plan will result in those activities not being prioritized. As we all know, technology standardization is mostly driven by the technology-oriented teams and functions of an incumbent bank. That being said, we also all know that it is the business partners of the incumbent who technically own the technological means of production, as they hold the funding for those. You need to be very convincing and/or creative to persuade them to do a brave prioritization, down-prioritizing new business features against migrating to the central ELK cluster. Last but not least, and quite needless to mention, it should be done at relevance.

The main DevOps technological ecosystem partners

We divide the main partners (organisms) of our DevOps 360° technological ecosystem into six main categories, with the Agile DevOps teams excluded, as we consider those to be consumers and not producers of technological utilities:

• DevOps platforms: Teams owning and offering utilities such as CI/CD pipelines, cloud services, test services, and so on
• Technological utility services: Teams offering utilities such as enterprise message buses, operational data storage, and tactical automation
• Core infrastructure teams: Responsible for mainframe platforms, network services, Linux and Windows platforms, middleware, and data centers
• Shared services: Include command and cyber security centers
• End user services: Workspace management, the IT help desk, and the virtual desktop belong in this category
• Governance services: Tools such as a Configuration Management Database (CMDB) and portfolio registry are included

In the rest of this chapter, we will focus on the DevOps platform teams, discussing them across a range of elements, characteristics, and approaches. It is important to clarify that we by no means downplay the importance of the rest of the technological partners, but we consider the DevOps platform teams as core first-level utility consumption partners to the Agile DevOps teams, as well as being at the core of the DevOps 360° SDLC and operating model, hence the focus.

The DevOps platform teams

Platform teams have, in recent years, become the mainstream way of enabling the DevOps 360° technological ecosystem and consequently support Agile DevOps teams in evolving their DevOps adoptions. We define platform teams as cross-functional teams that build frameworks, workflows, and technological utilities, enabling Agile DevOps teams to deliver end-to-end value to their clients through the utilization of technology. In this section, we attempt a rounded and broad deep dive into some of the core aspects of DevOps platform teams.

What is the value proposition of DevOps platform teams?

The DevOps platform teams’ value proposition in our DevOps 360° operating model evolution is rather obvious, I believe, and as expected, is closely aligned with the objective of standardization, while its aspects are more multidimensional. Let us look into some of its core elements:

• Economies of scale and maximization of operational and engineering efficiency: The motto you build it once and it gets consumed many times is one of the objectives behind the establishment of DevOps platform teams. For instance, suppose you have a CI/CD pipeline platform team that builds the critical path of your CI/CD capabilities and workflow for cloud-native applications, piloting that for your asset management portfolio of business applications. The solution, upon success, can scale, creating operational and engineering efficiencies, while onboarding more business lines, without them having to build and maintain their own solutions.
• Fulfill the comparative advantage theory: With comparative advantage in economics, we define an agent’s (DevOps platform team’s) advantage over others (every single Agile DevOps team) in producing a product or a service, faster, at scale, and with a relatively lower cost. This advantage is generated through expertise concentration, full dedication to the product or service, and operational efficiencies generated by economies of scale. In simple words, the CI/CD platform team, by concentrating on CI/CD expertise and skills, being fully dedicated to the product, and creating the necessary means to scale it fast, is expected to be able to deliver faster and cheaper compared to the Agile DevOps teams.
• Capture technological progress in relation to cost: With technological progress, we refer to technological advancements through investment but also the increase in efficiency and output that those bring. By concentrating technological offerings in DevOps platform teams, both progress and cost become more transparent.
• Operational data insights: The utilization of the platforms offered by the DevOps platform teams can support operational data consolidation such as technology utilization, compliance, velocity, and lead times that can be used for multiple purposes.
• SLDC adoption out of the box: As the DevOps platform teams’ utilities will be based on the DevOps 360° catalog, onboarding and utilization of those utilities will automatically imply the adoption of the SDLC for Agile DevOps means. That will also include the adoption of the fundamentals of fulfilling the time-to-market-reliability-compliance equilibrium.
• Broad “state-of-the-art” visibility: DevOps platform teams, as opposed to Agile DevOps teams or third-party vendors, have broad organizational visibility, which supports capturing the big DevOps picture and demand, and consequently translating that to utility offerings that can have a greater organizational impact.
• Dedicated DevOps 360° technological ecosystem: Bringing the dedicated DevOps platform teams close supports ensuring flawless, end-to-end DevOps value stream-based journeys, productivity, and experience, while fulfilling the four DevOps 360° qualities. In addition, the dedication of DevOps platform teams in specific technological domains will support excelling in those domains, establishing proven best-in-class practices, standards, frameworks, and methods.

Be aware of the “technology security dilemma”

Platform teams must be there to serve Agile DevOps teams while maintaining a balance of power and not dictating the technological vision. Observations of the latter will most probably result in Agile DevOps teams removing their consensus of support for DevOps platform teams, due to the effect of a security dilemma. In international relations, a security dilemma occurs when one state grows in military power, becoming a domination threat to other states. The latter then also invest in improving their military power, as well as building a coalition against the threat. I have seen this happen several times in Agile DevOps teams.

How does the DevOps platform team fit into the business enterprise agility model?

Before we deep dive into the technicalities of DevOps platform teams, let us first position them in the business enterprise agility context. Naturally, and as we saw in Chapter 5, Business Enterprise Agility and DevOps Ways of Working Reconciliation, you need to include DevOps platform teams in the enterprise business agility model of your organization and ensure that the respective organizing structures and principles also apply to them, along with the ability to be cross-functional in terms of skills, to achieve the best possible DevOps organizationally aligned autonomy, through the ability to self-organize where applicable. Two very good examples that I have seen in incumbents organizing DevOps platform teams under their business enterprise agility model are the following:

• DevOps journey value stream: Comprising the DevOps platform teams of the technological ecosystem, brought under the same organization and positioned in a sequence of steps based on the DevOps journey of the Agile DevOps teams.
• DevOps engineering tribe/cluster/ecosystem: Similarly, in this model, the DevOps platform teams are brought under the same organization, but this time are not organized as sequential value stream steps but as interrelated and integrated engineering capabilities.

I cannot stress enough the importance of embedding those teams in your enterprise agility model, as you need to achieve organizational cohesion and harmony, in the broader sense of WoWs, including your SDLC evolution and engineering, agility ceremonies, enterprise portfolio planning, and benefit realization. The embodiment is relatively intuitive, due to the agnostic reconciliation relationship of DevOps with business enterprise agility models. As we discussed in Chapter 5, Business Enterprise Agility and DevOps Ways of Working Reconciliation the key role of DevOps platform teams is recognized in every single business enterprise agility model, hence they all provide the necessary organizational structure means.

How does a proven operating model of DevOps platform teams look?

A fundamental aspect when designing DevOps platform teams is their operating model. By operating model, we refer to the way that the platform team is organized and operates on a day-to-day basis within its internal context, but also within its external context. Throughout my career, I have had the opportunity to experiment with various operating models. Through that experimentation, I am confident to say that I found what I call the triangular model (see Figure 7.1) to be the most effective operating model as I have literally never seen it not deliver and I have four examples of incumbents who are using it.

Figure 7.1 – Triangular platform team operating model

In the triangular operating model, depending on its size, scale, and clients, the team is either virtually or in person divided into three main parts, all belonging in essence to the same DevOps platform team. Each of the three parts has a special focus, while one complements the other. The naming of the three parts varies per adoption, though it is critical for the naming to be descriptive of the parts’ responsibilities:

• Innovation team: Takes care of improving the platform from both a functional and also a non-functional perceptive and owns the improvements backlog. The backlog normally includes tasks such as the provisioning of tools and services, interoperability with the rest of the DevOps technological ecosystem, upgrades, tactical automation, and product built-in knowledge management systems.
• Onboarding/task force team: Do you recall the client engagement role of the DevOps CoE? This is very similar, if not identical, and in certain cases is a joint force between the DevOps platform and the DevOps CoE. This arm of the platform team either supports the onboarding of Agile DevOps teams to its platform, through starting from scratch or migration from another platform, conducts incubation and training, or acts as a task force supporting either cases of reliability issues or vast and critical business go-lives. This part of the DevOps platform is dedicated to the platform’s client engagements and does not actually contribute to the platform’s backlog but provides feedback for improvements based on the engagement observations. A very important role of this team is to work together with the Agile DevOps teams and other DevOps stakeholders on the DevOps journey, engineering, and productivity initiatives.
• Operations engineering team: This team has a dual role, and it is the face of the DevOps platform team towards the outside to the platform team world. Its first responsibility is to keep the platform’s lights on in terms of incident management, monitoring, business continuity plans, and performing user request fulfillment. The second is to constantly conduct platform reliability improvements, as well as eliminating toil for the platform tenants, in collaboration with the platform innovation team, while gatekeeping the platform’s operational readiness.

The people across the triangular operating model must be engineers at heart and in skills. Even the ones primarily focused on platform hygiene and request fulfillment should be engineers under incubation. Who, when ready, will move to more advanced engineering responsibilities within the team? Career mobility is also very much possible and advisable across the three teams. Triangular teams, you will typically notice, due to the key role their people have in the evolution, will be working very closely with the DevOps CoE and be core members of the DevOps 360° design and advocacy authorities.

How does a proven service support model of a DevOps platform team look?

The core part of a DevOps platform team’s operating model is what is often called in the industry the service support model. By service support model, we refer to how the platform’s utility consumers interact with the platform team for day-to-day matters – a core aspect of improving DevOps journeys, productivity, and experience. In this sub-section, we will look into the anatomy of such a service support model, through the lens of proven practice.

A strategy that I often see incumbents deploying to support the backbone of service support models is building their own custom-made portals, as a single point of entry across DevOps 360° technological ecosystem teams. Those portals have the nature of a private cloud setup, and you will often find them labeled DevOps portal, Everything as Code, or Developer experience portal.

Single point of entry and Level 0

The first layer of a DevOps experience portal is a user interface, often built in React or Angular, which displays the available options to the end user. These options can range from knowledge base articles to raising incidents for a specific platform, and from raising a new feature request to self-service through automation. The latter is the one that, in terms of support levels, I call Level 0 (L0). L0 includes all the self-service capabilities, through chatbots, APIs, premade scripts, and templates, that DevOps platform teams have automated, eliminating manual interventions and lead times.

Service desk – Level 1

The next stage is what is often referred to in the industry as Level 1 (L1). Typically, as part of this level, we find a service desk team that is responsible for the fulfillment of requests that has not yet moved to L0 through automation. This team normally covers the technological stack of more than one platform team or the totality of those. It is the ideal team to have new joiners of the platform teams embedded in supporting their incubation through exposure to clients and a wide range of technologies. Furthermore, this team is the perfect candidate for moving its positions to nearshore or offshore locations, combining savings and around-the-clock support. Principally, this team consists of engineers who, based on the client request observations and when time allows, can focus on improving the L0 capabilities.

Reliability engineering – Level 2

Moving on to the next level, which is typically called Level 2 (L2), or reliability engineering in our context, this team is a very skilled and seasoned one and its primary responsibilities include the following:

• Emergency response to critical infrastructure incidents
• Catching the dispatched L1 tickets that cannot be resolved by the service desk
• Automation toward L0
• Proactive operational readiness and reliability engineering improvements

The reliability engineering team normally will be dedicated to a specific DevOps platform team as expertise and speed are vital to its operations. For instance, there can be a CI/CD reliability engineering team or a container platform reliability engineering team. It should should follow Google’s SRE paradigm of 50% of its time invested in supporting the platform and 50% in improving the platform. Both the service desk and reliability engineering team belong to the operations engineering team that we discussed earlier as part of the operating model.

Innovation – Level 3

Last but not least comes Level 3 (L3), which is in essence the innovation team that we discussed earlier in the operating model. L3 is mostly tasked with support that requires code/configuration changes in the platform or new interoperability capabilities, as well as with parsing requirements for new functionality to become available.

The following figure provides a visualization of the service support model, also providing some corresponding support examples per team:

Figure 7.2 – Sample support model for the platform team

When it comes to platform onboarding for Agile DevOps teams, you should strive to provide that through a combination of L0 (premade recipes/templates and interoperability APIs) and your platform knowledge base (how-to articles). The L1 team should primarily handle any inquiries, with the support of L2 and L3, depending on the support volumes and nature. Obviously, if you adopt the triangular operating model, and especially in tactical adoptions, there must be a dedicated team that is slightly detached from the business-as-usual service model, focusing exclusively on the flagship applications.

The DevOps platform product and service catalog

Obviously, each platform team’s service model will very much depend on its product and service catalog. We define a product and service catalog as the complete and concrete list of products and services that a DevOps platform team offers to its consumers. Product and service catalog clarity directly indicates clarity to the value proposition and scope of the respective platform team and therefore tactical positioning in the broader DevOps 360° evolution. That tactical positioning serves a purpose from multiple perspectives: the Agile DevOps team’s perspective on what utilities can be consumed centrally, the DevOps 360° technological ecosystem perspective on what the interoperability points and capability engineering demarcation are, as well as the broader DevOps 360° ecosystem on the DevOps 360° operating model enablement perspective. The following table provides an example of how a potential product and service catalog for a CI/CD platform could be formed.

Table 7.1 – Example of a CI/CD pipeline platform team’s product and service catalog

From DevOps platforms’ catalogs to enterprise technology menus

For many incumbents, the consolidation of all the technology catalogs across the DevOps 360° technological ecosystem forms what we often find called the enterprise technology menu. This consists of the totality of pre-approved and centrally offered as a service technologies that can be consumed when implementing the DevOps 360° SDLC.

Figure 7.3 – Example technology menu

In my experience, enterprise technology menus are easier to define than implement. Firstly, moving to a standard technology is not always straightforward and often has several complications. For instance, if the Agile DevOps teams are obliged by the technology menu to shift to a new ETL tool, this most probably will not be just a lift-and-shift overnight exercise. It is highly likely that they will need to refactor their applications, investigate upstream and downstream data dependencies, redesign interfaces, and conduct thorough integration testing. And what about dependencies on external legacy platforms of partners and regulators that can only parse data in specific formats, through specific channels? In these cases, you will most probably end up with a significant period of parallel application runs (old and new running in parallel) or the move will never happen, either because of lack of time and money or high operational risk. Certainly, your organization, as we will see in more detail in Chapter 10, Tactical and Organic Enterprise Portfolio Planning and Adoption during the evolution, will face an adoption dilemma. Are we happy to adopt the DevOps 360° SDLC using our own tools, or do we need to move to the standard ones? The dilemma’s scale obviously will depend on your DevOps context and strategic objectives and its real impact will become more apparent when you do the evolution adoption mathematics. But hold on to that thought, till we reach Chapter 10, Tactical and Organic Enterprise Portfolio Planning and Adoption.

To conclude, technology menus are without a doubt a solid approach to simplification and standardization, but in my opinion, only as long as they follow a pragmatic and tactical approach, as we discussed regarding cut-offs. And while cut-offs will not necessarily resolve your inherited DevOps complexity, they will most probably, as your modernization strategy progresses, make it more sustainable.

Why is the demarcation of responsibilities important for DevOps platforms?

As we saw in the previous chapter, the DevOps capability anatomy is rather complex. One of the main challenges I have seen in my career when it comes to DevOps platform teams is defining who is responsible for what in the platform, initially between the platform and its tenants, and afterward, across other stakeholders who have a role in a certain capability. Getting this demarcation clear through the platform enablement and onboarding process can save unnecessary waste and disagreements. Let us look into that through a practical example. Suppose that SonarQube is offered by the CI/CD platform team, and it is consumed by the Agile DevOps teams, while the DevOps CoE also comes into play as it owns the SDLC IT controls framework. A potential responsibility demarcation agreement could look like the following (R in the table stands for Responsible):

Table 7.2 – Example of technological capability responsibility demarcation for static code quality scans

Remember that when more than one team is responsible, no one is in essence, as responsibility comes with clear liabilities and authority. Hence, agreeing on the fundamental single responsibility of enabling and consuming a capability has multiple purposes.

Why you need a “DevOps platform – tenant” social contract

Good bills make good DevOps partners. Therefore, expectation clarity, transparency, and alignment need to be established between the DevOps platform teams and their tenants, through a formal social contract. This is even more of a necessity as most probably, and due to historical reasons, there is an unconscious bias of your Agile DevOps teams toward the platform teams. The situation is even worse if trust is already lost on certain teams and heavy reputational restoration is required, especially if there is no change on the people side of those teams. Committing to responsibilty in writing will have a positive and binding impact. The following are the main areas where social contract transparency is required:

• Availability, performance, continuity, and the resilience targets of the platform
• New feature and vendor version upgrade velocity and cycle times
• Operating and service model agreements
• Ensuring the platform’s out-of-the-box compliance
• The new capabilities prioritization process, product vision, and roadmap
• Platform interoperability with local home-grown solutions

It is inevitable that the nature of DevOps platform teams calls for focus in two directions in order to fulfill the social contract: to deliver and build the desirability of their products and services, while in parallel ensuring belief in their ability and credibility. But it’s not only the DevOps platform teams that have liabilities from the social contract, but also their tenants. The fulfillment of the social contract by the DevOps platform teams will imply that the Agile DevOps teams will need to stop the proliferation of local home-grown solutions and support the common standard offerings.

Treaty on the non-proliferation of DevOps technologies

In international relations, the most well-known threat of non-proliferation is the one of nuclear weapons. The signing members (platforms and tenants) commit to stopping the spread of nuclear weapon technology (DevOps tech) and promote cooperation in the domain of nuclear energy. It is just remarkable how DevOps relates to international relations.

How can you enable DevOps journeys, productivity, and experience?

Very much related to the social contract is the term developer experience (DX), which in recent years has been used to describe the experience of a developer in accessing and using technology across the SDLC. In this book, given our non-developer-centric DevOps approach, but also looking at experience holistically, we will replace DX with DevOps journeys, productivity, and experience. A focus on DevOps journeys, productivity, and experience is essential to the success of DevOps platform teams. That is, to the extent that I would propose your platform teams have dedicated DevOps platform people, working together with the Agile DevOps teams on the cause of reducing friction, idle times, and bottlenecks across the flow of technological utility consumption. The possibilities are endless in this topic, with the following being some of the proven practices:

• Understand and visualize DevOps journeys across the entire DevOps 360° SDLC value stream.
• Set performance indicator benchmarks across the DevOps 360° SDLC phases and optimize against them.
• Identify and optimize lead times that slow productivity down and cause cognitive state switches.
• Automate the consumption and provisioning of IT assets through streamlined portals, also providing onboarding estimates to support effective planning
• Increase the interoperability of services across the ecosystem.
• Shift toward process and policy engineering with automated evidence generation.
• Improve self-service and utilize off-the-shelf public cloud capabilities.
• Document standards and methods in code repositories.
• Build and utilize scalable and reusable solutions through templating and recipes.

How technology consumption defines the organizing principles of Agile DevOps teams

Technological advancements and capability engineering positively impact DevOps journeys, productivity, and experience, and consequently, the organizing principles of the Agile DevOps teams that we discussed in Chapter 5, Business Enterprise Agility and DevOps Ways of Working Reconciliation. Where from and how Agile DevOps teams are consuming technology adds another element of at relevance in a multispeed banking context to your DevOps 360° evolution. Understanding those technological dynamics and how they shape the organizing principles of your ways of working and influence your platform modernization journeys is an important aspect of your evolution. Being able to proactively assess impact so you can plan a gradual evolution will be game-changing.

In the following figure, using the portfolio classification of this book, we illustrate the most dominant scenarios that we see rising when technological advancements and democratization take effect during the DevOps evolution.

Figure 7.4 – Agile DevOps teams organizing principles illustration based on technology consumption

As is obvious, the influence of technology is profound in Agile DevOps teams’ WoWs, as it defines and also impacts the levels of autonomy and responsibilities. This situation undoubtedly creates different speed levels, which we will come back to in Chapter 10, Tactical and Organic Enterprise Portfolio Planning and Adoption.

What to look for in DevOps technological utility due diligence

I would like to start this sub-section with a real story of mine that has a very deep meaning if you think about it carefully.

The cheeseburger story

Some years ago, we initiated a CI/CD pipeline standardization process in a bank. It was not dramatic at all, actually, in terms of variations, and we were all aligned in the chosen technologies, though everyone had separate processes. One of the differences we had was in artifacts management, with half the organization using one technology and the other half using another. Our group CTO asked for a comparison of the two to be conducted during one of the meetings. Then, one of the business CIOs spoke, There is no reason to waste time. It is like comparing a cheeseburger from fast food A with a cheeseburger from fast food B. I’ll leave it up to your intuition to decode that ambiguous statement.

Sooner or later in your evolution, you will reach the stage of evaluation, due diligence, and comparison of various technologies. When you reach that stage, you need to make sure that you do not omit important aspects. What I have observed in several cases is that the functional requirements dominate the criteria of evaluation, with the non-functional being overlooked. In most cases though, it is the latter that proves to be a showstopper in the long run. In this sub-section, we will refrain from discussing functional requirements as they are heavily business context, technology, and capability dependent. Our focus will be on the often omitted non-functional qualities, as it is easier to use common and agnostic criteria for them. The following list is my favorite one from over the years.

Table 7.3 – Non-functional/quality requirements technology assessment sample

Having thorough due diligence evidence is important not only for decision traceability purposes but also for future fit-for-purpose re-assessment purposes. Also note that this exercise is obviously not only applicable when deciding on new solutions but also during the consolidation of existing ones.

Why commercial versions gain momentum in the FSI

The preceding assessment can also provide answers to the open source versus commercial version dilemma, which in recent years has dominated incumbents’ technological due diligence agenda. The overall observed tendency in the industry is to steer away from open source tools and toward enterprise and commercial versions. The main reasons are the following:

• DevOps platforms are mission-critical for running banks. Therefore, ultra-availability, continuity, and resiliency, along with vendor support, are required.
• Regulatory pressure on DevOps SDLC and platform audits, especially in a cloud-native context and particularly in the domains of security, compliance, and auditing.
• It’s easier to attract employees that already have expertise in certain technologies, and also to incubate internal ones.
• Advanced interoperability capabilities.
• The product roadmap, especially in the case of GSI incumbents, can be influenced, along with the ability to exclusively customize products.
• Commercial product packages’ bundling ability.

There is, as you can see, plenty of solid rationale behind this commercial shift. For all of you that have been part of a large DevOps transformation, you also know that community and open source technologies will not be sufficient when you start standardizing and institutionalizing at scale. Sooner or later, you will have to get into commercial discussions.

The industry’s platforms versus direct cloud native dilemma

While DevOps platform teams still advance in the industry with many incumbents not yet harvesting their full potential, there is yet another DevOps dilemma that is rising on the horizon – one of DevOps platform teams versus direct cloud native. Direct cloud native in this context refers to establishing the necessary organizational, governance, and technological means in enabling Agile DevOps teams to consume DevOps technological utilities directly from cloud technology providers in a secure and compliant way, without the DevOps platform teams acting as an intermediary. This is a very interesting development that requires extensive preparations in ensuring sustainability and compliance and, moreover, in finding the right balance, with the DevOps platform teams serving the Agile DevOps teams who will stay on-premises.

How communities of practice strengthen DevOps platforms

We referred earlier in this book to the strongly opinionated developer. You know, that guy who walks around the office wearing an I love Java T-shirt (kidding). There are also strongly opinionated platform engineers in organizations. It is advisable to bring people with strong opinions together in communities of practice and get them to contribute the most to your technological ecosystem. To cover every single core DevOps platform team you must have some kind of connection to your DevOps CoE and the DevOps 360° design and advocacy authority. There are plenty of reasons for doing so:

• Providing the Agile DevOps teams with a sense of and ability to influence the DevOps platform roadmaps and backlogs, and actually allowing them to do so
• Giving the DevOps platform teams the ability to understand the business context of Agile DevOps teams
• Increasing awareness and incubation through training sessions and demos
• Enabling inner sourcing synergies
• Conducting joint demos and live sessions to promote each other’s work

The main five strategic platforms incumbents focus on

Looking across incumbents, you will discover certain DevOps platform teams receive greater focus, and there are certain reasons for that: strategic positioning in the DevOps 360° SDLC, a strategic role in enabling the broader DevOps 360° operating model, high demand from Agile DevOps teams, and significant focus from a regulatory perspective. In this sub-section, we outline the most strategic DevOps platform teams we find in incumbents.

The ITSM platform

Perhaps it will come as a surprise to you that, often, we see ITSM platforms not being considered at the core of DevOps technologies. That is due to the dominance of the traditional ITIL interpretation of ITSM in the industry. In our book though, we approach ITSM from a process engineering and interoperability in the DevOps flow perspective. Moreover, we will examine it from a strong site reliability engineering (SRE) (more on this in Chapter 13, Site Reliability Engineering in the FSI) perspective, toward the enablement of autonomous operations. But let us save SRE for later and look at the following responsibilities of an ITSM DevOps platform team:

• Event, incident, and change management flow automation
• Simplification of the problem management procedures
• Configuration management database automation
• Automated calculation of availability for applications and services
• Automation of service onboarding to service catalog registries
• Automated release impact analysis, as part of release management
• Automation of production-readiness review assessments
• Tactical API automation and interoperability to the broader ecosystem
• Automation of service governance regulatory evidence

Through process engineering, those teams’ mission is to increase the ITSM platform engineering capabilities and shift them left in the DevOps 360° SDLC. Also, our ambition was in one bank to gradually ensure that Agile DevOps teams will forget what the ITSM tool GUI looks like. In recent years, there have also been big developments in the reconciliation of ITSM platforms and processes with DevOps. Platforms such as ServiceNow have started offering DevOps module functionality and platforms such as BMC Remedy offer a rich API interoperability catalog. There’s a lot to be done in this domain.

The CI/CD pipeline platform

The CI/CD pipeline platform team is the king of DevOps platforms, especially for organizations that have fallen into the misconception of equalizing CI/CD pipeline adoptions with DevOps adoptions. As is clearly described by its name, this team offers the core CI/CD capabilities required in enabling an evolved and engineered DevOps 360° SDLC. The offering of this platform team, I believe, is straightforward and, therefore, for the economy of your reading, I will refrain from mentioning code repositories and build tools. What I will focus on is an at relevance in a multi-speed banking context element that CI/CD platform teams have been observed as enabling in the reality of incumbents.

Do you recall the different paths that we mentioned in the previous chapter on the DevOps 360° SDLC? Think of your code and artifacts as a train wagon and those paths being railways. Depending on your programming language, platform technology, and portfolio classification, your code and artifacts can follow different paths, reaching different destinations where they are deployed. For many incumbents, different paths mean different or partially different CI/CD pipelines. The most dominant scenarios that I have identified in the industry are the following:

• Mainframe CI/CD: Dedicated to mainframe applications, the simplest of CI/CD versions
• Decoupled CI/CD: Dedicated to decoupled on-premises applications, covering mostly the CI/CD critical path
• Cloud-native CI/CD: Dedicated to cloud-native applications; very advanced and extending much further than the critical path
• Mobile CI/CD: Dedicated to the digital front tier of incumbent banks, which is mobile banking, especially in the case of greenfield initiatives; equally advanced as cloud-native CI/CD

Enterprise CI/CD versions gain momentum in incumbents’ adoptions from a critical path perspective, with the Atlassian Stack, GitHub, GitLab, Azure DevOps, and CloudBees (Jenkins’ enterprise version) being the dominant ones.

Naturally, the different pipelines have capability overlaps and to a certain extent utilize the same technologies, being extensions of the foundational critical path in essence.

The testing services platform

An often-overlooked platform, which lacks investment due to a struggle in understanding its value, complications in setting it up, as well as the cost of establishing it, is the testing services platform. All of us that have worked with DevOps realize the importance of testing services. In the absence of them, time to market and reliability aspects of software delivery can be jeopardized, SDLC implementation costs rise, regulatory evidence is hard to collect, and heavy manual interventions during testing increase the probability of defects. Now, I need to make clear one thing about the definitions. In this book, we do not refer to Testing as a Service (TaaS) as the practice of outsourcing testing-related activities to vendors or third-party companies in particular; we refer to the capabilities built by an incumbent in offering testing-related services and utilities to Agile DevOps teams.

Such a platform should focus on the following capabilities:

• Test management tools and test evidence generation
• Test automation frameworks, across test levels (regression, integration, performance, user acceptance, and so on) interoperated with the CI/CD pipeline
• Test environment provisioning both on-premises and also in the cloud in cooperation with DevOps platforms
• Develop disposable test and innovation/experimentation environments
• PaaS solutions for test data services (extract/generate – obfuscate – clone – load – clean)

Unfortunately, and primarily due to investment constraints, I have seen DevOps testing services platform teams either not being established in the first place, struggling to find their position in the organization, or having challenges in delivering on expectations. However, I strongly recommend you invest in them as they can prove to be game-changers.

The observability platform – telemetry

A domain where incumbents have invested in recent years is observability. Looking at an average representative example, we will find observability platform teams focus on delivering the following capabilities:

• Open telemetry across the SDLC: Through this capability, data is collected across the DevOps 360° technological ecosystem, serving multiple purposes, such as the following:
  • Technological capability adoption and utilization measurements
  • Operational efficiency metrics on velocity and lead times
  • Real-time audit and compliance evidence

A quite innovative solution that is currently trending is the usage of real-time enterprise message busses, such as Kafka, to collect metrics across the DevOps 360° technological ecosystem, which are either streamed directly through visualization tools such as Grafana or are stored in operational data storage and sourced by reporting tools such as Power BI.

• Application and infrastructure monitoring: This service will primarily offer technological utilities and capability engineering, on application and infrastructure monitoring, from a complete tiering perspective. The following tiers are the mainstream ones in the industry, which we complement with examples:
  • Tier 1 – Infrastructure: virtual machine CPU, memory, and network
  • Tier 2 – Application processes and services: morning checks and starting engines
  • Tier 3 – Critical application-specific logic: client asset portfolio valuation
  • Tier 4 – Cross-application value stream data flows: clearing and payments straight through processing
  • Tier 5 – Business process and activity flows: approval of risk numbers
  • Tier 6 – Customer activity: account opening end-to-end value stream

There are plenty of solutions available for incumbents in this space with Prometheus, ITRS Geneos, AppDynamics, Elasticsearch, Jaeger, and cloud-native solutions (where applicable) being the dominant ones used by incumbents.

• Application and infrastructure logging: This service covers the logging of technological utilities, primarily covering the following four capabilities, and is of absolute focus to incumbent banks from a standardization and advancement perspective:
  • Infrastructure logging: Activity on virtual machines
  • Operational logging: Activity on the business logic layer of the application
  • Security logging: Security events on the infrastructure and operational layer
  • Compliance and audit logging: IT controls and data retention

Splunk and Elasticsearch, as well as public cloud vendor-native solutions (where applicable) are the dominant solutions used in the industry.

On top of the technological utilities offered in the broader DevOps 360° technological ecosystem and to Agile DevOps teams, this platform team has a pivotal role from two relatively vital perspectives in the evolution:

• Providing dashboarding and visibility (often through the respective agile DevOps teams) to the business teams in order to increase the evolution’s transparency
• Providing mechanisms for capturing the DevOps 360° evolution’s technological but also broader adoption progress

As we will see in Chapter 11, Benefit Measurement and Realization, data is gold in measuring your evolution’s progress and realizing benefits and this team is a core partner of that cause.

Private cloud platforms

In essence, and technically, this offering consists of custom-made portals that are used by Agile DevOps teams, but also the broader stakeholder’s ecosystem, in materializing two main business cases:

• Infrastructure as Code utility provisioning: Various IT assets, from Linux and Windows servers to Oracle databases, and from Kafka instances to ELK clusters, including the necessary security policies, are provisioned automatically through premade templates, playbooks, and recipes (as code) that the private cloud platform team creates and maintains. The purpose is to eliminate lead times and the human effort of provisioning IT assets so software delivery can be sped up, as well as fulfilling standardization and out-of-the-box security and compliance aspects, along with enabling dynamic updates of IT asset utilization in the configuration management database and service governance tools. Terraform is the dominant technology currently deployed in provisioning templating solutions.
• Container platforms: These are used in order to host containerized applications, which, in this book, we include under the cloud-native portfolio classification category. Containerization is a method of building, deploying, and running applications in the form of containers, which are encapsulated with all the necessary dependencies of the application, such as libraries, binaries, and configuration files. Containerization is ideal for microservices and distributed architectures, following cloud-native principles, and can increase the speed and independency of delivering software. Container platforms provide an environment for applications to be built, deployed, and run, along with the corresponding deployment, orchestration, monitoring, security, and compliance mechanisms. Red Hat’s OpenShift and Docker Enterprise Edition are the dominant choices of incumbents on container platforms.

Private cloud platforms technically serve as forms of IaaS and PaaS utilities that, in most cases, run on-premises (in a data center of the incumbent).

Public cloud platforms

Public cloud adoptions, in recent years, have gained momentum in the industry, from a software, platform, and infrastructure as a service perspective. With the path to the public cloud not being that straightforward due to legacy procedures, governance, security, and compliance requirements, incumbents have been establishing cloud platform teams to facilitate this journey. The focus of these teams in most cases is multidimensional:

• Deliver public cloud solutions, from an infrastructure, platform, and tooling perspective, across SDLC environments.
• Define the public cloud operating model along with the Agile DevOps teams and the core infrastructure teams, including capability demarcation, CloudOps, and FinOps.
• Ensure adherence to compliance requirements, through policy engineering controls and built-in by-design security baselines.
• Conduct public cloud due diligence exercises across various public cloud vendors.
• Hands-on support in the onboarding of tenants, either through lift and shift or application “cloud-native re-engineering.”
• Definition of cloud-native standards and qualification assessments in cooperation with the enterprise architecture, core infrastructure, other DevOps platforms, and the DevOps CoE.
• In many cases, frontrunning of SRE standards implementation.
• Incubation and training of the agile DevOps teams.

Through my personal experience in working with incumbents, I have discovered two interesting findings when it comes to public cloud vendor choices and corresponding setups of public cloud teams. Firstly, their main focus from a diligence perspective is concentrated around the following dominant use cases and corresponding vendor choices. To make it clear, my intention is not to advertise any particular public cloud provider, and I appreciate that different incumbents make different choices. The following is, though, a very accurate, representative example of the industry’s choices:

Table 7.4 – Public cloud use cases and vendors in the FSI

As you can see from the preceding table, it looks quite balanced and confirms the multi-cloud approach of incumbents (my second observation), driven on the one hand by the business lines’ use cases, and on the other hand by regulatory requirements around continuity and data retention across different geographical regions of the world.

Following the reality of the multi-cloud context, to ensure dedication and building expertise, incumbents are forming dedicated cloud vendor DevOps public cloud platform teams. In many organizational structure cases, they belong under the same public cloud center of excellence or the same public cloud community of practice. The scope of these teams is not only to achieve excellence in the adoption of a certain cloud provider’s capabilities but also to ensure the highest possible degree of hybrid cloud solution agnosticness (compliance, security, orchestration, CI/CD, and observability), as well as enabling the pragmatic demarcation of the public cloud operating model’s organizing principles and demarcation across the totality of the broader DevOps ecosystem of stakeholders.

Summary

In this chapter, we focused on the technological enablement of the DevOps 360° operating model. We outlined the inevitably strong value proposition of technology for the DevOps evolution, and we analyzed the term technological ecosystem, also outlining the six main organisms that comprise it. We continued by looking into arguments on busting the myth of equalizing DevOps with technology, concluding that technology is just another enabler of DevOps and not its whole essence. Afterward, we moved on to discuss two initiatives that currently dominate incumbents’ DevOps technology agenda, which are engineering transformation and technology standardization. On the latter, apart from outlining its value proposition, we also discussed why it is important to take precautions. The DevOps platform teams dominated the core of this chapter and we discussed them across several dimensions. Starting by outlining their value proposition, we moved on to outlining proven operating and service models that incumbents have adopted, also discussing the importance of product and service catalogs. The latter we also connected with initiatives such as technology menus, which are presently gaining momentum in the industry. Continuing with talking about DevOps platform teams, we argued the importance of establishing responsibility demarcation, as well as a social contract between the platform and its tenants, highlighting their relationship to DevOps journeys, productivity, and experience. We closed the chapter by examining the five main domains of focus for incumbents when establishing DevOps platform teams.

In the next chapter, we will deep dive into the dark and misperceived world of DevOps compliance in banking.

Technology product	DevOps 360° SDLC capability engineering enablement	Platform team service
Azure DevOps	Task management Version control Code build Code deploy Test management	Platform maintenance and onboarding Standards, methodologies, and proven practices CI/CD control adoption Ecosystem interoperability Knowledge base Central compliance and adoption evidence collection
SonarQube	Static code analysis
Artifactory	Build and deploy artifacts’ storage