Shared Values

As we saw in Chapter 2, all organizations have found value in reducing the size of each production change. Ideas like Trunk-Based Development and decoupling deployment from release allow engineers to get closer to their ideal of single-piece flow.

Organizations that excel at Continuous Delivery all empower product engineers with autonomy, as well as accountability for their changes. The engineer who authors a change is the person responsible for shepherding that change into production and watching for any potential defects. Product delivery teams also have a lot of autonomy in terms of how they work—I repeatedly heard from participants that it was hard to describe the delivery process since different teams within the organization work in different ways.

In order to achieve this level of autonomy, there is a noticeably heavy investment in custom delivery platforms that provide self-service capabilities to product engineers (there is a summary of the capabilities of these platforms in Chapter 4). Many organizations make a distinction between their Continuous Integration system and their Continuous Delivery infrastructure, where their Continuous Integration system is responsible for initially validating a change and building a deployable artifact, while the Continuous Delivery infrastructure is responsible for moving a build artifact through various environments and monitoring those environments for potential issues. The Automotive Marketplace is the only organization I spoke with that has a unified system providing both Continuous Integration and Continuous Delivery.

Two Modes of Continuous Delivery

While participants shared a lot of software delivery practices, I did notice that all fell into one of two distinct modes of Continuous Delivery.

One mode is Branch-Based Continuous Delivery, where a team uses short-lived feature branches as their unit of change and manages code review with pull requests. The other mode is Trunk-Based Continuous Delivery, where teams practice Trunk-Based delivery, work directly on master, and do ad hoc code review.

Trunk-Based teams have a more rapid deployment tempo than Branch-Based teams with multiple new deployments per hour for Trunk-Based versus once or twice a day for Branch-Based.

There is an even starker distinction in the typical cycle times for a production change between these two modes, with changes being in progress for much longer for Branch-Based teams. This is partly because the unit of change for Branch-Based teams is much larger—often an entire feature, rather than an individual commit. In addition, Trunk-Based teams are typically doing code review out-of-band, rather than blocking a change from going out.

Trunk-Based teams also tend to rely much less on preproduction environments for quality checks, instead of testing in production. I suspect this is because faster change makes the sort of manual validation that is done in preproduction environments a lot less feasible or valuable.

Most of the larger engineering organizations I talked to had teams in both of these modes. Generally the teams working with larger, older systems use Branch-Based Continuous Delivery while teams working on new, smaller systems operate using Trunk-Based Continuous Delivery.

Monoliths Versus SOA

The high-level architecture of a system has a marked impact on how Continuous Delivery is implemented. Most notably, as discussed in Chapter 4, teams working on large, monolithic systems are forced to batch production changes up using mechanisms like Release Buses. This is due to the rate of changes from multiple teams, a relatively slow deployment, and cross-team coordination challenges. In contrast, in SOAs a team typically has full ownership of a small codebase, and can often achieve single-piece flow, where each unit of change rolls out to production independently. Several participants highlighted this as a big advantage of working in a service-oriented system.

Succeeding with Continuous Delivery

After studying how teams are succeeding with Continuous Delivery in the wild, themes emerged which inform your strategy.

Reducing batch size should be a guiding principle. Trunk-Based Development practices like Feature Flagging and Branch by Abstraction are also key. Keep the size of each unit of change as small as possible (e.g., by aggressively focusing on short feature branches), and avoid batching up production changes whenever possible. The ideal is to get all the way to pure Trunk-Based Development, eschewing feature branches entirely and reaching single-piece flow.

In general, branches should be viewed with suspicion. Long-lived feature branches should certainly be avoided, but so should the use of branches for release management. Instead, prefer delivery pipelines that flow a build artifact through environments, and eventually into production, as discussed in Chapter 2.

Organizations who want to move toward Continuous Delivery should plan to invest quite a lot in a delivery platform with self-service tooling for product engineers. Rather than having infrastructure or operations engineers manage deployment and monitor production themselves, they should focus on building the tools that enable product engineers to do this work.

The most consistent thing I heard from the participants was that Continuous Delivery has resulted in overwhelmingly positive outcomes. I hope that the experiences of those organizations will help you achieve the same.