The Road to Disciplined DevOps
Over the long term, the mortgage application portal (MAP) application proves to be a successful system for BigBank. The first release, which focuses on providing mortgage information for existing customers and information about mortgage options, is well accepted by BigBank customers. It results in a measurable decrease in support desk calls and information requests in branches for mortgage information, providing an important cost savings to BigBank. The second release, which adds the ability for new and existing customers to begin the mortgage application process online, provides both a significant cost savings, as well as a revenue increase through a greater renewal rate. Subsequent releases add new and more sophisticated offerings.
In this chapter, we overview both the team's and BigBank's process-improvement journey. Over the span of several releases, the team evolves from their initial Agile life cycle to the Lean life cycle, and then eventually to the Continuous Delivery: Lean life cycle. Given the success of the MAP deal with DAD, BigBank decides to try out DAD on several other development teams. They realize that the enterprise aspects already built into DAD would have taken them years to figure out on their own, had they stayed with Scrum. The team's process improvements, particularly around evolving into the Continuous Delivery: Lean life cycle, dovetail nicely into BigBank's overall Disciplined DevOps strategy.
Release 2
Release two begins with a two-week Inception phase. During this period, the team spends several hours sizing the new requirements that Patricia has identified the previous week. Debbie and Ashok each decide to take the first week off for vacation, and Dick takes off both Fridays so that he can have two long weekends. The team also takes a three-day introductory course in test-driven development (TDD) the second week, as this is a practice that has intrigued several team members ever since they started regression testing within the team during release one. This phase also overlaps with the warranty period for release one, where the team had to be available to fix any production problems. During the first week of Inception, a few severity two and severity three issues are identified and fixed. A patch is deployed into production, once again working with Oswald over the weekend.
The Construction phase consists of seven two-week iterations. Here are the highlights of what happens:
•TDD. The team's adoption of TDD is a bit rough at first, but goes well over time. Carlos spends a lot of time coaching team members in TDD, pairing with each team member for an hour a day for the first three iterations. Tara also spends similar amounts of time pairing with people to help them gain basic unit testing skills. By the end of the release, everyone is comfortable with taking a test-driven approach, although they feel that they still need to work on growing their skills. Team members with good design skills to start with find it much easier to pick up TDD than those without such skills.
•Remote work. Before agile was introduced into BigBank, it was common for people to work one or two days a week from home. The scrum coaches that BigBank hired insisted on colocated teams, which from the point of view of reducing overall risk is a great idea. However, from the point of view of the people who are used to working from home, sometimes it is a great inconvenience. During Iteration C2, Carlos, the DAD coach, suggests that now that the team has gelled, they can safely experiment with letting people work from home occasionally. After discussing it with the team, Danny and Ashok begin to work from home one day a week. During Iteration C4, Debbie also starts to work at home one day a week. To support dispersed team members, two tools are adopted, one for agile management and the other for chat. The agile management tool enables the team to manage their work item list and task board, and will generate critical management reports, including the iteration burndown and the release burnup charts. The chat tool supports both text-based chat and videoconferencing, a feature that team members frequently use for collaborating and joining the daily coordination meeting whenever someone is working from home.
•Improvement tracking. During Iteration C1's retrospective, Carlos suggests that the team begins to measure their improvement efforts. He describes a DAD technique called measured improvement, where at each retrospective the team rates how well they are doing at addressing the previously identified improvements. The way that they do this is that they go through the list one at a time, and each person is asked to rate, on a scale of 1–10, how well they think the team is doing. The votes are captured in a spreadsheet so that the trends can be publicly displayed in the team room as one of the information radiators. If an improvement starts to trend downward, that's an indication that the team needs to refocus on that improvement. When an improvement levels off for many iterations, that's an indication that the team has either plateaued and may need to consider another strategy, or that it has adopted the improvement as best it is going to and doesn't need to focus on it any longer.
•Guided continuous improvement (GCI). Carlos also introduces the team to the concept of GCI,1 where the team leverages the DA tool kit to help them increase their rate of process improvement. When the team identifies issues that need to be addressed within a retrospective, Carlos shares his advice, but more importantly, guides the team in using the book Choose Your WoW! to identify potential improvements to experiment with.
•The team evolves. The team interviews several potential candidates to join the team, and brings on David, a developer who is new to BigBank, during the second week of Iteration C3. At the start of Iteration C5, Doug, an existing BigBank employee, joins the team after being interviewed by them. Because collaboration and teamwork are so important to agile teams, they need to take responsibility for determining who will join them. Their personnel department, or “human resources” department in many cases, can often narrow down the number of likely candidates for the team to consider. However, the team itself should make the final decision as to who becomes a member, not a manager who is external to the team. Doug is joining the team because Debbie is rotating onto another team. While the team has grown to become extremely efficient and intends to stay together long term, it is a good idea to rotate team members to keep them stimulated and to give them new learning opportunities. The experience bringing Doug onto the team was different enough for the personnel department to realize that agile required a different approach by them. This, in turn, motivated them to start learning about Disciplined Agile (DA)'s People Management process blade2 and seek help from a qualified enterprise agile coach.
•Parallel independent testing. As Carlos suggested, toward the end of release one, the team works with BigBank's quality assurance (QA) team to have them begin testing new MAP builds in parallel to their development efforts. Negotiations to do this began in Iteration C1, but it takes almost two months for the QA team to make the right people, Thomas and Tatiana, available to do this testing. The challenge is finding people with sufficient skills to do this work. Many of the testers on the QA team are “old school,” and need to work from a detailed requirements specification. That isn't going to work for agile parallel independent testing, because it is unlikely that there is going to be such a specification. Nor is it likely that the team is going to take on the overhead of doing so, simply to support this type of testing, and that was certainly the case with MAP. Instead, people with a more sophisticated testing skill set are needed, particularly around exploratory testing and end-to-end, cross-system integration testing. Carlos needs to work with Quincy, the QA manager, to educate him on how agile teams work and on the implications for his team. All of this takes time, and it isn't until the end of Iteration 4 that parallel independent testing begins. To report potential issues back to the team, Thomas and Tatiana capture them using a commercial test management tool, which everyone on the QA team uses. Ideally, they want to use the agile management tool they adopted, but Quincy wouldn't support that. In the end, Tara and Patricia need to review the reported issues. Tara captures them in the agile management tool so that the team can easily access them, and Patricia prioritizes them so that the issues are treated as another type of requirement. Quincy is surprised by how few defects were found by the independent testing team. Carlos explains that due to the greater focus of all team members on quality and testing each other's work, good agile teams don't let many defects “escape” the iterations. For agile teams, done means done—built and tested. Having said that, the independent testing effort was valuable because it found defects that the team didn't, in particular around cross-system integration bugs that likely would have escaped into production. This, in turn, enabled the team to identify weaknesses in their test strategy and then address them.
•Vacations. Danny takes two weeks off for vacation during Iteration C4, and Terry takes one week during Iteration C6. Members of agile teams, just like members of other teams, get to take vacation time. When they are gone, the team's capacity to perform work is correspondingly lower, something that is taken into account during iteration planning.
•Sharing their learnings. During the middle of Construction, BigBank began to roll out DAD to other delivery teams. Because the MAP team members were the most experienced DAD practitioners within BigBank, they were asked to share their knowledge with these other teams. At first, there was a push to move some people out of the team to help seed DAD knowledge into the other teams. This would have helped those teams, but would have decimated this one, and the decision had been made previously to see whether a long-lived (stable) team would work well within BigBank. Instead, the team decided to hold several “lunch-and-learn” sessions, where a team member would give a 10-minute talk about their experiences on a DAD team and then answer people's questions for the rest of the hour. BigBank had lunch brought in for each of these sessions to help support the effort. Carlos began to work with another DAD team, and two other Certified Disciplined Agile Coaches (CDACs)—Constance and Carter—were hired to help with BigBank's agile transformation efforts.
The Transition phase takes one week this release, down from two weeks the previous release. This is the result of the need for less training required for the support staff and better overall quality due to improved testing during Construction. Additionally, the team benefits from their investment in automated deployment scripting and automated regression testing in the prior release. The training sessions were one hour each, compared with three hours each during the first release, allowing the team to schedule all of the training sessions on a single day. The team's approach to testing, in particular the adoption of TDD and parallel independent testing, dramatically reduces the need for hardening. However, there is still some hardening required, something that Carlos hopes to coach the team about in the next release.
Release 3
Release 3 is shortened to three months, with one week for Inception, six two-week Construction iterations, and one week for Transition. The following events of interest occurred during this release:
•Training. During Inception, the team attends a two-day training workshop in acceptance test-driven development (ATDD), also called behavior-driven development (BDD). After their experiences with unit-/design-level TDD during release two, team members feel that they should take the next step and start trying TDD at the requirements level.
•Requirements envisioning. Patricia runs a half-day requirements elicitation workshop with a group of stakeholders to flesh out new requirements. This occurs on the second day of the Inception phase, and the session is attended by all team members so that they can hear directly what the stakeholders wanted.
•Improved quality. A few minor issues, all severity three or four, are found during the warranty period following the second release. These issues are prioritized and added to the team's work item list. Victoria, the vice president of IT at BigBank, publicly commends the MAP team in one of her monthly newsletters for their high quality and stakeholder satisfaction ratings that they are consistently receiving.
•Continuous deployment. Although the team started to automate their deployment during the first release, it wasn't until now that it became truly continuous. By the end of Release 3, the team was able to set up deployment so that successful builds are automatically promoted from someone's workstation to their team integration environment, and from the team integration environment into their demo environment and a staging environment for the independent test team. Using feature toggles, the team starts to “turn on” smaller pieces of functionality on a more frequent basis so they can do more frequent deployments to production. By architecting these small changes as isolated microservices, the stakeholders feel more comfortable deploying these changes without large regression testing efforts.
•Adoption of acceptance test-driven development (ATDD). After their training, the team actively started to adopt ATDD tooling and techniques. Tara invested about half of her time during the first three iterations pairing with Barbara, Doug, and Danny, helping them to gain solid ATTD skills.
•Organizational support for continuous integration (CI)/continuous deployment (CD). BigBank's first step toward Disciplined DevOps is to form a small Center of Excellence (CoE)3 comprised of three DevOps engineers to start. Their initial focus is on setting up a continuous development tool chain and helping the existing DAD teams to adopt their infrastructure. The MAP application team had these tools set up already, although the other teams newer to agile were still in the process of doing so.
•Strategizing around DevOps. The MAP team began working with the DevOps CoE to share ideas for how they could rearchitect the MAP application to support more effective development and operations of it. The team commits to a detailed discussion of this at the beginning of their next release.
•Short Transition phase. Transition is, once again, one calendar week. However, it actually took a bit less than three hours to run the regression test suites one more time and then run the deployment script. The rest of the week was the warranty period for that release (although no major errors were found), and the start of the Inception phase for release 4.
Releases 4+
The team continues improving the way that they work over time. The following key events occurred during release 4:
•The team reworks MAP with DevOps capabilities. The team decides to invest most of a week at the beginning of release 4 to learn about BigBank's DevOps strategy and what they need to do to fit into it. The team was interested in DevOps for two reasons: First, they wanted to delight their customers by providing better levels of service to them. Second, because DAD teams work in an enterprise-aware manner, it made sense to take advantage of BigBank's DevOps resources wherever possible. In release 4, their implementation of feature toggles expanded to address all functionality, not just the new features that were implemented in release 3. This required a bit of refactoring on the part of the team, reducing their ability to delivery new functionality during release 4 to make this investment. In release 5, the team added operational monitoring capabilities into the MAP application and ran an A/B test to help them identify which implementation of a new feature was preferable to their end users. Interestingly, they found this very easy to do, as the result of their existing support of feature toggles and with a framework provided by the DevOps CoE. In release 6, they added real-time chat into MAP to enable end-user feedback and support.
•The release cadence was tightened. To support Berhard's desire to shorten time to market, and to reduce overall risk on the MAP endeavor, the team actively tightened their release cadence over time: Release 4 was three months in length; releases 5 and 6 were both two months; and for release 7 and beyond, they settle on releasing every four weeks.
•The life cycle evolves. As the result of discussions during several retrospectives during release 5, the team decides to begin moving toward following DAD's Lean life cycle. This is viable because the team has relatively easy access to stakeholders and because the team has gelled so well. At the beginning of release 6, they choose to drop the concept of working in two-week iterations in favor of a continuous stream of development. Their first improvements are to hold weekly planning sessions and demos, and hold retrospectives whenever someone on the team decides to call one (because they have an issue to discuss). During release 8, the team replaces the weekly planning session in favor of replenishment sessions and pulls one work item at a time into their process (instead of a week's worth of work). The team is now capable of taking a new feature from the backlog and quickly building and deploying it to customers. This short cycle time reduces work in process and provides a quick return on investment.
•Operating and supporting MAP. With the team building, fixing, and deploying their own work, they have, in effect, adopted the DevOps practice and mentality of “you build it, you run it.” Carlos points out that the team has successfully evolved into DAD's Continuous Delivery: Lean life cycle, which is the most advanced and effective delivery life cycle. This move dovetails well into BigBank's overall DevOps strategy.
•The team stops sizing. Part of the decision to settle on a one-month release cadence is to provide a consistent and predictable schedule to stakeholders. This enables the team to convince their stakeholders to allow them to adopt a #NoEstimates4 strategy, where the team no longer takes on the overhead of sizing their work.
BigBank's Overall Disciplined Agile Transformation
The experiment with DAD on the MAP team is clearly a success. This was due to:
•BigBank's willingness to invest in training and coaching the team in Disciplined Agile techniques;
•The team's willingness to learn together to adopt the agile mindset;
•The team's willingness to try nonsolo work techniques such as pair programming and agile modeling, which enabled them to learn new skills faster; and
•Team members with good technical skills initially who were also willing to expand those skills as needed.
In parallel, five other teams are also deploying successfully with DAD, prompting BigBank to start adopting it across many development teams. This wider adoption started while the MAP team was in the middle of release 2. A year later, BigBank now has 15 teams following a DAD-based approach.
Although BigBank's initial DAD adoption was successful, it was clear to them that their inefficiencies went far beyond software development. They had been hearing about DevOps for a while now, and several of their new hires were eager to go in that direction, so BigBank decided to shift gears.
Having not learned their lessons from jumping on the Scrum bandwagon, they immediately decided to jump on the DevOps bandwagon. They invested in building a DevOps CoE, staffed by DevOps engineers who had experience with DevOps at smaller startup companies in the past. They started by promoting the classic team-based vision, as depicted in Figure 12.1, and by building out the tooling infrastructure to support CI/CD within delivery teams. Interestingly, the MAP team had been unknowingly evolving their approach in this direction already, so they, in effect, became one of the DevOps pilot teams for BigBank. DevOps requires a mindset that promotes collaborative, learning-centric ways of working that are supported by agile practices. On the development side, teams tend to adopt a continuous delivery (CD) approach and also take on many of the responsibilities that were the purview of operations and support organizations in the past.
Although the strategy depicted in Figure 12.1 is great for individual teams, it quickly failed in BigBank's multiteam, multiparadigm environment, where regulatory compliance and working with legacy systems are the norm. BigBank needed to ensure that security, data management, and fundamental business operations concerns were addressed by their DevOps strategy. Furthermore, the realities of having hundreds of teams working in parallel requires a bit of release management/coordination that goes beyond the team-based strategy. Similarly, their support/help desk strategy needed to address the fact that they had over 2,000 systems in production, tens of thousands of internal users, and millions of external users. Luckily, their Certified Disciplined Agile Coaches (CDACs) pointed them to Disciplined DevOps,5 the workflow for which is depicted in Figure 12.2.
With the help of their DevOps Center of Excellence (CoE), BigBank was able to put the organizational, process, and tooling infrastructure in place to support a Disciplined DevOps strategy. This took time and effort, but eventually they were able to build out an approach that reflected their enterprise context. Throughout this, the MAP team was an active partner of the DevOps CoE in that the CoE ran experiments with them to try out potential strategies before rolling them out to other teams. For example, the MAP team was the first one to try out the A/B testing framework developed by the CoE (during release 5) and the real-time chat framework (during release 6). In effect, working with the MAP team to work with the chat framework was the agile CoE's way to run an experiment to explore its viability in practice. The end result was that the CoE convinced BigBank to purchase an enterprise license for it. Behind the scenes, from the MAP team's point of view, the DevOps CoE coached the data management and security teams in how to work in an agile manner with the MAP team. BigBank's Disciplined DevOps strategy proved to be a good start, but was insufficient in the long run. They quickly realized that the rest of their IT strategy needed to evolve to support business agility throughout their entire organization. This required them to adopt a Disciplined Agile IT (DAIT) strategy to support their overall Disciplined Agile Enterprise (DAE). These two aspects of the Disciplined Agile (DA) tool kit are overviewed in the appendix of this book.
1See PMI.org/disciplined-agile/gci.
2See Ambler, S., & Lines, M. (2017). An executive's guide to Disciplined Agile: Winning the race to business agility. Disciplined Agile Consortium.
3See PMI.org/disciplined-agile/people/centers-of-excellence.