4.1 Goals of Agile Process Maturity

This chapter focuses on helping you establish an agile development process that is light but effective and, most importantly, repeatable. This is not an easy goal to accomplish. In many ways, agile shifts the focus away from implementing processes that contain comprehensive controls, or as agile enthusiasts describe as being high in ceremony. Ceremony, in this context, really means bureaucracy or, more specifically, laden with excess controls and “red tape.” The goal of this chapter is to help you implement agile processes that are Lean,³ repeatable, clearly defined, measureable, and adhere to the principles defined in the agile manifesto.⁴ We will also discuss how to coexist (or perhaps survive) in non-agile environments. The first step is to understand process maturity in an agile development environment.

4.2 Why Is Agile Process Improvement Important?

Any software or systems development process must continually evolve to meet the ever-changing challenges and requirements of the real world. Agile is no different in this respect. Agile practitioners also know that agile is not perfect and many agile projects have failed for a variety of reasons. Agile processes need to evolve and improve using the very same values and principles that are expected in any agile development effort.

4.3 Where Do I Start?

The toughest part of implementing mature agile processes is figuring out where to start. I usually start by assessing existing practices and fully understand what works well and what needs to be improved. It is common for me to find that some practices work just fine in one organization that I would have expected were the source of problems. I find that sometimes less-than-perfect processes and procedures may not really be the pain point that one would expect—usually because of the organizational culture. Obviously, trying to fix something that isn’t broken will not be very successful, and you will likely find that you do not have much credibility with the key stakeholders if they just don’t feel that you are focused on solving the most important problems. In these situations, I communicate my concerns and then focus on what they want me to work on, although I know that they will come back to me and ask for help when things go wrong.

4.4 Understanding Agile Process Maturity

Agile process maturity can be understood in many different ways. The most obvious measure of agile process maturity could be in terms of the degree to which the practices adhere to the agile manifesto and the agile principles.⁵ I usually refer to this as a purity measure to indicate the degree to which the process follows authentic agile principles. As a consultant, I am usually called in to help with situations that are less than perfect. This pragmatic reality does change the fact that we want to approach implementing the agile ALM in a manner that adheres to and benefits from agile values and principles.

4.4.1 Adherence to the Principles

Mature agile requires that we adhere to the agile principles that we reviewed in Section 3.7. In this book we seek to educate you on software methodology in a way that empowers you to apply these principles and create a software lifecycle that is best for your project and your organization. One of the ironies that we often see is that some agile practitioners are the least “agile” people in the world, insisting on there being only one right way to become truly agile. I disagree, and we hope to share the best practices in creating an agile ALM that you can tailor to your own special requirements.

4.4.2 Repeatable Process

Agile processes, just like any other process, must be repeatable. It does not help to have an agile ALM unless it can be used repeatedly to achieve the desired results. We have seen many agile teams struggle with repeatability because they depended upon the guidance of individual players rather than understanding that agile is still a process that should yield the same results, regardless of who is performing the task—assuming the proper level of skills and training.

4.4.3 Scalability (Scrum of Scrums)

Organizations often pilot agile methodologies in one particular group with spectacular results. The truth is that the participants in the agile pilot are often hand-picked and among the best resources in the organization. But agile processes must be scalable so that other teams within the organization can also be successful. We discuss the critical success factors for scalability throughout this book and then tie them together in Chapter 19, “Integration across the Enterprise.” If you want a scalable process, then you need to start by ensuring that your approach is comprehensive.

4.4.4 Comprehensive (Items on the Right)

Agile processes must be comprehensive so that everyone understands what needs to be accomplished, including interdependencies and deadlines. The agile manifesto aptly notes the following:

4.4.5 Transparency and Traceability

Mature agile processes are transparent and traceable. Transparency is fundamental because you want everyone to understand what is being done and how their work impacts (and is impacted by) the work of others. You also want to be able to verify that steps have been completed. Processes that are transparent are easier to understand and follow, ensuring that everyone understands the rules of the road. Being able to go back and verify that each step was completed successfully is also essential, particularly when regulatory compliance is required. In addition, you want to be able to provide transparency to senior management through effective IT governance.

4.4.6 IT Governance

IT governance provides visibility into the organizational processes and existing operations so that senior management can make accurate decisions based upon the information that is available. I always explain to my colleagues that IT governance is essential because this function enables senior management to make the right decisions based upon accurate and up-to-date information. You can even look at IT governance as managing the right information “up” to those who are in the position of making decisions. In some large organizations, agile projects may be in progress at the same time as non-agile projects. Mature agile processes must be able to successfully coexist in these real-world hybrid environments.

4.4.7 Coexistence with Non-agile Projects

The elephant in the room for agile is the number of non-agile projects that exist within an organization that is working to implement agile. We have seen many organizations where existing non-agile projects were already underway, or perhaps existing team members were just not comfortable with taking an agile approach. Mature agile application lifecycle management often requires coexistence with non-agile projects. Coexistence is a sign of maturity, as is aligning with industry standards and frameworks.

4.4.8 Harmonization with Standards and Frameworks

Many organizations must follow the guidance provided in industry standards, including ISO 9000 or frameworks such as ISACA COBIT or the ITIL v3 framework. Mature agile processes can easily align and harmonize with the guidelines provided by these well-respected industry standards and frameworks. This includes meeting the requirements of Section 404 of the Sarbanes-Oxley Act of 2002 or safety standards such as those commonly required by the automotive, medical engineering, or defense industries. Mature agile helps improve quality and can align well with IT controls that are reasonable and appropriate.

4.4.9 Following a Plan

Planning is essential for the success of any significant endeavor. Too many agile enthusiasts erroneously think that they don’t need to create comprehensive plans to guide the software and systems development effort. The dark side of planning is that sometimes those creating the plan refuse to admit what they do not know. Mature agile processes plan as much as possible and communicate those plans effectively. Unknowns should be identified as risks, which are then mitigated as part of the risk management process. Many years ago W. Edwards Deming noted the importance of “driving out fear.” Agility admits when it does not have enough information to specify the details of a plan. Decisions are made at the “last responsible moment.” Mature agile processes embrace comprehensive plans but also do not attempt to plan out details that cannot yet be specified.

4.4.10 Continuous Process Improvement

Process improvement is a journey that must be continuously harnessed throughout the software and systems lifecycle. The mature agile process embraces continuous process improvement at both a deep technical level and at a pragmatic business level. Improving your technical processes is mostly focused on avoiding human error while maintaining a high level of productivity and quality. Improving your business processes can be a bit more complicated.

4.5 Applying the Principles

Implementing the agile ALM requires that you understand the agile values and principles and, more importantly, how to utilize them in practical terms. Technology projects require a deep understanding of exactly what the system should do and how it should work. These are important details that are typically expressed in terms of requirements. There are many different types of requirements, from system-level response time to functional usage, including navigation. Many professionals use epics⁷ and stories⁸ to describe requirements in high-level terms. Writing and maintaining a requirements document is often less than fruitful, with most requirements documents out of date even before they have been approved by the user. Agile takes a pragmatic approach to requirements management that focuses on working software instead of writing requirements documents that are often of limited value.

4.6 Recognition by the Agile Community

Agile development is part of a large ecosystem with an active and involved community. Mature agile processes are aligned with agile principles and are recognized by the agile community. Much of my work involves taking innovative and even risky approaches when I customize software methodology to meet the unique needs of often complex organizations. Although I always maintain confidentiality, I find it effective to write and publish articles that describe my approach to DevOps and agile process maturity. Sometimes, my views are well accepted by the agile community, and other times the reaction can be quite significant. I actually use my esteemed colleagues in the agile community as a feedback loop to continuously improve my own process methodologies.

4.7 Consensus within the Agile Community

The agile community can be both opinionated and very vocal. It can also be difficult to gain consensus. You need to expect that there will always be a diversity of views and opinions expressed in the agile community. Sometimes, views are expressed in rather emphatic terms. In fact, it is the great irony that some agile practitioners are the least agile people I have ever worked with, insisting that agility can be practiced in only one particular way. My view is to enjoy the plurality of opinions, looking for consensus when I can find it and also understand that sometimes experienced practitioners will have differing points of view. This is especially true when confronting some of the more thorny issues in understanding agility. One of these considerations is what agile process maturity is not.

4.8 What Agile Process Maturity Is Not

The agile process is not an excuse to skip documenting and tracking requirements, so agile process maturity is also not an excuse for failing to implement enough “ceremony” to get the job done. Although agile has boasted many fabulous successes, it is also not without its failures. One of the biggest problems with agile today is folks just going along with what they are told without questioning and reflecting upon the effectiveness of the agile process.

4.9 What Does an Immature Agile Process Look Like?

Immature agile processes can resemble software development in the Wild, Wild West. If your team delivers in an inconsistent way and lacks transparency and predictability, then you are likely dealing with a lack of process maturity. You might even be successful from time to time—but maturity involves a lot more than occasional heroics. There are many other potential problems with agile.

4.10 Problems with Agile

Too often agile has become an excuse to work in a very loose and ineffective way. We have seen agile teams use the agile manifesto as an excuse to not plan out their projects and also to not communicate their plans with others. Sometimes teams also fail to document and track requirements, which can lead to many problems, including a high incidence of defects. We have also seen teams that used agile as an excuse to not document their work. Mature agile processes provide the right balance of planning, requirements management, and documentation to avoid mistakes and get the job done. We recall one major incident in a large bank where a vendor claimed to be employing agile and shipped a release that was not really ready to be seen by the customer.

4.11 Waterfall Pitfalls

Agile enthusiasts have long described the many pitfalls and problems inherent in following the waterfall software methodology. In Chapter 14, “Agile in a Non-Agile World,” we will discuss these and other challenges as well, but also acknowledge that there are times when waterfall is the only choice. From the perspective of agile process maturity, we need to understand exactly where waterfall is problematic so that we do not make the same mistakes in our agile or hybrid agile processes.

4.11.1 Mired in Process

We often see organizations that are simply mired in their waterfall processes. These groups typically take a very rigid approach to planning and requirements gathering. Although sometimes waterfall makes sense, it is essential to always be pragmatic and avoid getting mired in your own processes. When this happens, we have seen teams where it actually became part of the culture to pretend to be following the process.

4.11.2 Pretending to Follow the Process

One of the most dysfunctional behaviors we often see is organizations that require complete adherence to waterfall processes, which results in team members being forced to pretend to be following these rigid waterfall processes. In these circumstances we find people who feel pressured into creating and following plans even when they really do not have all of the necessary details, or creating requirements specifications that document features that are not yet well understood. If management forces employees to follow waterfall in a rigid and dysfunctional way, then they really have no choice but to smile and pretend to follow the process. The better way is to create mature agile processes that include both the items on the left of the agile manifesto and the items on the right.

4.12 The Items on the Right

The agile manifesto teaches us to value individuals and interactions over processes and tools, working software over comprehensive documentation, customer collaboration over contract negotiation, and responding to change over following a plan. But mature agile processes must have robust processes and tools, adequate documentation, and plans. You also don’t want to try to engage with customers without well-written contracts and clear agreements. The items on the right side of the agile manifesto are actually very important. It is also important to adjust your ceremony for the environment and culture in which your organization is operating.

4.12.1 Adjusting Ceremony

Agile processes are said to be “light” in terms of ceremony, which means that they are not overly burdensome with rigid verbose rules and required procedures, which are inherent in creating IT controls. Mature agile processes are able to adjust the amount of ceremony required to avoid mistakes and still get the job done. Although right-sizing the amount of process is a must-have, so is coexisting with non-agile processes when necessary.

4.13 Agile Coexisting with Non-Agile

There are many times when agile simply must exist with non-agile processes. This is the real world that many agile practitioners find so difficult to accept. We work with many large banks and financial services firms where agile must coexist with non-agile processes. This is often the case when large organizations must have IT governance in place to ensure that senior management can make decisions based upon adequate information.

4.14 IT Governance

IT governance is all about providing information to senior management so that the right decisions can be made. Many agile processes suffer from failing to provide adequate information to senior managers. Mature agile processes provide enough information so that senior management can make the right decisions in support of the development effort. IT governance is closely aligned with providing transparency.

4.14.1 Providing Transparency

Mature agile processes provide the transparency that is essential to help all stakeholders understand the tasks that they have to complete and especially how their work affects the work of other members of the team. Processes, and especially workflows, help the entire team understand what needs to be done on a day-to-day basis. This is exactly where having just enough process can help you get the job done and avoid costly mistakes. Above all, you want to have an ALM that follows the agile principles.

4.15 ALM and the Agile Principles

Mature agile processes should obviously adhere to agile principles. The agile ALM is customer-centric and facilitates the early and continuous delivery of valuable software. We welcome changing requirements, even late in development, and harness change for the customer’s competitive advantage. The agile ALM should help deliver working software by frequently facilitating daily collaboration between all stakeholders, including businesspeople and developers. Projects should be built around motivated individuals with the environment and support they need while encouraging face-to-face interactions. Working software is the primary measure of progress.

4.16 Agile as a Repeatable Process

Mature agile processes must be repeatable above all else. Even the best process will be of little value if it cannot be used reliably across all of the projects and groups involved with completing the work. Closely related is the need for scalability.

4.16.1 Scalability

Scalability means that the mature agile process can be used reliably across the enterprise. We often find that this is exactly where organizations struggle the most. We will review some tactics to help ensure that your processes can scale to the enterprise in Chapter 19, “Integration across the Enterprise.” Another key aspect of agile process maturity is ensuring that you deliver on time and within budget.

4.16.2 Delivering on Time and within Budget

We see many agile teams struggling with the reality that no one is going to give them a blank check and tell them to take their time on delivering results. Mature agile processes should provide enough planning and structure to help ensure that the software can be delivered on time and within budget. Unless your senior management team is clairvoyant and just anticipates your team’s every whim, you will need to communicate what you need to get the job done. This should include a clear idea of the timeframe required and the budget that will help ensure success of the project. This is particularly essential when considering the quality of the software that you deliver.

4.16.3 Quality

Mature agile processes must ensure that quality is a top priority. This requires a strong focus on robust automated testing and benefits greatly from thorough test planning. Well-written test cases can help supplement even incomplete requirements documents. Mature agile processes cannot survive without a strong focus on quality and testing. W. Edwards Deming, regarded by many as the father of quality management, was well known for explaining that quality must be built in from the very beginning of the software and systems lifecycle. This is particularly true in mature agile processes.

4.17 Deming and Quality Management

Many of the lessons from Deming are a main focus of the agile ALM, and we will point them out throughout this book. Testing is essential, but there are many other ways to build quality into the agile ALM.

4.17.1 Testing versus Building Quality In

Application testing is a must-have. But quality has to be built in from the very beginning. Code reviews and inspections are among the tools that help ensure quality is built into the product from the very beginning. Ensuring that requirements are well defined is essential for ensuring high-quality systems. The agile ALM provides a comprehensive framework of best practices to help build quality into the product from the very beginning. It is also the best way to help improve productivity.

4.17.2 Productivity

Technology professionals often find themselves mired in the quagmire of trying to get work done efficiently. The mature agile ALM helps avoid mistakes and rework that is so often the reality of today’s software and systems development effort. One of the most effective practices to improve productivity is rapid iterative development.

4.18 Agile Maturity in the Enterprise

Implementing processes across any large organization can be very challenging, and agile process maturity should be measured across the enterprise. While we are not advocating comparing groups to each other, which could actually be counterproductive, we do want to have common criteria to help each team plan their own process improvement efforts. It is best to understand processes within the group context itself. We have seen teams that had technical flaws in their processes, tools, or procedures and in one group these issues presented a significant challenge, but for another it was almost irrelevant. For example, we have seen teams lack strong version control procedures but somehow manage to avoid problems that we would have expected through sheer force of will or even manual controls. Obviously these situations are optimal, but still each team may have a very different view of their priorities and pain points. We implement agile processes consistently across the enterprise, while still understanding that each team may have a somewhat different culture, environment, and priorities. We can manage this balance by establishing the goals and objectives while understanding that there could be some difference in processes, tools, and procedures.

4.18.1 Consistency across the Enterprise

Process maturity models can be helpful in establishing common criteria to help ensure consistency across the enterprise. We also use industry standards and frameworks as a source of consistent best practices to implement across the enterprise. For example, we might ask a team to explain how they implement automated build, package, and deployment, including their procedures to verify and validate that the correct code has been deployed. Teams are often quite up-front about what they are doing well and what could be improved. Helping each team to focus on its own perceived priorities is essential for successful process improvement. But there is also room for ensuring that industry best practices are implemented consistently across the firm. This work requires excellent communication and even some good marketing of the new approach.

4.18.2 Marketing the New Approach

We never assume that teams will just automatically agree to implement the best practices that we advocate. Sometimes, it is best to help a team create its own plan. We balance this approach with enterprise process improvement efforts to essentially market industry best practices using the latest processes and tools. Throughout this effort it is essential to continuously focus on process improvement.

4.19 Continuous Process Improvement

The most effective way to implement mature agile processes is to take an agile and iterative approach to implementing the agile ALM itself. This means that you need to be continuously working toward excellence. Learning from mistakes is par for the course, and effective processes should also be self-correcting.

4.19.1 Self-Correcting

Process improvement is not without its challenges. The important thing is to ensure that your processes correct themselves and evolve. Being able to improve your processes is much easier if you are able to measure them and demonstrate progress over time.

4.20 Measuring the ALM

We tend to be wary of overengineering the measurement process, as some teams tend to try to game the measurement. With any measurement approach, it is important to consider validation up-front. This is especially true with regard to metrics.

4.20.1 Project Management Office (PMO) Metrics

Metrics, including those used in project management, can be very important. More importantly, selecting valid and verifiable metrics is key to ensuring a successful measurement approach leading to quantifiable process improvement. Our experience has been that less is more in this case, and the best approach is to select a few metrics that are valid and verifiable. Establishing an in-house metrics program is very important. It is also important to ensure that your vendors do the same.

4.21 Vendor Management

Vendors often have strong sales and marketing functions that sometimes include information on their processes, which can include metrics. It is important for you to review and understand your vendors’ criteria. We have had many times when we were asked to review vendor programs and give our recommendations on ensuring that the vendor approach was aligned with our client’s requirements. It has been our experience that many vendors welcome this input and where there are gaps, they should be understood as well. Although agile process maturity is typically focused on software, we often review processes around hardware development as well.

4.22 Hardware Development

Hardware development is often dependent upon a waterfall approach because half an incomplete circuit chip is often not very helpful. Our effort is to align the agile ALM with the engineering lifecycle required to design and implement hardware. This is often required when we consult with firms that create firmware.

4.22.1 Firmware

Firmware is software that must be created and embedded in the hardware that consists of the complete hardware-software component. We view agile process maturity as part of this alignment and have seen teams succeed quite well even when using a hybrid waterfall approach for the hardware and an agile approach for the firmware.

4.23 Conclusion

There are many factors to consider when creating a mature agile process. We have introduced and reviewed many of the issues involved with creating mature agile processes. The agile ALM needs to be aligned with the technology, environment, and culture of the team and the organization within which it will operate. Rarely do we see teams get this right the first time, and the most successful groups take an agile iterative approach to creating their agile ALM.