Chapter 3. Agile Application Lifecycle Management
Agile application lifecycle management requires a flexible and creative approach to managing the software and system lifecycle that goes beyond the traditional software or systems development lifecycle (SDLC). Agile ALM builds upon the SDLC by empowering the team to customize their approach to meet the unique needs of the business and the project that is being undertaken. The agile ALM also typically takes an iterative approach. Unfortunately, agile ALM has suffered from a dark side of rigidity, with too many “experts” prescribing their view of the one true path to agility. This book presents a pragmatic approach to dealing with real-world challenges, which departs from the agile purist view. We are challenging the status quo in agile development by suggesting that many, if not most, companies choose to take a more hybrid approach than many agile purists would like. We take a very wide view of the agile ALM, including functions that many other authors overlook. We also believe that your colleagues are smart enough to create processes that are appropriate to their situation and the organizational culture in which they must operate. We also focus on taking a true DevOps view in defining our processes by including many stakeholders as we describe our own experience in defining the agile ALM. In this chapter we discuss the core strategies that will help you create a flexible and robust software and systems development lifecycle while ensuring that the values and principles of agility are understood and maintained.
3.1 Goals of Agile Application Lifecycle Management
The goal of establishing an agile ALM is to create a flexible and yet effective software and systems lifecycle that adheres to agile values and principles and ensures that you produce high-quality software within the budget and the agreed-upon timeframe. The agile ALM must ensure that your team knows precisely what to do at any point in time and maintains just enough process to avoid costly mistakes while achieving the desired goals. The agile ALM also has the essential goal of ensuring that you do not suffer from having so much ceremony that your team is bogged down in processes that hamper successful execution. Agile ALM has the essential goal of achieving a balance between process and flexibility. In this chapter, we also have the goal of helping you understand how to take a step back and define your own lifecycle.
3.2 Why Is Agile ALM Important?
Having too much process is just as bad as not having enough. But that said, the team needs to understand what to do on a daily basis. This includes providing a roadmap so that each member of the team understands exactly what they need to do on a daily basis. We take a very wide view by including all stakeholders who could potentially be affected, including the folks on the help desk and internal IT audit department who need to test and verify IT controls. The agile ALM is important because it gives you a methodology for scaling your controls up when necessary and keeping your processes light when it makes sense to do so. This is no easy task, and the agile ALM helps provide the guidelines for evaluating when a process step should be enforced to avoid costly mistakes and when extra ceremony is not only unnecessary, but could get in the way of accomplishing your goals.
3.3 Where Do I Start?
Achieving agility requires a cultural shift. You need to start by ensuring that your team understands what agility really means in very practical terms. You also need to ensure that agile principles become part of your organizational culture. The agile transformation is a journey that involves understanding and evolving behaviors and the basic assumptions inherent in the software and systems delivery process. This is precisely where each group in the organization may have a rather different experience and even a slightly different journey. Agility for the operations team may be a different experience from that of the developers—even in the same organization. You need to start by understanding the paradigm shift itself.
Undergoing a paradigm shift can be difficult and disruptive for many organizations that attempt but fail to achieve the full benefits of agility. The agile transformation is not just a change in what you do on a day-to-day basis—it is truly a cultural change as well.
3.4 Understanding the Paradigm Shift
Implementing an agile ALM often involves a major shift in the way that most folks view the software and systems lifecycle. Understanding what this shift requires in terms of attitude and behavior requires both a cultural transformation and a change in the way that the work is organized, tracked, and completed. The key to understanding the paradigm shift is to embrace agile principles and practices within the culture of your organization. Implementing agile in a bank or other large financial services company is not the same as implementing agility in a smaller, privately owned startup. Yet it may actually be the bank that reaps the benefits despite the challenges of right-sizing agility in a large organization that must adhere to audit and regulatory requirements. You need to understand the organizational culture that will affect any process improvement initiative.
At the core of understanding this paradigm shift is adopting a rapid and iterative approach to software and systems development.
3.5 Rapid Iterative Development
Rapid iterative development has been around a long time, and its meaning and methodology have evolved over time. The basic premise of rapid iterative development is that it is much easier to develop applications in an iterative fashion than to attempt to build the entire system in one cycle. Iterative development helps by allowing the team to get a piece of the code working without trying to complete all of the features in one lifecycle. This approach helps tame technical complexity by allowing the team to understand the technology and what they are trying to develop through a series of iterations. Many organizations today approach iterative development using scrum, but iterative development has been around a lot longer than scrum. Closely related is rapid application development (RAD).
We will examine RAD as it is discussed by industry leaders, including Barry Boehm and James Martin, in the next section. In Chapter 5 we will discuss how to create your own approach to rapid iterative development. But for this chapter you need to understand how rapid iterative development affects the development of agile application lifecycle management.
Understanding our mistakes is very important. It is equally important to learn from incidents and problems.
3.6 Remember RAD?
Rapid application development (RAD) became popular1 in the late 1980s and 1990s as technology professionals began to look for faster approaches2 to developing software. Industry experts, including James Martin and Barry Boehm, provided their views on how to develop software faster. Efforts to use computer-aided software engineering (CASE) were also intended to take us from the prevailing waterfall methodology to more efficient approaches to developing software. Getting the computer to write code for us was certainly ideal. But another approach was to take common everyday tasks such as report writing and make them easier to accomplish in less time. For this purpose we evolved from third-generation languages (e.g., Cobol, FORTRAN) to programming frameworks that promised more productivity without requiring as much training and expertise. Fourth-generation languages (4GL) were popular in the early eighties—often as report writers that did not require comprehensive software programming as was necessary with Cobol, FORTRAN, and PL/1. These efforts had some success, but more importantly, they provided a direction for thinking of better strategies to developing software and systems.
1. Martin, James (1991). Rapid Application Development, New York: Macmillan Coll Division.
2. www.casemaker.com/download/products/totem/rad_wp.pdf
Although RAD provided a path to thinking out of the box, agility provided a much more robust foundation that included effective principles and practices. As noted, DevOps consists of a set of principles and practices that help improve communication and collaboration. Although our practices may be different based upon the needs of the organization, the principles are always the same. We take the same approach to using agility to guide our approach to agile processes. This means that when we think through designing processes, we always keep in mind the agile manifesto and the agile principles. The next section explains how we use agile principles to guide our approach to application lifecycle management.
3.7 Focus on 12 Agile Principles
The agile ALM requires that we understand and apply agile principles. Let’s review these principles3 and suggest some ideas for how they can be applied in real-life situations:
3. http://agilemanifesto.org/principles.html
1. Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
We are always focused on meeting the customer needs, and the best way to accomplish this goal is to deliver working prototypes for review and verification during the software development process. This requires true partnership, but is not a blank check to modify any and all requirements. The best approach is fair and reasonable for both the customer and technology professionals creating the system.
2. Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.
Changing requirements come from improving our understanding of what the system needs to do and from changes in the surrounding environment. The customer may not actually know how to ask for what he or she really needs up-front. During the development process, the requirements become clearer to both the end user and the professionals writing the code.
3. Deliver working software frequently, from every couple of weeks to every couple of months, with a preference for the shorter timescale.
Delivering working software helps ensure that you fully validate and understand the requirements. Each iteration will bring you closer to creating a complete and comprehensive system. You will also discover what you forgot or did not completely understand with each iteration of the software.
4. Businesspeople and developers must work together daily throughout the project.
Many perspectives must be considered during the software development process, but the most important view is that of the business and product users. Creating a technical masterpiece is of little use if it does not meet the needs of the end users.
5. Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
People are your greatest resources. Companies today often forget this lesson and fail to develop their most important assets, which are smart, motivated individuals who know how to work as part of a team.
6. The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
Face-to-face conversation is indeed far superior to any other mode of communication. That said, many of us work remotely, and teams are often dispersed throughout the world. Successful endeavors place a premium on facilitating face-to-face conversations to build relationships and establish common understanding.
7. Working software is the primary measure of progress.
The goal of any effort should be to create working software. Requirements and test documents that sit on the shelf do not make a complete system.
8. Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
Working long hours can get old very quickly. The best working environments maintain a healthy work-life balance and can be sustained for a long time.
9. Continuous attention to technical excellence and good design enhances agility.
Technology changes quickly, and maintaining technical excellence can be difficult to achieve. Choosing the right design and architecture is essential for successful agile iterative efforts.
10. Simplicity—the art of maximizing the amount of work not done—is essential.
Technology professionals sometimes make things too complicated. Truly elegant design is simple and thereby much more capable of adapting to required change if necessary.
11. The best architectures, requirements, and designs emerge from self-organizing teams.
Self-organizing teams are often the most high performing, which results in the best architectures, requirements, and designs.
12. At regular intervals, the team reflects on how to become more effective and then tunes and adjusts its behavior accordingly.
Process improvement is a journey that must be improved upon and adapted over time. Achieving excellence requires that the team regularly reflect on how they can improve and evolve.
Agile principles help us understand and implement the guidance provided by the agile manifesto. Make sure that you understand the manifesto and can implement your processes in an effective way.
3.8 Agile Manifesto
Manifesto for Agile Software Development4
We are uncovering better ways of developing
software by doing it and helping others do it.
Through this work we have come to value:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on
the right, we value the items on the left more.
We discuss the agile manifesto and its practical usage throughout this book.
Agile makes effective use of fixed timebox sprints.
3.9 Fixed Timebox Sprints
The notion of developing code in iterations is not new to agile. Waterfall was envisioned by Winston Royce as being an iterative process. What is relatively new is the idea that we should fix the length of each iteration. Scrum has made popular the notion of sprints with durations of two to four weeks. I have seen engineering firms where they could not develop any useful hardware chip in less than two to three months, and then their iterations matched the lifecycle of the hardware. The real point here is to pick a length of time that allows you to produce a useable piece of code that can be shown to the end user or his representative for feedback and the collaborative evaluation of requirements. Many teams do very well with sprints of two to four weeks as made popular by scrum. But there are also very good reasons to sometimes have longer or shorter sprint durations. There are several factors to consider when choosing the length of the sprint.
The first is to consider how long it will take to produce a useable piece of code as we have described. Perhaps a more subtle consideration is how long can members of the team stay focused. A six-month sprint would certainly not seem very useful or viable. But sometimes four weeks is not enough time to really get very much done. My view is that choosing the length of time for a sprint and whether or not the timeframe really has to be fixed is a decision best made by the team and reviewed as needed during the sprint. Another consideration is technical risk. Sometimes you really don’t know how long a particular task will take. This occurs when you are working with a new technology and therefore there may be considerable technical risk that needs to be managed. Choosing the right length of the sprint and whether or not the timeframe should be fixed at all should be done in collaboration with the customer or end user.
Customer collaboration is also an important consideration.
3.10 Customer Collaboration
Customer collaboration is essential for any comprehensive and usable ALM. I always try to ask the customer or customer representative for their input on how often they can receive a release and the impact of software upgrades to the end user. It is my experience that successful DevOps-driven deployment pipelines are often able to generate changes much faster than customers can handle receiving the changes.
The customer is also the most important stakeholder to be involved with feedback loops by helping the team understand the priorities as well as risks that need to be mitigated. Customers often surprise me with their view of how we should approach the release and deployment of milestone releases. This input is essential and really helps the team stay on focus. Sometimes, I learn that it is worth taking risks in order to achieve time-to-market, and other times I am told that the end user cannot handle the constant flow of changes.
Customers are also the best resource for validating and clarifying requirements so that we focus on the right features to be delivered. Most importantly, customers have the best focus for guiding the quality assurance and testing efforts too. Collaboration and communication are essential for any successful organization. This is also why I believe that DevOps must include the end-user perspective. Customers should be collaborating directly with the developers, testers, and service desk personnel. DevOps is not just about development and operations, and customer collaboration is of paramount importance.
The most important area of collaboration with customers is ensuring that we all understand exactly what needs to be built.
3.11 Requirements
Many good books explaining how to write requirements include guidance on how to create agile user stories. The process of defining requirements provides important information to the developers so that they know what needs to be coded. The process also helps the customers get a better idea of what they have asked for. Requirements change often due to the business landscape, including essential regulatory and audit controls. Far more common is our understanding of the requirements themselves, which evolve as the customers see the system and get to start testing it.
Understanding and specifying requirements must be an iterative process because users themselves often do not fully understand their own requirements up-front. The agile ALM helps clarify requirements, enabling the technology professionals to understand what they are building while also helping the end user understand what is required.
No one wants to have books of documentation sitting on the shelf, and agile development has famously deemphasized the importance of having written documentation. I view this as a case of right-sizing the documentation process.
3.12 Documentation
Understanding how much documentation is required can be challenging. The agile manifesto notes that working software is more important than comprehensive documentation. But the truth is that the amount of documentation required depends largely upon what the system is used for and the potential impact of inadequate documentation. For example, having requirements specified for a missile defense system or a life support system may be an absolute requirement. When comprehensive documentation is not practical, then sometimes it can be enough to use test cases to supplement the available information. In fact, it is common for test cases to contain the same information that might be documented in a requirements document.
Release notes are essential for understanding exactly what changes are included in a version of the code. User documents can also help with training and support. It is important to have sufficient documentation to avoid mistakes without creating books that are simply going to sit on the shelf.
3.13 Conclusion
The agile ALM helps put structure around the entire software and systems lifecycle. How much structure may depend upon the purpose of the system, the industry involved, and the culture of the organization itself.