Preface
This is an amazing, and perhaps chaotic, time to be involved with the technology industry. The demand for talent, skills, and commitment to excellence has never been higher. Developing software and systems has become a remarkably complex task, with many factors affecting the success of the development effort. Learning new development frameworks and adapting legacy systems to meet the need for continued growth and flexibility require the modern IT professional to be able to press forward, while understanding the limitations imposed by earlier conditions. Teams may be located in one specific “war” room or distributed across the globe and frequently working at different hours of the day, with varying languages, cultures, and expectations for how they will operate on a daily basis. The project itself might involve writing complex application software or customizing a vendor package as part of a systems (versus software) development effort. The competition for specialized technical resources motivates many organizations to allow flexible work arrangements, including telecommuting along with choosing office locations convenient to attract local candidates. Technology professionals must often choose between the demands of high-paying (and often stressful) opportunities and trying to maintain a comfortable work-life balance. The Internet has clearly become the backbone of commerce, and companies are expected to continuously align themselves with growing Web capabilities in order to achieve and maintain success.
Successful organizations need to support complex technologies, most often with a significant Web presence. Even companies going out of business are expected to have a functioning Web presence capable of handling the peak transaction requirements of customers and other users. In practice, these complex development efforts necessitate effective processes and methodologies to meet both the demands of today and those that will surface in the future. This book will describe best practices for designing the appropriate application lifecycle management (ALM) processes necessary to successfully develop and implement complex software systems, whether your team is writing the code or customizing a system that you have purchased from a vendor. We will discuss both agile and non-agile methodologies to empower the reader to choose the best approach for your organization and the project that you are trying to complete. Our goal is to increase and enhance the reader’s understanding and ability to apply these principles and practices in your own environment. We often work in the imperfect world of having to support lifecycle methodologies that are not always optimal. In fact, we are usually called in when things get really bad and an organization needs to figure out how to incrementally improve processes to improve quality and productivity. In our opinion, the most effective methodology to emerge in the last decade has been agile.
Agile configuration management and, by extension, agile application lifecycle management have become two of the most popular software development methodologies in use today. Agile has resulted in indisputable successes boasting improved productivity and quality. My 25-year (and counting) career has always involved software process improvement with a particular focus on configuration management. As a practitioner, I am completely tools and process agnostic. I have seen projects that successfully employed agile methods and other efforts that thrived using an iterative waterfall approach. Still, all organizations need a reliable and repeatable way to manage work, allowing full traceability and clear, complete communication. Years ago, the IT community looked to the software development lifecycle (SDLC) to guide us in understanding what needed to be done by each member of the team on a daily basis, although the SDLC process documentation often sat on the shelf along with the outdated requirements specification from the latest software or systems development effort. When purchasing commercial off-the-shelf (COTS) products became popular, we began to use the term systems development lifecycle to refer to the work, at times spanning months or even years, to customize and configure COTS systems. We will discuss the differences between software and systems lifecycles further in Chapter 1. Whether applied to software development or systems customization and configuration, the SDLC, in practice, generally only referred to the required activities to create and maintain the system. Some vendors marketed efforts to customize and configure their solution as production lifecycle management (PLM) solutions. Many companies struggled with improving programmer productivity, and some tried to use the Software Engineering Institute’s (SEI) Capability Maturity Model (CMM). These efforts often had limited success, and even those that succeeded had limited return on their investment due to the excessive cost and effort involved. The SEI chartered a series of Software Process Improvement Networks (SPINs) throughout the United States, which provided speakers and opportunities to meet with other professionals involved with software process improvement. I had the pleasure of serving for many years on the steering committee of one of the SPINs located in a major city. Today, most of the SPIN presentations focus on agile practices, and most of the attendees are interested in establishing scrums, iterative development, and agile testing. Agile has certainly had a major impact on software process improvement, although not without its own inherent challenges and limitations. Application lifecycle management has emerged as an essential methodology to help clarify exactly how software development is conducted, particularly in large-scale distributed environments. ALM typically has a broader focus than was considered in scope for an SDLC and helped to resolve many of the most common challenges, such as providing a comprehensive software development methodology helping each member of the team understand what needed to be done on a daily basis. At its core, the ALM enhances collaboration and communication. DevOps is a closely related approach that is particularly effective at driving the entire ALM.
DevOps and the ALM
DevOps is a set of principles and practices that improve communication between the development and operations teams. Many DevOps thought leaders acknowledge that DevOps is also effective at helping development interact with other groups, including quality assurance (QA) and information security (InfoSec). In this book, we will broaden that definition to show that DevOps is essential to enhancing communication between every other group that participates in the ALM. We will be discussing DevOps throughout this book and in detail in Chapter 12. DevOps principles and practices are applicable to the interactions between any two or more groups within the organization and are essential in driving the ALM.
The initial goal of any ALM is to provide the transparency required to enable decision makers to understand what needs to be done and, most importantly, approve the project, including its budget and initial set of goals and objectives. Providing this transparency is precisely where IT governance plays an essential role in helping to get the project approved and started.
IT Governance
Effective software methodology today must have a strong focus on IT governance, which is essentially the control of the organizational structures through effective leadership and the hands-on management of organizational policies, processes, and structures that affect information, information-related assets, and technology. Fundamentally, IT governance provides the guidance necessary to ensure that the information technology organization is performing successfully and that policies, processes, and other organizational structures are in place so that essential organizational strategies and objectives are achieved. Organizations with excellent IT governance enjoy improved coordination, communication, and alignment of goals throughout the entire enterprise. IT governance is closely related to, and must align with, corporate governance in order to ensure that information technology can help drive the business to success and profitability. The initial goals of IT governance are to define policies, clarify the objectives of corporate governance, and ensure that the information technology organization aligns with the business to provide essential services that enable the business to achieve its goals. From an IT service management perspective, IT governance helps drive the development and deployment of services that help achieve value; these include fitness for purpose (utility) and fitness for use (warranty). IT governance is also concerned with establishing the most efficient organizational structure that will allow technology to be delivered successfully as a valued corporate asset. In this context, management is also responsible for providing adequate resources while maintaining necessary budget and financial controls.
IT governance cannot exist in a vacuum. Management requires accurate and up-to-date information in order to make the best possible decisions. Department managers and teams must provide valid and relevant information so that management understands the risks, challenges, and resources required for success. IT governance enables the business by ensuring that informed decisions are made, that essential resources are available, and that barriers to success are removed or identified as risks. Risk management is essential to effective IT governance. Risk is not always bad, and many organizations thrive on well-defined risk. IT governance provides the essential information that is needed to enable senior management to identify and mitigate risk so that the organization can successfully operate within the global business environment.
IT governance has the unique view of seeing the organization as part of an ecosystem, with the focus on competitors and outside forces, including regulatory requirements that affect the business and business objectives. Information security and business continuity are special areas of focus for IT governance, as it is essential to ensure that the business can operate and thrive regardless of challenges, such as competitive forces and other external pressures. Other considerations of IT governance include data privacy, business process engineering, and project governance.
Closely related to IT governance, and often mentioned in the same sentence, is compliance with regulatory requirements, industry standards, and internal audit requirements. IT governance helps all relevant stakeholders within the entire organization understand what they need to do in order to meet and comply with all regulatory requirements. Effective IT governance enables businesses to implement organizations with organizational structures that operate successfully, while providing the necessary information to help senior management make the decisions, which then propel the organization to achieve improved quality, productivity, and profitability. With this guidance from senior management, the next step is to ensure that all of the stakeholders understand their roles and what needs to be done on a day-to-day basis. This is exactly where application lifecycle management comes into the picture.
Application Lifecycle Management
Application lifecycle management (ALM) evolved from the early days of process improvement to provide a comprehensive software development methodology that provides guidance from requirements gathering, to design development, all the way through to application deployment. In practice, ALM takes a wide focus, with many organizations establishing an ALM to manage their entire software and systems delivery effort. Even nondevelopment functions such as operations and the help desk can benefit from a well-defined ALM. Some organizations implement ALM in a way that would not be considered agile, using a waterfall model that has a heavy focus on completing the tasks in each phase before moving on to the next. Configuration management, consisting of source code management, build engineering, environment configuration, change control, release management, and deployment, has been a key focus of ALM for some time now. Another central theme has been applying agile principles to support and improve configuration management functions.
Agile CM in an ALM World
Agile configuration management (CM) provides support for effective iterative development, including fast builds, continuous integration, and test-driven development (TDD), that is essential for successful agile development. In a comprehensive lifecycle methodology, agile CM can make the difference between success and failure.
The Definition of Agile ALM
Agile ALM is a comprehensive software development lifecycle that embodies the essential agile principles and provides guidance on all activities needed to successfully implement the software and systems development lifecycle. Agile ALM embodies agile CM and much more. Agile ALM starts with tracking requirements with “just-enough process” to get the job done without any extra steps, or what agile enthusiasts often call “ceremony.” This is often accomplished by creating user stories, which need to be under version control just like any other artifact. Testing throughout the lifecycle also plays a significant role in agile ALM and may even be used to supplement requirements documents that are often intentionally kept brief in an agile world. Agile ALM focuses on iterative development that requires a minimum amount of process, with an emphasis on proven practices that include iterative development, strong communication, and customer collaboration. Understanding agility is much easier when we examine the process methodologies that have come before.
Understanding Where We Have Come From
Understanding where we have come from should always start with reviewing the essential principles of process improvement. For example, most practitioners will confirm that process improvement needs to be iterative, pragmatic, and continuous. One excellent source of valid principles for process improvement may be found in the work of W. Edwards Deming. Many of Deming’s teachings1 provide principles that are practical and form the basis of quality management.
1. Deming, W. Edwards. (1982). Out of the Crisis. Cambridge, MA: Massachusetts Institute of Technology, Center for Advanced Engineering Study.
Principles of Process Improvement
Process engineering focuses on defining the roles, responsibilities, and essential tasks that need to be accomplished in order for the process to be completed successfully. Processes themselves need to be understood, clearly defined, and communicated to all stakeholders. Complex processes are most often created in a collaborative way and usually take several iterations before they are comprehensive or complete. Processes may need to change over time and may be loosely defined early in the lifecycle, but usually require greater clarity and discipline as the target delivery date approaches. Too much process is just as bad as not enough. Therefore, the best processes are Lean with few, if any, extra unnecessary steps. Quality must be built into the process from the very beginning, and it is essential to maintain an honest and open culture to achieve effective processes and process improvement.
Right-sizing processes is essential for organizational success, as is effective communication. Application lifecycle management has its own terminology, which needs to be understood for effective communication among all stakeholders.
Terminology
Every effort has been made to use terms that are consistent with industry standards and frameworks. Please contact us via social media with any questions or via the website for this book as noted on the next page.
Use of “I” versus “We”
Although we do everything as a team, there were quite a few places where it was much easier to write in the first-person singular. Bob is also much more technical and “hands-on” than Leslie, so when you see first-person singular “I” or “my” you can safely assume that this is a first-person account from Bob.
Why I Write About Agile CM, DevOps, and Agile ALM
Agile configuration management and agile application lifecycle management provide the basis for essential best practices that help a software or system development team improve their productivity and quality in many significant ways. DevOps and the agile ALM help ensure that teams can produce systems that are reliable and secure while maintaining high levels of productivity and quality. As is often the case, early life experiences have greatly shaped my view of the world.
Blindness and Process Improvement
Much of how I have approached my life and career has been influenced by the fact that I had a significant visual handicap growing up that could not be safely corrected until I was in my late teens. Consequently, I used Braille, a white cane, and lots of taped recordings (“talking books”). Even when I gained useable vision, at first it was only for short amounts of time because my eyes would fatigue quickly, and then for all practical purposes I would be temporarily blind again (or what we blind guys like to refer to as “blinking” out). My beloved ophthalmologist, Dr. Helen Grady Cole, once noted that my handicap made me successful because I learned to achieve against all odds and “move mountains” when necessary. No doubt, you will hear some of that fierce determination in these pages. I am very comfortable when approaching the seemingly impossible and viewing it as quite doable. You will get to hear about some of my experiences in the motorcycle gang of para- and quadriplegics with whom I proudly associated during my most formative years.
Classroom Materials
University professors who would like to use our book for a class in software engineering or software methodology are encouraged to contact us directly. We are glad to review classroom materials and would guest lecture (via Skype where travel is impractical) if appropriate and feasible. Obviously, we are glad to answer any and all questions related to the material in the book.
Website for this Book
Please register on our website at http://agilealmdevops.com and connect with us on social media to engage in discussions on Agile ALM and DevOps!
Who Should Read This Book
This book will be relevant for a wide variety of stakeholders involved in application lifecycle management.
Development managers will find guidance regarding best practices that they need to implement in order to be successful. We also discuss the many people issues involved with managing the software development process.
How This Book Is Organized
This book is organized into 22 chapters divided into four parts. Part I consists of five chapters defining the software development process, agile ALM and agile process maturity, and rapid iterative development. Part II covers automation, including build engineering, automating the ALM, continuous integration, delivery, and deployment. Part III covers establishing essential IT controls, including change management, operations, DevOps, retrospectives, agile in non-agile environments, IT governance, and audit and regulatory compliance. Part IV covers scalability, including integration across the enterprise, agile ALM in the cloud, ALM on the mainframe, QA and testing, personality, and the future of ALM.
Part I: Defining the Process
Chapter 1: Introducing Application Lifecycle Methodology
This chapter introduces application lifecycle management by explaining what you need to know in order to define an ALM that will help you implement a comprehensive and effective software or systems lifecycle. We discuss how to implement the ALM using agile principles in a real-world, pragmatic way that will help guide the activities of each member of your team, whether you are creating new software or customizing a commercial package. Systems lifecycles are a little different than a software development lifecycle and are usually associated with obtaining (and customizing) a project from a solution vendor. Commercial off-the-shelf (COTS) software is commonly used today to deliver robust technology solutions, but they often require some effort to customize and implement. In this chapter, we introduce the core concepts and then build upon them throughout the rest of the book
Chapter 2: Defining the Software Development Process
This chapter helps you understand the basic skills of how to define the software development process. Defining the software development process always sounds straightforward until you actually start trying to do the work. Many tasks are involved with any software or systems lifecycle, and a well-defined process must provide guidance on exactly what needs to get done and who is responsible for completing each task.
Chapter 3: Agile Application Lifecycle Management
In this chapter we discuss the core strategies that will help you create a flexible and robust software and systems development lifecycle while ensuring that the values and principles of agility are understood and maintained.
Chapter 4: Agile Process Maturity
In this chapter, we will examine the factors that affect agile process maturity from a number of different perspectives. Many technology professionals find that they must implement agile processes in a large organizational context, including managing teams that are composed of many scrums, totaling scores or even hundreds of developers working from a variety of locations. Scalability is certainly an essential aspect of agile process maturity. Mature agile processes must be repeatable for each project in the organization and have sufficient support for project planning. We also need to understand how process maturity affects non-agile development methodologies, including waterfall and other process models.
Chapter 5: Rapid Iterative Development
In this chapter, we discuss rapid iterative development and its impact on software methodology that came long before agile development reached its level of popularity common today. We consider what we learned from rapid iterative development and how it may be applied in practical situations today.
Part II: Automating the Process
Chapter 6: Build Engineering in the ALM
This chapter helps you understand the build within the context of application lifecycle management. We discuss essential aspects of automating the application build, with particular attention on techniques for creating the trusted application base. We also discuss baselining, compile dependencies, and embedding version IDs as required for version identification. We discuss the independent build and creating a fully automated build process. Building quality into the build through automated unit tests, code scans, and instrumenting the code is an important part of this effort. Finally, we will discuss the ever-challenging task of selecting and implementing the right build tools.
Chapter 7: Automating the Agile ALM
In this chapter we discuss how application lifecycle management touches every aspect of the software and systems lifecycle. This includes requirements gathering, design, development, testing, application deployment, and operations support. Automation plays a major role in the agile ALM, which sets it apart in many ways from other software development methodologies. We also explain why it is essential to appreciate the big picture at all times so that the functions that you implement are in alignment with the overall goals and structure of your ALM.
Chapter 8: Continuous Integration
In this chapter we explain that continuous integration (CI) is an essential practice that involves putting together code that has been created by different developers and ascertaining if the code components can compile and run together successfully. CI requires a robust automated testing framework, which we will discuss further in Chapter 20 and provides the basis for ensuring code quality through both static and instrumented code scanning. Continuous integration often involves merging together code that has been written by different developers and is essential for code quality. The fundamental value of this practice is that integrating a small amount of code as early as possible can avoid much bigger issues later. It would be difficult to imagine an effective ALM that does not embrace integrating code frequently, although I will also discuss a couple of situations where it is difficult or even impossible to achieve. When code cannot be integrated early and often, there is increased risk, which must be identified and addressed. It is also important to understand that continuous integration relies upon many other practices, including continuous testing, effective build engineering, static and instrumented code analysis, and continuous deployment, discussed in Chapter 9.
Chapter 9: Continuous Delivery and Deployment
In this chapter we explain how continuous deployment (CD) is a methodology for updating production systems as often as necessary and generally in very small increments on a continuous basis. It would be difficult to understand continuous deployment without discussing continuous integration and delivery. The terminology for CD has been confusing at best, with many thought leaders using the terms continuous delivery and continuous deployment interchangeably. We discussed continuous integration in Chapter 8. Continuous delivery focuses on ensuring that the code baseline is always in a state of readiness to be deployed at any time. With continuous delivery, we may choose to perform a technical deployment of code without actually exposing it to the end user, using a technique that has become known as feature toggle. Continuous deployment is different from continuous delivery in that the focus is on immediate promotion to a production environment, which may be disruptive and a poor choice from a business perspective. We will help you find the right balance so that you can support your business by promoting changes to production as often as desired.
Part III: Establishing Controls
Chapter 10: Change Management
In this chapter we examine how change management is a broad function that helps us plan, review, and communicate many different types of planned and emergency (unplanned) system modifications. Changes may be bugfixes or new features and can range from a trivial configuration modification to a huge infrastructure migration. The goal of change control is to manage all changes to the production (and usually QA) environments. Part of this effort is just coordination, and that is very important. But part of this effort is also managing changes to the environment that could potentially affect all of the systems in the environment. It is also essential to control which releases are promoted to QA and production. Change control can act as the stimulus to all other configuration management–related functions as well. Throughout this chapter we will discuss how to apply change management in the ALM.
Chapter 11: IT Operations
In this chapter, we will discuss how to create an effective IT operation group that is aligned with your agile ALM. The IT operations organization is responsible for maintaining a secure and reliable production environment. In large organizations, operations often resembles a small army with too many divisions to navigate that is also often held responsible when things go wrong. Developers, working on the bleeding edge of technology, often regard their colleagues in operations as lacking technical skills and ability, which is true in so far as operations resources tend to focus more on the day-to-day running of the systems. Understanding these different perspectives is a key aspect of our DevOps approach to the agile ALM.
Chapter 12: DevOps
In this chapter, we discuss DevOps as a set of principles and practices intended to help development and operations collaborate and communicate more effectively. DevOps is truly taking the industry by storm and, in some circles, reaching almost mythical proportions. I hear folks suggesting that DevOps can help solve almost any issue, which given the versatility of its cross-functional approach, is a view that has some merit, but some groups are losing sight of what this methodology is all about and how it can really help us implement the ALM.
Chapter 13: Retrospectives in the ALM
This chapter discusses the practical application of retrospectives to support application lifecycle management. The first section of this chapter will examine the main function of retrospectives, namely, to evaluate what went well and what needs to be improved. But that’s just the beginning. Getting accurate information from all stakeholders in a retrospective can be very challenging. If you are successful, the retrospective can help drive the entire ALM process. Retrospectives require leadership, and this chapter will provide guidance on how to succeed if you are responsible for implementing this function. We will discuss how to employ retrospectives to support ITIL incidents and problem management, along with other industry standards and frameworks. Crisis and risk management are also key considerations along with IT governance and compliance. Retrospectives take on a different tone when used as vendor management tool. We will complete this chapter by considering how much process is necessary, how to deal with politics (or, more accurately, relationships), and the use of effective metrics to drive the process improvement journey.
Part IV: Scaling the Process
Chapter 14: Agile in a Non-Agile World
In this chapter we discuss that being agile in a non-agile world can be very difficult, and at times even seem impossible to accomplish. We have often found ourselves in organizations that insisted on a waterfall approach. What is most difficult is trying to predict things that are just not possible to ascertain up-front. Many are unaware that waterfall was originally envisioned as an iterative process because today it seems that some organizations expect their employees to be able to predict the future to a degree that is simply not reasonable. The real problem is that these are the same organizations that expect you to make the project actually conform to the plan once it has been developed and approved. Any deviations may be perceived as a lack of planning and proper management. Being agile in a non-agile world can be very challenging and is fraught with its own set of risks and pitfalls.
Chapter 15: IT Governance
In this chapter we discuss how IT governance provides transparency to senior management so that they can make the best decisions based upon the most accurate and up-to-date information. The ALM provides unique capabilities for ensuring that managers have the essential information necessary for evaluating their options. From the CEO to the board of directors, information must often be compartmentalized due to the practical constraints of just how much information can be consumed at any point in time. Achieving this balance empowers your leadership to make informed decisions that help steer your organization to success.
Chapter 16: Audit and Regulatory Compliance
This chapter explains that audit and regulatory compliance require that you establish IT controls to guide the way in which the team works. Your auditors may be internal employees or external consultants engaged by your firm. The internal audit team usually focuses on internal policy, whereas external auditors are often engaged to ensure compliance with federal regulatory guidelines. Although many technology professionals look at audit and regulatory compliance as just something that you have to do, others view it as an obligatory yet unfortunate waste of time and effort. Our focus is on establishing effective IT controls that help avoid both defects and risk. This chapter will help you understand how to use audit and regulatory compliance to ensure that you prevent the sorts of major systems glitches and outages that we read about all too often.
Chapter 17: Agile ALM in the Cloud
This chapter explains how cloud-based computing promises, and often delivers, capabilities such as scalable, virtualized enterprise solutions; elastic infrastructures; robust services; and mature platforms. Cloud-based architecture presents the potential of limitless scalability, but it also presents many challenges and risks. The scope of cloud-based computing ranges from development tools to elastic infrastructures that make it possible for developers to use full-size test environments that are both inexpensive and easy to construct and tear down, as required. The first step to harnessing its potential is to understand how application lifecycle management functions within the cloud.
Chapter 18: Agile ALM on the Mainframe
This chapter explains how to apply the agile ALM in a mainframe environment. Application lifecycle management on the mainframe typically enjoys a specific workflow. Despite a culture that lends itself well to step-by-step defined procedures, ALM on the mainframe often falls short of its potential. Sure, we can specify steps of a process, and everyone accepts that process tollgates are necessary on the mainframe. But that does not mean that our mainframe processes help to improve productivity and quality. It is essential that ALM on the mainframe be agile and help the team reach their goals and the business achieve success.
Chapter 19: Integration across the Enterprise
This chapter explains that understanding the ALM across the entire organization requires an understanding of the organization at a very broad level. It also requires that you understand how each structure within the company interfaces with the others. In DevOps, we call this systems thinking when we are examining an application from its inception to implementation, operation, and even its deprecation. DevOps principles and practices are essential in integrating the ALM across the organization.
Chapter 20: QA and Testing in the ALM
In this chapter we discuss how quality assurance (QA) and testing are essential to any software or systems lifecycle. Most technology professionals view the QA and testing process as simply executing test cases to verify and validate that requirements have been met and that the system functions as expected. But there is a lot more to QA and testing, and this chapter will help you understand how to establish effective processes that help ensure your system functions as needed. DevOps helps us build, package, and deploy software much more quickly. Too often, the QA and testing process cannot keep up with the accelerated deployment pipeline. DevOps cannot succeed without excellent QA and testing.
Chapter 21: Personality and Agile ALM
In this chapter we examine key aspects of human personality in the context of the agile ALM. Top technology professionals often have remarkable analytical and technical skills. However, even the most skilled professionals often have great difficulty dealing with some of the interesting behaviors and personalities of their colleagues. Implementing an agile ALM requires that you are able to work with all of the stakeholders and navigate the frequently thorny people issues inherent in dealing with diverse groups of very intelligent, albeit somewhat idiosyncratic, and often equally opinionated, people
Chapter 22: The Future of ALM
In this chapter we discuss what lies ahead for the agile ALM.