Take agility out of its product development context and think of it in its more familiar usage, as descriptive of a person or animal that has speed, grace, and flexibility. Agile individuals can bend and twist, turn on a dime, and switch on the speed when appropriate. Speed is one of many characteristics of agility , but certainly not the only one. The word agilein the physical world indicates an organism that can adapt to its environment, not over eons but by its actions right now to suit the current circumstances. Those blessed with agility are more likely to escape the perils of the moment and land in a dominant position.
Agility is defined as1:
marked by ready ability to move with quick easy grace <an agile dancer>
having a quick, resourceful, and adaptable character <an agile mind>
If we move to a business context, the daily headlines illustrate over and over both the power of agility and the risks inherent in its absence. Those corporate giants, from Citicorp to Chrysler , that can’t transition from historic business models and business cultures are at risk of extinction. Those, from Apple to Amazon , that reinvent themselves and respond to marketplace changes with urgency and innovation are the likely survivors.
Agility is an attribute that now ranks with stability and governance as indicators of a well-run business. For the global consumer we all serve, the pace of change in technology and business models has been positive. New capabilities at lower costs, as we see in the personal computer (PC) and cellphone markets, has now become an expected benefit of the migration to digital technologies . The life cycle of a product is now measured in months, and this speed of product refresh has fueled a global market with high expectations and a short attention span, as new, science-fiction features like voice recognition quickly become old-hat.
The Agile Manifesto: Founding Document
An understanding of the Manifesto for Agile Software Development is critical to anyone wishing to grasp the concepts of agility. The Manifesto is structured as a simple value statement , comparing one type of value with another. For example, the statement that “We value working products over comprehensive documentation ” has been interpreted to mean that agile teams disdain all documentation. In fact, agile teams, and most developers, resist useless documentation that will never be referenced, the sort of ritualized document production that’s become part of many development life cycles. In my experience with high-performing agile teams , they’ll work hard to produce documents, like user stories or test scripts, that add value to the process. What they refuse is to spend time on non-value-producing activities like writing status reports that are never read. In this example, the key idea of the Manifesto is not that only “working products” have value and that “comprehensive documentation ” has none; rather, it challenges product developers and project managers to question, project by project, which processes, tools, documents, plans, and contracts add value to this particular effort, and which ones we might be doing out of habit, or to fulfill a bureaucratic requirement.
In product development , as in most other human endeavors, the pendulum keeps swinging, in this case from highly structured, prescriptive, and predictive methods to more loosely defined, adaptable, and flexible approaches. In my career as an information technology project manager, I’ve been around long enough to see the gyration from ad hoc unstructured “cowboy coding” by “super-programmers,” with no overall plan or project controls, to the tightly constrained and systematized practices of the Capability Maturity Model (CMM) and Project Management Institute (PMI) -sanctioned3 “best practices” that require every step, action, and decision to be documented, traceable, and consistent project to project. We’ve now landed at a position in the continuum that fits the moment: the quest for a specific product development model that suits the project, the team, the enterprise, and the culture.
In contrast with traditional project methods , agile methods emphasize the incremental delivery of working products or prototypes for client evaluation and optimization . Traditional, “waterfall,” sequential project management assumes that, at the beginning of the project, stakeholders can know, articulate, and document their entire set of requirements. It further assumes that developers can review those documented requirements and accurately predict the activities required, the resources needed, and the time and cost of that development. Both real-world experience and academic research indicate that this is a mistaken belief.
The famous Standish Group CHAOS Studies, 4 familiar to project managers worldwide, studied thousands of IT projects for conformity with scope, time, and cost projections. Repeated about every four years, these studies consistently show that about 40% of IT projects are in the “challenged” category, a polite way of saying “circling the drain.” At a Weapons Symposium in 1999 , the results of a Department of Defense (DoD) software spending study were presented. Of $35.7 billion spent by the DoD in 1995 for software, only 2% of the software was usable as delivered. 75% of the software was either never used or was cancelled prior to delivery.5
Other academic research challenged project management methods in use, especially in an IT context. In a 1998 Harvard Business School study of large software projects, authors Austin and Nichols challenged many of the fundamental ideas of IT project management. As they noted6:
The f irst flawed assumption is that it is possible to plan such a large project.
The second f lawed assumption is that it is possible to protect against late changes.
A third f lawed assumption is that it even makes sense to lock in big projects early!
At about the same time, Watts Humphrey, a respected IBM researcher, published his Requirements Uncertainty Principle7 stating that,
for a new software system, the requirements will not be completely known until after the users have used it.
Hadar Ziv of the University of California soon followed with his own Software Uncertainty Principle8:
Uncertainty is inherent and inevitable in software development processes and products.
In an influential article in 2001 that surveyed software development methods of innovative Internet companies, Alan MacCormack, assistant professor at the Harvard Business School, published a review of the history of software development methodologies .9
MacCormack’s “Evolutionary Model of Software Development Methods” illustrates the history of IT systems development methods:
Waterfall Model– sequential process, maintains a document trail
Rapid-Prototyping Model– disposable prototype, helps establish customer preference
Spiral Model– series of prototypes, identifies major risks
Incremental or Staged Delivery Model– system is delivered to customer in chunks
Evolutionary Delivery Model– iterative approach in which customers test an actual version of the software
These academic findings and observations from real-world efforts led many developers and project managers to question the assumptions of traditional product development methodologies. Recognizing problems with existing methods does not, however, solve the predicament of revamping, replacing, or enhancing those processes. In his article on IT project management for Internet companies, MacCormack recommended a set of practices that could begin to replace the traditional methods. These simple precepts have been cited as a central foundation of the movement toward Agile approaches:
Early release of evolving design and code,
Daily build of code and fast turnaround on changes,
Deeply skilled teams.
Developments in industry , especially the lean manufacturing systems pioneered by Japanese firms like Toyota , validated many of the ideas brewing in the software and project management communities. The Standish Group finding, that around 60% of features built in to IT projects are rarely or never used, was also noteworthy. Concepts like focusing on features the customer valued and specifically requested and building in quality upfront rather than “testing it in” later resonated with these communities.
The development of agile methods accelerated in the 1990s, as Scrum was developed at Easel Corporation and Extreme Programming evolved at Chrysler . In the mid-1990s Dynamic Systems Development Method was introduced and quickly adopted in Europe. Finally, in 2001, the Agile Manifesto was created, and agile development methods were on their way toward mainstream acceptance.
Agile methods assume that clients can’t know what they want in detail until they see a prototype. Agile assumes that, as we iterate toward a complete solution, small releases of functionality and value at a time, our stakeholders will guide us to a product that more closely fits their needs and expectations. Each iteration delivers a working product or prototype, and the response to that product serves as crucial input into the succeeding iterations.
Agile theory assumes that changes, improvements, and additional features will be incorporated throughout the product development life cycle, and that change, rather than perceived as a failing of the process, is seen as an opportunity to improve the product and make it more fit for its use and business purpose.
Let’s delve into each of these statements and clarify what they mean.
Individuals and interactions over processes and tools
Process and tools don’t create results, or add value for the user: only skilled people can do that. Processes and tools, when selected and applied judiciously, can offer guidelines and best practices, and can improve productivity, but agile project managers understand that it’s the human interactions, within the team and with the users and customers of the desired product, that grant us the insight to deliver the right results. Many agile advocates look at the Business Process Reengineering (BPR) movement of the late 1990s as the culmination of a trend toward valuing processes over people, as if the perfect process alone could enforce success, and the skills and characteristics of the people were secondary. Agile theory asserts that even the most perfect process can do no more than guide and support skilled, motivated, and driven individuals focused on delivering value.
Working products over comprehensive documentation
This is, as stated, one of the most frequently misinterpreted principles within the Agile Manifesto . The key idea it intends to convey is that a working product or prototype can grant users insight into the actual status of development efforts far more clearly and realistically than any status report or Gantt chart can. Experienced IT consultants know that every software product under development is always proclaimed by the developer to be “90% done” until it either is delivered or fails and is discarded. This agile principle insists that the only way to know the real status of an effort, and to gauge the users’ satisfaction and acceptance of the results, is to deliver something tangible that users can review and assess .
This preference of working software or product prototypes over documentation also serves to emphasize a key idea in agile development: documentation is no substitute for collaboration and interaction. Status reports are distributed through e-mail, Gantt charts are posted on a wall in the project office, and requirements documents are approved via an e-mail chain rather than face-to-face: all of these are examples of the sort of interaction-free exchanges that agile methods are expressly designed to replace with vivid, intensive collaboration.
Customer collaboration over contract negotiation
Projects based on contracts have traditionally implied that the entire scope and range of features required are documented before any development begins, and that the scope, schedule, resources, and often the price, are fixed and unchanging throughout the life of the project. Sometimes these contracts and scope documents are not even created by the teams building the product. To make matters worse, in many projects the contract becomes a substitute for collaboration with the client, and complex or innovative projects often become adversarial wrangles over terms of the contract.
Agile methods aim to move focus from the contract, and from predefined planning documents, to collaboration, with the customer and other stakeholders becoming a central element of the development effort. A key insight of this statement in the Manifesto is the idea that it’s only customers who can define the value of the features development teams deliver, and that they must take ownership of the output, and must be involved throughout the effort, not just at the beginning, when scope documents are developed, and at the end, when acceptance is required.
It’s important to note that this idea can create difficulties on both sides of the development effort. Project teams migrating from traditional to agile methods are often startled by the degree of client interaction required in an agile environment, and can often be troubled by the constant requirement to present their designs and prototypes to client review, and by the need to constantly incorporate changes and improvements as they work. Conversely, many clients, trained to “throw projects over the wall” to developers, often having little contact with the project team between specification and acceptance time, are often unprepared for the intensive interaction expected in an agile engagement. Preparation of teams, and expectation setting regarding this new way of engaging, is a key success factor.
Responding to change over following a plan
Developers and project teams are asked to participate in many types of projects. Some are production-style, low-uncertainty projects, like installing a new server in an existing data center, for which there may be well-known, documented rules and procedures. Others may be innovative, high-uncertainty projects, like creating a new business based on a web-based business model that is untried and untested. Agile theory contends that it’s a fallacy to believe that all types of projects, from production to exploration, can be accomplished using the same, anticipatory, Plan-Do type of project approach, and that exploratory, innovative projects require a more experimental Envision-Explore approach.
Another key element of this philosophy is the changing nature of business itself. As global competition heats up, and the pace of change speeds up, driven by the Internet and our instant communication culture, the requirements of a project can change dramatically from conception to execution. Project teams are no longer simply confronting a series of minor scope changes that can be handled with change control processes; they’re facing projects in which the entire set of business assumptions, technology, and infrastructure can change as the project is underway. Traditional measures of project success, such as compliance with an anticipatory plan and strict control of change requests, are a poor fit for this type of engagement.
Every project is not a candidate for an agile approach. Agile methods and approaches are uniquely suited for innovative, experimental, “never-been-done” projects, and are probably not the best fit for well-documented, low-variability projects . Projects like the 20th implementation of a bottling plant, or the 8th installation of an off-the-shelf software product, while having unique characteristics, probably don’t call for the exploratory, learn-as-we-go attributes in which agile methods excel. Projects such as an innovative software application , with requirements emerging as development proceeds, or a new product development effort for a quick-moving marketplace like consumer electronics, are probably better fits for the methods we are discussing.
In biology, organisms that are dynamic, evolving over time, and adapting to the environment in which they dwell are known as “emergent” organisms . This concept of emergence is frequently discussed in agile documents, applying this biological meaning to the group interaction of a project team, and to the evolving requirements and expectations of the customer. New software, or new products in general, are subject to changing business conditions, changing technologies, changing end-customer expectations, and changing economic conditions, and so are not well-matched with static, predictive methods. These innovative products or services invite the exploration, experimentation, and creativity that are enabled by iterative, incremental, time-bound methodologies.
In addition to the innovative, emergent qualities we’ve discussed, agile projects also require an actively engaged, available customer. Common to most agile methods is the requirement for the customer to be a constant participant in the work sessions, daily meetings, iteration reviews, and brainstorming that exemplify the agile approach. Unlike traditional project methods , the entire requirement and work plan is not derived upfront, and so the customer is not free to participate in a requirements workshop and then disappear until acceptance time. Since agile methods assume that customers can’t know what they want until they see a working example, the customer clearly needs to be there to review the deliverables and comment. The ultimate goal is to deeply integrate the customer into the process as a central member of the development team.
Projects with high regulatory compliance requirements are probably not a great fit for agile methods. Customers whose industry or culture requires that every change go through a formal committee, that every conversation be documented with detailed minutes, and that every new direction, taken in a speculative manner to improve the product, must go through a lengthy approval process, will probably lack tolerance for the lean, nimble, artifact-light approach that agile advocates.
Other important considerations when determining whether to adopt an agile approach relate to skills and culture. It’s inherent in agile theory that the teams being led by these methods are composed of skilled, motivated, driven professionals. In agile environments, leadership and management are focused on serving the team, removing barriers, and fostering a creative environment, not on managing individual granular tasks.
Methods like Extreme Programming explicitly call for highly skilled team members, and explicitly state that they are not suited for large teams. This creates difficulty in many environments, where the staff is composed of a mixture of seasoned veterans and less-experienced journeymen. Most agile advocates would caution against assigning rookies to a key role in an agile team, although bringing them in to observe the process is encouraged as a training tool.
The question of dispersed and virtual teams in agile environments is hotly debated. Since many of the agile methods require attendance at a daily session, there’s been some controversy about whether these can be held through video, collaboration tools, or other virtual techniques. Communication, collaboration, and customer interaction are key tenets of agility, so these concerns can’t be simply discounted. Teams in which project managers and developers are working on many projects at once add to these concerns. There has been effort by members of the agile community to address both scaling and virtual teaming; Jim Highsmith has proposed both scalable agile team compositions and hub-spoke team configurations to address these concerns.
Finally, some industries, and some corporate cultures, may not be suited for agile approaches; highly regulated industries like pharmaceuticals, and safety-conscious industries like transportation, may be wary of shedding the documentation and process associated with traditional methods, for example.
The developers who gathered to develop the Agile Manifesto were more than theorists; they were dedicated practitioners who went on to develop pragmatic development systems based on the agile principles. Kent Beck, Ward Cunningham, and Ron Jefferies, all signers of the Manifesto, are also the co-developers of Extreme Programming, the agile method that attracted the most attention at the time. Alistair Cockburn, another signer, is the architect of the Crystal methods , as well as an author of influential works on use cases . Manifesto signer Jim Highsmith has been the chronicler of the agile movement, and he has done the most to migrate agile concepts from the software to the project management community. Ken Schwaber and Jeff Sutherland, also signers, were the originators of the Scrum methods. We’ll explore each of these variants of agile methods, and discuss criteria for selecting one approach or another based on project specifics.
Extreme Programming
Often known simply as XP, Extreme Programming has gotten an outsize share of interest among agile methods. Extreme Programming drew attention because its audience of software developers was discovering many of its practices in real project work, and because its initial successes, including the well-known Chrysler Compensation project , coincided with the Internet movement , with its need for an approach geared to speed-to-value and exploratory, innovative projects.10
XP has some unique features among the agile methods surveyed here. It is focused on software development and is not touted as a project methodology. It explicitly admits that it doesn’t scale to teams larger than ten or so, and that it’s not well suited to virtual or dispersed teams. Like the Agile Manifesto itself, XP is presented as a series of principles. These include elements such as the following:
The planning game: A recurrent workshop in which developers and customers interact in order to create and refine the “stories” that describe their project.
Small releases: “Every release should be as small as possible, containing the most valuable business requirements.”11
Metaphor: The overall idea of the project; the broad goal told as a narrative or story, to keep the technical jargon to a minimum and build a collaborative vision between developers and customers .
Simplicity: The ideal of simplicity is central to XP. The delivered product itself should be simple, delivering the needed features in the simplest manner without trying to speculate about the bells and whistles that might be useful sometime in the future. The use of methodology and technique should be simple too; XP developers avoid documentation, other than stories and test cases, unless there’s a convincing demonstration of their value to the customer. Finally, XP practitioners strive for elegance and simplicity in their actual coding practices, which leads us to refactoring.
Refactoring: Refactoring is the optimization of the internal code and architecture of software, and it’s a key element of XP. It’s also a response to one of the hazards of iterative design, the danger that the separate iterations will be poorly integrated and internally incompatible. Refactoring is a disciplined approach to rebuilding system internals to ensure simplicity, elegance, and compatibility.
Pair programming: Pair programming takes the concept of software inspections and walkthroughs to the next level. Rather than periodic reviews, the key insight of pair programming is that two skilled practitioners can review and optimize each other’s code, and each other’s coding practices, as they work toward the customer’s goal.
Testing: The vital difference between XP testing practices and traditional practices is that XP insists that test cases for all features be developed upfront, with the stories.
Continuous builds: Going beyond the “daily build” that’s a common practice in many commercial software companies, XP practitioners prefer the continuous build, ensuring compatibility and functionality continuously as the product is created.
Sustainable development: A reaction to the 70-hour week, “death march” project that many developers have experienced, the 40-hour-week standard that XP espouses is consistent with the agile philosophy that creative developers do their best work when they’re committed, energized, clear, and focused.
Available customer: XP calls for the customer to be completely integrated with the development team, available to review features, builds, and tests and to review, assess, and optimize the product as it evolves .
XP’s broad exposure, and the debate it has engendered, has given the project management and software development communities the chance to consider these ideas and to explore their applicability to their work.
Scrum
Ken Schwaber, who, along with Jeff Sutherland, developed the Scrum Process, states that “the core of the Scrum approach is the belief that most systems development has the wrong philosophical basis . . . you can’t predict or definitely plan what you’ll deliver, when you’ll deliver it, and what the quality and cost will be.”12 Schwaber declares that development is not a defined process, which can be repeated time after time, like the implementation of a well-known system, but is instead an empirical process , in which each project is fundamentally different, in which new discoveries must be made every time, and which unfolds differently each time based on the findings of each unique project.
It’s important to distinguish the process from the content of the project in thinking about Scrum. While the Scrum process can be consistent from project to project, the content to be developed, whether a new bit of software, a new system, or a new product, will be sufficiently innovative and unique that the team will discover and integrate new ideas within each project as they work, create, and discover.
Unlike XP, which is designed to be a programming-centric approach , Scrum has more of a project management focus. With the uncertainty of predicting the outcomes quoted above, Scrum instead concentrates on creating a supportive environment for creative development, on close examination of features delivered, and on constant fine-tuning of the product as it is iteratively delivered. As in all agile methods, the supportive environment emphasizes collaboration, constant customer involvement, and the drive toward iteratively delivering real working features.
The Scrum process has three activities :
Pre-sprint , in which the team, in collaboration with the client (known in Scrum as the “product owner”) , creates a set of three “backlogs” that catalog the features visualized for the product under development. The product backlog is the “master” list of features discovered so far, which in turn divides into a release backlog and a sprint backlog. Also defined in the pre-sprint planning is the sprint goal, a business result for the sprint to come.
Sprint , strictly defined in Scrum as a 30-day work cycle in which team members sign up for tasks on the backlog, work with the sprint goal in mind, and participate in daily Scrum sessions. The sprint is the fixed center of the Scrum method, in which the features accepted into the sprint backlog remain constant during the sprint. Product owners can make changes or discard the entire effort once the product of the sprint is displayed for review, but during the sprint the scrummaster manages changes and suggestions to keep the team focused while “sprinting.”
Post-sprint , when the product is displayed to the product owners for review and optimization. At the end of this session, if the project is not complete, the Scrum cycle begins again with a new pre-sprint planning session.
In order to grasp Scrum, it’s critical to understand the backlog concept, as illustrated in Figure 17-1. The entire product, to be delivered when the project is complete, is defined by the features and functions captured in the product backlog. Those features are then categorized by the collaborative team and product owner to determine which features will be included in the next sprint. Those selected features compose the sprint backlog. Not pictured in the diagram but also used in some Scrum projects is the release backlog, for products that envision multiple releases leading up to a final product release.
Figure 17-1. Scrum process diagram
The philosophy of Scrum centers on a few simple ideas: that predicting the outcome of innovative, creative projects is folly, that dedicated, talented teams have the initiative and creativity to build the requested features and functions if enabled to do so by a supportive environment, and that delivery of working functions and constant oversight of the content (rather than the process) makes successful development of inventive products possible.
Lean Development
In 1989, Professor James Womack and consultant Dan Jones published Lean Thinking,13 a survey of the lean manufacturing techniques that helped create the “Japanese miracle” of the late 1980s and early 1990s. They chronicled the ideas of lean manufacturing, with their focus on eliminating waste, creating a smooth “flow” of work on the factory floor, and expecting workers to contribute high skill levels and an ownership mentality. These concepts helped Toyota , the exemplar of these techniques, vault over the traditional giants of the automotive industry. Lean manufacturing theories were highly influential in the creation of lean development (LD). Robert Charette, while not a signatory of the Agile Manifesto, has developed a methodology that has many commonalities with those mentioned so far. Similar, though distinct, ideas have also been put forward by Mary Poppendieck and Tom Poppendieck in their book Lean Software Development 14 (we focus on Charette’s version here) .
LD emphasizes four key success factors that clearly illustrate LD’s compatibility with other agile methods:
Create visible customer value rapidly,
Build change-tolerant software,
Create only necessary functionality and no more,
Aggressiveness, stubbornness, and belief in meeting stretch goals.15
Like Scrum, LD is more of a project management environment than simply a software development one; it consists of three distinct phases, startup, steady-state, and transition/renewal. Rather than the daily “scrum,” it recommends a time-boxed “whirlpool” that, like all agile methods, includes the analysis, design, test, and build activities in each iteration.
Lean development is important not just for its conformance to the ideals of agile development but because the underlying philosophies of lean manufacturing have been accepted by business leaders worldwide, and so come with existing acceptance. This makes the introduction of agile methods in a lean framework more easily accepted, and presents a strategic framework that executives are likely to accept with less resistance.
There are, of course, other variants we haven’t explored here, including Feature Driven Development and Jim Highsmith’s Adaptive Software Development . Dynamic Systems Development Method (DSDM) , for example, is a software development life cycle that embraces many of the foundation ideas of agile development, such as business value focus, iterative release, deep collaboration, and time-boxed delivery.16 Some, like Scrum, are primarily focused on the project management element of product development, while others, like Extreme Programming, are software-centric. The agile ideas outlined here, from the early theoretical findings to the Agile Manifesto, have not just created debate and discussion but have led to the crafting of a variety of disciplined, complete methodologies that bring the theories into real-world practice.
Footnotes
1 Merriam Webster Online Dictionary, www.merriam-webster.com .
2 As Highsmith has done in his book Agile Project Management, we’ve replaced “software” with “product” in the Agile Manifesto, and throughout the book, so agility can apply to all types of product development efforts.
3 The CMM is a development model created for the U.S. Department of Defense by Carnegie Mellon University, which created the Software Engineering Institute (SEI). The term maturity relates to the degree of formality and optimization of processes, from adhoc practices to formally defined steps to managed result metrics to active optimization of the processes.
The PMI is the certifying body of the project management profession and the developer and publisher of the Project Manager’s Body of Knowledge.
5 Stanley J. Jarzombek, “The 5th Annual Joint Aerospace Weapons Systems Support, Sensors, and Simulation Symposium ( JAWS S3).” Proceedings, 1999.
6 Robert D. Austin and Richard L. Nolan, “How to Manage ERP Initiatives,” Working Paper 99–024, 1998.
7 Watts S. Humphrey, A Discipline for Software Engineering (SEI Series in Software Engineering) (Boston: Addison-Wesley, 1995).
8 Hadar Ziv, Deborah J. Richardson, and René Klösch, “The Uncertainty Principle in Software Engineering” (Technical Report 96-33), University of California, Irvine, 1996.
9 Alan MacCormack, “Product-Development Practices That Work: How Internet Companies Build Software.” MIT Sloan Management Review, 42(2): 75–84 (Winter 2001).
11 Kent Beck, Extreme Programming Explained: Embrace Change (Boston: Addison-Wesley, 2000), p. 56.
12 Quoted in Jim Highsmith, Agile Project Management: Creating Innovative Products, Second Edition (New York: Pearson Educational, 2010).
13 James P. Womack and Daniel T. Jones, Lean Thinking: Banish Waste and Create Wealth in Your Corporation, Second Edition (New York: Free Press, 2003).
14 Lean Thinking: Banish Waste and Create Wealth in Your Corporation, Revised and Updated Hardcover – June 10, 2003.
15 Robert Charette, Foundations of Lean Development (Spotsylvania, VA: ITABHI Corp., 2002).