What’s Driving These Changes?

In a classical plan‐driven project management environment, a project was deemed to be successful if it delivered well‐defined requirements on‐time and within the approved budget. In today’s world:

• There is a much higher level of uncertainty, which, in many cases, makes it very difficult, if not impossible, to document firm and well‐defined requirements prior to the start of the project. In that environment, a much more flexible and adaptive approach is needed to further define and elaborate the detailed project requirements as the project is in progress.
• We also live in a very competitive environment where there is a much greater need for creativity and innovation to maximize the business value of the solution. An overemphasis on planning and control can stifle creativity and innovation. In fact, there have been many projects that have delivered well‐defined requirements and met their cost and schedule goals but failed to deliver an acceptable level of business value.

This does not mean that a classical plan‐driven project management is obsolete and no longer useful, but we need to recognize that it does have limitations and fit the project management approach to the nature of the project rather than force‐fitting all projects to a classical plan‐driven approach.

The important thing to consider is “value.” What is the “value” that the project is intended to produce? Meeting a cost and schedule goal for delivering well‐defined requirements certainly has some value in many situations, but it is not necessarily the most important (or only) value in a given project. The choice of an appropriate methodology for a project will depend on a number of factors:

1. The level of uncertainty in the project: A project that has a higher level of uncertainty in the requirements would naturally lean more towards a more flexible and adaptive approach.
2. The level of training and sophistication of the project team: It takes a considerable amount of skill and judgment to use an Agile approach successfully and it should not be attempted unless the team has been properly trained in Agile, the primary roles of Scrum Master and Product Owner are in place, and the necessary tools to support the project are also in place.
3. The relationship with the customer of the project: A classical plan‐driven project is typically based on somewhat of a contractual relationship with the customer:
   • The customer expects the project to be delivered as defined in the requirements within the approved cost and schedule goals.
   • The customer does not need to be heavily involved in the implementation of the project until it is time to approve the final deliverables.

     An Agile approach requires a much more collaborative approach with the customer. The customer needs to share responsibility for the successful completion of the project team by:

   • taking an active role in the project to provide feedback and inputs on incremental results
   • further defining and elaborating requirements as the project is in progress.

This means that there is no longer only one way to do project management and it takes a considerable amount of skill and organizational maturity to fit the most appropriate project management approach to the nature of the project:

• Not only is it important that the individuals responsible for product development are trained and skilled in an Agile approach.
• In addition, the businesspeople who are required to take an active role in the process need to understand how the process works.
• The overall organization needs to be committed to whatever level of organizational change that may be needed to make it successful.

The Impact on the Project Management Profession

This isn’t just a matter of getting another certification—it can require a major shift in thinking for many traditional project managers that will take time and experience to develop. The Project Management Institute (PMI) has created the PMI‐ACP^® (Agile Certified Practitioner) certification, which has been very successful and is a great step in the right direction—but it doesn’t go far enough, in my opinion.

• It doesn’t test whether a project manager knows how to blend Agile and classical project management principles and practices in the right proportions to fit a given situation, and that is the real challenge that many project managers face.
• PMI‐ACP is also not designed around a specific Agile role as many other Agile certifications are and the role that an Agile Project Manager might play is still somewhat undefined.

A lot of the polarization that has existed between the Agile and classical plan‐driven project management communities has been rooted in some well‐established stereotypes of what a project manager is that are based on how typical projects have been managed in the past. The role of a project manager has been so strongly associated with someone who plans and manages projects using classical plan‐driven project management approaches that many people cannot conceive of any other image of a project manager. It’s time to develop a new vision of what an Agile Project Manager is that goes beyond all those traditional stereotypes and fully integrates Agile within the overall portfolio of project management principles and practices.

It feels very similar to an evolution that took place when I worked in the quality management profession in the early 1990s. Up until that time, the primary emphasis in quality management had been on quality control, and inspection, and the image of a quality manager was heavily based on that role:

• The predominant quality management approach was based on final inspection of products prior to shipping them to the customer and rejecting any that didn’t meet quality standards. It’s easy to see how that approach was inefficient, because it resulted in a lot of unnecessary rework to correct problems after the fact, and it also wasn’t that effective because any inspection approach is based on sampling, and it is impractical to do a 100% sample. For that reason, it can result in mediocre quality.
• A far better approach was to go upstream in the process and eliminate defects at the source by designing the process to be inherently more reliable and freer of defects, and build quality into the inherent design of the products. That didn’t mean that the prior emphasis on quality control and inspection was obsolete and eliminated; it was just not the only way to manage quality and wasn’t the most effective approach in all situations.

That was a gut‐wrenching change for many in the quality management profession—instead of being in control of quality and being the gatekeeper with the inspection process, a good quality manager needed to become more of a coach and a consultant to influence others to build quality into the way they did their work. This changed the nature of the work dramatically for many in the quality management profession and eliminated a number of traditional quality management roles that were based on the old quality control and inspection approach. The similarity to the changes going on in the project management profession should be apparent:

• To be successful in more uncertain environments, project managers need to be able to take an adaptive approach that is appropriate to the level of uncertainty in the project and integrate quality into the process rather than relying on final acceptance testing at the end of the project to validate the product that is being produced.
• They also need to give up some of the control that has become associated with the project management profession—in some cases, they may need to become more of a coach and a consultant to influence others rather than being in absolute control of a project.

This can dramatically change the role of a project manager. In some situations, the role of a project manager as we’ve known it may no longer exist. For example, at a team level in an Agile project, you probably won’t find anyone with a title of Project Manager because the project management functions have been absorbed into other roles and are done very differently. That doesn’t mean that project management is no longer important, but it may cause us to dramatically rethink what project management is in a much broader context than the way we might have thought about it in the past.

Definition of Waterfall

The word Waterfall actually has a very specific meaning, but that’s often not how the word is really used:

Another aspect to the Waterfall model is that it is plan‐driven; it attempts to define and document detailed requirements and a plan for the entire project prior to starting the project.

• When someone makes a statement comparing Waterfall to Agile, the word Waterfall is often used very loosely to refer to any kind of plan‐driven methodology, and that’s not really a very accurate and meaningful comparison.
• In some other comparisons like this, the word Waterfall refers to a general style of project management that obsessively emphasizes predictability and control over agility, and that’s just bad project management. The Waterfall model will be discussed in more detail in Chapter 2.

Definition of Agile

Officially, Agile is defined by the principles and values of the Agile Manifesto of 2001 which will be discussed in Chapter 2. Agile is also an umbrella term used by many Agile practitioners to refer to different methods and frameworks that are based on adaptive, experimental, and extreme programming practices that have emerged since the mid‐to‐late 1990s.⁵

From a general perspective, Agile is a flexible and adaptive approach for developing and optimizing solutions in an uncertain environment. It is both incremental and iterative:

• “Incremental” means that the solution is broken up into “chunks” that are developed and tested individually and might also be released individually rather than waiting for the entire solution to be developed, tested, and released as a whole.
• “Iterative” means that the solution is progressively optimized and refined based on user feedback and inputs to maximize the value of the solution to the users.

It is particularly well suited for an environment with a high level of uncertainty because the process can start with only a high‐level view of the project goals and requirements and those goals and requirements can be further elaborated and refined as the project is in progress. Of course, that does not mean that all Agile projects start with only a high‐level view of the project goals and requirements. That will vary from one project to the next depending on the level of uncertainty in the project.

In actual practice, the meaning of the word Agile in this kind of comparison is also somewhat elusive because it has taken on some very strong connotations in actual usage. At a project level, the word Agile has frequently taken on a specific connotation associated with using the Scrum methodology on software development projects.

That definition has evolved over the years as Scrum has become somewhat of a de‐facto standard for Agile projects; however, the original definition of Agile conceived in the Manifesto for Agile Software Development,⁶ published in 2001, was much broader than that. Better known as the Agile Manifesto, it laid out some simple and general principles and values that can apply to any kind of project (not just software development) (see Chapter 2).

Comparison of Predictive (Plan‐Driven) and Adaptive (Value‐Driven) Approaches

Traditional, classical plan‐driven project management is a style of project management that is applied to projects where the requirements and plan for completing the project can be defined to a large extent prior to implementing the project. The emphasis in this style of project management is on predictability, and for that reason, the PMI calls it “predictive.” However, plan‐driven is a relative term, and you won’t find many projects that start out with an absolutely rigid plan that is not expected to change at all. This style of project management is often loosely called “Waterfall.”

In contrast, an adaptive (or value‐driven) style of project management starts the implementation of a project with a less well‐defined plan of how the project will be implemented and recognizes that the requirements and plan for the project are expected to evolve as the project progresses. Adaptive is also a relative term; you won’t find many projects that have no plan whatsoever of how the project will be done.

The important point is that the terms predictive (plan‐driven) and adaptive (value‐driven) are relative—they are not discrete, binary, mutually exclusive alternatives. They should imply a continuous range of approaches with different levels of upfront planning. Table 1.1 shows a comparison of the two approaches.

Which Approach Is Better?

You will often hear people say an Agile approach is inherently better than a Waterfall approach. That is often a very biased statement and it’s not necessarily accurate. Saying “Agile is better than Waterfall,” is like saying, “A car is better than a boat.”

• Agile and Waterfall are different kinds of approaches designed for different kinds of projects.
• The problem is not so much that Waterfall or Agile are inherently good or bad; the problem comes about when those methodologies are misused, and people try to use a single methodology (whatever it might be) for all projects.
• Using a “one‐size‐fits‐all” strategy to applying either Waterfall (plan‐driven) or Agile (adaptive) approaches to all projects is not likely to yield optimum results.

In my opinion, being able to objectively understand the difference between a plan‐driven approach and a more adaptive (value‐driven) approach—as well as the principles behind those approaches at a deeper level—is probably one of the most important skills an Agile Project Manager needs to have. An Agile Project Manager needs to recognize the following:

• There is a broad range of alternative approaches between being plan‐driven and being adaptive (value‐driven): The Agile Project Manager must choose the right level of upfront planning to be applied to a project, based on the level of uncertainty and other factors in the project.
• It takes some skill to make the right choice: There is nothing inherently wrong with either of these approaches (adaptive or plan‐driven). The problem comes about when people try to force‐fit a project into one of these approaches rather than selecting and tailoring the approach to fit the project. For example, if I were to set out to try to find a cure for cancer (which has a high level of uncertainty) and I attempted to apply a highly plan‐driven approach to that project, the results would probably be dismal.

The important point is that a heavily plan‐driven approach (what some loosely refer to as Waterfall) is not the only way to successfully manage a project. In many projects, a good approach is to use an adaptive (value‐driven) approach to start the design effort without fully defined and detailed requirements and perhaps prototype something quickly. Then, user feedback can be added to further refine the design as the project progresses. With a more adaptive (value‐driven) approach:

• The elements of the approach are much more concurrent than sequential: Instead of doing the entire design and then turning it over to quality assurance (QA) for testing, the design is done in small chunks called iterations or sprints that are typically two to four weeks long. During that time, developers and testers work collaboratively to design and test the software during each sprint.
• The customer also provides detailed inputs on the design during each sprint: The customer accepts the results of each sprint at the end of each two‐ to four‐week period rather than waiting for user acceptance testing (UAT) at the end of the project. That has the advantage of finding and resolving any problems quickly and early in the project.

The Early History of Project Management

In order to understand this transition and to put it in perspective, it is useful to understand how the project management profession has evolved over the years and how we got to where we are today.

• Project management has been practiced for many years in one way or another—I’m sure that there was some kind of “project management” approach to building the great pyramids of Egypt or the Great Wall of China or other similar large efforts many years ago, but it probably wasn’t even thought of as project management in those days.
• They didn’t have Gantt charts and Pert charts and other sophisticated project planning and management tools, because those things weren’t even invented until the twentieth century.

The Industrial Revolution created the need for a more disciplined approach to project management, and a well‐defined body of knowledge associated with project management began to evolve:

World War II brought about the need for more large‐scale project management for organizing very large projects like the Manhattan Project; however, it wasn’t until the 1950s and 1960s, that it became apparent that a much more well‐defined body of knowledge and a disciplined approach were needed to successfully manage some of the large and complex projects that were evolving at that time, which led to the following:

Many people probably assume that the project management profession is now reaching a stage of maturity and stabilizing, but I believe that we have only seen the beginning, and project management, as we’ve known it, will continue to grow in entirely new directions.

Transformation of the Project Management Profession

Sometimes we get so immersed in day‐to‐day activities that we don’t take time to step back and see some fundamental changes that are going on around us.

• It seems clear to me that the project management profession, as we know it, is going to go through such a major transformation. The exact nature of that transformation isn’t completely clear as it is still evolving; however, it does seem likely that it will cause us to rethink many of the things we have taken for granted in the project management profession for a long time in a much broader perspective.
• It feels very similar to the evolution that has taken place in other technology areas and disciplines. For example, there is a strong similarity to the evolution from classical physics to modern physics.

Up until that time, the study of physics had been heavily dominated by Newtonian physics, which defines some fundamental laws of how the universe behaves such as Newton’s laws of motion. These fundamental laws were taken for granted in the world of physics for many years, even though we didn’t fully understand why things in the universe behaved as they did.

As modern physics has evolved in the twentieth century, based on quantum mechanics and relativity, we began to develop a deeper understanding of the real dynamics behind these laws, and we began to understand that the universe is not as simple and deterministic as we might have thought it was.

The transition from classical Newtonian physics to a more complete and more dynamic model based on quantum mechanics provided a deeper understanding of the forces and principles behind those laws, as well as the limitations in those laws and when and where they are really applicable. That deeper understanding didn’t invalidate the laws of Newtonian physics in most situations—“on an ‘everyday’ scale; that is, situations in which energies are large enough to permit one to neglect quantum effects, but small enough to neglect relativistic effects.”¹¹

The similarity to the transition in the project management profession should be apparent—we’re moving from a world in which we had the impression that the behavior of the universe was highly predictable and controllable and totally subject to some well‐defined rules to a world that is much more dynamic, much more probabilistic, and much less predictable.

What's Driving This Change, and Why Now?

You might ask, “Why is it becoming so essential for the project management profession to make a change at this particular point in time?” There are several major factors that will force us to rethink the concept of project management:

1. The nature of projects is changing: The modern concepts of project management were developed as result of big projects like the transcontinental railroad. Today, we have new industries and a much broader range of projects such as web development, e‐commerce, large IT projects, etc., which weren’t common before the mid‐1990s. It is becoming increasingly apparent that applying a “one‐size‐fits‐all” approach to such a broad range of projects will not have optimum results.
2. Technology is rapidly changing: Figure 1.2 shows how the adoption rate of new technologies has changed over the past century. Project management approaches that worked in the 1950s and 1960s must be reexamined to adapt to the current fast pace of technology adoption.
3. The general level of uncertainty in the world is increasing.

A similar transformation took place in the quality management profession in the 1980s and early 1990s.

• At that time, the Japanese auto industry was demonstrating huge improvements in quality of products that made conventional quality management methods based primarily on quality control and inspection very inadequate.
• They forced people to rethink the whole strategy and approach for doing quality management. Without the leadership of people like W. Edwards Deming and the significant improvements in quality that were demonstrated in the automotive industry, the transformation of quality management might never have happened.

  

• What started primarily in the automotive industry has now become a more modern approach to quality management that is widely used in all industries.

A similar thing is happening with Agile and classical plan‐driven project management today:

• The leadership of W. Edwards Deming in establishing a Total Quality Management (TQM) philosophy can be compared to the thought leadership behind the Agile Manifesto in 2001.
• The broad‐based adoption of TQM starting in the automotive industry and eventually spreading to many other industries can be compared to how Agile has started out in software development today and is now beginning to spread to other areas.

Other researchers have come to a similar conclusion regarding this; Manfred Saynisch published his findings of a research project in Project Management Journal.

The article proposes a new approach to project management called “PM2” where classical plan‐driven project management will play an active and important role but will be “extended to consider dynamic, nonlinear, multi‐causal structures and processes as well as the principles of self‐organization, evolution, and networking.” The article goes on to say:

I agree with that view—we are on the verge of a new generation of project management that will cause us to rethink many of the accepted notions of what “project management is.” It requires blending classical plan‐driven project management principles and practices with a much more empirical and evolutionary approach to deal with the uncertain, dynamic, and fast‐paced environment we live in today.

The key message is to fit the project management approach to the nature of the project, as shown in Figure 1.3. That takes a lot more skill, but it definitely can be done.