The Scrum Framework was developed as a reaction to some frustrating limitations found in the “traditional” way software development work is organized. The traditional framework is codified in the PMI’s PMBOK and is commonly known as the system development life cycle (SDLC) or waterfall.
The SDLC framework envisions software development as a series of “projects.” Each project has a beginning, a middle, and an end. Throughout the life of a product, there may be many projects that are used to change or enhance the product’s functionality. Each project is a discrete activity with its own scope, its own goals, its own budget, and its own schedule. Each project is independent of every other project and has nothing necessarily in common with any other project other than the fact that several projects may enhance the same product.
Projects begin with a proposal that is then reviewed by an approving authority. The approvers may be high-level managers, an approval committee, or some other authority that makes budgetary decisions. The proposal includes a description of the new product or enhancement, a projected cost, a projected return on the proposed investment, and an estimated delivery date.
The approvers evaluate the proposal and consider both the potential return on the investment (ROI) and the risk that the return may not be realized. They check whether the proposal fits within the organization’s overall business strategy, and they consider whether it makes sense to proceed with this proposal rather than others that may be being considered as well.
If the approval authority gives its consent, budget money is set aside to work the project, and a due date is set for delivery of the project’s work product. (This is known as Gate 0 in the SDLC.)
The next step is to do detailed analysis and design, and create a detailed work breakdown schedule , or WBS. The WBS is usually created by software architects and engineering leads and is a list of technical tasks to be performed according to a specific sequence. Other planning activities also take place, including producing such things as a risk analysis and a risk mitigation plan.
The tasks in the WBS are usually represented as a Gantt chart that shows a schedule for the completion of each task. Each task is assigned an estimated duration and is arranged in a sequence that displays the prior tasks that must be completed before work can begin on the subsequent tasks dependent on them. Dependent tasks tend to form “chains” within the Gantt chart, with the longest chain called the critical path because its duration governs the project’s overall duration.
The SDLC framework is often called waterfall because the critical path shown in a Gantt chart often resembles water flowing over a cliff.
After the initial WBS is created, it is adjusted to fit the approved budget and the approved deadline. The resulting project plan shows a detailed list of technical tasks to be performed—all of which are scheduled to produce the project’s goal by the approved delivery date.
This plan then is reevaluated and approved by the approval authority, after which work can begin (Gate 1 in the SDLC). Development teams are assembled and assigned to complete tasks as scheduled in the WBS.
What Could Possibly Go Wrong?
It turns out that quite a bit can go wrong. The reason for this is that the SDLC is designed to solve complicated problems. It is not designed to deal with complex problems. It works best when the problem to solve is like building that 99th house described in Chapter 2. It is not very well adapted to building the first house.
The waterfall framework assumes that everything relevant to the project is either known at the beginning or can be predicted accurately before any work starts. By the time Gate 1 is passed, an entire plan listing every detail of the work has to have been drawn up. The plan will have to conform to the original budget and delivery date commitments made at Gate 0. The plan is, in effect, a “recipe” for creating the project deliverables. All the developers have to do is follow the recipe and it “should” be possible to achieve all of the project goals.
In reality, creating and following this kind of recipe rarely works. The reason is that the plan is based on the assumption that everything can be predicted before the project starts.
Any typical waterfall plan is filled with dozens (if not hundreds) of assumptions. Every predicted duration of every task in the WBS is an assumption. The completeness and accuracy of the task list itself is an assumption. The belief that the project requirements are correct and will not change is also an assumption. The availability of key developers and other contributors is, once again, an assumption.
There is a wry definition of the word assume. “ASSUME: making an ASS out of U and ME.” Waterfall plans pile assumptions on top of other assumptions. If any of those assumptions turn out to be unrealistic, the results are often quite bad—if not catastrophic.
The SDLC gating process and associated approvals set a specific expectation in the minds of the approvers. The expectation is that if the detailed plan is approved in Gate 1, then the benefits will be delivered on time and on budget. In effect, the approvers are agreeing to “buy” the results for a fixed price and delivery date. They expect to get those results as long as the project plan they approved is followed.
This expectation is so strong there is usually some kind of “change control” process that restricts the developers’ ability to deviate from the approved plan. The assumption is that the plan is correct, so any deviation from it must be incorrect until it is proved otherwise.
The detailed plan and associated Gantt charts give the illusion of certainty about the project’s outcome. Success is “guaranteed” if you “plan the work” and “work the plan.”
In reality, the detailed plan is usually wrong. The duration of each technical task is a guess, and the guess is usually made by someone who does not have to do the task him- or herself. They guess at how long a task should take. They have no idea how long a task will take.
The list of tasks to be performed is also a guess. It is based on the architect’s or engineering lead’s understanding of the requirement at the beginning of the project before any work has been done. As the work progresses, the need for new tasks may be uncovered. Each new task becomes an item for the change control authority either to approve or disapprove.
The goal of the project is also a guess. At the very beginning of the project, a proposal is reviewed and approved as part of Gate 0. It is approved on the basis of a perceived need that exists at that time. As the project progresses, the need may change. These changes take the form of “requirement change requests,” which are also subject to approval by the change control authority.
Waterfall works best for projects that don’t involve guesswork. If things such as task lists and durations are known, then the Gate mechanisms and the WBS work well.
In software development, almost all key factors are not known. Using waterfall causes the project plan to boil down to a large number of pure guesses that may be correct or incorrect in the end.
Studies such as the Standish Group 2015 Chaos Report1 have been made to determine how often waterfall-based software projects are successful. For this purpose, “successful” means the end customer got what they wanted for approximately what they expected to pay and within the approximate timeframe they originally requested.
These studies show that these projects are successful about 11% to 14% of the time. Of the other 86%, about half of those projects are classified as “seriously challenged.” “Seriously challenged” means these projects require at least a 50% cost adjustment, 50% rework, and/or a 50% increase in duration to achieve the project’s goals. The other half of the 86% represents projects that are total failures, with all costs written off.
In his book Agile Product Management with Scrum,2 Roman Pichler described working through a two-year software development project using the waterfall framework. Throughout the project, a great deal of attention was paid to following the plan. In the end, the project was completed on time and under budget. They planned the work well and worked the plan accurately. Unfortunately, they succeeded in creating a software tool that was perfect for the marketplace as it had existed two years earlier. By the time the project was finished, the end result was obsolete even before it was delivered. The assumption about what the marketplace would need at the end of the two-year project had been wrong. All the time and effort creating the product had been wasted.
Waterfall model
What is a system? A system (such as software development) can be analyzed based on its inputs and its outputs. In other words, the process of software development consumes things and produces things. As Figure 3-1 shows, the software development process consumes money (in the form of people, programming tools, equipment, and so on) and time (man-hours, calendar days). It produces software features and functionality. Most important, however, is that it produces functionality at a certain level of quality.
The important thing to realize is that, in any system, the inputs and outputs are balanced. They form an equilibrium. If the inputs are changed, the outputs change as well.
The WBS in a waterfall project forms a detailed plan to allocate and manage the time and money consumed by the software development process. It dictates that certain functionalities be completed at certain points in time. In effect, the WBS holds constant the inputs (time and resources), but only one of the outputs (functionality).
“Working the plan” means completing all tasks by the dates specified in the WBS using the resources allocated. This works fine if the complexities and durations of all tasks are known. If those complexities and durations are only guesses, however, bad consequences ensue if the guesses turn out to be wrong.
Let’s consider an example. Let’s assume there is a waterfall project plan for developing a product and that the development team discovers a key task on the plan’s critical path has been underestimated. The task had been planned to take two weeks, but the developers now believe it will take five weeks.
Because this task is on the plan’s critical path, the delay will affect the entire project. Everything scheduled to be completed after this particular task will be delayed three weeks. The delivery of the product will be delayed three weeks. The budget for the project will be exceeded by three weeks’ worth of cost.
Because this change in the schedule has such far-reaching consequences, it usually must be approved by the change control authority before being allowed. The developers must give a justification for taking five weeks instead of two weeks, and they may only take the extra time if the change control authority approves the request.
Often such a change control request is never made. The usual reaction to the news of the potential delay is for the project manager (PM) to tell the developers, “We can’t miss this deadline.” The PM tells the developers they must stick to the original two-week timeframe. The PM says, “I don’t care how you do it. You must get it done in two weeks [or else . . .].”
Faced with this kind of pressure from the PM, the developers often do “figure out” how to get the task done in two weeks. Whether the PM objects or the change control authority rejects the change request, developers are often put in a position in which they are expected to do the impossible. Because the PM doesn’t care how they do it, the developers use shortcuts, hard coding, patches, and other bad practices to make it appear they have done the job and met the deadline.
This behavior is clearly predicted by the diagram of the software development system in Figure 3-1. The developers have discovered that the time set aside for a particular task is not adequate. They are told they have to complete the task in the originally specified two weeks. The inputs (resources and time) as well as one of the outputs (product) are all being held constant and fixed. The only way to compensate is to reduce the quality of the product.
The developers do not intend to do a poor-quality job. They simply want to show they have met the deadline so that management will get off of their backs. They do their shortcuts, hard coding, and workarounds while saying to themselves, “We’ll come back and fix all of this later.” The trouble is, “later” never comes.
Strict adherence to deadlines causes bad code and other quality compromises to corrupt the product. What is worse is that this corruption is hidden. The developers know it’s there, but their managers do not.
There is a name for these hidden problems. It is called technical debt . Technical debt is work that is “owed” to the product. The truly bad thing about technical debt is not that it exists, but that it is hidden. Technical debt is a problem, but it will never be solved until decision makers realize it is there.
How serious an issue can technical debt become? The following pages present two examples.
The .CSS File from H*LL.
I consulted with a company with a main product that was an e-commerce web site. They had been developing this web site using an aggressive waterfall schedule.
I asked the various PMs and program managers how clean the code base was. They all assured me the quality of the code was “great.”
I then asked some of the developers how clean the code base was. They told me it was “awful.”
The developers gave me an example of a cascading style sheet (.css) file with which they had to work. (For those of you who are not web designers, a .css file contains rules about the fonts, color schemes, and button shapes used to control the appearance of a web page.)
These .css files are usually not very complicated. A typical .css file contains a few hundred lines of code. A truly complicated one can be 500 to 900 lines long.
The .css file the developers told me about was quite a bit bigger than 500 lines of code. That file had 40,000 lines of code in it! (I could barely stop laughing when I heard how big the file was.)
How did the file get so big? The answer lies in how a .css file works. When a web browser uses it, it reads the file from top to bottom. Whenever it encounters a new rule for displaying a particular part of the web page, it takes note of it. If it encounters a second version of the rule for displaying that particular part of the web page, it forgets the first rule and uses the second one.
When the .css file was relatively new, one of the developers was pressed for time. The job was to change the appearance of a particular part of the web page. Rather than take the time to find the original rule and change it, the developer simply added a new version of the rule to the end of the file.
As more and more changes were required, developers pressed for time simply chose to add new rules to the end of the file, rather than look through the file to find the original version and change it.
At some point in time, the file became too big for anyone to figure out. The only practical way to change a rule was to add a new definition to the end of the file. By the time I talked with the developers, the file contained 12 or more definitions of each and every rule.
The file was so big, it was affecting the performance of the entire web site. It took time for a web browser to read and interpret 40,000 lines of .css code rules. Some developers noted it was possible to log in to a competitor’s web application and start completing transactions before their own web site even came up for display.
The real problem with the.css file is not that it was so big. The issue is that the size problem was hidden. It was an “elephant in the living room” that the developers all knew about, but the architects and managers did not. The people who created the Work Breakdown Schedules for various projects assumed the .css file was of normal size. When they planned out the development schedule, they only allocated a small amount of time for changing the appearance of the web site. This forced the developers to take the “easy” way out and perpetuate the problem, rather than fix it once and for all.
A Small Change
Another example comes from another company with which I once consulted. The company’s product was also an e-commerce application with a web site, an Android app, and an Apple iPhone (IOS) app as ways for a customer to access it.
Once again, the focus at this company was about aggressive deadlines. The team I worked with once gave an estimate of 90 days to complete a particular enhancement. The top management of the company told the team that 90 days was “ridiculous” and they had only 40 days to complete the work.
One of the consequences of this could be seen in the execution of another enhancement. The team was responsible for enhancing a set of application program interfaces (APIs) that allowed the IOS and Android apps to access the functionality provided by the “core” systems. The Android and IOS people asked for a change in one of the APIs.
The requested change was relatively simple. There was an existing API that the mobile devices called and returned certain pieces of information. The app people simply wanted that API to return three new pieces of information. What’s more, those three new pieces of information were already present in the core systems.
The APIs were written in Java and, when asked, most Java programmers would say that this change was probably about an afternoon’s worth of work. A Java programmer should be able to sit down and produce this change in a matter of a few hours. Throw in a day or two for testing and the job would be done.
In reality, it took much longer than an afternoon for this enhancement to be made. In the end, it took four programmers six weeks to get the job done. It took that long because the API code was so bad it was almost impossible to figure out what was going on within it. Also, it was almost impossible to figure out where those three pieces of information were to be found in the core systems.
The whole process became a huge exercise in “trial and error.” The developers would return some values only to find out they were the wrong values. They made code changes only to find that the code they changed had been “branched around” and was not even being executed. When they did change some code that was in use, the quality assurance (QA) team reported their change had “broken” something else. It took four developers six weeks to (1) return the correct values and (2) restabilize the API.
How could the management of this company tolerate this kind of situation? The answer is that they didn’t notice that it was building up and they didn’t realize it was there.
The situation is similar to the story about putting a frog into boiling water. If a frog gets dropped into a pot of hot water, it jumps out immediately. If a frog is put into a pot of cold water that is then gradually heated to the boiling point, it will boil to death before it realizes things are changing.
The management of this company had learned it took a long time to get things done. They kept up constant pressure on the developers, but they also had learned to expect that changes took time. They didn’t notice things were getting worse and worse until they got to the point where it took weeks to do what ought to be accomplished in a few hours of work.
Technical debt can be a huge issue. If it is allowed to accumulate, technical debt can become very expensive. It can literally be the difference between the cost of one programmer working for a few hours and the cost of four programmers working for six weeks. Multiply that one example by the number of enhancements a company usually makes to a product and the costs can be staggering.
Technical debt is the result of shortcuts developers take when they can’t meet arbitrary deadlines any other way. These deadlines come from guesses that architects and planners make about how long various tasks “should” take. As we have seen, those guesses are usually wrong in the end. Nevertheless, management assumes the overall plan is correct, and all the guesses and assumptions that make it up are also correct. They insist developers “work the plan,” even when the plan proves to be unworkable.
Summary
The fundamental root cause of these difficulties is the fact that waterfall is not suitable for doing software development work. Waterfall is designed to solve complicated problems, not complex ones. It is well suited to organizing projects for which there are no uncertainties. If planners and architects really could be certain about the technical challenges and what it will take to solve them, then planning the work and working the plan makes sense.
The challenge is that, in software development, nothing is certain. It is not possible to make a comprehensive plan for developing software because it is impossible to know ahead of time all of the unexpected obstacles that will be encountered along the way,
The only approach that makes sense in software development is an “adaptive” one. First, identify the goal to be achieved (such as a certain piece of software produced for a certain overall cost and delivered within a general period of time). Next, figure out what the first step is and plan that step. Take the step, then look at the results. After taking into account any issues that have been encountered, plan and take the next step. Keep going until either the goal is achieved or it finally makes no sense to continue.
The only way to solve a complex problem is to learn and adapt as you go. The Scrum Framework is an elegant and ingenious way to organize people to do just that.