2.1.1 The Agile Revolution

Agile development was a grassroots revolution that began in engineering. Lack of success with Waterfall development frameworks drove engineering to find an alternative. In my view, the biggest attraction of Agile for engineering was the elimination of the fixed schedules of Waterfall to allow them time to build good software without punishments imposed for missed schedules.

We know that few organizations were successful in achieving the commitments of cost, schedule, and scope with Waterfall planning. The path to failure started in the planning stage where engineering was pressured to make long‐term development commitments based on vague or no requirements. Engineering had little power. Sales could always play the trump card at the CEO level by claiming impacts to their sales forecasts if they didn't get features they wanted. Product managers were under intense pressure from sales to get commitments from engineering.

As development progressed under Waterfall, new “must‐have” features came in with no relief in content delivery or schedule. Feature functionality grew from new requirements. This resulted in late releases, or, often, partially developed or tested features being dropped from the release as the deadline approached. A significant portion of the release development effort was wasted. The net effect was lower R&D Return on Investment (ROI).

There were also negative customer impacts with Waterfall. We really didn't want to get customer feedback during release development because it would put us further behind. We stood behind the 1000‐page detailed requirements document to fend off new requests during release development – “You signed off on this.” Customers were offered “workarounds” and were promised “enhancements” in a future release, which usually didn't happen. “That's on our roadmap” became the standard answer.

Agile development was the promised solution for these problems. Engineering was going to work more closely with users to build what they really valued, rather than just be implementers of prescribed features.

Agile, as we know it today, had its beginning in the Manifesto of Agile Software Development¹ proposed by a group of 17 influential people in the software industry. It was an engineering perspective of what was necessary to overcome a perceived lack of success caused by the impossible predictability demands of the business. Engineers wanted to have time to build software they could be proud of within a collaborative and motivational environment.

The Manifesto was based on 12 principles:

1. Customer satisfaction by early and continuous delivery of valuable software.
2. Welcome changing requirements, even in late development.
3. Deliver working software frequently (weeks rather than months).
4. Close, daily cooperation between business people and developers.
5. Projects are built around motivated individuals, who should be trusted.
6. Face‐to‐face conversation is the best form of communication (colocation).
7. Working software is the primary measure of progress.
8. Sustainable development, able to maintain a constant pace.
9. Continuous attention to technical excellence and good design.
10. Simplicity – the art of maximizing the amount of work not done – is essential.
11. Best architectures, requirements, and designs emerge from self‐organizing teams.
12. Regularly, the team reflects on how to become more effective, and adjusts accordingly.

This led to the creation of different Agile frameworks like Extreme Programming, Scrum, and Kanban. There were others, but Scrum has become dominant in the software industry, followed by Kanban. I will use the term “Agile” to represent both Scrum and Kanban, or any other development framework where development teams select small increments of work from a queue and develop them in two‐ to four‐week cycles.

2.1.2 Scrum

Scrum provides agility by allowing requirements to change throughout the development cycle. Requirements documentation was reduced to facilitate rapid change and reduce effort. Verbal communication was preferred. Requirements in Agile could be described in the form of a short “user story” that could be clarified by someone called a product owner during development.

A user story is a short statement of need in the form:

• As a <specific role>
• I want to <what the user wants to achieve>
• In order to <the reason behind the achievement>

For example, “As accounting clerk, I want to identify late invoices so I can follow‐up and obtain payment.”

The level of decomposition of a user story is determined by the requirement that it can be completed by the team within a sprint, a development cycle of two to four weeks. Agile accounted for definition of “epics” that are higher‐level statements of need that can be decomposed into user stories. The emphasis of Agile is on conveying the problem for engineering to solve rather than providing them with prescriptive functionality.

The product owner is responsible for what gets developed by an Agile team. Teams are comprised of five to seven engineering staff. The product owner prioritizes the work of the team and expands on the user stories during development to ensure that the right thing is built. The product owner has final approval and declares a user story “done” at a review at the end of each sprint. Engineers are given time in the next sprint to incorporate feedback, as well as any new requirements identified. Defect correction takes a higher priority than new development.

Engineering teams set their own goals for the sprint user stories. They are encouraged to lower sprint targets if they miss objectives.

Figure 2.1 shows a high-level view of Scrum.

The team selects user stories for each sprint from a prioritized list called the “product backlog” that contains stories targeted for the release “increment.” The product backlog content and priorities are controlled by the product owner. Sprint retrospectives are done by the team at the end of each sprint to find ways to increase throughput and improve quality.

There is one more important distinguishing factor of Scrum. Each story is supposed to provide working functionality that can be demonstrated, instead of just a chunk of development. For example, assume the user needs to drill down in a report to get the information they need to satisfy the user story. Before Scrum, a wireframe for the full report would be generated and reviewed, followed by the complete layout of the UI. Drilldown capability for the entire report would then be developed.

In Scrum, a user story would be defined around the need that is driving a specific drilldown case. For example, “As an accountant I want to determine the accounts that contribute to the accounts payable balance in order to follow up with the customer.” A page with a single button would be created in a sprint. However, the button is functional. When the product owner clicks on the button, a summary of accounts displays.

We can see that Scrum is incompatible with the long‐term feature‐level predictability demanded by the business. It is impossible to make feature‐level predictions when requirements haven't been developed, and how do we account for the extra time necessary to incorporate feedback during and after the sprint? We also can't account for the indeterminate time it will take to fix defects at a higher priority than new functionality in the next sprint to keep the software “potentially shippable.” Contingency can be built in to achieve predictions, but my experience is that contingency doesn't last long under the intense pressure on product management and engineering to deliver more features.

2.1.3 Kanban

It's easier to describe Kanban by comparing and contrasting it with Scrum. In terms of similarities, both involve small development teams pulling small increments of work from queues at their own pace. Both advocate small development increments. However, the product owner title is not used in Kanban. Similar positions are “service request managers” or “service delivery managers.”

One difference is that Kanban provides more flexibility in how requirements are defined. There is no requirement for user stories.

Work items are represented by cards on a board (see Figure 2.2). The columns represent the order of work. Work items can be “In Process” or “Done.” To add to the confusion, many Scrum teams adopt Kanban boards to focus the team's work, but it doesn't mean they're “doing Kanban.” As long as they are using user stories under the guidance of a product owner and conducting Scrum “ceremonies” like stand‐up meetings and retrospectives, they are still doing Scrum, which could be called “Scrum with a Kanban board.”

There is much available on the web on the history of Kanban. It started in Japanese auto manufacturing to control parts inventories. Big stacks of parts would build up at manufacturing workstations. For example, brake pads may pile up for wheel assembly. This represents money tied up in the manufacturing process that shows up on financial reports. It also leads to delays in the manufacturing line when upstream work must stop while a bubble of inventory is processed. Manufacturing needed some way to prevent the buildup of parts in the manufacturing line.

“Kanban” means signboard or billboard in Japanese. The simple solution was to create cards to signal the number of parts to be delivered to a workstation. The preceding workstation knows how many parts to provide to the subsequent workstation to prevent parts from piling up. It reduces the amount of capital tied up in parts sitting on the manufacturing floor and reduces delay by limiting queue sizes.

Kanban used the concept of “Work in Process (WIP)” to account for value of parts waiting in queues. It became an important manufacturing Key Performance Indicator (KPI). WIP limits could be established for the number of queued parts. WIP limits also defined a minimum number of parts to account for variation in the manufacturing process to prevent “work starvation.”

Kanban used in software today was heavily influenced by Don Reinertsen, the same person we can thank for Cost of Delay and Weighted Shortest Job First (WSJF) discussed in Chapter 4. He modeled software development as a series of “workstations” taking work delivered by preceding workstations and applying a development process step. For example, flow of Agile development artifacts may include multiple queues.

Work can build up at each interface and cause the same impediments to efficient flow that occur in manufacturing. One difference is that partially completed elements of the software process are not quantified in terms of inventory cost. It doesn't appear on the financial report.

The same book referenced for Cost of Delay and WSJF, The Principles of Product Development Flow, contributed much to optimizing the flow of work through the software development cycle. Reinertsen supports his principles with queuing theory – an interesting and insightful contribution to software development.

2.2.1 Schedule Pressure

Agile was predicated on variable content and fixed release cycles determined by a “timebox.” Requirements are expected to evolve during development. Feedback is provided throughout the development cycle to prioritize building what users really want over schedule. Agile teams pull work from queues at a pace they determine, and they prioritize correction of defects over new production to keep the software in a “potentially shippable state.” However, most large organizations today are still using a traditional Waterfall planning process that forces schedule and content constraints on Agile teams, undercutting the Agile tenet of variable content.

From the engineering perspective, Agile can actually improve predictability for the business. The fixed effort, schedule, and content model of Waterfall planning was an illusion. Companies made huge investments in large releases that rarely met schedules. Any that met effort, schedule, and content commitments usually compromised quality. As ship dates drew near, partially completed features were shed from the release like ballast released from a sinking balloon. Agile broke down software development investment into smaller increments with lower risk and improved predictability. Why would Waterfall planning be retained?

Agile failed to address the business need for multiyear financial predictability that the business sought from Waterfall planning. These needs must be addressed to make the variable content of Agile development a reality. To start, let's examine the drivers behind the business need for predictability that need to be satisfied. Fortunately, we only need to learn how a few line‐items in financial reports drive the quest for predictability and the perpetuation of Waterfall planning. You don't need a background in finance or accounting to understand it.

There are three documents that drive the need for business predictability. The first two are the income statement and balance sheet that can be found in the annual report of a public company. These documents also exist in nonpublic companies. Investors and boards of directors rely on them to justify investment in the company. The third document is the business plan, an internal document that describes how the company will attain the objectives of the financial statements.

2.2.2 The “Motivation” Factor

This section exposes an unspoken reason why the fixed schedule and scope of Waterfall planning persist in Agile development. In the eyes of the business, Waterfall schedules create deliverable accountability used to prod engineering to work harder to meet deadlines. The prods are in the form of negative project meetings that “motivate” engineering to avoid showing up with deliverables off‐schedule.

How do I know that? In every Agile transition I've been involved with, I've stressed to management that they have agreed to implement self‐directed teams that pull work from queues at a pace determined by them. The first reaction is invariably, “How do we know they're working as hard as they can?” My response is, “So, you're saying that the only way this company motivates people is to overload them with work and prod them when they drop behind?” Unfortunately, this is the sad truth. Executives publicly state that they accept self‐directed teams in Agile, but they still believe that pressure from fixed schedule and scope commitments are necessary to create motivation.

I've seen attempts to retain individual accountability in Agile. I've attended Scrum stand‐up meetings where attendees are asked one by one by managers what they've completed in the last day. I've seen many cases where the backlog for a fixed‐length sprint is chosen by product or project management. Missed the sprint objective? “Well, we need to load them up with even more next sprint, so we get closer to what we need to meet the release schedule.” I've seen project managers who continue to track at the task level. The problem is that most companies don't know how to motivate Agile teams without negative schedule accountability.

You may be wondering why engineering motivation is always questioned. Engineering leaders often hear comments from other department leaders that question their motivation, especially in front of the CEO – “I was on the engineering floor at 8:00 this morning and I didn't see any of your developers.” In most of the organizational assessments I did, product management described their major issue as poor engineering productivity. They had all the right plans if only engineering could develop them.

Engineering productivity is questioned because they miss the impossible commitments based on the fixed schedules and content of Waterfall planning. From the perspective of the outside world, they clearly need more motivation. Additional functionality is piled onto large releases and project management increases pressure to deliver. It's a no‐win situation for engineering.

The other major reason for questioning engineering motivation is that under pressure, many engineering leaders end up committing to additional features and functionality without insisting on trade‐offs, and they often deliver, at least from the perspective of the outside world. Clearly, engineering had slack that was only taken up under intense pressure.

Engineers know how this really happens. They resort to developing software that works for the “happy path.” Demonstrating the “happy path” requires only a fraction of the code required to create robust and reliable software that covers all situations. Under intense schedule pressure, the software to handle unusual cases and exceptions is sacrificed, resulting in error‐prone and unreliable software. Testing time is constrained to include only the most visible and rudimentary cases. The outside world can't tell the difference until customer complaints roll in. However, engineering did meet the schedule despite the addition of new features and functionality. They obviously just needed more pressure to increase their motivation.

I've been in the position of an engineering leader under intense pressure at the executive level, so I don't blame these engineering leaders. Careers are at stake. I do have a tip on how engineering leaders can better handle these situations. I realized once I got out of engineering that engineering leaders usually push back immediately when faced with the challenge of adding functionality within the current schedule. The engineering mind immediately starts to verbalize obstacles when presented with a problem. This gives the impression that they will not really try to find ways to get it done.

Try this. When confronted with a challenge, immediately respond that you understand the importance to the business, and you will get your team together to look at every conceivable way to get it done. No thinking out loud of why it may not be possible. Go back with your team and try to find ways to get it done. At the same time consider trade‐offs that might be presented as well as what you could deliver within the requested time. You will find the audience to be much more receptive to accept trade‐offs when clearly presented, and they will believe you and your team have tried your best to meet their needs.

Many organizations still believe they need the fixed schedules and content commitments of Waterfall planning to motivate engineering. A later chapter dedicated to motivation will show that pressure works, but it limits employee engagement. Motivation should not be an excuse for retention of Waterfall planning.

2.2.3 The Mythical Product Owner

The envisioned role of the all‐knowing product owner was not effectively implemented in most of the organizations I assessed. There were pockets of success, but larger product companies with established product management organizations struggled. A successful product owner implementation is crucial for successful Agile.

The product owner role defined in Scrum was a great idea. Engineers using Waterfall were tired of being blamed for missed requirements. At the end of the day, if the product didn't meet expectations, it was engineering's fault. “They should have known, or they should have asked.” The idea that engineering could have an individual assigned to their team to tell them everything they needed to know was attractive, and to take the blame when they didn't get it right. I have sometimes heard product owners referred to by engineering as “One Neck to Grab (ONTG),” or the “single wringable neck.” The expressions reflect the pressure engineers are under. Of course, it's better from the engineering perspective that the product manager gets their neck wrung when the wrong thing is built.

User stories reduced requirements documentation, but under the assumption that a product owner would be engaged with the team to add the detail necessary for engineers to write their code. Construx refers to user stories as “topics of discussion.” At a minimum, user story detail should be included in an acceptance test written prior to inclusion in the backlog. Any cases not explicitly described in the acceptance tests were supposed to be clarified by a product owner fully engaged with the team throughout sprint development.

Product managers were no longer required in the Agile vision. After all, they were the perceived source of the pressure on engineering. Engineers also didn't want product managers constraining them by telling them how to build it. Engineers just wanted someone who would work closely with the team to ensure they built the right thing. Tell them the problem to solve. Product managers were written out of the Agile vision. They could be replaced by product owners.

The name “product owner” led to confusion in organizations with established product management departments. Product managers have always been “product owners” in that they have ultimate responsibility for the success of the project, sometimes at the Profit and Loss (P&L) level. Therefore, in their minds, they were the logical “product owners.”

However, much of the time of product managers is taken up by the insatiable demands of sales. Failure to respond to a sales request could put a product manager in front of the CEO explaining why they have sabotaged this year's financial commitments. Product managers were also driven by other priorities, like preparing for trade shows or performing customer demonstrations. Near‐term priorities gave them little time to interact with Agile teams during a sprint development cycle, and often no time to thoroughly review their implementation at the end of a sprint. Agile teams took their best shots at building functionality based on a one‐sentence user story, only to find they're wrong as the release date drew closer, or often after it has been released.

Many organizations realized the futility of using product managers in the product owner role and assigned product owners from within engineering. Product owners had more time with the teams, but they lacked exposure to customers and the market. Now engineering was isolated from the customer through two layers, product owners and product managers, and sales often presented a third layer.

Even with the assignment of a qualified product owner, they rarely sat with the teams to give them the level of verbal communication expected in Agile. Engineering was left to interact through e‐mail, chat, or by requesting meetings. This way a product owner could handle multiple teams. Certainly, remote communication would be sufficient. Product owners and teams could be in different buildings, across the country, or even offshore in different time zones.

Let's look at what happens when engineers can't get the answers they need, especially when they're under schedule pressure. They make assumptions based on their peepholes into the domain, and they are often wrong. One of the scariest things in the world is listening to a couple of junior engineers with no domain experience discuss how something should be implemented. I used to tell product managers that if you don't specify what should happen, assume that decision will be made by an engineer fresh out of college. Engineers must make assumptions or miss schedules.

Product organizations struggled to establish the product owner envisioned in Agile because they build software for a diverse set of users who can't be represented by an individual. Product organizations must aggregate needs and constraints from worldwide markets with diverse product stakeholders and regulatory requirements. The product owner role did not include standard requirements elicitation processes to create and prioritize product requirements. Get it wrong? “Oh, just fix it in the next sprint.”

I worked with a large energy company that brought in field engineers as product owners. Engineering commented that their last product was always built for the last field engineer, missing many of the requirements for international deployment.

So Agile went forth without clarification of how the new Agile roles would fit within a larger established organization. The role of project management was also written out. They could be replaced by scrum masters who support the team instead of pestering engineers to get their project tasks checked off. Management roles were also excluded. “We don't really care what they do as long as they stay out of our hair.”

The net result is a reduction in software development productivity due to massive rework caused by incomplete requirements. Please hear me out. I'm not going to advocate going back to the Waterfall days. As I said, my goal is to make Agile successful in the eyes of the business. Agile has the potential to reduce wasteful rework to levels we could never attain in Waterfall. However, the failure to implement a successful product owner model results in high levels of waste, and what's worse is that this waste is no longer visible to the business because it just gets added to sprint backlogs.

The bottom line is that making Agile successful in larger organizations requires clear role descriptions for the product manager, product owner, project manager, and management that complement each other.

2.2.4 Feature Planning

Agile development was supposed to replace features with epics and stories. Agile teams could add value by understanding the problem to be solved, rather than just implementing features based on detailed requirements. Feature‐based planning is still the norm at most organizations. In my experience, the vast majority of Agile teams are adding predefined features to a code base with little opportunity to add value. Many organizations define their product backlog in terms of features.

Product managers need processes that allow them to focus on value prior to any discussions on features, and the value must be conveyed to engineering in terms of a problem for them to solve. We'll see later that there is a place for features, but starting there severely limits the ability to add business value.