2.1 Objectives

This chapter will help you

• Understand the histories of business analysis, agile development, and agile analysis and planning.

• Know the benefits of business analysis and agile development.

• Understand the value added by including an effective agile analysis competency in an agile organization.

2.2 What Is Agile Analysis and Planning?

The short definition of agile analysis and planning is that it is business analysis (BA) plus planning as practiced in an agile context. That definition only helps, though, if you know what the individual terms mean. The following is a more formal definition based on the description introduced earlier in the book:

Agile analysis and planning is the examination of a business or any aspect of it (culture, organizational structure, process, products, and more) to learn what needs to change and when in order to achieve a desired outcome in a context that places a high premium on adaptability, resilience, and continuous innovation and value delivery. It includes analyzing who the product is for (the stakeholders), defining their requirements, determining when the capabilities will be delivered, and estimating costs and resources.

As explained in Chapter 1, I bundled BA together with requirements implementation planning because both are in a continual state of flux in agile development and therefore are inextricably linked. Changes to requirements and priorities affect the implementation plan, and changes to the plan affect the timing of the analysis. A more precise term for the discipline is agile business analysis and planning, since analysis without that qualifier could refer to any sort of analysis (e.g., financial, data, or systems analysis). For brevity, however, I more often use agile analysis and planning in this book. You’ll occasionally see the even shorter term, agile analysis. Don’t attach any special meaning to these variants; they all refer to the same competency.

As the definition points out, agile analysis and planning occurs in an environment that places a high value on adaptability and continuous delivery (CD). This context distinguishes it from traditional analysis and planning, where the business sets a higher value on predictability and sticking to a preset plan. This shift in context has a profound impact on the way the competency is carried out. For example, in a waterfall context, the requirements specifications and requirements implementation plan are frozen prior to development. Changes can be made but only after following a rigorous change management process. In agile development, however, we expect and even encourage changes to the requirements and plan because they enable the business to respond quickly to learning and adapt to the market.

You might think that the kinds of businesses that would value adaptability and continuous delivery—and thus be natural candidates for agile development—would be disruptive innovators. You’d be partially right: those companies use the approach because it enables them to test hypotheses for the product and respond effectively to learning. But agile development is also widely practiced by established companies because it provides them with the strategic capability to react to changes in the market and technology without sacrificing reliability.

2.3 Who Is a Business Analyst?

This book is primarily concerned with agile BA as a competency and a practice and is less concerned with the title of the person doing the work. Unless otherwise specified, when I use the term business analyst in this book, I mean any person engaged in BA activities, regardless of their formal title. In practice, those who perform these activities may have the title of business analyst, IT business analyst, team analyst, business systems analyst—or even a non-BA title such as requirements engineer, product owner, data analyst, tester, UX professional, or programmer.

This usage is in line with the International Institute for Business Analysis (IIBA), which defines a business analyst as follows: “any person who performs business analysis tasks described in the BABOK^® Guide, no matter their job title or organizational role. Business analysts are responsible for discovering, synthesizing, and analyzing information from a variety of sources within an enterprise, including tools, processes, documentation, and stakeholders.”¹

1. International Institute of Business Analysis (IIBA), BABOK v3: A Guide to the Business Analysis Body of Knowledge, 3rd ed. (Toronto, Canada: IIBA, 2015), 2–3.

In practice, organizations often use more fine-grained role names for subsets of BA responsibilities. For example, in many of the organizations with which I work, the business analyst is typically someone concerned with high-level business requirements, whereas the team analyst is concerned with detailed product requirements. As a general rule, I avoid these distinctions in this book: a business analyst, or analyst, is simply any practitioner of BA. The exception is a context where it’s important to specify the role. For example, I may use the term team analyst when discussing guidelines that apply only to business analysts at the team level.

2.4 Why Agile Analysis and Planning?

In this chapter, you'll be looking at the research on the benefits of agile analysis and planning. To give you a more tangible feel, let’s follow the story of one specific, fictionalized company: Customer Engagement One (CEO) Inc.

CEO is developing an app, also called CEO, that enables any business to manage its engagements with customers across multiple platforms. The early version of the app was built as a simple, monolithic system using an agile methodology. The typical change request could be accommodated by a single agile team working independently. Consequently, teams got by with minimal preparation and planning.

As the product grew, however, it became more complex. A typical change now required more competencies than could fit within a single agile team. Teams now needed to collaborate more frequently, but doing so was difficult with their current minimal approach. They began to suffer from bottlenecks as they waited for others to perform work they were dependent on. Features delivered by individual teams were inconsistent, leading to rework and delays. Stories couldn’t be completed on time because there were essential requirements that were not well enough understood at the time estimates were made. The user experience of the app became fractured because teams didn’t have a shared understanding of the vision for the product. The company had tried to launch a product-wide, strategic initiative to provide a more consistent experience but had given up because there never seemed to be enough teams ready at the same time to pull off a change of that size.

At this point, the development lead for CEO realized that the organization needed to spend more time on analysis and planning and develop more effective processes for carrying it out. Staff were trained in the competency and deployed to agile teams. Product owners were more productive because they could focus on market-facing responsibilities while team-level analysts took the lead on day-to-day team-facing activities.

The planning process in the organization was changed so that a portion of each team’s budget was now reserved for strategic initiatives. These were planned quarterly. The rest of the budget was earmarked for customer-generated requests. These were planned using the Kanban approach to enable fast response to learning and customer feedback. Business analysts started to prepare features and user stories in advance of planning, resulting in improved estimates. This practice allowed teams to forecast completion dates better and coordinate their work in advance. As a result of these changes, the CEO was now able to take on large initiatives.

The previous story is not as fictional as I suggested. Other than the name, it’s the story of an actual company. The agile analysis benefits that it describes are available to any company willing to spend the time and energy to develop the competency.

2.5 The Parallel Histories of Agile and Business Analysis

Agile analysis and planning is, as noted earlier, a hybrid of two disciplines: agile planning and development and BA. Figure 2.1 indicates how these two approaches co-evolved from the 1940s to 2017. Significant events in agile development are indicated in roman; events in BA are indicated in italic; those related to agile BA are shown in bold.

Following is a brief survey of that history, as summarized in the figure. We focus first on the history of BA, then on the concurrent evolution of agile development.

2.6 Two Diagnoses for the Same Problem

It’s not entirely a coincidence that agile software development and BA both began around the same time—the 1990s and early 2000s. Both were responses to the same problem vexing software engineers at the time: a disturbingly high number of software projects were deemed failures. For example, the Standish Group’s 1995 CHAOS Report found that 31.1 percent of projects were canceled before they were completed and that only 16.2 percent of software projects were completed on time and on budget.⁴ Others reported that 64 percent of features were never, or rarely ever, used.⁵ Business analysis and agile development were both proposed as ways to resolve this issue. However, they proceeded from different diagnoses of the underlying problem. Let’s look at each diagnosis and the evidence regarding its success.

4. Project Smart, The Standish Group Report: CHAOS (Standish Group, 1995/2014), 3.

5. Mary Poppendieck and Tom Poppendieck, Lean Software Development: An Agile Toolkit (Boston: Addison-Wesley, 2003), 32. The authors quote a Standish Group study reporting 45 percent of developed features never used and 19 percent rarely used.

2.7 The Business Analysis Diagnosis

Business analysis began with the hypothesis that the root cause of project failure was communication—specifically, poor communication between business stakeholders and solution providers. There was ample research to back up this hypothesis. For example, the Standish Group reported at the time on the top ten causes of challenged projects.⁶ The top two factors pointed to poor communication problems: lack of user input and incomplete requirements and specifications.⁷ So did four other factors in the list: unrealistic expectations, unclear objectives, changing requirements, and unrealistic timeframes.

6. Project Smart, CHAOS, 9.

7. Project Smart, 9.

To address this root cause, some IT organizations began to define a new role—the IT business analyst. Initially, many business analysts were former systems analysts. I was one of those who made the transition—intrigued by the new competency because it included the stakeholder interactions I had come to enjoy as a systems analyst but without the technical aspects of the role. After years working at the cutting edge of technology, I was burning out and ready to make the switch.

Many other former IT developers did the same, but just as many moved to BA from business operations. The first business analysts that I trained were social workers who were about to be assigned to work as business analysts on a project to replace the case-management system they had been using. Today, I’d estimate that about half of today’s business analysts come from the business side. That’s a good thing because it means that those with in-depth knowledge of the users and business context for a software product can take a leadership role concerning its requirements.

2.8 The Business Analysis Track Record

The data indicate that the BA approach has been highly successful. Let’s begin with growth metrics. In 2004, one year after IIBA was founded, it had thirty-seven members. Today it has almost thirty thousand members, more than one hundred twenty chapters in over forty countries.⁸ PMI reports that “business analysis has become a competency of critical importance to project management.”⁹ It’s doubtful we would be seeing this kind of growth and acceptance unless the BA competency was providing significant value.

8. IIBA, “About IIBA,” 2020, https://www.iiba.org/about-iiba

9. Project Management Institute, “PMI Professional in Business Analysis (PMI-PBA)®,” 2020, https://www.pmi.org/certifications/business-analysis-pba

What do software development organizations themselves think about the importance of BA to their success? Some insight is provided by the Standish Group’s CHAOS Report 2015, which lists the main factors cited by organizations for project success. Table 2.1 summarizes the study’s results.

While the study doesn’t explicitly call out BA, three of the top ten factors listed are directly addressed by the competency:¹⁰

10. Shane Hastie and Stéphane Wojewoda, “Standish Group 2015 CHAOS Report—Q&A with Jennifer Lynch,” 2015, https://www.infoq.com/articles/standish-chaos-2015

1. User Involvement: Including user feedback, requirements review, basic research, prototyping, and other consensus-building tools.

2. Clear Business Objectives: The development of a shared understanding across all stakeholders and participants about the purpose of the project and how it aligns with the organization’s goals and strategy.

3. Optimization: A structured process for improving business effectiveness.

You might have noticed that one of the other factors is agile process. We’ll get back to that later in this chapter.

What does the data say about the impact of BA on cost? One of the most scientifically rigorous studies of its kind, the Business Analysis Benchmark,¹¹ found that organizations using poor requirements practices spent 62 percent more on similarly sized projects than organizations using the best requirements practices.¹² Figure 2.2 illustrates how this added cost, referred to by the study’s authors as the requirements premium, decreases to zero as requirements quality increases.

11. Keith Ellis, Business Analysis Benchmark—The Impact of Business Requirements on the Success of Technology Projects (Toronto, Canada: IAG, 2009), 2–3.

12. Ellis, Business Analysis Benchmark. Specifically, the study found that an organization with best requirements practices spent on average $3.63 million on a project originally estimated at $3 million, while an organization with poor requirements practices paid $5.87 million for a project estimated at $3 million—representing a 62 percent cost increase.

The benefits don’t end there. The report found that every measure in its study improved as requirements discovery and management maturity levels rose. These include:

• Percentage of projects delivered on time

• Percentage of projects delivered on budget

• Percentage of projects delivering all required features

• Percentage of projects deemed successful

The following are some of the other conclusions of the study regarding the benefits of BA:¹³

13. Ellis, 8.

• In companies that have only an average level of requirements competency, the lack of excellence will consume approximately 41.5 percent of the IT development budget on strategic projects.

• Companies in the upper third of requirements capability report 54 percent of projects on time, on budget and on function and pay 50 percent less for their applications. In contrast, companies with a poor requirements capability had three times as many project failures as successes.

• Companies with excellent requirements processes reported that over 70 percent of their projects were successful.

Importantly, these benefits apply across all software development approaches. So, if your team is already experiencing the benefits of agile development, it will experience even more benefits by adding an effective BA capability.

2.9 The Agile Diagnosis

In the late 1990s and early 2000s, as BA was being established, another hypothesis for project failures was evolving. It proposed that the root cause was the waterfall development process widely in use at the time and that the solution was to replace it with an approach referred to as agile development. Waterfall development is a predictive, sequential process wherein each step is completed before the next begins: first, a comprehensive needs and requirements analysis is performed up front; then a complete design is created and is followed by implementation, testing, and finally, release into production. Plans are made up front on the basis of requirements and other factors, and success is measured based on conformance to the plan. When projects failed, the waterfall answer was to enforce the process more strictly: do even more comprehensive analysis up front to ensure no critical requirements were missed and impose stricter controls on requirements changes. The problem was that no matter how hard companies tried, this approach didn’t do much to move the needle. (Recall that disappointing 1995 CHAOS Report.)

The early agilists began to wonder if the problem wasn’t with waterfall’s assumptions of stability and predictability—noting that the opposite assumption is more often true: that software development often occurs under volatile, unpredictable conditions. Requirements often change after they’ve been baselined because of changing customer behaviors, market competition, new technology, new learning, and other factors. Instead of spending more time on upfront analysis and planning, the agilists proposed spending less. Instead of relying on a preset plan that was doomed to obsolescence the moment it was signed, they offered an adaptive approach that uses feedback to revise planning decisions throughout development to optimize agility—the ability to create and respond to change in order to succeed in an uncertain and turbulent environment.¹⁴ The adjective agile can be used to describe a development process or organization that is responsive to change.

14. Derived from Agile Alliance, Agile 101 (Agile Alliance, 2020), https://www.agilealliance.org/agile101/

The agile toolkit draws from many sources. It includes workflow-management practices pulled from Kanban, guidelines for reducing waste from lean software development; events and techniques from the Scrum and XP frameworks; and scaled agile techniques from SAFe, DevOps, and continuous integration and continuous delivery (CI/CD). In the next chapter, we’ll look at all these foundational approaches and their impact on the practice of agile analysis.

2.10 The Agile Track Record

Agile development is no longer an interesting theory; it’s accepted practice. The 14th Annual State of Agile Report¹⁵ found that “95% of respondents report their software organizations practice Agile development methods” and that 61 percent have been doing so for three or more years. Fifty-one percent of respondents reported that more than half or all of their teams were agile. Today, it is the presumed approach in most development organizations that I encounter.

15. Digital.ai, 14th State of Agile Report, 2020, 8, https://explore.digital.ai/state-of-agile/14th-annual-state-of-agile-report

Agile process was also cited as one of the top ten success factors for software development initiatives in the Standish Group’s CHAOS Report 2015, as shown in Table 2.1. Table 2.2 summarizes the report’s findings concerning the success of projects using agile versus waterfall approaches.

As the table indicates, the use of an agile versus a waterfall method more than tripled the percentage of projects considered successful and reduced the percentage of failures to one-third that of the waterfall rate.

Agile development has also been shown to have a positive impact on productivity, with a 2015 QSM Associates study, commissioned by Rally Software, finding that “Agile projects experienced a 16% increase in productivity compared to [the] industry average.”¹⁶ It’s worth noting, though, that this metric doesn’t quite match the more extravagant claims made by Scrum’s authors when the framework was introduced: “When we say Scrum provides higher productivity, we often mean several orders of magnitude higher, i.e., several 100 percents higher.”¹⁷

16. Melinda Ballou, “As Agile Goes Mainstream, It’s Time for Metrics,” Rally Software Development, 2008, https://www.broadcom.com/products/software/agile-development/rally-software

17. Ken Schwaber and Mike Beedle, Agile Software Development with Scrum. (Upper Saddle River, NJ: Prentice Hall, 2002), viii.

One reason for the discrepancy is that if you do a feature-by-feature comparison, agile development probably costs almost as much as waterfall because savings are mostly offset by rework involved in agile’s trial-and-error approach. Its benefit, however, lies elsewhere: in the effectiveness of the investment. Agile methods help the business quickly determine which features to develop and then get that minimum marketable product (MMP) to market quickly. In that respect, agile has been highly successful: according to a QSMA study, “Agile projects are 37% faster to market than [the] industry average.”¹⁸

18. QSM Associates, The Agile Impact Report, Proven Performance Metrics from the Agile Enterprise (Boulder, CO: Rally Software, 2015), 4.

The evidence also suggests that agile development is leading to better products. In one study,¹⁹ a full 78 percent of participants believed that using agile had led to higher stakeholder satisfaction.

19. A Dr. Dobbs Journal survey quoted by Mike Cohn reported that 78 percent of participants believed that using agile had led to higher stakeholder satisfaction. See Mike Cohn, Succeeding with Agile (Boston: Addison-Wesley, 2010), 16.

2.11 Why Agile Teams Should Include an Effective BA Competency

With such a strong case for BA and agile development individually, you’d think the case for a competency that combined the two would be a slam dunk. The research bears this out. According to the Business Analysis Benchmark,²⁰ requirements maturity is closely correlated with success outcomes across all development approaches, including agile. Figure 2.3 illustrates its findings for organizations using agile development.

20. Ellis, Business Analysis Benchmark, from a file provided by the author on research for the report.

The figure clearly shows that if your agile team doesn’t yet include a strong BA competency, it should introduce one. For example, if your team is currently at the lowest requirements maturity level, it stands to more than double its success rates by moving to the highest maturity level. A single-step increase from level 1 to level 2 results in a 49 percent increase in success rates.²¹

21. Ellis, Business Analysis Benchmark.

Interestingly, the report also showed that the impact on success rates of requirements maturity level alone was far more significant than the effect of development methodology alone.²² If you hold the maturity level constant, the maximum improvement for organizations using agile versus waterfall methods is an 11.6 percent increase in project success rates (at maturity level 2)—not nearly as high as the doubling of rates that are possible when improving requirements maturity levels alone.

22. According to the report, the greatest improvement for organizations using agile versus waterfall methods occurs at maturity level 2, where there is an 11.6 percent increase in project success rates (from 54.77% to 61.94% successful). The benefit is down to 4.4 percent at level 3, while at the extremes (levels 1 and 4), agile success rates are actually slightly worse than those of waterfall.

2.12 Chapter Summary

Here are the key points covered in this chapter:

• Agile development was launched with the publication of The Agile Manifesto in 2001.

• IIBA was founded in 2003.

• Research shows that increasing BA maturity levels results in improvements in the percentage of projects delivered on time, percentage of projects on budget, and percentage of projects delivering all required features.

• Research shows that an agile team that goes from a low requirements maturity level can double its success rates.

2.13 What’s Next?

In this chapter, we looked at the business case for agile analysis and planning. In Chapter 3, “Fundamentals of Agile Analysis and Planning,” we look at the fundamental concepts behind the competency.