Agile didn’t begin as a single idea or as the product of a single person’s experience – it evolved over decades, and we can trace elements of Agile to numerous other ideas, approaches, concepts and theories dating back as far as the 1600s. However, it was at a single event in 2001 when the word ‘Agile’ came to be applied to these things.

The mid-20th century saw the early days of software development, and the methods being used tended to mirror those of earlier manufacturing and hardware approaches. They were linear, with each aspect of development being completed before the next would begin. This made sense with hardware, but some of those applying this approach to software began to see problems with it. They started experimenting with different methods to order and manage the work, and they began to talk about their findings with each other. The ideas emerging from this small minority of developers gradually spread through papers, articles, conferences and word of mouth.

As their ideas matured, many of them began developing those ideas into brands and methods they could market to others. Yet, despite their differences, these people and their approaches shared much in common. One trait they shared was their desire to be collaborative and the notion that working together can reach a better outcome than working individually. This was partly what drew them together in February 2001 at a ski lodge in Utah, where the common ground they shared emerged as the Agile Manifesto.

This chapter explores that journey and helps you to understand how the richness of Agile’s heritage can explain why it is so successful. We describe the traditional, linear, ‘heavyweight’ approaches that were (and to some extent still are) dominant, and explain some of the alternative approaches that influence Agile. Finally, we describe how the 17 authors of the Agile Manifesto managed to reach such profound and influential common ground despite seemingly being in competition with one another.

Even 20 years ago, the only thing new about ‘Agile’ was its name. This chapter helps you to understand why and provides you with a solid basis to the rest of the book where we help you to understand Agile in more depth.

THE ESTABLISHED APPROACH TO DEVELOPMENT

Before we start to think about Agile, it is worth first considering the alternatives to developing systems and solutions prior to the Agile Manifesto. These are still relevant and have their place in systems development today.

The linear approach

There are many approaches to delivering solutions, but they mainly break down into two types: an Agile approach or a linear approach. Prior to the emergence of Agile, most development took a linear approach in which the main building blocks of the solution were carried out sequentially, with one being completed – and usually signed off – before the next was started. By putting enough effort and talent into each stage, and by applying rigorous processes and scrutiny, the team expectation is that each element (requirements elicitation, analysis, design, test, etc.) will be correct and therefore flow into the next step. Change is managed as an exception activity within the steps and is minimised so as to not cause delay. If there are multiple versions or releases, each version would follow its own linear approach.

In 1970, Dr Winston Royce wrote a paper in which he reflected on the previous nine years he had spent at Lockheed building software packages for spacecraft mission planning, commanding and post-flight analysis.¹³ In the paper he drew several diagrams to illustrate his points. One has proved particularly influential and been coined the Waterfall model (see Figure 2.1).

This model describes the seven steps that are required to successfully deliver a software system. Although Royce did not intend this to be a recommendation of how software should be built, that is how it was interpreted and how many organisations apply it. In fact, Royce describes this approach as ‘risky and invites failure’ and that it has ‘never worked on large software development efforts’. The rest of his paper proposed alternative approaches that bear more similarities to Agile approaches than most people think.

Despite his reservations at the time, Royce’s Waterfall model is still a popular way to develop software and deliver solutions. Even when teams try to be Agile, they often end up with an approach that resembles the Waterfall, with most of the analysis at the start and delivery delayed until the end.

When considering linear approaches, there are two main facets that are distinctly different from Agile approaches. They are:

• how the work is decided;
• how teams are managed and led.

How the work is decided

In linear approaches, such as Waterfall, the work is decided in advance based on up-front requirement and analysis steps. This aims to achieve a level of certainty and accuracy through rigorous analysis. In some ways this makes sense: do each stage once and do it well. It will be cheaper to fix mistakes in analysis than in design and cheaper to fix mistakes in design than after release.

So, it makes sense to put a lot of focus on getting analysis and design correct before moving onto implementation, test and release. However, as we will discuss throughout this book, this depends on it being possible for up-front analysis to be (and remain) true. This is increasingly rare in today’s real world projects and distinctly different from the pace of change encountered through the mainframe systems of the 1960s and 1970s. Deciding a complete project’s worth of work in advance seems implausible in the volatile, uncertain, complex and ambiguous world we live in today and hence why Agile adoption has become far more appealing.

How teams are managed and led

Management in linear approaches is typically top-down command and control, with managers instructing team members what to do. Work is planned in advance, assigned to teams and monitored by managers. More experienced engineers are promoted to lead or manager roles, and it is generally assumed that the more senior you are, the more skilled and experienced you are. Decisions tend to be escalated up to managers to make and cascaded down to the workers to implement. Chapter 9 discusses this type of leadership (Theory X) in more detail.

Work is typically managed as a project, with each step detailed in a plan, usually in advance. The project manager creates a product breakdown structure and a work breakdown structure. Each work item is then estimated and a project duration established. To reduce risk, each stage includes completion sign-off, after which the artefacts created are generally not changed without invoking a change process. Since change gets more expensive the further down the waterfall you go, change is heavily resisted and getting confidence in each step before starting the next is really important.

INFLUENCES ON AGILE

There is a rich mix of influences on Agile, dating back to 1620 when Francis Bacon first described the scientific method of empiricism that is at its heart (see Chapter 6). Many of the influences are from the software world, but by no means all of them. Rigid production lines with each machine carrying out a fixed task may seem like the antithesis of Agile, but the early 20th-century pioneers of manufacturing such as Henry Ford laid some of the foundations for it. Their focus on efficiency, standardisation and automation made sense for relatively simple problem spaces; so too with Agile, where continuous improvement and delivering value as quickly as possible means automating processes where possible.

The Toyota Production System

The Toyota Production System evolved from Toyota’s origins making weaving looms and creating automated production lines for textile mills.¹⁵ Based on the philosophy of eliminating all waste, it has a strong culture of continuous improvement and empowering workers on the production line.

One example of empowerment is the Andon Cord – a rope or button that will stop the production line. Any worker who spots a problem is expected to pull the Andon Cord so that the abnormality can be addressed. Any use of the cord results in a ‘thank you’, not a reprimand, thus encouraging workers to pull the cord not just when they are sure something is wrong, but as soon as they suspect something is not right. This focus on quality over speed is what sets Toyota apart from its competitors.

However, Toyota also focuses on speed. The philosophy of jidoka (automation with a human touch) means that the ingenuity of humans design the system, but once shown to have value, processes are automated. Then a kaizen (continuous improvement) approach is applied to steadily simplify, improve and make the element more efficient. Many of these improvement ideas come from the workers, who are empowered to suggest and test possible improvements.

The Lean movement was largely born out of the innovations of Toyota and has itself been hugely influential, not just on Agile but throughout manufacturing and beyond.

The New New Product Development Game

In 1986 the Harvard Business Review published an article by Hirotaka Takeuchi and Ikujiro Nonaka called The New New Product Development Game.¹⁶ They had observed that the rules of the game of product development were changing. It wasn’t good enough to take years to bring a new product to market anymore; the most successful companies were bringing products to market much more quickly, and they were cheaper and better.

When they looked at some of these products, they found that none of them were applying a linear approach to development – that is, one where research and development pass ideas onto engineering, who pass designs onto production, who pass products into marketing, which they characterised as being like a relay race, where the baton is passed from one team to another. Instead, products were being developed by an integrated team; one that behaved more like a rugby team, all moving forward together, passing the ball between them.

The paper didn’t just introduce the rugby metaphor that would inspire Scrum, it also described several core behaviours that can be found in the Agile Manifesto and in the Agile methods that would follow:

• self-organising teams;
• iterative development;
• subtle leadership;
• empowered teams;
• engineers encouraged to go out and talk to customers;
• striving for constant iteration in a world of constant change.

What’s interesting about the New New Product Development Game is that none of the products investigated were software products – they were paper copiers (Fuji-Xerox), cameras (Canon), a car (Honda) and a PC (NEC) – yet their conclusions were remarkably similar to those from the software community who created Agile.

Iterative development

The integrated ‘rugby’ method recommended an iterative approach to development with overlapping development phases and iterative experimentation. Sixty years earlier, at Bell Labs, the engineer and scientist Walter Shewhart was describing the scientific method as ‘specification – production – inspection’ and using a scientific, empirical approach to improve manufacturing quality. His success led to him publishing a book in 1931,¹⁷ and later turning his control chart ideas into the Shewhart Cycle in 1939.¹⁸ W. Edwards Deming would later popularise this in Japan in 1950, where it became known as the Deming Cycle, and as the plan-do-study-act (PDSA) cycle that defines continuous improvement¹⁹ (see Figure 2.2).

This iterative approach was also advocated for large software projects. In 1986, Barry Boehm was working on US defence software projects when he published a paper on his ‘Spiral Model’ (see Figure 2.3).²⁰ The solution is developed across several iterations, each going through the same development stages with each iteration building on the lessons learned from the previous iterations.

Around the same time, Dr Ivar Jacobson was using his experience at Ericsson to come up with a new way to manage requirements he called ‘Use Cases’.²¹ A decade later he joined forces with James Rumbough and Grady Booch to create the Rational Unified Process – a tailorable framework for iterative software development with a focus on modelling and component architectural design.²²

The lightweight software development methods

Over the 1990s, software development exploded in popularity and sophistication. What had been the preserve of large organisations with large mainframe computers and bespoke languages was democratised with the advent of the PC. Even within those large organisations, pioneers were finding alternative ways to develop software that didn’t follow the traditional linear and Waterfall development approach.

Approaches such as rapid application development (RAD), extreme programming (XP), Crystal, feature driven development, adaptive software development and Scrum were emerging. They were termed ‘lightweight’ development methods as they were not as ‘heavyweight’ as the established approaches such as the Waterfall approach or the V-Model.

Although ‘lightweight’, many of these approaches emerged from large product development activities: XP from the C3 Project at Chrysler (a payroll system);²³ Scrum from the New New Product Development Game, meeting culture at Borland²⁴ and the complex process control theory at DuPont;²⁵ and DSDM (formally known as the Dynamic System Development Method) has its origins in RAD that emerged from blue chip organisations like British Airways, Oracle and ANZ Bank.²⁶

THE MANIFESTO STORY

Many of the influences mentioned earlier had been around for a long time, but it was one event in February 2001 that catalysed them into what we now know as ‘Agile’ and resulted in the Agile Manifesto. On its 20th anniversary, Fadi Stephan published an article describing the event from behind the scenes.²⁷ In that meeting, 17 people got together in a ski lodge in Utah where, over the course of three days, they managed to find common ground for their different perspectives and products and created the Agile Manifesto.

What is surprising is that they were able to find common ground at all, since they represented several different theories and approaches and were, in some respects, competitors of one another. Indeed, one of the participants, Alistair Cockburn, observed ‘I personally didn’t expect that this particular group of agilists to ever agree on anything substantive.’²⁸

The various backgrounds

The original idea to get together to identify common ground came as a result of some discussions at an XP retreat about whether XP should extend beyond its focus on technical software practices. This led to Martin Fowler reaching out to Alistair Cockburn, who was developing the Crystal method and had arranged similar meetings in the past. They thought they should get themselves, and their respective contact lists, together to talk about this more.

The invite list grew, and included people such as Jeff Sutherland, Ken Schwaber and Mike Beedle (who were popularising Scrum), Joe Kern (feature driven development), Arie Van Benneken (DSDM), Jim Highsmith (adaptive software development) and many others, including some specifically to bring in different perspectives such as testing or more formal processes. 17 of the invitees were able to attend. They settled on Utah as the venue largely for logistical reasons (the airport was easier to get to and some of the main organisers lived there).

The lightweight methods conference

They arrived on Sunday 11 February and would spend the next three days in a fairly relaxed agenda that included time to ski and laze in the hot-tub but also several hours of discussion a day. The aim was to understand what everyone wanted to achieve, set up a website to share their thinking and to come up with a manifesto they could all commit to signing. In particular, they wanted to counter the notion that chief information officers needed large, heavyweight frameworks, lots of documentation and lots of people to ensure their projects would ‘be safe’ and deliver.

Having got to know one another better and learned about each other’s different methods, they approached the first hurdle: what to call this thing. Nobody was keen on being described as ‘lightweight’, and some of the other candidates, such as ‘lean’ or ‘adaptive’, had already become associated with specific approaches – they settled on ‘Agile’.

There was a lot of good discussion, but it took Dave Thomas and Martin Fowler to grab a pen and propose some values to catalyse the group into focusing on something tangible to agree on. The group discussed and iterated the values through several versions (including debating which order they should take) before settling on the four that we describe in Chapter 4.

The idea of a set of principles was discussed, but they didn’t have time to discuss them in more depth; that happened over email exchanges over the following weeks, with a draft published in April 2001. Ward Cunningham (inventor of the Wiki) then set up a website to promote the manifesto and allow visitors to show their support by signing up. They formed the Agile Alliance later that year as a non-profit organisation to further Agile methods.

SUMMARY

Although it formally emerged in 2001, the roots of Agile can be traced back as far as Francis Bacon’s scientific method, first described in the 1600s, and to the manufacturing innovations that revolutionised textiles and motor manufacturing in the early 1900s.

However, it was the surge in new ways to develop software in the 1990s that were the strongest influences. As various innovators and pioneers of software rebelled against the established ‘heavyweight’ processes that dominated at the time, they came together and found that they could all agree on a simple expression of their values and beliefs that became the Agile Manifesto.

What is perhaps most remarkable is that 17 people with different approaches, many trying to sell or promote their approach in competition with one another, were able to agree on common ground in just three days. And that common ground has stood the test of time.

13 Royce, W. W. (1970) Managing the development of large software systems. Proc. IEEE Wescon, 26, 328–388. https://www-scf.usc.edu/~csci201/lectures/Lecture11/royce1970.pdf.

14 Jesko, G., Lamm, S. R. and Walker, M. (2016) Model-Based System Architecture. John Wiley & Sons, Hoboken, NJ.

15 https://global.toyota/en/company/vision-and-philosophy/
production-system/.

16 Takeuchi, H. and Nonaka, I. (1986) The new new product development game. Harvard Bus. Rev., January. https://hbr.org/1986/01/the-new-new-product-
development-game.

17 Shewhart, W. A. (1931) Economic Control of Quality of Manufactured Product. Van Nostrand Co. Inc., New York. Reprinted in 2015 by Martino Fine Books, Eastford, CT.

18 Shewhart, W. A. (1939) Statistical Method from the Viewpoint of Statistical Control. Graduate School, Department of Agriculture, Washington, DC.

19 Deming, W. E. (1982) Out of the Crisis. MIT Press, Cambridge, MA. Earlier in Deming’s career he used plan-do-check-act (PDCA) but changed ‘check’ to ‘study’ as he felt that ‘check’ emphasised inspection over analysis.

20 Boehm, B. (1986) A spiral model of software development and enhancement. ACM SIGSOFT Software Eng. Not., 11 (4), 14–24.

21 Jacobson, I. (1992) Object Oriented Software Engineering: A Use Case Driven Approach. ACM Press, Addison-Wesley, Boston, MA.

22 Jacobson, I., Booch, G. and Rumbaugh, J. (1999) The Unified Software Development Process. Addison Wesley, Boston, MA.

23 https://martinfowler.com/bliki/C3.html.

24 Rigby, D., Sutherland, J. and Takeuchi, H. (2016) The secret history of Agile innovation. Harvard Bus. Rev., April. https://hbr.org/2016/04/the-secret-history-
of-agile-innovation.

25 http://leanmagazine.net/wordpress/wp-content/uploads/
The-Scrum-Story.pdf; https://kenschwaber.wordpress.com/2010/06/10/waterfall-leankanban-and-scrum-2/; Highsmith, J. (2002) Agile Software Development Ecosystems. Addison-Wesley, Hoboken, NJ.

26 https://thedigitalprojectmanager.com/dsdm-consortium/; https://www.agilebusiness.org/page/history/.

27 Stephan, F. (2021) A behind the scenes look at the writing of the Agile Manifesto. Kaizenko. https://www.kaizenko.com/a-behind-the-scenes-look-at-the-
writing-of-the-agile-manifesto/.

28 History: The Agile Manifesto. Agile Manifesto.org. https://agilemanifesto.org/history.html.