The term Agile began with the Manifesto for Agile Software Development: a powerful document that is the foundation of all Agile approaches and remains just as valid today as it was when it was signed in 2001. For many, the Manifesto is a footnote in an Agile training course or certification where the main focus is on applying a process or framework. Unfortunately, skipping over the Manifesto values, or giving then scant regard, risks misunderstanding or, worse, abusing them. This is a common problem. Many believe they are ‘Agile’ following completion of an Agile course, yet when we compare their behaviours against the Agile Manifesto values, we find that they are not really Agile at all.

Every Agile book will mention the Agile Manifesto, but most just list the values and principles with little regard for the changes in behaviours expected by the reader to achieve those values. We believe that the Agile Manifesto values are so critical that they deserve more explanation and consideration. This chapter takes you through the Manifesto and discusses each value statement in depth. As you will see, each value statement has two parts – one side, the left-hand side, is a statement that Agile teams value highly; the other side, the right-hand side, is a statement that Agile teams want to de-emphasise and value less.

For each value, we will help you understand it from three perspectives:

• What the value is all about and why it is useful.
• What people find challenging about the value.
• Some anti-patterns we have observed that cause people to value the left-hand side less or the right-hand side more.

By understanding the Manifesto in this way, you will gain a deeper understanding of what it means to be Agile and how the Manifesto values can help you achieve that.

THE MANIFESTO FOR AGILE SOFTWARE DEVELOPMENT

As we discussed in Chapter 2, the Agile Manifesto was the result of three days of discussions in 2001 between a group of thought leaders and development framework experts. Despite coming from different backgrounds and having different experiences, they agreed on a short form of words that summed up how they all felt software development ought to be done to have the greatest chance of success, as shown in Figure 4.1.

In just 68 words they managed to convey a number of powerful concepts and proposed an approach to developing products, services and solutions that was radically different from how most projects were being run. Many teams and organisations still continue to value the attributes on the right-hand side of the Manifesto, even thought they would claim they are following an Agile approach. You will see us reiterate many times throughout this book the difficulty in adhering to the Agile Manifesto values and principles despite how straightforward it seems. In fact, they are very hard to stick to and we constantly see individuals and teams struggle with the concepts.

Each element of the Manifesto deserves some attention and thought, starting with the first sentence, which sets the tone:

We are uncovering better ways of developing software by doing it and helping others do it.

It makes clear that we are practitioners that ‘do’ the development of software ourselves and not managers or consultants. We are directly involved in the work. Agile teams are teams where everyone is involved in creating the value. That doesn’t mean everyone needs to be expert software coders, but it does mean there isn’t room for people whose sole role is to set tasks for others or oversee the work.

THE VALUES

At the heart of the Agile Manifesto are four value statements. They are particularly powerful because of the way they are framed. Each sets out two contrasting viewpoints with one valued more than the other. Unlike most sets of values, this not only sets out what we should do, but also what we should try to avoid.

The final sentence of the Manifesto is often misunderstood:

That is, while there is value in the items on the right, we value the items on the left more.

While it is clear that the items on the right of the Manifesto have value, that doesn’t mean that they need as much attention as those on the left. One reason for this is that many organisations and teams are already very comfortable with the items on the right; our working practices and management approaches overwhelmingly favour values such as planning, control, consistency and formal documentation. This means that, by default, many people will naturally favour the statements on the right of the Manifesto. The challenge isn’t reminding teams that there is value in the items on the right – they already value those things. The challenge is in helping teams shift their default behaviours towards the items on the left.

We will now explore each of the values in turn, understanding what underpins them and why they are so significant. We will share practical advice and tips and look at common challenges of both the left- and the right-hand statements contained within the Manifesto.

Individuals and interactions over processes and tools

This statement is all about people – because it is people who make the biggest impact on the success of delivery (see Figure 4.2). It is people who create the solution, and it is people who will be using the solution. Getting from the problem to a good enough solution therefore requires lots of communication and interaction between lots of people. Some of this interaction can be automated or standardised, but it is possible to take this too far – to the extent that the process becomes more important than the people. You may have experienced this yourself if you have ever become lost in the multiple menus of automated call centre systems, struggling to find a combination of options to match the reason for your call.

To understand this value, let’s consider the software domain that the Manifesto authors were familiar with. The creation of software products to meet a user need is a human endeavour. It requires people to make decisions, consider options and apply judgement. Every software product is the result of a unique combination of actions taken by people. This uniqueness means that what works well in manufacturing does not work well for software or other types of knowledge work; following the same standard process each time will not predictably meet the needs of the customer. Many people are engaged in knowledge work that, like software, overwhelmingly requires decisions to be taken, data to be analysed and judgement to be applied – work that is unique, just like the software domain the Manifesto originated from.

This Manifesto value reaches far beyond the domain of software development. When we overlay the fact that our world is more VUCA – volatile, uncertain, complex and ambiguous – it is clear that trying to predict and codify everything into a standard process or tool will be futile. Tools and processes have their place, but can only help to a certain extent – delivering the best outcome for the customer will always rely more on people and human interaction.

Where we can get value from processes and tools is when we apply them to the right type of problem – those that are simple or repetitive, such as automating tests, tracking version control of artefacts, delivering simple training as e-learning or describing ways of working. They can also be valuable for more complicated situations that do not stretch into the complex – for example, tracking dependencies between system components, managing build pipelines, analysing performance data or controlling financial data.

Valuing individuals also means trusting them

An implication of valuing individuals is that we must also trust them. When things go wrong, it is rarely because people are not good enough, yet that’s often our first thought. In his book, The New Economics,³¹ Dr W. Edwards Deming attributes the blame for problems on the system, rather than other factors such as the people, 94 per cent of the time. We should assume noble intent in our people. The vast majority of us come to work intending to do a good job, are trustworthy and will make decisions we believe are the right ones. This is a powerful statement and important for Agile teams.

Placing high degrees of trust in our people can be unnerving. We are used to systems that place protections aiming to prevent unwanted behaviour; however, such processes affect everyone, not just the few exhibiting such behaviour, and this can have a negative effect. Even seemingly benign processes exhibit this behaviour. For example, a human resources (HR) performance management process that rewards individual effort is intended to incentivise people to work to their maximum effort, but it can have a negative effect on overall team performance because people are not incentivised to work for the good of the team.

Instead, it is preferable to use processes that only trigger if the unwanted behaviour occurs. These then only affect the small minority of people who are not behaving with noble intent. Trusting people and balancing that trust with setting direction and leading is core to good Agile leadership, as we discuss in Chapter 9.

What makes this value hard to apply?

The key to this value is to treat people as people: value their opinion, remove barriers to communication and work collaboratively. However, this isn’t always as easy as it sounds.

Existing processes constrain Agile behaviour Many organisations have existing processes, some of which may be mandatory, that inhibit a focus on people. Some common examples include staff management processes, quality assurance checks, internal review stages, progress reporting, corporate tooling and standardised job roles.

Processes can become ingrained in organisational culture, which can lead to a focus more on adhering to the processes rather than adapting them or challenging them. Just because a process exists doesn’t mean that it must be followed every time; not all processes are mandatory, and even those that are ought to be subject to review and continuous improvement. However, all too often that isn’t the case, and it isn’t always easy to propose changes.

Over-reliance on tools Tools can also become dominant. When we are engaged in complex work, tools can help to manage the complexity and make things easier, but this is not always true. Sometimes tools can make work more complex, not less. One example is job tracking or work management tools. Because they can handle hundreds of tickets or thousands of work items, teams that use them can end up struggling to manage huge backlogs of work.

Teams that stick to simple processes – such as paper boards and sticky notes – can’t track such volumes and so are forced to prioritise better and focus on finishing work to create space for new work to be captured. This makes them more efficient and more Agile, even when they are a similar size to the teams with hundreds of tickets in the job tracking tool.

Face-to-face collaboration isn’t always feasible Valuing individuals and interactions is easier when engaging with people face to face, where we can look each other in the eye and pick up non-verbal communication such as body language. Being physically in the same place enables humans to more quickly establish trust, which is essential to good team performance.

Remote collaboration and more distributed and homeworking has become more commonplace now. When teams are distributed, particularly over different time zones, it can be harder to work well together, and processes and common tooling can really help; however, be careful the tools don’t dominate at the expense of human relationships.

It isn’t just teams that can be more successful with high levels of interaction; products can be too. When a product is collaboratively developed by multiple people together, it tends to have higher quality, be easier to support and to perform better. This is because the constant communication within the team as they develop the solution brings quality reviews, redesigns and deeper customer understanding for free.

Anti-patterns

Table 4.1 shows some of the anti-patterns and problems we have observed when trying to value individuals and interactions or avoid over-valuing processes and tools.

Working software over comprehensive documentation

This statement is all about the specifics of what is delivered at the end of each iteration of work, and how valuable it is (see Figure 4.3). Unfortunately, the statement suffers from including the ‘software’ word, which can detract from its real impact, which is the other word – ‘working’. ‘Working’ is the significant word. Software isn’t working if users aren’t able to use it to solve their problems, and that is true for all types of product or solution.

For users to get value from the solution, it must be available, they must be allowed to use it, and it must use real data to solve real problems. This can mean that in order to be ‘working,’ the solution requires a certain amount of documentation. For example, you may need to document data policies to be allowed to access another system, provide training material for users or create images for marketing.

Documentation is just another aspect of the solution and should be updated and evolve just like the other solution elements – with just enough documentation created just in time for the users who need it. Each iteration of the solution should include just enough documentation to support deployment, maintenance and customer needs.

How much documentation is enough?

Achieving ‘just enough’ documentation is a delicate balance that requires constant iteration and evolution. There is often a tension between the team and their stakeholders over how much documentation is enough. If in doubt, identify who the documentation is for (e.g. users, system support) and ask them how much documentation is enough, and what format they want it in.

Too much documentation can be a poor investment of time, so how can we get the balance right? There are broadly two types of documentation most teams need to care about:

• governance documentation to provide information on the health, progress and viability of the work or delivery;
• product documentation to provide information to the users, supporters and future developers of the product.

Understand your audience

For both types of documentation, we should try to minimise the effort it takes to produce it. Often the development approaches chosen can be self-documenting, particularly for software products, removing the need for additional design documents; or, instead of creating a progress update report for a stakeholder, they could just be given access to the work tracking system used by the team, or come into their room and ask them.

Understanding your audience in this way is important for all documentation. For example:

• The needs of early adopters familiar with the problem space are different from customers forced to use a new tool they aren’t used to.
• Technical users have different needs from non-technical.
• Documenting in an accessible way for disabilities requires additional thought.
• Expanding to new markets may require knowledge of different languages and cultures.

This variety of documentation should also be approached in an iterative and incremental way, with just enough documentation for the current product and its users. It can be tempting to do too much because you know it will be needed for later iterations – but that assumes later iterations will happen and they will be what you expect. Neither of these things are guaranteed, so it is better to wait until you are sure before creating that documentation.

One exception to this is accessibility. Even if you don’t have any disabled users for early iterations, building in good accessibility at the start can make things much easier, and in some sectors (such as within UK Government³²) can be a legal requirement. Meeting accessibility standards³³ tends to improve a product for all users, whether they need accessibility features themselves or not.

What makes this value hard to apply?

This should be one of the easier values to get right – after all, delivering a product to customers is clearly a beneficial thing to do – yet we frequently see teams spend months without their customers using the product. Additionally, development tools make it easy to produce lots of documentation that is tempting to publish and maintain.

We see documentation as more valuable than it is One of the biggest challenges is that creating documentation and attaching importance to it has become commonplace in modern development practices. Traditional development approaches place high levels of value on documents, since they are deemed to be a primary way to reduce risk and ensure delivery, and are even used as a measure of progress. Documenting requirements, detailing a plan to deliver them and getting sign-off from the customer before the expensive work of development begins seems to make sense. It is only when you bring in the challenges of a VUCA world described in Chapter 1 that this approach seems wrong.

While documentation can often be an indicator of potential value, we can only confirm true business value when the solution is actually being used by a user or customer. No amount of documentation can achieve the same result.

Not delivering working solutions The second major challenge we observe is teams who continue to deliver solutions at each iteration that cannot be used properly by their customer – in other words, they are not ‘working’. A solution cannot be used by its customer if it is still in a development environment, isn’t approved to be deployed, can’t be bought, only solves half a problem or the users aren’t trained or authorised to use it.

This manifests in several ways. When there are separate development, testing and deployment teams, it is common for the development team to think they are finished when the development is passed onto the test team. Unfortunately, the customer doesn’t have value yet and, worse still, when the test team find problems, they may not be passed to the development team until the next iteration. This can mean a delay of two iterations or more, as shown in Figure 4.4. In contrast, a cross-functional Agile team (see Chapter 7) delivers complete features each iteration (see Figure 4.5).

Cross-functional teams delivering every iteration isn’t always enough. Many teams use multiple environments to avoid development versions of the product causing problems in a production environment. When a team deploys a solution to one of these non-customer locations, it cannot be working for the customer since the customer cannot access it themselves for their real job.

Access to live data can also be a challenge for some teams, particularly in highly regulated or secure environments or where the product hasn’t launched to market yet. This is another example where it cannot properly ‘work’. Although the impact can be mitigated a little through test or synthetic data, it is always better to deploy a live product to live customers, even when there is more development to do.

Delaying deployment Sometimes teams could deploy their product but choose not to. This is often because the increment only includes a subset of the solution or just one aspect is ready. The team or the customer suggests it would be better to wait until it is more complete before allowing it to be used.

This misses the fundamental point of Agile approaches: deliver value early to the customer so that you can validate your solution and gather feedback on how well it is meeting the user need. The more frequently we do this, the quicker we get feedback on what is good and bad about our product, and the quicker we can respond to change. If the customer has changed what they need, we will know about it faster and can pivot, reprioritise or even stop the delivery.

Delaying this will not only delay any value to the customer but, more importantly, delay any feedback that we may be on the wrong track. And when that feedback does come, it will be longer and more expensive to fix. We often see this in organisations that value perfection when, ironically, delaying release until the product is perfect increases the likelihood that it will not be perfect.

Wherever possible, we should deploy the version of the product that we have, and that deployment should be to help real customers do their real job. We should ensure that our customers can use each increment of our product by knowing the criteria for use; split the work into small, user-value centric goals rather than components of the solution; and have all the skills, approvals and authorities needed to take the product live within the team or within easy reach.

Anti-patterns

Table 4.2 shows some of the anti-patterns and problems we have observed when trying to value working software or avoid over-valuing comprehensive documentation.

It is hard to overstate the importance of this value and the frequency with which it is ignored, even in teams who are trying hard to be Agile. We see teams create a backlog and earnestly work their way through it. Each iteration they make some progress, but don’t have a version of the product that customers can use. Reviews focus on stories closed rather than on demonstrating the product or getting feedback from it being used by customers. Months pass before a version is presented that the customers can use, and inevitably it doesn’t quite do what they need it to.

However, this is also the simplest value to do well and to improve. Simply focusing on having a working version of your solution available at each iteration is enough; preferably one that your customer will use. And that’s a simple thing to test and measure.

Customer collaboration over contract negotiation

This value is all about people, the power of talking and face-to-face communications over written agreements (see Figure 4.6). It also returns to the theme of assuming that we can’t know everything we would like to in advance.

By definition, a contract negotiation has a point when it is finished: when both parties sign. That implies that neither party expects it to change after this point. Contracting for something requires 100 per cent certainty, yet we know that 100 per cent certainty isn’t possible in a VUCA world and it would be foolish to assume otherwise. We know that, in reality, very little is absolutely clear – most things aren’t a binary yes or no and we expect frequent change. In order to navigate high change and low certainty situations, collaboration is critical and the best people to talk to will usually be our customers: high change requires high collaboration.

This value only has four significant words, and each one carries nuance and detail that is worth exploring.

Who are our customers?

The first question is what do we mean by the word ‘customer’? For commercial products that might be quite simple: they are the people who will purchase and use the product. It isn’t as clear-cut for internal products or services, where the person funding the work may be different from the person using the product. At least, internal customers are usually easy to identify; potential purchasers of a product that isn’t yet on the market are less visible.

The customers with whom we should collaborate are many and varied. If someone has (or should have) a stake or an interest in the product, they should probably be regarded as a customer. That includes other internal departments, such as audit, marketing, assurance or sales, who may not be direct users of the product but are still interested in it. This is because their perspectives or requirements could significantly affect how the product needs to behave, and it’s important to know whether this is the case or not.

Collaboration is more than just talking

To collaborate we must work together. It implies an equal relationship and a shared goal. It isn’t just keeping someone informed or asking them to keep you updated. It is a partnership.

Having a common goal that both the development team and the customers are striving for is important. It helps keep everyone aligned and honest. Agile teams don’t create products or software; they solve their customers’ problems. They don’t build a feature; they know why that feature is needed and what impact it will have. Getting that insight requires teams and customers to work together throughout the product development. It requires them to truly collaborate.

While we can collaborate asynchronously and using written documents, we will be much more successful in achieving a shared goal when that collaboration is in person. Trust is easier to build when we have as many paths of communication open as possible. Conversation is often richer than text and less likely to be misunderstood.

Non-verbal communication is also important for the message to be understood properly. There is a widely quoted statistic,³⁴ based on research by Albert Mehrabian,³⁵ that only 7 per cent of communication is verbal. Even if 7 per cent feels too low, there is still a significant contribution to the message that comes from the non-verbal elements. Non-verbal communication is particularly valuable when establishing trust. We can use it to show interest, agreement, confusion and many other emotions that are hard to convey as quickly or powerfully in written exchanges. This means that where teams are working remotely, or have fewer face-to-face meetings, they need to find other ways to build the trust necessary for effective collaboration.

There are lots of different types of contract

The second part to this value is often dismissed without much thought, especially when people assume the contract in question is the legal type of contract that not all teams have. The contract, however, does not need to be a legal agreement. Instead, think of a contract as any agreement that we make and will be held to account for by another party.

These include commercial or legal contracts, but they also include anything that has been signed off, such as requirements specifications, non-functional requirements catalogues, definitions of done and ready, test specifications, interface definitions, product features, performance figures, marketing plans, build schedules, communications plans and many, many more. Each one of these is an opportunity to over-value the agreement and undervalue the collaboration with the other parties.

But teams do need some contracts, and they need to adhere to them. Ideally, these are contracts that they also have a stake in creating and maintaining. Examples of beneficial ‘contracts’ include:

• definitions of done and ready;
• acceptance criteria for deliverables, features or stories;
• quality, security and assurance thresholds or standards;
• accessibility standards, and documentation formats and standards.

Even with these, we still want to keep options open to change them in the future. The more locked down and harder to change our ‘contracts’ become, the harder it is to be Agile.

This is similar to ‘processes’ in the first value. Agreeing things in advance can bring consistency that may be valuable to many teams, especially those in large organisations; however, there must be a balance. For anything fixed, it is important to know the criteria for changing it. Each time a standard or process is invoked is potentially an opportunity to improve (or remove) it. This means that if we treat them as contracts, then we must know when we can renegotiate those contracts. This is also true for agreements that are wider than one team, and that requires collaboration.

Negotiation implies winners and losers

The final word of this value – negotiation – is another interesting choice. Negotiation implies that the two parties start from different standpoints and that some give and take will be required to reach an agreement. This is because negotiation also implies that both parties have different goals. If we didn’t have different goals, we wouldn’t need to negotiate. That’s not what we want in Agile teams. Agile teams and their customers should have the same goal. Agile teams collaborate with their stakeholders to get win–win outcomes; they don’t negotiate with them to get to win–lose or lose–lose outcomes.

What makes this value hard to apply?

The biggest challenge of this value is getting the balance right – the balance between how much we collaborate with our customers, and how much we write down and agree.

Enough collaboration but not too much or too little While collaboration with customers is a desirable thing, it is possible to have too much collaboration. Constantly asking your customer’s opinion can leave them feeling that you aren’t listening to them, or that you are asking them to do all your thinking for you.

A challenge of too much collaboration is knowing when to stop. Particularly where there are differing opinions, teams still need to reach some consensus. Constantly revisiting the same topics without change or new information to justify it is damaging.

Everyone’s opinion is important and should be listened to, but not everyone needs to agree with every decision. We see this sometimes in organisations where a team doesn’t agree with the strategic direction of their product or its relationship with other products. In this situation, it is important to have your view heard, but then to commit to the decision even when you disagree with it. This practice is sometimes called ‘disagree and commit’.

The converse of this is when there is too little collaboration – either too infrequent or too ineffective. This can lead to the team making too many decisions themselves and making assumptions about what the customer needs. We sometimes see this as the result of customers who are too busy for effective frequent collaboration. In these cases, getting the most value from the time you do have is critical, for instance by focusing their time on making decisions, presenting options for their consideration or encouraging them to delegate or get help from colleagues.

Too many customers Another problem too much collaboration can cause is when accountability for product decisions is not clear. Agile teams have a single person who is accountable for deciding what the team should work on next, usually called the Product Owner, but they are rarely the only person with a view on the product. The level of engagement with those other stakeholders can be a struggle to get right. When there are lots of people interested in the product, we can see conflicting advice, confusion over who to listen to and assumptions on who has or has not been involved.

Sometimes, we see teams attempting to address this by creating additional governance. This can result in extra meetings or approvals processes to approve new requests, or requiring email confirmation from all stakeholders before a decision can be reached. These behaviours have the effect of creating additional process contracts and more documentation. A better approach is to focus on improving the collaboration and helping the Product Owner to be the single accountable role. Setting goals the right size can help – as big as possible without being too big for the team to deliver. Too big and additional clarification will be needed from the Product Owner; too small and creating them is a lot more work for the Product Owner.

Where is my customer? The opposite problem can also be a challenge – finding customers to collaborate with, especially when creating a new or innovative product or service. Startups have several techniques that can help, such as those discussed by Eric Ries in The Lean Startup,³⁶ Alex Osterwalder in Value Proposition Design and Business Model Generation³⁷ and Jake Knapp in Sprint.³⁸ If you don’t know who your customers are, you must go and find them. Don’t assume you know their needs better than they do. At the very least, test that assumption.

Treating ‘requirements’ as ‘required’ The Product Backlog is a critical artefact for Agile teams. It describes the work that could be done by the team to meet the product goals. However, it also looks like a list of product requirements. To stakeholders familiar with more traditional approaches, where requirements are agreed and committed to before work begins, this can be a problem – because the term implies that they are all required. They are not.

Once customers and teams think like this, it is easy to assume that the backlog won’t change and must all be delivered. As each iteration is delivered, because there are still ‘requirements’ to do, it is tempting for the team to avoid releasing the increment until they are all complete. As the customer knows all the things they ‘require’ are not yet delivered, they are also not keen to have a version of the product to use – after all, it isn’t finished. The effect of this is that the Product Backlog becomes regarded and treated as if it were a contract, which inhibits collaboration and inhibits change.

We frequently observe teams where their backlog is regarded as fixed – a contract – and there is even a form of change management process required to change it. This is common when the individuals in the Product Owner role have previously been senior users or principal users in traditional, linear approaches. There, their primary role was to decide what is in scope or out of scope. Once in scope it shall be delivered.

This is a completely opposing viewpoint from the Agile view, which assumes that everything not currently being delivered in an iteration is liable to change. Although the roles are similar – they are both focused on the users of the product – it is important to understand their differences and for the Product Owner to be well trained and supported.

To avoid this pitfall, consider the Product Backlog to be a list of options or ideas, not commitments. Refer to things the product may need to do as items on the Product Backlog and treat them as things that evolve over time, with some collaboration required each time. Do not refer to them as ‘requirements’. This aspect is described in more detail in Chapter 10.

Seniority is valued over role Collaboration between the team and their customers requires respect for each other’s opinions and equality. This can be a challenge in hierarchical organisations; it can be an issue when the Product Owner is relatively junior and has more senior stakeholders who think they can control the team or the backlog directly. This is not only a challenge for effective collaboration, but can also introduce implied or actual contracts that stakeholders expect to hold the team to account over.

A variant of this challenge is when the technical views of the team are disregarded by the Product Owner, perhaps favouring new feature development instead of infrastructure or architectural improvements. There is often a tension between technical excellence and rapid feature delivery. The Product Owner’s job is to balance these, which means strong collaboration to help understand the value in the work and the impact or risk of deferring it. Deferring technical debt can lead to later increments not being deployable because they start failing quality thresholds.

Anti-patterns

Table 4.3 shows some of the anti-patterns and problems we have observed when trying to value customer collaboration or avoid over-valuing contract negotiation.

Responding to change over following a plan

Of all the values, this is probably the one people remember the quickest and relate most to Agile (see Figure 4.7). This is probably a good thing as it is the one that speaks most directly to the reasons why Agile emerged. It describes a direct outcome of Agile approaches – the ability to respond to change – that is more tangible than the other values.

It is also the most challenging and controversial when comparing Agile with more traditional delivery approaches – it downplays the significance of having a plan. This leads to two major responses, depending on your perspective:

• that planning is not important for Agile teams and they don’t need to create plans;
• that the Manifesto says we still value the things on the right, so Agile teams still need to have a plan.

Both perspectives are right; and both are wrong.

In a VUCA world we know that plans, especially detailed plans, are usually wrong. But the activity of planning, especially collaboratively, can be hugely useful.

All plans are lies – they almost never describe what will happen

The single thing that causes most projects to fail is that something happened that the project manager and the team didn’t expect. In fact, it is usually lots of things. This means that it is not just difficult to come up with an accurate plan for a complex delivery, it’s impossible. In a meeting with Scott Ambler, co-developer of Disciplined Agile,³⁹ he said something to us that describes this situation well:

When somebody asks you for a plan, they are asking you to lie to them. You will be lying about the scope, the schedule or the cost. Or all three. You won’t know what you are lying about at the time, but you will definitely be lying to them.

We know this is true, otherwise we wouldn’t need change management processes and expect to use them frequently. We know that we are bad at predicting the future and this includes our ability to predict what features or requirements we will need.

The half-life of requirements is the amount of time it takes 50 per cent of your requirements to become obsolete – wrong, superfluous or missing. In an article from May 2011,⁴⁰ Susan Atkinson and Gabrielle Benefield explain that requirements’ half-life is reducing drastically: from five years or more in the 1980s, to two to three years in 2000, to just six months in 2011. Given that we know the pace of change has continued to increase, we can expect it to be even shorter today. As Figure 4.8 shows, for projects longer than a few weeks the impact of this is stark.

The impact of this is that we need to assume that unless we spoke to the customers today, there is a growing probability that what they told us is no longer correct. These are precisely the opposite conditions we need for accurate planning. Without input data we can trust, we cannot plan. Traditionally we try to overcome this by requiring ‘sign-off’ to validate that the requirements are right and transfer responsibility (or blame) to the customer if they are not, but, arguably, that is an unreasonable thing to ask of our customers.

We can make accurate plans – when they are short term or high level

If we can’t predict the future well enough to be able to produce a plan we can trust, what can we do? There are two elements to the solution:

• plan in detail only for a timescale we can be confident in;
• plan longer term by increasing the levels of abstraction until we can trust them.

In Agile teams we set our iterations to be small enough so that we can commit to work that we know will be valuable and highly unlikely to change – usually one to four weeks. Within this timeframe, we fix the work and don’t accept change. Because the time is short, the requirements’ half-life problem doesn’t arise, and we can be quite sure that our customers need the things they are asking for; especially when they will be using the solution once the iteration completes.

Most products are not complete in a few weeks – some will take months or years to be complete – so we need a way to describe the purpose of the product without trying to describe the detailed features or capabilities. We can be confident that things such as a product vision or a high-level roadmap won’t change, and if they do, there are probably bigger problems.

Our roadmaps contain the goals and user needs that we would like the product to deliver against, but don’t need to be very detailed. If keeping the goals high level works for the team, then that’s what we do. Otherwise, we break our big goals into smaller goals; we don’t break big goals down into big plans. This is explained in more detail in Chapter 10.

The level of abstraction should be just enough to be able to describe the product, and give confidence to stakeholders or investors that the product should be developed, but not so detailed that the details are likely to turn out to be wrong. As the product develops, planning for each iteration uncovers a little more detail and, because it is done just before implementation, we can be confident it is correct. This multi-level planning is shown in Figure 4.9 and described in more detail in Chapter 10.

Deciding the detail incrementally just before we implement it means that it is easy to respond to change. This same principle applies with scaled approaches – agree the detail as late as possible, keeping as many options as possible. For instance, roadmaps don’t need to position features against dates – Now, Next and Later are just as useful, but far more likely to be true, more honest. In this way, we can have a high-level plan, be able to respond to change and reduce the lies we are asked to tell our stakeholders.

Incrementally develop everything

Developing our solutions incrementally doesn’t just apply to the finished product, it also applies to its design, architecture, interfaces, marketing, testing – in fact everything. Yet this isn’t the natural way to develop some of these. Take architecture, for example. Traditionally, we understand our requirements and constraints in depth and then develop an architecture and design that takes them all into account and provides confidence that our product will be able to meet them all.

That won’t work with Agile products, and attempting to do so introduces more risks than it mitigates. It risks locking in assumptions made early in the product development and inhibiting the ability of the product to respond to the changes in requirements that we expect will emerge in time. Instead, we apply the same incremental approach to architecture and design as we do to product goals and features – we do just enough architecture necessary to allow an increment of the product to deliver value to its users. Each iteration of the product is an opportunity to revisit those architecture and design decisions, each time armed with more information, and evolve it for the next iteration.

Repeatedly revisiting (or ‘refactoring’) the architecture and design each iteration isn’t as wasteful as it may sound. By doing just enough design for each iteration we don’t include things ‘just in case’ or create complex structures because ‘we will need to use them later’. We keep things simple and only add complexity when we know for certain it is required. This increases quality by making the product simpler and less likely to break, and each time we revisit that part of our solution, we have another opportunity to spot any errors and correct them.

In this way we apply the ‘responding to change’ value to our architecture and all other elements of the product just as we do to the user needs.

What makes this value hard to apply?

By far the biggest challenge with this value is managing the expectations of stakeholders who think they need a greater level of planning detail and accuracy.

People like knowing what to expect Our brain craves certainty. As David Rock explains in his paper ‘SCARF: a brain based model for collaborating with and influencing others’,⁴¹ there are five domains of human social experience that are capable of inducing the same stress response we evolved to respond to life-or-death situations. One of those is ‘certainty’.⁴² When we are faced with uncertainty, our brain invokes the fight or flight response. Conversely, having a feeling of certainty or familiarity can trigger the reward response and cause us to feel good.

The lack of such certainty with Agile teams can be profoundly unsettling, especially when people are used to the perceived certainty of detailed plans. Many stakeholders of Agile teams believe they have experienced success with traditional delivery approaches that are very plan driven. Both the lack of certainty with Agile teams and the stark contrast with their prior experiences induce stress. Worse, because this stress originates in our primitive, limbic system in the brain, it overpowers our logical brain, making it harder to overcome.

This assumption that an accurate plan exists for this problem – we just need to find it – is flawed. As we explained earlier, an unfortunate artefact of this volatile, uncertain, complex and ambiguous world is that hindsight tends to remove the complexity. In other words, the problems look simpler once we have finished. We can look back at what happened, rationalise and justify it, and decide what we could have done better.

This gives a false sense of confidence about future work – and a false assumption that we just need to learn from previous deliveries, analyse more thoroughly and we will be able to come up with a plan that will definitely succeed. The problem is that this ignores the fact that at the point of planning and making decisions, the information we had at the time was very different from the information we have months later.

It is like the requirements’ half-life discussion. When we are making decisions in a VUCA world, the information we have is a mix of correct, incorrect, misleading and missing data. With hindsight, we know which category every data point fits into – but at the time we don’t. This is the fundamental problem with relying on plans, and why Agile approaches work better for complex problems.

External pressure to produce detailed plans Sometimes external artefacts require more detailed plans than we would like. Business case templates often expect plans, milestones and detailed lists of features. Contracts with suppliers may wish to detail the ‘outcomes’ in terms of features or seek to specify the work at a level of detail that we can’t provide without up-front analysis and planning. Ideally, we can recast these requests into things that we can be more certain of, perhaps higher-level features, or we can use outcomes that we can be confident are unlikely to change.

The Endowment Effect causes us to resist change In psychology and behavioural economics, the ‘Endowment Effect’⁴³ describes the observation that people demand more to give up an object than they would be willing to pay to acquire it. In other words, we think that things we already have are more valuable than they are. We see this with systems that won’t be decommissioned, yet if they didn’t exist we would never prioritise creating them. We also see it with items on backlogs, roadmaps or project plans. Once we have put something on a plan, we attach more value to it than it deserves.

This leads to prioritising the completion of existing work over adding new stories, and a reluctance to remove items from a backlog. The easiest way to avoid the Endowment Effect is to have as little as possible on your backlog and to avoid describing backlog items as ‘requirements’.

Sadlines and deadlines A common challenge of plans is that they often contain deadlines that the work is expected to be organised around. People who like plans love deadlines. They create urgency, importance and certainty.

However, not all deadlines are equal. Agile consultant Liz Keogh⁴⁴ takes the word deadline literally: if nothing dies as a result of missing the date, then it’s not a deadline. Often, all that will happen is somebody will become a little bit sad. So, she coined the word ‘sadline’.⁴⁵ Genuine deadlines are important to know about and factor into our work, but we should first be confident they aren’t really a sadline.

Anti-patterns

Table 4.4 shows some of the anti-patterns and problems we have observed when trying to value responding to change or avoid over-valuing following a plan.

SUMMARY

The four value statements of the Manifesto for Agile Software Development are short and concise. Yet, beneath their surface lies complexity, nuance, risk and temptation that are easy to overlook. Whenever we see Agile teams run into trouble or hear claims that ‘Agile isn’t working’, we invariably also see the Manifesto values not being adhered to – we see the items on the right valued highly, and the items on the left being neglected or downplayed.

When we investigate the values in more depth, we can see why this is the case. Our organisations and our default behaviours have been conditioned through history to value the items on the right and find those on the left unsettling. Understanding this is the first step to overcoming it.

Agile values people and collaboration more highly than repeatable processes, consistency and certainty. Rather than fight against change through detailed analysis, planning and sign-offs, we expect and embrace change, aiming to do just enough to deliver some value and get feedback on what we can change – because we know it is highly unlikely we will have got everything right this time around.

This inevitably leads to higher levels of uncertainty than we are used to from traditional project approaches, and this is uncomfortable for many people. We can overcome this by seeking certainty in other ways: planning at higher levels of abstraction, using short iterations focused on small goals, expecting to refactor designs and architectures frequently, bringing our customers on the journey with us and measuring progress by seeing customers get value from using our product throughout its development.

Understanding this is the first step to being Agile, and is a strong foundation for successful application of any Agile approach.

31 Deming, W. E. (1994) The New Economics, Massachusetts Institute of Technology, Cambridge, MA.

32 The ‘Public Sector Bodies (Websites and Mobile Applications) (No. 2) Accessibility Regulations 2018’ places a legal obligation on all public sector websites and mobile applications to be accessible and include an accessibility statement. https://www.legislation.gov.uk/uksi/2018/852/contents/made.

33 Such as the Web Content Accessibility Guidelines (known as WCAG 2.1). https://www.w3.org/TR/WCAG21/.

34 The 7 per cent statistic also has its critics: Yaffe, P. (2011) The 7% rule: fact, fiction or misunderstanding. Ubiquity (October), 1–5. https://ubiquity.acm.org/article.cfm?id=2043156.

35 Mehrabian, A. (1981) Silent Messages: Implicit Communication of Emotions and Attitudes. Wadsworth, Belmont, CA.

36 Ries, E. (2011) The Lean Startup: How Constant Innovation Creates Radically Successful Businesses. Portfolio Penguin, London and New York.

37 Osterwalder, A., Pigneur, Y., Bernarda, G., Smioth, A. and Papadakos, T. (2014) Value Proposition Design: How to Create Products and Services Customers Want. John Wiley & Sons, Hoboken, NJ; Osterwalder, A. and Pigneur, Y. (2010) Business Model Generation: A Handbook for Visionaries, Game Changers and Challengers. John Wiley & Sons, Hoboken, NJ.

38 Knapp, J., Zeratsky, J. and Kowitz, B. (2016) Sprint: How to Solve Big Problems and Test New Ideas in Just 5 Days. Bantam Press, London and New York.

39 Disciplined Agile is the framework of the Project Management Institute and is described in Chapter 11.

40 Atkinson, S. and Benefield, G. (2011) The curse of the change control mechanism. InfoQ. https://www.infoq.com/articles/the-curse-of-
the-change-control-mechanism/.

41 Rock, D. (2008) SCARF: A brain-based model for collaborating with and influencing others. NeuroLeadership, 1, 44–57.

42 The other domains in the SCARF Model are described in Chapter 9.

43 Thaler, R. (1980) Toward a positive theory of consumer choice. J. Economic Behavior Organiz., 1, 39–60.

44 https://lizkeogh.com.

45 https://twitter.com/lunivore/status/
301007423106322434.