Agile: A Potted Landscape

Agile (whether spelled with a small “a” or a big “A”) means different things to different people. Very few Agile teams work in the same way, partly because there is a choice of Agile methodologies, and partly because all Agile methodologies encourage you to adapt and improve the process to better suit your team and your context.

“Agile” is a catch-all term for a variety of different iterative and incremental software development methodologies. It was created as a term when a small group of thought leaders went away on a retreat to a ski lodge in Snowbird, Utah, back in 2001 to discuss issues with modern software development. Large software projects routinely ran over budget and over schedule, and even with extra time and money, most projects still failed to meet business requirements. The people at Snowbird had recognized this and were all successfully experimenting with simpler, faster, and more effective ways to deliver software. The “Agile Manifesto” is one of the few things that the 17 participants could agree on.

What is critical about the manifesto is that these are simply value statements. The signatories didn’t believe that working software was the be-all and end-all of everything, but merely that when adopting a development methodology, any part of the process had to value the first values more than the second ones.

For example, negotiating a contract is important, but only if it helps encourage customer collaboration rather than replace customer collaboration.

Behind the value statements are 12 principles which form the backbone of the majority of Agile methodologies.

These principles tell us that Agile methods are about delivering software in regular increments, embracing changes to requirements instead of trying to freeze them up front, and valuing the contributions of a team by enabling decision making within the team, among others.

So what does “Agile” development look like?

Most people who say they are doing Agile tend to be doing one of Scrum, Extreme Programming, Kanban, or Lean development—or something loosely based on one or more of these well-known methods. Teams often cherry-pick techniques or ideas from various methods (mostly Scrum, with a bit of XP, is a common recipe), and will naturally adjust how they work over time. Normally this is because of context, but also the kind of software we write changes over time, and the methods need to match.

There are a number of other Agile methods and approaches, such as SAFe or LeSS or DAD for larger projects, Cynefin, RUP, Crystal, and DSDM. But looking closer at some of the most popular approaches can help us to understand how to differentiate Agile methodologies and to see what consistencies there actually are.

Scrum, the Most Popular of Agile Methodologies

Scrum is, at the time of writing, by far the most popular Agile methodology, with many Certified Scrum Masters, and training courses continually graduating certified Scrum practitioners and trainers. Scrum is conceptually simple and can integrate into many existing project and program management frameworks. This makes it very popular among managers and senior executives, as they feel they can understand more easily what a team is doing and when the team will be finished doing it.

Scrum projects are delivered by small, multidisciplinary product development teams (generally between 5 and 11 people in total) that work off of a shared requirements backlog. The team usually contains developers, testers, and designers, a product manager or Product Owner, and someone playing the Scrum Master role, a servant leader and coach for the team.

Extreme Programming

Extreme Programming (XP) is one of the earliest Agile methodologies, and is one of the most Agile, but it tends to look the most different from traditional software development.

These days, teams using XP are comparatively rare, since it’s incredibly disciplined and intense; but because many of the technical practices are in active use by other Agile teams, it’s worth understanding where they come from.

The following are the core concepts of Extreme Programming:

• The team has the customer accessible to it at all times.

• It commits to deliver working code in regular, small increments.

• The team follows specific technical practices, using test-driven development, pair programming, refactoring, and continuous integration to build high-quality software.

• The whole team shares the work to be done through collective code ownership, common coding standards, and a shared design metaphor.

• People work at a sustainable pace to prevent team members from burning out.

Kanban

Kanban is very different from Scrum or XP in that it is not a methodology for building a software product, but is instead a method for running a high-functioning team. Kanban comes from W. Edwards Deming’s work with Toyota and the Toyota Production System, which revolutionized how work would move around a factory floor.

Most of the ongoing work on a manufacturing floor is done at individual stations, and in most manufacturing spaces, each station has an in-queue and an out-queue. Work is processed from the in-queue and results placed into the out-queue.

Deming noted that work was often pushed through the system, so orders for specific parts of systems would be pushed in at the front and make their way through the entire system. Critically, he identified that partially complete work spends the majority of its time in one of the queues at a station or being transferred from one station to another (or even worse, put into storage).

Instead, he proposed a system based on a just-in-time approach to each station, enabling each station to request (or pull) work from the preceding station when it was ready for the next piece of work.

This means that work in progress at any given station is highly limited, which creates a more optimal flow through the system. Deming’s work was primarily about identifying waste in the system, where work was partially done and turned out not to be needed, or done prematurely.

Kanban systems prioritize flow, or cycle time, which is a measure of how fast a piece of work can travel from the beginning to the end of the system, and the number of touch points it has as it does so.

In IT processes there may be potentially dozens or even hundreds of touch points between an idea being conceived and getting it into production. Many of these touch points have queues, where work waits until the process can be achieved, each of which creates delays in flow.

A common example is a change control board for reviewing and approving changes to production systems. Because it is inefficient for such a board to meet each day, they often meet only weekly or monthly. Work that needs to be done must sit at the proposal stage waiting for the change control board to approve or process the request.

Because of the number of changes that are queued up for the change control board, sometimes they cannot review each change and have to postpone changes until the next board, which can cause even greater delays. Once a change is approved, it then moves to the implementation team, who may receive many such changes, which means they get queued there, and so on.

Kanban systems are based on three key practices, detailed in the following sections.

Lean

Lean development, and more recently Eric Ries’s Lean startup model, are based, again, on Lean manufacturing, which came out of Kanban and the Toyota Production System. However, Lean development has gone a slightly different direction over time.

One of the key differences in Lean is the emphasis on analyzing what you have done and learning from it in order to iterate.

Lean as a methodology prioritizes the principle cycle of build → measure → learn.

It suggests that iterative learning is the key to building a successful product, and that to iterate successfully, you need to not just build and build again, but also take the time and effort to measure the impact of each change, and then learn from measurements.

By explicitly calling out the measure and learn parts of this iteration, the focus for technical teams includes far more emphasis on building measurable systems, and including monitoring and analytics in the design.

Lean teams tend to use Hypothesis Driven Development where instead of writing work units in terms of the value it will deliver to the customer (like user stories), they state a hypothesis on how the change will affect a business value measure.

Stories are not done simply when they are coded and deployed to production, but are instead only considered done when the data has been gathered and analyzed to find out if the feature actually delivers the value hypothesized.

Secondly, Lean teams tend to encourage the use of experimental models that can be statistically compared, such as A/B testing frameworks, which allow multiple implementations of a feature to be put in front of customers for evaluation.

An example of a Lean story might be “Providing customers a buy button on every page instead of once added to the cart will encourage customers to make more purchases.” The story might be implemented by showing the buy button on every page to just 20% of customers, with the other 80% seeing the current behavior in an A/B test. Results will be gathered and compared to see if the work was worthwhile.

Lean also emphasizes early delivery of a Minimum Viable Product (MVP). The team starts by designing and delivering only the minimum feature set necessary to start gathering data on what users want and are willing to pay for, and then rapidly iterates and evolves the product based on real user feedback.

Agile Methods in General

Regardless of which Agile methodology your development team is using—and many teams are using hybrid methodologies, where they started with a well-known methodology but have adjusted it to fit their needs—we tend to find that all Agile teams value and practice the following:

Prioritizing feedback

Agile teams place a massive value on getting feedback on their products as soon as possible. This generally means increasing the speed at which software is in a demonstrable state, as well as decreasing the barriers to communication that prevent real decision makers from seeing the results of their decisions.

Agile teams also value feedback loops within their development methods, encouraging retrospectives each iteration or continuous improvement capabilities that ensure that the process itself is adaptive and delivering value.

Any mechanism to speed up feedback is the hallmark of Agile software. That can include practices such as continuous integration, rapid deployment systems, and production testing facilities.

Minimizing the pathway to production is key to getting feedback as fast as possible.

Speedy delivery of small batches

Most Agile methods have some basis in the Lean manufacturing industry, or awareness of it, and the key learning is about reducing batch size. Almost all Agile teams prefer to deliver software in small iterative chunks rather than in single large batches of features.

In some cases, delivery may be into a pre-production or staging environment rather than directly to production, but the team is able to concentrate on small numbers of features at a time, and to iterate on the feedback given on the completed work.

Agile teams measure their effectiveness in terms of velocity, the number of features delivered to production. Teams will invest time in automating their systems and tools if that will improve the team’s velocity.

Iterative development

Agile methods all aim to allow some form of iteration—the feedback gained from delivering features early is only useful if it can be acted upon quickly. Most Agile methods have facilities to reduce the impact of rework, and also give strong feedback to the decision makers as to the cost of the rework. For example, in a Scrum team, rework is added to the product backlog, and the team then needs to estimate and prioritize this work against other features.

Team ownership

Agile methods empower development teams, pushing decision making down to the team level, making the team responsible and accountable for how to do its own work, and for its own success or failure. Teams are also free—and expected—to find ways to improve their processes and practices on their own.

Many Agile teams have a coach, an experienced person who can help them to understand the practice framework and guide them through the rituals and retrospectives. Coaches are there to help the team self-direct and orient toward high-functioning capability, not force people to work a specific way.

If it hurts, do it more often

Agile tends to force us to find the pain points in development processes; and where we find difficult or painful areas, we encourage doing it more often.

While this may sound ridiculous and not at all sensible, we recognize that many actions that we find awkward or difficult are often because of unfamiliarity. Deployment is the classic case. When a team deploys its system only once every six months, we find processes that don’t match the reality, and staff members who may have done one or two deploys in their working lifetime. When teams deploy multiple times per day, we find team members who are intimately familiar with the process and the context.

A process that is done rarely tends to be manual because the cost of automating it and maintaining the automation is high compared to the number of executions. But once you start doing a process an order of magnitude more often, automation can start to reduce the pain and can earn huge dividends in terms of effort expended. Furthermore, automation increases repeatability and quality, giving further benefits to the team that uses it.

Inspect and adapt

Agile methods require teams to iterate not only on the product or service that they are building, but also on their methodology and internal processes. In order to do that, just as we monitor the effectiveness of the product or service, we need to inspect our own processes and identify the value delivered.

Teams use concepts such as value stream mapping, time logs, velocity, and retrospectives to examine and adjust the process. This continual learning culture and openness to changing the process enables teams to adapt effectively to changing contexts of an organization.

What About DevOps?

This Agile stuff is all well and good, providing it actually results in working software. As more and more development teams across the world moved to this way of working, they started to encounter problems in operations.

Agile teams are almost exclusively measured by the amount of working software delivered, the “velocity” of the development team. However, most operations teams are valued and rewarded on the basis of system stability. This can be measures of uptime or incidents.

Development and operations teams can end up almost at war with each other because of their opposing priorities.

Developers often have to make decisions that trade off time to delivery, and cost of delivery against long-term operability concerns. If developers don’t share some accountability for operations and support, this creates an externality, rewarding short-term thinking and encouraging developers to cut corners. While taking extra time to think about improving operability means that stories are delivered slower, penalizing the development team.

The DevOps movement came from the recognition of this as a problem, combined with a massive structural shift in automation of large operations platforms, such as the move to cloud computing, virtualization, and programmable automated tooling for operations teams.

Operations teams began to recognize that they were seen as a blocker to Agile development teams, and that in many cases their jobs had to change from being the doers to being the enablers in the organization.

Organizations that practice DevOps tend to organize their infrastructure and operations teams into different focus areas:

• Infrastructure teams that buy and manage infrastructure

• Tooling teams that build automated tooling for self-provisioning and management of said infrastructure

• Support teams that respond to incidents

Some companies have moved their infrastructure wholesale to large cloud providers, essentially outsourcing the infrastructure teams, and putting most of their operations work into tooling teams that enable developers to self-service in the provision, operation, and maintenance of the infrastructure.

Things like logging, monitoring, alerting, patching, and so forth tend to be problems that most organizations should only need to solve once (per infrastructure provider), and then developers can be provided with the APIs and tools to manage themselves.

Support is a much bigger problem, but some DevOps organizations moved quickly toward requiring development teams to support their own applications. This de-siloization caused development teams to in-source the operational pain that their decisions could cause, and removed the externalities, creating more robust services as a result.

Enabling development teams to maintain and operate their own services requires a level of organizational maturity that not all organizations are capable of. But organizations that succeed at this can reach incredible levels of speed and efficiency.

Many DevOps teams are releasing code to production multiple times per day, with some organizations reaching levels of hundreds or even thousands of deploys per day.

Looking back at Lean and Kanban, we can see that if the largest queue in your system is around waiting to get the code into production to get feedback, eliminating this bottleneck can be a huge business enabler.

Improving feedback and reducing time to market aren’t the only benefits of this approach. We also see that organizations that are capable of releasing more often are significantly more reliable.

Organizations that deploy infrequently tend to focus on mean time between failures (MTBF), which is a risk-averse strategy, but means that they are much less prepared to deal with failures when they do happen. Instead, the key metric that we start to look at is the mean time to recovery (MTTR) from operational failures. Organizations that release smaller changes more often can identify the cause of problems much faster and are significantly more capable of getting fixes out, which reduces their MTTR.

But the extreme velocity of change in DevOps does create major challenges for security and requires a new way of thinking about security. This is something that we will look at throughout this book, especially in the chapters on operations, risk management, compliance, and testing.

Agile and Security

Getting Agile teams to work well with security people has historically been quite difficult. Part of the problem is that most security processes and practices have been built for large Waterfall projects with requirements set up front, instead of for small teams working quickly and iteratively.

Many security professionals have a hard time adapting their existing practices to a world where requirements can change every few weeks, or where they are never written down at all. Where design and risk-management decisions are made by the team just in time, instead of being planned out and directed from top down. And where manual testing and compliance checking cannot possibly keep up with the speed of delivery.

Worse still, too many security teams work with a worldview where their goal is to inhibit change as much as possible, in an effort to minimize the change in risk profile of the application or environment: if there is no change in risk, then the security team cannot be blamed for new security issues that may arise.

Security teams that try to reduce risk by minimizing change, rather than supporting development teams to realize their ideas in a secure way, are doomed to be increasingly irrelevant in an Agile world, and will therefore be bypassed. The result will be systems that are not secure, safe, or compliant, because security becomes removed from the development process.

Despite all this, there are teams that are successfully following Agile methods to deliver secure software, and the rest of this book is going to show you techniques and tools that can work well with Agile teams and practices that we think can improve the entire process.

For this to work, everyone needs to do their part.

Agile teams need to understand and choose to adopt security practices, and take more responsibility for the security of their systems.

Product Owners for these teams need to give the teams enough time to do this properly, and they need to understand and prioritize security and compliance requirements.

Security professionals have to learn to accept change, to work faster and more iteratively, and be able to think about security risks, and how to manage risks, in incremental terms. And most important, security needs to become an enabler, instead of a blocker.

Agile and DevOps are not fads. The future of IT is going to be faster, more responsive, more collaborative, and more automated.

Security has to face these challenges and ensure that the future is not only fast, but safe, too. In the next chapter, we’ll start by looking at how and where security can be added into the Agile development life cycle.