Chapter 1. Feature Management
Feature management solutions enable businesses to dynamically control the availability of application features to end users. In this chapter, I introduce feature flags, a fundamental technology in feature management solutions. I then discuss the various feature management use cases, including progressive delivery, infrastructure migrations, experimentation, and testing.
What Is a Feature Flag?
In simple terms, a feature flag is a decision point in your code that can change the behavior of your application. Feature flags—also known as feature toggles—have long been used to trigger “hidden” code or behaviors without having to ship new versions of your software.
A feature flag is like a powerful “if” statement:
if(enableFeature(one.click.checkout, {...}))
then
/*show the one-click checkout feature */
else
/* show the old feature/
In the early days of software development, this might have been a command-line argument, an undocumented variable in a configuration file, or a hidden value in a registry. The value of each feature flag was typically set early in the life cycle, at compile-time, deploy-time, or runtime. After the value of the flag was set, the software would run that way until it was recompiled, redeployed, or restarted. Modern feature management is different.
There are two parts to the seemingly simple enableFeature call highlighted in the preceding example that make it special. The second parameter to the function (elided for simplicity in our code snippet) is a context. It typically identifies a specific user, but it can contain any arbitrary data. This is important because it makes the behavior of the function context sensitive. The code path taken can change based on the context provided; for example, the user’s identity, the plan they’ve paid for, or any other data.
Second, the enableFeature function calls out to the feature management platform, which takes the context and a set of rules defined in the feature management system to determine which code path is executed. This makes the behavior of the function dynamic—the code path can change whenever the rollout rules are changed, without redeploying a new version of the software.
Feature flags can either be temporary or permanent. Temporary flags are often used to safely deploy changes to your application or to test new behaviors against old ones. After a new behavior is being used by 100% of your users, the flag is intended to be removed. Permanent flags give you a way to control the behavior of your application at any time. You might use a permanent flag to create a kill-switch or to reveal functionality only to specific users.
This simple function is the source of enormous benefit for your team. After you begin to build feature management into your application, you’ll develop new superpowers to ship when you’re ready, test in production, stay in control, experiment with everything, and customize on the fly.
Ship When You’re Ready
With the rise of continuous delivery, software teams ship new code to production all the time. But the rapid pace of change brings new kinds of risk. In the old days, there might have been months of testing and validation for each new deployment. But for a team practicing continuous delivery, new code goes from the development team, through automated tests, and then directly to users in minutes.
Deploy != Release
Software developers attempt to mitigate this risk in two ways. First, when deploying frequently, each change is less risky because it is smaller and more isolated. Secondly, feature flags can be used to control which users can see each change. This decouples the act of deploying from the act of releasing.
When deployment is decoupled from release, feature flags give you the opportunity to create a release strategy. Your team—developers, product managers, DevOps, and marketers—can decide together, at the beginning of the project, how a new change should actually roll out to customers:
How will the feature be released?
Should anyone see it now?
Who will be getting this feature first?
Who will beta test it?
Will it be rolled out progressively?
Do I need to compare it against the old behavior?
Do I need to hit a particular date for an external event?
What do I do if something doesn’t go right?
Feature flags allow your team to create an agreed-upon process for deployment with success criteria that you can define at each stage. The flags give you the ability to move forward, or backward, based on the success at each critical decision point.
Teams using feature management are no longer constrained by users seeing new code the moment it hits production. Developers can merge their work, even in a partially finished state, deploy it to production, and be confident that no users will experience it until they turn on the feature flag. This also means that you can do true continuous delivery, where new code is deployed as it is written. Without some way to decouple deployment from release, teams are stuck in a maze of long-running feature branches.
With feature management, not only are developers able to fully adopt continuous delivery, but the business can now take advantage of progressive delivery—the ability to strategically control when individual users or user segments gain access to new features. Developers can build at the pace of innovation, operations can deliver continuously, product teams can iterate quickly, and marketing can safely release features to the right users at the right time.
Test in Production
Test in production started out as a joke in the days of Waterfall development because software development schedules almost always slipped. Testing, which always came at the end of the life cycle, was often compressed or skipped entirely. To meet the deadline, teams often sarcastically joked that testing could happen after the feature was released in production.
It sounds risky to test in production; you don’t want to expose your users to untested software. That’s why software teams have built elaborate patterns for testing and why most developers assume that deployment must always involve difficult, multistage deployments to testing and staging servers. But there are surprising benefits to skipping the staging servers and testing in production deliberately and safely.
True Test Environments Are Difficult to Create
The software industry has moved toward distributed microservices and an ever-expanding roster of external service providers. The volume of messages, transactions, traffic, and data has increased significantly, to the point that it is often impossible to replicate in another environment.
Unlike 10 years ago, your software can’t be tested realistically on a developer’s laptop. And even your testing and staging environments are unlikely to be true representations of your service.
Real users are unpredictable and so is their data. Your test data, even at high volume, will never capture every edge case. And the closer you come to production-level traffic volumes, the more expensive testing becomes, especially for projects that rely on external vendors like Amazon Web Services.
So, kill your staging server! Test in production.
Test in Production
Testing with real, live users on your production environment will always yield better data than any test or staging system. By testing in production, you can gain a much more accurate understanding of your system’s behavior.
The question is how to do this testing safely. With a feature management platform, teams use powerful targeting rules to control access to new features and can instantly turn access to any feature off, bringing back the old behavior instantly, without the need to rollback code or redeploy.
Canary releases, ring deployments, and percentage rollouts are different patterns for safely testing in production by progressively increasing the exposure of any new change. Instead of cutting over completely, as in a blue/green deployment,1 feature management allows you to roll out a change gradually to an increasing population, monitoring as you go.
Percentage Deployments
In a percentage-based rollout, small numbers of users are selected randomly to experience the new feature. That percentage is then increased over time until eventually everyone has access to the new feature. By starting small and gradually increasing the number of users, you have the opportunity to observe the behavior of the system under new conditions, and advance only if the signs are healthy and user feedback is positive.
Percentage-based rollouts are useful when there is little variation in your targeted user base or you are more concerned with operational impact of your change.
Companies often automate percentage rollouts to gradually increase unless there is an alert or monitoring alarm.
Ring Deployments
Ring deployments, a term coined by Microsoft, is another method to gradually expose features to different groups of users. The difference is that the groups are selected specifically to manage the risk of deployment. Microsoft deployments often start with small groups of low-risk users and expand through larger, higher-risk populations. This helps identify problems early while limiting the “blast radius” of disruption if something goes wrong.
A typical ring deployment begins by releasing first to internal users. After you have verified that the change has been successful for that set of users, you can expose the next set of users: the canary group.
Teams use canaries to measure the reaction from real users in production and look for early indicators of danger or success (similar to how actual canaries were used in coal mines many years ago to test for toxic gas). Even testing with a small number of users can increase your confidence dramatically. If a feature generates an unexpected or negative reaction, you can pause the deployment to address problems or even turn the feature off entirely until you are ready to try again.
Then, you can move on to the third ring: beta testers or early adopters. These are users who are interested in having early access to new features and are more prepared for problems or rough edges.
After you’re satisfied with your beta test, you can move on to a general release. But even this you can do in stages. You might start with users of your free product and then move on to paying customers. Or, you might start with small geographies and move on to larger ones. Or, you might start with accounts with small numbers of users or load, and expand to larger-load customers.
Not every ring deployment needs to follow this exact pattern. But the defining characteristic of a ring deployment is using feature management targeting rules to release strategically to users in order to identify problems in the smallest, lowest-risk population.
Stay in Control
Creating and deploying new software is risky. Bugs can be introduced accidentally, the software can be delivered badly or to the wrong people, or it can interact in unexpected and unfortunate ways with existing software or hardware. It’s important to have safety nets in place when things don’t go as planned. Feature management provides the necessary control to reduce, and often eliminate, the risks associated with deploying new software.
Flag Early and Often
I’ve found that many teams have a too-narrow definition of what they can feature flag. The misconception comes from thinking that a “feature” refers only to customer-visible features. In fact, there are many benefits to feature flagging every significant change, even those that are not visible to the customer, such as new backend improvements or infrastructure changes. As a complement to customer-visible feature flags, these “operational feature flags” give DevOps teams powerful controls that they can use to improve availability and mitigate risk.
Safety Valves and Circuit Breakers
A well-wrapped feature means that you can quickly turn it off if it is performing poorly, which can mean the difference between a public relations disaster and a minor hiccup with minimal impact. Enter the safety valve.
Safety valves are permanent feature flags that you can use to quickly disable or limit nonessential parts of your application to keep the rest of the application healthy. On an ecommerce site, for example, increased page load time can dramatically reduce conversion rates. If nonessential features like product recommendations or reviews are found to increase latency, a DevOps team can use a safety valve to quickly disable the feature and reduce latency, all without requiring any change in code.
Safety valves are powerful tools that you can design into a system from the outset, or introduce later, after a need has been identified. It’s important to properly document these flags. Because they’re permanent, you must note them as such so that they’re not accidentally removed. You also should add them to service runbooks, with a clear description of the symptoms that should trigger the use of the flag as well as the impact to end users. Going a step further, if a safety valve disables user-visible functionality, you should bake it into incident management processes so that status page updates, emails to impacted customers, and other relevant communications are triggered.
Simple safety valves are first managed manually, but the next logical step is to automate them, triggering changes to the flag’s rollout rules based on error monitoring or other metrics. This kind of automation is powerful, but you must use it with caution—a false positive caused by faulty error monitoring could cause unnecessary customer impact.
Atlassian uses feature flags heavily in its development processes. With a combination of release flags and safety valves, it is able to recover from 90% of its incidents within seconds. Only a small fraction of incidents require redeploys or code reverts.
Infrastructure Migrations
Infrastructure migrations, like database schema changes, are some of the riskiest changes in software systems. They often involve complex interactions between different services and backing stores. They can be difficult to roll back, and in modern applications, they must usually be completed with no downtime.
With some advanced planning, you can use feature flags to remove the risk from most infrastructure migrations. The key idea is to decompose the problem into multiple steps, with each step controlled by a feature flag that you can roll back. At each step, you use correctness checks to validate that it’s safe to proceed to the next step. This approach works well for database migrations, but you also can apply it to other infrastructure changes (e.g., changing cloud service providers, vertically scaling a database, or swapping the implementation of a specific algorithm).
Experiment with Everything
Feature flags can serve different features or different feature versions to different people, based on rules that you create. This is a powerful new tool that allows you to experiment with different behavior in your applications. You can perform extremely detailed and granular targeting of your users based on their self-reported characteristics, historical usage patterns, or any attribute you specify.
By defining a goal and comparing which behavior in your application leads to more successful completions of that goal, you can validate your hypotheses about product improvements and protect yourself against unexpected changes.
Monitoring/Baselines
Every change that you make to your software is a kind of experiment, even if you don’t explicitly design them as such. When you release a new change using a feature flag, you have a built-in capability to measure the impact of the change.
You begin by establishing important baselines for your product, such as the number of signups, dollars in sales, or searches performed. Then, as you progressively roll out, you can compare user activity in the old cohort with the behavior in the new one to ensure you are affecting your baseline metrics positively. Your application is a complex system, and it isn’t always easy to predict the way users will react to changes, even changes that might seem unrelated. Measuring your new features against existing baselines helps to reduce the risk of change.
A/B Testing
A/B testing, or experimentation, is a commonly used method of validating new ideas. This kind of experiment tests one thing against another—variation A versus variation B, or new versus old. Multivariate feature flags allow you to serve multiple variations of a feature to different user segments: testing variation A versus B versus C, or more. Most people use experimentation to test UI features, such as placement of buttons, design layout, or word choice. But it’s equally useful for testing backend functionality, such as evaluating the effect of different content delivery networks (CDNs) or different site search providers.
User Feedback
The ability to release changes to a limited set of users makes it much easier to gather feedback about your product. You can create a beta group of users and target feature flags specifically to that group who have offered to provide you feedback. Testing new features with a subset of users allows developers to find and address bugs as well as glean valuable feedback about the features they’ve built.
As your team continuously delivers new features of your application, feedback is continuously flowing back to you. You have the opportunity to harvest this feedback and develop your product to meet the needs of real users.
Customize on the Fly
Customizing how users experience your application can be a powerful way to improve engagement and increase conversions, sales, and other business objectives. Historically, customization has required significant investments in developer resources. With feature management, nontechnical teams can now manage customization initiatives without requiring direct support from developer resources.
Entitlements and Plan Management
There are many kinds of features that are better controlled with a permanent flag. Often, features in your code might need to change because the status of a user or organization changes; for example, when a user upgrades from a free to a premium service or when a specific organization needs a custom feature. In many cases, these changes can and should be managed by sales and support, not the development organization.
Dynamic Configuration Management
Configuration parameters might need to be changed over time. Adding feature flags to those settings allows you to change the way your software operates without having to update or redeploy code. This results in much quicker response times to changing conditions.
Summary
There are a many uses for feature management that benefit different departments in an organization. Your feature management solution will empower your software development, technical operations, and business teams to deliver value to customers faster, with less risk. But how do you bring feature management to your team? In the next chapter, you’ll find best practices for adopting feature management across your organization.
1 Blue/green deployment is a technique that includes running two identical production environments. Only one of the environments serves production traffic while the other environment is updated with the new software. When ready, the updated environment takes on all production traffic, and the original environment is available for rollback if necessary.