Your First Feature Flag

A kill switch is a technique for quickly turning off a feature or routing users to a previous version of a feature. Kill switches are the simplest use case for feature flags and make for a great first flag in your software. I’ve found that teams derive immediate value by introducing kill switches around a new feature. It’s also best to start with a small feature; your first flag shouldn’t be part of a large, multimonth delivery effort. Find quick wins to begin building the muscle memory in your team for flagging.

It’s also important to follow these first flags through the entire life cycle, including removing them after they’re no longer needed. After this process has been completed successfully a few times, you can begin introducing feature flags into your everyday workflow.

Feature Flags in Your Workflow

Feature management is valuable to many teams in an organization, not just the development team. Depending on the need, features might be managed by product managers, marketing, support, or DevOps teams. When you’re starting a new project, you should include feature management in the planning process. In the kickoff meeting, it’s useful to ask the following questions to help define your flagging strategy:

• What’s the release strategy for this feature? How risky is it? Are there segments of users who should see the feature first?

  Consider whether there should be a beta, canary release, or a ring deployment for this feature.

• Are you gating access to the feature based on pricing plans or entitlements?

  Consider adding a permanent flag and specifying rules to determine who should have access to the feature.

• Are there any circuit breakers or operational controls that you should flag? What will you do if something goes wrong?

  Consider including operational flags to adjust the configuration of your new feature.

• Is this feature nonessential and possibly resource intensive?

  Consider adding a kill switch.

• What does success mean for this feature? Are there any experiments that you should perform, baselines you should measure, or metrics to track?

  Consider introducing a flag that compares new behavior against the current behavior of a control group.

Based on your answers, you can define the flags that the development team should introduce. You’ll know which flags are intended to be temporary, exactly how long they should live, and whether any permanent flags will be introduced. It’s much easier to do this early, before you’ve begun to build your new feature.

Flags and Branching Strategies

You can use feature flags with any branching strategy. They are an important step in moving toward trunk-based development because the ability to flag code and keep it hidden allows you to merge aggressively, even before the branched code is ready for release.

Feature flags also complement feature-branch–based development and DVCS. Teams that incorporate flags in conjunction with feature branches can eliminate the need for long-lived branches, along with the headaches associated with integrating long-lived branches back into master.

Should I Test All Combinations of Flags?

It isn’t necessary (or even possible) to test every combination of feature flags. Because each feature flag has at least two variations, you’d have an exponential number of test cases to write. Testing each variation of a flag in isolation (using default values for the other flags) is usually enough, unless there’s some known interaction between certain flags. But you should identify combinations of feature flags that are known to create user issues in test cases and guard against using them in your code.

Flags and Libraries

Another decision that affects testing is whether you should use feature flags in reusable library code. I think the answer is no—flags are an application-level concern, not a library concern. Let’s illustrate this with a small example. Imagine you have a search library that is backed by Elasticsearch, with an API like the following:

function searchES(query, lastId) {
  ...
}

Imagine that you’re feature-flagging between two different search systems, using a feature flag called use.algolia. You could make the feature flag the library’s concern, which would require us to pass the user context to the library, like so:

function flaggedSearch(query, lastId, user) {
   if (enableFeature("use.algolia", user)) {
     return searchAlgolia(query, lastId);
   }
   else {
     return searchES(query, lastId);
   }
}

This approach has a few problems. First, it introduces into the library a dependency on the feature management system, making it potentially more difficult to reuse and test. Second, it ties the application’s data model (the user context) into the library. The user context isn’t relevant to search; it’s relevant only to the feature flag. A better way to structure this is to move the feature flag to the uses of the search function and add a parameter to the search function indicating which engine to use:

enum SearchEngine {
  ALGOLIA,
  ES
}
function search(query, lastId, searchEngine) {
   if (searchEngine == ALGOLIA) {
     return searchAlgolia(query, lastId);
   }
   else (searchEngine == ES) {
     return searchES(query, lastId);
   }
}

Now, feature flagging is an application-level concern. You check the flag at the call sites to the search function:

  if (enableFeature("use.algolia", user)) {
     engine = ALGOLIA   
   }
   else {
     engine = ES
   }
   results = search(query, lastId, engine);

Using this method, you’ve separated the flagging concern from search and avoided introducing dependencies on the feature management system and the application-level user object, which simplifies testing considerably.

Unit Testing

Even after structuring your code to keep feature flags in the application level, you’re still going to need to unit-test in the presence of feature flags. The easiest way to write unit tests for code containing feature flags is to mock-out the feature management service. Mocking is a simple way to avoid interactions with the external feature management service, which is important for unit testing.

Integration Testing

In most cases, it’s still best to eliminate external dependencies in end-to-end integration tests. You can accomplish this either by using a mock object library or by having the feature management service return default values or hardcoded values from a file or other local source.

If external dependencies are not an issue for your tests, and you want to modify feature flags for testing purposes, a feature management service with an API is a viable approach. The API should allow you to programmatically create flags and change rollout rules. Ideally, it should allow you to also create and destroy environments so that tests can start from a clean slate with each run.

Appoint a Maintainer

The developer who introduces a flag should be responsible for cleaning it up. To enforce this, appoint a maintainer for the flag. This person is responsible for recording the purpose of the flag, ensuring that it has a well-defined rollout plan, and eventually cleaning it up.

Set Expiration Dates

When I create a flag, I usually have some idea of how long it’s intended to remain in the code. It’s common to have temporary flags that can’t be removed quickly; for example, when running a weeks-long A/B test or creating a flag that hides a months-long work in progress. These flags are the most likely to slip through the cleanup cracks. A strategy here is to give feature flags expiration dates at the time of creation. When a flag reaches its expiration date, you can be notified that it’s time to delete that flag. Some teams file issues to track this, some teams use tags or naming conventions, and some teams even create pull requests to remove the feature flags in advance.

Flag Removal Branches and Pull Requests

One of the biggest barriers to deleting temporary flags is remembering what the flag did, where it needs to be removed, and what dead code can be deleted as a result. Too many stale flags are a form of technical debt and an antipattern that you should avoid. Surprisingly, this amnesia kicks in quickly—even if a developer circles back only a week later to delete a flag, it’s time-consuming to remember the original context and locate all the places where the flag is referenced. One way to help overcome this challenge is to create a branch and pull request (PR) to remove the flag at the same time the flag is introduced. This extra branch or PR is kept unmerged until it is time to delete the flag. After the flag has served its purpose, cleanup consists of merging the PR, deploying the change, and deleting the (now-inactive) flag from the flag management dashboard.

Finding Flag References

Eventually, you’ll run into a flag that nobody knows anything about. Perhaps the maintainer has left the company or the flag itself wasn’t documented well. You might not even know which codebase (or codebases) reference the flag. In that situation, you’ll need a good strategy for finding references to the flag in code. Some best practices make this easier:

• Use a naming convention that makes feature flag strings stand out. For example, if feature flags always have the sentinel prefix fflag., a simple grep search is much more likely to identify flags without many false positives.

• Name feature flags in commit messages. When you introduce a flag, add a commit message like “Introduces feature flag flag-key.” Similarly, when you delete a flag, add a commit message like “Removes feature flag flag-key.” By doing this, you can quickly find commits that affect feature flags with a Git log.

• If you need to do a comprehensive search, use your repository hosting tool’s code search feature. You can also search Git history with a command like git rev-list --all | xargs git grep flag-key.

Permissions and Role-Based Access Controls

Feature management is a practice that can benefit your entire organization. Your feature management console is the source of truth, and it’s useful for anyone in your organization to be able see the current, accurate state of the world for your application for any given customer.

You can more tightly control just who can modify a feature flag using role-based access controls (RBACs). You can limit changing specific flags, any flag in a given project, or environment (like your production environment). You might want to limit write permissions to your DevOps team or to just the engineers who built a given feature.

That said, I would encourage you to be as open as possible in providing access to feature management throughout your team and across all disciplines from engineering to marketing to support. Feature flags are valuable as much for the collaboration they enable as the simple technical capability of enabling or disabling something, and those benefits should be shared as widely as possible.

Collaboration

After your team is using feature management, there are a handful of practices to observe that will ensure smooth collaborating:

Document changes

It’s good practice to maintain a log of flag updates. It’s even more helpful to leave a comment with every change. When something is going unexpectedly wrong, being able to quickly see if anything has changed recently (and why it did) is an invaluable asset.

Make updates visible

Likewise, it’s good practice to make sure that your team knows whenever changes are made. You can try sending notifications to Slack or a similar collaboration tool so that the team knows about flag changes in real time.

Integrate

After you begin using feature management, answering the question “is this done yet” becomes a bit more complicated. The code might be written and it might be deployed, but only a small fraction of users might be able see it. So, it’s good practice to connect your feature management platform—which is the source of truth for feature deployment—to your other collaboration tools, like issue trackers or code repositories. It’s super helpful to see the current deployed state of a flag directly from an issue, ticket, or PR.

Organizing Flags

As feature management spreads across a larger organization, one of the challenges is simply how to deal with the sheer number of flags. Here are a couple of methods that you can employ:

Separate concerns

The most important step is to divide your flags into smaller, related chunks of functionality. You might choose to divide by microservice, by application tier, or by related functionality. But separating your flags into projects or buckets can avoid the need for developers to understand every flag in the system.

Name your flags well

It’s also important to help your team understand what flags are for as easily as possible. So, adopt a naming convention that makes it clear at first glance what a flag is for, what part of the system it affects, and what it does. The more consistent you can be across teams, the smoother your collaboration will be.