Chapter 7. Building Environments With Stacks
In Chapter 5, I described an infrastructure stack as a collection of infrastructure resources that you manage as a single unit. An environment is also a collection of infrastructure resources. So is a stack the same thing as an environment? In this chapter, I explain that maybe it is, but maybe it isn’t.
The difference between an environment and a stack is that an environment is conceptual, while a stack is concrete. You define an environment to fulfill a particular purpose, like deploying and testing software. You define a stack with code, and then provision it, change it, and destroy it using a tool. As I’ll explain in this chapter, you might implement an environment as a single stack, or you might compose an environment from multiple stacks. You could even create several environments in one stack, although you shouldn’t.
What environments are all about
The concept of an environment is one of those things that we take for granted in IT. But we often mean slightly different things when we use the term in different contexts. In this book, an environment is a collection of operationally related infrastructure resources. That is, the resources in an environment support a particular activity, such as testing or running a system. Most often, multiple environments exist, each running an instance of the same system.
There are two typical use cases for having multiple environments running instances of the same system. One is to support a release delivery process, and the other is to run multiple production instances of the system.
Release delivery environments
The most familiar use case for multiple environments is to support a progressive software release process - sometimes called the path to production. A given build of an application is deployed to each environment in turn to support different development and testing activities until it is finally deployed to the production environment.
Figure 7-1. Foodspin delivery environments
I’ll use this set of environments throughout this chapter to illustrate patterns for defining environments as code.
Multiple production environments
You might also use multiple environments for complete, independent copies of a system in production. Reasons to do this include:
- Fault tolerance
-
If one environment fails, others can continue to provide service. Doing this could involve a failover process to shift load from the failed environment. You can also have fault tolerance within an environment, by having multiple instances of some infrastructure, as with a server cluster. Running an additional environment duplicates all of the infrastructure, creating a higher degree of fault tolerance, although at a higher cost. See [Link to Come] for more on this.
- Scalability
-
You can spread a workload across multiple environments. People often do this geographically, with a separate environment for each region. Multiple environments may be used to achieve both scalability and fault tolerance. If there is a failure in one region, the load is shifted to another region’s environment until the failure is corrected.
- Segregation
-
You may run multiple instances of an application or service for different user bases, such as different clients. Running these instances in different environments can strengthen segregation. Stronger segregation may help meet legal or compliance requirements and give greater confidence to customers.
Environments, consistency, and configuration
Since multiple environments are meant to run instances of the same system, the infrastructure in each environment should be consistent. Consistency across environments is one of the main drivers of Infrastructure as Code.
Differences between environments create the risk of inconsistent behavior. Testing software in one environment might not uncover problems that occur in another. It’s even possible that software deploys successfully in some environments but not others.
On the other hand, you typically need some specific differences between environments. Test environments may be smaller than production environments. Different people may have different privileges in different environments. Environments for different customers may have different features and characteristics. At the very least, names and IDs may be different (appserver-test, appserver-stage, appserver-prod). So you need to configure at least some aspects of your environments.
A key consideration for environments is your testing and validation strategy. When the same infrastructure code is applied to every environment, testing it in one environment tends to give confidence that it will work correctly in other environments. You don’t get this confidence if the infrastructure varies much across instances, however.
You may be able to improve confidence by testing infrastructure code using different configuration values. However, it may not be practical to test many different values. In these situations, you may need additional validation, such as post-provisioning tests or monitoring production environments. I’ll go more in-depth on testing and validation in Chapter 9.
Patterns for building environments
As I mentioned earlier, an environment is a conceptual collection of infrastructure elements, and a stack is a concrete collection of infrastructure elements. A stack project is the source code you use to create one or more stack instances. So how should you use stack projects and instances to implement environments?
I’ll describe two antipatterns and one pattern for implementing environments using infrastructure stacks. Each of these patterns describes a way to define multiple environments using infrastructure stacks. Some systems are composed of multiple stacks, as I described in “Patterns and antipatterns for structuring stacks”. I’ll explain what this looks like for multiple environments in “Building environments with multiple stacks”.
Antipattern: Multiple-Environment Stack
A Multiple-Environment Stack defines and manages the infrastructure for multiple environments as a single stack instance.
For example, if there are three environments for testing and running an application, a single stack project includes the code for all three of the environments.
Figure 7-3. A multiple-environment stack manages the infrastructure for multiple environments as a single stack instance.
Also Known As
-
One stack with many environments
-
All your environments in one basket
Motivations
Many people create this type of structure when they’re learning a new stack tool because it seems natural to add new environments into an existing project.
Consequences
When running a tool to update a stack instance, the scope of a potential change is everything in the stack. If you have a mistake or conflict in your code, everything in the instance is vulnerable1.
When your production environment is in the same stack instance as another environment, changing the other environment risks causing a production issue. A coding error, unexpected dependency, or even a bug in your tool can break production when you only meant to change a test environment.
Related patterns
You can limit the blast radius of changes by dividing environments into separate stacks. One obvious way to do this is the copy-paste environment (“Antipattern: Copy-Paste Environments”), where each environment is a separate stack project, although this is considered an antipattern.
A better approach is the reusable stack pattern (“Pattern: Reusable Stack”). A single project is used to define the generic structure for an environment and is then used to manage a separate stack instance for each environment. Although this involves using a single project, the project is only applied to one environment instance at a time. So the blast radius for changes is limited to that one environment.
Antipattern: Copy-Paste Environments
The Copy-Paste Environments antipattern uses a separate stack source code project for each infrastructure stack instance.
In our example of three environments named test, staging, and production, there is a separate infrastructure project for each of these environments. Changes are made by editing the code in one environment and then copying the changes into each of the other environments in turn.
Figure 7-4. A copy-paste environment has a separate copy of the source code project for each instance.
Also Known As
-
Singleton stack
-
Snowflake stack
-
Copy-pasta
Motivation
Copy-paste environments are an intuitive way to maintain multiple environments. They avoid the blast radius problem of the multi-headed stack antipattern. You can also easily customize each stack instance.
Applicability
Copy-paste environments may be appropriate when you want to maintain and change different instances and aren’t worried about code duplication or consistency.
Consequences
It can be challenging to maintain multiple copy-paste environments. When you want to make a code change, you need to copy it to every project. You probably need to test each instance separately, as a change may work in one but not another.
Copy-paste environments often suffer from configuration drift (“Configuration Drift”). Using copy-paste environments for delivery environments reduces the reliability of the deployment process and the validity of testing, due to inconsistencies from one environment to the next.
Implementation
You create a copy-paste environment by copying the project code from one stack instance into a new project. You then edit the code to customize it for the new instance. When you make a change to one stack, you need to copy and paste it across all of the other stack projects, while keeping the customizations in each one.
Related patterns
In cases where stack instances are meant to represent the same stack, the reusable stack pattern (“Pattern: Reusable Stack”) is usually more appropriate. The wrapper stack pattern (“Pattern: Wrapper Stack”) avoids the disadvantages of a copy-paste stack by having all of the logic in a module and only using each stack project for configuration.
Pattern: Reusable Stack
A Reusable Stack is an infrastructure source code project that is used to create multiple instances of a stack.
Figure 7-5. A Reusable Stack is an infrastructure source code project that is used to create multiple instances of a stack.
Also known as
-
Cookie Cutter Stack
-
Template Stack
-
Library Stack
Motivation
You create a reusable stack to maintain multiple consistent instances of infrastructure. When you make changes to the stack code, you can apply and test it in one instance, and then use the same code version to create or update multiple additional instances. You want to provision new instances of the stack with minimal ceremony, maybe even automatically.
Applicability
You can use a reusable stack for delivery environments or for multiple production environments. This pattern is useful when you don’t need much variation between the environments. It is less applicable when environments need to be heavily customized.
Consequences
The ability to provision and update multiple stacks from the same project enhances scalability, reliability, and throughput. You can manage more instances with less effort, make changes with a lower risk of failure, and roll changes out to more systems more rapidly.
You typically need to configure some aspects of the stack differently for different instances, even if it’s just what you name things. I’ll spend a whole chapter talking about this (Chapter 8).
You should test your stack project code before you apply changes to business-critical infrastructure. I’ll spend multiple chapters on this, including Chapter 9 and Chapter 10.
Implementation
You create a reusable stack as an infrastructure stack project, and then run the stack management tool each time you want to create or update an instance of the stack. Use the syntax of the stack tool command to tell it which instance you want to create or update. With Terraform, for example, you would specify a different state file or workspace for each instance. With CloudFormation, you pass a unique stack ID for each instance.
The following example command provisions two stack instances from a single project using a fictional command called stack. The command takes an argument env that identifies unique instances:
>stack upenv=test--source mystack/srcSUCCESS: stack 'test' created>stack upenv=staging --source mystack/srcSUCCESS: stack 'staging' created
As a rule, you should use simple parameters to define differences between stack instances - strings, numbers, or in some cases, lists. Additionally, the infrastructure created by a reusable stack should not vary much across instances.
Related patterns
The reusable stack is an improvement on the copy-paste environment antipattern (“Antipattern: Copy-Paste Environments”), making it easier to keep multiple instances consistent.
Stack code modules (Chapter 6) allow you to define code once and then share it across multiple stack projects. Unlike a reusable stack, a stack module is not used directly to create infrastructure-instead, a stack project imports modules. So modules are more of a supplement to reusable stacks than a substitute.
The wrapper stack pattern (“Pattern: Wrapper Stack”) uses modules to implement a combination of copy-paste environments (one project per instance) and a reusable stack (one copy of the code that defines the stack).
The patterns I describe in Chapter 8 offer ways to configure instances of reusable stacks.
Building environments with multiple stacks
The reusable stack pattern describes an approach for implementing multiple environments. In Chapter 5 I described different ways to structuring a system’s infrastructure across multiple stacks (“Patterns and antipatterns for structuring stacks”). There are several ways you can implement your stacks to combine these two dimensions of environments and system structure.
The simple case is implementing the complete system as a single stack. When you provision an instance of the stack, you have a complete environment. I depicted this in the diagram for the reusable stack pattern (Figure 7-5).
But you should split larger systems into multiple stacks. For example, if you follow the service stack pattern (“Pattern: Service Stack”) you have a separate stack for each service:
Figure 7-6. Example using a separate infrastructure stack for each service
To create multiple environments, you provision an instance of each service stack for each environment:
Figure 7-7. Using multiple stacks to build each environment
You would use commands like the following to build a full environment with multiple stacks:
>stack upenv=staging --source product_browse_stack/srcSUCCESS: stack 'product_browse-staging' created>stack upenv=staging --source product_search_stack/srcSUCCESS: stack 'product_search-staging' created>stack upenv=staging --source shopping_basket_stack/srcSUCCESS: stack 'shopping_basket-staging' created
In [Link to Come] I describe strategies for splitting systems into multiple stacks, and how to integrate infrastructure across stacks.
Conclusion
Reusable stacks should be the workhorse pattern for most teams who need to manage large infrastructures. This pattern creates challenges and opportunities. One challenge is how to configure each stack instance, which is the topic of Chapter 8, the next chapter.
However, reusable stacks create the opportunity to test infrastructure code before you apply it to environments you care about. In Chapter 9 I’ll explain the core practice of continuously testing infrastructure code, and in Chapter 10 I’ll describe implementation patterns following this practice with infrastructure stacks.
1 Charity Majors shared her painful experiences of working with a multiple-environment stack in a blog post.