One of the principles of good software design is the DRY principle (“Don’t Repeat Yourself”)1. As you write code for different infrastructure stacks, you may find yourself writing very similar code multiple times. Rather than maintaining the same code in multiple places, it is often easier to maintain a single copy of the code.
Most stack management tools support modules, or libraries, for this reason. A Stack Code Module is a piece of infrastructure code that you can use across multiple stack projects. You can version, test, and release the module code independently of the stack that uses it.
As an example, the Foodspin team has several servers in their system-container host servers, application servers, and a CD server. The code for each of these is similar:
virtual_machine
:
name
:
foodspin-clusterhost
source_image
:
hardened-linux-base
memory
:
16GB
provision
:
tool
:
servermaker
maker_server
:
maker.foodspin.io
role
:
clusterhost
virtual_machine
:
name
:
foodspin-appserver
source_image
:
hardened-linux-base
memory
:
8GB
provision
:
tool
:
servermaker
maker_server
:
maker.foodspin.io
role
:
appserver
virtual_machine
:
name
:
foodspin-gocd-master
source_image
:
hardened-linux-base
memory
:
4GB
provision
:
tool
:
servermaker
maker_server
:
maker.foodspin.io
role
:
appserver
The Foodspin team writes a module to declare a server, with placeholders for parameters:
declare module
:
foodspin-server
parameters
:
name
:
STRING
memory
:
STRING
server_role
:
STRING
virtual_machine
:
name
:
${name}
source_image
:
hardened-linux-base
memory
:
${memory}
provision
:
tool
:
servermaker
maker_server
:
maker.foodspin.io
role
:
${server_role}
Each stack that needs a server can then use this module:
use module
:
foodspin-server
name
:
foodspin-clusterhost
memory
:
16GB
server_role
:
clusterhost
use module
:
foodspin-server
name
:
foodspin-appserver
memory
:
8GB
server_role
:
appserver
use module
:
foodspin-server
name
:
foodspin-gocd-master
memory
:
4GB
server_role
:
gocd_master
The code for each server is simpler using modules since it only declares the elements that are different rather than duplicating the declarations for the server image and server provisioner. Most usefully, they can make changes to how they provision servers in one location, rather than needing to change code in different codebases.
Modules can be useful for reusing code. However, they also add complexity. So it’s essential to use modules in a way that creates more value than complexity. A useful module simplifies stack code, improving readability and maintainability. A poor module reduces flexibility and is difficult to understand and maintain.
Here are a few patterns and antipatterns that illustrate what works well and what doesn’t:
A Facade Module provides a simplified interface for a resource provided by the stack tool language, or the underlying platform API.
An Anemic Module wraps the code for an infrastructure element but does not simplify it or add any particular value.
A Domain Entity Module implements a high-level concept by combining multiple low-level infrastructure resources.
A Spaghetti Module is configurable to the point where it creates significantly different results depending on the parameters given to it
An Obfuscation Layer is composed of multiple modules. It is intended to hide or abstract details of the infrastructure from people writing stack code but instead makes the codebase as a whole harder to understand, maintain, and use.
A One-shot Module is only used once in a codebase, rather than being reused.
A Facade Module provides a simplified interface for a resource provided by the stack tool language, or the underlying platform API.
The facade module presents a small number of parameters to code that uses it:
use module
:
foodspin-server
name
:
burgerbarn-appserver
memory
:
8GB
The module itself includes a larger amount of code, that the calling code doesn’t need to worry about:
declare module
:
foodspin-server
virtual_machine
:
name
:
${name}
source_image
:
hardened-linux-base
memory
:
${memory}
provision
:
tool
:
servermaker
maker_server
:
maker.foodspin.io
role
:
application_server
network
:
vlan
:
application_zone_vlan
gateway
:
application_zone_gateway
firewall_inbound
:
port
:
22
from
:
management_zone
firewall_inbound
:
port
:
443
from
:
webserver_zone
A facade module simplifies and standardizes a common use case for an infrastructure resource. The stack code the uses a facade module should be simpler and easier to read. Improvements to the quality of the module code are rapidly available to all of the stacks which use it.
Facade modules work best for simple use cases, usually involving a low-level infrastructure resource.
A facade module limits how you can use the underlying infrastructure resource. Doing this can be useful, simplifying options and standardizing around good quality implementations. However, it can also reduce flexibility.
A module is an extra layer of code between the stack code and the code that directly specifies the infrastructure resources. This extra layer adds at least some overhead to maintaining, debugging, and improving code. It can also make it harder to understand the stack code.
Implementing a facade module generally involves specifying an infrastructure resource with a number of hard-coded values, and a small number of values that are passed through from the code that uses the module.
An anemic module (“Antipattern: Anemic Module”) is a facade module that doesn’t hide much, so adds complexity without adding much value. A domain entity module (“Pattern: Domain Entity Module”) is similar to a facade, in that it presents a simplified interface to the code that uses it. But a domain entity combines multiple lower-level elements to present a single higher-level entity to the calling code. In contrast, a facade module is a more direct mapping of a single element.
An Anemic Module wraps the code for an infrastructure element but does not simplify it or add any particular value. It may be a facade module (“Pattern: Facade Module”) gone wrong, or it may be part of an obfuscation layer (“Antipattern: Obfuscation Layer”).
use module
:
any_server
server_name
:
burgerbarn-appserver
ram
:
8GB
source_image
:
base_linux_image
provisioning_tool
:
servermaker
server_role
:
application_server
vlan
:
application_zone_vlan
gateway
:
application_zone_gateway
firewall_rules
:
firewall_inbound
:
port
:
22
from
:
management_zone
firewall_inbound
:
port
:
443
from
:
webserver_zone
The module itself passes the parameters directly to the stack management tool’s code:
declare module
:
any_server
virtual_machine
:
name
:
${server_name}
source_image
:
${origin_server_image}
memory
:
${ram}
provision
:
tool
:
${provisioning_tool}
role
:
${server_role}
network
:
vlan
:
${server_vlan}
gateway
:
${server_gateway}
firewall_inbound
:
${firewall_rules}
Value-Free Wrapper, Pass-Through Module, Obfuscator Module.
Sometimes people write this kind of module aiming to follow the DRY (Don’t Repeat Yourself) principle. They see that code that defines a common infrastructure element, such as a virtual server, load balancer, or security group, is used in multiple places in the codebase. So they create a module that declares that element type once and use that everywhere. But because the elements are being used differently in different parts of the code, they need to expose a large number of parameters in their module. The result is that the code that uses the module is no simpler than directly using the stack tool’s code. The codebase still has repeated code, only now it’s repeated use of the module.
Nobody intentionally writes an anemic module. You may debate whether a given module is anemic or is a facade, but the debate itself is useful. You should consider whether a module adds real value and, if not, then refactor it into code that uses the stack language directly.
Writing, using, and maintaining module code rather than directly using the constructs provided by your stack tool adds overhead. It makes your stack code more difficult to understand, especially for people who know the stack tool but are new to your codebase. It adds more code to maintain, usually with separate versioning and release. You can end up with different versions of a module being used by different stacks, causing people to waste time and energy managing releases, upgrades, and testing.
If a module does more than pass parameters, but still presents too many parameters to code that uses it, you might consider splitting it into multiple modules. Doing this makes sense when there are a few common but different use cases for the infrastructure element the module defines. For example, rather than a single module for defining firewall rules, you may want one module to define a firewall rule for public HTTPS traffic, another for internal HTTPS traffic, and a third for SSH traffic. Each of these may need fewer parameters than a single module that handles multiple protocols and scenarios.
Some anemic modules are “Pattern: Facade Module” that grew out of control. Others are part of an “Antipattern: Obfuscation Layer”. It may also be a “Pattern: Domain Entity Module” that maps a bit too directly to a single underlying infrastructure element, rather than to a useful combination of elements.
When organizing code of any kind into different components, you need to consider dependencies. A stack project that uses a module depends on that module2. The level of coupling describes how much a change to one part of the codebase affects other parts. If a stack is tightly coupled to a module, then changes to the module will probably require changing the stack code as well. This is a problem when multiple stacks are tightly coupled to a module, or when there is tight coupling across multiple modules and stacks. Tight coupling creates friction and risk for making changes to code.
You should aim to make your code loosely coupled. You can draw on lessons from software architecture to find ways to identify and avoid tight coupling3.
The level of cohesion of a component describes how well-focused it is. A module that does too much, like a spaghetti module (“Antipattern: Spaghetti Module”), has low cohesion. A module has high cohesion when it has a clear focus. A facade module (“Pattern: Facade Module”) can be highly cohesive around a low-level infrastructure resource, while a domain entity module (“Pattern: Domain Entity Module”) can be highly cohesive around a clear, logical entity.
A Domain Entity Module implements a high-level concept by combining multiple low-level infrastructure resources. An example of a higher level concept is the infrastructure needed to run a specific Java application. This example shows how a module that implements a Java application infrastructure instance might be used from stack project code:
use module
:
java-application-infrastructure
name
:
"shopping_app_cluster"
application
:
"shopping_app"
application_version
:
"4.20"
network_access
:
"public"
The module creates a complete set of infrastructure elements, including a cluster of virtual servers with load balancer, database instance, and firewall rules.
Infrastructure stack languages provide language constructs that map directly to resources and services provided by infrastructure platforms (the things I described in Chapter 3). Teams combine these low-level constructs into higher-level constructs to serve a particular purpose, such as hosting a Java application or running a data pipeline.
It can take a fair bit of low-level infrastructure code to define all of the pieces needed to meet that high-level purpose. As I described above, to create the infrastructure to run a Java application, you may need to write code for a cluster of virtual servers, for a load balancer, for a database instance, and for firewall rules. That’s a lot of code. Although you may write modules so you can share code for each low-level resource, if you need multiple different applications with similar infrastructure, it would be useful to share code at this higher level as well.
A domain entity module is useful for things that are fairly complex to implement, and that are used in pretty much the same way in multiple places in your system. Don’t create a domain entity module that you only use once. And don’t create one of these modules for multiple instances of the same thing, when each instance is significantly different. The first case is a one-shot module (“Antipattern: One-shot Module”), the second risks becoming a spaghetti module (“Antipattern: Spaghetti Module”).
For example, you may run multiple application servers that use the same stack-operating system, application language, and application deployment method. This is a candidate for a domain entity module.
As a counter-example, you may have multiple application servers, but some run Windows, some run Linux. One runs a stateless PHP application, another runs a Java application on Tomcat with a MySQL database, while a third hosts a .Net application that uses message queues to integrate with other applications. Although all three of these are application servers, their implementation is quite different. Any module that can create a suitable infrastructure stack for all of these is unlikely to have a clean design and implementation.
On a concrete level, implementing a domain entity model is a matter of writing the code for a related grouping of infrastructure in a single module. But the best way to create a high-quality codebase that is easy for people to learn and maintain is to take a design-led approach.
I recommend drawing from lessons learned in software architecture and design. The domain entity module pattern derives from Domain Driven Design (DDD)5, which creates a conceptual model for the business domain of a software system, and uses that to drive the design of the system itself. Infrastructure, especially one designed and built as software, can be seen as a domain in its own right. The domain is building, delivering, and running software.
A particularly powerful approach is for an organization to use DDD to design the architecture for the business software, and then extend the domain to include the systems and services used for building and running that software.
The code to implement a Java application server might look like Example 6-9. This example creates and assembles three different resources: a DNS entry, which points to a load balancer, which routes traffic to a server cluster.
declare module
:
java-application-infrastructure
dns_entry
:
id
:
"${APP_NAME}-hostname"
record_type
:
"A"
hostname
:
"${APP_NAME}.foodspin.io"
ip_address
:
{
$load_balancer.ip_address
}
server_cluster
:
id
:
"${APP_NAME}-cluster"
min_size
:
1
max_size
:
3
each_server_node
:
source_image
:
base_linux
memory
:
4GB
provision
:
tool
:
servermaker
role
:
appserver
parameters
:
app_package
:
"${APP_NAME}-${APP_VERSION}.war"
app_repository
:
"repository.foodspin.io"
load_balancer
:
protocol
:
https
target
:
type
:
server_cluster
target_id
:
"${APP_NAME}-cluster"
Code to provision a stack that runs a search application using this module might look like this:
use module
:
java-application-infrastructure
app_name
:
foodspin_search_app
app_version
:
1.23
If you find that some stack projects have very little code other than importing a single module, you might consider putting the code directly into the stack project. Reusing code in the form of an entire stack, rather than a module, is the reusable stack pattern (“Pattern: Reusable Stack”). In some cases, having the entire stack code in a module is deliberate, as in the wrapper stack pattern (“Pattern: Wrapper Stack”).
A facade module (“Pattern: Facade Module”) is like a domain entity module that only creates a single low-level infrastructure element. A spaghetti module (“Antipattern: Spaghetti Module”) is similar to a domain entity module but has too many different options.
A Spaghetti Module is configurable to the point where it creates significantly different results depending on the parameters given to it. The implementation of the module is messy and difficult to understand, because it has too many moving parts:
declare module
:
application-server-infrastructure
variable
:
network_segment = {
if ${parameter.network_access} = "public"
id
:
public_subnet
else if ${parameter.network_access} = "customer"
id
:
customer_subnet
else
id
:
internal_subnet
end
}
switch ${parameter.application_type}
:
"java"
:
virtual_machine
:
origin_image
:
base_tomcat
network_segment
:
${variable.network_segment}
server_configuration
:
if ${parameter.database} != "none"
database_connection
:
${database_instance.my_database.connection_string}
end
...
"NET"
:
virtual_machine
:
origin_image
:
windows_server
network_segment
:
${variable.network_segment}
server_configuration
:
if ${parameter.database} != "none"
database_connection
:
${database_instance.my_database.connection_string}
end
...
"php"
:
container_group
:
cluster_id
:
${parameter.container_cluster}
container_image
:
nginx_php_image
network_segment
:
${variable.network_segment}
server_configuration
:
if ${parameter.database} != "none"
database_connection
:
${database_instance.my_database.connection_string}
end
...
end
switch ${parameter.database}
:
"mysql"
:
database_instance
:
my_database
type
:
mysql
...
...
The example code above assigns the server it creates to one of three different network segments, and optionally creates a database cluster and passes a connection string to the server configuration. In some cases, it creates a group of container instances rather than a virtual server. This module is a bit of a beast.
Swiss Army module.
As with other antipatterns, people create a spaghetti module by accident, often over time. You may create a facade module (“Pattern: Facade Module”) for a common infrastructure element, such as a server, that grows so that it can create radically different types of servers. Or you may create a domain entity module (“Pattern: Domain Entity Module”) for an application’s infrastructure. But then you find that the module needs to optionally include a variety of elements, such as databases, message queues, load balancers, and public Internet gateways, depending on the application that runs on it.
A module that does too many things is less maintainable than one with a tighter scope. The more things a module does, and the more variations there are in the infrastructure that it can create, the harder it is to change it without breaking something. These modules are harder to test. As I explain in a later chapter (Chapter 9), better-designed code is easier to test, so if you’re struggling to write automated tests and build pipelines to test the module in isolation, It’s a sign that you may have a spaghetti module.
A spaghetti module’s code often contains conditionals, that apply different specifications in different situations. For example, a database cluster module might take a parameter to choose which database to provision.
When you realize you have a spaghetti module on your hands, you should refactor it. Often, you can split it into different modules, each with a more focused remit. For example, you might decompose your single application infrastructure module into different modules for different parts of the application’s infrastructure. An example of a stack that uses decomposed modules in this way, rather than using the spaghetti module from the earlier example (Example 6-10) might look like this:
use module
:
java-application-servers
name
:
burgerbarn_appserver
application
:
"shopping_app"
application_version
:
"4.20"
network_segment
:
customer_subnet
server_configuration
:
database_connection
:
${module.mysql-database.outputs.connection_string}
use module
:
mysql-database
cluster_minimum
:
1
cluster_maximum
:
3
allow_connections_from
:
customer_subnet
Each of the modules is smaller, simpler, and so easier to maintain and tested than the original spaghetti module.
A spaghetti module is often a domain entity module (“Pattern: Domain Entity Module”) gone wrong.
Unlike the other patterns and antipatterns in this chapter, an Obfuscation Layer is composed of multiple modules. Intended to hide or abstract details of the infrastructure from people writing stack code, it instead makes the codebase as a whole harder to understand, maintain, and use.
Abstraction layer, portability layer, in-house infrastructure model.
An obfuscation layer is usually intended to simplify or standardize implementation. A team might create a library of modules as an in-house model for building infrastructure. Sometimes the intention is for people to write stack code without needing to learn the stack tool itself. In other cases, the layer tries to abstract the specifics of the infrastructure platform so that code can be written once and used across multiple platforms.
If your team uses a heavily customized model for infrastructure code, then it becomes a barrier to new people joining your team. Even someone who knows the stack tools you use has a steep learning curve to understand your special language. An in-house obfuscation layer tends to be inflexible, forcing people to either work around its limitations or spend time making changes to the obfuscation layer itself. Either way, people waste time creating and maintaining extra code.
People, especially managers and hands-off architects6, often overestimate the benefit of hiding an underlying platform or tools from people writing infrastructure code. I have yet to see a cloud abstraction layer that adds noticeable value, or that avoids adding cost and complexity. Most fail at both.
Domain entity modules (“Pattern: Domain Entity Module”) may be used to create an obfuscation layer at a higher level, while facade modules (“Pattern: Facade Module”) would be used to obfuscate lower-level resources. If each module in the layer handles multiple platforms, they are probably spaghetti modules (“Antipattern: Spaghetti Module”).
A one-shot module is only used once in a codebase, rather than being reused.
People usually create one-shot modules as a way to organize the code in a project.
If a stack project includes enough code that it becomes difficult to navigate, you have a few options. Splitting the stack into modules is one approach. If the stack is conceptually doing too much, it might be better to divide it into multiple stacks, using an appropriate stack structural pattern (see “Patterns and antipatterns for structuring stacks”). Otherwise, merely organizing code into different files and, if necessary, different folders, can make it easier to navigate and understand the codebase without the overhead of the other options.
Organizing the code into modules adds the overhead of declaring the module and passing parameters. You add even more complexity if you manage the module code separately, with separate versioning. This overhead is worthwhile when you share code across multiple stack projects. The benefits of code reuse make up for the added time and energy of maintaining a separate module. Paying that cost when you’re not using the benefit is a waste.
A one-shot module may map closely to lower-level infrastructure elements, like a facade module (“Pattern: Facade Module”), or to a higher-level entity, like a domain entity module (“Pattern: Domain Entity Module”).
This chapter has explored the use of modules to share code across stacks. Modules are a popular mechanism to manage the growth of a codebase, but as you can see, it’s easy to create an unmaintainable mess.
The next chapter (Chapter-Building-Environments) is devoted to one of the most common situations that require multiple instances of the same or similar infrastructure. This need leads to some patterns and antipatterns for implementing shared code at the level of the stack.
1 The DRY principle, and others, can be found in The Pragmatic Programmer: From Journeyman to Master by Andrew Hunt and David Thomas
2 In later chapters ([Link to Come]) we’ll see that a stack can depend on another stack, as well.
3 Two resources to get started include Martin Fowler’s paper Reducing Coupling, and Mohamed Sanaulla’s post Cohesion and Coupling: Two OO Design Principles.
4 Later on (“Example of how Foodspin moves to reusable stacks”) we’ll see how the Foodspin team evolves this to use a single reusable stack project
5 See Domain-Driven Design: Tackling Complexity in the Heart of Software, by Eric Evans, 2003 Addison Wesley
6 For a thoughtful view on how architecture, and architects, relates to implementation, I recommend the Architect Elevator, by Gregor Hohpe.
3.135.207.129