Choose tools that are configured with code

Infrastructure as Code, by definition, involves specifying your infrastructure in text-based files. You manage these files separately from the tools that you use to apply them to your system. You can read, edit, analyze, and manipulate your specifications using any tools you want.

Other infrastructure automation tools store your infrastructure specifications as data that you can’t access directly. Instead, you can only use and edit the specifications by using the tool itself. The tool may have some combination of GUI, API, and command-line.

The issue with these black-box tools is that they limit the practices and workflows you can use:

• You can only version your infrastructure specifications if the tool has built-in versioning,

• You can only use Continuous Integration if the tool has a way to trigger a job automatically when you make a change,

• You can only create delivery pipelines if the tool makes it easy to version and promote your infrastructure specifications,

• You can only split monolithic infrastructure into independent pieces if the tool supports it.

It is challenging to use agile engineering practices such as Test-Driven Development, Continuous Integration, and Continuous Delivery with black-box infrastructure management tools.

A tool that uses external code for its specifications doesn’t constrain you to use a specific workflow. You can use an industry-standard source control system, text editor, CI server, and automated testing framework. You can build delivery pipelines using the tool that works best for you.

Manage your code in a version control system

If you’re defining your stuff as code, then putting that code into a version control system (VCS) is simple and powerful. By doing this, you get:

Traceability

VCS provides a history of changes, who made them, and context about why¹. This history is invaluable when debugging problems.

Rollback

When a change breaks something—and especially when multiple changes break something—it’s useful to be able to restore things to exactly how they were before.

Correlation

Keeping scripts, specifications, and configuration in version control helps when tracing and fixing gnarly problems. You can correlate across pieces with tags and version numbers.

Visibility

Everyone can see each change committed to the version control system, giving the team situational awareness. Someone may notice that a change has missed something important. If an incident happens, people are aware of recent commits that may have triggered it.

Actionability

The VCS can trigger an action automatically for each change committed. Triggers enable CI jobs and CD pipelines.

Secrets and source code

Systems need various secrets. Your stack tool may need a password or key to use your platform’s API to create and change infrastructure. You may also need to provision secrets into environments, for example making sure an application has the password for its database.

It’s essential to handle these types of secrets in a secure way from the very beginning. Whether you are using a public cloud or a private cloud a leaked password can have terrible consequences. So even when you are only writing code to learn how to use a new tool or platform, you should never put secrets into code. There are many stories of people who checked a secret into a source repository they thought was private, only to find it had been discovered by hackers who exploited it to run up huge bills.

One solution to this is to encrypt secrets in order to store them in code². Infrastructure developers and unattended systems need to be able to decrypt these secrets without storing a secret. So you still have at least one secret to manage outside of source control!

There are a few approaches for handling secrets needed by infrastructure code without actually putting them into code. The include secretless authorization, runtime secret injection, and disposable secrets.

Scripting your infrastructure

Before standard tools appeared for provisioning cloud infrastructure declaratively, we wrote scripts in general-purpose, procedural languages. These used SDK (Software Development Kit) libraries to interact with the cloud provider’s API.

Example 4-1 uses pseudo-code, but is similar to scripts that I wrote in Ruby with the AWS SDK. It creates a server named my_application_server and then runs the (fictional) servermaker tool to configure it.

import 'cloud-api-library'

network_segment = CloudApi.find_network_segment('private')

app_server = CloudApi.find_server('my_application_server')
if(app_server == null) {
  app_server = CloudApi.create_server(
    name: 'my_application_server',
    image: 'base_linux',
    cpu: 2,
    ram: '2GB',
    network: network_segment
  )
  while(app_server.ready == false) {
    wait 5
  }
  app_server.provision(
    provisioner: servermaker,
    role: tomcat_server
  )
}

This script combines what and how. It specifies attributes of the server, including the CPU and memory resources to provide it, what OS image to start from, and what Ansible role to apply to the server. It also implements logic: it checks whether the server named my_application_server already exists, to avoid creating a duplicate server, and then it waits for the server to become ready before running Ansible on it. The script would need additional logic to handle errors, which I didn’t include in the example.

The example code also doesn’t handle changes to the server’s attributes. What if you need to increase the RAM? You could change the script so that if the server exists, the script will check each attribute and change it if necessary. Or you could write a new script to find and change existing servers.

More realistic scenarios include multiple servers of different types. In addition to our application server, my team had web servers and database servers. We also had multiple environments, which meant multiple instances of each server.

Teams I worked with often turned simplistic scripts like the one in this example into a multi-purpose script. This kind of script would take arguments specifying the type of server and the environment, and use these to create the appropriate server instance. We evolved this into a script that would read configuration files that specify various server attributes.

I was working on a script like this, wondering if it would be worth releasing it as an open-source tool, when Hashicorp released the first version of Terraform.

Building infrastructure with declarative code

Terraform, like most other stack provisioning tools and server configuration tools, uses a declarative language. Rather than a procedural language, which executes a series of statements using control flow logic like if statements and while loops, a declarative language is a set of statements that declare the result you want.

Example 4-2 creates the same server as Example 4-1. The code in this example (as with most code examples in this book) is a fictional language⁶.

virtual_machine:
  name: my_application_server
  source_image: 'base_linux'
  cpu: 2
  ram: 2GB
  network: private_network_segment
  provision:
    provisioner: servermaker
    role: tomcat_server

This code doesn’t include any logic to check whether the server already exists or to wait for the server to come up before running Ansible. The tool that applies the code implements this logic, along with error-handling. The tool also checks the current attributes of infrastructure against what is declared, and work out what changes to make to bring the infrastructure in line. So to increase the RAM of the application server in this example, you can edit the file and re-run the tool.

Declarative infrastructure tools like Terraform and Chef keep the what and how separate. The code you write as a user of the tool only declares the attributes of your infrastructure. The tool implements the logic for how to make it happen. As a result, your code is cleaner and more direct.

DSLs for infrastructure

In addition to being declarative, infrastructure tools often use their own DSL⁷.

The advantage of a DSL for infrastructure code is keeping the code simple and focused.

A general-purpose language needs extra syntax, such as declarations of variables and references to class structures in a DSL:

import 'cloud-api-library'
app_server = CloudApi.find_server('my_application_server')
if(app_server == null) {
  app_server = CloudApi.create_server(name: 'my_application_server')

A DSL can strip this to the most relevant elements:

virtual_machine:
  name: my_application_server

The return of general-purpose languages for infrastructure

Newer tools, such as Pulumi and the AWS CDK, bring general-purpose languages back to infrastructure coding. There are several arguments for doing this, some more convincing than others.

Configuration isn’t real code

Some folks take the phrase “Infrastructure as Code” to heart. They argue that declarative languages are just configuration, not a “real” language. Personally, I’m not bothered if someone disparages my code as being mere configuration. I still find it useful to keep “what” and “how” separate, and to avoid writing repetitive, verbose code.

It’s useful not to have to learn a new language

Using a popular language like JavaScript means more people can learn to write infrastructure as code since they don’t need to learn a peculiar special-purpose language. I have sympathy for making it easy for people to adopt infrastructure as code. But I don’t think a new language syntax, especially a simple declarative language, is as hard to learn as the domain-specific aspects of infrastructure code like networking constructs.

Infrastructure DSLs are not well-supported by development tools

This is generally true. There are many mature IDE’s⁸ for languages like JavaScript, Python, and Ruby. These have loads of features, like syntax highlighting and code refactoring, that help developers to be more productive. Rather than discouraging the use of new languages (of any kind), I would love to see vendors improve their support for infrastructure coding languages.

Proper support for libraries helps you to simplify code

Most infrastructure DSLs let you write modules (as I’ll discuss in Chapter 6). But these are not as rich and flexible as libraries in mature, procedural languages like JavaScript, Python, and Ruby. I discuss reusable modules and libraries for stacks in Chapter 6.

Using the same language lets you combine infrastructure and application code

Examples of this show application code that provisions its own infrastructure. People who’ve been developing web applications for a while recall how nifty it was to be able to embed SQL statements into HTML code. Experience has shown that this does not lead to cleanly designed, maintainable systems. It is often useful to specify actions for a tool to trigger on specific changes to infrastructure, such as provisioning newly created resources. But this can be done without intermingling the different styles of code.

Infrastructure languages are less testable

Also true, although there are several reasons for this. One is that the ecosystem of testing tools for infrastructure is not as mature as that for other types of code. Another reason is that testing declarations requires a different approach to testing logic. A third reason is that testing code that provisions infrastructure has much longer feedback loops. I delve into these in “Challenges with testing infrastructure code”.

The swing back towards general-purpose languages for infrastructure is new⁹. Some of the arguments threaten to regress us to codebases filled with verbose spaghetti code mingling configuration, application logic, and repetitive utility code. But I expect that this is just one step on a path to languages that support better coding.

For many teams today, the challenges with their codebase do not come from the language they use. Regardless of language, they need ways to keep their code clean, well-organized, and easy to maintain.

Implementation Principle: Avoid mixing different types of code

As discussed earlier, some infrastructure coding languages are procedural, and some are declarative¹⁰. Each of these paradigms has its strengths and weaknesses.

A particular scourge of infrastructure is code that mixes both declarative and procedural code, as in Example 4-3. This code includes declarative code for defining a server as well as procedural code that determines which attributes to assign to the server depending on the server role and environment.

for ${env} in ["test", "staging", "prod"] {
  for ${server_role} in ["app", "web", "db"] {
    server:
      name: ${server_role}-${env}
      image: 'base_linux'
      cpu: if(${env} == "prod" || ${env} == "staging") {
        4
      } else {
        2
      }
      ram: if(${server_role} == "app") {
        "4GB"
      } else if (${server_role} == "db") {
        "2GB"
      } else {
        "1GB"
      }
      provision:
        provisioner: servermaker
        role: ${server_role}
  }
}

Mixed declarative and procedural code is a design smell¹¹https://martinfowler.com/bliki/CodeSmell.html. A “smell” is some characteristic of a system that you observe that suggests there is an underlying problem. In the example from the text, code that mixes declarative and procedural constructs is a smell. This smell suggests that your code may be trying to do multiple things and that it may be better to pull them apart into different pieces of code, perhaps in different languages.]. It’s not easy to understand this code, which makes it harder to debug, and harder to change without breaking something.

The messiness of this code comes from intermingling two different concerns, which leads to the next implementation principle.

Implementation Principle: Separate infrastructure code concerns

A common reason for messy code is that it is doing multiple things. These things may be related, but teasing them apart can make them easier to distinguish, and improve the readability and maintainability of the code.

Example 4-3 does two things: it specifies the infrastructure resource to create, and it configures that resource differently in different contexts. The example illustrates two of the four most common concerns of infrastructure code:

Specification

Specifications define the shape of your infrastructure. Your server has specific packages, configuration files, and user accounts. Declarative languages work well for this because specifications are what you want. Specifications are what most people mean when they talk about infrastructure code.

Configuration

Configuration defines the things that vary when you provision different instances of infrastructure. Different application servers built from a single specification may need different amounts of RAM. You may want to deploy different applications onto otherwise identical servers. Configuration is almost always declarative.

Execution

Execution applies specification and configuration to the actual infrastructure resources. Together, the specification and configuration declare what you want. The orchestration is about how to make that happen. Procedural or functional code usually works best for orchestration. Ideally, you should use an off the shelf tool for this rather than writing your own code.

Orchestration

Orchestration combines multiple specifications and configurations. For example, you may need to create a server in the cloud platform, and then install packages on the server. Or you may need to create networking structures, and then create multiple servers attached to those structures.

virtual_machine:
  name: application_server_${CUSTOMER}
  source_image: 'base_linux'
  provision:
    tool: servermaker
    role: foodspin_application
  network: $(switch ${CUSTOMER}) {
    "bomber_burrito": us_network
    "curry_hut": uk_network
    "burger_barn": au_network
  }

virtual_machine:
  name: application_server_${CUSTOMER}
  source_image: 'base_linux'
  provision:
    tool: servermaker
    role: foodspin_application
  network: ${NETWORK}

switch (${CUSTOMER}) {
  case 'bomber_burrito':
    return 'us_network'
  case 'curry_hut':
    return 'uk_network'
  case 'burger_barn':
    return 'au_network'
}

Implementation Principle: Treat infrastructure code like real code

To keep an infrastructure codebase clean, you need to treat it as a first-class concern. Too often, people don’t consider infrastructure code to be “real” code. They don’t give it the same level of engineering discipline as application code.

Design and manage your infrastructure code so that it is easy to understand and maintain. Follow code quality practices, such as code reviews, pair programming, and automated testing. Your team should be aware of technical debt and strive to minimize it.