1 Introduction

Besides black art, there is only automation and mechanization. - Federico Garcia Lorca

Building infrastructure is (pretty) easy. Building it in an automated, repeatable, and reusable way is much, much harder.

Perhaps you are a systems administrator, a developer, or you’re doing DevOps work, and you’ve discovered that you need to manage your infrastructure components more efficiently and effectively. Maybe you’ve inherited a bespoke Cloud, Docker, or virtualized environment, or maybe you need to migrate to the Cloud or a virtualization environment—but you’re finding the tools available to you aren’t sufficient.

In some cases you might have a mix of manually constructed infrastructure, hand-edited configuration files, and 10,000-line shell scripts copied from one host to another, with undocumented fixes accumulating each time. If you’re lucky, you might have a configuration management tool like Puppet or Chef to deploy your applications and services, or one or more bespoke deployment tools written to deploy infrastructure and applications. Solutions are often undocumented, not scalable, and boutique to your existing infrastructure and architecture.

In addition to a lack of tooling, you have colleagues and customers who want their environments built more quickly than ever before. They want changes to compute and storage and networking to happen promptly—all error-free and done as automatically as possible. The combination of these challenges means you need to find a better way of doing things.

Thankfully in the last few years a lot of tools have been developed to help you out. These include tools that allow you to build applications and infrastructure like you write code. These tools enable a new concept:

1.1 Infrastructure as Code

“Infrastructure as Code” (IaC) enables us to describe our infrastructure and applications in source code. This enables us to treat our infrastructure like we treat our applications and take advantage of development practices and tools.

Infrastructure represented, and programmatically manipulatable, in code can be versioned, shared, re-used, debugged, and potentially reverted when something goes wrong. We can apply the tenets of our application development life cycle to our infrastructure including code reviews, linting, and automated testing. Instead of poorly and infrequently maintained documentation or state stored in the minds of your teammates, we can create managed environments that we can easily introspect, monitor, and debug.

Once our infrastructure is articulated in code it becomes equivalent to our application code. Instead of roadblocks or detours to handle infrastructure differently from applications we can instead:

  • Build and provision automated development, testing, and production environments.
  • Provision our applications and services on top of these environments.
  • Automate the deployment workflow between them, beginning to head towards the Holy Grail of continuous deployment and delivery.

1.1.1 Wait … what’s infrastructure?

The first generation of these IaC tools, like Puppet and Chef, focused heavily on configuring operating systems and applications. They did branch into building infrastructure but generally in a limited form. You could often build the operating system’s settings and application configuration for complex applications but not the hosts, services, and networking that they were built on.

Infrastructure is the building blocks upon which you construct your applications. The dividing line between what’s infrastructure and what isn’t is blurry in many cases. Generally the data center services (or Cloud services in other environments), hosts, virtual machines or Docker containers, networking (including routing, switching, and firewalls), and storage were all traditionally considered infrastructure. In many modern environments, infrastructure also includes more complex services or Software-as-a-Service (SaaS) products delivered by third parties such as DNS, Content Delivery Networks (CDN), databases, job scheduling, queues, and monitoring.

To fully realize the promise of IaC, we’re starting to see tools that cater for these building blocks and integrate with some of the existing tools to bridge into managing operating systems and applications. In this book we’re going to see one of those tools: Terraform.

1.2 Introducing Terraform

Terraform is an “Infrastructure as Code” tool for building and managing infrastructure efficiently and elegantly. Terraform provides declarative execution plans for building and running applications and infrastructure. What does declarative mean? Declarative plans are ones in which you specify outcomes. These outcomes are high-level states, problems, or definitions of the infrastructure you want to build.

In a traditional procedural build process you specify the steps (and sub-steps) to achieve these outcomes:

  • Create host using VM management tool.
    • Launch VM management tool.
    • Click on “New VM”.
    • …
  • Install operating system using PXE boot.
  • Configure networking using ifconfig.

In a declarative tool you declare the required end state:

  • A virtual machine with two CPUs and 1GB of RAM.
  • Ubuntu 16.04 installation.
  • Two network interfaces with IP addresses 10.0.0.1/24 and 10.0.1.1/24.

You let the tool take care of the steps required and focus on the outcome. This means you no longer need to worry about what specific commands to run, buttons to push, or settings to tweak. The tool underneath takes care of those details.

This is how Terraform operates: you rely on it to take care of the details while you articulate the end state you want. You define the list of resources and configuration required for your infrastructure. Terraform examines each resource and uses a graph-based approach to model and apply the desired state. Each resource is placed inside the graph, its relationships with other resources are calculated, and then each resource is automatically built in the correct order to produce your infrastructure. This means you never have to think about modeling these complex relationships. Terraform does it for you.

You can also import existing resources to get a jump start on your configuration. As your infrastructure grows and changes you can perform incremental updates and changes.

Terraform also integrates with configuration management and provisioning tools to allow you to control application and service level components. You can easily leverage and reuse existing configuration management content you may already have that defines your applications and infrastructure.

Right now, Terraform is primarily used to manage Cloud-based and Software-as-a-Service (SaaS) infrastructure but also supports various on-premise resources such as Docker and VMWare vSphere. Terraform can manage hosts, compute resources, data and storage, networking, as well as SaaS services like CDNs, monitoring tools, and DNS.

Terraform is written by the team at HashiCorp. It is open source and licensed under the Mozilla Public License, version 2.0. HashiCorp also sells an enterprise version of Terraform.

1.3 So why not a configuration management tool?

Using Terraform versus configuration management is not an either/or. It’s more “Why not just a configuration management tool?” As we discussed above, configuration management tools—like Puppet, Chef, and Ansible—are excellent at managing applications and services, but they’re often focused on software configuration rather than building the infrastructure components that underpin our applications. Terraform is instead focused on building and deploying infrastructure.

This makes Terraform a very complementary tool in most organizations when building and deploying applications and infrastructure. We recommend using Terraform to build your infrastructure and then using your existing configuration management tools, with their pre-existing configuration stores, to configure your applications.

1.4 So why not CloudFormation et al?

Many Cloud tools and SaaS services come with their own bootstrapping tools. An example is AWS CloudFormation, which performs a similar function to Terraform, exclusively for AWS. Why not use one of these tools instead of Terraform? There are two big downsides to these tools.

First, they (generally) only work for their own resources. This can lead to a vendor lock-in, where you are so tightly tied to the vendor’s tool that you can’t use or migrate to other vendors’ products. Vendor lock-in isn’t always terrible. But it should be a choice that you consciously make, not something accidentally get locked into because you chose a specific vendor’s tool.

Second, also because they just provision their own infrastructure, these tools alone don’t necessarily make your environment fully functional. Some of these tools are best-of-breed for managing their own resources and don’t need to extend beyond that. In some cases this means you may have a substantial coverage of your infrastructure. In others, you have to glue multiple tools together or write your own tool to manage disparate resources.

In comparison, Terraform provides a holistic solution that allows you to skip building that tool and limits the amount of infrastructure glue you need to create to manage resources from different services.

1.5 So what can you use Terraform for?

There are a wide variety of potential use cases for Terraform.

1.5.1 Multi-tier applications

Terraform is focused on infrastructure deployment and supports a diverse collection of components including compute, storage, and networking assets. This makes it ideal to build N-tier applications. It’s ideal for classic web applications with database back ends right up to multi-tier applications with web, cache, application, middleware, and database tiers. Each tier can be configured in Terraform, independent and isolated, and because it’s all articulated in code, they can be easily scaled and managed by changing that code.

1.5.2 Self-service infrastructure

Often operations teams aspire to being infrastructure coordinators and controllers rather than bottlenecks in the infrastructure provisioning and management process. In this world, operations provides scalable and cost-effective self-service infrastructure to their customers.

The operations team can then focus on providing quality and value to consumers while managing risk through standardizing resources. Terraform enables operations folks to configure those standardized resources, creating standard stacks and packages that can then be provided to consumers as self-service infrastructure.

1.5.3 Production, development, and testing environments

A common problem in complex environments is that development and testing environments do not always reflect the state of the production environments. With Terraform, as your infrastructure is now codified, it’s easy to ensure that your production configuration can be shared with your development and testing environments to ensure consistency and compatibility.

We can also start to move away from the idea of fixed environments, where production is always the same environment. Instead we can see infrastructure and environments like we see application code: as versioned and iterative. Instead of being locked to a single conception of “production,” we can deploy a new environment that can evolve into or become a new “production.”

1.5.4 Continuous delivery

If you’re operating in an environment where applications are packaged in artifacts and your application configuration is done with configuration management tools, then adding Terraform allows you to also codify your infrastructure. This means all of the pieces of your environment are now automatically deployable, scalable, and manageable. You can chain together your infrastructure, your configuration management, and your build artifacts to produce continuous delivery pipelines.

1.5.5 Managing your management tools

In addition to managing Cloud-based resources, Terraform also configures many of the tools and infrastructure components you use to manage your infrastructure. Terraform can configure components like GitHub organizations and repositories, PagerDuty alerts, Grafana monitoring consoles, and DataDog metrics. This means you can not only use Terraform to build your infrastructure, but you can also start to codify the configuration of your management tools. Want to spin up a new environment? With Terraform you not only get the compute, network, and storage components, but also the management framework around it.

1.6 What’s in the book?

This is a hands-on introduction to Terraform. We’ll use Terraform to build actual infrastructure and application stacks. The book will take you through Terraform basics into more advanced features and examples. It’s an introductory book designed for folks who have never used Terraform, so if you’re already a Terraform guru then some of the material may not be of interest.

Let’s look at what’s in each chapter.

  • Chapter 1: This introduction.
  • Chapter 2: Installation & Basics. Shows you how to install Terraform and introduces you to using it.
  • Chapter 3: Building an infrastructure stack with Terraform. Takes you through building an initial infrastructure stack with Terraform.
  • Chapter 4: Provisioning with Terraform. Shows you how you can provision applications on resources deployed with Terraform, including integrating with existing configuration management tools.
  • Chapter 5: Collaborating with Terraform. Introduces you to collaborating and sharing Terraform configuration.
  • Chapter 6: Building a multi-environment architecture. Demonstrates an architecture and workflow that allows you to develop and manage infrastructure and safely make change to that infrastructure.
  • Chapter 7: Testing infrastructure changes. Takes your architecture and shows you how to write tests to validate it and how to use those tests and workflow to safely manage change in your infrastructure.
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
18.191.202.72