Chapter 1: Getting to Know IaC

In this chapter, we are going to discuss what Infrastructure as Code (IaC) is in detail. We will be discussing the benefits of using IaC. Furthermore, we will start describing the basics of Terraform, and then undertake a comparison of Terraform with other available IaC options, including AWS CloudFormation, Azure Resource Manager (ARM), and Google Cloud Deployment Manager. Moving on, we will then discuss Terraform architecture.

Throughout this chapter, we will be focusing on Terraform for major cloud providers, including GCP, AWS, and Azure. The whole chapter will help you in terms of achieving a fair understanding of Terraform (IaC) and how you can overcome the old lego system of executing manual changes to your environment. This will help you get an idea of how you can start using Terraform for infrastructure automation in your organization.

The following topics will be covered in this chapter:

• Introduction to IaC
• Advantages of IaC
• Introduction to Terraform
• A comparison with other IaC
• An understanding of Terraform architecture

Technical requirements

To follow along with this chapter, you need to have a basic knowledge of different IaC options, including ARM templates and AWS CloudFormation, while a basic knowledge of major cloud providers, such as GCP, AWS, and Azure, will be beneficial.

Introduction to IaC

Before learning about Terraform, let's try to understand what it basically means for us and how it helps make users' lives easier in the IT industry. The first thing that always comes to consumers' minds is that when they need an IT infrastructure, for example, if they want a virtual machine, they need to raise a ticket on some ticket portal such as ServiceNow, and someone from the backend would log in to that ticketing portal and take that ticket from the queue and deploy the virtual machine for the consumer, either in VMware or a HyperV environment through the management portal using some click jobs. That is the traditional approach for infrastructure deployment, which is somewhat fine if they need to manage infrastructure in their private data center and there is very little possibility of performing scaling of those deployed resources, which means once it gets provisioned, after that no one is requesting further changes to the created resource.

In all these cases, it is fine if they easily go ahead and perform all the operations manually but what about if they need to deploy a very large infrastructure consisting of more repeatable work in the cloud? Then it would be a really tedious job for the administrator to provision those resources manually and also it is a very time-consuming job for them. So, to overcome this challenging situation, most cloud vendors have come up with an approach of IaC, which is mostly an API-driven approach. Most cloud vendors have published APIs for all their resources. Using that API, we can easily get the resource deployed in the cloud.

Nowadays, as most customers are moving toward the cloud, and as we all know, cloud platforms provide us with more elasticity and scalability in terms of their infrastructure, this means you can easily utilize the resources and pay for what you use; you don't need to pay anything extra. Just think down the line of an administrator needing to perform the scaling up and down of resources and how difficult it would be for them. Let's suppose there are 1,000 resources that need to be scaled up during the day and scaled down at night.

In this case, consumers need to raise 1,000 tickets for performing the scale-up and again 1,000 more tickets for scaling down, which means by the end of the day, the system administrator who is managing the infrastructure will get flooded with so many requests and it would be really impossible for them to handle this. So, here we have something called IaC, which is a way of deploying or managing the infrastructure in an automated way. All the resources that need to be managed will be defined in code format and we can keep that code in any source control repository, such as GitHub or Bitbucket. Later, we can apply a DevOps approach to manage our infrastructure easily. There are many advantages of using IaC; we are going to discuss a few of them.

Advantages of IaC

Let's discuss a few of the advantages of using IaC.

Simple and speedy

Using IaC, you would able to spin up a complete infrastructure architecture by simply running a script.

Suppose you need to deploy infrastructure in multiple environments, such as development, test, preproduction, and production. It would be very easy for you to provision it with just a single click. Not only this, but say you need to deploy the same sort of infrastructure environments in other regions where your cloud provider supports backup and disaster recovery.

You can do all this by writing simple lines of code.

Configuration consistency

The classical approach of infrastructure deployment is very ugly because the infrastructure is provisioned and managed using a manual approach that helps to maintain some consistency in the infrastructure deployment process. But it always introduces human error, which makes it difficult to perform any sort of debugging.

IaC standardizes the process of building and managing the infrastructure so that the possibility of any errors or deviations is reduced. Definitely, this will decrease any incompatibility issues with the infrastructure and it will help you to run your application smoothly.

Risk minimization

When we were managing infrastructure manually, it was observed that only a handful of Subject Matter Experts (SMEs) knew how to do it and the rest of the team members remained blank, which introduces dependency and security risks for the organization. Just think of a situation where the person who is responsible for managing the complete infrastructure leaves the organization; that means whatever they knew they might not have shared with others or they may not have updated the documents. At the end of the day, risk has been introduced to the organization because that employee is leaving. Many times, in such cases, the organization needs to undergo some reverse engineering to fix any issues.

This challenging situation can be controlled by using IaC for the infrastructure. IaC will not only automate the process, but it also serves as a form of documentation for your infrastructure and provides insurance to the company in cases where employees leave the company with institutional knowledge. As you know, we generally keep IaC to source control tools such as Azure Repos or GitHub. So, if anyone makes any configuration changes to the infrastructure, they will get recorded in the source control repository. So, if anyone leaves or goes on vacation, it won't impact the manageability of the infrastructure because the version control tool will have kept track of the changes that have been performed on the infrastructure and this would definitely reduce the risk of failure.

Increased efficiency in software development

Nowadays, with the involvement of IaC for infrastructure provisioning and managing, developers get more time to focus on productivity. Whenever they need to launch their sandbox environments to develop their code, they are easily able to do so. The quality analyst (QA) will be able to have a copy of the code and test it in separate environments. Similarly, security and user acceptance testing can also be done in different staging environments. With a single click, both the application code and infrastructure code can be done together following Continuous Integration and Continuous Deployment (CI/CD) techniques.

Even if we want to get rid of any infrastructure, we can include an IaC script that will spin down the environments when they're not in use. This will shut down all the resources that were created by the script. So, we won't end up performing a cleanup of the orphan resources that are not in use. All this would help to increase the efficiency of the engineering team.

Cost savings

IaC would definitely help to save costs for the company. As we mentioned earlier, IaC script can help us create and automate the infrastructure deployment process, which allows engineers to stop doing manual work and start spending more time in performing more value-added tasks for the company, and because of this, the company can save money in terms of the hiring process and engineers' salaries.

As we mentioned earlier, IaC script can automatically tear down environments when they're not in use, which will further save companies cloud computing costs.

After getting a fair understanding of IaC and the benefits of using it, let's move ahead and try to learn some details about one IaC technology – Terraform.

Introduction to Terraform

Welcome to this introductory guide to Terraform. For anyone who is new to Terraform and unaware of what it is, as well as for the purpose of comparison with other IaC tools that are currently associated with major cloud providers including AWS, Azure, and Google, we believe that this is the best guide to begin with. In this guide, we will be focusing on what Terraform is and what problems it can solve for you, undertaking a comparison with other software tools, including ARM templates, AWS CloudFormation, and Google Cloud Deployment Manager, and explaining how you can start using Terraform effectively in your day-to-day jobs related to the provisioning and maintenance of your IT infrastructure.

What is Terraform?

Terraform is one of the open source tools that was introduced to the market by HashiCorp in 2014 as IaC software (IaC means we can write code for our infrastructure) that is mainly used for building, changing, and managing infrastructure safely and efficiently. Terraform can help with multi-cloud environments by having a single workflow, in other words, terraform init, terraform plan, terraform apply, and so on, for all clouds. The infrastructure that Terraform manages can be hosted on public clouds such as AWS, Microsoft Azure, and GCP, or on-premises in private clouds such as VMware vSphere, OpenStack, or CloudStack. Terraform handles IaC, so you never have to worry about your infrastructure drifting away from its desired configuration.

Terraform mainly uses Terraform files ending with .tf or .tf.json that hold detailed information about what infrastructure components are required in order to run a single application or your entire data center. Terraform generates an execution plan, which describes what it is going to do to reach the desired state, and then executes it to build the infrastructure described. If there is any change in the configuration file, Terraform is able to determine what has been changed and create incremental execution plans that can be applied.

Terraform can not only manage low-level components, such as compute instances, storage, and networking; it can also support high-level components, such as DNS and SaaS features, provided that the resource API is available from the providers.

After learning about what Terraform is, you might have one more question in your mind: what exactly makes this Terraform so popular? To answer that question, first and foremost, Terraform is cloud-agnostic, which means you can provision or manage your infrastructure in any cloud platform. The second thing that makes Terraform very much in demand is its standard workflow. You don't need to remember N number of parts of the workflow; a simple init, plan, and apply from Terraform's point of view would be enough and it is the same across any platform. The third factor is the Terraform syntaxing. Terraform uses uniform code syntaxing whether you work on any cloud or on-premises. There are many more exceptional factors that could encourage enterprise customers to start using Terraform.

Features of Terraform

Let's now try to get an understanding of all of Terraform's features, which are pushing up market demand for the product.

Infrastructure as code

Infrastructure is defined in a code format based on proper syntax in a configuration file that can be shared and reused. The code defined in the configuration file will provide a blueprint of your data center or the resource that you are planning to deploy. You should be able to deploy a complete infrastructure from that configuration file following the Terraform workflow.

Execution plans

The Terraform workflow has three steps – init, plan, and apply. During the planning step, it generates an execution plan. The execution plan gives you information about what Terraform will do when you call apply. This means you do not get any sort of surprise when you perform terraform apply.

Note

We are going to cover in detail the Terraform workflow in the upcoming chapters. So, stay tuned.

Resource graph

Terraform builds a graph of all your resources and parallelizes the creation and modification of any non-dependent resources. Because of this resource graph, Terraform manages to build infrastructure as efficiently as possible that is sufficiently intelligent to understand dependencies in its infrastructure.

Changing automation

Complex changes to your defined infrastructure can be applied with minimal human interaction. With the aforementioned execution plan and resource graph, you know exactly what Terraform will change and in what order, thereby avoiding multiple possible human errors.

Terraform use cases

As we have got to know what Terraform is, let's now learn about some of the use cases of Terraform in the enterprise world. A few of them have been discussed as follows.

Heroku app setup

Heroku is one of the most popular Platforms as a Service (PaaS) for hosting web apps. Developers use it to create an app and then attach other services, such as a database or email provider. One of the best features of the Heroku app is its ability to elastically scale the number of dynos or workers. However, most non-trivial applications quickly require many add-ons and external services:

Figure 1.1 – Heroku app architecture

By using Terraform, entire things that are required for a Heroku application setup could be codified in a configuration file, thereby ensuring that all the required add-ons are available, and the best part of this is that with the help of Terraform, all of this can be achieved in just 60 seconds. Any changes requested by the developer in the Heroku app can be actioned immediately using Terraform, whether it be a complex task related to configuring DNS to set a CNAME or setting up Cloudflare as a content delivery network (CDN) for the app, and so on and so forth.

Multi-tier applications

N-tier architecture deployment is quite common across the industry when thinking about the required infrastructure for an application. Generally, two-tier architecture is more in demand. This is a pool of web servers and a database tier. As per the application requirements, additional tiers can be added for API servers, caching servers, routing meshes, and so on. This pattern is used because each tier can be scaled independently and without disturbing other tiers:

Figure 1.2 – N-tier application architecture

Now, let's try to understand how Terraform can support us in achieving N-tier application infrastructure deployment. In the Terraform configuration file, each tier can be described as a collection of resources, and the dependencies between the resources for each tier can either be implicit or we can define them explicitly so that we can easily control the sequence of the resource deployment. This helps us to manage each tier separately without disturbing the others.

Self-service clusters

In a large organization, it's quite challenging for the central operation team to provide infrastructure to the product team as and when needed. The product team should be able to create and maintain their infrastructure using tooling provided by the central operations team:

Figure 1.3 – Self-service cluster

In the preceding requirement, the entire infrastructure can be codified using Terraform, which will focus on building and scaling the infrastructure, and a Terraform configuration file can be shared within an organization, enabling product teams to use the configuration as a black box and use Terraform as a tool to manage their services. During deployment of the infrastructure, if the product team encounters any issues, they can reach out to the central operations team for help and support.

Software demos

Nowadays, software development is increasing by the day, and it is very difficult to get the infrastructure required to test that software. We have tools such as Vagrant at our disposal to help us build virtualized environments, and while you may be able to use that environment for demonstration purposes, it is really difficult to perform software demos directly on production infrastructure:

Figure 1.4 – Software demo example

A software developer can provide a Terraform configuration to create, provision, and bootstrap a demo on cloud providers such as Azure, GCP, and AWS. This allows end users to easily demo the software on their infrastructure, and it even allows them to perform scale-in or scale-out of the infrastructure.

Disposable environments

In the industry, it is quite common to have multiple landscapes, including production, staging, or development environments. These environments are generally designed as a subset of the production environment, so as and when any application needs to be deployed and tested, it can easily be done in the smaller environment; but the problem with the increase in complexity of the infrastructure is that it's very difficult to manage it:

Figure 1.5 – Multiple environments

Using Terraform, the production environment that you constructed can be written in a code format, and then it can be shared with other environments, such as staging, QA, or dev. This configuration code can be used to spin up any new environments to perform testing, and can then be easily removed when you are done testing. Terraform can help to maintain a parallel environment and it can provide an option in terms of its scalability.

Software-defined networking

Software-Defined Networking (SDN) is quite famous in data centers, as it allows operators to operate a software-defined network very smoothly and developers are able to develop their applications, which can easily be run on top of the network infrastructure provided. The control layer and infrastructure layer are the two main components for defining a software-defined network:

Figure 1.6 – Software-defined network

Software-defined networks can be transformed into code using Terraform. The configuration code written in Terraform can automatically set up and modify settings by interfacing with the control layer. This allows the configuration to be versioned and changes to be automated. As an example, Azure Virtual Network is one of the most commonly used SDN implementations and can be configured by Terraform.

Resource schedulers

In large-scale infrastructures, the static assignment of applications to machines is very challenging. In terms of Terraform, there are many schedulers available, such as Borg, Mesos, YARN, and Kubernetes, that can be used to overcome this problem. These can be used to dynamically schedule Docker containers, Hadoop, Spark, and many other software tools:

Figure 1.7 – Kubernetes with Terraform

Terraform is not just limited to cloud providers such as Azure, GCP, and AWS. Resource schedulers can also behave as providers, enabling Terraform to request resources from them. This allows Terraform to be used in layers, to set up the physical infrastructure running the schedulers, as well as provisioning them on the scheduled grid. There is a Kubernetes provider that can be configured using Terraform to schedule any Pod deployment. You can read about Kubernetes with Terraform at https://learn.hashicorp.com/collections/terraform/kubernetes.

Multi-cloud deployment

Nowadays, every organization is moving toward multi-cloud, and one of the challenging tasks is to deploy the entire infrastructure in a different cloud. Every cloud provider has its own defined manner of deployment, such as ARM templates for Azure or AWS CloudFormation. Hence, it is very difficult for an administrator to learn about all of these while maintaining the complexity of the environment deployment:

Figure 1.8 – Multi-cloud deployment

Realizing the complexity of multi-cloud infrastructure deployments using already-existing tools that are very specific to each cloud provider, HashiCorp came up with an approach known as Terraform. Terraform is cloud-agnostic. A single configuration can be used to manage multiple providers, and it can even handle cross-cloud dependencies. This simplifies management and orchestration, helping administrators to handle large-scale, multi-cloud infrastructures.

So far, we have covered IaC, namely, Terraform, its features, and the different use case scenarios where we can apply Terraform. Furthermore, we have covered how Terraform differs from other IaCs mainly used in the major cloud providers, including AWS, Azure, and Google.

A comparison with other IaC

In the preceding section, you got to know about Terraform, and similar to Terraform, there are many other IaC options that are more specific to individual cloud providers, such as ARM templates for the Azure cloud, CloudFormation for AWS, and Cloud Deployment Manager for GCP. Likewise, each provider has come up with their own IaC that is used for their infrastructure provisioning. Now, the challenge for an administrator or developer is to learn and remember a vast number of different template syntaxes. To overcome this challenge, Terraform came up with a solution involving common workflows and syntax that allows operators to operate complex infrastructure. Let's try to understand how Terraform is different in terms of being cross-platform, as well as its modularity, language of code, validation, readability, maintainability, workflow, error management, state management, and so on for major cloud providers including AWS, Azure, and GCP.

CloudFormation versus Terraform

In this section, we will see how Terraform differs from CloudFormation based on various parameters:

Figure 1.9 – CloudFormation versus Terraform

Let's get started.

Cross-platform

A major benefit of Terraform is that you can use it with a variety of cloud providers, including AWS, GCP, and Azure. CloudFormation templates are more centric to AWS. So here, Terraform would be more preferred than AWS CloudFormation.

Language

Terraform is written in HashiCorp Configuration Language (HCL). HCL's syntax is very powerful and allows you to reference variables, attributes, and so on, whereas CloudFormation uses either JSON or YAML, which are bare notation languages and are not as powerful as HCL. With CloudFormation, you can also reference parameters, such as other stacks, but overall, we think HCL makes Terraform more productive.

On the other hand, CloudFormation has good support for various conditional functionalities. Terraform has the count, for_each, and for loops, which make certain things easy (such as creating N identical resources) and certain things a bit harder (such as the if-else type conditional structure; for example, count = var.create_eip == true ? 1: 0). CloudFormation has also a wait condition and creation policy, which can be important in certain deployment situations where you may have to wait before a certain condition is satisfied.

In terms of language, Terraform is simple and easy to draft, which would encourage developers and administrators to start exploring Terraform, but it practically depends upon the use case scenario, which might in some cases require you to use CloudFormation for AWS resource deployment and manageability.

Modularity

Modularity is generally defined as how much easier it is for you to implement your infrastructure code using Terraform or CloudFormation. Terraform's modules can be easily written and reused as infrastructure code in multiple projects by multiple teams. Hence, it is quite easy to modularize Terraform code. You have the option to modularize CloudFormation code using nested stacks. You cannot modularize your CloudFormation stack itself in the same way you can prepare stacks of stacks in Terraform. You can use cross-stack references in CloudFormation and in the same way, it can be referenced in Terraform using the terraform_remote_state data source, which is used for deriving root module output from the other Terraform configuration.

Comparing both AWS CloudFormation and Terraform in terms of modularity, we can see that Terraform seems to be more friendly and easy to use.

Validation

Both tools allow you to validate infrastructure, but there is a difference. Using CloudFormation, you can validate AWS CloudFormation stacks, which means it will check only the syntax of your stack. If you want to update some resources, you can use CloudFormation change sets, which let you review the changes, meaning whether that specific resource defined in AWS CloudFormation will get replaced or updated, before you actually execute AWS CloudFormation stacks.

In the Terraform workflow, there is a plan phase that provides complete information about what resources are getting modified, deleted, or created before you deploy the infrastructure code.

In AWS CloudFormation, you have CloudFormation Designer, which can help you to know all the resources you have defined in AWS CloudFormation. This will provide you with insight before you actually go ahead and perform the deployment.

In terms of validation, both Terraform and AWS CloudFormation have certain options to perform it. The only major difference that can be noticed is that in AWS CloudFormation, you would be required to use multiple AWS services whereas in Terraform, we are easily able to validate with just its workflow.

Readability

There is no doubt that when you write a Terraform configuration file you will find it easier compared to AWS CloudFormation. Terraform configuration files are written in HCL, meaning they are more presentable when you understand the syntax. When you think about CloudFormation, which is generally written in JSON or YAML, it is quite complex in terms of its readability, especially JSON; YAML is somewhat fine to read and understand but the problem with YAML is regarding its whitespace and linting. In Terraform also, there is the linting concern but Terraform has a smart way of performing linting by just running terraform fmt -recursive, which will perform linting to all the Terraform configuration files present in the directory, whereas in YAML, you have to perform something manually or use some YAML validator tool that can help you with the validations.

So, in a nutshell, Terraform has more preferred than to AWS CloudFormation.

License and support

AWS CloudFormation is a managed service from AWS offered for free whereas Terraform is an open source infrastructure automation tool provided by HashiCorp. There is the Terraform Cloud Enterprise product from HashiCorp that is a licensed version and Enterprise customers need to purchase it from HashiCorp.

In terms of support plans, AWS has their own support plan that can be taken from AWS, in the same way that Terraform has their support option that can used by customers.

AWS console support

As AWS CloudFormation is a native AWS IaC tool, it naturally provides excellent support in the AWS console. In the AWS console, you can monitor the progress of deployments, see all deployment events, check various errors, integrate CloudFormation with CloudWatch, and so on. In the Terraform CLI, we don't have any console option, but in the Terraform Cloud Enterprise version, there is a console where you can see all the progress of the resource deployments.

Workflow

You must be thinking that AWS CloudFormation will be more mature from an operational perspective, and this is true to a certain extent, but here you need to understand the pain area. Let's take an example where you have a complex infrastructure of around 50 resources and you are creating it using AWS CloudFormation. After running for around 20-30 minutes, your AWS CloudFormation stack receives an error in one of the resources and because of that, the stack fails. In that case, it may roll back and destroy whatever it had created in AWS or sometimes, it may be able to deploy some of the resources even though it failed provided you have disabled rollback on failure in AWS CloudFormation. For further reading on the rollback on failure option in AWS CloudFormation, you can refer to https://aws.amazon.com/premiumsupport/knowledge-center/cloudformation-prevent-rollback-failure/. But to be very honest, if it did roll back and destroyed what it had created, then as an operator, you won't be looking for something like this because this adds more time and effort. Now, when you provision your AWS infrastructure using the Terraform workflow, this will provision all the required infrastructure in AWS. Even though, let's suppose, the Terraform workflow failed, then it will maintain the existing resources that had already been created prior to the failure. You can re-run the Terraform configuration, and this will simply provision the remaining resources rather than creating all the resources from scratch, which will save a significant amount of time and effort from the point of view of reworking.

So, based on the preceding paragraph, it appears that both AWS CloudFormation and Terraform workflows can be used. It's totally depends upon the consumer and which one they prefer.

Error message understandability

Errors thrown by Terraform can be tough for a novice to understand because sometimes they provide a strange error that is generated from the respective provider's APIs. In order to have an understanding of all different sorts of errors, you need to have an understanding of the arguments that are supported by the providers in Terraform.

So, comparing the understandability of error messages between Terraform and AWS CloudFormation, on occasion you may find that AWS CloudFormation is straightforward as you will have experience of working with the latter, and then, with your knowledge of Terraform, you will easily be able to figure out most of the errors.

Infrastructure state management

Both AWS CloudFormation and Terraform maintain the state of the infrastructure in their own way. In Terraform, state files get stored in the local disk or you can provide a remote backend location such as AWS S3 where you wish to store your Terraform state file. This state file would be written in JSON and would hold the information of the infrastructure that is defined in the Terraform configuration file. So, it is quite easy for Terraform to validate and check any sort of configuration drift.

In contrast, AWS CloudFormation is native to AWS, so we don't need to worry about how AWS CloudFormation would maintain its state. AWS CloudFormation used to get a signal from the resources that got provisioned or managed through AWS CloudFormation. If there is any configuration drift, then AWS CloudFormation may not be able to detect it.

Validation

Both tools allow you to have validation of infrastructure, but there is a difference. Using the Azure CLI or PowerShell, you can validate a template by running az group deployment validate, which means it will only check the syntax of your template. To validate what all the resources are that get provisioned, you have to deploy the complete ARM template.

Terraform provides a validation with the plan phase that checks the current deployment in the Terraform state file, refreshes the status of the actual cloud configuration, and then calculates the delta between the current configuration and the target configuration.

In terms of validation, if you were to compare both, then here too, Terraform wins out over ARM templates.

Azure ARM templates versus Terraform

In this section, we will see how Terraform differs from ARM based on various parameters:

Figure 1.10 – ARM templates versus Terraform

Let's get started.

Cross-platform

The major benefit of Terraform is that you can use it with many cloud providers, including AWS, GCP, and Azure. The ARM template is more central to Azure whereas for Terraform, you just need to learn the HCL language syntax and Terraform workflow and start using any cloud provider, such as GCP, Azure, or AWS. So from this, we can understand that Terraform is cloud-agnostic and it would be easy for the user to implement infrastructure on any of these platforms using Terraform.

Language

Terraform is written in HCL. HCL's syntax is very powerful and allows you to reference variables, attributes, and so on, whereas ARM templates use JSON and is a little bit complex but very powerful, because you can use conditional statements, nested templates, and so on.

Keeping all of this in mind, here too, we would encourage end users and developers to use Terraform.

Modularity

Modularity generally means how much easier for you it is to implement your infrastructure code using Terraform or ARM templates. Terraform's modules can be easily written and reused as infrastructure code in multiple projects by multiple teams. Hence, it is quite easy to modularize the Terraform code.

You have the option to modularize ARM templates using a nested ARM template, although you also have the option of forming a nested template, the first one in line in your main template, and the second one using separate JSON files called from your main template.

Here, comparing both ARM templates and Terraform, Terraform scores more points than ARM templates, although in this case, ARM templates are still a viable option for end users if they are planning to create resources in Azure.

Readability

There is no doubt that when you are intending to write a Terraform configuration file, you will find it more readable compared to ARM templates. Terraform configuration files written in HCL are more presentable when you understand their syntax, and the best part is that if you have learned the syntax, you would be able to write and use it anywhere in any cloud provider, including Azure, GCP, and AWS. When you consider ARM templates, which are generally written in JSON, they are quite complex in terms of their readability.

License and support

ARM templates use an Azure-native IaC approach and are available for free, whereas Terraform is an open source tool available from HashiCorp. There is the Terraform Cloud Enterprise version of Terraform that enterprise customers can purchase. Regarding support, both Azure and Terraform have their own support plans that can be utilized by the customer.

Azure portal support

As an ARM template is a native Azure IaC tool, it naturally provides excellent support in the Azure portal. You can monitor deployment progress in the Azure portal.

There is no Azure portal support for Terraform but if a customer is willing to use the Terraform Cloud Enterprise product, then they can utilize a portal that shows enough information about the infrastructure that they are planning to manage through Terraform.

Workflow

In terms of workflow, both ARM templates and Terraform have very good features. If the deployment of resources is interrupted halfway through, both Terraform and ARM templates have the capability to deploy the remaining resources during the second run, ignoring existing resources.

There is one minor difference, however. ARM template deployment requires an Azure resource group in place before you run the ARM template code, whereas Terraform configuration code is able to create an Azure resource group during the runtime itself, just as you would be required to define a resource group code block in the configuration file. Hence, it seems that both ARM templates and Terraform are quite satisfactory in terms of workflow.

Error message understandability

Terraform code can generate some weird errors. There is no doubt that in the backend, it would be using ARM provider APIs, which would show those errors, and it might be difficult for those end users who are new to Terraform to understand them, whereas in ARM templates, errors can be easily identified and fixed as end users may have experience of working with ARM templates.

So, in terms of comparing error message understandability between Terraform and ARM templates, this process may be easier in ARM templates.

Infrastructure state management

Both ARM templates and Terraform provide good state management. Terraform provides state file management using the backend state mechanism. In the state file, Terraform maps the resource that's been defined in the configuration file with what has actually been deployed so that it can easily trace any configuration drift. ARM is able to roll back to a previous successful deployment. You may have read that ARM is stateless as it does not store any state files.

Google Cloud Deployment Manager versus Terraform

In this section, we will see how Terraform differs from Google Cloud Deployment Manager based on various parameters:

Figure 1.11 – Cloud Deployment Manager versus Terraform

Let's get started.

Cross-platform

As you have already seen a comparison of AWS and Azure, you should now have an understanding of Terraform, which can be used on any platform, including GCP too. You just need to know how you can write Terraform configuration files using HCL and their workflow.

Cloud Deployment Manager is an infrastructure deployment service that automates the creation and management of Google Cloud resources. So here, Terraform wins out over Cloud Deployment Manager.

Language

Terraform is written in HCL. HCL's syntax is very easy and powerful and it allows you to reference variables, attributes, and so on, whereas Cloud Deployment Manager uses YAML to create a configuration file and you can define multiple templates in the configuration file, which is written using Jinja or Python.

Keeping all of this in mind, here too we would encourage end users or developers to use Terraform because in order to use cloud deployment, you would be required to have command of Python or Jinja and YAML.

Modularity

Modularity means how easily you can define configuration files and reuse them for infrastructure provisioning and management using Terraform or Cloud Deployment Manager. Terraform's modules can be written and published easily, and these modules can easily be consumed by other product teams in their projects. So, modularizing Terraform code is quite straightforward.

You have the option to modularize Cloud Deployment Manager using a nested template inside a configuration file. These templates can be written in either Python or Jinja. A single configuration can import both Jinja and Python templates. These template files should be present locally or placed on a third-party URL so that Cloud Deployment Manager can access them.

Here, comparing both Cloud Deployment Manager and Terraform, Terraform scores more points than Google Cloud Deployment Manager, although Google Cloud Deployment Manager remains a viable option for end users here, too, if they are planning to create resources in Google.

Validation

Both tools allow you to validate infrastructure, but there is a difference. To figure out any syntax errors in the configuration file, you would be required to run it from Google Cloud Deployment Manager. This will throw any possible errors, as well as validating what resources get provisioned. In this case, you have to deploy a complete Google Cloud Deployment Manager configuration file in Google Cloud.

Terraform provides a validation with the plan phase that tells you what resources it is going to create, destroy, or update. It is used for comparison purposes with the existing Terraform state file and accordingly provides detailed insights to the administrator as to when they will run terraform plan before being deployed to Google Cloud.

In terms of validation, if you were to compare both, then here, too, Terraform wins out over Google Cloud Deployment Manager.

Readability

If you are intending to write a Terraform configuration file, you will find it more readable compared to a Google Cloud Deployment Manager configuration file. Terraform configuration files written in HCL are more presentable, and the best part is that if you have learned Terraform syntaxes, you would be able to write and use them anywhere in any cloud, including Google, GCP, and AWS, or even on-premises.

When you think about Cloud Deployment Manager, which is generally written in YAML, it is also easy to understand, but when you need to introduce a template in the configuration file, which is written in Jinja or Python, then it becomes complex to read and understand.

Therefore, I am definitely of the opinion that Terraform is easier to use compared with Cloud Deployment Manager.

Maintainability

At a certain level, I think Terraform is more maintainable because of its superior modularity and how you can create different resources in Google Cloud and store its state file in Google Cloud Storage.

A Cloud Deployment Manager configuration file that is written in YAML and whose template is written in Python or Jinja is not that easy to read, which means the consumer needs to learn multiple coding languages. The nested template appears very complex and you need to store the template file in localhost or a third-party URL. Defining its template code locally in the configuration file and consuming it in the configuration file is not possible.

Cloud Deployment Manager doesn't have any kind of state file whereas Terraform has a state file that is stored in the remote backend.

The Terraform code configuration can be stored in any version control tool, such as Git, Azure Repos, or Bitbucket, if we are planning to use CI/CD DevOps pipelines.

Maintaining both Terraform code and Google Cloud Deployment Manager templates would be a little bit complex because both need to be stored either locally or in the source control tools.

Google console support

As Google Cloud Deployment Manager is a native Google IaC tool used for the provisioning and maintenance of infrastructure, it provides excellent support in the Google Cloud console. You can see the progress in terms of deployment in the Google Cloud console.

There is no Google Cloud console support for Terraform, though there is a separate portal if you are using the Terraform Cloud Enterprise version.

Workflow

In terms of workflow, both Cloud Deployment Manager and Terraform have very good features. If the deployment of resources is interrupted halfway through, both Terraform and Cloud Deployment Manager have the capability to deploy the remaining resources during the second run, ignoring existing resources.

There is one minor difference, however. Cloud Deployment Manager deployment requires a project in place before you run the configuration file using Cloud Deployment Manager, whereas Terraform configuration code is able to create a Google project during the runtime itself, just as you would be required to define a Google project resource block code in the configuration file. Therefore, I feel both of them are quite satisfactory when it comes to the workflow.

Error message understandability

In the previous AWS and Azure comparisons, we mentioned that errors used to emanate from the respective providers' APIs, and learning how to handle those errors comes with experience. If you were already equipped with that knowledge, then you would easily be able to rectify those issues. Again, compared to Google Cloud Deployment Manager, Terraform code exhibits a number of difficult errors that would be difficult for you to understand if you were new to Terraform. It doesn't mean that you can easily understand Google Cloud Deployment Manager errors simply because they involve multiple coding languages, such as YAML, JSON, or Jinja.

So, in terms of comparing the understandability of error messages between Terraform and Google Cloud Deployment Manager, if you had experience with Google Cloud, then it would be instinctive for you and this will help you to understand the errors easily. Regarding handling errors with Terraform, it may take some time for you to figure it out as Google Cloud resource APIs release errors specific to that.

Infrastructure state management

Both Cloud Deployment Manager and Terraform provide good state management. Terraform provides state file management using a backend state mechanism and state files can be stored in Google Cloud Storage. Cloud Deployment Manager is capable of rolling back to a previous successful deployment, since deployment used to happen incrementally by default, and it is quite easy to implement any changes you wish to make in terms of Google resources using Cloud Deployment Manager.

We covered how Terraform differs from other IaC options, mainly in terms of being cross-platform, as well as its modularity, readability, infrastructure state management, language, maintainability, workflow, cloud console support, and so on, and how it can be used by major cloud providers such as AWS, Azure, and GCP. All these comparisons will help you to understand how Terraform is a more powerful IaC and how you can effectively choose Terraform IaC for your infrastructure provisioning rather than selecting any specific IaC. Moving on, let's try to gain an understanding of Terraform architecture.

An understanding of Terraform architecture

With the help of the preceding section, we learned and became familiar with Terraform, which is just a tool for building, changing, and versioning infrastructure safely and efficiently. Terraform is entirely built on a plugin-based architecture. Terraform plugins enable all developers to extend Terraform usage by writing new plugins or compiling modified versions of existing plugins:

Figure 1.12 – Terraform architecture

As you can see in the preceding Terraform architecture, there are two key components on which Terraform's workings depend: Terraform Core and Terraform plugins. Terraform Core uses Remote Procedure Calls (RPCs) to communicate with Terraform plugins and offers multiple ways to discover and load plugins to use. Terraform plugins expose an implementation for a specific service, such as AWS, or a provisioner, and so on.

Terraform Core

Terraform Core is a statically compiled binary written in the Go programming language. It uses RPCs to communicate with Terraform plugins and offers multiple ways to discover and load plugins for use. The compiled binary is the Terraform CLI. If you're interested in learning more about this, you should start your journey from the Terraform CLI, which is the only entry point. The code is open source and hosted at github.com/hashicorp/Terraform.

The responsibilities of Terraform Core are as follows:

• IaC: Reading and interpolating configuration files and modules
• Resource state management
• Resource graph construction
• Plan execution
• Communication with plugins via RPC

Terraform plugins

Terraform plugins are written in the Go programming language and are executable binaries that get invoked by Terraform Core via RPCs. Each plugin exposes an implementation for a specific service, such as AWS, or a provisioner, such as Bash. All providers and provisioners are plugins that are defined in the Terraform configuration file. Both are executed as separate processes and communicate with the main Terraform binary via an RPC interface. Terraform has many built-in provisioners, while providers are added dynamically as and when required. Terraform Core provides a high-level framework that abstracts away the details of plugin discovery and RPC communication, so that developers do not need to manage either.

Terraform plugins are responsible for the domain-specific implementation of their type.

The responsibilities of provider plugins are as follows:

• Initialization of any included libraries used to make API calls
• Authentication with the infrastructure provider
• The definition of resources that map to specific services

The responsibilities of provisioner plugins are as follows:

• Executing commands or scripts on the designated resource following creation or destruction

Plugin locations

By default, whenever you run the terraform init command, it will be looking for the plugins in the directories listed in the following table. Some of these directories are static, while some are relative to the current working directory:

You can visit the following link for more information on plugin locations:

https://www.terraform.io/docs/extend/how-terraform-works.html#plugin-locations

Important note

<OS> and <ARCH> use the Go language's standard OS and architecture names, for example, darwin_amd64.

Third-party plugins should usually be installed in the user plugins directory, which is located at ~/.terraform.d/plugins on most OSes and %APPDATA% erraform.dplugins on Windows.

If you are running terraform init with the -plugin-dir=<PATH> option (with a non-empty <PATH>), this will override the default plugin locations and search only the path that you had specified.

Provider and provisioner plugins can be installed in the same directories. Provider plugin binaries are named with the scheme terraform-provider-<NAME>_vX.Y.Z, while provisioner plugins use the scheme terraform-provisioner-<NAME>_vX.Y.Z. Terraform relies on filenames to determine plugin types, names, and versions.

Selecting plugins

After finding any installed plugins, terraform init compares them to the configuration's version constraints and chooses a version for each plugin as defined here:

• If there are any acceptable versions of the plugin that have already been installed, Terraform uses the newest installed version that meets the constraint (even if releases.hashicorp.com has a newer acceptable version).
• If no acceptable versions of plugins have been installed and the plugin is one of the providers distributed by HashiCorp, Terraform downloads the newest acceptable version from releases.hashicorp.com and saves it in .terraform/plugins/<OS>_<ARCH>.

  This step is skipped if terraform init is run with the -plugin-dir=<PATH> or -get-plugins=false options.

• If no acceptable versions of plugins have been installed and the plugin is not distributed by HashiCorp, then the initialization fails and the user must manually install an appropriate version.

Upgrading plugins

When you run terraform init with the -upgrade option, it rechecks releases.hashicorp.com for newer acceptable provider versions and downloads the latest version if available.

This will only work in the case of providers whose only acceptable versions are in .terraform/plugins/<OS>_<ARCH> (the automatic downloads directory); if any acceptable version of a given provider is installed elsewhere, terraform init -upgrade will not download a newer version of the plugin.

We have now covered Terraform architecture and learned about its core components, in other words, Terraform Core and Terraform plugins, and how Terraform Core communicates with Terraform plugins (provisioners and providers) using RPCs. In conjunction with this, you now have an understanding of the different sources from where Terraform will attempt to download plugins, because without plugins, you will not be able to use Terraform.

Summary

With the help of this chapter, you will now have a fair understanding of what Terraform is. Terraform is an IaC orchestration tool introduced by HashiCorp that is mainly used for the provisioning of infrastructure for your applications in code format. It is written in HCL, which is easily readable and understood, and we have also examined a number of different use cases, including multi-tier applications, SDNs, multi-cloud deployment, and resource schedulers, as well as examples of where you can use Terraform.

Moving on, you got to understand how Terraform differs from other IaCs, such as with ARM templates and CloudFormation, in terms of language, readability, modularity, error handling, and suchlike. Later, we explained Terraform architecture, and from there we got to know about how Terraform Core is used to communicate with Terraform plugins using RPCs, and by using Terraform Core, we would be able to run different Terraform CLI commands, including init, plan, and apply.

In the next chapter, we are going to discuss how you can install terraform.exe on a different machine, such as Windows, macOS, and Linux, which will help you in getting started with Terraform.

Questions

The answers to these questions can be found in the Assessments section at the end of this book:

1. What do you understand the definition of Terraform to be?

   a) It is a virtual box.

   b) It is an orchestration tool that is used for the provisioning of infrastructure.

   c) It is a cloud.

   d) It is a Google Chrome extension.

2. Which of the following are Terraform plugins? Select one or more:

   a) Terraform providers

   b) Terraform provisioners

   c) Terraform resources

   d) Terraform plan

3. A Terraform configuration file is written in which language?

   a) Python

   b) HCL

   c) YAML

   d) Go

4. Which of the following is not a Terraform provider?

   a) Azure

   b) AWS

   c) GCP

   d) SAP

5. Terraform is an orchestration tool developed by which company?

   a) Microsoft

   b) HashiCorp

   c) Amazon

   d) Google

Further reading

You can check out the following links for more information about the topics covered in this chapter:

• What is Terraform? https://www.Terraform.io/intro/index.html
• Terraform use cases: https://www.Terraform.io/intro/use-cases.html
• How Terraform works: https://www.Terraform.io/docs/extend/how-Terraform-works.html
• Terraform releases: https://github.com/hashicorp/Terraform/releases