If we just count stars on GitHub, then Consul is probably the most popular HashiCorp tool. It's one of the few service discovery tools on the market and probably the only one that can be considered modern. The closest alternatives are Zookeeper and etcd. Zookeeper is known for being hard to set up and maintain and for being kind of too slow (written in Java) and old already (and you don't want to use anything older than few years these days, do you?). The etcd, though it's a very popular choice, lacks most of the features required for service discovery, and it's rather just a storage than a complete solution.

Consul is both service discovery tool and a key/value storage. You need to install Consul agent on each node in order to get it working. As Consul is written in Go, the installation is not very complex: drop the binary and few configuration files to the server and configure system service (systemd, upstart or whatever you prefer) to run it. No other prerequisites are required. In addition, you need to have multiple Consul masters: each Consul agent will join the Consul cluster by connecting to each.

After you have a cluster running, you can define services on your nodes via simple JSON configuration. For each service, there could be health checks, making sure that, for example, database is still available at a certain port. There is also a way to trigger events inside Consul cluster, there is an API to access Consul, and, most importantly, all of this is baked by key/value store and gossip protocol (provided by Serf--backbone of Consul and also a tool from HashiCorp). All these things together allow you to build systems that react to changes very fast and that are easily discoverable. Consul can even replace internal DNS because it has a DNS server built in.

Consul might not be the most sophisticated data storage, especially for huge amounts of data. But it is perfect for small sets of configuration, such as aforementioned services and health checks. You can easily write your own data to this storage, via API, for example.

If you are using both Terraform and Consul, it makes a lot of sense to connect them together. There is a Consul provider for Terraform that allows you to create Consul services, nodes, keys, and so on. More importantly, it has a data source consul_keys. With this data source, you can fetch any data from your Consul cluster, which means that you get a real data backend for your templates. It's up to you what you are going to store there and how you are going to use it, but here are few examples:

We are not going to set up the complete Consul cluster as it is really out of scope of this book. Nevertheless, let's look at a small example of using consul_keys to set the AMI instead of using data source for AMI:

provider "consul" { 
    address = "consul.example.com:80" 
    datacenter = "frankfurt" 
} 
data "consul_keys" "amis" { 
    # Read the launch AMI from Consul 
    key { 
        name = "mighty_trousers" 
        path = "ami" 
    } 
} 
resource "aws_instance" "app-server" { 
  ami = "${consul_keys.amis.var.mighty_trousers}" 
  instance_type = "${lookup(var.instance_type, var.environment)}" 
  subnet_id = "${var.subnet_id}" 
  vpc_security_group_ids = ["${concat(var.extra_sgs, aws_security_group.allow_http.*.id)}"] 
  user_data = "${data.template_file.user_data.rendered}" 
  tags { 
    Name = "${var.name}" 
  } 
} 

Consul is a powerful data backend for Terraform and, being also a product of HashiCorp, they are more likely to work nicely together. Consul can also be used as storage for state files, which we will discuss in later chapters. Other data backends might (and probably will) appear in future, but we should seriously consider Consul at least because of it being the part of the same technology stack.