Enabling CloudWatch Logs for Docker with Terraform
by Pierre Pomes, Stephane Jourdan
Infrastructure as Code (IAC) Cookbook
- Infrastructure as Code (IAC) Cookbook
- Table of Contents
- Infrastructure as Code (IAC) Cookbook
- Credits
- About the Authors
- About the Reviewer
- www.PacktPub.com
- Customer Feedback
- Preface
- 1. Vagrant Development Environments
- Introduction
- Adding an Ubuntu Xenial (16.04 LTS) Vagrant box
- Using a disposable Ubuntu Xenial (16.04) in seconds
- Enabling VirtualBox Guest Additions in Vagrant
- Using a disposable CentOS 7.x with VMware in seconds
- Extending the VMware VM capabilities
- Enabling multiprovider Vagrant environments
- Customizing a Vagrant VM
- Using Docker with Vagrant
- Using Docker in Vagrant for a Ghost blog behind NGINX
- Using Vagrant remotely with AWS EC2 and Docker
- Simulating dynamic multiple host networking
- Simulating a networked three-tier architecture app with Vagrant
- Showing your work on the LAN while working with Laravel
- Sharing access to your Vagrant environment with the world
- Simulating Chef upgrades using Vagrant
- Using Ansible with Vagrant to create a Docker host
- Using Docker containers on CoreOS with Vagrant
- 2. Provisioning IaaS with Terraform
- Introduction
- Configuring the Terraform AWS provider
- Creating and using an SSH key pair to use on AWS
- Using AWS security groups with Terraform
- Creating an Ubuntu EC2 instance with Terraform
- Generating meaningful outputs with Terraform
- Using contextual defaults with Terraform
- Managing S3 storage with Terraform
- Creating private Docker repositories with Terraform
- Creating a PostgreSQL RDS database with Terraform
- Enabling CloudWatch Logs for Docker with Terraform
- Managing IAM users with Terraform
- 3. Going Further with Terraform
- Introduction
- Handling different environments with Terraform
- Provisioning a CentOS 7 EC2 instance with Chef using Terraform
- Using data sources, templates, and local execution
- Executing remote commands at bootstrap using Terraform
- Using Docker with Terraform
- Simulating infrastructure changes using Terraform
- Teamwork – sharing Terraform infrastructure state
- Maintaining a clean and standardized Terraform code
- One Makefile to rule them all
- Team workflow example
- Managing GitHub with Terraform
- External monitoring integration with StatusCake
- 4. Automating Complete Infrastructures with Terraform
- Introduction
- Provisioning a complete CoreOS infrastructure on Digital Ocean with Terraform
- Provisioning a three-tier infrastructure on Google Compute Engine
- Provisioning a GitLab CE + CI runners on OpenStack
- Managing Heroku apps and add-ons using Terraform
- Creating a scalable Docker Swarm cluster on bare metal with Packet
- 5. Provisioning the Last Mile with Cloud-Init
- Introduction
- Using cloud-init on AWS, Digital Ocean, or OpenStack
- Handling files using cloud-init
- Configuring the server's time zone using cloud-init
- Managing users, keys, and credentials using cloud-init
- Managing repositories and packages using cloud-init
- Running commands during boot using cloud-init
- Configuring CoreOS using cloud-init
- Deploying Chef Client from start to finish using cloud-init
- Deploying a remote Docker server using cloud-init
- 6. Fundamentals of Managing Servers with Chef and Puppet
- Introduction
- Getting started (notions and tools)
- Installing the Chef Development kit and Puppet Collections
- Creating a free hosted server Chef account and a Puppet server
- Automatically bootstrapping a Chef client and a Puppet agent
- Installing packages
- Managing services
- Managing files, directories, and templates
- Handling dependencies
- More dynamic code using notifications
- Centrally sharing data using a Chef data bag and Hiera with Puppet
- Creating functional roles
- Managing external Chef cookbooks and Puppet modules
- 7. Testing and Writing Better Infrastructure Code with Chef and Puppet
- Introduction
- Linting Chef code with Foodcritic and Puppet code with puppet-lint
- Unit testing with ChefSpec and rspec-puppet
- Getting ready
- How to do it…
- The Spec Helper
- Testing a successful Chef run context
- Testing a package installation
- Testing services status
- Testing another recipe from the same cookbook
- Testing directory creation
- Testing file creation
- Testing templates creation
- Stubbing data bags for searches
- Testing recipes inclusion
- Intercepting errors in tests
- There's more…
- See also
- Testing infrastructure with Test Kitchen for Chef and Beaker for Puppet
- Integration testing with ServerSpec
- 8. Maintaining Systems Using Chef and Puppet
- Introduction
- Maintaining consistent systems using scheduled convergence
- Creating environments
- Using Chef encrypted data bags and Hiera-eyaml with Puppet
- Using Chef Vault encryption
- Accessing and manipulating system information with Ohai
- Automating application deployment (a WordPress example)
- Using a TDD workflow
- Planning for the worse – train to rebuild working systems
- 9. Working with Docker
- Introduction
- Docker usage overview
- Choosing the right Docker base image
- Getting ready
- How to do it…
- Starting from an Ubuntu image
- Starting from a CentOS image
- Starting from a Red Hat Enterprise Linux (RHEL) image
- Starting from a Fedora image
- Starting from an Alpine Linux image
- Starting from a Debian image
- Linux distributions container image size table
- Starting from a Node JS image
- Starting from a Golang image
- Starting from a Ruby image
- Starting from a Python image
- Starting from a Java image
- Starting from a PHP image
- See also
- Optimizing the Docker image size
- Versioning Docker images with tags
- Deploying a Ruby-on-Rails web application in Docker
- Building and using Golang applications with Docker
- Networking with Docker
- Creating more dynamic containers
- Auto-configuring dynamic containers
- Better security with unprivileged users
- Orchestrating with Docker Compose
- Linting a Dockerfile
- Deploying a private Docker registry with S3 storage
- 10. Maintaining Docker Containers
- Introduction
- Testing Docker containers with BATS
- Test-Driven Development (TDD) with Docker and ServerSpec
- The workflow for creating automated Docker builds from Git
- The workflow for connecting the Continuous Integration (CI) system
- Scanning for vulnerabilities with Quay.io and Docker Cloud
- Sending Docker logs to AWS CloudWatch logs
- Monitoring and getting information out of Docker
- Debugging containers using sysdig
- Index
CloudWatch Logs is a log aggregation service by Amazon you can use to send your logs to. It's very useful to keep some logs centralized, share access to them, receive alarms when errors happen, or simply store them safely. We'll see how to create a CloudWatch Log group and use it to stream logs from a Docker container logs inside it.
To step through this recipe, you will need the following:
- A working Terraform installation
- An AWS provider configured in Terraform (refer to the previous recipes)
- An Internet connection
- A Docker Engine running on Linux for the optional usage demonstration
Let's say we want the log group to be named docker_logs, and that we want to keep those logs for seven days. In the variables.tf file, that would look like this:
variable "log_group_name" {
default = "docker_logs"
}
variable "log_retention_days" {
default = "7"
}Also, in a new cloudwatch.tf file, we can use the simple aws_cloudwatch_log_group resource:
resource "aws_cloudwatch_log_group" "docker_logs" {
name = "${var.log_group_name}"
retention_in_days = "${var.log_retention_days}"
}After a terraform apply, if you navigate to the AWS CloudWatch page, you'll see the newly created group under the Log Groups entry on the left (https://eu-west-1.console.aws.amazon.com/cloudwatch/).
You can now use this group to create a log stream from an application or a container. Using it as recommended by AWS is well documented, so let's use it with Docker instead. It only requires to give the Docker daemon access to the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables (configuring the Docker daemon is out of the scope of this chapter, but that's under /etc/sysconfig/docker for Red Hat-based systems such as Fedora or CentOS, and /etc/default/docker for Debian/Ubuntu systems). Restart the daemon and start logging your containers output using a new Docker logging driver, using the log group name specified in Terraform earlier (docker_logs):
$ docker run -it --rm -p 80:80 --log-driver=awslogs --log-opt awslogs-region=eu-west-1 --log-opt awslogs-group=docker_logs --log-opt awslogs-stream=nginx nginx:stable
Generate some activity on the container:
$ curl -IL http://localhost HTTP/1.1 200 OK
Refresh the AWS CloudWatch page and you'll see a new entry named nginx with the container logs. You can run all your containers in your infrastructure like this and get centralized logging very easily!
-
No Comment