CloudWatch provides an agent that can be configured with different filesystem sources to collect, aggregate and stream log files to CloudWatch Logs. The first step is to create an IAM policy that will provide the access required to write log streams into CloudWatch Log groups.
- Navigate to IAM and choose Policies | Create policy:
- In the Create policy menu use the JSON editor and paste the following IAM Policy: https://github.com/gabanox/Certified-Solution-Architect-Associate-Guide/blob/master/chapter21/CloudWatchAgentPolicy.txt.
- Review and name the policy CloudWatchAgentPolicy and click on CreatePolicy.
- Our example will use a file to configure the CWL agent. Create an S3 bucket, and upload this configuration file. Make sure to replace my-bucket with your own bucket name and make it readable (you can use IAM roles).
- Now navigate to IAM Roles and choose Create Role. Choose EC2 Role and click Next: Permissions.
- Filter the IAM policy using the name CloudWatchAgentPolicy. Use the check button and proceed to the next step.
- You will be prompted to give the role a name, use CloudWatchAgentRole and make sure the IAM Policy is attached.
- Now launch an EC2 instance using the IAM Role created (CloudWatchAgentRole).
- Provide the following user data as initialization for your EC2 Instance. This data can be downloaded from https://github.com/gabanox/Certified-Solution-Architect-Associate-Guide/blob/master/chapter21/bootstrap.txt.
#!/bin/bash
curl https://s3.amazonaws.com//aws-cloudwatch/downloads/latest/awslogs-agent-setup.py -O
chmod +x ./awslogs-agent-setup.py
./awslogs-agent-setup.py -n -r us-east-1 -c s3://cloudwatch-bucket-config-file/my-config-file
The agent will be downloaded locally, and the Python script will be configured to use the my-config-file provided. When the service becomes active it will start streaming syslog data to CloudWatch as shown in the following screenshot:
Congratulations you have created log data ingestion solution that stores activity information from instances in a centralized way. You'll want to create an AMI with this configuration and use it as your baseline.