We'll use Jenkins EC2 plugin (https://plugins.jenkins.io/ec2) to create agent nodes when needed and destroy them after a period of inactivity. The plugin is already installed. However, we'll need to configure it to use a specific Amazon Machine Image (AMI), so creating one is our first order of business.

Before we proceed, please make sure that the environment variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_DEFAULT_REGION are set. If you followed the instructions for setting up the cluster with kops, the environment variables are already defined in source cluster/kops.

We'll build the image with Packer (https://www.packer.io/), so please make sure that it is installed in your laptop.

Packer definition we'll explore soon will require a security group. Please execute the command that follows to create one.

 1  export AWS_DEFAULT_REGION=us-east-2
 2
 3  aws ec2 create-security-group 
 4      --description "For building Docker images" 
 5      --group-name docker 
 6      | tee cluster/sg.json

For convenience, we'll parse the output stored in cluster/sg.json to retrieve GroupId and assign it in an environment variable. Please install jq (https://stedolan.github.io/jq/) if you don't have it already.

 1  SG_ID=$(cat cluster/sg.json 
 2      | jq -r ".GroupId")
 3
 4  echo $SG_ID

The output of the latter command should be similar to the one that follows.

sg-5fe96935

Next, we'll store the security group export in a file so that we can easily retrieve it in the next chapters.

 1  echo "export SG_ID=$SG_ID" 
 2      | tee -a cluster/docker-ec2

The security group we created is useless in its current form. We'll need to authorize it to allow communication on port 22 so that Packer can access it and execute provisioning.

 1  aws ec2 
 2      authorize-security-group-ingress 
 3      --group-name docker 
 4      --protocol tcp 
 5      --port 22 
 6      --cidr 0.0.0.0/0

We're done with the preparation steps and we can proceed to create the AMI.

Let's take a quick look at the Package definition we'll use.

 1  cat jenkins/docker-ami.json

The output is as follows.

{
  "builders": [{
    "type": "amazon-ebs",
    "region": "us-east-2",
    "source_ami_filter": {
      "filters": {
        "virtualization-type": "hvm",
        "name": "*ubuntu-xenial-16.04-amd64-server-*",
        "root-device-type": "ebs"
      },
      "most_recent": true
    },
    "instance_type": "t2.micro",
    "ssh_username": "ubuntu",
    "ami_name": "docker",
    "force_deregister": true
  }],
  "provisioners": [{
    "type": "shell",
    "inline": [
      "sleep 15",
      "sudo apt-get clean",
      "sudo apt-get update",
      "sudo apt-get install -y apt-transport-https ca-certificates nfs-common",
      "curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -",
      "sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"",
      "sudo add-apt-repository -y ppa:openjdk-r/ppa",
      "sudo apt-get update",
      "sudo apt-get install -y docker-ce",
      "sudo usermod -aG docker ubuntu",
      "sudo apt-get install -y openjdk-8-jdk"
    ]
  }]
}

Most of the definition should be self-explanatory. We'll create an EBS image based on Ubuntu in the us-east-2 region and we'll use the shell provisioner to install Docker and JDK.

Let's create the AMI.

 1  packer build -machine-readable 
 2      jenkins/docker-ami.json 
 3      | tee cluster/docker-ami.log

The last lines of the output are as follows.

...
...,amazon-ebs,artifact,0,id,us-east-2:ami-ea053b8f
...,amazon-ebs,artifact,0,string,AMIs were created:
us-east-2: ami-ea053b8f

...,amazon-ebs,artifact,0,files-count,0
...,amazon-ebs,artifact,0,end
...,,ui,say,--> amazon-ebs: AMIs were created:
us-east-2: ami-ea053b8f

The important line is the one that contains artifact,0,id. The last column in that row contains the ID we'll need to tell Jenkins about the new AMI. We'll store it in an environment variable for convenience.

 1  AMI_ID=$(grep 'artifact,0,id' 
 2      cluster/docker-ami.log 
 3      | cut -d: -f2)
 4
 5  echo $AMI_ID

The output of the latter command should be similar to the one that follows.

ami-ea053b8f

Just as with the security group, we'll store the AMI_ID export in the docker-ec2 file so that we can retrieve it easily in the next chapters.

 1  echo "export AMI_ID=$AMI_ID" 
 2      | tee -a cluster/docker-ec2

Now that we have the AMI, we need to move to Jenkins and configure the Amazon EC2 plugin.

 1  open "http://$JENKINS_ADDR/configure"

Please scroll to the Cloud section and click the Add a new cloud drop-down list. Choose Amazon EC2. A new form will appear.

Type docker-agents as the Name, and expand the Add drop-down list next to Amazon EC2 Credentials. Choose Jenkins.

From the credentials screen, please choose AWS Credentials as the Kind, and type aws as both the ID and the Description.

Next, we need to return to the Terminal to retrieve the AWS access key ID.

 1  echo $AWS_ACCESS_KEY_ID

Please copy the output, return to Jenkins UI, and paste it into the Access Key ID field.

We'll repeat the same process for the AWS secrets access key.

 1  echo $AWS_SECRET_ACCESS_KEY

Copy the output, return to Jenkins UI, and paste it into the Secret Access Key field.

With the credentials information filled in, we need to press the Add button to store it and return to the EC2 configuration screen.

Please choose the newly created credentials and select us-east-2 as the Region.

We need the private key next. It can be created through the aws ec2 command create-key-pair.

 1  aws ec2 create-key-pair 
 2     --key-name devops24 
 3      | jq -r '.KeyMaterial' 
 4      >cluster/devops24.pem

We created a new key pair, filtered the output so that only the KeyMaterial is returned, and stored it in the devops24.pem file.

For security reasons, we should change the permissions of the devops24.pem file so that only the current user can read it.

 1  chmod 400 cluster/devops24.pem

Finally, we'll output the content of the pem file.

 1  cat cluster/devops24.pem

Please copy the output, return to Jenkins UI, and paste it into the EC2 Key Pair's Private Key field.

To be on the safe side, press the Test Connection button, and confirm that the output is Success.

We're finished with the general Amazon EC2 configuration, and we can proceed to add the first and the only AMI.

Please click the Add button next to AMIs, and type docker as the Description.

We need to return to the Terminal one more time to retrieve the AMI ID.

 1  echo $AMI_ID

Please copy the output, return to Jenkins UI, and paste it into the AMI ID field.

To be on the safe side, please click the Check AMI button, and confirm that the output does not show any error.

We're almost done.

Select T2Micro as the Instance Type, type docker as the Security group names, and type ubuntu as the Remote user. The Remote ssh port should be set to 22. Please write docker ubuntu linux as the labels, and change the Idle termination time to 10.

Finally, click the Save button to preserve the changes.

We'll use the newly created EC2 template soon. Feel free to skip the next section if this was the way you're planning to create agents for building container images.