As you make more services, you will notice that manual deployments of changes from source control and builds are taking up more time due to the need to figure out which image dependencies belong where, which image actually needs rebuilding (if you run a mono-repo), if the service changed at all, and many other ancillary issues. In order to simplify and streamline our deployment process, we will need to find a way to make this whole system fully automated so that the only thing needed to deploy a new version of services is a commit of a change to a branch of your code repository.

As of today, the most popular automation server called Jenkins is generally used in such function to do this build automation and deployment of Docker images and infrastructure but others like Drone, Buildbot, Concoure, etc have been rising fast through the ranks of very capable software CI/CD tooling too but none have so far reached the same acceptance levels from the industry yet. Since Jenkins is also relatively easy to use, we can do a quick demonstration of its power, and while the example is a bit simplistic, it should make it obvious on how this can be used for much more.

Since Jenkins will need awscli, Ansible, and python-boto, we have to make a new Docker image based on the Jenkins that is available from Docker Hub. Create a new folder and add a Dockerfile with the following content in it:

FROM jenkins

USER root
RUN apt-get update && 
    apt-get install -y ansible 
                       awscli 
                       python-boto

USER jenkins

Now we build and run our server:

$ # Let's build our image
$ docker build -t jenkins_with_ansible

Sending build context to Docker daemon 2.048kB
Step 1/4 : FROM jenkins
<snip>
Successfully tagged jenkins_with_ansible:latest

$ # Run Jenkins with a local volume for the configuration
$ mkdir jenkins_files
$ docker run -p 8080:8080 
             -v $(pwd)/jenkins_files:/var/jenkins_home 
             jenkins_with_ansible

Running from: /usr/share/jenkins/jenkins.war
<snip>
Jenkins initial setup is required. An admin user has been created and a password generated.
Please use the following password to proceed to installation:

3af5d45c2bf04fffb88e97ec3e92127a

This may also be found at: /var/jenkins_home/secrets/initialAdminPassword
<snip>
INFO: Jenkins is fully up and running

While it is still running, let's go to the main page and enter the installation password that we got a warning for during the image start. Go to http://localhost:8080 and enter the password that was in your logs:

Click on Install Suggested Plugins on the next window and then after the relevant downloads are finished, select Continue as admin on the last installer page, which should lead you to the main landing page:

Click on create new jobs, name it redeploy_infrastructure, and make it a Freestyle project:

Next, we will configure the job with our Git repository endpoint so that it builds on any commits to the master branch:

As our build step, when the repository trigger activates, we will destroy and deploy the infrastructure, effectively replacing it with a newer version. Add a new build step of Execute Shell type and add the following to it:

# Export needed AWS credentials
export AWS_DEFAULT_REGION="us-west-1"
export AWS_ACCESS_KEY_ID="AKIABCDEFABCDEF"
export AWS_SECRET_ACCESS_KEY="123456789ABCDEF123456789ABCDEF"

# Change to relevant directory
cd chapter_8/aws_deployment

# Redeploy the service by cleaning up the old deployment
# and deploying a new one
ansible-playbook destroy.yml
ansible-playbook deploy.yml

The job should look quite a bit similar to this:

Save the changes with Save, which should take you to the build's main page. Here, click on the Build Now button and once the build appears on the left side build list, click on its progress bar or the dropdown  next to its name and select View Log:

Success! As you can see, with Jenkins and a small configuration, we just made an automated deployment of our simple infrastructure. It is crude but effective though normally you would not want to redeploy everything but just the pieces that have changed and have the Jenkins live in-cluster, but that are somewhat more-involved endeavors that will be left to the reader as possible points of improvement.