What Is a Docker Container?

A Docker container encapsulates an application together with other software packages and libraries it requires to run. People sometimes use the terms Docker container and Docker image interchangeably, but there is a difference. The filesystem-level object that encapsulates the application is called a Docker image. When you run the image, it becomes a Docker container.

You can run many Docker containers, all using the same OS kernel. The only requirement is that you must install a server-side component called the Docker engine or the Docker daemon on the host where you want to run the containers. In this way, the host resources can be split and utilized in a more granular way across the containers, giving you more bang for your buck.

Docker containers provide more isolation and resource control than regular Linux processes, but provide less than full-fledged virtual machines would. To achieve these properties of isolation and resource control, the Docker engine makes use of Linux kernel features such as namespaces, control groups (or cgroups), and Union File Systems (UnionFS).

The main advantage of Docker containers is portability. Once you create a Docker image, you can run it as a Docker container on any host OS where the Docker server-side daemon is available. These days, all the major operating systems run the Docker daemon: Linux, Windows, and macOS.

All this can sound too theoretical, so it is time for some concrete examples.

Creating, Building, Running, and Removing Docker Images and Containers

Since this is a book on Python and DevOps, we will take the canonical Flask “Hello World” as the first example of an application that runs in a Docker container. The examples shown in this section use the Docker for Mac package. Subsequent sections will show how to install Docker on Linux.

Here is the main file of the Flask application:

$ cat app.py
from flask import Flask
app = Flask(__name__)

@app.route('/')
def hello_world():
    return 'Hello, World! (from a Docker container)'

if __name__ == '__main__':
    app.run(debug=True, host='0.0.0.0')

We also need a requirements file that specifies the version of the Flask package to be installed with pip:

$ cat requirements.txt
Flask==1.0.2

Trying to run the app.py file directly with Python on a macOS laptop without first installing the requirements results in an error:

$ python app.py
Traceback (most recent call last):
  File "app.py", line 1, in <module>
    from flask import Flask
ImportError: No module named flask

One obvious way to get past this issue is to install the requirements with pip on your local machine. This would make everything specific to the operating system you are running locally. What if the application needs to be deployed on a server running a different OS? The well-known issue of “works on my machine” could arise, where everything works beautifully on a macOS laptop, but for some mysterious reason, usually related to OS-specific versions of Python libraries, everything breaks on the staging or production servers running other operating systems, such as Ubuntu or Red Hat Linux.

Docker offers an elegant solution to this conundrum. We can still do our development locally, using our beloved editors and toolchains, but we package our application’s dependencies inside a portable Docker container.

Here is the Dockerfile describing the Docker image that is going to be built:

$ cat Dockerfile
FROM python:3.7.3-alpine

ENV APP_HOME /app
WORKDIR $APP_HOME

COPY requirements.txt .

RUN pip install -r requirements.txt

ENTRYPOINT [ "python" ]
CMD [ "app.py" ]

A few notes about this Dockerfile:

• Use a prebuilt Docker image for Python 3.7.3 based on the Alpine distribution that produces slimmer Docker images; this Docker image already contains executables such as python and pip.

• Install the required packages with pip.

• Specify an ENTRYPOINT and a CMD. The difference between the two is that when the Docker container runs the image built from this Dockerfile, the program it runs is the ENTRYPOINT, followed by any arguments specified in CMD; in this case, it will run python app.py.

To create the Docker image for this application, run docker build:

$ docker build -t hello-world-docker .

To verify that the Docker image was saved locally, run docker images followed by the name of the image:

$ docker images hello-world-docker
REPOSITORY               TAG       IMAGE ID            CREATED          SIZE
hello-world-docker       latest    dbd84c229002        2 minutes ago    97.7MB

To run the Docker image as a Docker container, use the docker run command:

$ docker run --rm -d -v `pwd`:/app -p 5000:5000 hello-world-docker
c879295baa26d9dff1473460bab810cbf6071c53183890232971d1b473910602

A few notes about the docker run command arguments:

• The --rm argument tells the Docker server to remove this container once it stops running. This is useful to prevent old containers from clogging the local filesystem.

• The -d argument tells the Docker server to run this container in the background.

• The -v argument specifies that the current directory (pwd) is mapped to the /app directory inside the Docker container. This is essential for the local development workflow we want to achieve because it enables us to edit the application files locally and have them be auto-reloaded by the Flask development server running inside the container.

• The -p 5000:5000 argument maps the first port (5000) locally to the second port (5000) inside the container.

To list running containers, run docker ps and note the container ID because it will be used in other docker commands:

$ docker ps
CONTAINER ID  IMAGE                      COMMAND         CREATED
c879295baa26  hello-world-docker:latest  "python app.py" 4 seconds ago
STATUS          PORTS                    NAMES
Up 2 seconds    0.0.0.0:5000->5000/tcp   flamboyant_germain

To inspect the logs for a given container, run docker logs and specify the container name or ID:

$ docker logs c879295baa26
 * Serving Flask app "app" (lazy loading)
 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 647-161-014

Hit the endpoint URL with curl to verify that the application works. Because port 5000 of the application running inside the Docker container was mapped to port 5000 on the local machine with the -p command-line flag, you can use the local IP address 127.0.0.1 with port 5000 as the endpoint for the application.

$ curl http://127.0.0.1:5000
Hello, World! (from a Docker container)%

Now modify the code in app.py with your favorite editor. Change the greeting text to Hello, World! (from a Docker container with modified code). Save app.py and notice lines similar to these in the Docker container logs:

 * Detected change in '/app/app.py', reloading
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 647-161-014

This shows that the Flask development server running inside the container has detected the change in app.py and has reloaded the application.

Hitting the application endpoint with curl will show the modified greeting:

$ curl http://127.0.0.1:5000
Hello, World! (from a Docker container with modified code)%

To stop a running container, run docker stop or docker kill and specify the container ID as the argument:

$ docker stop c879295baa26
c879295baa26

To delete a Docker image from local disk, run docker rmi:

$ docker rmi hello-world-docker
Untagged: hello-world-docker:latest
Deleted:sha256:dbd84c229002950550334224b4b42aba948ce450320a4d8388fa253348126402
Deleted:sha256:6a8f3db7658520a1654cc6abee8eafb463a72ddc3aa25f35ac0c5b1eccdf75cd
Deleted:sha256:aee7c3304ef6ff620956850e0b6e6b1a5a5828b58334c1b82b1a1c21afa8651f
Deleted:sha256:dca8a433d31fa06ab72af63ae23952ff27b702186de8cbea51cdea579f9221e8
Deleted:sha256:cb9d58c66b63059f39d2e70f05916fe466e5c99af919b425aa602091c943d424
Deleted:sha256:f0534bdca48bfded3c772c67489f139d1cab72d44a19c5972ed2cd09151564c1

This output shows the different filesystem layers comprising a Docker image. When the image is removed, the layers are deleted as well. Consult the Docker storage drivers documentation for more details on how Docker uses filesystem layers to build its images.

Publishing Docker Images to a Docker Registry

Once you have a Docker image built locally, you can publish it to what is called a Docker registry. There are several public registries to choose from, and for this example we will use Docker Hub. The purpose of these registries is to allow people and organizations to share pre-built Docker images that can be reused across different machines and operating systems.

First, create a free account on Docker Hub and then create a repository, either public or private. We created a private repository called flask-hello-world under our griggheo Docker Hub account.

Then, at the command line, run docker login and specify the email and password for your account. At this point, you can interact with Docker Hub via the docker client.

When building the Docker image in the previous section, it was automatically tagged as latest, and the repository was set to the name of the image, signifying that the image is local:

$ docker images hello-world-docker
REPOSITORY               TAG       IMAGE ID            CREATED          SIZE
hello-world-docker       latest    dbd84c229002        2 minutes ago    97.7MB

To tag a Docker image, run docker tag:

$ docker tag hello-world-docker hello-world-docker:v1

Now you can see both tags for the hello-world-docker image:

$ docker images hello-world-docker
REPOSITORY               TAG      IMAGE ID           CREATED          SIZE
hello-world-docker       latest   dbd84c229002       2 minutes ago    97.7MB
hello-world-docker       v1       89bd38cb198f       42 seconds ago   97.7MB

Before you can publish the hello-world-docker image to Docker Hub, you also need to tag it with the Docker Hub repository name, which contains your username or your organization name. In our case, this repository is griggheo/hello-world-docker:

$ docker tag hello-world-docker:latest griggheo/hello-world-docker:latest
$ docker tag hello-world-docker:v1 griggheo/hello-world-docker:v1

Publish both image tags to Docker Hub with docker push:

$ docker push griggheo/hello-world-docker:latest
$ docker push griggheo/hello-world-docker:v1

If you followed along, you should now be able to see your Docker image published with both tags to the Docker Hub repository you created under your account.

Running a Docker Container with the Same Image on a Different Host

Now that the Docker image is published to Docker Hub, we are ready to show off the portability of Docker by running a container based on the published image on a different host. The scenario considered here is that of collaborating with a colleague who doesn’t have macOS but likes to develop on a laptop running Fedora. The scenario includes checking out the application code and modifying it.

Launch an EC2 instance in AWS based on the Linux 2 AMI, which is based on RedHat/CentOS/Fedora, and then install the Docker engine. Add the default user on the EC2 Linux AMI, called ec2-user, to the docker group so it can run docker client commands:

$ sudo yum update -y
$ sudo amazon-linux-extras install docker
$ sudo service docker start
$ sudo usermod -a -G docker ec2-user

Make sure to check out the application code on the remote EC2 instance. In this case, the code consists only of app.py file.

Next, run the Docker container based on the image published to Docker Hub. The only difference is that the image used as an argument to the docker run command was griggheo/hello-world-docker:v1 instead of simply hello-world-docker.

Run docker login, then:

$ docker run --rm -d -v `pwd`:/app -p 5000:5000 griggheo/hello-world-docker:v1

Unable to find image 'griggheo/hello-world-docker:v1' locally
v1: Pulling from griggheo/hello-world-docker
921b31ab772b: Already exists
1a0c422ed526: Already exists
ec0818a7bbe4: Already exists
b53197ee35ff: Already exists
8b25717b4dbf: Already exists
d997915c3f9c: Pull complete
f1fd8d3cc5a4: Pull complete
10b64b1c3b21: Pull complete
Digest: sha256:af8b74f27a0506a0c4a30255f7ff563c9bf858735baa610fda2a2f638ccfe36d
Status: Downloaded newer image for griggheo/hello-world-docker:v1
9d67dc321ffb49e5e73a455bd80c55c5f09febc4f2d57112303d2b27c4c6da6a

Note that the Docker engine on the EC2 instance recognizes that it does not have the Docker image locally, so it downloads it from Docker Hub, then runs a container based on the newly downloaded image.

At this point, access to port 5000 was granted by adding a rule to the security group associated with the EC2 instance. Visit http://54.187.189.51:5000¹ (with 54.187.189.51 being the external IP of the EC2 instance) and see the greeting Hello, World! (from a Docker container with modified code).

When modifying the application code on the remote EC2 instance, the Flask server running inside the Docker container will auto-reload the modified code. Change the greeting to Hello, World! (from a Docker container on an EC2 Linux 2 AMI instance) and notice that the Flask server reloaded the application by inspecting the logs of the Docker container:

[ec2-user@ip-10-0-0-111 hello-world-docker]$ docker ps
CONTAINER ID  IMAGE                           COMMAND         CREATED
9d67dc321ffb  griggheo/hello-world-docker:v1  "python app.py" 3 minutes ago
STATUS        PORTS                    NAMES
Up 3 minutes  0.0.0.0:5000->5000/tcp   heuristic_roentgen

[ec2-user@ip-10-0-0-111 hello-world-docker]$ docker logs 9d67dc321ffb
 * Serving Flask app "app" (lazy loading)
 * Debug mode: on
 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 306-476-204
72.203.107.13 - - [19/Aug/2019 04:43:34] "GET / HTTP/1.1" 200 -
72.203.107.13 - - [19/Aug/2019 04:43:35] "GET /favicon.ico HTTP/1.1" 404 -
 * Detected change in '/app/app.py', reloading
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 306-476-204

Hitting http://54.187.189.51:5000² now shows the new greeting Hello, World! (from a Docker container on an EC2 Linux 2 AMI instance).

It is worth noting that we did not have to install anything related to Python or Flask to get our application to run. By simply running our application inside a container, we were able to take advantage of the portability of Docker. It is not for nothing that Docker chose the name “container” to popularize its technology—one inspiration was how the shipping container revolutionized the global transportation industry.

Running Multiple Docker Containers with Docker Compose

In this section we will use the “Flask By Example” tutorial that describes how to build a Flask application that calculates word-frequency pairs based on the text from a given URL.

Start by cloning the Flask By Example GitHub repository:

$ git clone https://github.com/realpython/flask-by-example.git

We will use compose to run multiple Docker containers representing the different parts of the example application. With Compose, you use a YAML file to define and configure the services comprising an application, then you use the docker-compose command-line utility to create, start, and stop these services that will run as Docker containers.

The first dependency to consider for the example application is PostgreSQL, as described in Part 2 of the tutorial.

Here is how to run PostgreSQL in a Docker container inside a docker-compose.yaml file:

$ cat docker-compose.yaml
version: "3"
services:
  db:
    image: "postgres:11"
    container_name: "postgres"
    ports:
      - "5432:5432"
    volumes:
      - dbdata:/var/lib/postgresql/data
volumes:
  dbdata:

A few things to note about this file:

• Define a service called db based on the postgres:11 image published on Docker Hub.

• Specify a port mapping from local port 5432 to the container port 5432.

• Specify a Docker volume for the directory where PostgreSQL stores its data, which is /var/lib/postgresql/data. This is so that the data stored in PostgreSQL will persist across restarts of the container.

The docker-compose utility is not part of the Docker engine, so it needs to be installed separately. See the official documentation for instructions on installing it on various operating systems.

To bring up the db service defined in docker-compose.yaml, run the docker-compose up -d db command, which will launch the Docker container for the db service in the background (the -d flag):

$ docker-compose up -d db
Creating postgres ... done

Inspect the logs for the db service with the docker-compose logs db command:

$ docker-compose logs db
Creating volume "flask-by-example_dbdata" with default driver
Pulling db (postgres:11)...
11: Pulling from library/postgres
Creating postgres ... done
Attaching to postgres
postgres | PostgreSQL init process complete; ready for start up.
postgres |
postgres | 2019-07-11 21:50:20.987 UTC [1]
LOG:  listening on IPv4 address "0.0.0.0", port 5432
postgres | 2019-07-11 21:50:20.987 UTC [1]
LOG:  listening on IPv6 address "::", port 5432
postgres | 2019-07-11 21:50:20.993 UTC [1]
LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres | 2019-07-11 21:50:21.009 UTC [51]
LOG:  database system was shut down at 2019-07-11 21:50:20 UTC
postgres | 2019-07-11 21:50:21.014 UTC [1]
LOG:  database system is ready to accept connections

Running docker ps shows the container running the PostgreSQL database:

$ docker ps
dCONTAINER ID   IMAGE   COMMAND    CREATED   STATUS   PORTS   NAMES
83b54ab10099 postgres:11 "docker-entrypoint.s…"  3 minutes ago  Up 3 minutes
        0.0.0.0:5432->5432/tcp   postgres

Running docker volume ls shows the dbdata Docker volume mounted for the PostgreSQL /var/lib/postgresql/data directory:

$ docker volume ls | grep dbdata
local               flask-by-example_dbdata

To connect to the PostgreSQL database running in the Docker container associated with the db service, run the command docker-compose exec db and pass it the command line psql -U postgres:

$ docker-compose exec db psql -U postgres
psql (11.4 (Debian 11.4-1.pgdg90+1))
Type "help" for help.

postgres=#

Following “Flask by Example, Part 2”, create a database called wordcount:

$ docker-compose exec db psql -U postgres
psql (11.4 (Debian 11.4-1.pgdg90+1))
Type "help" for help.

postgres=# create database wordcount;
CREATE DATABASE

postgres=# l

                            List of databases
     Name  |  Owner | Encoding |  Collate |   Ctype  |   Access privileges
-----------+--------+----------+----------+----------+--------------------
 postgres  | postgres | UTF8   | en_US.utf8 | en_US.utf8 |
 template0 | postgres | UTF8   | en_US.utf8 | en_US.utf8 | =c/postgres +
           |          |        |            |            |postgres=CTc/postgres
 template1 | postgres | UTF8   | en_US.utf8 | en_US.utf8 | =c/postgres +
           |          |        |            |            |postgres=CTc/postgres
 wordcount| postgres | UTF8| en_US.utf8 | en_US.utf8 |
(4 rows)
postgres=# q

Connect to the wordcount database and create a role called wordcount_dbadmin that will be used by the Flask application:

$ docker-compose exec db psql -U postgres wordcount
wordcount=# CREATE ROLE wordcount_dbadmin;
CREATE ROLE
wordcount=# ALTER ROLE wordcount_dbadmin LOGIN;
ALTER ROLE
wordcount=# ALTER USER wordcount_dbadmin PASSWORD 'MYPASS';
ALTER ROLE
postgres=# q

The next step is to create a Dockerfile for installing all the prerequisites for the Flask application.

Make the following modifications to the requirements.txt file:

• Modify the version of the psycopg2 package from 2.6.1 to 2.7 so that it supports PostgreSQL 11

• Modify the version of the redis package from 2.10.5 to 3.2.1 for better Python 3.7 support

• Modify the version of the rq package from 0.5.6 to 1.0 for better Python 3.7 support

Here is the Dockerfile:

$ cat Dockerfile
FROM python:3.7.3-alpine

ENV APP_HOME /app
WORKDIR $APP_HOME

COPY requirements.txt .

RUN 
 apk add --no-cache postgresql-libs && 
 apk add --no-cache --virtual .build-deps gcc musl-dev postgresql-dev && 
 python3 -m pip install -r requirements.txt --no-cache-dir && 
 apk --purge del .build-deps

COPY . .

ENTRYPOINT [ "python" ]
CMD ["app.py"]

Run docker build -t flask-by-example:v1 . to build a local Docker image. The output of this command is not shown because it is quite lengthy.

The next step in the “Flask By Example” tutorial is to run the Flask migrations.

In the docker-compose.yaml file, define a new service called migrations and specify its image, its command, its environment variables, and the fact that it depends on the db service being up and running:

$ cat docker-compose.yaml
version: "3"
services:
  migrations:
    image: "flask-by-example:v1"
    command: "manage.py db upgrade"
    environment:
      APP_SETTINGS: config.ProductionConfig
      DATABASE_URL: postgresql://wordcount_dbadmin:$DBPASS@db/wordcount
    depends_on:
      - db
  db:
    image: "postgres:11"
    container_name: "postgres"
    ports:
      - "5432:5432"
    volumes:
      - dbdata:/var/lib/postgresql/data
volumes:
  dbdata:

The DATABASE_URL variable uses the name db for the PostgreSQL database host. This is because the name db is defined as a service name in the docker-compose.yaml file, and docker-compose knows how to link one service to another by creating an overlay network where all services defined in the docker-compose.yaml file can interact with each other by their names. See the docker-compose networking reference for more details.

The DATABASE_URL variable definition refers to another variable called DBPASS, instead of hardcoding the password for the wordcount_dbadmin user. The docker-compose.yaml file is usually checked into source control, and best practices are not to commit secrets such as database credentials to GitHub. Instead, use an encryption tool such as sops to manage a secrets file.

Here is an example of how to create an encrypted file using sops with PGP encryption.

First, install gpg on macOS via brew install gpg, then generate a new PGP key with an empty passphrase:

$ gpg --generate-key
pub   rsa2048 2019-07-12 [SC] [expires: 2021-07-11]
      E14104A0890994B9AC9C9F6782C1FF5E679EFF32
uid                      pydevops <[email protected]>
sub   rsa2048 2019-07-12 [E] [expires: 2021-07-11]

Next, download sops from its release page.

To create a new encrypted file called, for example, environment.secrets, run sops with the -pgp flag and give it the fingerprint of the key generated above:

$ sops --pgp BBDE7E57E00B98B3F4FBEAF21A1EEF4263996BD0 environment.secrets

This will open the default editor and allow for the input of the plain-text secrets. In this example, the contents of the environment.secrets file are:

export DBPASS=MYPASS

After saving the environment.secrets file, inspect the file to see that it is encrypted, which makes it safe to add to source control:

$ cat environment.secrets
{
	"data": "ENC[AES256_GCM,data:qlQ5zc7e8KgGmu5goC9WmE7PP8gueBoSsmM=,
  iv:xG8BHcRfdfLpH9nUlTijBsYrh4TuSdvDqp5F+2Hqw4I=,
  tag:0OIVAm9O/UYGljGCzZerTQ==,type:str]",
	"sops": {
		"kms": null,
		"gcp_kms": null,
		"lastmodified": "2019-07-12T05:03:45Z",
		"mac": "ENC[AES256_GCM,data:wo+zPVbPbAJt9Nl23nYuWs55f68/DZJWj3pc0
    l8T2d/SbuRF6YCuOXHSHIKs1ZBpSlsjmIrPyYTqI+M4Wf7it7fnNS8b7FnclwmxJjptBWgL
    T/A1GzIKT1Vrgw9QgJ+prq+Qcrk5dPzhsOTxOoOhGRPsyN8KjkS4sGuXM=,iv:0VvSMgjF6
    ypcK+1J54fonRoI7c5whmcu3iNV8xLH02k=,
    tag:YaI7DXvvllvpJ3Talzl8lg==,
    type:str]",
		"pgp": [
			{
				"created_at": "2019-07-12T05:02:24Z",
				"enc": "-----BEGIN PGP MESSAGE-----

hQEMA+3cyc
        g5b/Hu0OvU5ONr/F0htZM2MZQSXpxoCiO
WGB5Czc8FTSlRSwu8/cOx0Ch1FwH+IdLwwL+jd
        oXVe55myuu/3OKUy7H1w/W2R
PI99Biw1m5u3ir3+9tLXmRpLWkz7+nX7FThl9QnOS25
        NRUSSxS7hNaZMcYjpXW+w
M3XeaGStgbJ9OgIp4A8YGigZQVZZFl3fAG3bm2c+TNJcAbl
        zDpc40fxlR+7LroJI
juidzyOEe49k0pq3tzqCnph5wPr3HZ1JeQmsIquf//9D509S5xH
        Sa9lkz3Y7V4KC
efzBiS8pivm55T0s+zPBPB/GWUVlqGaxRhv1TAU=
=WA4+
        
-----END PGP MESSAGE-----
",
				"fp": "E14104A0890994B9AC9C9F6782C1FF5E679EFF32"
			}
		],
		"unencrypted_suffix": "_unencrypted",
		"version": "3.0.5"
	}
}%

To decrypt the file, run:

$ sops -d environment.secrets
export DBPASS=MYPASS

$ GPG_TTY=$(tty)
$ export GPG_TTY

The goal here is to run the migrations service defined previously in the docker-compose.yaml_ file. To tie the +sops secret management method into docker-compose, decrypt the environments.secrets file with sops -d, source its contents into the current shell, then invoke docker-compose up -d migrations using one command line that will not expose the secret to the shell history:

$ source <(sops -d environment.secrets); docker-compose up -d migrations
postgres is up-to-date
Recreating flask-by-example_migrations_1 ... done

Verify that the migrations were successfully run by inspecting the database and verifying that two tables were created: alembic_version and results:

$ docker-compose exec db psql -U postgres wordcount
psql (11.4 (Debian 11.4-1.pgdg90+1))
Type "help" for help.

wordcount=# dt

                  List of relations
 Schema |      Name       | Type  |       Owner
--------+-----------------+-------+-------------------
 public | alembic_version | table | wordcount_dbadmin
 public | results         | table | wordcount_dbadmin
(2 rows)

wordcount=# q

Part 4 in the “Flask By Example” tutorial is to deploy a Python worker process based on Python RQ that talks to an instance of Redis.

First, Redis needs to run. Add it as a service called redis into the docker_compose.yaml file, and make sure that its internal port 6379 is mapped to port 6379 on the local OS:

  redis:
    image: "redis:alpine"
    ports:
      - "6379:6379"

Start the redis service on its own by specifying it as an argument to docker-compose up -d:

$ docker-compose up -d redis
Starting flask-by-example_redis_1 ... done

Run docker ps to see a new Docker container running based on the redis:alpine image:

$ docker ps
CONTAINER ID   IMAGE   COMMAND    CREATED   STATUS   PORTS   NAMES
a1555cc372d6   redis:alpine "docker-entrypoint.s…" 3 seconds ago  Up 1 second
0.0.0.0:6379->6379/tcp   flask-by-example_redis_1
83b54ab10099   postgres:11  "docker-entrypoint.s…" 22 hours ago   Up 16 hours
0.0.0.0:5432->5432/tcp   postgres

Use the docker-compose logs command to inspect the logs of the redis service:

$ docker-compose logs redis
Attaching to flask-by-example_redis_1
1:C 12 Jul 2019 20:17:12.966 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 12 Jul 2019 20:17:12.966 # Redis version=5.0.5, bits=64, commit=00000000,
modified=0, pid=1, just started
1:C 12 Jul 2019 20:17:12.966 # Warning: no config file specified, using the
default config. In order to specify a config file use
redis-server /path/to/redis.conf
1:M 12 Jul 2019 20:17:12.967 * Running mode=standalone, port=6379.
1:M 12 Jul 2019 20:17:12.967 # WARNING: The TCP backlog setting of 511 cannot
be enforced because /proc/sys/net/core/somaxconn
is set to the lower value of 128.
1:M 12 Jul 2019 20:17:12.967 # Server initialized
1:M 12 Jul 2019 20:17:12.967 * Ready to accept connections

The next step is to create a service called worker for the Python RQ worker process in docker-compose.yaml:

  worker:
    image: "flask-by-example:v1"
    command: "worker.py"
    environment:
      APP_SETTINGS: config.ProductionConfig
      DATABASE_URL: postgresql://wordcount_dbadmin:$DBPASS@db/wordcount
      REDISTOGO_URL: redis://redis:6379
    depends_on:
      - db
      - redis

Run the worker service just like the redis service, with docker-compose up -d:

$ docker-compose up -d worker
flask-by-example_redis_1 is up-to-date
Starting flask-by-example_worker_1 ... done

Running docker ps will show the worker container:

$ docker ps
CONTAINER ID   IMAGE   COMMAND    CREATED   STATUS   PORTS   NAMES
72327ab33073  flask-by-example "python worker.py"     8 minutes ago
Up 14 seconds                             flask-by-example_worker_1
b11b03a5bcc3  redis:alpine     "docker-entrypoint.s…" 15 minutes ago
Up About a minute  0.0.0.0:6379->6379/tc  flask-by-example_redis_1
83b54ab10099  postgres:11      "docker-entrypoint.s…"  23 hours ago
Up 17 hours        0.0.0.0:5432->5432/tcp postgres

Look at the worker container logs with docker-compose logs:

$ docker-compose logs worker
Attaching to flask-by-example_worker_1
20:46:34 RQ worker 'rq:worker:a66ca38275a14cac86c9b353e946a72e' started,
version 1.0
20:46:34 *** Listening on default...
20:46:34 Cleaning registries for queue: default

Now launch the main Flask application in its own container. Create a new service called app in docker-compose.yaml:

  app:
    image: "flask-by-example:v1"
    command: "manage.py runserver --host=0.0.0.0"
    ports:
      - "5000:5000"
    environment:
      APP_SETTINGS: config.ProductionConfig
      DATABASE_URL: postgresql://wordcount_dbadmin:$DBPASS@db/wordcount
      REDISTOGO_URL: redis://redis:6379
    depends_on:
      - db
      - redis

Map port 5000 from the application container (the default port for a Flask application) to port 5000 on the local machine. Pass the command manage.py runserver --host=0.0.0.0 to the application container to ensure that port 5000 is exposed correctly by the Flask application inside the container.

Start up the app service with docker compose up -d, while also running sops -d on the encrypted file containing DBPASS, then sourcing the decrypted file before calling docker-compose:

source <(sops -d environment.secrets); docker-compose up -d app
postgres is up-to-date
Recreating flask-by-example_app_1 ... done

Notice the new Docker container running the application in the list returned by docker ps:

$ docker ps
CONTAINER ID   IMAGE   COMMAND    CREATED   STATUS   PORTS   NAMES
d99168a152f1   flask-by-example "python app.py"  3 seconds ago
Up 2 seconds    0.0.0.0:5000->5000/tcp   flask-by-example_app_1
72327ab33073   flask-by-example "python worker.py" 16 minutes ago
Up 7 minutes                             flask-by-example_worker_1
b11b03a5bcc3   redis:alpine     "docker-entrypoint.s…" 23 minutes ago
Up 9 minutes    0.0.0.0:6379->6379/tcp   flask-by-example_redis_1
83b54ab10099   postgres:11      "docker-entrypoint.s…"  23 hours ago
Up 17 hours     0.0.0.0:5432->5432/tcp   postgres

Inspect the logs of the application container with docker-compose logs:

$ docker-compose logs app
Attaching to flask-by-example_app_1
app_1         |  * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)

Running docker-compose logs with no other arguments allows us to inspect the logs of all the services defined in the docker-compose.yaml file:

$ docker-compose logs
Attaching to flask-by-example_app_1,
flask-by-example_worker_1,
flask-by-example_migrations_1,
flask-by-example_redis_1,
postgres
1:C 12 Jul 2019 20:17:12.966 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 12 Jul 2019 20:17:12.966 # Redis version=5.0.5, bits=64, commit=00000000,
modified=0, pid=1, just started
1:C 12 Jul 2019 20:17:12.966 # Warning: no config file specified, using the
default config. In order to specify a config file use
redis-server /path/to/redis.conf
1:M 12 Jul 2019 20:17:12.967 * Running mode=standalone, port=6379.
1:M 12 Jul 2019 20:17:12.967 # WARNING: The TCP backlog setting of 511 cannot
be enforced because /proc/sys/net/core/somaxconn
is set to the lower value of 128.
1:M 12 Jul 2019 20:17:12.967 # Server initialized
1:M 12 Jul 2019 20:17:12.967 * Ready to accept connections
app_1         |  * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
postgres      | 2019-07-12 22:15:19.193 UTC [1]
LOG:  listening on IPv4 address "0.0.0.0", port 5432
postgres      | 2019-07-12 22:15:19.194 UTC [1]
LOG:  listening on IPv6 address "::", port 5432
postgres      | 2019-07-12 22:15:19.199 UTC [1]
LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres      | 2019-07-12 22:15:19.214 UTC [22]
LOG:  database system was shut down at 2019-07-12 22:15:09 UTC
postgres      | 2019-07-12 22:15:19.225 UTC [1]
LOG:  database system is ready to accept connections
migrations_1  | INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
migrations_1  | INFO [alembic.runtime.migration] Will assume transactional DDL.
worker_1      | 22:15:20
RQ worker 'rq:worker:2edb6a54f30a4aae8a8ca2f4a9850303' started, version 1.0
worker_1      | 22:15:20 *** Listening on default...
worker_1      | 22:15:20 Cleaning registries for queue: default

The final step is to test the application. Visit http://127.0.0.1:5000 and enter python.org in the URL field. At that point, the application sends a job to the worker process, asking it to execute the function count_and_save_words against the home page of python.org. The application periodically polls the job for the results, and upon completion, it displays the word frequencies on the home page.

To make the docker-compose.yaml file more portable, push the flask-by-example Docker image to Docker Hub, and reference the Docker Hub image in the container section for the app and worker services.

Tag the existing local Docker image flask-by-example:v1 with a name prefixed by a Docker Hub username, then push the newly tagged image to Docker Hub:

$ docker tag flask-by-example:v1 griggheo/flask-by-example:v1
$ docker push griggheo/flask-by-example:v1

Change docker-compose.yaml to reference the new Docker Hub image. Here is the final version of docker-compose.yaml:

$ cat docker-compose.yaml
version: "3"
services:
  app:
    image: "griggheo/flask-by-example:v1"
    command: "manage.py runserver --host=0.0.0.0"
    ports:
      - "5000:5000"
    environment:
      APP_SETTINGS: config.ProductionConfig
      DATABASE_URL: postgresql://wordcount_dbadmin:$DBPASS@db/wordcount
      REDISTOGO_URL: redis://redis:6379
    depends_on:
      - db
      - redis
  worker:
    image: "griggheo/flask-by-example:v1"
    command: "worker.py"
    environment:
      APP_SETTINGS: config.ProductionConfig
      DATABASE_URL: postgresql://wordcount_dbadmin:$DBPASS@db/wordcount
      REDISTOGO_URL: redis://redis:6379
    depends_on:
      - db
      - redis
  migrations:
    image: "griggheo/flask-by-example:v1"
    command: "manage.py db upgrade"
    environment:
      APP_SETTINGS: config.ProductionConfig
      DATABASE_URL: postgresql://wordcount_dbadmin:$DBPASS@db/wordcount
    depends_on:
      - db
  db:
    image: "postgres:11"
    container_name: "postgres"
    ports:
      - "5432:5432"
    volumes:
      - dbdata:/var/lib/postgresql/data
  redis:
    image: "redis:alpine"
    ports:
      - "6379:6379"
volumes:
  dbdata:

To restart the local Docker containers, run docker-compose down followed by docker-compose up -d:

$ docker-compose down
Stopping flask-by-example_worker_1 ... done
Stopping flask-by-example_app_1    ... done
Stopping flask-by-example_redis_1  ... done
Stopping postgres                  ... done
Removing flask-by-example_worker_1     ... done
Removing flask-by-example_app_1        ... done
Removing flask-by-example_migrations_1 ... done
Removing flask-by-example_redis_1      ... done
Removing postgres                      ... done
Removing network flask-by-example_default

$ source <(sops -d environment.secrets); docker-compose up -d
Creating network "flask-by-example_default" with the default driver
Creating flask-by-example_redis_1      ... done
Creating postgres                 ... done
Creating flask-by-example_migrations_1 ... done
Creating flask-by-example_worker_1     ... done
Creating flask-by-example_app_1        ... done

Note how easy it is to bring up and down a set of Docker containers with docker-compose.

Porting the docker-compose Services to a New Host and Operating System

We will now show how to take the docker-compose setup from the preceding section and port it to a server running Ubuntu 18.04.

Launch an Amazon EC2 instance running Ubuntu 18.04 and install docker-engine and docker-compose:

$ sudo apt-get update
$ sudo apt-get remove docker docker-engine docker.io containerd runc
$ sudo apt-get install 
  apt-transport-https 
  ca-certificates 
  curl 
  gnupg-agent 
  software-properties-common
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
$ sudo add-apt-repository 
  "deb [arch=amd64] https://download.docker.com/linux/ubuntu 
  $(lsb_release -cs) 
  stable"
$ sudo apt-get update
$ sudo apt-get install docker-ce docker-ce-cli containerd.io
$ sudo usermod -a -G docker ubuntu

# download docker-compose
$ sudo curl -L 
"https://github.com/docker/compose/releases/download/1.24.1/docker-compose-
$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
$ sudo chmod +x /usr/local/bin/docker-compose

Copy the docker-compose.yaml file to the remote EC2 instance and start the db service first, so that the database used by the application can be created:

$ docker-compose up -d db
Starting postgres ...
Starting postgres ... done

$ docker ps
CONTAINER ID   IMAGE   COMMAND    CREATED   STATUS   PORTS   NAMES
49fe88efdb45 postgres:11 "docker-entrypoint.s…" 29 seconds ago
  Up 3 seconds        0.0.0.0:5432->5432/tcp   postgres

Use docker exec to run the psql -U postgres command inside the running Docker container for the PostgreSQL database. At the PostgreSQL prompt, create the wordcount database and wordcount_dbadmin role:

$ docker-compose exec db psql -U postgres
psql (11.4 (Debian 11.4-1.pgdg90+1))
Type "help" for help.

postgres=# create database wordcount;
CREATE DATABASE
postgres=# q

$ docker exec -it 49fe88efdb45 psql -U postgres wordcount
psql (11.4 (Debian 11.4-1.pgdg90+1))
Type "help" for help.

wordcount=# CREATE ROLE wordcount_dbadmin;
CREATE ROLE
wordcount=# ALTER ROLE wordcount_dbadmin LOGIN;
ALTER ROLE
wordcount=# ALTER USER wordcount_dbadmin PASSWORD 'MYPASS';
ALTER ROLE
wordcount=# q

Before launching the containers for the services defined in docker-compose.yaml, two things are necessary:

1. Run docker login to be able to pull the Docker image pushed previously to Docker Hub:

   $ docker login

2. Set the DBPASS environment variable to the correct value in the current shell. The sops method described in the local macOS setup can be used, but for this example, set it directly in the shell:

   $ export DOCKER_PASS=MYPASS

Now launch all the services necessary for the application by running docker-compose up -d:

$ docker-compose up -d
Pulling worker (griggheo/flask-by-example:v1)...
v1: Pulling from griggheo/flask-by-example
921b31ab772b: Already exists
1a0c422ed526: Already exists
ec0818a7bbe4: Already exists
b53197ee35ff: Already exists
8b25717b4dbf: Already exists
9be5e85cacbb: Pull complete
bd62f980b08d: Pull complete
9a89f908ad0a: Pull complete
d787e00a01aa: Pull complete
Digest: sha256:4fc554da6157b394b4a012943b649ec66c999b2acccb839562e89e34b7180e3e
Status: Downloaded newer image for griggheo/flask-by-example:v1
Creating fbe_redis_1      ... done
Creating postgres    ... done
Creating fbe_migrations_1 ... done
Creating fbe_app_1        ... done
Creating fbe_worker_1     ... done

$ docker ps
CONTAINER ID   IMAGE   COMMAND    CREATED   STATUS   PORTS   NAMES
f65fe9631d44  griggheo/flask-by-example:v1 "python3 manage.py r…" 5 seconds ago
Up 2 seconds        0.0.0.0:5000->5000/tcp   fbe_app_1
71fc0b24bce3  griggheo/flask-by-example:v1 "python3 worker.py"    5 seconds ago
Up 2 seconds                                 fbe_worker_1
a66d75a20a2d  redis:alpine     "docker-entrypoint.s…"   7 seconds ago
Up 5 seconds        0.0.0.0:6379->6379/tcp   fbe_redis_1
56ff97067637  postgres:11      "docker-entrypoint.s…"   7 seconds ago
Up 5 seconds        0.0.0.0:5432->5432/tcp   postgres

At this point, after allowing access to port 5000 in the AWS security group associated with our Ubuntu EC2 instance, you can hit the external IP of the instance on port 5000 and use the application.

It’s worth emphasizing one more time how much Docker simplifies the deployment of applications. The portability of Docker containers and images means that you can run your application on any operating system where the Docker engine runs. In the example shown here, none of the prerequisites needed to be installed on the Ubuntu server: not Flask, not PostgreSQL, and not Redis. It was also not necessary to copy the application code over from the local development machine to the Ubuntu server. The only file needed on the Ubuntu server was docker-compose.yaml. Then, the whole set of services comprising the application was launched with just one command:

$ docker-compose up -d

Although docker-compose makes it easy to run several containerized services as part of the same application, it is only meant to be run on a single machine, which limits its usefulness in production scenarios. You can really only consider an application deployed with docker-compose to be “production ready” if you are not worried about downtime and you are willing to run everything on a single machine (this being said, Grig has seen hosting providers running Dockerized applications in production with docker-compose). For true “production ready” scenarios, you need a container orchestration engine such as Kubernetes, which will be discussed in the next chapter.

Exercises

• Familiarize yourself with the Dockerfile reference.

• Familiarize yourself with the Docker Compose configuration reference.

• Create an AWS KMS key and use it with sops instead of a local PGP key. This allows you to apply AWS IAM permissions to the key, and restrict access to the key to only the developers who need it.

• Write a shell script that uses docker exec or docker-compose exec to run the PostgreSQL commands necessary for creating a database and a role.

• Experiment with other container technologies, such as Podman.