Chapter 5, Concurrency and Parallelism in RxPY, detailed the use of multithreading to handle CPU-bound operations such as audio encoding. It also explained the limitations of this model with Python code, which was due to the limitations implied by the GIL. Python has never been good at multithreading, and will probably never be. However, in web applications or microservices, this is not necessarily an issue. Instead of using multithreading to deal with CPU bound operations, it is possible to use multiprocessing.

With multithreading, only one instance of the service is running, and this service uses multithreading to parallelize task executions. With multiprocessing, several single-threaded instances of the service are executed in parallel. However, this has some implications. The biggest one is that the service should be stateless; any request should be independent of other requests, or the internal state of the service. Since several instances of the service are running in parallel, there is usually no guarantee that some related requests will be served by the same instance (Traefik allows us to do this in some cases). Should the service be stateful, then the management of its state must be done by another service since it cannot be done in the context of the service instance (multiple processes do not share memory, so they cannot share their state directly).

With Docker Compose, starting new instances of the service is done with the up command and an additional parameter that indicates how many instances of the container must be running. For example, starting two more instances of the audio-encoder container is done the following way:

$ docker-compose up -d --scale audio-encoder=3
audio-encode-server-1_reverse-proxy_1 is up-to-date
audio-encode-server-1_minio_1 is up-to-date
Starting audio-encode-server-1_audio-encoder_1 ... done
Creating audio-encode-server-1_audio-encoder_2 ... done
Creating audio-encode-server-1_audio-encoder_3 ... done

Here, the existing services are kept as they are, and two new instances of the audio-encoder container are started. The Starting log on the existing instance is misleading; this instance is not stopped when executing this command, but it keeps running while the other instances are started.

Since Traefik monitors Docker instances, it sees that new instances are available and it adds them to the list of servers of the audio-encoder backend. Several instances are now running and the load balancer is now enabled. This can be seen in the web interface of Traefik.

Scaling down the audio-encode backend to one instance is done the same way:

$ docker-compose up -d --scale audio-encoder=1
audio-encode-server-1_reverse-proxy_1 is up-to-date
audio-encode-server-1_minio_1 is up-to-date
Stopping and removing audio-encode-server-1_audio-encoder_2 ... done
Stopping and removing audio-encode-server-1_audio-encoder_3 ... done
Starting audio-encode-server-1_audio-encoder_1 ... done

The two most recently added instances of the audio-encoder container are stopped, and only one instance remains. Here again, Traefik sees these changes and adjusts its configuration accordingly.