To increase performance, it is usually highly advantageous to increase the size of the socket buffers because they are no longer doing the work of a single machine but the work of as many Docker containers as you have running on top of regular machine connectivity. For this, there are a few settings that you should probably set to make sure that the socket buffers are not struggling to keep up with all the traffic passing through them. At the time of writing this book, most of these default buffer settings are generally pretty tiny when the machine starts (200 KB in a few machines that I've checked) and they are supposed to be dynamically scaled, but you can force them to be much larger from the start.

On an Ubuntu LTS 16.04 installation, the following are the default ones for the buffer settings (though yours may vary):

net.core.optmem_max = 20480
net.core.rmem_default = 212992
net.core.rmem_max = 212992
net.core.wmem_default = 212992
net.core.wmem_max = 212992
net.ipv4.tcp_rmem = 4096 87380 6291456
net.ipv4.tcp_wmem = 4096 16384 4194304

We will dial these values up to some sensible defaults by adding the following to /etc/sysctl.d/10-socket-buffers.conf, but be sure to use values that make sense in your environment:

net.core.optmem_max = 40960
net.core.rmem_default = 16777216
net.core.rmem_max = 16777216
net.core.wmem_default = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 87380 16777216

By increasing these values, our buffers start large and should be able to handle quite a bit of traffic with much better throughput, which is what we want in a clustering environment.