Sadly, the settings we have seen so far are not the only things that need tweaking with increased network connections to your server. As you increase the load on your server, you may also begin to see nf_conntrack: table full errors in your dmesg and/or kernel logs. For those unfamiliar with netfilter, it is a kernel module that tracks all Network Address Translation (NAT) sessions in a hashed table that adds any new connections to it and clears them after they are closed and a predefined timeout is reached, so as you increase the connection volume from and to a single machine, you will most likely find that the majority of these related settings are defaulted rather conservatively and are in need of tweaking (though your distribution may vary--make sure to verify yours!):

$ sysctl -a | grep nf_conntrack
net.netfilter.nf_conntrack_buckets = 65536
<snip>
net.netfilter.nf_conntrack_generic_timeout = 600
<snip>
net.netfilter.nf_conntrack_max = 262144
<snip>
net.netfilter.nf_conntrack_tcp_timeout_close = 10
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 60
net.netfilter.nf_conntrack_tcp_timeout_established = 432000
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_last_ack = 30
net.netfilter.nf_conntrack_tcp_timeout_max_retrans = 300
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 60
net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 120
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_unacknowledged = 300
<snip>

Quite a few of these can be changed, but the usual suspects for errors that need tweaking are as follows:

• net.netfilter.nf_conntrack_buckets: Controls the size of the hash table for the connections. Increasing this is advisable, although it can be substituted with a more aggressive timeout. Note that this cannot be set with regular sysctl.d settings, but instead needs to be set with a kernel module parameter.
• net.netfilter.nf_conntrack_max: The number of entries to hold. By default, this is four times the value of the previous entry.
• net.netfilter.nf_conntrack_tcp_timeout_established: This keeps the mapping for an open connection for up to five days (!). This is generally almost mandatory to reduce in order to not overflow your connection tracking table, but don't forget that it needs to be above the TCP keepalive timeout or you will get unexpected connection breaks.

To apply the last two settings, you need to add the following to /etc/sysctl.d/10-conntrack.conf and adjust the values for your own infrastructure configuration:

net.netfilter.nf_conntrack_tcp_timeout_established = 43200
net.netfilter.nf_conntrack_max = 524288

For a bucket count, you need to directly change the nf_conntrack hashsize kernel module parameter:

echo '131072' | sudo tee /sys/module/nf_conntrack/parameters/hashsize

Finally, to ensure that the right order is followed when loading the netfilter module so these values persist correctly, you will probably also need to add the following to the end of /etc/modules:

nf_conntrack_ipv4
nf_conntrack_ipv6

If everything was done correctly, your next restart should have all of the netfilter settings we talked about set.