We devoted a great deal of content to the concept of upstreams. Just to remind you, upstreams are entities that generate HTTP responses that Nginx sends to the clients. Usually, an upstream contains several (or at least one) servers speaking one of the supported protocols, such as FastCGI or plain HTTP. Nginx uses sophisticated client code to very efficiently proxy the communication between the clients and the upstream in a transparent way by also optimizing the number of idle connections and wasted memory. There are a number of algorithms that Nginx uses to balance the client load on all the members of an upstream block, and one of those algorithms is known to bite the unsuspecting web administrators.

The configuration under discussion looks like this:

proxy_pass http://backend;

upstream backend {
    server server1.example.com;
    server server2.example.com;
    server server3.example.com;
    ip_hash;
}

The first line sets up the handling of some location or the whole server by proxying everything to a named upstream. The mentioned upstream is configured in the block later. It consists of three servers and a special directive ip_hash, which turns on an algorithm to use when choosing one of the three servers that will actually process each incoming request.

The default algorithm is the so-called weighted round-robin. In our simple configuration without any weights, the round-robin would choose the servers one after the other in that order and rewind back to the first after the last. It is an efficient and simple algorithm that will surely balance the load in a good fashion. One significant disadvantage of it is that the requests from the same client may end up being processed on different upstream servers, which sometimes is not good, for example, because of the caching in RAM on the individual upstream servers. The directive ip_hash will turn the round-robin off, and instead, servers will be chosen based on a hash value computed from the IP address of the client.

One of the consequences is that the same client will always talk to the same server (unless that server is down, in which case the hash value will point to another server trying to minimize the effect on the rest of the servers in the upstream). The other consequence is that your client load will be distributed between servers only as evenly as your client IP addresses. Usually, when you have enough load to justify proper upstream blocks with many servers, your client IP pool will already be big and diverse enough. Sometimes there is another proxy in front of Nginx, and all your incoming requests look like they come from a very limited set of addresses. In this case, you have a subtle and hard-to-debug problem, which may or may not lead to a disaster.

If you are lucky, you will note that the load on your upstream servers is very uneven. For example, if one of those three servers is completely idle although there are no problems with it, and it happily responds to direct requests.

The quick and dirty workaround here is to remove the ip_hash directive. The proper solution will require you to employ the ngx_http_realip module and provide better data for the IP-hashing algorithm. The idea is to save the real client IP address to a special HTTP request header on the proxy that is located in front of the Nginx and then take it from there instead of the real endpoint address of incoming TCP connections. This module may not be compiled in your version of Nginx.

There are also other consistent hashing strategies that you might consider. Refer to the full documentation for hashing at http://nginx.org/en/docs/http/ngx_http_upstream_module.html#hash.