Problem

You need to automate your tests with a load driver to gain consistency and repeatability in your testing.

Solution

Use an HTTP load-testing tool such as Apache JMeter, Locust, Gatling, or whatever your team has standardized on. Create a configuration for your load-testing tool that runs a comprehensive test on your web application. Run your test against your service. Review the metrics collected from the run to establish a baseline. Slowly ramp up the emulated user concurrency to mimic typical production usage and identify points of improvement. Tune NGINX and repeat this process until you achieve your desired results.

Discussion

Using an automated testing tool to define your test gives you a consistent test to build metrics from when tuning NGINX. You must be able to repeat your test and measure performance gains or losses to conduct science. Running a test before making any tweaks to the NGINX configuration to establish a baseline gives you a basis to work from so that you can measure if your configuration change has improved performance or not. Measuring for each change made will help you identify where your performance enhancements come from.

Problem

You need to increase the number of requests allowed to be made over a single connection from clients and increase the amount of time that idle connections are allowed to persist.

Solution

Use the keepalive_requests and keepalive_timeout directives to alter the number of requests that can be made over a single connection and change the amount of time idle connections can stay open:

http {
    keepalive_requests 320;
    keepalive_timeout 300s;
    # ...
}

The keepalive_requests directive defaults to 100, and the keepalive_timeout directive defaults to 75 seconds.

Discussion

Typically, the default number of requests over a single connection will fulfill client needs because browsers these days are allowed to open multiple connections to a single server per FQDN. The number of parallel open connections to a domain is still limited typically to a number less than 10, so in this regard, many requests over a single connection will happen. A trick for HTTP/1.1 commonly employed by content delivery networks is to create multiple domain names pointed to the content server and alternate which domain name is used within the code to enable the browser to open more connections. You might find these connection optimizations helpful if your frontend application continually polls your backend application for updates, because an open connection that allows a larger number of requests and stays open longer will limit the number of connections that need to be made.

Problem

You need to keep connections open to upstream servers for reuse to enhance your performance.

Solution

Use the keepalive directive in the upstream context to keep connections open to upstream servers for reuse:

proxy_http_version 1.1;
proxy_set_header Connection "";

upstream backend {
  server 10.0.0.42;
  server 10.0.2.56;
  
  keepalive 32;
}

The keepalive directive in the upstream context activates a cache of connections that stay open for each NGINX worker. The directive denotes the maximum number of idle connections to keep open per worker. The proxy modules directives used above the upstream block are necessary for the keepalive directive to function properly for upstream server connections. The proxy_http_version directive instructs the proxy module to use HTTP version 1.1, which allows for multiple requests to be made in serial over a single connection while it’s open. The proxy_set_header directive instructs the proxy module to strip the default header of close, allowing the connection to stay open.

Discussion

You want to keep connections open to upstream servers to save the amount of time it takes to initiate the connection, allowing the worker process to instead move directly to making a request over an idle connection. It’s important to note that the number of open connections can exceed the number of connections specified in the keepalive directive because open connections and idle connections are not the same. The number of keepalive connections should be kept small enough to allow for other incoming connections to your upstream server. This small NGINX tuning trick can save some cycles and enhance your performance.

Problem

You need to buffer responses between upstream servers and clients in memory to avoid writing responses to temporary files.

Solution

Tune proxy buffer settings to allow NGINX the memory to buffer response bodies:

server {
    proxy_buffering on;
    proxy_buffer_size 8k;
    proxy_buffers 8 32k;
    proxy_busy_buffer_size 64k;
    # ...
}

The proxy_buffering directive is either on or off; by default it’s on. The proxy_buffer_size denotes the size of a buffer used for reading the first part of the response, headers, from the proxied server and defaults to either 4k or 8k, depending on the platform. The proxy_buffers directive takes two parameters: the number of buffers and the size of the buffers. By default, the proxy_buffers directive is set to a number of 8 buffers of size either 4k or 8k, depending on the platform. The proxy_busy_buffer_size directive limits the size of buffers that can be busy, sending a response to the client while the response is not fully read. The busy buffer size defaults to double the size of a proxy buffer or the buffer size. If proxy buffering is disabled, the request cannot be sent to the next upstream server in the event of a failure because the NGINX has already started sending the request body.

Discussion

Proxy buffers can greatly enhance your proxy performance, depending on the typical size of your response bodies. Tuning these settings can have adverse effects and should be done by observing the average body size returned and performing thorough and repeated testing. Extremely large buffers, set when they’re not necessary, can eat up the memory of your NGINX box. You can set these settings for specific locations that are known to return large response bodies for optimal performance.  

Also See

NGINX proxy_request_buffering Documentation

Problem

You need to buffer logs to reduce the opportunity of blocks to the NGINX worker process when the system is under load.

Solution

Set the buffer size and flush time of your access logs:

http {
    access_log /var/log/nginx/access.log main buffer=32k
        flush=1m gzip=1;
}

The buffer parameter of the access_log directive denotes the size of a memory buffer that can be filled with log data before being written to disk. The flush parameter of the access_log directive sets the longest amount of time a log can remain in a buffer before being written to disk. When using gzip, the logs are compressed before being written to the log—values of level 1 (fastest, less compression), through 9 (slowest, best compression) are valid.

Discussion

Buffering log data into memory may be a small step toward optimization. However, for heavily requested sites and applications, this can make a meaningful adjustment to the usage of the disk and CPU. When using the buffer parameter to the access_log directive, logs will be written out to disk if the next log entry does not fit into the buffer. If using the flush parameter in conjunction with the buffer parameter, logs will be written to disk when the data in the buffer is older than the time specified. When tailing the log, and with buffering enabled, you may see delays up to the amount of time specified by the flush parameter.

Problem

You need to tune your operating system to accept more connections to handle spike loads or highly trafficked sites.

Solution

Check the kernel setting for net.core.somaxconn, which is the maximum number of connections that can be queued by the kernel for NGINX to process. If you set this number over 512, you’ll need to set the backlog parameter of the listen directive in your NGINX configuration to match. A sign that you should look into this kernel setting is if your kernel log explicitly says to do so. NGINX handles connections very quickly, and, for most use cases, you will not need to alter this setting.

Raising the number of open file descriptors is a more common need. In Linux, a file handle is opened for every connection; and, therefore, NGINX may open two if you’re using it as a proxy or load balancer because of the open connection upstream. To serve a large number of connections, you may need to increase the file descriptor limit system-wide with the kernel option sys.fs.file_max, or, for the system user, NGINX is running as in the /etc/security/limits.conf file. When doing so, you’ll also want to bump the number of worker_connections and worker_rlimit_nofile. Both of these configurations are directives in the NGINX configuration.

Enable more ephemeral ports. When NGINX acts as a reverse proxy or load balancer, every connection upstream opens a temporary port for return traffic. Depending on your system configuration, the server may not have the maximum number of ephemeral ports open. To check, review the setting for the kernel setting net.ipv4.ip_local_port_range. The setting is a lower- and upper-bound range of ports. It’s typically OK to set this kernel setting from 1024 to 65535. 1024 is where the registered TCP ports stop, and 65535 is where dynamic or ephemeral ports stop. Keep in mind that your lower bound should be higher than the highest open listening service port.

Discussion

Tuning the operating system is one of the first places you look when you start tuning for a high number of connections. There are many optimizations you can make to your kernel for your particular use case. However, kernel tuning should not be done on a whim, and changes should be measured for their performance to ensure the changes are helping. As stated before, you’ll know when it’s time to start tuning your kernel from messages logged in the kernel log or when NGINX explicitly logs a message in its error log.