We'll run httperf with the options --hog (use as many TCP ports as needed), --uri /index.html (request the static web page index.html) and we'll use --num-conn 1000 (initiate a total of 1000 connections). We will be varying the number of requests per second (specified using --rate) to see how the server responds under different workloads.

This is what the typical output from httperf looks like when run with the above options:

$ ./httperf --hog --server=bytelayer.com --uri /index.html --num-conn 1000 --rate 50
Total: connections 1000 requests 1000 replies 1000 test-duration 20.386 s
Connection rate: 49.1 conn/s (20.4 ms/conn, <=30 concurrent connections)
Connection time [ms]: min 404.1 avg 408.2 max 591.3 median 404.5 stddev 16.9
Connection time [ms]: connect 102.3
Connection length [replies/conn]: 1.000
Request rate: 49.1 req/s (20.4 ms/req)
Request size [B]: 95.0
Reply rate [replies/s]: min 46.0 avg 49.0 max 50.0 stddev 2.0 (4 samples)
Reply time [ms]: response 103.1 transfer 202.9
Reply size [B]: header 244.0 content 19531.0 footer 0.0 (total 19775.0)
Reply status: 1xx=0 2xx=1000 3xx=0 4xx=0 5xx=0
CPU time [s]: user 2.37 system 17.14 (user 11.6% system 84.1% total 95.7%)
Net I/O: 951.9 KB/s (7.8*10^6 bps)
Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0

The output shows us the number of TCP connections httperf initiated per second ("Connection rate"), the rate at which it requested files from the server ("Request rate"), and the actual reply rate that the server was able to provide ("Reply rate"). We also get statistics on the reply time—the "reply time response" is the time taken from when the first byte of the request was sent to the server to when the first byte of the reply was received—in this case around 103 milliseconds. The transfer time is the time to receive the entire response from the server.

The page we will be requesting in this case, index.html, is 20 KB in size which is a pretty average size for an HTML document. httperf requests the page one time per connection and doesn't follow any links in the page to download additional embedded content or script files, so the number of such links in the page is of no relevance to our test.

The following chart shows the response time, in milliseconds, of the server when it is running without ModSecurity. The number of requests per second is on the horizontal axis:

As we can see, the server consistently delivers response times of around 300 milliseconds until we reach about 75 requests per second. Above this, the response time starts increasing, and at around 500 requests per second the response time is almost a second per request. This data is what we will use for comparison purposes when looking at the response time of the server after we enable ModSecurity.

Finding the memory usage on a Linux system can be quite tricky. Simply running the Linux top utility and looking at the amount of free memory doesn't quite cut it, and the reason is that Linux tries to use almost all free memory as a disk cache. So even on a system with several gigabytes of memory and no memory-hungry processes, you might see a free memory count of only 50 MB or so.

Another problem is that Apache uses many child processes, and to accurately measure the memory usage of Apache we need to sum the memory usage of each child process. What we need is a way to measure the memory usage of all the Apache child processes so that we can see how much memory the web server truly uses.

To solve this, here is a small shell script that I have written that runs the ps command to find all the Apache processes. It then passes the PID of each Apache process to pmap to find the memory usage, and finally uses awk to extract the memory usage (in KB) for summation. The result is that the memory usage of Apache is printed to the terminal.

The actual shell command is only one long line, but I've put it into a file called apache_mem.sh to make it easier to use:

#!/bin/sh
# apache_mem.sh
# Calculate the Apache memory usage
ps -ef | grep httpd | grep ^apache | awk '{ print $2 }' |  xargs pmap -x | grep 'total kB' | awk '{ print $3 }' |  awk '{ sum += $1 } END { print sum }'

Now, let's use this script to look at the memory usage of all of the Apache processes while we are running our performance test. The following graph shows the memory usage of Apache as the number of requests per second increases:

Apache starts out consuming about 300 MB of memory. Memory usage grows steadily and at about 150 requests per second it starts climbing more rapidly.

At 500 requests per second, the memory usage is over 2.4 GB—more than the amount of physical RAM of the server. The fact that this is possible is because of the virtual memory architecture that Linux (and all modern operating systems) use. When there is no more physical RAM available, the kernel starts swapping memory pages out to disk, which allows it to continue operating. However, since reading and writing to a hard drive is much slower than to memory, this starts slowing down the server significantly, as evidenced by the increase in response time seen in the previous graph.

In both of the tests above, the server's CPU usage was consistently around 1 to 2%, no matter what the request rate was. You might have expected a graph of CPU usage in the previous and subsequent tests, but while I measured the CPU usage in each test, it turned out to run at this low utilization rate for all tests, so a graph would not be very useful. Suffice it to say that in these tests, CPU usage was not a factor.

The following graph shows the server response time with the core ruleset loaded:

At first, the response time is around 340 ms, which is about 35 ms slower than in previous tests. Once the request rate gets above 50, the server response time starts deteriorating. As the request rates grows, the response time gets worse and worse, reaching a full 5 seconds at 100 requests per second. I have capped the graph at 100 requests per second, as the server performance has already deteriorated enough at this point to allow us to see the trend.

We see that the point at which memory usage starts increasing has gone down from 75 to 50 requests per second now that we have enabled the core ruleset. This equals a reduction in the maximum number of requests per second the server can handle of 33%.

What could be causing this deterioration in response time? Let's take a look at the memory usage of Apache and see if we can find any clues:

Aha! We see that once we hit 50 requests per second, the memory usage goes up dramatically. The server only has 2 GB of memory, so it's a pretty good bet that the increase in memory usage and subsequent swapping of memory pages to and from the hard drive is what causes the server performance to deteriorate.

For comparison purposes, take a look at these graphs, which show the response rate and memory usage for ModSecurity with no rules (dotted line) and ModSecurity with the core ruleset (solid line):

The memory usage with the core ruleset loaded is represented by the solid line in the following graph:

The conclusion we can draw from this is that once the request rate goes over a certain threshold, the memory usage grows to the extent that the kernel has to start swapping data to disk. Once this happens, the response time grows larger and larger.

We see that under 50 requests per second the response times are virtually identical, indicating that ModSecurity does not incur any significant performance penalty as long as there is a sufficient amount of free memory available. This is important, because it shows that the underlying limit being encountered is the amount of free memory, and this could have happened just as easily without ModSecurity enabled. ModSecurity just lowers the threshold at which this happens.

Is it possible that the deterioration in response time seen in the previous graphs is caused by a ModSecurity configuration setting? Two likely candidates would be request and response body buffering. Both of these settings, when set to On, cause ModSecurity to allocate extra memory to hold the buffered request and response bodies.

Let's set both RequestBodyAccess and ResponseBodyAccess to Off and run the same tests again and see if there's any difference.

When buffering is disabled, the response time doesn't start increasing until we get above 75 requests per second, indicating that turning off buffering indeed improved performance—the server is able to handle 25 extra requests per second before performance starts going downhill.

Let's see what the memory usage looks like with buffering disabled:

As expected, we see that with buffering turned off, memory usage doesn't start increasing significantly until we reach 75 requests per second, again showing that memory load is a crucial factor in the number of requests per second the server can handle.

We have seen that loading the core ruleset caused a decrease in the number of simultaneous connections the server could handle before being overwhelmed. CPU usage was not an issue at all, but the server was running a pretty fast dual-core processor, so the results may be different on systems with older hardware.

In the case of the limitation we ran into here, optimizing two things should enable us to squeeze a significant amount of extra performance out of the server:

In the next section we will be looking at how to decrease the amount of memory Apache uses, and we will also be looking at ways to optimize your rules for efficiency.

Non-capturing parentheses, as we have mentioned earlier, are this awkward-looking construct:

(?: )

They perform the same function as regular parentheses, with the difference that the non-capturing ones do not capture backreferences. This means the regular expression engine doesn't need to keep track of backreferences or allocate memory for them, which saves some processing time as well as memory.

If you look at the core ruleset, you'll notice that many of the rules in it contain these parentheses, which shows you that if performance is something you have in mind then you should be using them, too.

It is usually more efficient to use a single regular expression instead of a lot of smaller ones. So if for example you wanted to match a filename extension, the following rule would be faster than using one rule for each extension:

SecRule REQUEST_FILENAME ".(?:exe|bat|pif)" deny

You'll have to weigh writing one-line, complex rules against the readability of your ruleset. For a small amount of rules in the ruleset, the difference in speed won't matter, and you should prefer to use simple, readable rules. If your ruleset starts growing into the hundreds of rules, you may want to consider using the above technique.