Take a moment to have a closer look at just which variables are available for use. There are a lot, so I have placed them in Appendix A.

ModSecurity uses two types of variables: Standard variables, which simply contain a single value, and collections, which can contain more than one value. One example of a collection is REQUEST_HEADERS, which contains all the headers sent by the client, such as for example User-Agent or Referer.

To access a field in a collection, you give the collection name followed by a colon and then the name of the item you want to access. So if for example we wanted to look at the referrer in a particular request we would use the following:

SecRule REQUEST_HEADERS:Referer "bad-referer.com"

Most collections can also be used on their own, without specifying a field, in which case they refer to the whole of the data in the collection. So for example, if you wanted to check all argument values for the presence of the string script you could use the following:

SecRule ARGS "script"

In practice, if the query string submitted was ?username=john&login=yes then the above would expand to this when the rule is evaluated:

SecRule ARGS:john|ARGS:login "script"

The following collections are available in ModSecurity 2.5:

Some collections have fixed fields, such as the GEO collection, which contains fields such as COUNTRY_NAME and CITY. Other collections, such as REQUEST_HEADERS have variable field names—in the case of REQUEST_HEADERS it depends on which headers were sent by the client.

It is never an error to specify a field name that doesn't exist or doesn't have a value set—so specifying REQUEST_HEADERS:Rubber-Ducky always works—the value would not be tested against if the client hasn't sent a Rubber-Ducky header.

The TX collection is also known as the transaction collection. You can use it to create your own variables if you need to store data during a transaction:

SecRule REQUEST_URI "passwd" "pass,setvar:tx.hackscore=+5"
SecRule REQUEST_URI "<script" "pass,setvar:tx.hackscore=+10"
SecRule TX:HACKSCORE "@gt 10" deny

In the first two rules we use the setvar action to set the collection variables. You use this action whenever you want to create or update a variable. (You can also remove variables by using the syntax setvar:!tx.hackscore as prefixing the variable with an exclamation mark removes it.)

The TX collection also contains the built-in fields TX:0 and TX:1 through TX:9. TX:0 is the value that matched when using the @rx or @pm operators (we will learn more about the latter operator later). TX:1 TX:9 contain the captured regular expression values when evaluating a regular expression together with the capture action.

There are three types of collections in ModSecurity that can be used as persistent storage. We have already seen that it is possible to use setvar to create a variable and assign a value to it. However, the variable expires and is no longer available once the current request has been handled. In some situations you would like to be able to store data and access it in later requests.

There are three collections that can be used for this purpose:

The IP collection is used to store information about a user from a specific IP address. It can be used to store such things as the number of failed access attempts to a resource, or the number of requests made by a user.

Before we can use one of these collections, we need to initialize it. This is done by using the initcol action:

SecAction initcol:ip=%{REMOTE_ADDR},nolog,pass

We also need to make sure that we have configured a data directory for ModSecurity to use:

SecDataDir /var/log/httpd/modsec_data

Make sure that the directory is writable by the Apache user or the initcol action will not work properly. Now that this is done we can use the IP collection in conjunction with setvar to store user-specific data.

It is possible to look in several variables at once to see if a matching string can be found. If for example we wanted to examine both the request headers and the request arguments passed for the string park and deny any matching requests we could use the following rule:

SecRule ARGS|REQUEST_HEADERS "park" deny

As can be seen the pipe character (|) is used to separate the variable names, and it functions a lot like the logical or you might be familiar with if you've done any programming.

You may be wondering what the difference is between the following:

SecRule REQUEST_URI "secret" "deny"

and this:

SecRule REQUEST_URI secret deny

In this case there is no difference. If both the operator expression and action list don't contain any whitespace then they don't need to be enclosed in quotes. However, if the rule was modified to match the string secret place then we would need to enclose this string in quotes:

SecRule REQUEST_URI "secret place" deny

The essence of quotes as they apply to ModSecurity is that anything enclosed in quotes is considered as "one part", meaning that the"secret place" string is considered to be part of the operator expression of the rule.

What if we need to specify a string within the operator or action list, and it is already enclosed in quotes? This happens if for example we use the msg: action to write a log message. In this case we would use single quote characters to enclose the string we want to log:

SecRule REQUEST_URI "secret place" "deny,log,msg:'Someone tried to access the secret place!'"

What if even the quoted message needed to include quotes? Let's say that you wanted to log the message "Someone's trying to hack us!". In that case you would need to escape the innermost quote (the one in "someone's") with a backslash. The rule would now look like this:

SecRule REQUEST_URI "secret place" "deny,log,msg:'Someone's trying to hack us!'"

In general throughout this book I tend to enclose operators and action lists in quotes even when not strictly necessary. It makes it easier to later expand on rules without forgetting the quotes.

Remember that you must restart Apache to reload the ModSecurity ruleset. If you were to forget to restart or were distracted by another task then a broken ModSecurity configuration file (resulting, for example, from forgetting to wrap an action list in quotes) would result in Apache refusing to start. This might not be a big deal so long as the server is running along nicely, but if anything such as log rotation were to cause the server to restart then the restart would fail and your web server would be down (and yes, the reason I mention this is because it happened to me in exactly the manner described—I wouldn't want you making the same mistake).

I'm not going to provide any formal specification of how regular expressions work here, but instead I will give a few short examples to allow you to get a better understanding of how they work. For a more complete overview, please see Appendix B which contains a primer on regular expressions.

The following examples cover some of the most common forms of regular expressions:

Regular expressions can contain metacharacters. We have already seen an example of these in the table above: ^, $, and "dot" don't match any one character but have other meaning within regular expressions (start of string, end of string and match any character in this case). The following table lists some additional metacharacters that are frequently used in regexes:

So for example if we wanted to match favorite or favourite, we could use the regex favou?rite. Similarly, if we wanted to match either previous or previously we could use the regex previous(ly)?. The parentheses—()—group the ly characters and then apply the ? operator to the group to match it 0 or 1 times (therefore making it optional).

So what if we really do want to match a dot literally, and not have it interpreted as any character? In that case we need to escape the dot with a backslash. Referring back to our previous example, if we really did want to match the string p.t literally, we would use the regex p.t to ensure that the dot is interpreted like a literal character and not a metacharacter by the regex engine.

If you are already familiar with regular expressions, the preceding examples probably didn't teach you anything new. If, however, you feel that you need to learn more about regexes before you feel comfortable with them, I encourage you to read Appendix B for a more in-depth look at how regexes work.

As we will see, there are several ways to match strings using operators other than @rx, but in many situations, regular expressions are the only tool that will get the job done, so it definitely pays to learn as much as you can about them.

Should you find that the appendix tickles your fancy and that you really want to learn about regexes then I can heartily recommend that you get a hold of "Mastering Regular Expressions" by Jeffrey E. F. Friedl (published by O'Reilly), which is the definitive guide to the subject.

To get a better understanding of the default regular expression mode (@rx) of matching, consider the following two rules, which are both equivalent:

# Rule 1 SecRule REMOTE_HOST "@rx .microsoft.com$" deny
# Rule 2 SecRule REMOTE_HOST ".microsoft.com$" deny

Both of the above rules do exactly the same thing—block any access attempts from users at Microsoft Corporation. The @rx operator is omitted in the second rule, but since the ModSecurity engine interprets the provided string as a regular expression if no other operator is specified, the rules will both match any domain name ending in .microsoft.com.

As we just learned, the reason there is a backslash before the dots (".") in the above rules is that the dot is a special character in regular expressions. On its own, a dot will match any character, which means that the regular expression .microsoft.com would match hostnames ending in .microsoft.com as well as xmicrosoft.com and others. To avoid this, we escape the dot with a backslash, which instructs the regular expression engine that we really do want it to match a dot, and not just any character.

You may also wonder why there is a $ sign at the end of the regular expressions above. Good question! The $ sign matches the end of a line. If we had not specified it, the regular expression would have matched other hostnames such as microsoft.com.mysite.com as well, which is probably not what we want.

You can count the number of items in a collection by prefixing it with an ampersand (&). For example, the following rule matches if the client does not send any cookies with his request:

SecRule &REQUEST_COOKIES "@eq 0"

You can also use the count operator to make sure that a certain field in a collection is present. If we wanted a rule to match if the User-Agent header was missing from a request we could use the following:

SecRule &REQUEST_HEADER:User-Agent "@eq 0"

The above will match if the header is missing. If instead there is a User-Agent header but it is empty the count operator would return 1, so it is important to be aware that there is a difference between a missing field and an empty one.

You can also use a regular expression to filter out only certain fields in a collection. For example, to select all arguments that contain the string arg, use the following construct:

SecRule ARGS:/arg/ "secret" deny

The regular expression filters out any arguments whose name contains arg, so the above rule will match query strings such as arg1=secret phrase which contain the value secret, but it will not match if no argument name contains the string arg, since in that case the regular expression construct doesn't select any arguments at all from the collection.

You'll notice that the syntax used to filter out arguments differs from a normal collection declaration by the slashes surrounding the regular expression. We use the forward slashes to tell the rule engine to treat the string within the slashes as a regular expression. Had we omitted the slashes, only parameters with the exact name arg would have been selected and matched against.

The collections IP, SESSION, and USER contain a number of built-in fields, that can be used to get statistics about the creation time and update rate of each collection:

The CREATE_TIME and LAST_UPDATE_TIME fields contain a UNIX timestamp (number of seconds since January 1st, 1970), so keep that in mind if you ever need to convert these values to a human-readable format.

The KEY field contains the value stored in the collection variable when the collection was first initialized with initcol. The IP.KEY field would for example contain the IP address of the client.

We have seen how to write regular expressions that match one of several alternative words. For example, to match red, green, or blue we would use the regex (red|green|blue). ModSecurity has two "phrase matching" operators that can be used to match a set of words: @pm and @pmFromFile.

The @pm version of our color-matching example would look like this:

SecRule ARGS "@pm red green blue" deny

This will trigger if an argument contains any of the strings red, green, or blue. As with the regex operator, a partial match is enough, so a query string of the form ?color=cobaltblue would trigger a match since the argument value contains the string blue.

Set-based pattern matching has several advantages:

red
green
blue
...

SecRule ARGS "@pmFromFile /usr/local/colors.txt" deny

Executing operator "rx" with param "(?:red|green|blue)" against ARGS:x.
Target value: "red"
Operator completed in 11 usec.

Executing operator "pm" with param "red green blue" against ARGS:x.
Target value: "red"
Operator completed in 6 usec.

In later chapters we will learn more about using a positive secure model. A positive security model means that instead of trying to detect malicious data, we assume that all data is malicious and then only allow through exactly that which we determine to be valid requests. The operator @validateByteRange is useful for this purpose—you can use it to make sure that a character is only within a certain allowed range. For example, you would probably want an argument that contains a username to only contain the letters a-z, A-Z, and 0-9. Ensuring this is easy using @validateByteRange:

# Only allow reasonable characters in usernames SecRule ARGS:username "@validateByteRange 48-57, 65-90,  97-122, 45, 95"

The range 48-57 corresponds to ASCII characters 0..9, 65-90 is A..Z, and 97-122 is a..z. The ASCII codes for dash (45) and underscore (95) are also included so that these characters can be used in a username.

The above rule will block any attempt to provide a username argument that contains any characters except those allowed. Consult an ASCII chart to find out which ranges you need to block. Separate ranges and numbers using commas and make sure that all numbers are input in decimal notation.

The way the allow action works differs depending on how a rule is written. An allow action can be configured to work in one of three ways:

To block a request you use the deny action. This has the effect of immediately stopping any further rule processing and denying the request with the HTTP error code that was specified in the rule or inherited as the default action.

There is another action called block, which sounds like it would be similar to deny. This action is deceptive however, as in its current form it can be used to both deny and allow requests, depending on what the default action specifies. One way this would help is not having to modify every rule if you wanted to change the disruptive action from deny to allow, for example. The intention of the ModSecurity authors is to expand the capabilities of block in the future, but I do not recommend using it at the current time.

Sometimes we want rule processing to continue even when a rule matches. In this case, we use the pass action to tell the rule engine to continue processing the next rule even if this one matches, like so:

SecRule REMOTE_ADDR "^192." "pass,log,logdata:'Suspicious 
IP address'"

Any rule that is in place to perform an action (such as executing a script) but where you don't want to block the request should have a pass action in its action list.

Using the drop action results in the active TCP connection to the client immediately being closed by sending a TCP FIN packet. This action is useful when responding to Denial of Service attacks since it will preserve server resources such as limited-size connection tables to the greatest extent possible.

Requests that you think should be handled by another server can be redirected. This is done using the redirect action, and the effect is that the rule engine immediately stops any further processing and sends a HTTP status 302 redirect response to the client. The following rule redirects any matching requests to Google:

SecRule REQUEST_BASENAME "search.php"  "redirect:http://www.google.com"

It is important that you specify http:// before the server hostname if you want to redirect a request to a different server. If, in the example above, we had written the redirect string as redirect:www.google.com, the visitor would have ended up being redirected to http://www.ourserver.com/www.google.com, which is not what we intended.

You can also proxy requests. In web server terms, proxying means forwarding a request to another server and letting it deal with it. After the forwarded request has been handled the result is returned to the client via the original server. This means that to the client, it looks like the original server handled the request all along.

To proxy a request using ModSecurity we use the proxy action:

SecRule IP:Attacker "1" proxy:http://10.10.10.101/

Proxying allows you to do sneaky things like redirect any requests that you consider to be attacks to a honeypot web server and let it deal with it. A honeypot, when the term is used in information security, is a dedicated server that is allowed to attract hackers. The goal is to have the honeypot lure the hackers in, all the while making them think that they are trying to hack into a legitimate server. This serves the purpose of deflecting any attacks away from your real servers, and can also be used to create excellent deceptive effects by planting plausible-looking but completely false data on the honeypot server.

To proxy requests, your Apache server must have the mod_proxy module dynamically loaded or statically compiled in.

As an example, if during a request you notice that you don't want the rule engine to process any further rules you can use ctl:ruleEngine=off in the action list of a rule to stop the engine for the remainder of the request.

The ctl:requestBodyProcessor action doesn't correspond to any directive. Instead, this action can be used to set the module used to parse request bodies. Currently, this is used to allow the parsing of XML request bodies. If you need to be able to parse XML data submitted by clients, you should use the following rule to enable the XML processor and instruct it to parse XML request bodies:

SecRule REQUEST_HEADERS:Content-Type "^text/xml$" "nolog,pass,ctl:requestBodyProcessor=XML, ctl:requestBodyAccess=On"

With the above rule in place, any POST request with the content type text/xml will be parsed by the XML parser. You can then access the data by specifying the XML collection together with an XPath expression:

SecRule XML:/person/name/text() "Neo"

XPath expressions are a way to get to specific data in an XML document. The above rule would evaluate all of the<name> nodes contained in the following XML document and trigger a match since one of the nodes contained the string Neo:

<persons>
<person>
<name>John</name> </person>
<person> <name>Neo</name>
</person>
</persons>

The three most commonly used HTTP request methods are GET, POST and HEAD. You might be surprised to learn that the HTTP specification actually implements many more methods—if a web server supports the WebDAV (Web-based Distributed Authoring and Versioning) extensions, the total number of methods becomes almost 30. As an example, here are the request methods implemented by the latest version of Apache:

Unless we had good reason to allow any of the less common methods, it would be good practice to block any but the commonly used ones. This instantly blocks any potential vulnerability that might be present in the Apache source code for the handling of non-standard methods.

This rule blocks all HTTP methods except for GET, POST, and HEAD:

SecRule REQUEST_METHOD "!^(GET|POST|HEAD)$" "deny,status:405"

We use the HTTP error code 405—Method not allowed for blocking any such non-standard method access attempts.

Suppose we wanted to restrict access to our web site so that it was only available during business hours (don't laugh, a certain UK government web site actually closes its company information search service at night). To accomplish this, we can use the variable TIME_HOUR together with a regular expression so that our site can only be accessed between the hours of 8 AM and 5 PM:

SecRule TIME_HOUR !^(8|9|10|11|12|13|14|15|16|17)$ deny

This rule contains a list of the "allowed" hours during which users can access the site. The hour format is based on the 24-hour clock, so 1 means 1 o'clock at night and 13 is used to represent 1 PM. The pipe character (|) is a feature of regular expressions that specifies that any of the hours can match—in effect making it an "or" operator.

There are three additional important characters in the above regex that we'd do well to explore a bit more. They are the exclamation mark (!), the caret (^) and the dollar sign. Let's start with the caret and dollar sign, since they are related. The caret matches the beginning of a string. If we hadn't used it in this example then the 8 would have matched both the hour 8 as well as the hour 18, which would have given users access to the site during the hour starting at 6 PM even though that wasn't our intention.

Similarly, the dollar sign ($) matches the end of a string. By preceding the list of allowed hours with a caret, and terminating it with a dollar sign, we make sure that only the listed hours will match, and so avoid any unpleasant surprises.

You may notice that preceding the list of allowed hours is an exclamation mark. This is used to negate the list of given operators. Since we want the rule to match when the hour is outside the list of allowable hours (and thus block the request), we use the exclamation mark to trigger the rule whenever the hour is outside the given range.

We could of course also have done away with the negation operator and simply specified the "forbidden" hours 0-7 and 18-23, but this would have created a slightly longer regular expression where we would have had to specify 14 separate hours instead of just the 10 in the example above.

An important point to consider is that the negation applies to the whole regular expression. Thus, the exclamation mark above does not apply solely to the first number (8) above, but to the entire regular expression that follows, namely the list of all the hours between 8 and 17.

Suppose you had a database on your network that contains customer information such as the names and addresses of customers as well as the credit card numbers they used when making purchases. It would be a bad thing if a hacker was able to get a hold of the credit card numbers stored in the database, so of course you would want to use best practices such as encrypted database files and a separate database server to store the card numbers.

However, suppose that in spite of all this, a hacker was able to leverage a programming error in your web site's administrative interface to get a hold of database records. If this were to happen, he could simply use a web browser to access credit card numbers, perhaps by using some clever SQL injection techniques. Fortunately, we can use ModSecurity as a last line of defense against this kind of disaster!

We do this by examining the HTTP response data that is sent back to clients. ModSecurity contains an operator called @verifyCC. It takes as an argument a regular expression. When this regular expression matches, the argument is passed to another algorithm to validate it as a credit card number. If the algorithm returns true we can block the response from being sent back, because it likely contains a credit card number. This is the way to write a rule to do that:

SecRule RESPONSE_BODY "@verifyCC d{13,16}" "phase:4,deny,t:removeWhitespace,log,msg:'Possible credit card number leak detected'"

1 8 8 1 8 8 8 8 8 8 8 8 2 1 0 4
x 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2
-------------------------------------------------
1 16 8 2 8 16 8 16 8 16 8 16 2 2 0 8

1+1+6+8+2+8+1+6+8+1+6+8+1+6+8+1+6+2+2+0+8 = 90

An IP address or a hostname by itself doesn't give much information about where in the world a visitor to your web site is located. Sure, a hostname such as host-18-327.92.broadband.comcast.net might give you a hint that the visitor is from the USA, but that only works with some hostnames and it's not very specific.

Enter geographical lookup databases. These map IP addresses to their geographical location. A company called MaxMind (http://www.maxmind.com) has such a database available, both in a free version and in a paid version. Their free database, GeoLite Country, is accurate enough for most applications (certainly when all you want to do is find out which country a visitor is from), and should you require greater accuracy you can purchase a license for their paid version, GeoIP Country.

So what does this have to do with ModSecurity? As of version 2.5, ModSecurity supports geographical location (or geolocation) of your visitors by referencing a geographical database such as the one published by MaxMind. This means that you can write rules that take into account where in the world your visitor is located. This is useful for many applications. If for example you processed credit card payments you could match the geographical location of the IP to the country in which the credit card was issued. If an American credit card is suddenly used in Taiwan, that should raise suspicions and potentially cause you to decline processing the order.

ModSecurity uses the GEO collection to store geographical information. Let's take a closer look at this collection and the fields it contains.

$ wget  http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz
$ gunzip GeoIP.dat.gz
$ mkdir /usr/local/geoip
$ mv GeoIP.dat /usr/local/geoip

SecGeoLookupDb "/usr/local/geoip/GeoIP.dat"

# Block users from China
SecRule REMOTE_ADDR "@geoLookup" "deny,nolog,chain"
SecRule GEO:COUNTRY_CODE "@streq CN"

# Block users from China, Russia and Pakistan
SecRule REMOTE_ADDR "@geoLookup" "deny,nolog,chain"
SecRule GEO:COUNTRY_CODE "@pm CN RU PK"

# Redirect European visitors to EU server SecRule REQUEST_URI "^/download/(.*)$"  "phase:1,capture,chain,  redirect:http://europe.example.com/download/%{TX.1}"
SecRule REMOTE_ADDR "@geoLookup" "chain"
SecRule GEO:COUNTRY_CONTINENT "@streq EU"

ModSecurity allows you to pause requests for a specified period of time. This is done via the pause action. This can be useful if for example you have detected suspicious behavior, such as a potential spammer submitting POST requests to a comment form at a rate far higher than normal.

To pause a request you specify the time, in milliseconds, that you want to delay it. If we wanted to pause any request after a user has submitted more than five POST requests within the last minute we could use the following rules:

SecAction "initcol:ip=%{REMOTE_ADDR},pass,nolog"
SecRule REQUEST_METHOD "@streq POST"  "pass,setvar:ip.posts_made=+1,expirevar:ip.posts_made=12"
SecRule IP:POSTS_MADE "@gt 5" "pause:5000"

This will pause POST requests for 5 seconds if a user has submitted more than five of them within a one-minute interval (and after the pause the request will be denied if that is what SecDefaultAction specifies). The expirevar action instructs the ModSecurity rule engine to expire the variable ip.posts_made after 12 seconds, so users can not submit more than five POST requests in a minute.

Take care when using pause, as it will cause an Apache child process to sleep for the specified amount of time. If you were under a Denial of Service attack, and the attacker submitted requests that caused a pause to occur, this could make your site go down more quickly than if the pause action had not been in place.

As an example, suppose that we wanted to execute a script to email us an alert message whenever an attempted SQL injection exploit was detected. To do this, we need two things:

For the script, we will use a standard shell script that invokes /bin/sh, though we could have easily used Perl or any other scripting language. We will email the alert to [email protected].

Create a file named email.sh in the directory /usr/local/bin and type the following in it:

#!/bin/sh
echo "An SQL injection attempt was blocked" | mail s "ModSecurity Alert" [email protected]
echo Done.

The script invokes the mail binary to send an email with the subject ModSecurity Alert to [email protected]. The last line of the script writes the string Done. to stdout. This is so that ModSecurity will recognize that the script has executed successfully.

We now have to make the script executable so that it can be invoked when a rule matches:

$ chmod a+rx /usr/local/bin/email.sh

Now all that is left is to create a rule that will trigger the alert script:

SecRule ARGS "drop table" "deny,exec:/usr/local/bin/email.sh"

You can now test out this rule by attempting to access http://yourserver/?test=drop%20table. If you've substituted your own email address in the example above you should get an email telling you that an SQL injection attempt has just been blocked.

Receiving such an email can be useful to quickly be alerted of any ongoing attacks. However, what if we wanted the email to contain a little more information on the attempted exploit; would that be possible? Yes, it's not only possible, it's also a very good idea, since more information about an alert can allow us to decide whether it is something to investigate more in-depth (such as when we detect that it's not just an automated vulnerability scanner pounding away at our server but actually a hacker probing for weaknesses with manually crafted exploit URLs).

ModSecurity allows us to set environment variables via the setenv action. By populating environment variables with suitable data we can record more information about the request that was blocked.

Suppose we would like to gather the following data when an attempted SQL injection is detected:

We will place this information in six separate environment variables, which we will call HOSTNAME, REMOTEIP, REMOTEHOST, REQUESTURI, ARGS, and UNIQUEID. Our modified rule now looks like this:

SecRule ARGS "drop table" "deny,t:lowercase,  setenv:HOSTNAME=%{SERVER_NAME},  setenv:REMOTEIP=%{REMOTE_ADDR},  setenv:REQUESTURI=%{REQUEST_URI},  setenv:ARGS=%{ARGS},  setenv:UNIQUEID={%UNIQUE_ID},  exec:/usr/local/bin/email.sh"

Now all we have to do is modify the email script so that it places the environment variables in the email body:

#!/bin/sh
echo "
An SQL injection attempt was blocked:
Server: $HOSTNAME
Attacking IP: $REMOTEIP
Attacking host: $REMOTEHOST
Request URI: $REQUESTURI
Arguments: $ARGS
Unique ID: $UNIQUEID
Time: `date '+%D %H:%M'`
" | mail s 'ModSecurity Alert' [email protected]
Echo Done.

As you can see, we use a multi-line echo statement to get all the information nicely formatted. Since this is a shell script, it will replace $HOSTNAME and the other environment variables with the value we set the variables to in our ModSecurity rule. The last line of the echo statement also adds a timestamp with today's date and the current time by invoking the date command and placing backticks (`) around it, which causes the shell to execute the command and substitute the command's output for it. Finally, the data is piped into the mail binary, which sends an email with the subject line ModSecurity Alert to the specified email address.

Again, at the end of the script we make sure to echo a dummy text to stdout to make ModSecurity happy. If you test this script you should get a nicely formatted email with all of the attacker's details.

ModSecurity makes it possible to solve problems that you thought were hard or impossible to solve using your standard web application. And often in a very elegant way, too.

A common problem webmasters face is counting the number of downloads of a binary file, such as an executable file. If the resource on the web server had been a normal web page, we could easily just add a server-side script to the page to update the download counter in a database. However, being binary, the file can be accessed and linked to directly, with no chance for any server-side script to log the download or otherwise take note that a download is taking place.

We will see how to create a ModSecurity rule that will invoke a shell script when a binary file is downloaded. This shell script contains some simple code to increment a download counter field in a MySQL database.

First, let's start by creating a new SQL database named stats and add a simple table to it that contains the columns date and downloads:

mysql> CREATE DATABASE stats; Query OK, 1 row affected (0.00 sec)
mysql> USE stats; Database changed
mysql> CREATE TABLE download (day DATE PRIMARY KEY, downloads INT); Query OK, 0 rows affected (0.05 sec)

The day column holds a date and the downloads column is the number of downloads in that day. So far so good—let's move on to the code that will update the database.

To get this right, we need to know a little about how modern web browsers and servers handle file downloads. The HTTP/1.1 protocol allows a client to specify a partial content range when performing a GET request. This range can be used to specify a byte interval of the file that the client wants to download. If successful, the server responds with HTTP status code 206—Partial content and sends only the data in the requested range to the client. For large files the web browser may perform tens or hundreds of GET requests with a partial content range specified before it has downloaded the entire file. This is useful because it allows a web browser or download manager to re-download the missing parts of an interrupted download without having to resort to downloading the file in its entirety again.

If we were to create a rule that triggers on any GET request for a particular file then a browser that uses partial content GET requests would increase the download counter many times for a single file download. This is obviously not what we want, and therefore we will write our rule to trigger only on requests that result in a standard HTTP 200—OK response code.

We will name the shell script that gets invoked /usr/local/bin/newdownload.sh. The shell script in newdownload.sh is a simple statement that invokes MySQL, passing it an SQL statement for updating the table:

#!/bin/sh
mysql -uweb -ppassword -e "INSERT INTO download  (day, downloads) VALUES (CURRENT_DATE, 1) ON DUPLICATE  KEY UPDATE downloads = downloads + 1;" stats

The ON DUPLICATE KEY statement is a construct special to MySQL. It instructs the database to ignore the INSERT statement and instead update the database field if the primary key already exists. In this way a row with today's date will get inserted into the database if it's the first download of the day (setting the download counter to 1), or updated with downloads = downloads + 1 if a row with today's date already exists. For this to work we must make the field day a primary key, which we did above when the table was created.

After creating the shell script, we need to make sure it is marked as executable:

# chmod a+rx /usr/local/bin/newdownload.sh

We will put this rule in phase 5, since this is the most certain phase to read response codes and we don't need to take any disruptive action. We use two chained rules since there are two conditions that need to be fulfilled—that the request URI is the path to our file, and that the response code is 200. We do a little smart optimization here by specifying the rule that matches the filename as the first rule in the chain. Had we instead looked at the response code first, both rules would have to be invoked every time the server generated a response code of 200—by doing it the other way around the second rule in the chain is only considered if the Request URI matches our filename.

The rules look like this:

SecRule REQUEST_URI "^/File.zip$"  "phase:5,chain,pass,nolog"
SecRule RESPONSE_STATUS 200  "exec:/usr/local/bin/newdownload.sh"

As we have learned previously, the ^ and $ characters are regular expression markers that match the start of, and end of, a line, respectively. Using them in this way ensures that only the File.zip found at the exact location /File.zip matches, and not any other file such as /temp/File.zip.

We use the pass action since we want to allow the request to go through even though the rule chain matches. Even though disruptive actions such as deny or drop cannot be taken in phase 5 (logging), we need to specify pass, or we would get an error message about the inherited SecDefaultAction not being allowed in phase 5 when we tried to reload the rules.

Now let's test our download counter. Upload any ZIP file to the web site root directory and name it File.zip. Then before we download the file for the first time let's make sure that the download table is empty:

mysql> USE stats; Database changed mysql> SELECT * FROM download; Empty set (0.00 sec)

Alright, now for the real test—we will download File.zip and see if the download counter is set to 1:

# wget http://localhost/File.zip --2009-01-29 16:47:11-- http://localhost/File.zip
Resolving localhost... 127.0.0.1 Connecting to localhost|127.0.0.1|:80... connected. HTTP request sent, awaiting response... 200 OK
Length: 100476 (98K) [application/zip]
Saving to: `File.zip'
shell scriptsfile downloads, counting2009-01-29 16:59:03 (142 MB/s) - `File.zip' saved [100476/100476]
mysql> SELECT * FROM download; +------------+-----------+
| day | downloads |
+------------+-----------+
| 2009-01-29 | 1 |
+------------+-----------+
1 row in set (0.00 sec)

Success! Now try downloading the file one more time and verify that the counter goes up to 2 and you will be sure that the shell script is working as intended.

I hope this example has showed you the power of ModSecurity and its exec statement. The shell script could easily be expanded to add additional functionality such as sending an email to you when the number of daily downloads of the file reaches a new all-time high.

Now let's see how we can use the IP collection to block a user from trying to brute-force a protected page on our server. To do this we need to keep track of how many times the user unsuccessfully tries to authenticate.

One attempt would be the following:

# This looks good, but doesn't work
SecAction initcol:ip=%{REMOTE_ADDR},pass SecRule REQUEST_URI "^/protected/" "pass,chain,phase:2"
SecRule RESPONSE_STATUS "^401$" "setvar:ip.attempts=+1"
SecRule IP:ATTEMPTS "@gt 5" deny

The intention of the above rules is that if someone tries an unsuccessful username/password combination more than 5 times for any resource under /protected, he will be denied access. We use the setvar:ip.attempts=+1 syntax to increase the counter each time an access attempt fails.

This looks good, but if you try it out you will find that it does not work. The reason is that when Apache notices a Require directive (which is what is used to password-protect a resource), it generates a 401—Authentication Required response and immediately sends it back to the client. This happens right after ModSecurity phase 1 (request headers) and causes the rule engine to immediately jump to phase 5 (logging). This is a caveat that applies to certain internal Apache redirects and also applies to 404—Not Found responses, so we need to work around it.

The solution is to also keep track of accesses to the resource where the response code is in the 200-299 range (meaning the response was successful). When we detect such a response on a protected resource we know that the client has authenticated successfully, and can set the counter to 0 so that he will not be blocked.

This is how the rules look with our new try:

# Initialize IP collection
SecAction "initcol:ip=%{REMOTE_ADDR},pass,phase:1"
# Track accesses to the protected resource
SecRule REQUEST_URI "^/protected/" "pass,phase:1,setvar:ip.attempts=+1"
# Was this an authenticated access? (Chained rule)
SecRule REQUEST_URI "^/protected/" "chain,pass,phase:3"
# Yes, user is logged in, set counter to 0 SecRule RESPONSE_STATUS "^2..$" "setvar:ip.attempts=0"
# Block if more than 5 non-authenticated access attempts
SecRule IP:ATTEMPTS "@gt 5" "phase:1,deny"

We put all of the rules that need to trigger on a 401—Authentication Required response in phase 1 so that the rule engine is able to process them. The above now works, but suffers from a shortcoming: If someone legitimately doesn't remember his password and tries various combinations more than five times, he will be locked out of the server for good. To solve this, we modify our previous rule so that in addition to increasing the counter, it also contains an expirevar action to expire the variable after a certain number of seconds:

SecRule REQUEST_URI "^/protected" "pass,phase:1,setvar:ip.attempts=+1, expirevar:ip.attempts=600"

We set the expiration time in seconds to 600, which equals ten minutes. This means that after five failed access attempts, any further requests will be blocked for ten minutes. If the attacker should return nine minutes after being blocked and try another password, the expirevar action will trigger again and reset the timer back to ten minutes. Any legitimate user who accidentally forgot his password would have to wait the full ten minutes before he would be given a further five attempts to remember his password.

The full rule listing to block access after five failed attempts with the reset on the block after ten minutes now looks like this:

# Initialize IP collection
SecAction "initcol:ip=%{REMOTE_ADDR},pass,phase:1"
# Track accesses to the protected resource
SecRule REQUEST_URI "^/protected" "pass,phase:1,setvar:ip.attempts=+1,expirevar:ip.attempts=600"
# Was this an authenticated access? (Chained rule)
SecRule REQUEST_URI "^/protected/" "chain,pass,phase:3"
# Yes, user is logged in, set counter to 0
SecRule RESPONSE_STATUS "^2..$" "setvar:ip.attempts=0"
# Block if more than 5 non-authenticated access attempts
SecRule IP:ATTEMPTS "@gt 5" "phase:1,deny"

If you think that the above solution looks like a bit of a hack then I agree. However, you need to be aware of and know how to work around problems like the one with the 401—Authentication Required response in the rule engine.

This chapter contained a lot of information, and you will no doubt want to refer back to it when writing rules. It will take a while to get used to the ModSecurity syntax if you haven't written rules before, so make sure you try out as many examples as possible and write rules of your own to get the hang of the process of creating new rules.

In this chapter we first looked at the basic SecRule syntax, and then learned how to match strings using either regular expressions or simple string comparison operators. We learned in which order the rule engine executes rules and why it's important to know about this to be able to write rules properly. We also learned about all the other things we need to know to successfully write rules such as transformation functions, macro expansion and the actions that can be taken when a rule matches.

In the second half of the chapter we looked at practical examples of using ModSecurity, including how to use a geographical database to locate visitors and how to execute shell scripts when a rule matches. We also saw how to intercept uploaded files and how to use Clam AntiVirus in conjunction with a shell script to scan uploaded files for viruses.

In the next chapter we look at the performance of ModSecurity and how to write rules so as to minimize any performance impact on our web applications.