If you are under attack, it is very important to get a picture of what your attacker is trying to do. Is he using a pre-packaged script to try to get into your server? Is it just a bot hammering away using known exploit code? Or is someone attempting to hack in by using handcrafted SQL injection requests via a proxy server in a foreign country?
Perusing logs of ModSecurity alerts on a regular basis is important to see what kind of exploits are being tried against your server—in some cases you may find that there's a new vulnerability out there that you need to patch against simply by paying some attention to the generated log data.
The standard Apache log does not give much more information than the time and date of a request, and the first line of the request (that is you'll see what resource the GET or POST was made to, but not much more than that). ModSecurity introduces audit logging, which gives you the ability to log much more detailed information about the requests made to your server. Using audit logging, you can get information on the request headers and request body, as well as information on the response headers and body and all the rules that matched the request.
In this chapter, we will learn how audit logging works, and also take a look at a very helpful tool called the ModSecurity Console, which provides a web-based interface to viewing audit logs and generating reports from the log data.
The audit logging capabilities of ModSecurity are switched off by default. You can enable the audit log engine by placing a SecAuditEngine directive in the ModSecurity configuration file. Here are the possible values for SecAuditEngine:
In most cases you will probably want to use SecAuditEngine RelevantOnly to only log those transactions that are actually considered relevant—that is those that match a ModSecurity rule or have a relevant HTTP status code. Using the On parameter instead would enable logging for all transactions which can use up a lot of disk space as well as slow down the server if it is under heavy load.
The SecAuditLogRelevantStatus directive takes as a parameter a regular expression that is matched against the HTTP response code for the transaction. So to log transactions that generate an HTTP error (status code 400-599), you would use the following:
SecAuditEngine RelevantOnly SecAuditLogRelevantStatus ^[45]
If the SecAuditLogRelevantStatus directive is not configured then the SecAuditEngine RelevantOnly setting will still log any transactions that match a ModSecurity rule.
There are two types of audit log formats that can be used: serial and concurrent. Serial logging logs all audit log events to a single file whereas concurrent logging places the log data for each request into a separate file.
The type of logging to be done is configured via the SecAuditLogType directive, which takes the value serial for serial logging, and concurrent for multiple-file logging.
There are advantages to both types of logging. With serial logging, all the audit log data is conveniently available in a single file. However, serial logging is slower and it is more cumbersome to parse the serial log file if you want to do further processing on each logged event. Concurrent logging, on the other hand, places log files in separate directories corresponding to the time that the log was generated. This can make it easier to parse the logs using automated tools. As we will see later, concurrent logging is also the required setting to forward audit log data to a ModSecurity console.
When using concurrent logging, we need to configure the directory where ModSecurity will create the individual log files. This is done using the SecAuditLogStorageDir directive, like so:
# Set the directory for concurrent file logging SecAuditLogStorageDir /var/log/audit/
It is important that the specified directory exists before Apache is restarted, and that it is writable by the Apache user. If Apache can't create files in this directory, no audit logging will take place, and the only way you will be able to find out why is to enable debug logging, so make sure you set the permissions appropriately.
The SecAuditLogParts directive controls which information is included in each audit log entry. The directive takes a string of characters as an argument and each character represents one part of the log data.
These are the characters available together with an explanation of which part of the transaction they represent:
|
Character |
Description |
|---|---|
|
A |
Audit log header Boundary that signifies the start of the audit log entry. Contains the time and date stamp of the log entry as well as the client and server IP address. Also contains the unique ID for the log entry, which makes it easy to find the request in the Apache log files. This option is mandatory and will be implicitly included if you don't specify it. |
|
B |
Request headers Contains all of the headers in the request, as sent by the client. |
|
C |
Request body Contains the request body. Only available if request body access is enabled in ModSecurity. |
|
E |
Response body Contains the response body of the request. Only available if response body access is enabled in ModSecurity. If the request was denied by a rule, this instead contains the error page sent to the client. |
|
F |
Response headers Contains the response headers, excluding the date and server headers as these are added late in the response delivery process by Apache. |
|
H |
Audit log trailer Contains information on whether the request was allowed or denied, and the relevant HTTP status code as well as the ModSecurity message as it appears in the Apache error log. Also contains a timestamp and the server string (as it would appear without any of the modifications that may have been made to it using |
|
I |
Request body without files Contains the same information as C—the request body—except when the encoding used is |
|
K |
Matched rules A list of all rules that matched this event, one per line, in the order that the rules matched. Each listed rule includes any default action lists. |
|
Z |
End of audit log entry Boundary that signifies the end of the audit log entry. This option is mandatory and will be implicitly included if you don't specify it. |
So for example to log the request headers, request body, response headers, and audit log trailer you would use the following configuration directive:
SecAuditLogParts ABCFHZ
The following is a summary of the typical configuration to enable audit logging and setting the log type to serial:
# Enable serial audit logging SecAuditEngine RelevantOnly SecAuditLog logs/modsec_audit.log SecAuditLogType serial SecAuditLogParts ABCFHZ
Now, let's take a look at what an audit log entry looks like. The following entry was generated with the above configuration, and shows details relating to a denied request to access the URI /test on the server at www.bytelayer.com.
--5759e83f-A-- [27/Mar/2009:14:22:32 +0000] dqIu7V5MziQAAEpPAWwAAAAE 94.76.206.36 38037 94.76.206.36 80 --5759e83f-B-- GET /test HTTP/1.0 User-Agent: Wget/1.11.1 (Red Hat modified) Accept: */* Host: www.bytelayer.com Connection: Keep-Alive --5759e83f-F-- HTTP/1.1 403 Forbidden Content-Length: 275 Connection: close Content-Type: text/html; charset=iso-8859-1 --5759e83f-H-- Message: Access denied with code 403 (phase 2). Pattern match "test" at REQUEST_URI. [file "/etc/httpd/conf.d/mod_security.conf"] [line "34"] Action: Intercepted (phase 2) Stopwatch: 1238163752365805 926 (481 695 -) Producer: ModSecurity for Apache/2.5.7 (http://www.modsecurity.org/). Server: Apache/2.2.8 (Fedora) mod_jk/1.2.27 DAV/2 --5759e83f-Z--
Each log part starts with a separator of the form --5759e83f-A--. The separator begins and ends with two dashes. The hexadecimal string is a unique identifying string for this audit log entry. The uppercase letter before the last two dashes corresponds to the audit log part character, as given in the previous table.
The different audit log parts are all present as configured via SecAuditLogParts (except for the request body, which is empty since this is a GET request). For example, the heading --5759e83f-B-- is followed by the request headers as sent by the client.
Concurrent logging logs the same information as serial logging—the difference is that each log entry is placed in a separate file. The following is a typical configuration to enable concurrent logging:
# Enable concurrent audit logging SecAuditEngine RelevantOnly SecAuditLogType concurrent SecAuditLogStorageDir /var/log/audit/ SecAuditLog logs/modsec_audit.log SecAuditLogParts ABCFHZ
With concurrent logging, the main audit log file acts as an index file, pointing to the individual log files.
ModSecurity will create a specific directory structure in which the individual log files are placed. The directory structure looks as follows:
/var/log/audit/ |-- 20090331 | |-- 20090331-1530 | | |-- 20090331-153030-Cei44F5MziQAAFKTAIcAAAAA | | |-- 20090331-153030-Cei5115MziQAAFKUAM0AAAAB | | |-- 20090331-153030-CgEmHV5MziQAAFKVAS0AAAAC | | |-- 20090331-153054-C1JA815MziQAAFKoBfIAAAAV | | `-- 20090331-153054-C1JIqV5MziQAAFKVAS4AAAAC | |-- 20090331-1531 | | |-- 20090331-153100-C6skpV5MziQAAFKUAM4AAAAB | | |-- 20090331-153105-C@gA6F5MziQAAFKgBD0AAAAN | | |-- 20090331-153109-DBTxLV5MziQAAFKhBKAAAAAO | | `-- 20090331-153118-DLyqv15MziQAAFKeA8AAAAAL | |-- 20090331-1532 |-- 20090401 | |-- 20090401-0208 | | |-- 20090401-020802-8H1Ox15MziQAAFPGEv4AAAAI | | |-- 20090401-020802-8esmr15MziQAAF0tGXAAAAAD | | |-- 20090401-020805-8RWbIF5MziQAAFNbCN8AAAAy ...
ModSecurity generates a new directory for each day (for example the 20090331 directory, which contains log files generated on the 31st of March, 2009). Each of these directories then contains a separate subdirectory for those minutes of the day when log entries were generated (20090331-1530 is the first of those directories in the tree above, and contains all log files for requests generated at 3:30 in the afternoon on March 31st). Each individual log entry is then contained within its own file, which has a filename consisting of the date, time, and unique ID for the request.
To make sure that certain rules do not trigger logging, we can use the nolog and noauditlog directives. The nolog directive causes a match of the current rule to not be considered a criterion for writing log data either to the Apache error log or the ModSecurity audit log. Similarly, the noauditlog directive causes a match of the current rule to not be considered a criterion for recording the request in the audit log.
For both the nolog and noauditlog directives, a rule that has matched before or after the current rule can still trigger logging for the transaction. To disable logging for all rules in a transaction, use the directive ctl:auditEngine=off.
ModSecurity includes actions to sanitize audit log data. The purpose of this is to prevent things such as user passwords from showing up in the audit logs.
These are the sanitization actions that ModSecurity supports:
|
Action |
Description |
|---|---|
|
|
Sanitizes the named argument value of a |
|
|
Sanitize the variable that caused the rule to match. This can be either a request argument, request header or response header. |
|
|
Sanitize named request header. |
|
|
Sanitize named response header. |
As an example, if a web page accepted an argument named "password" and it also matched a ModSecurity rule then the following would make sure that the password is replaced by asterisks when data is written to the audit log:
SecRule login.php allow,auditlog,sanitiseArg:password
Accessing /login.php?password=123456 on the server would result in the following request header part being written to the audit log:
--e8d98139-B--
GET /login.php?password=****** HTTP/1.1
Host: bytelayer.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Cookie: JSESSIONID=4j4gl8be129l6
In the above log entry, the password has been replaced by asterisks (*) which prevents those viewing the audit log from finding out the provided password. The sanitization does not apply to the debug log, as the full data will always be available there.
We can also use sanitiseMatched to achieve the same effect. The following will sanitize the value provided for the password argument:
SecRule ARGS_NAMES password allow,auditlog,sanitiseMatched
Finally, we can instruct ModSecurity to sanitize a named request or response header. Suppose that we didn't want persons with access to the audit log to be able to view cookie data sent by a client. We could use the following to sanitize the cookie information in the logs:
SecAction phase:1,allow:phase,sanitiseRequestHeader:Cookie,nolog
The above rule works in phase 1, and allows access to the current phase (so that any later rules get a chance to deny the request). Note the nolog directive, which instructs ModSecurity that a successful rule match against this rule should not by itself mean that the request gets written to the error or audit log. Since SecAction implies an unconditional match, all requests would have been logged if we had left the nolog action out. As it is, the effect of the above rule will only get triggered if another rule causes data to be written to the audit log, in which case it will sanitize any cookie information in the request headers.
The log data we have seen so far can be tedious to look at, as it will most likely require you logging into the server and manually examining the various log files. In particular if you have many servers running ModSecurity you would probably not want to manually examine the log files on each one to determine what attacks, if any, your servers have blocked.
Luckily, there is an excellent tool called the ModSecurity Console that allows you to view audit logs using your web browser. The console is able to collect audit log data from several servers running ModSecurity—each server that provides log data to ModSecurity is referred to as a sensor.
The console has a number of attractive features that greatly simplify the viewing and management of audit logs:
Overview of all sensors, including the number of unhandled (active) alerts on each
Ability to view detailed information about each event, including the full request headers and body, IP address of the client that generated the event, and information on which ModSecurity rule triggered the alert
Email reports can be sent for alerts that you consider serious enough
Scheduled reporting allows you to receive emails with an overview of generated alerts on a daily, weekly, or monthly basis
The console is provided as a binary file that requires the Java runtime environment (JRE) version 1.4 or later to function, so before we begin you should make sure that you have this installed. You can obtain the Java runtime environment from http://www.java.com/en/download/manual.jsp.
The ModSecurity Console contains a built-in web server, so there is no need to integrate it with Apache or any other server—simply running the executable will start up a web server on port 8888 where the console data can be viewed using any web browser.
You can download the console from http://www.breach.com/products/ModSecurity-Community-Console.html. You will have to fill out a form providing some details about yourself, and after doing this and accepting the license agreement for the console, you are presented with a download page where you have the choice of downloading either an RPM or a .tar.gz file of the console. We will be using the .tar.gz archive.
Once you have the download link to the .tar.gz file for the console, simply use wget or a similar tool to download the archive to your server:
$ cd /home/download $ wget http://www.breach.com/resources/modsecurity/ downloads/modsecurity-console.tar.gz
Note
The above is not a functioning download link—substitute the URL that you find on the download page for the above one.
Make sure you also follow the link to the license for the console—the console is not open source software, however Breach Security are offering a free perpetual license to it that allows for up to three sensors to be used. The license is simply a block of text that needs to be pasted in the appropriate configuration edit box once the console has been installed.
The next step is to unpack the archive:
$ tar xfvz modsecurity-console.tar.gz
The above command unpacks the archive into a folder named modsecurity-console. This folder contains the ModSecurity Console executable as well as configuration files. It's a good idea to move the folder to a more permanent location:
$ mv modsecurity-console /opt/
The ModSecurity Console executable can now be started with the following command:
$ /opt/modsecurity-console/modsecurity-console start
Being a stand-alone Java program, the console will continue running even if you shut down Apache. If there is any firewall protecting your server, make sure you enable access to TCP port 8888, as that is the default port used by the console. If necessary, you can change the port that the server listens on by modifying the console configuration file, available in /opt/modsecurity-console/etc/console.conf, and restarting the console.
Once you have started the console executable, you can access the ModSecurity Console by visiting the URL https://yourserver:8888/ in a web browser. Note the https in the URL—it's important that you specify this as the console web server is configured to only allow HTTPS access out of the box. If necessary, you can change the protocol used to standard HTTP in the console settings under Administration | Web Server Configuration once you have logged in.
Use the default username admin and password admin when logging in for the first time. You may get a security warning from your web browser since the server uses a self-signed certificate for the secure connection; however, it is perfectly safe to ignore this.
This is what the ModSecurity Console looks like after you log in for the first time:
As you can see there is a warning message urging us to change the default password—make sure you do this after setting up the console as the default credentials are the same for all installations and could easily allow anyone to access the console if these are left unchanged.
There are three main headings in the console window home page:
Sensor Overview
Shows information about the sensors (that is, servers running ModSecurity) that are sending data to the console, including statistics on the number of active alerts on each sensor. (An active alert is one that has not been dismissed by selecting a "resolution" in the alert screen.) The sensor overview also showsthe highest severity among the alerts.
Recently Observed Transactions
Shows you the transactions (alerts) that you have recently viewed
Administrative Events
Displays any administrative events, such as someone trying to log into the console with an incorrect username/password combination or console configuration errors that need to be corrected
After logging in, you will notice that there are no alerts since we haven't configured ModSecurity to forward audit logs to the console yet. To do this, we need to install and configure a program called mlogc—short for ModSecurity Log Collector. mlogc is distributed together with the main ModSecurity source code—once we have compiled it we can use it together with the SecAuditLog directive to forward audit logs to the console.
mlogc requires the curl-devel package—it uses this to communicate with the console when it sends its log data. You can install the package using your favorite package manager. Once this is installed, run the following commands to compile mlogc:
$ cd /home/download/modsecurity-apache/ $ ./configure $ make mlogc make[1]: Entering directory `/home/download/modsecurity-apache/apache2/mlogc-src' Building dynamically linked mlogc... Build finished. Please follow the INSTALL instructions to complete the install. make[1]: Leaving directory `/home/download/modsecurity-apache/apache2/mlogc-src' Successfully built "mlogc" in ../tools. See: mlogc-src/INSTALL
It is important to run configure again if you did not have the curl-devel package installed when you compiled ModSecurity. If you don't do this, you may get error messages relating to curl as the compiler won't be able to find the curl library files.
Once the compilation finishes, the mlogc binary will be found in the tools subdirectory of the root directory for the ModSecurity source code.
Now copy the mlogc binary to a more permanent location so that it is available for use by ModSecurity:
$ cp /home/download/modsecurity-apache/tools/mlogc /usr/local/bin/
The log collector needs some basic configuration, such as which server to submit the audit log data to, as well as which directory is used to store the log files.
There is a sample configuration file for mlogc called mlogc-default.conf in the mlogc-src directory. Let's copy this file to /etc/, renaming it mlogc.conf:
$ cp /home/download/modsecurity-apache/apache2/mlogc-src/mlogc-default.conf /etc/mlogc.conf
The following are the lines that need to be modified in the new /etc/mlogc.conf file to get the log collection working:
CollectorRoot /var/log/mlogc
This is the root directory that mlogc will use for its files. We will be creating this directory and setting permissions on it in the next section, but for now we'll just configure it to the above.
The next line to modify sets the IP address for the console:
ConsoleURI "https://CONSOLE_IP_ADDRESS:8888/rpc/auditLogReceiver"
Change CONSOLE_IP_ADDRESS to the IP address of the server where the ModSecurity console is running. If the ModSecurity Console is running on the same server as the sensor, use 127.0.0.1 for the IP address, which is the address for localhost.
You also need to edit the following two lines so that they match the credentials for the sensor as configured in the ModSecurity console:
SensorUsername "SENSOR_USERNAME" SensorPassword "SENSOR_PASSWORD"
Now all that's left to do is create the mlogc root directory and tell ModSecurity to use mlogc to forward its audit log data to the console.
The first thing to do is create the directories where mlogc will store the audit log data it receives from ModSecurity before submitting it to the console:
$ mkdir /var/log/mlogc $ mkdir /var/log/mlogc/data $ chown apache:apache /var/log/mlogc $ chown apache:apache /var/log/mlogc/data
The /var/log/mlogc/data directory is where mlogc expects to find the individual log files, so we need to change the SecAuditLogStorageDir directive in the main ModSecurity configuration to point to this directory:
SecAuditLogStorageDir /var/log/mlogc/data
The final step is to change the SecAuditLog directive to invoke mlogc instead of writing to a plain index file:
SecAuditLog "|/usr/local/bin/mlogc /etc/mlogc.conf"
Now restart Apache to apply the new settings. When mlogc is invoked, it fetches the specified audit log files and forwards them to the ModSecurity Console in real time, where they will be immediately available for viewing. You can now verify this by triggering a rule match and checking the ModSecurity Console for a corresponding alert.
In this chapter, we looked at how audit logging works in ModSecurity. We learned how to configure audit logging in ModSecurity and about the difference between serial and concurrent logging. We learned that audit log sanitization actions can be applied to prevent certain information from showing up in the audit logs, and we learned how to disable logging for specific rules or HTTP requests.
The last half of the chapter was devoted to the ModSecurity Console which is an excellent tool to collate and view log data. We learned how to use the console as well as how to send log data to the console using mlogc.
In the next chapter we will be looking at virtual patching—a technique to block newfound vulnerabilities without having to rely on the vendor to supply a software update.