Good logging is important for real-time incident detection and after-the-fact auditing. By religiously watching your log file, you will often get warnings that an outage is about to occur or that an attacker is analyzing your network for vulnerabilities. This allows you to take action to correct or prevent the problem. Log files also provide an audit trail for determining what went wrong or what an attacker accomplished.
Logging is a balance between collecting as much useful information as possible and not collecting so much information that it overwhelms you. An administrator overwhelmed by log files will ignore them until after an incident occurs. This negates the first benefit of log files—early warning. Many network administrators keep the default logging setting on routers and never take advantage of all of the additional information that can be logged. The rest of this chapter covers the logging capabilities of Cisco routers and discusses how to avoid being overwhelmed by your log files.
To achieve maximum benefit from log files, you must monitor them regularly. On systems of medium importance, log review can be done daily with the results emailed to the administrator. On highly secure systems, log analysis is often done in real time with the results sent to a pager.
Numerous commercial and public domain software packages can help you analyze your log files. Use them. These packages automate the process of analyzing log files by filtering out uninteresting messages and highlighting the dangerous ones. Such programs are the only way an administrator can keep up with log files to use them for incident detection.
One of the most useful ways to set up your log-watching programs is to have them:
Highlight everything you know is dangerous.
Ignore everything you know is safe.
Print out everything else.
Before you run the log checker the first time, go through your current logs and configure the software to highlight or ignore each different entry. Each day, run the log-check software, taking note of any highlighted (dangerous) messages. Next, for each message the software didn’t highlight or ignore, configure the software to do one or the other. After about two weeks of doing this daily, you will have a custom configuration that will automatically inform you of dangerous log messages while stripping out all extraneous ones. If all your routers log to a central syslog server, monitoring your logs becomes even easier, since everything can be monitored from one location and you need only a single log-checking configuration.
Cisco routers can log information in six ways:
Console log messages are configured to be displayed on the console port only. You must be connected to the console port to see these messages.
Buffered logging keeps the log messages in RAM on the router. A logging buffer must be configured on the router, and this buffer is circular, meaning that when it fills up, the oldest log message is deleted to make room for the new message.
Using the terminal monitor command, routers can be configured to send log messages to the VTY terminals.
Cisco routers can be configured to send their log messages to one or more external syslog servers.
If enabled, SNMP can be configured to send traps to an external SNMP server to log specific conditions.
If you are using AAA, you can configure the router to log network connections and even command execution to the Network Access Server (NAS).
Log messages are given a priority from 0 to 7. The lower the number, the more critical the message. These levels are defined as shown in Table 11-1.
Level | Title | Description |
0 | Emergencies | System is unusable. |
1 | Alerts | Immediate action is needed. |
2 | Critical | A critical condition has occurred. |
3 | Errors | An error condition has occurred. |
4 | Warning | A warning message. |
5 | Notifications | Normal but significant events. |
6 | Informational | Information messages. |
7 | Debugging | Debugging messages. |
These levels are a hierarchy in which messages of your chosen level and above are logged and printed. Choosing level 7 (debugging) will print out all messages from level 7 to 0, while choosing level 2 (critical) will only print out messages of levels 2 to 0.
The importance of accurate time was addressed in Chapter 10. It is also important to make sure that all log messages include detailed time information. Without this information, it becomes impossible to precisely correlate various log files to determine when an incident happened. To configure Cisco routers to include detailed time information in all log messages, use the service timestamps log datetime command. Use the following options to make sure as much information as possible is captured:
This option includes milliseconds in each log timestamp. Without this option, log messages are timestamped only to the nearest second.
This option configures the routers to use the local time to timestamp each log message. Doing so is generally advisable to make the logs easier for a human to interpret, but may be left out when routers across different time zones are logging to a central syslog server.
This option configures the router to indicate the time zone when timestamping each message. This step makes comparing log messages from various sources much easier.
To enable this detailed timestamping:
RouterOne#config terminal
Enter configuration commands, one per line. End with CNTL/Z. RouterOne(config)#service timestamps log datetime msec localtime show-timezone
RouterOne(config)#^Z
To see console messages, you must be logged into the console. The console is configured by default to level 5 (notifications), meaning that it only displays messages of notification level or above.
To change the logging level of the console, use the logging console command, followed by the number or name of the new severity level. To see all logging messages, set the console to level 7 (debug) by:
RouterOne#config terminal
Enter configuration commands, one per line. End with CNTL/Z. RouterOne(config)#logging console
7 RouterOne(config)#^Z
or
RouterOne#config terminal
Enter configuration commands, one per line. End with CNTL/Z. RouterOne(config)#logging console
debugging RouterOne(config)#^Z
As you begin logging more information, you may have trouble using the console as the display is covered up by numerous log messages. You can eliminate some of these messages by changing to a higher severity level using the commands described or you can disable logging to the console altogether. Since each console log message requires a CPU interrupt to be delivered, disabling logging in the console port can increase a router’s performance. Disabling console logging is done by:
RouterOne#config terminal
Enter configuration commands, one per line. End with CNTL/Z. RouterOne(config)#no logging console
RouterOne(config)#^Z
Buffered logging keeps copies of log messages in the router’s RAM. This makes it very easy for an administrator logged into the router to display current log messages. The RAM buffer is circular. The buffer is a fixed size, and as new messages are added to the buffer, the older messages are deleted. This keeps the most recent messages in the buffer without filling up the router’s memory.
The size of each router’s logging buffer should be configured with regard to the amount of RAM on the router. These buffers should be large enough to keep useful log entries, but small enough not to use up the router’s RAM and interfere with performance. A size of 16,000 or 32,000 bytes is usually sufficient and doesnt overwhelm the router. Play with the values if you find that you need log messages kept longer.
To enable buffered logging:
Make sure logging is turned on with the logging on command.
Configure the buffer with the logging buffered command.
Configure the logging buffer severity level with the logging buffered command.
The following example configures the router to use a 32,000-byte logging buffer and to log messages of level 6 (informational) and above:
RouterOne#config terminal
Enter configuration commands, one per line. End with CNTL/Z. RouterOne(config)#logging on
RouterOne(config)#logging buffered
32000 RouterOne(config)#logging buffered
informational RouterOne(config)#^Z
While logging is enabled to the console by default, log messages are not normally sent to the VTYs. If you are logged in through the network on a VTY and want to see log messages on your terminal, use the terminal monitor command:
Use the logging monitor command to configure the severity of messages sent to the VTY terminal.
At the enable prompt of the terminal, type terminal monitor
.
The following example configures the terminal to receive level 3 (error) messages and above:
RouterOne#config terminal
Enter configuration commands, one per line. End with CNTL/Z. RouterOne(config)#logging monitor error
RouterOne(config)#^Z
RouterOne#terminal monitor
Note that the terminal monitor command was run not from configuration mode, but rather from privileged or enable mode.
Turning on terminal monitor, especially
at the debug level, can overwhelm your connection, so be careful. If
you find your terminal session scrolling with so many log messages
that you are overwhelmed, use the terminal
no monitor command to stop the terminal from receiving
the log messages. Even if you can’t see what you are typing, try
typing terminal
no
monitor
and pressing Enter; the router
will still take the command and stop the log messages to your
terminal.
syslog logging is the most important non-AAA method of logging. All previous logging methods print the log message to a terminal or keep the log in memory. Neither of these records the log messages long term. Long-term storage is critical for keeping audit trails and is required for analyzing logs after messages have left the terminal screen or have buffered out of memory.
Almost all Unix servers come with syslog servers configured by default. Almost all security-conscious organization have one or two central syslog servers in which all logging information is sent and recorded. These messages are sent across the network to the syslog server where the server records the message to the hard drive. Centralizing your log files in this method makes correlating log information from different routers and systems much easier.
Remember to configure your router to send all log messages to a syslog server so that messages are permanently recorded and not lost when the router runs out of memory or messages scroll off the terminal screen.
syslog servers rely on the severity levels defined earlier and on another variable called the facility. syslog facilities separate different services so that log messages can be separated into different log files. This separation makes reviewing and auditing log files much easier. Table 11-2 lists some standard syslog facilities.
syslog facility | Description |
User | Log messages generated by user processes |
Kern | Log messages generated by the kernel |
Log messages regarding the mail system | |
Daemon | Log messages generated by system daemons |
Auth | Log messages regarding the authorization system |
LPR | Log messages regarding the printing system |
News | Log messages regarding Usenet news processes |
UUCP | Reserved for the UUCP system |
Cron | Log messages regarding the cron system |
Local0 | Reserved for your use |
Local1 | Reserved for your use |
Local2 | Reserved for your use |
Local3 | Reserved for your use |
Local4 | Reserved for your use |
Local5 | Reserved for your use |
Local6 | Reserved for your use |
Local7 | Reserved for your use (default Cisco log level) |
Cisco routers are usually configured to use one of the reserved facilities local0 through local7.
The syslog server must be configured to receive and record the log messages to a file. On almost all Unix systems, the syslog configuration file is /etc/syslog.conf. The standard format for syslog.conf files is (with tabs, not spaces):
Facility.Severity logfile
If you are unfamiliar with syslog, the easiest way to get a router logging to the syslog server is to use a facility that is already configured. However, this facility mixes your Cisco log messages with server log messages. The recommended way to get the router logged into the server is to configure a unique facility, such as local6 or local7, to send messages to a different file. For example, to configure the syslog server to send all log messages it receives that have a facility of local6 and a severity of informational to the file /var/log/cisco, your /etc/syslog.conf file might contain:
local6.info /var/log/cisco
You would then configure each router to use facility local6 when sending messages to the syslog server.
To configure syslog logging on your router:
Configure the syslog facility using the logging facility command.
Configure the syslog severity level with the logging trap command.
Configure the syslog server to which log messages will be sent with the logging command.
To configure your router to send log messages to the server
13.18.10.4
using facility
local6 and severity
informational:
RouterOne#config terminal
Enter configuration commands, one per line. End with CNTL/Z. RouterOne(config)#logging facility
local6 RouterOne(config)#logging trap
informational RouterOne(config)#logging
13.18.10.4 RouterOne(config)#^Z
You can specify additional syslog servers with additional logging <ip-address> commands and specify that log messages will be sent to each server for redundancy.
By default, many syslog servers accept messages from any IP address. While this won’t let an attacker modify log messages already recorded, it does allow an attacker to send bogus log messages that get mixed in with your good messages. It can also allow an attacker to use up all of the disk space on your syslog server so that valid log messages can’t be recorded because of lack of space.
It is important to make sure your syslog server does not accept syslog messages from systems external to your network. This can be accomplished by using ACLs at your network boundaries to deny all incoming syslog packets (UDP port 514) and/or by configuring the syslog server to accept log messages only from your internal network.
Cisco has recently added a feature that stamp each syslog message with a sequence number. These numbers increase by one for each syslog message sent. These sequence number can provide a way for an administrator to determine whether the log files have been tampered with. A log file that is missing a set of sequence numbers indicates that the log files are either incomplete or have been tampered with. This example enables syslog sequence numbers:
RouterOne#config terminal
Enter configuration commands, one per line. End with CNTL/Z. RouterOne(config)#service sequence-numbers
RouterOne(config)#^Z
Sequence numbers do little if intruders accesses the syslog server—they can simply resequence the messages in the log file. However, sequence numbers can be useful in determining whether bogus messages are being forwarded to a syslog server. A log file that contains log messages without sequence numbers or with duplicate sequence numbers indicates that someone is sending bogus message to your syslog server.
IOS Version 12.1(3)T introduced the ability to limit the number of syslog messages sent to and from the router in a specific time frame. Some logging, such as ACL violation logging, if not implemented correctly, can cause the router to overwhelm the syslog server with thousands of messages a second. You can limit the number of messages sent with the logging rate-limit command. This example limits message output to 10 messages a second:
RouterOne#config terminal
Enter configuration commands, one per line. End with CNTL/Z. RouterOne(config)#logging rate-limit all
10 RouterOne(config)#^Z
You can append the except keyword followed by a severity level to rate-limit only low-priority messages.
For example, to avoid rate-limiting messages of severity error or higher, enter:
RouterOne#config terminal
Enter configuration commands, one per line. End with CNTL/Z. RouterOne(config)#logging rate-limit all
10except
error RouterOne(config)#^Z
If you run SNMP (see Chapter 8), you can use SNMP traps to log additional information. Traps are packets sent to an SNMP server when specific events occur. Such events include high temperatures, configuration changes, and down interfaces. If you are familiar with SNMP and want to enable SNMP traps on your router, you must:
Use the snmp-server host command to configure which SNMP server will receive the traps.
Use the snmp-server enable traps command to enable SNMP traps.
The following example configures the router to send traps to the
SNMP server 13.145.6.5
:
RouterOne#config terminal
Enter configuration commands, one per line. End with CNTL/Z. RouterOne(config)#snmp-server host
13.145.6.5 public RouterOne(config)#snmp-server enable traps
RouterOne(config)#^Z
This example configures the router to send all possible traps to the SNMP server. You can supply additional arguments to the snmp-server enable traps command to limit the traps sent. See Cisco’s documents on SNMP traps for more information.
Aside from system logging, ACL violation logging is an important area to configure. By logging ACL violations, you can record any time your ACLs block access. This is extremely useful for determining when attackers are trying to spoof, log in, or access your router from external or untrusted systems.
ACL violation logging can be enabled by simply adding the keyword log or log-input to the end of your ACL statements. Be selective about what you choose to log. Adding the log keyword to the wrong filters can end up logging every packet in and out of your networks.
The log keyword provides information on the type, date, and time of the ACL violation and is the only option for standard ACLs. The log-input keyword is available on extended ACLs and logs additional information about the input interface and source MAC address.
The log keyword works only for standard access lists. Both the log and log-input keywords can be used for extended and named ACLs.
The following examples configure your ACL antispoofing
filters to log both inbound and outbound spoofing attempts. Assuming
our internal network was 130.18.0.0/16
, an ingress filter that logs
all spoofed packets looks like:
RouterOne#config terminal
Enter configuration commands, one per line. End with CNTL/Z. RouterOne(config)#access-list
115 deny ip 130.18.0.0 0.0.255.255 any log-input RouterOne(config)#access-list
115 permit ip any any RouterOne(config)#^Z
This filter would be placed inbound on each interface that borders an external network. Assuming this was Serial 0/0 on RouterOne, enter:
RouterOne#config terminal
Enter configuration commands, one per line. End with CNTL/Z. RouterOne(config)#int
Serial 0/0 RouterOne(config-if)#ip access-group
115 in RouterOne(config-if)#^Z
An egress filter that logs all attempts at sending spoofed packets out of the network would look like:
RouterOne#config terminal
Enter configuration commands, one per line. End with CNTL/Z. RouterOne(config)#access-list
116 permit ip 130.18.0.0 0.0.255.255 any RouterOne(config)#access-list
116 deny ip any any log-input RouterOne(config)#^Z
And would be applied to all outgoing interfaces:
RouterOne#config terminal
Enter configuration commands, one per line. End with CNTL/Z. RouterOne(config)#int
Serial 0/0 RouterOne(config-if)#ip access-group
116 out RouterOne(config-if)#^Z
Logging router login attempts through VTYs is very
important. It can give you an early indication that someone is trying
to access your router without authorization. Assuming you want only
the IP 130.18.5.6
to be able to
access the router through VTYs, your ACL would look like:
RouterOne#config terminal
Enter configuration commands, one per line. End with CNTL/Z. RouterOne(config)#access-list
117 permit ip host 130.18.5.6 any RouterOne(config)#access-list
117 deny ip any any log-input RouterOne(config)#^Z
You would apply it the VTY lines 0 through 4 with:
RouterOne#config terminal
Enter configuration commands, one per line. End with CNTL/Z. RouterOne(config)#line vty 0 4
RouterOne(config-line)#access-class
117 in RouterOne(config-line)#^Z
Now someone from an unauthorized IP trying to access a router VTY will generate a log message. The following message is an sample log entry of an unauthorized person trying to telnet to the router:
Oct 13 21:10:44.185 EDT: %SEC-6-IPACCESSLOGP: list 120 denied tcp 19.8.59.41(63104) -> 0.0.0.0(23), 1 packet
Many sites also choose to log all permitted VTY access attempts in addition to blocked access ones. Since all VTY access is through TCP, you can use the ACL keyword established to log successful access attempts without overwhelming your router or syslog server. This can be done effectively if you:
Pass, without logging, all established TCP connections from permitted IPs.
Pass and log the first packet for all TCP sessions from permitted IPs.
Deny and log all other packets.
This example logs both access attempts from approved IPs and access attempts from unapproved IP addresses:
RouterOne#config terminal
Enter configuration commands, one per line. End with CNTL/Z. RouterOne(config)#access-list
117permit tcp host
130.218.5.6any established
RouterOne(config)#access-list
117permit tcp host
130.218.5.6any log-input
RouterOne(config)#access-list
117deny ip any any log-input
RouterOne(config)#^Z
AAA accounting allows you to log additional information that cannot be obtained from the previous logging methods. Using Cisco’s TACACS+, AAA accounting can log every command executed on the router to the Network Access Server (NAS).
There are five methods of AAA accounting—EXEC, System, Command, Connection, and Network. Here is a short description of each method:
EXEC accounting records information about each EXEC, or shell, session on the router. It records information such as username, date, time, and IP of the system connecting to the router.
System accounting logs information about system-level events, such as reboots or when accounting is turned on or off.
Command accounting logs information about the commands typed at the EXEC shell prompt. It records information (including what command was executed, who executed the command, the privilege level, and the date and time). Please note that older versions of Cisco’s implementation of RADIUS do not support Command accounting, so you will need to use TACACS+ or upgrade.
Connection accounting logs information about outgoing connections made from the router. These connections include telnet, rlogin, tn3270, and LAT.
Network accounting logs information about PPP, SLIP, and ARAP sessions.
To log the largest amount of useful information, you should turn on accounting for all five methods. Each method supports three types of logging:
The start-stop keyword configures the router to generate a log message when a service starts and stops. For example, using the start-stop keyword for EXEC, accounting generates a log message when a user first accesses the router and another log messages when the user disconnects from the router.
The stop-only keyword configures the router to generate log messages only when a service ends. With EXEC accounting, this keyword generates log messages only when a user disconnects from the router.
The wait-start keyword delays the start of the requested service until confirmation is received from the NAS server that a log message has been received. This is typically reserved for higher-security installations that want to guarantee that each connection and command that run on a system is recorded. If the NAS server doesn’t indicate that it successfully received the log message, the router will not start the requested service or command.
Recommended initial settings for each method are shown in Table 11-3.
Method | Recommended starting type | Explanation |
EXEC | start-stop | This method makes determining exactly when someone accessed the router and exactly when they disconnected much easier. |
System | stop-only | System messages are usually atomic events that don’t need the start and stop times recorded. The stop-only keyword is usually sufficient for System accounting. |
Command | stop-only | Commands are usually run in such short time periods that start and stop times are of limited use and can double the amount of log messages you have to analyze. |
Connection | start-stop | Logging the start and stop times of outgoing connections from the router makes analyzing log files easier. |
Network | start-stop | Again, logging start and stop times of these network connections makes log file analysis less of a chore and more intuitive. |
The rest of this section provides examples on configuring each logging method.
If you have AAA configured to use TACACS+, you can enable EXEC, System, Connection, and Network accounting to your NAS server with:
RouterOne#config terminal
Enter configuration commands, one per line. End with CNTL/Z. RouterOne(config)#aaa accounting exec default
start-stopgroup
tacacs+ RouterOne(config)#aaa accounting system default
stop-onlygroup
tacacs+ RouterOne(config)#aaa accounting connection default
start-stopgroup
tacacs+ RouterOne(config)#aaa accounting network default
start-stopgroup
tacacs+ RouterOne(config)#^Z
To configure Command accounting, you also need to supply the privilege level you want logged. Unless you have changed the default privilege levels, privilege levels 1 and 15 should capture everything. If you have custom privilege-level configurations, make sure you add commands to log all levels you are interested in:
RouterOne#config terminal
Enter configuration commands, one per line. End with CNTL/Z. RouterOne(config)#aaa accounting commands
1default
stop-onlygroup
tacacs+ RouterOne(config)#aaa accounting commands
15default
stop-onlygroup
tacacs+ RouterOne(config)#^Z
If you have AAA configured to use RADIUS, you can enable EXEC, System, Connection, and Network accounting to your NAS server with:
RouterOne#config terminal
Enter configuration commands, one per line. End with CNTL/Z. RouterOne(config)#aaa accounting exec default
start-stopgroup
radius RouterOne(config)#aaa accounting system default
stop-onlygroup
radius RouterOne(config)#aaa accounting connection default
start-stopgroup
radius RouterOne(config)#aaa accounting network default
start-stopgroup
radius RouterOne(config)#^Z
You also need to enable Command accounting for all privilege levels you are interested in. Older versions of Cisco’s IOS don’t support Command accounting through RADIUS. If you determine that your router doesn’t support RADIUS command logging, then either migrate to TACACS+ or upgrade your IOS:
RouterOne#config terminal
Enter configuration commands, one per line. End with CNTL/Z. RouterOne(config)#aaa accounting commands
1default
stop-onlygroup
radius RouterOne(config)#aaa accounting commands
15default
stop-onlygroup
radius RouterOne(config)#^Z
AAA also needs to be configured to log authentication failures. The following example will generate an AAA log message each time someone attempts to access the router but fails:
RouterOne#config terminal
Enter configuration commands, one per line. End with CNTL/Z. RouterOne(config)#aaa accounting send stop-record authentication failure
RouterOne(config)#^Z
This checklist summarizes the important security information presented in this chapter. A complete security checklist is provided in Appendix A.
Actively monitor all logs for indications of attacks, misconfigurations, and failures.
Configure logging timestamps to include milliseconds using the service timestamp log datetime msec localtime command.
Enable RAM buffer logging with the logging buffered command. The default and recommended level is 6 (informational).
Enable logging sequence numbers with the service sequence-numbers command.
Configure routers to send log messages to a syslog server to preserve the messages:
Make sure that sites requiring higher levels of security and auditability send router log messages to multiple syslog servers for redundancy.
Filter out syslog messages from external systems through ACLs at your network’s border or with the syslog server itself.
Configure key ACLs to record access violations. Recommended ACL logging includes:
Antispoofing violations
VTY access attempts
HTTP access attempts
SNMP access attempts
Route filter violations
ICMP violations
Any other important filters
In environments requiring additional security, use AAA and enable AAA accounting:
Configure EXEC, System, Connection, and Network accounting to record information on system events and user sessions.
Configure AAA accounting to record authentication failures.
If a record of each command executed on the router is required, configure command accounting.
18.225.55.151