ICMP Misconceptions

The traffic generated by pings and traceroutes uses the much-maligned ICMP, with UDP to a lesser extent for DNS lookups if required.

However, it's worth mentioning that ICMP was created for very good reasons and is still used for very important tasks in the day-to-day operation of the Internet. For example, it's needed to tell devices the size to set their Maximum Transmission Unit (MTU) in order to allow packets to traverse smoothly across heterogeneous network links. As a result, after reading what follows, you should avoid committing one of the most common junior sysadmin mistakes: blocking all ICMP traffic to your servers.

tcpdump

Let's have a cursory look at your options for monitoring pings and traceroutes around the clock. Straight from the sysadmin's tool kit of reliable utilities, you might consider the powerful packet-sniffing tool, tcpdump. This tool has long been trusted to split traffic into smaller pieces in order to offer an insight into what is travelling across a network.

For example, if you want to pick up pings, then the following command works when I ping my paranoid server from another machine:

# /usr/sbin/tcpdump -i eth0 icmp and icmp[icmptype]=icmp-echo

This next example, showing tcpdump's ICMP packet-sniffing abilities, also picks up traceroutes:

# /usr/sbin/tcpdump ip proto ∖∖icmp

As you can see in the following code, pings rely on both replies and requests; however, because my paranoid server's firewall is blocking certain ICMP traffic, an admin prohibited error is logged when traceroutes appear.

listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
17:06:47.925923 IP recce.chrisbinnie.tld > noid.chrisbinnie.tld: ICMP echo request, id 21266, seq 1, length 64
17:06:47.925979 IP noid.chrisbinnie.tld > recce.chrisbinnie.tld: ICMP echo reply, id 21266, seq 1, length 64
17:06:48.927871 IP recce.chrisbinnie.tld > noid.chrisbinnie.tld: ICMP echo request, id 21266, seq 2, length 64
17:06:48.927921 IP noid.chrisbinnie.tld > recce.chrisbinnie.tld: ICMP echo reply, id 21266, seq 2, length 64
17:06:49.928069 IP recce.chrisbinnie.tld > noid.chrisbinnie.tld: ICMP echo request, id 21266, seq 3, length 64
17:06:49.928136 IP noid.chrisbinnie.tld > recce.chrisbinnie.tld: ICMP echo reply, id 21266, seq 3, length 64
17:06:52.215139 IP noid.chrisbinnie.tld > recce.chrisbinnie.tld: ICMP host noid.chrisbinnie.tld unreachable - admin prohibited, length 68
17:06:52.215179 IP noid.chrisbinnie.tld > recce.chrisbinnie.tld: ICMP host noid.chrisbinnie.tld unreachable - admin prohibited, length 68
17:06:52.215194 IP noid.chrisbinnie.tld > recce.chrisbinnie.tld: ICMP host noid.chrisbinnie.tld unreachable - admin prohibited, length 68
17:06:52.215210 IP noid.chrisbinnie.tld > recce.chrisbinnie.tld: ICMP host noid.chrisbinnie.tld unreachable - admin prohibited, length 68
17:06:52.215220 IP noid.chrisbinnie.tld > recce.chrisbinnie.tld: ICMP host noid.chrisbinnie.tld unreachable - admin prohibited, length 68
17:06:52.215231 IP noid.chrisbinnie.tld > recce.chrisbinnie.tld: ICMP host noid.chrisbinnie.tld unreachable - admin prohibited, length 68

In the code snippet, a server called noid experiences probes from a host called recce. Recce is reconnoitering to see if the server is online.

Iptables

You can also run the iptables command, which uses Netfilter's kernel-based firewall:

# iptables -I INPUT -p icmp --icmp-type 8 -m state --state NEW,ESTABLISHED,RELATED -j LOG --log-level=1 --log-prefix "Pings Logged "

If you're paying attention, you may have spotted the number 8 being used as the --icmp-type value. In Table 6.1, you can see the codes that are used by ICMP. You can find more information at its Request for Comments (RFC) page, at https://tools.ietf.org/html/rfc792. According to this page, ICMP has been around since 1981 or thereabouts, when the Internet was a very different animal.

As a reaction to the attacks that used ICMP, changes were made to the Linux kernel over time. Thanks to the abuse of ICMP, the functions marked with an asterisk in Table 6.1 are rate limited by default in modern implementations of the kernel (since Linux 2.4.10).

In the following code snippet, you see the source and destination IP addresses (SRC=10.10.10.200 and DST=10.10.10.10) involved in the ping traffic exchange logged to the /var/log/messages file.

Feb 31 17:19:34 noid.chrisbinnie.tld kernel: Pings Logged IN=eth0 OUT= MAC=00:61:24:3e:1c:ef:00:30:16:3c:14:3b:02:10 SRC=10.10.10.200 DST=10.10.10.10 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=40978 SEQ=1
Feb 31 17:19:35 noid.chrisbinnie.tld kernel: Pings Logged IN=eth0 OUT= MAC=00:61:24:3e:1c:ef:00:30:16:3c:14:3b:02:10 SRC=10.10.10.200 DST=10.10.10.10 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=40978 SEQ=2
Feb 31 17:19:36 noid.chrisbinnie.tld kernel: Pings Logged IN=eth0 OUT= MAC=00:61:24:3e:1c:ef:00:30:16:3c:14:3b:02:10 SRC=10.10.10.200 DST=10.10.10.10 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=40978 SEQ=3

If I used either tcpdump or iptables to log traffic, I would be keen to reduce the log files. The first reason for this would be to stop an attacker (knowingly or otherwise) from causing disk space problems by creating massive log files, after inundating a server with ICMP traffic. The second reason would be to keep the significant logging noise levels to a minimum so that I could quickly reference the log and spot what I was looking for.

Assuming that you wisely felt uncomfortable having tcpdump running in the background all year round, for reasons of system stability, let's use the iptables example to strip out the information that you need.

Let's look at a high-level overview of how you might go about stripping the noise out of a log. For clarity, you would probably want your reconnoiter logs to be dumped to a file separate from syslog.

I'm going to use the excellent, “rocket-fast” syslog server, rsyslog, as an example (you can find more information at www.rsyslog.com). This is because Red Hat and Debian (and their many derivatives) currently use rsyslog by default, so there's a good chance you will have access to it.

Let's take another look at the iptables pings example from a moment ago:

# iptables -I INPUT -p icmp --icmp-type 8 -m state --state NEW,ESTABLISHED,RELATED -j LOG --log-level=1 --log-prefix "Pings Logged "

You're now paying closer attention to the --log-prefix option, as shown here:

--log-prefix "Pings Logged "

If you want to save all kernel warnings to a new log file, then it is relatively easy to set up. Open the file /etc/rsyslog.conf and add the following line to the RULES section or close to it (assuming there's not already an entry for kern.warning that you might disrupt; otherwise, decide if you can overwrite or append to it):

kern.warning                                 /var/log/iptables.log

Of course, when it comes to manipulating text files, you can achieve a lot of different results with a quick shell script (or by using some grep, awk, or sed command-line tools). However, for the sake of avoiding even temporary disk space issues, let's try not to log every warning from the kernel in case a piece of hardware starts misbehaving and logs thousands of warnings in a short period of time.

You're going to create a new syslog config file and call it /etc/rsyslog.d/iptables-pings-logging.conf. Incidentally, you can probably name this file whatever you like if your main config file (that's the /etc/rsyslog.conf file) picks up all config files that it finds in that directory. By default, there's an entry in that file that looks like this to read all the config files at once:

$IncludeConfig /etc/rsyslog.d/*.conf

I should say, however, that I had trouble getting the filters to work from a new file in that directory. That was despite checking permissions and having remote-syslog-logging working fine from within one of those files (and trying startswith and regex instead of the contains operator that you will see next).

If you run into the same trouble, then instead of adding the following two lines to your new config file, simply navigate to the line mentioning kern.* under RULES again in the main config file (/etc/rsyslog.conf) to avoid using a separate config file altogether, and add these two lines:

:msg, contains, "Pings"                 /var/log/iptables-pings.log
& ∼

The first line catches entries that include the string “Pings” and then asks syslog to write them to the file /var/log/iptables-pings.log. The second line is a little unusual and tells the syslog software to ignore any entries caught by the previous line so that you're not doubling up with your logs by writing the same content to another file. You can, of course, leave that second line out if you'd like to log elsewhere too.

Now that you're able to drop content to a log file and filter out specific iptables events (adding your own labels as you prefer), let's look at other examples.

Multipart Rules

If you want to allow pings to reach your servers and also allow outbound pings from your server, then each operation involves slightly different iptables configuration.

The following web page explains how to allow pings into a certain IP address on your server (assuming you have more than one bound to your server) along with the different outbound pings config: www.cyberciti.biz/tips/linux-iptables-9-allow-icmp-ping.html.

Note that the -d for destination on the inbound ping rules means a specific IP address.

Log Everything for Forensic Analysis

If you were concerned that you were experiencing an attack and you wanted to log all inbound connections to your machine, then you could (briefly) enable this command:

# iptables -I INPUT -m state --state NEW -j LOG --log-prefix "Logged Traffic: "

Be warned that your /var/log/messages file would soon be very large, so you would have to disable the logging by flushing the rule as soon as you could. (You can check out the example later in this section to find out how to do this.) You might expect the output to look something like this from the iptables command:

Nov  11 01:11:01 ChrisLinuxHost kernel: New Connection: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:41:23:4f:4d:1f:08:00 SRC=10.10.10.10 DST=10.10.10.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=28621 PROTO=UDP SPT=137 DPT=137 LEN=58

This extract is taken from my now sizeable /var/log/messages file. The traffic appears to have been generated by a Netbios packet. Simply swap INPUT with OUTPUT if you would prefer to track egested traffic.

On that note, what if you want to log traffic but also rate-limit what hits your server logs? Here's an example:

# iptables -I INPUT -p icmp -m limit --limit 5/min -j LOG --log-prefix "Blocked ICMP Traffic: " --log-level 7

It's simple enough to change -p icmp to -p tcp or -p udp and pick up TCP and UDP packets, respectively. This example means that you are only logging five packets per minute of this type of traffic. This can be useful because, generally, the first few probes are informative before repetition starts.

Incidentally, if you find that your logs are filling up too quickly, then you can just flush every iptables rule like this:

# iptables -F
# iptables -X
# iptables -t nat -F
# iptables -t nat -X
# iptables -t mangle -F
# iptables -t mangle -X
# iptables -t raw -F
# iptables -t raw -X
# iptables -t security -F
# iptables -t security -X
# iptables -P INPUT ACCEPT
# iptables -P FORWARD ACCEPT
# iptables -P OUTPUT ACCEPT

My preference is to put this string of commands into a script (or very long Bash alias, with the commands separated by semicolons), and as the root user, I can type flush whenever I need a quick, fail-safe removal of all the firewall rules in use.

Hardening

If you're worried about your system being overloaded with ICMP traffic, then there are a couple of relatively simple things you can check. This is thanks to the power of Unix-type operating systems. First (and your build may already have this set as default), you can ignore ICMP broadcasts by adding the following line to the bottom of the file /etc/sysctl.conf:

net.ipv4.icmp_echo_ignore_broadcasts = 1

This sysctl.conf example will persist a reboot, whereas the following command will set it live immediately:

# echo "1" > /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts

The following command will also set it live immediately if you have followed the sysctl.conf example (it reloads all the config settings found inside the file sysctl.conf):

# sysctl -p

The icmp_echo_ignore_broadcasts setting stops ICMP broadcasts from bringing down your network with unwanted broadcast traffic.

In reality, this is a deprecated attack (you saw that modern kernels use rate-limiting as standard), and the kernel setting is really only useful for ping attacks to the broadcast address that force every device in the local network's broadcast domain to respond. If every device responds at the same time and then responds to others' responses, then a denial of service is caused by too much traffic on the network. As with all kernel settings, though, it's useful to know the background to help you understand how networking works.

Another attack (harking from 1996) was called the Ping of Death, and crafted massive ICMP packets with the hope of crashing the remote machine. It affected many of the popular networking stacks (at release 18 operating systems were found to be vulnerable). Simply knowing an IP address and firing ICMP packets greater than 65,536 bytes at that IP address caused the machine to which it was bound to crash. There's more information from the excellent Insecure.org website, at http://insecure.org/sploits/ping-o-death.html.

Another commonly discussed attack from yesteryear involved a man-in-the-middle (MITM), and was called a smurf attack. Smurf attacks are a good illustration of how spoofing can work, and belong to a type of attack that has recently become more frequent, called an amplification attack.

The middleman is put to good use so that the original source of the attack can't be identified, and the attack's aim is to saturate bandwidth, ultimately causing a denial of service. Worryingly, the barrage of ICMP traffic actually appears to come from the victims themselves. The middleman (also sometimes called the amplifier) receives these spoofed packets and sends a normal echo-response reply to the victim, who didn't even ask for a response. They also have little way of stopping the packets from arriving. Back in the earlier days of the Internet, your ISP could only disable all ICMP traffic to stop the massive flood of data. Because the packets were spoofed, even the logs were of little help in tracking down where they originated.

One solution to this problem is to rate-limit the packets yourself. This leaves ICMP traffic free to arrive, and your network running relatively normally, although possibly a little slower.

This can be achieved on proprietary router hardware, such as Cisco or Juniper, but also on individual hosts by using iptables with a set of rules similar to these:

# iptables -I INPUT -p icmp -m limit --limit 30/minute --limit-burst 60 -j ACCEPT
# iptables -I INPUT -p icmp -m limit --limit 6/minute --limit-burst 10 -j LOG
# iptables -I INPUT -p icmp -j DROP

From the config shown on the first line, the --limit-burst option allows up to 60 packets to arrive initially before the rate limit of one packet every other second is enabled (the 30/minute value). The second line then says that you will log (though much more strictly) how much traffic is accepted before being logged to syslog: six entries per minute after a burst of ten packets. Finally, you'll discard those packets in order to mitigate any effect.

If I'm reading the documentation correctly, the --limit-burst value refers back to the specified period used by --limit. When that limit is not reached, --limit-burst benefits by an increment — in other words, by the value of one — every time the limit isn't reached, up to the number configured.

Summary

Some of the changes you have explored can be damaging to both systems and networks. You should know what you're doing (and ideally, experiment on a test machine) before trying any of these settings in production. As mentioned, the classic mistake of becoming panicked by the bad reputation that ICMP has and simply banning all mentions of the protocol on your network is ill-advised.

You have looked at attacks and how to prevent them from affecting the normal operations of your network and servers. You also looked at rate-limiting inbound ICMP traffic.

In addition, you looked at the logging of pings and traceroutes to pick up any recces taking place with nefarious intent on your server, and more importantly, how to safely log these unwanted probes to a file so that an attack won't fill your disks with sizable log files.

It's safe to say that some of this knowledge may only be needed rarely, but the next time someone mentions that they're blocking all ICMP traffic on their server, you can give them a knowing look and say that you're confident that ICMP issues will no longer cause you problems.