Chapter 6. Turning the Tables for Proactive Defense

Turning the Tables for Proactive Defense

In the previous chapter, you saw how you might need to spend considerable time and energy making sure that the services you want to offer will be available even when you have strict packet filtering in place. Now, with your working setup in place, you’ll soon notice that some services tend to attract a little more unwanted attention than others.

Here’s the scenario: You have a network with packet filtering to match your site’s needs, including some services that need to be accessible to users from elsewhere. Unfortunately, when services are available, there’s a risk that someone will want to exploit them for some sort of mischief.

You’ll almost certainly have remote login via SSH (Secure Shell), as well as SMTP email running on your network—both are tempting targets. In this chapter, we’ll look at ways to make it harder to gain unauthorized access via SSH, and then we’ll turn to some of the more effective ways to deny spammers use of your servers.

Turning Away the Brutes

The Secure Shell service, commonly referred to as SSH, is a fairly crucial service for Unix administrators. It’s frequently the main interface to the machine and a favorite target of script kiddie attacks.

SSH Brute-Force Attacks

If you run an SSH login service that’s accessible from the Internet, you’ve probably seen entries like this in your authentication logs:

Sep 26 03:12:34 skapet sshd[25771]: Failed password for root from 200.72.41.31 port 40992 ssh2
Sep 26 03:12:34 skapet sshd[5279]: Failed password for root from 200.72.41.31 port 40992 ssh2
Sep 26 03:12:35 skapet sshd[5279]: Received disconnect from 200.72.41.31: 11: Bye Bye
Sep 26 03:12:44 skapet sshd[29635]: Invalid user admin from 200.72.41.31
Sep 26 03:12:44 skapet sshd[24703]: input_userauth_request: invalid user admin
Sep 26 03:12:44 skapet sshd[24703]: Failed password for invalid user admin from 200.72.41.31
port 41484 ssh2
Sep 26 03:12:44 skapet sshd[29635]: Failed password for invalid user admin from 200.72.41.31
port 41484 ssh2
Sep 26 03:12:45 skapet sshd[24703]: Connection closed by 200.72.41.31
Sep 26 03:13:10 skapet sshd[11459]: Failed password for root from 200.72.41.31 port 43344 ssh2
Sep 26 03:13:10 skapet sshd[7635]: Failed password for root from 200.72.41.31 port 43344 ssh2
Sep 26 03:13:10 skapet sshd[11459]: Received disconnect from 200.72.41.31: 11: Bye Bye
Sep 26 03:13:15 skapet sshd[31357]: Invalid user admin from 200.72.41.31
Sep 26 03:13:15 skapet sshd[10543]: input_userauth_request: invalid user admin
Sep 26 03:13:15 skapet sshd[10543]: Failed password for invalid user admin from 200.72.41.31
port 43811 ssh2
Sep 26 03:13:15 skapet sshd[31357]: Failed password for invalid user admin from 200.72.41.31
port 43811 ssh2
Sep 26 03:13:15 skapet sshd[10543]: Received disconnect from 200.72.41.31: 11: Bye Bye
Sep 26 03:13:25 skapet sshd[6526]: Connection closed by 200.72.41.31

This is what a brute-force attack looks like. Someone or something is trying by brute force to find a username and password combination that lets them get into your system.

The simplest response would be to write a pf.conf rule that blocks all access, but that leads to another class of problems, including how to let people with legitimate business on your system access it. Setting up your sshd to accept only key-based authentication would help but most likely would not stop the kiddies from trying. You might consider moving the service to another port, but then again, the ones flooding you on port 22 would probably be able to scan their way to port 22222 for a repeat performance.[30]

Since OpenBSD 3.7 (and equivalents), PF has offered a slightly more elegant solution.

Setting Up an Adaptive Firewall

To thwart brute-force attacks, you can write your pass rules so they maintain certain limits on what connecting hosts can do. For good measure, you can banish violators to a table of addresses to which you deny some or all access. You can even choose to drop all existing connections from machines that overreach your limits. To enable this feature, first set up the table by adding the following line to your configuration before any filtering rules:

table <bruteforce> persist

Then, early in your rule set, block brute forcers, as shown here:

block quick from <bruteforce>

Finally, add your pass rule:

pass proto tcp to $localnet port $tcp_services 
     keep state (max-src-conn 100, max-src-conn-rate 15/5, 
         overload <bruteforce> flush global)

This rule is very similar to what you’ve seen in earlier examples. The interesting part in this context is the contents of the parentheses, called state-tracking options:

  • max-src-conn is the number of simultaneous connections allowed from one host. In this example, it’s set to 100. You may want a slightly higher or lower value, depending on your network’s traffic patterns.

  • max-src-conn-rate is the rate of new connections allowed from any single host. Here, it’s set to 15 connections per 5 seconds, denoted as 15/5. Choose a rate that suits your setup.

  • overload <bruteforce> means that the address of any host that exceeds the preceding limits is added to the table bruteforce. Our rule set blocks all traffic from addresses in the bruteforce table. Once a host exceeds any of these limits and is put in the overload table, the rule no longer matches traffic from that host. Make sure that overloaders are handled, if only by a default block rule or similar.

  • flush global says that when a host reaches the limit, all states for its connections are terminated (flushed). The global option means that for good measure, flush applies to all states created by traffic from that host, no matter which rule created a state.

As you can imagine, the effect of this tiny addition to the rule set is dramatic. After a few tries, brute forcers end up in the bruteforce table. That means that all their existing connections are terminated (flushed) and any new attempts will be blocked, most likely with Fatal: timeout before authentication messages at their end. You have created an adaptive firewall that adjusts automatically to conditions in your network and acts on undesirable activity.

Note

These adaptive rules are effective only for protection against the traditional, rapid-fire type of brute-force attempts. The low-intensity, distributed password-guessing attempts that were first identified as such in 2008 and have been recurring ever since (known among other names as The Hail Mary Cloud[31]) don’t produce traffic that will match these rules.

It’s likely that you will want some flexibility in your rule set and want to allow a larger number of connections for some services, but you also might like to be a little more tight-fisted when it comes to SSH. In that case, you could supplement the general-purpose pass rule with something like the following one early in your rule set:

pass quick proto { tcp, udp } to port ssh 
     keep state (max-src-conn 15, max-src-conn-rate 5/3, 
        overload <bruteforce> flush global)

You should be able to find the set of parameters that’s just right for your situation by reading the relevant man pages and the PF User Guide (see Appendix A).

Note

Remember that these sample rules are intended as illustrations and your network’s needs may be better served by different rules. Setting the number of simultaneous connections or the rate of connections too low may block legitimate traffic. There’s a potential risk of self-inflicted denial of service when the configuration includes many hosts behind a common NATing gateway and the users on the NATed hosts have legitimate business on the other side of a gateway with strict overload rules.

The state-tracking options and the overload mechanism don’t need to apply exclusively to the SSH service, and blocking all traffic from offenders isn’t always desired. You could, for example, use a rule like this:

pass proto { tcp, udp } to port $mail_services 
     keep state (max 1500, max-src-conn 100)

Here, max specifies the maximum number of states that can be created for each rule with no overload to protect a mail or Web service from receiving more connections than it can handle (keep in mind that the number of rules loaded depends on what the $mail_services macro expands to). Once the max limit is reached, new connections will not match this rule until the old ones terminate. Alternatively, you could remove the max restriction, add an overload part to the rule, and assign offenders to a queue with a minimal bandwidth allocation (see the discussion of traffic shaping in Chapter 7 for details on setting up queues).

Some sites use overload to implement a multitiered system, where hosts that trip one overload rule are transferred to one or more intermediate “probation” tables for special treatment. It can be useful in Web contexts not to block traffic from hosts in the overload tables outright but rather to redirect all HTTP requests from these hosts to specific Web pages (as in the authpf example near the end of Chapter 4).

Tidying Your Tables with pfctl

With the overload rules from the previous section in place, you now have an adaptive firewall that automatically detects undesirable behavior and adds offenders’ IP addresses to tables. Watching the logs and the tables can be fun in the short run, but because those rules only add to the tables, we run into the next challenge: keeping the content of the tables up-to-date and relevant.

When you’ve run a configuration with an adaptive rule set for a while, at some point, you’ll discover that an IP address one of your overload rules blocked last week due to a brute-force attack was actually a dynamically assigned address, which is now assigned to a different ISP customer with a legitimate reason to communicate with hosts in your network.[32] If your adaptive rules catch a lot of traffic on a busy network, you may also find that the overload tables will grow over time to take up an increasing amount of memory.

The solution is to expire table entries—to remove entries after a certain amount of time. In OpenBSD 4.1, pfctl acquired the ability to expire table entries based on the time since their statistics were last reset.[33] (In almost all instances, this reset time is equal to the time since the table entry was added.) The keyword is expire, and the table entry’s age is specified in seconds. Here’s an example:

$ sudo pfctl -t bruteforce -T expire 86400

This command will remove bruteforce table entries that had their statistics reset more than 86,400 seconds (24 hours) ago.

Note

The choice of 24 hours as the expiry time is a fairly arbitrary one. You should choose a value that you feel is a reasonable amount of time for any problem at the other end to be noticed and fixed. If you have adaptive rules in place, it’s a good idea to set up crontab entries to run table expiry at regular intervals with a command much like the preceding one to make sure your tables are kept up-to-date.

Giving Spammers a Hard Time with spamd

Email is a fairly essential service that needs special attention due to the large volume of unwanted messages, or spam. The volume of unsolicited commercial messages was already a painful problem when malware makers discovered that email-borne worms would work and started using email to spread their payload. During the early 2000s, the combined volume of spam and email-borne malware had increased to the point where running an SMTP mail service without some sort of spam countermeasures had become almost unthinkable.

Spam-fighting measures are almost as old as the spam problem itself. The early efforts focused on analysis of the messages’ contents (known as content filtering) and to some extent on interpretation of the messages’ rather trivially forgeable headers, such as the purported sender address (From:) or the store and forward paths of intermediate deliveries recorded in the Received: headers.

When the OpenBSD team designed its spam-fighting solution spamd, first introduced with OpenBSD 3.3 in 2003, the developers instead focused on the network level and the immediate communication partner in the SMTP conversations along with any available information about hosts that tried to deliver messages. The developers set out to create a small, simple, and secure program. The early implementation was based almost entirely on creative use of PF tables combined with data from trusted external sources.

Note

In addition to the OpenBSD spam-deferral daemon, the content-filtering-based antispam package SpamAssassin (http://spamassassin.apache.org/) features a program called spamd. Both programs are designed to help fight spam, but they take very different approaches to the underlying problem and don’t interoperate directly. However, when both programs are correctly configured and running, they complement each other well.

Network-Level Behavior Analysis and Blacklisting

The original spamd design is based on the observation that spammers send a lot of mail and the incredibly small likelihood of you being the first person to receive a particular message. In addition, spam is sent via a few spammer-friendly networks and numerous hijacked machines. Both the individual messages and the machines that send them will be reported to blacklist maintainers quickly, and the blacklist data consisting of known spam senders’ IP addresses forms the basis for spamd’s processing.

When dealing with blacklisted hosts, spamd employs a method called tarpitting. When the daemon receives an SMTP connection, it presents its banner and immediately switches to a mode where it answers SMTP traffic at the rate of 1 byte per second, using a tiny selection of SMTP commands designed to make sure that mail is never delivered but rather rejected back into the sender’s queue once the message headers have been transferred. The intention is to waste as much time as possible on the sending end while costing the receiver pretty much nothing. This specific tarpitting implementation with 1-byte SMTP replies is often referred to as stuttering. Blacklist-based tarpitting with stuttering was the default mode for spamd up to and including OpenBSD 4.0.

Note

On FreeBSD and NetBSD, spamd is not part of the base system but is available through ports and packages as mail/spamd. If you’re running PF on FreeBSD or NetBSD, you need to install the port or package before following the instructions over the next few pages.

Setting Up spamd in Blacklisting Mode

To set up spamd to run in traditional, blacklisting-only mode, you first put a special-purpose table and a matching redirection in pf.conf and then turn your attention to spamd’s own spamd.conf. spamd then hooks into the PF rule set via the table and the redirection.

The following are the pf.conf lines for this configuration:

table <spamd> persist
pass in on $ext_if inet proto tcp from <spamd> to 
      { $ext_if, $localnet } port smtp rdr-to 127.0.0.1 port 8025

And here is the pre–OpenBSD 4.7 syntax:

table <spamd> persist
rdr pass on $ext_if inet proto tcp from <spamd> to 
         { $ext_if, $localnet } port smtp -> 127.0.0.1 port 8025

The table, <spamd>, is there to store the IP addresses you import from trusted blacklist sources. The redirection takes care of all SMTP attempts from hosts that are already in the blacklist. spamd listens on port 8025 and responds s-l-o-w-l-y (1 byte per second) to all SMTP connections it receives as a result of the redirection. Later on in the rule set, you would have a rule that makes sure legitimate SMTP traffic passes to the mail server. spamd.conf is where you specify the sources of your blacklist data and any exceptions or local overrides you want.

Note

On OpenBSD 4.0 and earlier (and by extension, ports based on versions prior to OpenBSD 4.1), spamd.conf was in /etc. Beginning with OpenBSD 4.1, spamd. conf is found in /etc/mail instead. The FreeBSD port installs a sample configuration in /usr/local/etc/spamd/spamd.conf.sample.

Near the beginning of spamd.conf, you’ll notice a line without a # comment sign that looks like all:. This line specifies the blacklists you’ll use. Here is an example:

all:
:uatraps:whitelist:

Add all blacklists that you want to use below the all: line, separating each with a colon (:). To use whitelists to subtract addresses from your blacklist, add the name of the whitelist immediately after the name of each blacklist, as in :blacklist:whitelist:

Next is the blacklist definition:

uatraps:
        :black:
        :msg="SPAM. Your address %A has sent spam within the last 24 hours":
        :method=http:
        :file=www.openbsd.org/spamd/traplist.gz

Following the name (uatraps), the first data field specifies the list type—in this case, black. The msg field contains the message to be displayed to blacklisted senders during the SMTP dialogue. The method field specifies how spamd-setup fetches the list data—in this case, via HTTP. Other possibilities include fetching via FTP (ftp), from a file in a mounted filesystem (file), or via execution of an external program (exec). Finally, the file field specifies the name of the file spamd expects to receive.

The definition of a whitelist follows much the same pattern but omits the message parameter:

whitelist:
        :white:
        :method=file:
        :file=/etc/mail/whitelist.txt

Note

The suggested blacklists in the current default spamd.conf are actively maintained and have rarely, if ever, contained false positives. However, earlier versions of that file also suggested lists that excluded large blocks of the Internet, including several address ranges that claim to cover entire countries. If your site expects to exchange legitimate mail with any of the countries in question, those lists may not be optimal for your setup. Other popular lists have been known to list entire /16 ranges as spam sources, and it’s well worth reviewing the details of the list’s maintenance policy before putting a blacklist into production.

Put the lines for spamd and the startup parameters you want in your /etc/ rc.conf.local on OpenBSD or in /etc/rc.conf on FreeBSD or NetBSD. Here’s an example:

spamd_flags="-v -b" # for normal use: "" and see spamd-setup(8)

Here, we enable spamd and set it to run in blacklisting mode with the -b flag. In addition, the -v flag enables verbose logging, which is useful for keeping track of spamd’s activity for debugging purposes.

On FreeBSD, the /etc/rc.conf settings that control spamd’s behavior are obspamd_enable, which should be set to "YES" in order to enable spamd, and obspamd_flags, where you fill in any command-line options for spamd:

obspamd_enable="YES"
obspamd_flags="-v -b" # for normal use: "" and see spamd-setup(8)

Note

To have spamd run in pure blacklist mode on OpenBSD 4.1 or newer, you can achieve the same effect by setting the spamd_black variable to “YES” and then restarting spamd.

Once you’ve finished editing the setup, start spamd with the options you want and complete the configuration with spamd-setup. Finally, create a cron job that calls spamd-setup to update the blacklist at reasonable intervals. In pure blacklist mode, you can view and manipulate the table contents using pfctl table commands.

spamd Logging

By default, spamd logs to your general system logs. To send the spamd log messages to a separate log file, add an entry like this to syslog.conf:

!!spamd
daemon.err;daemon.warn;daemon.info;daemon.debug         /var/log/spamd

Once you’re satisfied that spamd is running and doing what it’s supposed to do, you’ll probably want to add the spamd log file to your log rotations, too. After you’ve run spamd-setup and the tables are filled, you can view the table contents using pfctl.

Note

In the sample pf.conf fragment at the beginning of this section, the redirection (rdr-to) rule is also a pass rule. If you opted for a match rule instead (or if you’re using an older PF version and chose to write a rdr rule that doesn’t include a pass part), be sure to set up a pass rule to let traffic through to your redirection. You may also need to set up rules to let legitimate email through. However, if you’re already running an email service on your network, you can probably go on using your old SMTP pass rules.

Given a set of reliable and well-maintained blacklists, spamd in pure blacklisting mode does a good job of reducing spam. However, with pure blacklisting, you catch traffic only from hosts that have already tried to deliver spam elsewhere, and you put your trust in external data sources to determine which hosts deserve to be tarpitted. For a setup that provides a more immediate response to network-level behavior and offers some real gains in spam prevention, consider greylisting, which is a crucial part of how the modern spamd works.

Greylisting: My Admin Told Me Not to Talk to Strangers

Greylisting consists mainly of interpreting the current SMTP standards and adding a little white lie to make life easier.

Spammers tend to use other people’s equipment to send their messages, and the software they install without the legal owner’s permission needs to be relatively lightweight in order to run undetected. Unlike legitimate mail senders, spammers typically don’t consider any individual message they send to be important. Taken together, this means that typical spam and malware sender software aren’t set up to interpret SMTP status codes correctly. This is a fact that we can use to our advantage, as Evan Harris proposed in his 2003 paper titled “The Next Step in the Spam Control War: Greylisting.” [34]

As Harris noted, when a compromised machine is used to send spam, the sender application tends to try delivery only once, without checking for any results or return codes. Real SMTP implementations interpret SMTP return codes and act on them, and real mail servers retry if the initial attempt fails with any kind of temporary error.

In his paper, Harris outlined a practical approach:

  • On first SMTP contact from a previously unknown communication partner, do not receive email on the first delivery attempt, but instead, respond with a status code that indicates a temporary local problem, and store the sender IP address for future reference.

  • If the sender retries immediately, reply as before with the temporary failure status code.

  • If the sender retries after a set minimum amount of time (1 hour, for example) but not more than a maximum waiting period (4 hours, for example), accept the message for delivery and record the sender IP address in your whitelist.

This is the essence of greylisting. And fortunately, you can set up and maintain a greylisting spamd on your PF-equipped gateway.

Setting Up spamd in Greylisting Mode

OpenBSD’s spamd acquired its ability to greylist in OpenBSD 3.5. Beginning with OpenBSD 4.1, spamd runs in greylisting mode by default.

In the default greylisting mode, the spamd table used for blacklisting, as described in the previous section, becomes superfluous. You can still use blacklists, but spamd will use a combination of private data structures for blacklist data and the spamdb database to store greylisting-related data. A typical set of rules for spamd in default mode looks like this:

table <spamd-white> persist
table <nospamd> persist file "/etc/mail/nospamd"
pass in log on egress proto tcp to port smtp 
            rdr-to 127.0.0.1 port spamd
pass in log on egress proto tcp from <nospamd> to port smtp
pass in log on egress proto tcp from <spamd-white> to port smtp
pass out log on egress proto tcp to port smtp

This includes the necessary pass rules to let legitimate email flow to the intended destinations from your own network. The <spamd-white> table is the whitelist, maintained by spamd. The hosts in the <spamd-white> table have passed the greylisting hurdle, and mail from these machines is allowed to pass to the real mail servers or their content-filtering frontends. In addition, the nospamd table is there for you to load addresses of hosts that you don’t want to expose to spamd processing, and the matching pass rule makes sure SMTP traffic from those hosts passes.

In your network, you may want to tighten those rules to pass SMTP traffic only to and from hosts that are allowed to send and receive email via SMTP. We’ll get back to the nospamd table in Handling Sites That Do Not Play Well with Greylisting.

The following are the equivalent rules in pre–OpenBSD 4.7 syntax:

table <spamd-white> persist
table <nospamd> persist file "/etc/mail/nospamd"
rdr pass in log on egress proto tcp to port smtp 
            -> 127.0.0.1 port spamd
pass in log on egress proto tcp from <nospamd> to port smtp
pass in log on egress proto tcp from <spamd-white> to port smtp
pass out log on egress proto tcp to port smtp

On FreeBSD, in order to use spamd in greylisting mode, you need a file descriptor filesystem (see man 5 fdescfs) mounted at /dev/fd/. To implement this, add the following line to /etc/fstab and make sure the fdescfs code is in your kernel, either compiled in or by loading the module via the appropriate kldload command.

fdescfs /dev/fd fdescfs rw 0 0

To begin configuring spamd, place the lines for spamd and the startup parameters you want in /etc/rc.conf.local. Here’s an example:

spamd_flags="-v -G 2:4:864" # for normal use: "" and see spamd-setup(8)

On FreeBSD, the equivalent line should go in /etc/rc.conf:

obspamd_flags="-v -G 2:4:864" # for normal use: "" and see spamd-setup(8)

You can fine-tune several of the greylisting-related parameters via spamd command-line parameters trailing the -G option.

Why Greylisting Works

A significant amount of design and development effort has been put into making essential services, such as SMTP email transmission, fault-tolerant. In practical terms, this means that the best effort of a service such as SMTP is as close as you can get to having a perfect record for delivering messages. That’s why we can rely on greylisting to eventually let us receive email from proper mail servers.

The current standard for Internet email transmission is defined in RFC 5321.[35] The following are several excerpts from Section 4.5.4.1, “Sending Strategy”:

“In a typical system, the program that composes a message has some method for requesting immediate attention for a new piece of outgoing mail, while mail that cannot be transmitted immediately MUST be queued and periodically retried by the sender. . . .

“The sender MUST delay retrying a particular destination after one attempt has failed. In general, the retry interval SHOULD be at least 30 minutes; however, more sophisticated and variable strategies will be beneficial when the SMTP client can determine the reason for non-delivery.

“Retries continue until the message is transmitted or the sender gives up; the give-up time generally needs to be at least 4–5 days.”

Delivering email is a collaborative, best-effort process, and the RFC clearly states that if the site you are trying to send mail to reports that it can’t receive at the moment, it is your duty (a must requirement) to try again later, giving the receiving server a chance to recover from its problem.

The clever wrinkle to greylisting is that it’s a convenient white lie. When we claim to have a temporary local problem, that problem is really the equivalent of “My admin told me not to talk to strangers.” Well-behaved senders with valid messages will call again, but spammers won’t wait around for the chance to retry, since doing so increases their cost of delivering messages. This is why greylisting still works, and since it’s based on strict adherence to accepted standards,[36] false positives are rare.

The colon-separated list 2:4:864 represents the values passtime, greyexp, and whiteexp:

  • passtime denotes the minimum number of minutes spamd considers a reasonable time before retry. The default is 25 minutes, but here we’ve reduced it to 2 minutes.

  • greyexp is the number of hours an entry stays in the greylisted state before it’s removed from the database.

  • whiteexp determines the number of hours a whitelisted entry is kept. The default values for greyexp and whiteexp are 4 hours and 864 hours (just over 1 month), respectively.

Greylisting in Practice

Users and administrators at sites that implement greylisting tend to agree that greylisting gets rid of most of their spam, with a significant drop in the load on any content filtering they have in place for their mail. We’ll start by looking at what spamd’s greylisting looks like according to log files and then return with some data.

If you start spamd with the -v command-line option for verbose logging, your logs will include a few more items of information in addition to IP addresses. With verbose logging, a typical log excerpt looks like this:

Oct 2 19:53:21 delilah spamd[26905]: 65.210.185.131: connected (1/1), lists: spews1
Oct 2 19:55:04 delilah spamd[26905]: 83.23.213.115: connected (2/1)
Oct 2 19:55:05 delilah spamd[26905]: (GREY) 83.23.213.115: <[email protected]> ->
<[email protected]>
Oct 2 19:55:05 delilah spamd[26905]: 83.23.213.115: disconnected after 0 seconds.
Oct 2 19:55:05 delilah spamd[26905]: 83.23.213.115: connected (2/1)
Oct 2 19:55:06 delilah spamd[26905]: (GREY) 83.23.213.115: <[email protected]> ->
<[email protected]>
Oct 2 19:55:06 delilah spamd[26905]: 83.23.213.115: disconnected after 1 seconds.
Oct 2 19:57:07 delilah spamd[26905]: (BLACK) 65.210.185.131: <bounce-3C7E40A4B3@branch15.
summer-bargainz.com> -> <[email protected]>
Oct 2 19:58:50 delilah spamd[26905]: 65.210.185.131: From: Auto lnsurance Savings <noreply@
branch15.summer-bargainz.com>
Oct 2 19:58:50 delilah spamd[26905]: 65.210.185.131: Subject: Start SAVlNG M0NEY on Auto
lnsurance
Oct 2 19:58:50 delilah spamd[26905]: 65.210.185.131: To: [email protected]
Oct 2 20:00:05 delilah spamd[26905]: 65.210.185.131: disconnected after 404 seconds. lists:
spews1
Oct 2 20:03:48 delilah spamd[26905]: 222.240.6.118: connected (1/0)
Oct 2 20:03:48 delilah spamd[26905]: 222.240.6.118: disconnected after 0 seconds.
Oct 2 20:06:51 delilah spamd[26905]: 24.71.110.10: connected (1/1), lists: spews1
Oct 2 20:07:00 delilah spamd[26905]: 221.196.37.249: connected (2/1)
Oct 2 20:07:00 delilah spamd[26905]: 221.196.37.249: disconnected after 0 seconds.
Oct 2 20:07:12 delilah spamd[26905]: 24.71.110.10: disconnected after 21 seconds. lists:
spews1

The first line is the beginning of a connection from a machine in the spews1 blacklist. The next six lines show the complete records of two connection attempts from another machine, which each time connects as the second active connection. This second machine isn’t yet in any blacklist, so it’s grey-listed. Note the rather curious delivery address () in the message that the greylisted machine tries to deliver. There’s a useful trick that we’ll look at in Greytrapping. The (GREY) and (BLACK) before the addresses indicate greylisting or blacklisting status. Then there’s more activity from the blacklisted host, and a little later we see that after 404 seconds (or 6 minutes and 44 seconds), the blacklisted host gives up without completing the delivery.

The remaining lines show a few very short connections, including one from a machine already on a blacklist. This time, though, the machine disconnects too quickly to see any (BLACK) flag at the beginning of the SMTP dialogue, but we see a reference to the list name (spews1) at the end.

Roughly 400 seconds is about the amount of time that naive blacklisted spammers hang around (according to data from various sites) and about the time it takes (at the rate of 1 byte per second) to complete the EHLO ... dialogue until spamd rejects the message. However, while peeking at the logs, you’re likely to find some spammers that hang around significantly longer. For example, in the data from our office gateway, one log entry stood out:

Dec 11 23:57:24 delilah spamd[32048]: 69.6.40.26: connected (1/1), lists:
spamhaus spews1 spews2
Dec 12 00:30:08 delilah spamd[32048]: 69.6.40.26: disconnected after 1964
seconds. lists: spamhaus spews1 spews2

This particular machine was already on several blacklists when it made 13 attempts at delivery from December 9 through December 12. The last attempt lasted 32 minutes and 44 seconds, without completing the delivery. Relatively intelligent spam senders drop the connection during the first few seconds, like the ones in the first log fragment. Others give up after around 400 seconds. A few hang on for hours. (The most extreme case we’ve recorded hung on for 42,673 seconds, which is almost 12 hours.)

Tracking Your Real Mail Connections: spamlogd

Behind the scenes, rarely mentioned and barely documented, is one of spamd’s most important helper programs: the spamlogd whitelist updater. As the name suggests, spamlogd works quietly in the background, logging connections to and from your mail servers to keep your whitelist updated. The idea is to make sure that valid mail sent between hosts you communicate with regularly goes through with a minimum of fuss.

Note

If you’ve followed the discussion up to this point, spamlogd has probably been started automatically already. However, if your initial spamd configuration didn’t include greylisting, spamlogd may not have been started, and you may experience strange symptoms, like the greylist and whitelist not being updated properly. Restarting spamd after you’ve enabled greylisting should ensure that spamlogd is loaded and available, too.

In order to perform its job properly, spamlogd needs you to log SMTP connections to and from your mail servers, just as we did in the sample rule sets in Chapter 5:

emailserver = "192.0.2.225"
pass log proto tcp to $emailserver port $email
pass log proto tcp from $emailserver to port smtp

On OpenBSD 4.1 and higher (and equivalents), you can create several pflog interfaces and specify where rules should be logged. Here’s how to separate the data spamlogd needs to read from the rest of your PF logs:

  1. Create a separate pflog1 interface using ifconfig pflog1 create, or create a hostname.pflog1 file with just the line up.

  2. Change the rules to the following:

    pass log (to pflog1) proto tcp to $emailserver port $email
pass log (to pflog1) proto tcp from $emailserver to port smtp

  3. Add -l pflog1 to spamlogd’s startup parameters.

This separates the spamd-related logging from the rest. (See Chapter 9 for more about logging.)

With the preceding rules in place, spamlogd will add the IP addresses that receive email you send to the whitelist. This isn’t an ironclad guarantee that the reply will pass immediately, but in most configurations, it helps speed things significantly.

Greytrapping

We know that spam senders rarely use a fully compliant SMTP implementation to send their messages, which is why greylisting works. We also know that spammers rarely check that the addresses they feed to their hijacked machines are actually deliverable. Combine these facts, and you see that if a greylisted machine tries to send a message to an invalid address in your domain, there’s a good chance that the message is spam or malware.

This realization led to the next evolutionary step in spamd development—a technique dubbed greytrapping. When a greylisted host tries to deliver mail to a known bad address in our domains, the host is added to a locally maintained blacklist called spamd-greytrap. Members of the spamd-greytrap list are treated to the same 1-byte-per-second tarpitting as members of other blacklists.

Greytrapping as implemented in spamd is simple and elegant. The main thing you need as a starting point is spamd running in greylisting mode. The other crucial component is a list of addresses in domains your servers handle email for, but only ones that you’re sure will never receive legitimate email. The number of addresses in your list is unimportant, but there must be at least one, and the upper limit is mainly defined by how many addresses you wish to add.

Next, you use spamdb to feed your list to the greytrapping feature and sit back to watch. First, a sender tries to send email to an address on your grey-trap list and is simply greylisted, as with any sender you haven’t exchanged email with before. If the same machine tries again, either to the same, invalid address or another address on your greytrapping list, the greytrap is triggered, and the offender is put into spamd-greytrap for 24 hours. For the next 24 hours, any SMTP traffic from the greytrapped host will be stuttered, with 1-byte-at-a-time replies.

That 24-hour period is short enough not to cause serious disruption of legitimate traffic because real SMTP implementations will keep trying to deliver for at least a few days. Experience from large-scale implementations of the technique shows that it rarely produces false positives. Machines that continue spamming after 24 hours will make it back to the tarpit soon enough.

To set up your traplist, use spamdb’s -T option. In my case, the strange address[37] I mentioned earlier in Greylisting in Practice was a natural candidate for inclusion:

$ sudo spamdb -T -a [email protected]

The command I actually entered was $ sudo spamdb -T -a "<[email protected]>". In OpenBSD 4.1 and newer, spamdb doesn’t require the angle brackets or quotes, but it will accept them.

Add as many addresses as you like. I tend to find new additions for my local list of spamtrap addresses by looking in the greylist and mail server logs for failed attempts to deliver delivery failure reports to nonexistent addresses in my domains (yes, it really is as crazy as it sounds).

Warning

Make sure that the addresses you add to your spamtrap lists are invalid and will stay invalid. There’s nothing quite like the embarrassment of discovering that you made a valid address into a spamtrap, however temporarily.

The following log fragment shows how a spam-sending machine is grey-listed at first contact and then comes back and clumsily tries to deliver messages to the curious address I added to my traplist, only to end up in the spamd-greytrap blacklist after a few minutes. We know what it will be doing for the next 20-odd hours.

Nov 6 09:50:25 delilah spamd[23576]: 210.214.12.57: connected (1/0)
Nov 6 09:50:32 delilah spamd[23576]: 210.214.12.57: connected (2/0)
Nov 6 09:50:40 delilah spamd[23576]: (GREY) 210.214.12.57: <[email protected]> ->
<[email protected]>
Nov 6 09:50:40 delilah spamd[23576]: 210.214.12.57: disconnected after 15 seconds.
Nov 6 09:50:42 delilah spamd[23576]: 210.214.12.57: connected (2/0)
Nov 6 09:50:45 delilah spamd[23576]: (GREY) 210.214.12.57: <[email protected].
com> -> <[email protected]>
Nov 6 09:50:45 delilah spamd[23576]: 210.214.12.57: disconnected after 13 seconds.
Nov 6 09:50:50 delilah spamd[23576]: 210.214.12.57: connected (2/0)
Nov 6 09:51:00 delilah spamd[23576]: (GREY) 210.214.12.57: <[email protected]> ->
<[email protected]>
Nov 6 09:51:00 delilah spamd[23576]: 210.214.12.57: disconnected after 18 seconds.
Nov 6 09:51:02 delilah spamd[23576]: 210.214.12.57: connected (2/0)
Nov 6 09:51:02 delilah spamd[23576]: 210.214.12.57: disconnected after 12 seconds.
Nov 6 09:51:02 delilah spamd[23576]: 210.214.12.57: connected (2/0)
Nov 6 09:51:18 delilah spamd[23576]: (GREY) 210.214.12.57: <[email protected]> ->
<[email protected]>
Nov 6 09:51:18 delilah spamd[23576]: 210.214.12.57: disconnected after 16 seconds.
Nov 6 09:51:18 delilah spamd[23576]: (GREY) 210.214.12.57: <[email protected].
com> -> <[email protected]>
Nov 6 09:51:18 delilah spamd[23576]: 210.214.12.57: disconnected after 16 seconds.
Nov 6 09:51:20 delilah spamd[23576]: 210.214.12.57: connected (1/1), lists: spamd-greytrap
Nov 6 09:51:23 delilah spamd[23576]: 210.214.12.57: connected (2/2), lists: spamd-greytrap
Nov 6 09:55:33 delilah spamd[23576]: (BLACK) 210.214.12.57: <[email protected]> ->
<[email protected]>
Nov 6 09:55:34 delilah spamd[23576]: (BLACK) 210.214.12.57: <bounce-3C7E40A4B3@branch15.
summer-bargainz.com> -> <[email protected]>

As a side note, it looks like even though the spammer moved to send from a different machine, both the From: and To: addresses stayed the same. The fact that he’s still trying to send to an address that’s never been deliverable is a strong indicator that this spammer doesn’t check his lists frequently.

Managing Lists with spamdb

There may be times when you need to view or change the contents of blacklists, whitelists, and greylists. These records are located in the /var/db/spamdb database, and an administrator’s main interface to managing those lists is spamdb.

Early versions of spamdb simply offered options to add whitelist entries to the database or update existing ones (spamdb -a nn.mm.nn.mm). You could delete whitelist entries (spamdb -d nn.mm.nn.mm) to compensate for shortcomings in either the blacklists used or the effects of the greylisting algorithms. Recent versions of spamdb offer some interesting features to support greytrapping.

Updating Lists

If you run spamdb without any parameters, it lists the contents of your spamdb database, and it lets you add or delete both spamtrap addresses and traplist entries. You can also add whitelist entries on the fly.

If you want to add a host to your whitelist without adding it to your permanent nospamd file and reloading your rule set or the table, you could do it from the command line instead, like this:

$ sudo spamdb -a 213.187.179.198

If a spam sender managed to get a message delivered despite your best efforts, you could correct the situation by adding the spam sender to the spamd-greytrap list like this:

$ sudo spamdb -a -t 192.168.2.128

Adding a new trap address is just as simple:

$ sudo spamdb -a -T [email protected]

If you want to reverse either of these decisions, you would simply substitute -d for the -a option in both these commands.

Keeping spamd Greylists in Sync

Beginning with OpenBSD 4.1, spamd can keep greylisting databases in sync across any number of cooperating greylisting gateways. The implementation is via a set of spamd command-line options:

  • The -Y option specifies a sync target—that is, the IP address(es) of other spamd-running gateways you want to inform of updates to your greylisting information.

  • On the receiving end, the -y option specifies a sync listener, which is the address or interface where this spamd instance is prepared to receive greylisting updates from other hosts.

For example, our main spamd gateway mainoffice-gw.example.com might have the following options added to its startup command line to establish a sync target and sync listener, respectively:

-Y minorbranch-gw.example.com -y mainoffice-gw.example.com

Conversely, minorbranch-gw.example.com at the branch office would have the hostnames reversed:

-Y mainoffice-gw.example.com -y minorbranch-gw.example.com

The spamd daemon also supports shared-secret authentication between the synchronization partners. Specifically, if you create the file /etc/mail/ spamd.key and distribute copies of it to all synchronization partners, it’ll be used to calculate the necessary checksums for authentication. The spamd.key file itself can be any kind of data, such as random data harvested from /dev/arandom, as suggested by the spamd man page.

Note

In situations where direct synchronization of spamd-related data isn’t practical or if you simply want to share your spamd-greytrap with others, exporting the contents of your list of locally trapped spam senders to a text file may be desirable. The list format spamd-setup expects is one address per line, optionally with comment lines starting with one or more # characters. Exporting your list of currently trapped addresses in a usable format can be as simple as putting together a one-liner with spamdb, grep, and a little imagination.

Detecting Out-of-Order MX Use

OpenBSD 4.1 gave spamd the ability to detect out-of-order MX use. Contacting a secondary mail exchanger first instead of trying the main one is a fairly well-known spammer trick and one that runs contrary to the behavior we expect from ordinary email transfer agents. In other words, if someone tries the email exchangers in the wrong order, we can be pretty sure that they’re trying to deliver spam.

For our example.com domain with main mail server 192.0.2.225 and backup 192.0.2.224, adding -M 192.0.2.224 to spamd’s startup options would mean that any host that tries to contact 192.0.2.224 via SMTP before contacting the main mail server at 192.0.2.225 will be added to the local spamdgreytrap list for the next 24 hours.

Handling Sites That Do Not Play Well with Greylisting

Unfortunately, there are situations where you’ll need to compensate for the peculiarities of other sites’ email setups.

The first email message sent from any site that hasn’t contacted you for as long as the greylister keeps its data around will be delayed for some random amount of time, which depends mainly on the sender’s retry interval. There are times when even a minimal delay is undesirable. If, for example, you have some infrequent customers who demand your immediate and urgent attention to their business when they do contact you, an initial delivery delay of what could be up to several hours may not be optimal. In addition, you are bound to encounter misconfigured mail servers that either don’t retry at all or retry too quickly, perhaps stopping delivery retries after just one attempt.

Also, some sites are large enough to have several outgoing SMTP servers, and they don’t play well with greylisting because they’re not guaranteed to retry delivery of any given message from the same IP address used with the prior delivery attempt. Even though those sites comply with the retry requirements, it’s obvious that this is one of the few remaining downsides of greylisting.

One way to compensate for such situations is to define a table for a local whitelist to be fed from a file in case of reboots. To make sure SMTP traffic from the addresses in the table is not fed to spamd, add a pass rule to allow the traffic to pass:

table <nospamd> persist file "/etc/mail/nospamd"
pass in log on egress proto tcp from <nospamd> to port smtp

In pre–OpenBSD 4.7 syntax, add a no rdr rule at the top of your redirection block and a matching pass rule to let SMTP traffic from the hosts in your nospamd table through, as shown here:

no rdr proto tcp from <nospamd> to $mailservers port smtp
pass in log on egress proto tcp from <nospamd> to port smtp

Once you’ve made these changes to your rule set, enter the addresses you need to protect from redirection into the /etc/mail/nospamd file. Then reload your rule set using pfctl -f /etc/pf.conf. You can then use all the expected table tricks on the <nospamd> table, including replacing its content after editing the nospamd file. In fact, this approach is strongly hinted at in both man pages and sample configuration files distributed with recent versions of spamd.

At least some sites with many outgoing SMTP servers publish information about which hosts are allowed to send email for their domain via Sender Policy Framework (SPF) records as part of the domain’s DNS information.[38] To retrieve the SPF records for our example.com domain, use the host command’s -ttxt option as follows:

$ host -ttxt example.com

This command would produce an answer roughly like the following:

example.com descriptive text "v=spf1 ip4:192.0.2.128/25 -all"

Here, the text in quotes is the example.com domain’s SPF record. If you want email from example.com to arrive quickly and you trust the people there not to send or relay spam, choose the address range from the SPF record, add it to your nospamd file, and reload the <nospamd> table contents from the updated file.

Spam-Fighting Tips

When used selectively, blacklists combined with spamd are powerful, precise, and efficient spam-fighting tools. The load on the spamd machine is minimal. On the other hand, spamd will never perform better than its weakest data source, which means you’ll need to monitor your logs and use whitelisting when necessary.

It’s also feasible to run spamd in a pure greylisting mode, with no blacklists. In fact, some users report that a purely greylisting spamd configuration is about as effective a spam-fighting tool as configurations with blacklists and sometimes significantly more effective than content filtering. One such report posted to openbsd-misc claimed that a pure greylisting configuration immediately rid the company of approximately 95 percent of its spam load. (This report is accessible via http://marc.info/, among other places; search for the subject “Followup – spamd greylisting results.”)

I recommend two very good blacklists. One is Bob Beck’s traplist based on “ghosts of usenet postings past.” Generated automatically by computers running spamd at the University of Alberta, Bob’s setup is a regular spamd system that removes trapped addresses automatically after 24 hours, which means that you get an extremely low number of false positives. The number of hosts varies widely and has been as high as 670,000. While still officially in testing, the list was made public in January 2006. The list is available from http://www.openbsd.org/spamd/traplist.gz. It’s part of recent sample spamd.conf files as the uatraps blacklist.

The other list I recommend is heise.de’s nixspam, which has a 12-hour automatic expiry and extremely good accuracy. It’s also in the sample spamd.conf file. Detailed information about this list is available from http://www.heise.de/ix/nixspam/dnsbl_en/.

Once you’re happy with your setup, try introducing local greytrapping. This is likely to catch a few more undesirables, and it’s good, clean fun. Some limited experiments—carried out while writing this chapter (chronicled at http://bsdly.blogspot.com/ in entries starting with http://bsdly.blogspot.com/2007/07/hey-spammer-heres-list-for-you.html)—even suggest that harvesting the invalid addresses spammers use from your mail server logs, from spamd logs, or directly from your greylist to put in your traplist is extremely efficient. Publishing the list on a moderately visible Web page appears to ensure that the addresses you put there will be recorded over and over again by address-harvesting robots and will provide you with even better greytrapping material, as they’re then more likely to be kept on the spammers’ list of known good addresses.


[30] At the time this chapter was first written, this was purely theoretical; I hadn’t yet had any credible reports that this was happening. That changed during 2012 when reliable sources started reporting the appearance of brute-force sequences at odd ports. See http://bsdly.blogspot.com/2013/02/theres-no-protection-in-high-ports.html for more.

[31] For an overview of the Hail Mary Cloud sequence of brute-force attempts, see the article “The Hail Mary Cloud and the Lessons Learned” at http://bsdly.blogspot.com/2013/10/the-hail-mary-cloud-and-lessons-learned.html. More resources are referenced there and in Appendix A.

[32] From a longer-term perspective, it’s fairly normal for entire networks and larger ranges of IP addresses to be reassigned to new owners in response to events in the physical, business-oriented world.

[33] Before pfctl acquired the ability to expire table entries, table expiry was more likely than not handled by the special-purpose utility expiretable. If your pfctl doesn’t have the expire option, you should seriously consider upgrading to a newer system. If upgrading is for some reason not practical, look for expiretable in your package system.

[34] The original Harris paper and a number of other useful articles and resources can be found at http://www.greylisting.org/.

[35] The relevant parts of RFC 5321 are identical to the corresponding parts of RFC 2821, which is obsolete. Some of us were more than a little disappointed that the IETF didn’t clarify these chunks of the text, now moving forward on the standards track. My reaction (actually, it’s quite a rant) is at http://bsdly.blogspot.com/2008/10/ietf-failed-to-account-for-greylisting.html.

[36] The relevant RFCs are mainly RFC 1123 and RFC 5321, which made obsolete the earlier RFC 2821. Remember that temporary rejection is an SMTP fault-tolerance feature.

[37] Of course, this address is totally bogus. It looks like the kind of message ID the GNUS email and news client generates, and it was probably lifted from a news spool or some unfortunate malware victim’s mailbox.

[38] SPF records are stored in DNS zones as TXT records. See http://www.openspf.org/ for details.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
52.15.42.128