Starting with OpenBSD 5.0, a priority scheme for classifying network traffic on a per-rule basis is available. The range of priorities is from 0 to 7, where 0 is lowest priority. Items assigned priority 7 will skip ahead of everything else, and the default value 3 is automatically assigned for most kinds of traffic. The priority scheme, which you’ll most often hear referred to as prio after the PF syntax keyword, is always enabled, and you can tweak your traffic by setting priorities via your match or pass rules.

For example, to speed up your outgoing SSH traffic to the max, you could put a rule like this in your configuration:

pass proto tcp to port ssh set prio 7

Then your SSH traffic would be served before anything else.

You could then examine the rest of your rule set and decide what traffic is more or less important, what you would like always to reach its destination, and what parts of your traffic you feel matter less.

To push your Web traffic ahead of everything else and bump up the priority for network time and name services, you could amend your configuration with rules like these:

pass proto tcp to port { www https } set prio 7
pass proto { udp tcp } to port { domain ntp } set prio 6

Or if you have a rule set that already includes rules that match criteria other than just the port, you could achieve much the same effect by writing your priority traffic shaping as match rules instead:

match proto tcp to port { www https } set prio 7
match proto { udp tcp } to port { domain ntp } set prio 6

In some networks, time-sensitive traffic, like Voice over Internet Protocol (VoIP), may need special treatment. For VoIP, a priority setup like this may improve phone conversation quality:

voip_ports="{ 2027 4569 5036 5060 10000:20000 }"
match proto udp to port $voip_ports set prio 7

But do check your VoIP application’s documentation for information on what specific ports it uses. In any case, using match rules like these can have a positive effect on your configuration in other ways, too: You can use match rules like the ones in the examples here to separate filtering decisions—such as passing, blocking, or redirecting—from traffic-shaping decisions, and with that separation in place, you’re likely to end up with a more readable and maintainable configuration.

It’s also worth noting that parts of the OpenBSD network stack set default priorities for certain types of traffic that the developers decided was essential to a functional network. If you don’t set any priorities, anything with proto carp and a few other management protocols and packet types will go by priority 6, and all types of traffic that don’t receive a specific classification with a set prio rule will have a default priority of 3.

In the examples just shown, we set different priorities for different types of traffic and managed to get specific types of traffic, such as VoIP and SSH, to move faster than others. But thanks to the design of TCP, which carries the bulk of your traffic, even a simple priority-shaping scheme has more to offer with only minor tweaks to the rule set.

As readers of RFCs and a few practitioners have discovered, the connection-oriented design of TCP means that for each packet sent, the sender will expect to receive an acknowledgment (ACK) packet back within a preset time or matching a defined “window” of sequence numbers. If the sender doesn’t receive the acknowledgment within the expected limit, she assumes the packet was lost in transit and arranges to resend the data.

One other important factor to consider is that by default, packets are handled in the order they arrive. This is known as first in, first out (FIFO), and it means that the essentially dataless ACK packets will be waiting their turn in between the larger data packets. On a busy or congested link, which is exactly where traffic shaping becomes interesting, waiting for ACKs and performing retransmissions can eat measurably into effective bandwidth and slow down all transfers. In fact, concurrent transfers in both directions can slow each other significantly more than the value of their expected data sizes.^[39]

Fortunately, a simple and quite popular solution to this problem is at hand: You can use priorities to make sure those smaller packets skip ahead. If you assign two priorities in a match or pass rule, like this:

match out on egress set prio (5, 6)

The first priority will be assigned to the regular traffic, while ACK packets and other packets with a low delay type of service (ToS) will be assigned the second priority and will be served faster than the regular packets.

When a packet arrives, PF detects the ACK packets and puts them on the higher-priority queue. PF also inspects the ToS field on arriving packets. Packets that have the ToS set to low delay to indicate that the sender wants speedier delivery also get the high-priority treatment. When more than one priority is indicated, as in the preceding rule, PF assigns priority accordingly. Packets with other ToS values are processed in the order they arrive, but with ACK packets arriving faster, the sender spends less time waiting for ACKs and resending presumably lost data. The net result is that the available bandwidth is used more efficiently. (The match rule quoted here is the first one I wrote in order to get a feel for the new prio feature—on a test system, of course—soon after it was committed during the OpenBSD 5.0 development cycle. If you put that single match rule on top of an existing rule set, you’ll probably see that the link can take more traffic and more simultaneous connections before noticeable symptoms of congestion turn up.)

See whether you can come up with a way to measure throughout before and after you introduce the two-priorities trick to your traffic shaping, and note the difference before you proceed to the more complex traffic-shaping options.

Underlying any queue system you define using the queue system in OpenBSD 5.5 and later is the Hierarchical Fair Service Curve (HFSC) algorithm. HFSC was designed to allocate resources fairly among queues in a hierarchy. One of its interesting features is that it imposes no limits until some part of the traffic reaches a volume that’s close to its preset limits. The algorithm starts shaping just before the traffic reaches a point where it deprives some other queue of its guaranteed minimum share.

Now that you know at least the basics of the theory behind the OpenBSD queue system, let’s see how queues work.

You’ll often find that certain traffic should receive a higher priority than other traffic. For example, you’ll often want important traffic, such as mail and other vital services, to have a baseline amount of bandwidth available at all times, while other services, such as peer-to-peer file sharing, shouldn’t be allowed to consume more than a certain amount. To address these kinds of issues, queues offer a wider range of options than the pure-priority scheme.

The first queue example builds on the rule sets from earlier chapters. The scenario is that we have a small local network, and we want to let the users on the local network connect to a predefined set of services outside their own network while also letting users from outside the local network access a Web server and an FTP server somewhere on the local network.

In the following example, all queues are set up with the root queue, called main, on the external, Internet-facing interface. This approach makes sense mainly because bandwidth is more likely to be limited on the external link than on the local network. In principle, however, allocating queues and running traffic shaping can be done on any network interface.

This setup includes a queue for a total bandwidth of 20Mb with six subqueues.

queue main on $ext_if bandwidth 20M
      queue defq parent main bandwidth 3600K default
      queue ftp parent main bandwidth 2000K
      queue udp parent main bandwidth 6000K
      queue web parent main bandwidth 4000K
      queue ssh parent main bandwidth 4000K
        queue ssh_interactive parent ssh bandwidth 800K
        queue ssh_bulk parent ssh bandwidth 3200K
queue icmp parent main bandwidth 400K

The subqueue defq, shown in the preceding example, has a bandwidth allocation of 3600K, or 18 percent of the bandwidth, and is designated as the default queue. This means any traffic that matches a pass rule but that isn’t explicitly assigned to some other queue ends up here.

The other queues follow more or less the same pattern, up to subqueue ssh, which itself has two subqueues (the two indented lines below it). Here, we see a variation on the trick of using two separate priorities to speed up ACK packets, and as we’ll see shortly, the rule that assigns traffic to the two SSH subqueues assigns different priorities. Bulk SSH transfers, typically SCP file transfers, are transmitted with a ToS indicating throughput, while interactive SSH traffic has the ToS flag set to low delay and skips ahead of the bulk transfers. The interactive traffic is likely to be less bandwidth consuming and gets a smaller share of the bandwidth, but it receives preferential treatment because of the higher-priority value assigned to it. This scheme also helps the speed of SCP file transfers because the ACK packets for the SCP transfers will be assigned a higher priority.

Finally, we have the icmp queue, which is reserved for the remaining 400K, or 2 percent, of the bandwidth from the top level. This guarantees a minimum amount of bandwidth for ICMP traffic that we want to pass but that doesn’t match the criteria that would have it assigned to the other queues.

To tie the queues into the rule set, we use the pass rules to indicate which traffic is assigned to the queues and their criteria.

set skip on { lo, $int_if }
pass log quick on $ext_if proto tcp to port ssh 
    queue (ssh_bulk, ssh_interactive) set prio (5,7)
pass in quick on $ext_if proto tcp to port ftp queue ftp
pass in quick on $ext_if proto tcp to port www queue http
pass out on $ext_if proto udp queue udp
pass out on $ext_if proto icmp queue icmp
pass out on $ext_if proto tcp from $localnet to port $client_out ➊

The rules for ssh, ftp, www, udp, and icmp assign traffic to their respective queues, and we note again that the ssh queue’s subqueues are assigned traffic with two different priorities. The last catchall rule ➊ passes all other outgoing traffic from the local network, lumping it into the default defq queue.

You can always let a block of match rules do the queue assignment instead in order to make the configuration even more flexible. With match rules in place, you move the filtering decisions to block, pass, or even redirect to a set of rules elsewhere.

match log quick on $ext_if proto tcp to port ssh 
    queue (ssh_bulk, ssh_interactive) set prio (5,7)
match in quick on $ext_if proto tcp to port ftp queue ftp
match in quick on $ext_if proto tcp to port www queue http
match out on $ext_if proto udp queue udp
match out on $ext_if proto icmp queue icmp

Note that with match rules performing the queue assignment, there’s no need for a final catchall to put the traffic that doesn’t match the other rules into the default queue. Any traffic that doesn’t match these rules and that’s allowed to pass will end up in the default queue.

Fixed bandwidth allocations are nice, but network admins with traffic-shaping ambitions tend to look for a little more flexibility once they’ve gotten their feet wet. Wouldn’t it be nice if there were a regime with flexible bandwidth allocation, offering guaranteed lower and upper bounds for bandwidth available to each queue and variable allocations over time—and one that starts shaping only when there’s an actual need to do so?

The good news is that the OpenBSD queues can do just that, courtesy of the underlying HFSC algorithm discussed earlier. HFSC makes it possible to set up queuing regimes with guaranteed minimum allocations and hard upper limits, and you can even have allocations that include burst values to let available capacity vary over time.

Working from a typical gateway configuration like the ones we’ve altered incrementally over the earlier chapters, we insert this queue definition early in the pf.conf file:

queue rootq on $ext_if bandwidth 20M
        queue main parent rootq bandwidth 20479K min 1M max 20479K qlimit 100
             queue qdef parent main bandwidth 9600K min 6000K max 18M default
             queue qweb parent main bandwidth 9600K min 6000K max 18M
             queue qpri parent main bandwidth 700K min 100K max 1200K
             queue qdns parent main bandwidth 200K min 12K burst 600K for 3000ms
        queue spamd parent rootq bandwidth 1K min 0K max 1K qlimit 300

This definition has some characteristics that are markedly different from the previous one in Introducing Queues for Bandwidth Allocation. We start with this rather small hierarchy by splitting the top-level queue, rootq, into two. Next, we subdivide the main queue into several subqueues, all of which have a min value set—the guaranteed minimum bandwidth allocated to the queue. (The max value would set a hard upper limit on the queue’s allocation.) The bandwidth parameter also sets the allocation the queue will have available when it’s backlogged—that is, when it’s started to eat into its qlimit, or queue limit, allocation.

The queue limit parameter works like this: In case of congestion, each queue by default has a pool of 50 slots, the queue limit, to keep packets around when they can’t be transmitted immediately. Here, the top-level queues, main and spamd, both have larger-than-default pools set by their qlimit setting: 100 for main and 300 for spamd. Cranking up these qlimit sizes means we’re a little less likely to drop packets when the traffic approaches the set limits, but it also means that when the traffic shaping kicks in, we’ll see increased latency for connections that end up in these larger pools.

The next step is to tie the newly created queues into the rule set. If you have a filtering regime in place already, the tie-in is simple—just add a few match rules:

match out on $ext_if proto tcp to port { www https } 
    set queue (qweb, qpri) set prio (5,6)
match out on $ext_if proto { tcp udp } to port domain 
    set queue (qdns, qpri) set prio (6,7)
match out on $ext_if proto icmp 
    set queue (qdns, qpri) set prio (6,7)

Here, the match rules once again do the ACK packet speedup trick with the high- and low-priority queue assignment, just as we saw earlier in the pure-priority-based system. The only exception is when we assign traffic to our lowest-priority queue (with a slight modification to an existing pass rule), where we really don’t want any speedup.

pass in log on egress proto tcp to port smtp 
    rdr-to 127.0.0.1 port spamd set queue spamd set prio 0

Assigning the spamd traffic to a minimal-sized queue with 0 priority here is intended to slow down the spammers on their way to our spamd. (See Chapter 6 for more on spamd and related matters.)

With the queue assignment and priority setting in place, it should be clear that the queue hierarchy here uses two familiar tricks to make efficient use of available bandwidth. First, it uses a variation of the high- and low-priority mix demonstrated in the earlier pure-priority example. Second, we speed up almost all other traffic, especially the Web traffic, by allocating a small but guaranteed portion of bandwidth for name service lookups. For the qdns queue, we set the burst value with a time limit: After 3000 milliseconds, the allocation goes down to a minimum of 12K to fit within the total 200K quota. Short-lived burst values like this can be useful to speed connections that transfer most of their payload during the early phases.

It may not be immediately obvious from this example, but HFSC requires that traffic be assigned only to leaf queues, or queues without subqueues. That means it’s possible to assign traffic to main’s subqueues—qpri, qdef, qweb, and qdns—as well as rootq’s subqueue—spamd—as we just did with the match and pass rules, but not to rootq or main themselves. With all queue assignments in place, we can use systat queues to show the queues and their traffic:

    6 users Load 0.31 0.28 0.34                   Tue May 19 21:31:54 2015

QUEUE            BW SCH  PR PKTS BYTES DROP_P DROP_B QLEN BORR SUSP P/S B/S
rootq           20M            0     0      0      0    0
main            20M            0     0      0      0    0
 qdef            9M        48887   15M      0      0    0
 qweb            9M        18553 8135K      0      0    0
 qpri          600K        37549 2407K      0      0    0
 qdns          200K        15716 1568K      0      0    0
spamd            1K        10590 661K     126   8772   47

The queues are shown indented to indicate their hierarchy, from root to leaf queues. The main queue and its subqueues—qpri, qdef, qweb, and qdns—are shown with their bandwidth allocations and number of bytes and packets passed. The DROP_P and DROP_B columns, which show the number of packets and bytes dropped, would appear if we had been forced to drop packets at this stage. QLEN is the number of packets waiting for processing, while the final two columns show live updates of packets and bytes per second.

For a more detailed view, use pfctl -vvsq to show the queues and their traffic:

queue rootq on xl0 bandwidth 20M qlimit 50
  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/ 50 ]
  [ measured:     0.0 packets/s, 0 b/s ]
queue main parent rootq on xl0 bandwidth 20M, min 1M, max 20M qlimit 100
  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/100 ]
  [ measured:     0.0 packets/s, 0 b/s ]
queue qdef parent main on xl0 bandwidth 9M, min 6M, max 18M default qlimit 50
  [ pkts:       1051  bytes:     302813  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/ 50 ]
  [ measured:   2.6 packets/s, 5.64Kb/s ]
queue qweb parent main on xl0 bandwidth 9M, min 6M, max 18M qlimit 50
  [ pkts:       1937  bytes:    1214950  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/ 50 ]
  [ measured:     3.6 packets/s, 13.65Kb/s ]
queue qpri parent main on xl0 bandwidth 600K, max 1M qlimit 50
  [ pkts:       2169  bytes:     143302  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/ 50 ]
  [ measured:     6.6 packets/s, 3.55Kb/s ]
queue qdns parent main on xl0 bandwidth 200K, min 12K burst 600K for 3000ms qlimit 50
  [ pkts:        604 bytes:       65091  dropped pkts:      0 bytes:      0 ]
  [ qlength:    0/ 50 ]
  [ measured:      1.6 packets/s, 1.31Kb/s ]
queue spamd parent rootq on xl0 bandwidth 1K, max 1K qlimit 300
  [ pkts:        884 bytes:       57388  dropped pkts:      0 bytes: 0 ]
  [ qlength: 176/300 ]
  [ measured:     1.9 packets/s, 1Kb/s ]

This view shows that the queues receive traffic roughly as expected with the site’s typical workload. Notice that only a few moments after the rule set has been reloaded, the spamd queue is already backed up more than halfway to its qlimit setting, which seems to indicate that the queues are reasonably dimensioned to actual traffic.

In Chapter 5, we set up a network with a single gateway and all externally visible services configured on a separate DMZ (demilitarized zone) network so that all traffic to the servers from both the Internet and the internal network had to pass through the gateway. That network schematic, illustrated in Chapter 5, is shown again in Figure 7-1. Using the rule set from Chapter 5 as the starting point, we’ll add some queuing in order to optimize our network resources. The physical and logical layout of the network will not change.

Figure 7-1. Network with DMZ

The most likely bottleneck for this network is the bandwidth for the connection between the gateway’s external interface and the Internet. Although the bandwidth elsewhere in our setup isn’t infinite, of course, the available bandwidth on any interface in the local network is likely to be less limiting than the bandwidth actually available for communication with the outside world. In order to make services available with the best possible performance, we need to set up the queues so that the bandwidth available at the site is made available to the traffic we want to allow. The interface bandwidth on the DMZ interface is likely either 100Mb or 1Gb, while the actual available bandwidth for connections from outside the local network is considerably smaller. This consideration shows up in our queue definitions, where the actual bandwidth available for external traffic is the main limitation in the queue setup.

queue ext on $ext_if bandwidth 2M
        queue ext_main parent ext bandwidth 500K default
        queue ext_web parent ext bandwidth 500K
        queue ext_udp parent ext bandwidth 400K
        queue ext_mail parent ext bandwidth 600K

queue dmz on $dmz_if bandwidth 100M
        queue ext_dmz parent dmz bandwidth 2M
                queue ext_dmz_web parent ext_dmz bandwidth 800K default
                queue ext_dmz_udp parent ext_dmz bandwidth 200K
                queue ext_dmz_mail parent ext_dmz bandwidth 1M
        queue dmz_main parent dmz bandwidth 25M
        queue dmz_web parent dmz bandwidth 25M
        queue dmz_udp parent dmz bandwidth 20M
        queue dmz_mail parent dmz bandwidth 20M

Notice that for each interface, there’s a root queue with a bandwidth limitation that determines the allocation for all queues attached to that interface. In order to use the new queuing infrastructure, we need to make some changes to the filtering rules, too.

The main part of the filtering rules could end up looking like this after adding the queues:

pass in on $ext_if proto { tcp, udp } to $nameservers port domain 
    set queue ext_udp set prio (6,5)
pass in on $int_if proto { tcp, udp } from $localnet to $nameservers 
    port domain
pass out on $dmz_if proto { tcp, udp } to $nameservers port domain 
    set queue ext_dmz_udp set prio (6,5)
pass out on $dmz_if proto { tcp, udp } from $localnet to $nameservers 
    port domain set queue dmz_udp
pass in on $ext_if proto tcp to $webserver port $webports set queue ext_web
pass in on $int_if proto tcp from $localnet to $webserver port $webports
pass out on $dmz_if proto tcp to $webserver port $webports 
    set queue ext_dmz_web
pass out on $dmz_if proto tcp from $localnet to $webserver port $webports 
    set queue dmz_web
pass in log on $ext_if proto tcp to $mailserver port smtp
pass in log on $ext_if proto tcp from $localnet to $mailserver port smtp
pass in log on $int_if proto tcp from $localnet to $mailserver port $email
pass out log on $dmz_if proto tcp to $mailserver port smtp set queue ext_mail
pass in on $dmz_if proto tcp from $mailserver to port smtp set queue dmz_mail
pass out log on $ext_if proto tcp from $mailserver to port smtp 
    set queue ext_dmz_mail

Notice that only traffic that will pass either the DMZ or the external interface is assigned to queues. In this configuration, with no externally accessible services on the internal network, queuing on the internal interface wouldn’t make much sense because that’s likely the part of the network with the least restricted available bandwidth. Also, as in earlier examples, there’s a case to be made for separating the queue assignments from the filtering part of the rule set by making a block of match rules responsible for queue assignment.

In Turning Away the Brutes, we used a combination of state-tracking options and overload rules to fill a table of addresses for special treatment. The special treatment we demonstrated in Chapter 6 was to cut all connections, but it’s equally possible to assign overload traffic to a specific queue instead. For example, consider the rule from our first queue example, shown here.

pass log quick on $ext_if proto tcp to port ssh flags S/SA 
    keep state queue (ssh_bulk, ssh_interactive) set prio (5,7)

To create a variation of the overload table trick from Chapter 6, add state-tracking options, like this:

pass log quick on $ext_if proto tcp to port ssh flags S/SA 
    keep state (max-src-conn 15, max-src-conn-rate 5/3, 
    overload <bruteforce> flush global) queue (ssh_bulk, ssh_interactive) 
    set prio (5,7)

Then, make one of the queues slightly smaller:

queue smallpipe parent main bandwidth 512

And assign traffic from miscreants to the small-bandwidth queue with this rule:

pass inet proto tcp from <bruteforce> to port $tcp_services queue smallpipe

As a result, the traffic from the bruteforcers would pass, but with a hard upper limit of 512 bits per second. (It’s worth noting that tiny bandwidth allocations may be hard to enforce on high-speed links due to the network stack’s timer resolution. If the allocation is small enough relative to the capacity of the link, packets that exceed the stated per-second maximum allocation may be transferred anyway, before the bandwidth limit kicks in.) It might also be useful to supplement rules like these with table-entry expiry, as described in Tidying Your Tables with pfctl.

Chapter 6 covered several ways to use spamd to cut down on spam. If running spamd isn’t an option in your environment, you can use a queue and rule set based on the knowledge that machines that send spam are likely to run a particular operating system. (Let’s call that operating system Windows.)

PF has a fairly reliable operating system fingerprinting mechanism, which detects the operating system at the other end of a network connection based on characteristics of the initial SYN packets at connection setup. The following may be a simple substitute for spamd if you’ve determined that legitimate mail is highly unlikely to be delivered from systems that run that particular operating system.

pass quick proto tcp from any os "Windows" to $ext_if 
    port smtp set queue smallpipe

Here, email traffic originating from hosts that run a particular operating system get no more than 512 bits per second of your bandwidth.

Priority-based queues are defined purely in terms of priority within the total declared bandwidth. For priq queues, the allowed priority range is 0 through 15, where a higher value earns preferential treatment. Packets that match the criteria for higher-priority queues are serviced before the ones matching lower-priority queues.

Class-based queues are defined as constant-sized bandwidth allocations, as a percentage of the total available or in units of kilobits, megabits, or gigabits per second. A cbq queue can be subdivided into queues that are also assigned priorities in the range 0 to 7, and again, a higher priority means preferential treatment.

The hfsc discipline uses the HFSC algorithm to ensure a “fair” allocation of bandwidth among the queues in a hierarchy. HFSC comes with the possibility of setting up queuing regimes with guaranteed minimum allocations and hard upper limits. Allocations can even vary over time, and you can even have fine-grained priority with a 0 to 7 range.

Because both the algorithm and the corresponding setup with ALTQ are fairly complicated, with a number of tunable parameters, most ALTQ practitioners tend to stick with the simpler queue types. Yet the ones who claim to understand HFSC swear by it.

On OpenBSD 5.5, all supported queue disciplines are compiled into the GENERIC and GENERIC.MP kernels. Check that your OpenBSD version still supports ALTQ. If so, the only configuration you need to do involves editing your pf.conf.

On FreeBSD, make sure that your kernel has ALTQ and the ALTQ queue discipline options compiled in. The default FreeBSD GENERIC kernel doesn’t have ALTQ options enabled, as you may have noticed from the messages you saw when running the /etc/rc.d/pf script to enable PF. The relevant options are as follows:

options            ALTQ
options            ALTQ_CBQ        # Class Bases Queuing (CBQ)
options            ALTQ_RED        # Random Early Detection (RED)
options            ALTQ_RIO        # RED In/Out
options            ALTQ_HFSC       # Hierarchical Packet Scheduler (HFSC)
options            ALTQ_PRIQ       # Priority Queuing (PRIQ)
options            ALTQ_NOPCC      # Required for SMP build

The ALTQ option is needed to enable ALTQ in the kernel, but on SMP systems, you also need the ALTQ_NOPCC option. Depending on which types of queues you’ll be using, you’ll need to enable at least one of these: ALTQ_CBQ, ALTQ_PRIQ, or ALTQ_HFSC. Finally, you can enable the congestion-avoidance techniques random early detection (RED) and RED In/Out with the ALTQ_RED and ALTQ_RIO options, respectively. (See the FreeBSD Handbook for information on how to compile and install a custom kernel with these options.)

ALTQ was integrated into the NetBSD 4.0 PF implementation and is supported in NetBSD 4.0 and later releases. NetBSD’s default GENERIC kernel configuration doesn’t include the ALTQ-related options, but the GENERIC configuration file comes with all relevant options commented out for easy inclusion. The main kernel options are these:

options    ALTQ         # Manipulate network interfaces' output queues
options    ALTQ_CBQ     # Class-Based queuing
options    ALTQ_HFSC    # Hierarchical Fair Service Curve
options    ALTQ_PRIQ    # Priority queuing
options    ALTQ_RED     # Random Early Detection

The ALTQ option is needed to enable ALTQ in the kernel. Depending on the types of queues you’ll be using, you must enable at least one of these: ALTQ_CBQ, ALTQ_PRIQ, or ALTQ_HFSC.

Using ALTQ requires you to compile PF into the kernel because the PF loadable module doesn’t support ALTQ functionality. (See the NetBSD PF documentation at http://www.netbsd.org/Documentation/network/pf.html for the most up-to-date information.)

All queues are set up on the external, Internet-facing interface. This approach makes sense mainly because bandwidth is more likely to be limited on the external link than on the local network. In principle, however, allocating queues and running traffic shaping can be done on any network interface. The example setup shown here includes a cbq queue for a total bandwidth of 2Mb with six subqueues.

altq on $ext_if cbq bandwidth 2Mb queue { main, ftp, udp, web, ssh, icmp }
      queue main bandwidth 18% cbq(default borrow red)
      queue ftp bandwidth 10% cbq(borrow red)
      queue udp bandwidth 30% cbq(borrow red)
      queue web bandwidth 20% cbq(borrow red)
      queue ssh bandwidth 20% cbq(borrow red) { ssh_interactive, ssh_bulk }
            queue ssh_interactive priority 7 bandwidth 20%
            queue ssh_bulk priority 5 bandwidth 80%
      queue icmp bandwidth 2% cbq

The subqueue main has 18 percent of the bandwidth and is designated as the default queue. This means any traffic that matches a pass rule but isn’t explicitly assigned to some other queue ends up here. The borrow and red keywords mean that the queue may “borrow” bandwidth from its parent queue, while the system attempts to avoid congestion by applying the RED algorithm.

The other queues follow more or less the same pattern up to the subqueue ssh, which itself has two subqueues with separate priorities. Here, we see a variation on the ACK priority example. Bulk SSH transfers, typically SCP file transfers, are transmitted with a ToS indicating throughput, while interactive SSH traffic has the ToS flag set to low delay and skips ahead of the bulk transfers. The interactive traffic is likely to be less bandwidth consuming and gets a smaller share of the bandwidth, but it receives preferential treatment because of the higher-priority value assigned to it. This scheme also helps the speed of SCP file transfers because the ACK packets for the SCP transfers will be assigned to the higher-priority subqueue.

Finally, we have the icmp queue, which is reserved for the remaining 2 percent of the bandwidth from the top level. This guarantees a minimum amount of bandwidth for ICMP traffic that we want to pass but that doesn’t match the criteria for being assigned to the other queues.

To make it all happen, we use these pass rules, which indicate which traffic is assigned to the queues and their criteria:

set skip on { lo, $int_if }
pass log quick on $ext_if proto tcp to port ssh queue (ssh_bulk, ssh_
interactive)
pass in quick on $ext_if proto tcp to port ftp queue ftp
pass in quick on $ext_if proto tcp to port www queue http
pass out on $ext_if proto udp queue udp
pass out on $ext_if proto icmp queue icmp
pass out on $ext_if proto tcp from $localnet to port $client_out

The rules for ssh, ftp, www, udp, and icmp assign traffic to their respective queues. The last catchall rule passes all other traffic from the local network, lumping it into the default main queue.

First, working from the same configuration we altered slightly earlier, we insert this queue definition early in the pf.conf file:

altq on $ext_if bandwidth $ext_bw hfsc queue { main, spamd }
       queue main bandwidth 99% priority 7 qlimit 100 hfsc (realtime 20%, linkshare 99%) 
            { q_pri, q_def, q_web, q_dns }
          queue q_pri bandwidth 3% priority 7 hfsc (realtime 0, linkshare 3% red )
          queue q_def bandwidth 47% priority 1 hfsc (default realtime 30% linkshare 47% red)
          queue q_web bandwidth 47% priority 1 hfsc (realtime 30% linkshare 47% red)
          queue q_dns bandwidth 3% priority 7 qlimit 100 hfsc (realtime (30Kb 3000 12Kb), 
                  linkshare 3%)
       queue spamd bandwidth 0% priority 0 qlimit 300 hfsc (realtime 0, upperlimit 1%, 
            linkshare 1%)

The hfsc queue definitions take slightly different parameters than the simpler disciplines. We start off with this rather small hierarchy by splitting the top-level queue into two. At the next level, we subdivide the main queue into several subqueues, each with a defined priority. All the subqueues have a realtime value set—the guaranteed minimum bandwidth allocated to the queue. The optional upperlimit sets a hard upper limit on the queue’s allocation. The linkshare parameter sets the allocation the queue will have available when it’s backlogged—that is, when it’s started to eat into its qlimit allocation.

In case of congestion, each queue by default has a pool of 50 slots, the queue limit (qlimit), to keep packets around when they can’t be transmitted immediately. In this example, the top-level queues main and spamd both have larger-than-default pools set by their qlimit setting: 100 for main and 300 for spamd. Cranking up queue sizes here means we’re a little less likely to drop packets when the traffic approaches the set limits, but it also means that when the traffic shaping kicks in, we’ll see increased latency for connections that end up in these larger than default pools.

The queue hierarchy here uses two familiar tricks to make efficient use of available bandwidth:

Next, we tie the newly created queues into the rule set. If you have a filtering regime in place already, which we’ll assume you do, the tie-in becomes amazingly simple, accomplished by adding a few match rules.

match out on $ext_if from $air_if:network nat-to ($ext_if) 
    queue (q_def, q_pri)
match out on $ext_if from $int_if:network nat-to ($ext_if) 
    queue (q_def, q_pri)
match out on $ext_if proto tcp to port { www https } queue (q_web, q_pri)
match out on $ext_if proto { tcp udp } to port domain queue (q_dns, q_pri)
match out on $ext_if proto icmp queue (q_dns, q_pri)

Here, the match rules once again do the ACK packet speedup trick with the high- and low-priority queue assignment, just as you saw earlier in the pure-priority-based system. The only exception is when we assign traffic to our lowest-priority queue, where we really don’t care to have any speedup at all.

pass in log on egress proto tcp to port smtp rdr-to 127.0.0.1 port spamd queue spamd

This rule is intended to slow down the spammers a little more on their way to our spamd. With a hierarchical queue system in place, systat queues shows the queues and their traffic as a hierarchy, too.

  2 users    Load 0.22 0.25 0.25                    Fri Apr 3 16:43:37 2015

QUEUE         BW  SCH  PRIO    PKTS    BYTES  DROP_P  DROP_B QLEN  BORROW SUSPEN P/S    B/S
root_nfe0     20M hfsc    0       0        0       0       0    0                  0      0
 main         19M hfsc    7       0        0       0       0    0                  0      0
  q_pri      594K hfsc    7    1360    82284       0       0    0                 11    770
  q_def     9306K hfsc          158    15816       0       0    0                0.2     11
  q_web     9306K hfsc          914   709845       0       0    0                 50  61010
  q_dns      594K hfsc    7     196    17494       0       0    0                  3    277
  spamd         0 hfsc    0     431    24159       0       0    0                  2    174

The root queue is shown as attached to the physical interface—as nfe0 and root_nfe0, in this case. main and its subqueues—q_pri, q_def, q_web, and q_dns—are shown with their bandwidth allocations and number of bytes and packets passed. The DROP_P and DROP_B columns are where number of packets and bytes dropped, respectively, would appear if we had been forced to drop packets at this stage. The final two columns show live updates of packets per second and bytes per second.

Think back to Turning Away the Brutes, where we used a combination of state-tracking options and overload rules to fill up a table of addresses for special treatment. The special treatment we demonstrated in Chapter 6 was to cut all connections, but it’s equally possible to assign overload traffic to a specific queue instead.

Consider this rule from our class-based bandwidth example in Class-Based Bandwidth Allocation for Small Networks.

pass log quick on $ext_if proto tcp to port ssh flags S/SA 
   keep state queue (ssh_bulk, ssh_interactive)

We could add state-tracking options, as shown in here.

pass log quick on $ext_if proto tcp to port ssh flags S/SA 
   keep state (max-src-conn 15, max-src-conn-rate 5/3, 
   overload <bruteforce> flush global) queue (ssh_bulk, ssh_interactive)

Then, we could make one of the queues slightly smaller.

queue smallpipe bandwidth 1kb cbq

Next, we could assign traffic from miscreants to the small-bandwidth queue with the following rule.

pass inet proto tcp from <bruteforce> to port $tcp_services queue smallpipe

It might also be useful to supplement rules like these with table-entry expiry, as described in Tidying Your Tables with pfctl.

Chapter 6 covered several ways to use spamd to cut down on spam. If running spamd isn’t an option in your environment, you can use a queue and rule set based on the common knowledge that machines that send spam are likely to run a particular operating system.

PF has a fairly reliable operating system fingerprinting mechanism, which detects the operating system at the other end of a network connection based on characteristics of the initial SYN packets at connection setup. The following may be a simple substitute for spamd if you’ve determined that legitimate mail is highly unlikely to be delivered from systems that run that particular operating system.

pass quick proto tcp from any os "Windows" to $ext_if port smtp queue smallpipe

Here, email traffic originating from hosts that run a particular operating system get no more than 1KB of your bandwidth, with no borrowing.