A good place to start is with software compiling, as you want to compile software and updates as efficiently as possible. Whenever you compile, your compiler makes assumptions about your hardware in order to create binaries. If you have an x86-compliant CPU, for example, your compiler will create binaries that can run on any CPU from a 386 onward. While this allows portability, it won’t take advantage of any new abilities of your CPU, such as the extended MMX, SSE, SSE2, or 3DNow! instruction sets. This is also why using precompiled binaries on your system is a surefire way to reduce your overall performance.

To ensure that software will be compiled efficiently, update your compiler flags in /etc/make.conf . This file does not exist on new systems, but you can copy it from /usr/share/examples/etc/defaults/make.conf.

Start by editing the CPUTYPE= line to reflect your CPU type; you’ll find supported types listed as comments just before this line. While this will take advantage of your CPU’s features, the disadvantage is that your compiled binaries may not run on different CPU types. However, if all of your systems run the same CPU platform, any optimizations you make to shared binaries will affect all of your systems equally well.

Next, change the CFLAGS line to CFLAGS= -O2 -pipe -funroll-loops. The -pipe option can significantly decrease the amount of time it takes to compile software, by using pipes to communicate between compiler processes instead of temporary files, but at the expense of using slightly more memory. The -funroll-loops saves one CPU register that would otherwise be tied up in tracking the iteration of the loop, but at the expense of making a slightly larger binary.

In your kernel configuration, add the following line after the machine i386 line:

makeoptions    COPTFLAGS="-O2 -pipe -funroll-loops -ffast-math"

This is similar to the CLAGS option in /etc/make.conf, except that it optimizes kernel compilation.

You can also add this line:

TOP_TABLE_SIZE=number

where number is a prime number that is at least twice the number of lines in /etc/passwd. This statement sets the size of the hash that top uses.

Set the following option if you have an AMD K5/K6/K6-2 or Cyrix 6x86 chip. It enables cache write allocation for the L1 cache, which is disabled by default for these chips.

options         CPU_WT_ALLOC

This option will disable NFS server code, so include it when you know that you will not be acting as an NFS server:

options        NFS_NOSERVER

Another way of saving kernel memory is to define the maximum number of swap devices, as shown in the next example. Your kernel needs to allocate a fixed amount of bitmapped memory so that it can interleave swap devices. I set the number to 1 on my workstation and 2 on my servers. If I need to add more to a server, I can easily create another partition.

options         NSWAPDEV=number

If you plan on compiling all your requisites into the kernel (NIC driver, IPF/IPFW, etc.) and won’t be loading any of these options as modules, you can include this line to skip module compiling. This saves significantly on the time taken to compile a kernel (sometimes reducing it by two-thirds).

makeoptions     MODULES_OVERRIDE=""

By default, all kernel options are compiled as modules. This allows you to use kldload to load a module even though it isn’t specified in your kernel configuration file.

The advantage of MODULES_OVERRIDE is the decrease in kernel compilation time. The disadvantage is that you’ll need to recompile your kernel if you ever need to add additional functionality, since you will have lost the ability to load the kernel module separately.

Most modern network cards and switches support the ability to auto-negotiate the communication speed. While this reduces administration, it comes at the cost of network throughput. If your switch, server, or workstation is set to use auto-negotiation, it will stop transferring network traffic every few moments to renegotiate its speed.

If your network driver supports it, you can set network speed with ifconfig at runtime or in /etc/rc.conf at boot time. Here is an example:

% grep fxp0 /etc/rc.conf
ifconfig_fxp0="inet x.x.x.x netmask x.x.x.x media 100BaseTX mediaopt 
    full-duplex"

Next, you can enable DEVICE_POLLING in your kernel, which changes the method by which data travels from your network card to the kernel. Without this setting, frequent interrupt calls may never free the kernel. This is known as livelock and can leave your machine unresponsive. Those of us unfortunate enough to be on the wrong side of certain denial-of-service attacks know about this.

The DEVICE_POLLING option causes the kernel to poll the network card at certain predefined times, during idle loops, or on clock interrupts. This allows the kernel to decide when it is most efficient to poll a device for updates and for how long, and ultimately results in a significant increase in performance.

To take advantage of DEVICE_POLLING, you need to compile two options into your kernel: options DEVICE_POLLING and options HZ=1000. The latter option slows the clock interrupts to 1,000 times per second, which prevents the kernel from polling too often.

Once you’ve recompiled your kernel, you’ll still need to enable the feature. Add this line to /etc/sysctl.conf:

kern.polling.enable=1

The DEVICE_POLLING option does not work with SMP-enabled kernels by default. If you are compiling an SMP kernel with DEVICE_POLLING, first remove the following lines from /usr/src/sys/kern/kern_poll.c:

#ifdef SMP
#include "opt_lint.h"
#ifndef COMPILING_LINT
#error DEVICE_POLLING is not compatible with SMP
#endif
#endif

Mail servers typically have a very large number of network connections, during which they transfer a small amount of data for a short period of time before closing the connection. In this case, it is useful to have a large number of small network buffers.

Network connections have two buffers, one for sending and one for receiving. The size of the buffer dictates how quickly data will funnel through the network and, in the event of a network delay, how much data can back up the server for that connection before there is a problem. Having a network buffer that is too small will cause a data backlog as the CPU waits for the network to clear, which causes greater CPU overhead. Having a network buffer that is too large wastes memory by using the buffer inefficiently. Finding a balance is the key to tuning.

I find that multiplying the number of established connections by 32 leaves me with room to breathe in the event that I see an abnormally high surge of traffic. I’ve come to this number over time through trial and error. So, if you expect to have a peak of 128 servers sending you mail, having 8,192 network buffer clusters would be good (128 2 per connection 32). Also, remember that connections can take up to two full minutes or more to close completely. If you expect more than 128 emails in any given two-minute period, increase the number accordingly.

Another important value to control is the maximum number of sockets. Start with the same number of sockets as there are network buffers, and then tune as appropriate.

You can find out how many network buffer clusters are in use with the command netstat -m. You can specify the values you want in /boot/loader.conf. For example:

kern.ipc.nmbclusters=8192
kern.ipc.maxsockets=8192

As with any performance tuning, monitor your system after making changes. Did you go overboard or underestimate what you would need? Always check and adjust accordingly.

File servers generally have longer-lived and less frequent network connections than those on mail servers. They usually transfer larger files.

To determine the optimal number of network buffer clusters, consider how many clients you have. Multiplying the number of network buffers by two is good practice, though some admins prefer to multiply by four to accommodate multiple file transfers. If you have 128 clients connecting to the file server, set the number of network buffer clusters to 1,024 (128 2 per connection 4).

If you have more than one element on your web page (for example, multiple images or frames), expect web browsers to make multiple connections to your web server. It’s common to see four connections per page served. Also count any database or network connections made in server-side scripting.

Web servers go through periods of highs and lows. While you might serve 100 pages per minute on average, at your low you might serve 10 pages per minute and at peak over 1,000 pages per minute. At a peak of 1,000 pages per minute, your clusters and sockets should be around 16,384 (1,000 pages 2 per connection 4 connections 2 for growth).

To take advantage of the traffic shaping functionality of your FreeBSD system, you need a kernel with the following options:

options IPFIREWALL
options DUMMYNET
options HZ=1000

dummynet does not require the HZ option, but its manpage strongly recommends it. See [Hack #69] for more about HZ and [Hack #54] for detailed instructions about compiling a custom kernel.

The traffic-shaping mechanism delays packets so as not to exceed the transfer speed limit. The delayed packets are stored and sent later. The kernel timer triggers sending, so setting the frequency to a higher value will smooth out the traffic by providing smaller delays. The default value of 100 Hz will trigger sends every 10 milliseconds, producing bursty traffic. Setting HZ=1000 will cause the trigger to happen every millisecond, resulting in less packet delay.

Traffic shaping occurs in three stages:

Pipes are the basic elements of the traffic shaper. A pipe emulates a network link with a certain bandwidth, delay, and packet loss rate.

Queues implement weighted fair queuing and cannot be used without a pipe. All queues connected to a pipe share the bandwidth of that pipe in a certain configurable proportion.

The most important parameter of a pipe configuration is its bandwidth. Set the bandwidth with this command:

# ipfw pipe 1 config bw 120kbit/s

This command creates pipe 1 if it does not already exist, assigning it 120 kilobits per second of bandwidth. If the pipe already exists, its bandwidth will be changed to 120 Kbps.

When configuring a queue, the two most important parameters are the pipe number it will connect to and the weight of the queue. The weight must be in the range 1 to 100, and it defaults to 1. A single pipe can connect to multiple queues.

# ipfw queue 5 config pipe 1 weight 20

This command instructs dummynet to configure queue 5 to use pipe 1, with a weight of 20. The weight parameter allows you to specify the ratios of bandwidth the queues will use. Queues with higher weights will use more bandwidth.

To calculate the bandwidth for each queue, divide the total bandwidth of the pipe by the total weights, and then multiply each weight by the result. For example, if a 120 Kbps pipe sees active traffic (called flows) from three queues with weights 3, 2, and 1, the flows will receive 60 Kbps, 40 Kbps, and 20 Kbps, respectively.

If the flow from the queue with weight 2 disappears, leaving only the flows with weights 3 and 1, those will receive 90 Kbps and 30 Kbps, respectively. (120 / (3+1) = 30, so multiply each weight by 30.)

The weight concept may seem strange, but it is rather simple. Queues with equal weights will receive the same amount of bandwidth. If queue 2 has double the weight of queue 1, it has twice as much bandwidth. Queues that have no traffic are not taken into account when dividing traffic. This means that in a configuration with two queues, one with weight 1 (for unimportant traffic) and the other with weight 99 (for important business traffic), having both queues active will result in 1%/99% sharing, but if there is no traffic on the 99 queue, the unimportant traffic will use all of the bandwidth.

Another very useful option is to create a mask by adding mask mask-specifier at the end your config line. Masks allow you to turn one flow into several flows; the mask will distinguish the different flows.

The default mask is empty, meaning all packets fall into the same flow. Using mask all would make all connections significant, meaning that every TCP or UDP connection would appear as a separate flow.

When you apply a mask to a pipe, each of that pipe’s flows acts as a separate pipe. Yet, each of those flows is an exact clone of the original pipe, in that they all share the same parameters. This means that the three active flows from our example pipe will use 360 Kbps, or 120 Kbps each.

For a queue, the flows will act as several queues, each with the same weight as the original one. This means you can use the mask to share a certain bandwidth equally. For our example with three flows and the 120 Kbps pipe, each flow will get a third of that bandwidth, or 40 Kbps.

This hack assumes that you will integrate these rules in your firewall configuration or that you are using ipfw only for traffic shaping. In the latter case, having the IPFIREWALL_DEFAULT_TO_ACCEPT option in the kernel will greatly simplify your task.

In this hack, we sometimes limit only incoming or outgoing bandwidth. Without this option, we would have to allow traffic in both directions, traffic through the loopback interface, and through the interface we will not limit.

However, you should consider disabling the IPFIREWALL_DEFAULT_TO_ACCEPT option, as it will drop packets that your policy does not specifically allow. Additionally, enabling the option may cause you to accept potentially malicious traffic you hadn’t considered. The example configurations in this hack were tested with an ipf-based firewall that had an explicit deny rule at the end.

When integrating traffic shaping into an existing ipfw firewall, keep in mind that an ipfw pipe or ipfw queue rule is equivalent to “ipfw accept after slow down . . . " if the sysctl net.inet.ip.fw.one_pass is set to 1 (the default). If the sysctl is set to 0, that rule is just a delay in a packet’s path to the next rule, which may well be a deny or another round of shaping. This hack assumes that the default behavior of the pipe and queue commands is to accept or an equivalent action.

There are several ways of limiting bandwidth. Here are some examples that assume an external interface of ed0:

# only outgoing gets limited
ipfw pipe 1 config bw 100kbits/s

ipfw add 1 pipe 1 ip from any to any out xmit ed0

To limit both incoming and outgoing to 100 and 50 Kbps, respectively:

ipfw pipe 1 config bw 100kbits/s
ipfw pipe 2 config bw 50kbits/s

ipfw add 100 pipe 1 ip from any to any in  recv ed0
ipfw add 100 pipe 2 ip from any to any out xmit ed0

To set a limitation on total bandwidth (incoming plus outgoing):

ipfw pipe 1 config bw 100kbits/s

ipfw add 100 pipe 1 ip from any to any in  recv ed0
ipfw add 100 pipe 1 ip from any to any out xmit ed0

In this example, each host gets 16 Kbps of incoming bandwidth (outgoing is not limited):

ipfw pipe 1 config bw 16kbits/s mask dst-ip 0xffffffff

ipfw add 100 pipe 1 ip from any to any in recv ed0

Here are a couple of real-life examples. Let’s start by limiting a web server’s outgoing traffic speed, which is a configuration I have used on one of my servers. The server had some FreeBSD ISO files, and I did not want it to hog all the outgoing bandwidth. I also wanted to prevent people from gaining an unfair advantage by using download accelerators, so I chose to share the total outgoing bandwidth equally among 24-bit networks.

# pipe configuration, 2000 kilobits maximum
ipfw pipe 1 config bw 2000kbits/s

# the queue will be used to enforce the /24 limit mentioned above
ipfw queue 1 config pipe 1 mask dst-ip 0xffffff00

# with this mask, only the first 24 bits of the destination IP
# address are taken into consideration when generating the flow ID

# divert outgoing traffic from the web server (at 1.1.1.1)
ipfw add queue 1 tcp from 1.1.1.1 80 to any out

Another real-life example involves limiting incoming traffic by department. This configuration limits the incoming bandwidth for a small company behind a 1 Mbps connection. Before this was applied, some users were using peer-to-peer clients and download accelerators, and they were hogging almost all the bandwidth. The solution was to implement some weighted sharing between departments and let the departments take care of their own hogs.

# Variables we will use
# External interface
EXTIF=fxp0

# My IP address
ME=192.168.1.1

# configure the pipe, 95% of total incoming capacity
ipfw pipe 1 config bw 950kbits/s

# configure the queues for the departments
# departments 1 and 2 heavy net users
ipfw queue 1 config pipe 1 weight 40
ipfw queue 2 config pipe 1 weight 40

# accounting, they shouldn't use the network a lot
ipfw queue 3 config pipe 1 weight 5

# medium usage for others
ipfw queue 4 config pipe 1 weight 20

# incoming mail (SMTP) to this server, HIGH priority
ipfw queue 10 config pipe 1 weight 100

# not caught by the previous categories - VERY LOW bandwidth
ipfw queue 11 config pipe 1 weight 1

# classify the traffic
# only incoming traffic is limited, outgoing is not affected.
ipfw add 10 allow ip from any to any out xmit via $EXTIF

# department 1
ipfw add 100 queue 1 ip from any to 192.168.0.16/28 in via $EXTIF

# department 2
ipfw add 200 queue 2 ip from any to 192.168.0.32/28 in via $EXTIF

# accounting
ipfw add 300 queue 3 ip from any to 192.168.0.48/28 in via $EXTIF

# mail
ipfw add 1000 queue 10 ip from any to $ME 25 in via $EXTIF

# others
ipfw add 1100 queue 11 ip from any to any in via $EXTIF

The incoming limit is set to 95% of the true available bandwidth. This will allow the shaper to delay some packets. If this were not the case and the pipe had the same bandwidth as the physical link, all of the delay queues for the pipe would have been empty. The extra 5% of bandwidth on the physical link fills the queues. The shaper chooses packets from the queues based on weight, passing through packets from queues with a higher weight before packets from queues with lower weight.

When you install a system, particularly a server, invest some time preparing for an emergency. On a FreeBSD system, your kit should include:

Place these items in an envelope and store it in a secure location with your backup tapes. Make a note on the envelope of the system to which this kit should apply, along with the version of the operating system. Ideally, you should have two copies of both your emergency kit and backup media. Store the second copy off-site.

Regardless of how you install a system, take a few minutes to download the *.flp files found in the floppies directory. This is especially important if you use cvsup to upgrade a system, as you can go months or years without the installation CD-ROM or floppy media. Your aim is to test these floppies on your system before a disaster strikes. The last thing you want to be doing in an emergency is scurrying around creating floppies only to find that an essential driver is missing.

Here, I’ll connect to the main FreeBSD FTP server and download the files for an i386, 5.1-RELEASE system:

# ftp ftp.freebsd.org
Trying 62.243.72.50...
Connected to ftp.freebsd.org.
<snip banner>
220 
Name (ftp.freebsd.org:dlavigne6): anonymous
331 Guest login ok, send your complete e-mail address as password.
Password:
ftp> cd pub/FreeBSD/releases/i386/5.1-RELEASE/floppies 
250 CWD command successful.
ftp> binary
200 Type set to I.
ftp> mget *.flp 
mget boot.flp [anpqy?]? a
Prompting off for duration of mget.
<snip transfer of five files>
ftp> bye 
221 Goodbye.

I find it convenient to create a floppies directory with subdirectories for each version of FreeBSD I have running in my network. I then download the appropriate *.flp files to the appropriate subdirectory so they are available when I wish to create an emergency repair kit for a new system.

Once you have all five files, you can decide which ones you’ll need for your particular system. To perform an emergency repair, you’ll need some way to load your version of the operating system into memory so you can access the utilities on the fixit floppy and restore whatever damage has happened to your own operating system. There are several ways to load an operating system.

The first approach is to boot directly from the install CD-ROM, assuming it is bootable and your BIOS supports this. If this is your scenario, you don’t need boot.flp, kern.flp, or mfsroot.flp.

If booting from the CD-ROM isn’t an option, you can use either boot.flp or both kern.flp and mfsroot.flp. boot.flp is basically the contents of the other two floppies placed onto one floppy. The kicker is that you need a floppy capable of holding 2.88 MB of data.

Depending upon your hardware, you may or may not need drivers.flp. If the installer detected all of your hardware, you won’t need this floppy. Otherwise, you will. Finally, if you don’t have a CD containing the live filesystem, you’ll need fixit.flp, as this floppy contains the actual repair utilities.

Use dd to transfer these files to floppies. Repeat this for each *.flp file you require, using a different floppy for each file:

# dd if=fixit.flp of=/dev/fd0

Label each floppy with its name and version of FreeBSD and write protect the floppies.

Before testing your floppies, print some important system information—you won’t remember all of these details in an emergency. First, you’ll want a copy of your filesystem layout:

# more /etc/fstab
# Device       Mountpoint          FStype     Options      Dump  Pass#
/dev/ad0s1b    none                swap       sw           0     0
/dev/ad0s1a    /                   ufs        rw           1     1
/dev/ad0s1e    /tmp                ufs        rw           2     2
/dev/ad0s1f    /usr                ufs        rw           2     2
/dev/ad0s1d    /var                ufs        rw           2     2
/dev/acd0      /cdrom              cd9660     ro,noauto    0     0
proc           /proc               procfs     rw           0     0
linproc        /compat/linux/proc  linprocfs  rw           0     0
/dev/fd0       /floppy             msdos      rw,noauto    0     0

Here, I’ve just sent the output to a pager for viewing. Depending upon how printing is set up on your system, redirect that output either directly to lpr or to a file that you can send to a printer.

Notice that all of my hard drive partitions start with /dev/ad0s1. The name of your hard drive is needed in order to view the partition table, or what FreeBSD calls the disklabel:

# bsdlabel ad0s1
# /dev/ad0s1:
8 partitions:
#        size   offset  fstype  [fsize bsize bps/cpg]
  a:   524288        0  4.2BSD    2048 16384 32776 
  b:  1279376   524288    swap                   
  c: 30008097        0  unused       0     0 # "raw" part, don't edit
  d:   524288  1803664  4.2BSD    2048 16384 32776 
  e:   524288  2327952  4.2BSD    2048 16384 32776 
  f: 27155857  2852240  4.2BSD    2048 16384 28512

Once you have a printout of your disklabel, complete your kit by printing the contents of /var/run/dmesg.boot. This file contains your startup messages, including the results of the kernel probing your hardware.

Now you’re ready to test that your kit works before sealing the envelope and sending it off for secure storage. First, boot the system using either your CD-ROM or the emergency floppies. Once the kernel has loaded and probed your hardware, the screen will ask: Would you like to load kernel modules from the driver floppy? If you choose yes, you will be asked to insert the drivers.flp floppy and will be presented with a list of modules to choose from:

cd9660.ko   
if_awi.ko
if_fwe.ko
if_sk.ko
if_sl.ko
if_sn.ko
<snip>

Taking a look at those modules, aren’t you glad you’re testing your kit before an emergency? While the modules don’t have the most descriptive names, it’s easy to find out what each module represents if you have access to a working system. For example, the modules that begin with if are interfaces. To see what type of interface if_awi.ko is:

% whatis awi
awi(4)      - AMD PCnetMobile IEEE 802.11 PCMCIA wireless network driver

You can whatis each name; just don’t include the beginning if or the trailing .ko. If you do need any of these drivers, save yourself some grief and write yourself a note explaining which drivers to choose off of the drivers.flp. The lucky bloke who has to repair the system will thank you for this bit of homework.

Once you exit from this menu, you’ll be prompted to remove the floppy. You’ll then be presented with the sysinstall Main Menu screen. Choose Fixit from the menu and insert fixit.flp. You should be prompted to press Alt F4, and you should then see a Good Luck! screen with a Fixit# prompt. Excellent, your floppy is good and your repair kit is complete. Type exit to return to the menu and exit your way out of the install utility.

If this had been an actual emergency, you’d definitely want to read the next hack [Hack #72] .

Let’s see what you can repair with the fixit floppy. When you choose that option, follow the prompts: insert the floppy, then press Alt F4. Do make note of the message you receive:

+-----------------------------------------------------------------------+
| You are now running from FreeBSD "fixit" media.                       |
| --------------------------------------------------------------------- |
| When you're finished with this shell, please type exit.               |
| The fixit media is mounted as /mnt2.                                  |
|                                                                       |
| You might want to symlink /mnt/etc/*pwd.db and /mnt/etc/group         |
| to /etc/ after mounting a root filesystem from your disk.             |
| tar(1) will not restore all permissions correctly otherwise!          |
|                                                                       |
| Note: you might use the arrow keys to browse through the              |
| command history of this shell.                                        |
+-----------------------------------------------------------------------+

Good Luck!

Fixit#

It’s not a bad idea to create those symlinks now, before you forget. You’ll have to mount your root slice first, so refer to your /etc/fstab printout for the proper name of that slice. In this example, / is on /dev/ad0s1a. I’ll mount it with the read-write option:

Fixit# mount -o rw /dev/ad0s1a /mnt
Fixit#

If your command is successful, you’ll receive the prompt back. A quick ls through /mnt should convince you that you now have access to the hard disk’s root filesystem.

If your command is not successful, run fsck_ffs until the filesystem is clean, then mount the filesystem:

Fixit# fsck_ffs /dev/ad0s1
** /dev/ad0s1
** Last Mounted on /mnt
** Phase 1 - Check blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
821 files, 27150 used, 99689 free (985 frags, 12338 blocks, 0.8% fragmentation)
Fixit# mount -u -o rw /dev/ad0s1 /mnt

Now for those symlinks:

Fixit# ln -f -s /mnt/etc/*pwd.db /etc
Fixit# ln -f -s /mnt/etc/group /etc

Note that you need to include the force (-f) switch when you make your symbolic (-s) links. You need to overwrite the existing link that links mnt2, or the fixit floppy, to /etc. You instead want to link the files on your hard drive (/mnt) to /etc.

You’ll also notice that while in the Fixit# prompt, the up arrow will recall history, but tab completion does not work.

At that Fixit# prompt, you have two command sets available to you. The first is that limited command set that comes with the sysinstall utility. Note that these are the only commands available at that holographic shell prompt:

Fixit# ls stand
-sh*               gunzip*       route*
[*                 gzip*         rtsol*
arp*               help/         sed*
boot_crunch*       hostname*     sh*
camcontrol*        ifconfig*     slattach*
cpio*              minigzip*     sysinstall*
dhclient*          mount_nfs*    test*
dhclient-script*   newfs*        tunefs*
etc/               ppp*          usbd*
find*              pwd*          usbdevs*
fsck_ffs*          rm*           zcat*

The second command set is on the floppy itself, mounted as mnt2:

Fixit# ls mnt2/stand
bsdlabel*    dd*         fixit_crunch*    mount_cd9660*    sleep*
cat*         df*         ftp*             mount_msdosfs*   swapon*
chgrp*       disklabel*  kill*            mv*              sync*
chmod*       dmesg*      ln*              reboot*          tar*
chown*       echo*       ls*              restore*         telnet*
chroot*      ex*         mkdir*           rm*              umount*
clri*        expr*       mknod*           rmdir*           vi*
cp*          fdisk*      mount*           rrestore*        view*

You’ll also find a minimal set of notes in:

Fixit# ls stand/help

One of the first things you’ll notice, especially if you try to read one of those help documents, is the lack of a pager. You won’t have any luck with more or less. However, cat and view are available for viewing files. If you’ve never used view before, remember to type :q to quit the viewer.

Also note that all of the restore utilities are on hand, unless you’ve used pax as your backup utility.

Let’s pause here for a moment and compare the fixit floppy to the live filesystem. There’s one CD marked as live in a purchased set. If you burn your own ISO images, the second image for your release will contain the live filesystem. For example, here is the listing for ftp://ftp.freebsd.org/pub/FreeBSD/releases/i386/ISO-IMAGES/5.1/:

5.1-RELEASE-i386-disc1.iso      630048 KB    06/05/03    00:00:00
5.1-RELEASE-i386-disc2.iso      292448 KB    06/05/03    00:00:00
5.1-RELEASE-i386-miniinst.iso   243488 KB    06/05/03    00:00:00
CHECKSUM.MD5                         1 KB    06/05/03    00:00:00

disc1.iso is the install CD, and disc2.iso is the live filesystem CD.

There are several advantages to using the live filesystem. First, you don’t have to make any floppies. In fact, your entire kit can be as simple as this one CD and your printouts specific to that system. Second, the CD is bootable, so you can reach that Fixit# prompt in under a minute.

Third, you have the entire built-in command set available to you. When you enter the Fixit screen, you’ll see the same welcome message as before. This time, it is the CD that is mounted as /mnt2, which is really a link to /dist:

Fixit# ls -l /mnt2
lrwxr-xr-x  1 root  wheel  5 Dec  8 08:22 /mnt2@ -> /dist

Fixit# ls /dist
.cshrc        boot/          etc/        root/        tmp/
.profile      boot.catalog   floppies/   rr_moved/    usr/
COPYRIGHT     cdrom.inf      mnt/        sbin/        var/
bin/          dev/           proc/       sys@

A quick ls /dist/bin and ls /dist/sbin will display all of the commands that come with a FreeBSD system. There isn’t a limited command set with the live filesystem.

Now that I’ve shown you the various ways to enter the Fixit facility, you’re probably wondering what you should be doing at that prompt. FreeBSD is quite robust and is usually capable of booting your hard drive to some sort of prompt. However, if the disk fails completely or is somehow incapable of booting to a prompt, the fixit facility is one of your options.

From here, you can run fsck on your various filesystems, which may fix the problem. You can see which filesystems are still mountable, allowing you to assess the extent of the damage. If some files were damaged, you can restore those files from backup.

If it turns out that the drive is damaged beyond repair, you can rest easy in the fact that you have a printout of your hardware and partitioning scheme, a floppy containing any necessary drivers, and a backup of all of your data. Above all, you were prepared.

Analyzing a malfunctioning process starts with basic information, such as error messages and return values. Sometimes those aren’t enough, though. Some error messages are unclear. In the case of security vulnerabilities, there may not be an error code or return value, because the program may crash or misbehave silently.

The BSDs provide several tools to analyze a program’s execution. You can monitor system calls with ktrace and resources with fstat. You can run a debugger such as GDB, the GNU Debugger, and watch your operating system’s internal operation.

In some cases, a program must run in a particular environment, which may make it difficult to analyze due to the limitations of some tools. For example, a telnetd advisory from 2001 (http://www.cert.org/advisories/CA-2001-21.html) affected most Unix operating systems. This particular vulnerability came to light when a group called TESO released an example exploit for it.

On Unix systems, telnetd runs as root, so that once the system authenticates the user, the process has the privileges required to set the user ID of the login shell to that of the user who logged in. This means that a remote entity who can cause telnetd to misbehave by sending it carefully designed input could execute processes as root on your system.

On most Unix systems, telnetd does not run as a standalone daemon. Since logins are relatively infrequent (on the system timescale compared to thousands of interrupts per second), the inetd service starts telnetd as needed.

This is a simple example of the data stream sufficient to crash vulnerable telnetds using perl and nc (netcat):

% perl -e 'print "377366"x512' |  nc testhost telnet

This was the example I used to diagnose the problem and test the fix. If you run this command against an impervious Telnet daemon, you’ll see the following output:

% perl -e 'print "377366"x512' | nc testhost telnet

[Yes]

[Yes]

[Yes]

The [Yes] message will repeat 512 times because the characters you sent, 377366, represent the Telnet protocol’s “ARE YOU THERE” control message, and you asked the question 512 times.

If you run this command against a vulnerable telnetd, the output can vary. In some cases, your connection may close before you get 512 [Yes] responses because telnetd crashed. In other cases, you may receive seemingly random output from portions of the telnetd memory space. These both indicate that the program did something it was not supposed to, due to the specific input you gave it.

In order to fix the problem, we need to find out where the executable did something incorrectly. We would like to run the program under the control of GDB, but we cannot start telnetd from the command line the way we usually would when debugging most executables. Normally, GDB is invoked in one of three ways.

First, to run a program and debug it, type:

% gdb 
               programname

GNU gdb 5.3nb1
Copyright 2002 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you 
are welcome to change it and/or distribute copies of it under certain 
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386--netbsdelf"...(no debugging symbols found)...
(gdb) run

Second, to examine the core file of a program that has already crashed, use:

% gdb 
               programname
                
               programname
               .core

Third, to examine a program that is already running, type:

% gdb 
               programname
                
               processid

In the case of telnetd, we cannot use the first method, because inetd must start telnetd in order to attach it to a network socket and operate properly. We cannot use the second method, because processes that run with root privileges do not leave core files, since the program’s memory image could contain sensitive data.

That leaves the third method. Attaching to a running process is problematic because telnetd isn’t running until someone connects. We’ll need to modify our attack script:

% perl -e 'sleep 30; print "377366"x512' |  nc testhost telnet

Now nc opens a socket to the testhost, inetd spawns a telnetd in response, and perl waits for 30 seconds before sending the attack string.

In another terminal, on the testhost, we say:

% ps -ax | grep telnetd
27857 ??  S      0:00.05 telnetd
27859 pd  S+     0:00.02 grep telnetd

% gdb /usr/libexec/telnetd 27857
GNU gdb[...]
Attaching to program `/usr/libexec/telnetd', process 27857

From here we can allow telnetd to crash and observe the exact type of error that caused the crash. If we’ve built telnetd with debugging information, GDB will even display the line of source code the program was executing when it crashed. Now we can use our favorite debugging techniques and either insert debugging messages or use GDB and set breakpoints and watchpoints to discover at what point the program went off course. We can then determine what changes to make to correct the error and prevent the exploit.

If you’re not a programmer, you can save the information and send it to the developers.

We were fortunate in this example because we had details of the exploit. That made it easy to experiment and try different approaches. In many cases, however, you won’t know the details of an exploit, and you may only know that there is a problem because of error messages in your logs.

You can use tcpdump to capture the traffic on the relevant port. Once you can correlate the timestamp of the log’s error message with some of your tcpdump traffic, you can take the data sent in an attack and create a Perl script to resend it. You can then apply the techniques already described to analyze and correct the problem.

Download logproc from http://www.peterson.ath.cx/~jlp/software/logproc.tar.gz. Then, extract it:

% gunzip logproc.tar.gz
% tar xvf logproc.tar
% ls -F logproc
./    ../    logserver.bin/    webserver.bin/

% ls -F logserver.bin
./    apache_rrd*    cleantmp*    logwatch*    mining/
../   arclogs*       collect*     meter*

% ls -F webserver.bin
./    ../    batcher*    cleantmp*    copier*

As you can see, there are two parts. One runs on each web server and the other runs on the log server.

The logs are fed to a process called batcher that runs on the web server and writes the log lines to a batch file as they are received. The batch file stays small, containing only five minutes’ worth of logs. Each completed batch file moves off to a holding area. A second script on each web server, the copier , takes the completed batch files and copies them to the centralized log host. It typically runs from cron. On the log host, the collect process, also run from cron, collects the batches and sorts the log lines into the appropriate daily log files.

The system can also monitor log information in real time. Each batcher process dumps the log lines as it receives them out to a multicast group. Listener processes can retrieve those log lines and provide real-time analysis or monitoring. See the sample logwatch script included with logproc for details.

First, create a home directory for the web server user. In this case, we’ll call the user www. Make sure that www’s home directory in /etc/master.passwd points to that same location, not to /nonexistent. If necessary, use vipw to modify the location in the password file.

# mkdir ~www
# chown www:www ~www

Next, log in as the web server user and create a public/private SSH keypair:

# su www
% ssh-keygen -t dsa

Create the directories used by the log processing tools, and copy the scripts over:

% cd ~www
% mkdir -p bin logs/{work,save}/0 logs/tmp logs/work/1

% cp $srcdir/logproc/webserver.bin/* bin/

Examine those scripts, and edit the variables listed in Table 7-1 to reflect your situation.

Then, make sure you have satisfied all of the dependencies for these programs:

# perl -wc batcher; perl -wc cleantmp; perl -wc copier

The only dependency you likely won’t have is IO::Socket::Multicast. Install it via the /usr/ports/net/p5-IO-Socket-Multicast port on FreeBSD systems or from the CPAN site (http://www.cpan.org/).

Next, configure httpd.conf to log to the batcher in parallel with normal logging. Note that the batcher command line must include the instance (site, virtual, secure) and type (access, error, ssl) of logging:

LogFormat "%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i" 
    "%{Cookie}i" %v" full
CustomLog "|/home/www/bin/batcher site access" full
ErrorLog  "|/home/www/bin/batcher site error"

You can adjust the LogFormat directive as necessary to log the information you or your log summarization software needs.

Finally, restart Apache and verify that the batchers are creating batches:

# apachectl configtest 
# apachectl graceful
# cd $wwwhome/logs/
# ls tmp       
               Should list error log files for each batcher instance
# ls work/0    
               Should list the working batches for each batcher instance
# ls save/0    
               Verify that batches have moved into the save directory after a 
                                                 five-minute batch interval
# ls work/0    
               and that new batches are currently being created

Start by creating a log user to receive the logs, complete with a home directory. Become the log user and copy the public key from the web server into ~log/.ssh/authorized_keys2. Then, as the log user, create the directories the log collection tools use:

# su log
% cd ~log
% mkdir -p bin web/{work,save}/{0,1} web/tmp web/{current,archive}

From a web server (as the web server’s user), ssh to the log host manually to verify the configuration of the authorized_keys2:

# su www
% ssh loghost -l loguser date

Then, run copier manually to verify that the log files actually make it to the log server. Watch your run output on the web server, then check that save/0 on the log server contains the newly copied logs.

Once you’re satisfied with these manual tests, schedule a cron job that copies and cleans up log files. These jobs should run as the web server user:

# crontab -e -u www

----------------------------- cut here -----------------------------
# copy the log files down to the collector host every 15 minutes
0,15,30,45 * * * * /home/www/bin/copier

# clean the tmp directory once an hour
0 * * * * /home/www/bin/cleantmp
----------------------------- cut here -----------------------------

Finally, wait until the next copier run and verify that the batches appear on the log host.

You should now have several batches sitting in save/0 in the log tree. Each batch contains the log lines collected over the batch interval (by default, five minutes) and has a filename indicating the instance (site, virtual, secure), type (access, error, ssl), web server host, timestamp indicating when the batch was originally created, and PID of the batcher process that created each batch.

Now, install the log processing scripts into bin/:

# cp $srcdir/collector/{arclogs,cleantmp,collect} bin/

Edit them to have valid paths for their new location and any OS dependencies, as shown in Table 7-2.

Again, make sure all dependencies are satisfied:

# perl -wc arclogs; perl -wc cleantmp; perl -wc collect

If you don’t have Time::ParseDate, then install it from the /usr/ports/devel/p5-Time-modules port on FreeBSD or from CPAN.

Run collect manually as the log user to verify that the log batches get collected and that log data ends up in the appropriately dated log file. Once you’re satisfied, automate these tasks in a cron job for the log user:

# crontab -e -u log

----------------------------- cut here -----------------------------
# run the collector once an hour
0 * * * * /home/log/bin/collect

# clean the tmp directory once an hour
0 * * * * /home/log/bin/cleantmp
----------------------------- cut here -----------------------------

Wait until the next collect run and verify that the batches are properly collected.

Compare the collected log files with the contents of your old logging mechanism’s log file on the web servers. Make sure every hit makes it into the collected log files for the day. You might want to run both logging mechanisms for several days to get a good feel that the system is working as expected.

The log server programs provide additional tools for monitoring and summarizing live log data. On a traditional single web server environment, you can always tail the log file to see what’s going on. This is no longer easy to do, because the logs are now written in small batches. (Of course, if you have multiple web servers, multiple tail processes would have to run on each web server.)

The batcher process helps with this by multicasting the logs out to a multicast group. Use the logwatch tool on the log server to view the live log data:

% cd ~log/bin
% ./logwatch
<lines of log data spew out here>

On a high-volume web site, there is likely to be too much data to scan manually. logwatch accepts arguments to specify which type of log data you want to see. You can also specify a Perl regular expression to limit the output.

The meter script watches the log data on the multicast stream, in real time, and summarizes some information about the log data. It also stores information in an RRDTool (http://www.rrdtool.org/) database.

The mining directory contains a checklog script that produces a “top ten clients” and “top ten vhosts” report. Alternatively, you can feed the collected log files to your existing web server log processing tools.

During the process of generating a GPG key, the program asks the user several questions. We may wish to impose constraints so that a set of users ends up with keys with similar parameters. We could train the users, but that would not guarantee correct results. Scripting the generation makes the process easier and eliminates errors.

First, let’s look at a typical key generation session:

% gpg --gen-key
gpg (GnuPG) 1.2.4; Copyright (C) 2003 Free Software Foundation, Inc.
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions. See the file COPYING for details.

Please select what kind of key you want:
   (1) DSA and ElGamal (default)
   (2) DSA (sign only)
   (4) RSA (sign only)
Your selection? 4
What keysize do you want? (1024) 2048
Requested keysize is 2048 bits
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 0
Key does not expire at all
Is this correct (y/n)? y

You need a User-ID to identify your key; the software constructs the user id
from Real Name, Comment and Email Address in this form:
    "Heinrich Heine (Der Dichter) <[email protected]>"

Real name:

Let’s pause there to consider the elements we can constrain.

You probably want to specify the cryptographic algorithm and key length for all users consistently, based on your security and interoperability requirements. I’ll choose RSA signing and encryption keys, but GPG doesn’t provide a menu option for that. I’ll have to create the signing key first and then add the encryption subkey.

Here’s an expect script that would duplicate the session shown so far:

#!/usr/pkg/bin/expect -f

set timeout -1
spawn gpg --gen-key
match_max 100000
expect "(4) RSA (sign only)"
expect "Your selection? "
send "4
"
expect "What keysize do you want? (1024) "
send "2048
"
expect "Key is valid for? (0) "
send -- "0
"
expect "Key does not expire at all"
expect "Is this correct (y/n)? "
send -- "y
"
expect "Real name: "

The script begins by setting timeout to infinite, or -1, so expect will wait forever to match the provided input. Then we spawn the process that we’re going to control, gpg --gen-key. match_max sets some buffer size constraints in bytes, and the given value is far more than we will need.

After the initial settings, the script simply consists of strings that we expect from the program and strings that we send in reply. This means that the script will answer all of the questions GPG asks until Real name: , without waiting for the user’s input.

Note that in several places we expect things besides the prompt. For example, before responding to the Your selection? prompt, we verify that the version of GPG we have executed still has the same meaning for the fourth option, by expecting that the text of that menu choice is still RSA (sign only). If this were a real, production-ready script, we should print a warning message and terminate the script if the value does not match our expectations, and perhaps include a check of the GPG version number. In this simple example, the script will hang, and you must break out of it with Ctrl-c.

There are several ways of handling the fields we do want the user to provide. For the greatest degree of control over the user experience, we could use individual expect commands, but here we will take a simpler approach. Here’s some more of the script:

interact "
" return
send "
"
expect "Email address: "
interact "
" return
send "
"
expect "Comment: "
interact "
" return
send "
"
expect "Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? "
interact "
" return
send "
" 
expect "Enter passphrase: "
interact "
" return
send "
"
expect "Repeat passphrase: "
interact "
" return
send "
"

The interact command allows the user to interact directly with the spawned program. We place a constraint that the user’s interaction ends as soon as the user presses the Enter key, which sends the carriage return character, . At that point, the interact command returns and the script resumes. Note that we have to send the from the script; expect intercepted the carriage return and GPG did not see it.

Again, a correct script would have a more complex flow of execution and allow for cases where the spawned program rejects the user’s input with an error message. For example, the Real Name field must be more than five characters long. If a user types less than five characters, GPG will prompt him to retype his username. However, the expect script just shown will not accept the new user input, because it is now waiting for the Email address: prompt.

Alternatively, we could replace these three lines:

interact "
" return
send "
"
expect "Email address: "

with:

interact -o "Email address: " return
send_user "Email address: "

Instead of stopping interaction when the user presses return, we stop interaction when the program outputs the Email address: prompt. That’s the difference between interact and interact -o; the former stops interaction based on input from the user, and the latter on output from the program. This time, we don’t need to send the carriage return, because the user’s keypress is passed through to GPG. However, we do need to echo the prompt, because expect has consumed it. This method lets GPG handle the error conditions for us:

Real name: abc
Name must be at least 5 characters long
Real name: abcde
Email address:

After GPG receives the information it needs to generate the key, it might not be able to find enough high-quality random data from the system. The script ought to handle that by spawning a process to generate more system activity, such as performing a lot of disk activity by running a find across the entire disk.

After generating the signing key, the script could spawn a new instance of GPG with the --edit-key option, to generate the desired RSA encryption key.

Although the final script may end up executing three processes, the whole process is seamless to the user. You can hide even more of the guts by using expect’s log_user setting to hide the output of the programs at points where the user does not need to see them.

You can use a script like this in conjunction with any Unix command-line program. By combining expect with telnet or ssh, you can control non-Unix systems, thereby leveraging the flexibility of Unix into a non-Unix domain. This even works with programs for which you do not have source code, such as control utilities for commercial databases or application software.

In the case of GPG, we do have source code, so we could modify the program, but writing an expect script is easier. A carefully designed expect script may not require changes when a new version of GPG is released. Source code changes to GPG would require integration with any new version of GPG.

Normally in Enlightenment, to switch from one virtual desktop to the next, you move the mouse pointer to the edge of the screen and then push past it, or you use a key sequence to move to an adjacent desktop. For an unattended demo, we need to automate this process. Enlightenment provides an undocumented utility called eesh that can control most aspects of the Enlightenment window manager. You can write scripts to move windows, resize them, or flip between desktops.

Note that eesh isn’t a friendly utility; it doesn’t even produce a prompt when you run it. Type help for the menu or exit to quit:

% eesh
               help
Enlightenment IPC Commands Help
commands currently available:
use "help all" for descriptions of each command
use "help <command>" for an individual description

actionclass             active_network          advanced_focus   sfa  
autosave                background              border                
button                  button_show             colormod              
configpanel             copyright               current_theme    tc   
cursor                  default_theme           dialog_ok        dok  
dock                    dump_mem_debug          exit             q    
focus_mode       sf     fx                      general_info          
geominfo_mode    sgm    goto_area        sa     goto_desktop     sd   
group            gc     group_info       gl     group_op         gop  
help             ?      imageclass              internal_list    il   
list_class       cl     list_remember           list_themes      tl   
module                  move_mode        smm    nop

Unfortunately, the eesh utility seems to be untested. It sometimes behaves inconsistently by not accepting commands until you enter them a second time or by withholding output until you press Enter again. As an example, there are actually more commands than those indicated in the help listing. Look in the Enlightenment source’s ipc.c file for a complete list.

We’ll start our script by making sure that Enlightenment is configured the way we want for our demo. We want six work spaces (3 by 2) to display our programs. Within eesh, try the following commands:

               num_areas ?
Number of Areas: 2 2
help num_areas
Enlightenment IPC Commands Help : num_areas (sna)
--------------------------------
Change the size of the virtual desktop
Use "num_areas <width> <height>" to change the size of the virtual desktop.
Example: "num_areas 2 2" makes 2x2 virtual destkops
Use "num_areas ?" to retrieve the current setting

num_areas 3 2

Now we have the number of areas we want. areas is the Enlightenment name for virtual desktops, since Enlightenment also supports multiple desktops, but that’s different. Now we’d like our screen to display the first area, so that the programs our script runs will open there:

               goto_area 0 0

If your terminal wasn’t on the first area, it just moved off the screen. Use the mouse to return to that area.

eesh also lets us write commands on the command line with the -e (execute command) flag:

% eesh -e "goto_area 0 0"

Now we know enough to write a simple demo script:

#!/bin/sh

eesh -e "num_desks 1"
eesh -e "num_areas 3 2"
sleep 1
eesh -e "goto_area 0 0"

# Configure the default gqmpeg playlist to play your desired music
gqmpeg

# Show an interesting avi file.
xanim -geometry +50x+10 netbsd3.avi &

# Give the programs time to start, to make sure they 
# open on the correct area.
# Also, lets people watching see what started up.
sleep 3
eesh -e "goto_area 1 0"

# Word Processing
abiword sampledoc.abw &
sleep 2
eesh -e "goto_area 2 0"

# Spreadsheet
gnumeric samplesheet.gnumeric &
sleep 2
eesh -e "goto_area 0 1"

# A lively game
battleball &
sleep 2
eesh -e "goto_area 1 1"

# Web Browsing (of a local hierarchy, in case you don't have net 
# connectivity at a trade show)
firebird file://index.html &
sleep 3
eesh -e "goto_area 2 1"
sleep 1

# Insert your favorite application here
# Leave screen back at page 1.
eesh -e "goto_area 0 0"

When you run the script, the screen will slide around to the various areas and pause a few seconds between program launches. We have most of the things we wanted: music, video, and applications. The next step is to keep it moving. Try the following script:

#!/bin/sh
while [ 1 ]
do
        eesh -e "goto_area 0 0"
        sleep 2
        eesh -e "goto_area 1 0"
        sleep 2
        eesh -e "goto_area 2 0"
        sleep 2
        eesh -e "goto_area 0 1"
        sleep 2
        eesh -e "goto_area 1 1"
        sleep 2
        eesh -e "goto_area 2 1"
        sleep 2
done

To stop the moving display, you have to get your keyboard focus into the xterm where the script is running so that you can press Ctrl-c. That can be difficult, but we’ll address it shortly.

For a complex demonstration, you can have different sets of these scripts that visit different sets of areas. You can also change the delay so that complex areas display for a longer period. I also made a script that clears all of the viewing areas. That way, when visitors to the booth play around with the machine, I can easily reset to a clean state and then start the demo again.

Since many of the utilities you’ll demonstrate don’t create .pid files, I find it easiest to use pkill, the “kill process by name” utility. (FreeBSD provides killall.)

I’ll also leave you with two example scripts that show how to extract information about Enlightenment’s current settings for use in a more complex script.

The first script is retitle:

#!/bin/sh

WIN=`eesh -ewait "set_focus ?" | sed 's/^focused: //' `
xterm -geometry 47x7+227+419 -fn -*-courier-*-o-*-*-34-*-*-*-*-*-*-* -e 
/home/david/bin/retitle2 $WIN

The second is retitle2:

#!/bin/sh
WIN=$1
echo "enter new title:"
read TITLE
eesh -e "win_op $WIN title $TITLE"

With these scripts and e16keyedit , you can bind a key combination to change the title of any window. This makes it much easier to keep track of xterms, if you prefer task-oriented titles.

Now back to the control issue. When I first wrote this demo, I used a switch wired to a serial port to start and stop the demo so that keyboard focus did not matter. However, wiring switches is more work than configuring software, so I found a better way.

The e16keyedit utility, written by Geoff “Mandrake” Harrison and Carsten “Raster” Haitzler (the primary developers of Enlightenment), allows you to bind function keys and Meta keys to run programs or perform the same functions that you can with eesh. Using e16keyedit, you can define function keys to set up the demo, clean up the demo, and start and stop the area rotations. Since the function keys can be bound to work anywhere within Enlightenment, keyboard focus no longer matters. You’re ready to give a fantastic demo!

e16keyedit is not part of the main Enlightenment distribution. Download it from SourceForge (http://sourceforge.net/project/showfiles.php?group_id=2).