This chapter's light relief is a true story about networks and system administration.

Trey Harris is a senior systems engineer, and a vice president of SAGE, the System Administrators Guild (www.sage.org). Trust me, if you have a mission-critical server, you want a guy like Trey to keep it running.

A few years ago, Trey was responsible for email at a university in the Research Triangle area of North Carolina. One day, a peculiar problem report came in from the Dean of the Statistics Department. The dean reported that there was a 500-mile limit on their email. Email to places closer than about 500 miles was usually delivered just fine. But as the Dean said, “500 miles, or a little bit more, is our current email limit.”

College deans are creatures with god-like powers on campus. Even when they report something blatantly ridiculous they have to be treated with care. Trey cautiously pointed out that email didn't really work that way. The dean replied that, no, he had all the data that proved it did. When they had first noticed the problem a few days ago, the dean assigned one of the geostatisticians to gather data. After all, this was a Statistics department and that's what they do.

The geostatistician had experimented with a great many email addresses, and had drawn up a map correlating geography and email results. The map showed that the Statistics department could send email to most sites closer than 500 miles, but there was a hard cutoff much beyond that. There were some places within that radius that they couldn't reach or could only reach sporadically, but they could never send email farther than that distance.

Trey knew that whatever this problem was, it was probably caused by someone changing the system configuration. Sure enough, the dean acknowledged that the problem started after a consultant patched the Statistics department server and rebooted it. However, the consultant was certain that he hadn't touched the mail system.

Trey logged into the department's mails server and sent a few test mails. Email to his own local test account went fine. The same thing for email to Raleigh, Atlanta, and Washington. Then he emailed a user in Memphis (600 miles away). It failed. Boston failed. So did Detroit. Trey was seeing exactly the ridiculous problem that the dean reported! At this point, Trey pulled out his address book and an atlas and tried to narrow it down. New York (420 miles) worked, but Providence (580 miles) failed.

One of the first things to check on a mail delivery problem when the system has been patched and rebooted, is the sendmail.cf configuration file. It was fine. Thinking about what to try next, Trey telnetted to the SMTP port to issue a few mail commands by hand. Ah! The first clue! The system responded with the old “Sendmail 5” response that was standard with Solaris at that time, even though Trey had installed the more up-to-date Sendmail 8 throughout the campus.

The pieces quickly came together after that. The consultant had upgraded the version of Solaris, which had wiped out the mail upgrade. The version of Sendmail 5 that Sun shipped as standard could deal with most of the Sendmail 8 sendmail.cf. But the new configuration options that Trey had written with more meaningful names were all ignored and therefore defaulted to zero. One of the settings that became zero was the timeout value to allow while connecting to a remote SMTP server. Some experimentation established that on the Stats department mail server, a zero timeout would abort a connect call in slightly over three milliseconds.

Now, an odd feature of the campus network at the time was that it was 100% switched. An outgoing packet wouldn't incur any processing delay until it reached a router on the far side. So the time to connect to a lightly loaded remote host on a nearby network would actually largely be governed by the speed of light distance to the destination rather than by incidental router delays.

As Trey says, “Feeling slightly giddy, I fired up the units command.” Units is a Unix command that converts between different measurement units. You tell it what units you have (like centimeters) and what units you want (like fathoms), and it tells you the conversion factor. Trey typed in the critical numbers to find out how many miles the speed-of-a-signal-in-a-wire (the physics constant "c") goes in 3 milliseconds:

There you go! “500 miles, or a little bit more.”