Before Vineyard.NET, I had always handled the security of my own computer system with a few simple and reliable policies: don’t run any services that might be unsecure; don’t give out accounts to people you don’t know; and disconnect your computer from the Internet when you are not around. The monitoring that I did, insofar as I did any, was haphazard.
Clearly, those techniques would not work for a commercial service such as Vineyard.NET. Instead, I needed to design and deploy a system that could be readily maintained, defended, and scaled.
From the start, Vineyard.NET’s security was one of my primary concerns. As the coauthor of several books on computer security, I knew any Internet service that I was involved with might be a target.
But there were other reasons that I was concerned about security. Vineyard.NET is a company that depended on computers that were connected to the Internet. If these computers were broken into and compromised, our reputation would be damaged, we would lose customers, and we would lose time required to put our systems back in order. We might lose so much in the way of customers, reputation, and time, that we might even go out of business.
Because of these problems, we followed a few simple rules for our system and networks: minimize our vulnerabilities and plan for break-ins.
From the beginning, we avoided running programs that had a history of security problems. The primary offender here is sendmail, the UNIX SMTP server and mail delivery agent. Sendmail is an especially dangerous program because it runs as the superuser, it has full access to the UNIX filesystem, it implements a complicated command language, and it talks to any other computer on the Internet.
Organizations like the CERT are continually sending out notices of new security problems discovered with sendmail. People are advised to download upgrades immediately from their vendors, if possible. Vineyard.NET runs the BSDI operating system, BSD/OS, and BSDI is very good about responding to security problems in a timely manner. But a much better way to deal with the problem is to use smap, a sendmail proxy, so that programs on the outside world never connect directly to our sendmail program. (Full instructions on how to install smap are in the book Practical UNIX & Internet Security.)
Another thing we did to protect us was to make frequent backups of our system. These backups protected us against both break-ins and accidents on the part of myself or Eric.
At Vineyard.NET we make two kinds of backups. The first are tape backups. We started out using the GNU tar facility, but switched over to UNIX dump because it ran at nearly twice the speed.
Each day we make a full dump of every file on the system. To accommodate all of the files, we got a DDS-II DAT tape drive. That holds 8 GB with compression, and, as we only have 6 GB online, things should be okay for a while. Our tapes are labeled Monday, Tuesday, Wednesday, and so on. At the end of each month, we make a “permanent” backup that is taken offsite.
We also make daily backups of the entire /usr/local/etc/httpd/htdocs directory. These backups are kept online in a single compressed tar file. These backups are designed to protect ourselves from accidentally deleting a file.
It turns out that we’ve used all of the backups with more or less successful results. The backups have been quite useful when files have been accidentally deleted. The only time that we had a problem that our backups couldn’t solve was when we were changing a disk formatting parameter on a disk drive that was being used to store Usenet files. We backed up the entire file system to the DAT drive twice, reformatted the disk, and then tried to restore it. It wouldn’t. It turns out that the Berkeley restore command has some hard-coded limits inside it, and it simply couldn’t restore 100,000+ files at once. The result: we lost all of our netnews. In retrospect, it was no big deal.
From the beginning, I decided that Vineyard.NET would not offer “shell” accounts —that is, accounts that allowed people to log into our UNIX server. We also resolved that all administration of the web server would be done through the FTP command. Although this was a little awkward for some people, and I think it cost us a few customers, in the end it was the right decision. It is far easier to run a UNIX machine securely if you do not allow people to log into it.
TCP/IP spoofing is a rather arcane attack, but it is one that has been successful in at least one widely publicized attack. To protect ourselves against it, we configured our external routers to reject any incoming packets that indicated they were being sent from our internal network.
As soon as we could, Eric and I set up ssh, the secure shell, so we could log on from one computer to another with the knowledge that our password was not being sent over the network. We also required that friends on the Internet who wanted guest accounts on our machines use ssh to access those accounts as well.
As for our home machines, we installed Kerberos on our BSDI box and started using a Kerberized-version of the Macintosh Telnet program. About a year later, we switched to a commercial version of ssh sold by Data Fellows. The program works quite well, and runs on PCs and Macs!
In the interest of security, we recently configured our Cisco routers so that they could only be logged into from a particular computer on our internal network. Although this slightly increased the difficulty of our administering the system, it dramatically improved the security of the routers. Prior to the configuration change, a person could have broken into our router by trying to guess thousands or millions of passwords. (Cisco’s operating system makes this procedure easier than that statement makes it sound.) After the change, the only way to break into the routers was first to identify and break into the one host on the subnet that could Telnet to the routers. Still, it would have been better if Cisco had simply added support for Kerberos, S/Key, and ssh into their products.
In the interest of improved security, we started reducing the access permissions of many directories on our UNIX system. Directories that contained programs were changed from mode 755 (read/write access for the superuser, read and chdir access for everybody else) to mode 711 (read/write access for the superuser, but only search chdir for others). Configuration files were changed from mode 644 (world-readable access) to mode 600 (access only for superuser). The idea of reducing permissions in this manner was to make it harder for somebody who had an account on the system to use that account to probe for further weaknesses.
Although I am an advocate of free software, there were many times in the development of Vineyard.NET that we went with commercially-available versions of free programs. For example, we chose to use BSD/OS, a commercial operating system from BSDI, rather than using FreeBSD, NetBSD, OpenBSD, or Linux. The reason: We wanted BSDI to track security problems for us and provide us with bug fixes, rather than having to do it ourselves. Likewise, we chose to purchase our web server from Community ConneXion, rather than simply using the freely available Apache web server. Yes, our primary reason for purchasing the Community ConneXion server was to obtain an SSL-enabled web server. But we were also eager to have somebody else tracking reported security problems with the web server and providing us with patches.
On the other hand, we were more than happy to use free software on occasions where it was clearly better than the commercial alternative. We replaced the Perl 5 interpreter that came with BSDI with one that we downloaded from the Internet because BSDI’s version of Perl 5 was out of date. We use GNU emacs as our text editor and GCC as our compiler, rather than purchasing expensive development tools that may not work as well.
We had continual (and continuing) problems with the configuration of our telephone lines. Prudent security practices dictate that lines used for incoming telephone calls should not be able to make outgoing calls—and they certainly should not be able to place long distance calls. Otherwise, somebody who breaks into your computer might use the ability to place long distance calls to charge tens of thousands of dollars in phone calls to your account.
It turns out that Vineyard.NET is particularly vulnerable to this sort of toll fraud, because we have been placing Centrex lines in places like schools and libraries, so these organizations could call the Vineyard.NET phone lines without having to pay message units. Every few months, we notice between $10 and $30 of phone calls on these lines to other phone numbers.
We have repeatedly asked our phone company, NYNEX, to disable 9+ dialing on both the dial-in telephone lines and the phone lines that are located on customer premises. Each time we’ve been assured by the phone company that the problem has been corrected, yet every time we check, we discover that the phones still can dial local and long-distance calls.
Frankly, this is still an unsolved problem. Right now, I’m trying to get a letter from NYNEX saying that our lines cannot place outgoing calls. Then I’m going to wait for another $100 phone bill and refuse to pay it.
Another problem that we have had with NYNEX is billing disputes. For several months we were not billed for one of our high-speed serial lines. Then we were double-billed. Payments made were not credited. We were billed for lines after they were disconnected. New phones that were installed were not billed for many months, then we had months of back invoices suddenly posted to our account. And we have never been able to decipher our Centrex bills to our satisfaction.
After having our office manager spend more than 100 hours trying to resolve these problems with NYNEX, we came to the realization that some amount of incorrect billing is probably unavoidable when dealing with NYNEX. So we are careful about what charges we dispute. Some are challenged right away. Others we let slide, because there are only so many hours in the day. (If our billing is that far off, we wonder what it is like for more substantial commercial firms?)
One of the biggest problems for any business is getting paid: after we had been operating for a few months, we had more than 200 customers and we were owed more than $10,000 in late fees. We didn’t want to make things worse by charging people finance charges. We just wanted our money.
It turned out that a lot of our customers wanted to pay us, they were just lazy. Writing a check is, after all, a pain. Receiving them was a pain too: entering each check into the computer, putting it on a deposit ticket, and taking it to the bank all took a considerable amount of time.
Fortunately, we found our savior: a technique for billing people’s credit cards and for withdrawing money directly from people’s credit cards using a system called ACH (the U.S. Automated Clearing House).
These days, it’s quite easy for a legitimate merchant to get a credit card merchant account. You go to your bank, fill out some forms, and wait. A few days later you’ll get a phone call from an investigator, who wants to stop by your place of business and make sure that you are legitimate. If that goes well, and it almost always does, you get a machine.
I had a credit card merchant account once before, when I was running a company that was selling computer software. But having to run people’s credit card numbers through a machine—and having to do it every month—is a real pain. We probably would have had to hire somebody to do the job, and that would have cost time and money.
So instead, I started looking around for software that could charge people’s credit cards automatically, and submit all of the charges by modem. My bank gave me a list of companies that sell this software. Most of it runs on PCs, some of it runs on Macs, and perhaps one or two programs run on a UNIX-based system. But nothing was quite right.
Then, by chance, I visited a trade show and found out about CheckFree’s merchant services. Based in Ohio, CheckFree has made a name for itself in the wireless bill payment business. In fact, I’ve used CheckFree for years to pay my various bills. I simply type all of the bills into Quicken, type a code, and my computer makes a phone call. CheckFree electronically removes the money from my account and transfers it into my merchant’s. If the merchant isn’t set up for ACH, CheckFree can write a check on my account. If several CheckFree customers try to pay the same merchant during the same payment period, CheckFree may transfer the money out of my account and into a CheckFree account, then send the merchant a single check for all of the CheckFree customers (with an accompanying check indicating who paid what).
It turned out that CheckFree also had merchant services. Checkfree could process credit card transactions for us. And CheckFree could do Merchant ACH: we could type in the checking account numbers of our customers and pull the money out directly. Even better, Checkfree had set up a system that allowed merchants to submit charges over the Internet encrypted using PGP.
Clearly, CheckFree’s Gateway system was exactly what Vineyard.NET had been looking for. It would allow us to take credit card numbers on our web server, store them online, and then automatically use credit cards and ACH charges to settle our customer’s accounts. Unfortunately, when I asked CheckFree to send me a program that implemented their protocols, they sent me a three-ring binder instead. There was no off-the-shelf code which they could provide me.
Over the next four months, I wrote a complete implementation of CheckFree’s Gateway 3.0 protocols. Although it was a fun experience and I learned a lot about Perl, bank card processing, and the intricacies of the nation’s ACH networks, I wished that I had been able to buy off-the-shelf the programs that I was looking for. In fact, there was a freelance computer consultant who sold such a package but he wanted $1,000 for the software. It was all written in C and I didn’t feel comfortable working with the person. So I ended up writing it all myself. Doing this had the added advantage of teaching me a lot about how credit cards are processed, which was useful for writing , of this book.
One of the scariest things was working with real live credit card numbers. If we screwed up, we might charge somebody too much money. If we got a sign reversed, we might transfer money from our account into a customer’s, rather than the other way around. If we had a break-in, somebody might steal the credit card numbers of our customers, and we could be liable. For the first six months, I went over every credit card batch for half an hour before submitting it to our bank.
So we tried to protect the live credit card numbers the best we could. Perhaps the most important thing we did was to store them encrypted on our web server. To do the encryption, we used PGP. The credit card encryption system has a special public key/private key. The web server and CGI scripts know the public key, but not the secret key. So when people enter their credit card numbers, the numbers are encrypted with the public key. The numbers are only decrypted when an actual billing run happens. We do this once or twice a month. To initiate the billing run, either Eric or myself must type the PGP passphrase that decrypts the secret key that, in turn, decrypts the credit card numbers.
As an added measure of protection, we set up numerous logs and reports that would be created every time a credit card run was processed. Not only did this help us in writing and debugging the software, it also made it easier for us to track down problems when they occurred.
Each time the software runs, Eric and I get sent numerous reports that are issued from CheckFree. We’re used to seeing this sequence of reports, and when we don’t get them, we know that there is something amiss. The reports have also notified us when the other person runs the billing system.
When customers pay by credit cards, merchants are assessed a surcharge between two and three percent. (Don’t think about passing this surcharge on to your customers: that’s a violation of your merchant agreement.) Still, despite this surcharge, it’s often cheaper for businesses to accept payment by credit cards than by check or cash. That’s because it’s difficult, and therefore expensive, to handle large amounts of cash or large numbers of checks. And when customers pay by cash or check, there’s always a chance that something will go wrong: the check will bounce, or it will get credited to the wrong account, or you’ll simply lose the money. (All of these have happened to us at Vineyard.NET.) None of these problems happen when you are charging somebody a monthly fee for a credit card that you’ve got on file. A further advantage is that you are immediately told if the charge is approved or not. If the charge is approved, it’s nearly a sure thing that you will get the money.
We prefer ACH to credit cards. Instead of being charged a fee between two and three percent, we are charged a flat fee of roughly 27 cents. But ACH is not without its problems. Transactions can fail for a variety of reasons, and sometimes we don’t hear about failed transactions for up to 10 days. Our ACH system requires more by-hand intervention than our credit card system. Sometimes we receive a Notification Of Change (NOC) on ACH, which means that we have to change somebody’s account number. Other times we receive a NOC, which means that the account has been deleted.
Our enthusiasm for these new electronic payment systems were not shared by our customers. After all, they had been able to pay by check whenever they wanted—and not paying when they didn’t. We had essentially been giving our customers percent interest loans on the overdue balances.
After nearly eight months of asking our customers to pay by ACH because it saved us money, we hit upon a better approach. We started giving a $1 monthly discount for customers who paid by credit card, and a $2 monthly discount for those who paid by ACH. Soon customers were calling us up, volunteering their credit card numbers and their bank account information. They thought they were getting a terrific deal. As far as we were concerned, the $2 per month discount was a lot cheaper for us than typing the checks into two accounting systems, taking the checks to the bank, and hounding customers who forgot to make their payments.
Vineyard.NET had been operational as a commercial service for about two weeks when I started wondering if we had enough phone lines. Unfortunately, there wasn’t any way to know for sure: neither our UNIX operating system nor our Cisco routers came with monitoring software that would accurately report usage over time.
One way to plan for capacity is simply to count how many users you have and make an educated guess about how many resources each person requires. For instance, one rule of thumb in the ISP industry is that you need 1 dial-up phone line for every 10 customers, for a ratio of 1:10. But another rule of thumb is that you need 1 modem for every 20 customers.
There is not much in the way of good hard data behind these numbers. It depends on a lot of things, such as whether your customers are dialing in from work or from home, what time of the year they are dialing in usage is higher in the winter, when the nights are longer); whether your customers are married or not; whether they have children; and whether there is anything especially interesting on the Internet. We might want to have a lower ratio because there isn’t a whole lot to do on Martha’s Vineyard after the sun goes down, but have a higher ratio because a lot of the computer users on the island aren’t very sophisticated—and thus aren’t likely to stay online for extended periods of time.
Another approach is simply to buy more phone lines when people start getting busy signals. But adding capacity isn’t cheap. The phone company charges us $50 to install each new phone line. Off-the-shelf 28.8 modems can be had for as low as $160 today, but if you don’t want a maintenance nightmare you’ll buy rack-mounted modems, which cost $600 for the rack (holds 16) and $240 for each modem. Connected to the modems you need a terminal concentrator such as a Livingston Portmaster or a Cisco 2511. We had more experience with the Cisco, so we bought one of those for $3000; it can handle 16 modems at a time. Crunch these numbers and you get a total cost of $515 to install a new port. Then there is the monthly cost of $20 per phone line. And every time you add another phone line, you increase the amount of capacity that’s needed for the ISP’s connection to the rest of the Internet.
It turns out that the only sensible way to gauge your capacity is to monitor how much you are using and make your decisions accordingly. We ended up developing a number of reports to do this.
The first report is a modem report. This report is sent nightly and includes the number of users who dialed up during the last 24-hour period, the number of calls per modem, a histogram of the number of users logged on at a time, and a hourly graph that shows the number of people logged on by time of day. This report is built by examining the login/logout records.
The second report we developed shows the usage of our high-speed connection to the Internet. This report is built by querying our external router every 5 minutes and recording the results.
When we started looking at our reports over time, we were somewhat surprised by the results:
Dial-up usage was steadily increasing, but not because people were staying on significantly longer. Instead, people were calling up more frequently and staying on for roughly the same amount of time.
Peak usage dropped considerably after daylight savings time kicked in.
Need for bandwidth may be illusory: we ran an ISP with 32 people dialed-up simultaneously and never came near to saturating our connection to the outside Internet.
Here is what one of Vineyard.NET’s daily reports looks like
Example A-1. Vineyard.NET’s Daily Modem Report Standard Example Format
From: "Mr. Logger" <logger>
Subject: Logger
Generating report for last 24 hours
Total number of calls: 530
Average time per call: 24 minutes
Longest call: 472 minutes (ericx) (16:06 - 23:59)
*** Bad Lines: A-14 (1)
Samples with 0 callers: 0
Samples with 1 callers: 3
Samples with 2 callers: 7
Samples with 3 callers: 27
Samples with 4 callers: 13
Samples with 5 callers: 14
Samples with 6 callers: 19
Samples with 7 callers: 11
Samples with 8 callers: 6
Samples with 9 callers: 19
Samples with 10 callers: 18
Samples with 11 callers: 20
Samples with 12 callers: 27
Samples with 13 callers: 19
Samples with 14 callers: 21
Samples with 15 callers: 15
Samples with 16 callers: 18
Samples with 17 callers: 11
Samples with 18 callers: 7
Samples with 19 callers: 4
Samples with 20 callers: 4
Samples with 21 callers: 1
Samples with 22 callers: 2
Samples with 23 callers: 1
Sample size: 5 minutes
Line Reports
Short Total Average Longest
Line Calls Calls Min. Length Call
_________________________________________________________________
A-01 21 2 7'55'40'' 22'39'' 2'14'34''
A-02 22 0 8'43'29'' 23'47'' 1'36'53''
A-03 17 1 10'29'33'' 37'01'' 7'52'33''
A-04 25 0 6'15'32'' 15'01'' 1'42'24''
A-05 22 1 10'03'05'' 27'24'' 2'49'48''
A-06 28 3 7'12'31'' 15'26'' 1'58'29''
A-07 23 0 7'09'47'' 18'41'' 1'27'24''
A-08 28 2 9'03'33'' 19'24'' 1'50'38''
A-09 17 1 6'41'55'' 23'38'' 1'14'20''
A-10 22 2 7'54'02'' 21'32'' 1'57'00''
A-11 15 1 11'15'53'' 45'03'' 6'36'20''
A-12 16 0 11'45'13'' 44'04'' 3'52'53''
A-13 24 2 7'05'54'' 17'44'' 1'20'37''
A-15 24 1 7'42'27'' 19'16'' 1'48'53''
A-16 21 0 8'30'44'' 24'19'' 1'47'59''
B-01 25 3 6'46'32'' 16'15'' 1'57'28''
B-02 17 0 7'23'25'' 26'05'' 2'18'28''
B-03 26 0 6'59'04'' 16'07'' 1'17'53''
B-04 21 1 9'37'57'' 27'31'' 3'15'40''
B-05 21 1 6'10'08'' 17'37'' 1'11'24''
B-06 18 0 10'04'32'' 33'35'' 2'32'54''
B-07 24 2 8'04'27'' 20'11'' 1'56'46''
B-08 23 2 8'16'24'' 21'34'' 3'18'23''
C-01 14 0 8'53'42'' 38'07'' 3'20'50''
C-02 16 1 12'53'02'' 48'18'' 4'28'25''
Total lines: 25
Short calls are calls that are less than 30 seconds.
Usage by hour:
Time average number logged in
0:00 - 0:59 7.6 **************
1:00 - 1:59 7.1 *************
2:00 - 2:59 4.7 ********
3:00 - 3:59 2.2 ***
4:00 - 4:59 3.1 *****
5:00 - 5:59 3.1 *****
6:00 - 6:59 4.2 *******
7:00 - 7:59 5.2 *********
8:00 - 8:59 10.1 *******************
9:00 - 9:59 12.9 ************************
10:00 - 10:59 15.1 *****************************
11:00 - 11:59 13.0 *************************
12:00 - 12:59 10.3 *******************
13:00 - 13:59 11.2 *********************
14:00 - 14:59 12.5 ************************
15:00 - 15:59 17.8 **********************************
16:00 - 16:59 14.2 ***************************
17:00 - 17:59 16.2 *******************************
18:00 - 18:59 13.4 *************************
19:00 - 19:59 13.5 **************************
20:00 - 20:59 9.8 ******************
21:00 - 21:59 9.4 *****************
22:00 - 22:59 15.4 *****************************
23:00 - 23:59 15.2 *****************************