Would anyone reading this book say that losing data is OK? I don’t believe so. Then why do we treat backups so lightly? Sometimes I feel like Rodney Dangerfield when I’m arguing for better backups—“I tell ya, I don’t get no respect, no respect.” Backups often aren’t considered during systems design. When a new server is purchased, does anyone ask for the impact on the current backup methodology? Some IT departments do not even have control over the purchase of new systems, because they are sometimes bought by other cost centers. Have you ever tried to explain to another department manager why his terabyte-sized database server isn’t going to get backed up to the standalone, gigabyte-sized tape drive that came with it?

Another often-overlooked issue is backup personnel. Have you ever tried to find the person in charge of backups? It’s often an extra duty that gets passed around, in a manner similar to the way my sister and brother and I argued over whose turn it was to wash the dishes. If you are lucky enough to have a dedicated person, it’s usually the most junior person in the company. I know, because that’s how I got my first job. In fact, that’s how many people get their first jobs. How can we give such low priority to something so important? Perhaps we should change that. Will one book change this long-standing hiring tradition? Probably not, but maybe it will help. At the very least, if the person in charge of backups has this book, that person has a complete guide to accomplishing the immense task that lies ahead.

What’s the big deal, you say? With modern computer systems and reliable disk drives, why are backups still so important? Because computers still go down, that’s why. Also, companies are placing more reliance than ever on computers functioning reliably. I don’t care how good your Unix vendor is or how reliable your disk drives are or even if you have Dogbert himself as your network administrator, systems go down. Murphy’s Law thrives in computer systems. Not only will your computer systems go down occasionally, but they will do so at the time most inconvenient to you and your customers. At that moment, and that moment will come, it is the job of the backup person to replace the data on the disk or disks that have stopped the show. “How long will it take?” is a typical question. The only acceptable response is “it’s already done.”

Who wants to be the person who messed up the restore and caused the customer database to be offline for three extra hours? Who wants to be the person who has to send a memo to the entire company saying that any purchase orders entered in the last two days have to be reentered? Who wants to be the person who has that in mind every day as they are checking the results of last night’s backups? If you do your job well, and no data is lost, you are just doing what you’re supposed to do. If you mess up, you’re in big trouble. Who wants that job? No one, that’s who.

You’re reading this book because you’ve got the impossible job that nobody wants. Whether you’ve been doing it for a while or have just started down the backup road, you can see that the task that lies ahead is immense. The volume of data is tremendous, the nature of the data changes constantly, and the utilities at your disposal never seem to be up to the job. I know because I’ve been there. I’ve spent months trying to implement “solutions” from operating systems and database products that weren’t ready. I’ve seen companies spend money on expensive commercial utilities, only to buy the wrong utility for their application. I’ve watched newer and bigger servers roll in the door without a single backup drive among them. I’ve also spent long nights and weekends in computer rooms trying to recover data in a “reasonable” amount of time. Unfortunately, “reasonable” is defined by the end user who has no idea how difficult this job is.

There are now solutions to almost every backup problem out there. If you run a small shop with just a few systems, all of which run the same operating system, there’s a solution for you. If you work in a huge shop with hundreds of boxes in the various flavors of Unix, Linux, Windows, and Mac OS, or just a few multiterabyte databases, there’s a solution for you. The biggest part of the problem is misinformation. Most people simply do not know what is available, so they either suffer without a solution or settle for an inferior one—usually the one with the best salesperson. The six important questions that you have to continually ask yourself and others are why, what, when, where, who, and how:

When trying to decide what files to include in your backups, take the most pessimistic technical person in your company out to lunch. In fact, get a few of them together. Ask them to come up with scenarios that you should protect against. Use these scenarios in deciding what should be included, and they will help you plan the “how” section as well. Ask your guests: “What are the absolute worst scenarios that could cause data loss?” Here are some possible answers:

What do you do if one of these scenarios actually happens? Do you even know where to start? Do you know:

First, you need to recover your backup server, because it has all the information you need. OK, so now you found the backup company’s card in your wallet, and you’ve pulled back every volume they had. Since your media database is lost, how will you know which one has last night’s backup on it? Time is wasting....

All right, you’ve combed through all the volumes, and you’ve found the one you need to restore the backup server (easier said than done!). Through your skill, cunning, and plenty of help from tech support, you restore the thing. It’s up and running. Now, how many disks were on the systems that blew up? What models were they? How were they partitioned? Weren’t some of them striped together into bigger volumes, and weren’t some of them mirroring one another? Where’s that information stored? Do you even know how big the drives or filesystems were? Man, this is getting complicated....

Didn’t you just install that big jumbo kernel patch last week on three of these systems? (You know, the one that stopped all those network broadcast storms that kept bringing your network down in the middle of the day.) You did make a backup of the kernel after you did that, didn’t you? Of course, the patch also updated files all over the OS drive. You made a full backup, didn’t you? How will you restore the operating system drive, anyway? Are you really going to go through the process of reinstalling the operating system just so you can run the restore command and overwrite it again?

Filesystems aren’t picky about size, as long as you make them big enough to hold the data that you restore to them, so it’s not too hard to get those filesystems up and running. But what about the database? It was using raw partitions. You know it’s going to be much pickier. It’s going to want /dev/rdsk/c7t3d0s7, /dev/dsk/c8t3d0s7, and /dev/dsk/c8t4d0s7 right where they were and partitioned just as they were before the disaster. They also need to be owned by the database user. Do you know which drives were owned by that user before the crash? Which disks were those again?

It could happen.

Make sure you can access essential information in the event of a disaster:

/opt/app/opus/transactions/portfolio/third-party/...etc...etc.../USD/CAI/GBP/BONY/ask/19970615120453.2372149821335

I remember an administrator at one of my previous employers who used to say, “Are we getting this on tape?” He always said it with his trademark smirk, and it was his way of saying “Hi” to the backup guy. His question makes a point. There are some global ways that you can approach backups that may drastically improve their effectiveness. Before we examine whether to back up part or all of the system, let us examine the common practice of using include lists and why they are dangerous. Also, let’s consider some of the ways that you can avoid using include lists.

What are include and exclude lists? Generically speaking, there are two ways to back up a system:

Looking at these examples, ask yourself what happens when you create /data4 or the F: drive? Someone has to remember to add it to the include list, or it will not be backed up. This is a recipe for disaster. Unless you’re the only one who adds drives or filesystems and you have perfect memory, there will always be a forgotten drive or filesystem. As long as there are other administrators and there is gray matter in your head, something will be left out.

However, unless your backup utility supports automated drive or filesystem discovery, it takes a little effort to say, “Back up everything.” How do you make the list of what systems, drives, filesystems, and databases to back up? What you need to do is look at files such as /etc/vfstab or the Windows registry and parse out a list of drives or filesystems to back up. You can then use exclude lists to exclude any drives or filesystems you don’t want backed up.

Oracle has a similar file in Unix, called oratab, which can be used to list all Oracle instances on your server.^[1] Windows stores this information in the registry, of course. You can use oratab to list all instances that need backing up. Unfortunately, Informix and Sybase databases have no such file unless you manually make one. I do recommend making such a file for many reasons. It is much easier to standardize system startup and backups when you have such a file. If you design your startup scripts so that a database does not get started unless it is in this file, you can be reasonably sure that any databases that anyone cares about will be in this file. This means, of course, that any important databases are backed up without any manual intervention from you. It also means that you can use the same Informix and Sybase startup scripts on every system, instead of having to hardcode each database’s name into the startup scripts.

How do you know what systems to back up? Although I never got around to it, one of the scripts I always wanted to write was a script that monitored the various host databases, looking for new systems. I wanted to get a complete list of all hosts from Domain Name System (DNS) and compare it against a master list. Once I found a new IP address, I would try to determine if the new IP address was alive. If it was alive, that would mean that there was a new host that possibly needed backing up. This would be an invaluable script; it would ensure there aren’t any new systems on the network that the backups don’t know about. Once you found a new IP address, you could use nmap to find out what type of system it is. nmap sends a malformed TCP packet to the IP address, and the address’s response to that packet reveals which operating system it is based on.

Assuming you’ve covered things that are not covered by normal system backups, you are now in a position to decide whether you are going to back up your entire systems or just selected drives or filesystems from each system. These are definitely two different schools of thought. As far as I’m concerned, there are too many gotchas in the selected-filesystem option. Backing up everything is easier and safer than backing up from a list. You will find that most books stop right there and say “It’s best to back up everything, but most people do something else.” You will not see those words here. I think that not backing up everything is very dangerous. Consider the following comparison between the two methods.

The following are various backup levels. These terms are not used the same way by everyone.

A question that I am often asked is, “You want me to back up every night?” What the question really means is, “Even on the weekend?” Nobody’s working on the weekend, right? Right...except for your noisiest customer last weekend. You know the customer I’m talking about: the one who calls your boss instead of the help desk when there’s a problem. And if your boss isn’t in or doesn’t fix the problem fast enough, this customer will call your boss’s boss. Well, last weekend this customer was really behind, so she spent the entire weekend at work, working around the clock on next year’s budget. She finally got it straightened out at about 1:00 a.m. Monday. At around 4:00 a.m., the disk where her home directory resides stopped working. (Everything dies Monday morning, doesn’t it?) You haven’t run a backup since Friday night. Your phone is ringing, and it’s your boss. Any guesses as to what he wants to talk to you about? Do you want to be the one to tell this customer that you could have saved her file, but you don’t run backups on the weekend?

There are several schools of thought on this question. The following are some suggested backup schedules.

0 3 2 5 4 7 6 9 8 9
0 3 2 4 3 5 4 6 5 7 6 8 7 9 8

This phrase from a Billy Joel song indicates the usual best time to do backups. Backups should be scheduled in such a way that they do not run during normal business hours. Sometimes you cannot avoid it, but it should not be a regular occurrence. There are two main reasons for this:

As stated earlier, how you want to do your restores determines how you want to do your backups. One of the questions that you must ask yourself is, “What are you going to protect yourself from?” Are the users in your environment all “power users” who use their computers intelligently and never make dumb mistakes? Would your company lose a lot of essential data if the files on your users’ PCs were accidentally deleted? If a hurricane took out your whole company, would it be able to continue doing business? Make sure that you are aware of all the potential causes for data loss, and then make sure your backup methods are prepared for all of them. The most exhaustive list of potential causes of data loss that I have seen is in another O’Reilly book called Practical Unix and Internet Security by Simson Garfinkel and Gene Spafford. Their list, with my comments attached, follows:

If you work in a shop with a modest budget, you probably looked at this heading and said, “Sure, if I could afford it.” Although automation that involves expensive jukeboxes and autochangers is nice, that is not the type of automation I am talking about. There are two types of automation. One type allows your backups to complete an entire cycle without requiring any manual intervention from you, such as ejecting and loading new volumes. This type of automation can make things much easier but can also make them much more expensive. If you can’t afford it, a less expensive alternative is to have your backup system notify you when you need to do something manually. At the very least, it should notify you if you need (or forgot) to change a volume. If things aren’t going right, you need to know. Too many times people look at their backup logs only when they need to do a restore. That’s when they find out that their backups have failed for days or weeks. A slightly intelligent backup system could email you or page you if things don’t go the way you expect them to go.

The second type of automation is actually much more important. This type of automation refers to how your backups “think.” Your backup process should know what to back up without you telling it. If a DBA installs a new database, your backups should know about it. If a system administrator installs a new drive or filesystem, your backups should automatically include it. This is the type of automation that is essential to safe backups. A good backup system should not depend on a human brain to remember to do something.

Another common problem happens as a backup system grows over time. What works for one or two boxes doesn’t necessarily work for 200. As the volume of data grows, the need for a standardized backup system becomes greater and greater. This is a problem because most administrators, as they are writing their shell script to back up five or six boxes, do not think ahead to the time when there may be many more. I can remember my early days as the backup guy. I had 10 or 11 systems, and the “monster” was an Ultrix box. It was “huge,” we said in those days. (It was almost 8 gigabytes!) The smallest tape drive we had was a 10 GB (with compression) Exabyte. We used the big 10 GB tape drive for the 8 GB system. We had what I considered to be a pretty good in-house backup script that worked without modification for two years.

Then came the HPs. The smallest system was 20 GB, and the biggest was much bigger than that. But these big systems came with a little 2 GB (4 with compression) DDS drive. Our backup script author never dreamed of a system that was bigger than a tape. One day I woke up, and our system was broken. I then spent months and months hacking up that shell script to support splitting the drive or filesystem into two tapes. Eventually, I gave up and bought a commercial product. My point is that if I had thought of that ahead of time, I might have been able to overcome the limitation without losing so much sleep.

When you are designing your backup system—or your data center, for that matter—plan on your systems getting bigger and more numerous. Plan for what you will do when that happens—trust me, it will happen. It will be much better for your mental health (not to mention your job security) if you can foresee the inevitable and plan for it when you design the system the first time. Your backup system is something that should be done right the first time. And if you spend a little time dreaming about how to break it before you design it, you can save yourself a lot of money in antacids and sleeping pills.

Unix, Linux, and Mac OS systems record three different times for each file. The first is mtime , or modification time. The mtime value is changed whenever the contents of the file have changed, such as when you add lines to a logfile. The second is atime , or access time. The atime value is changed whenever the file is accessed, such as when a script is run or a document is read. The last is ctime , or change time. The ctime value is updated whenever the attributes of the file, such as its permissions or ownership, are changed.

Administrators use ctime to look for hackers because they may change permissions of a file to try to exploit your system. Administrators also monitor atime to look for large files that have not been accessed for a long time. (Such files can be archived and deleted.)

Windows files stored on an NTFS filesystem and some files stored on modern Linux filesystems use access control lists (ACLs) to grant or restrict permissions to users. ACLs say who can read, write, execute, modify, or have full control over a file. Figure 2-1 shows an example of such ACLs.

Figure 2-1. Access control list example

You need to investigate how your backup product is handling ACLs. The proper answer is that they are backed up and restored. This is a feature common with commercial products, but unfortunately, not all open-source products do this. Make sure you look into this when evaluating open-source tools.

Mac OS files stored in MFS, HFS, or HFS Plus filesystems have two forks: the data fork and the resource fork. The data fork contains the actual data for the file, such as its text. The resource fork contains related structured data, such as offsets, menus, dialog boxes, and icons for the file. The two forks are tightly bound into a single file. While they are typically used by executables, every file can have a resource fork, and other applications can use it as well. For example, a word processing program may store a file’s text in the data fork and the file’s images in the file’s resource fork.

These resource forks, like Windows ACLs, need to be backed up, and not all backup products back them up properly. Make sure you investigate what your backup system does with the data fork and resource fork.

K.I.S.S. Have you seen this acronym before? It applies double or triple to backups. The more complicated your backup scheme is, the more likely it is to fail. If you do not understand it, you cannot implement it. Remember this every time you consider adding a new bell or whistle to your backup system. Every change puts your data at risk. Also, every change might make your backup system that much more complex—and more difficult to explain to the new backup person. One of the heads of support for a commercial backup product said that he sees the same thing over and over again. One person gets to know the software really well and writes various scripts to automate this and that. Backups become a well-oiled machine—until they are turned over to the trainee. The trainee doesn’t understand all the bells and whistles, and things start breaking. All of a sudden, your data is in danger. Keep that in mind the next time you think about adding some cool new feature to your backup script.

This next comment also relates to the previous section about “thinking big.” One of the common judgment errors is to not automate in the beginning. It’s so much easier to just put a hardcoded include list in a file somewhere or put it in the cron or scheduled task entry itself. However, that creates many different backup methods. If each box has its own special customized backup system, it is very hard to monitor your backups and explain them to the new person.

It’s not such a big deal when you have two or three systems, but it is when you grow to 200 systems. If you have to remember every system’s idiosyncrasies every time you look at your logs, things inevitably get out of control. Exceptions for each system also can mean that things get overlooked. Do you remember that nine months ago you excluded /home* on apollo? I hope so, if apollo just became your primary NFS server, and it now has seven home directories.

If you cannot explain your backups to a stranger in less than a few hours, things are probably too complex. You should look at implementing things like centralized logging, standardized backup scripts, and some level of automation.

If you’ve read this far, you know that I consider your backups very important. If your backups are important, isn’t the media on which they reside just as important? That goes without saying, right? Well, you’d never know it from most volume “libraries.” Volume “piles” is probably a more accurate term. How many computer rooms have you seen that have volumes spread out all over the place? They get stacked, piled, fall behind the systems, and a tape cartridge works really well as a coaster for a coffee mug. (We wouldn’t want to get any coffee rings on the new server, right?)

Have you ever really needed a volume and couldn’t find it? I’ve been there. It’s a horrible feeling to know that you’ve got the file on a volume, but can’t find the darn volume! Why, then, do we treat our backup volumes like so much dirty laundry? Organize your backup volumes! Label them, catalog them, give them unique names or numbers, and put them in some sort of logical order in some kind of storage container. Do it, or the backup demon will come to haunt you!

What about that media cabinet that you’re using for your on-site volume storage? You don’t have one, you say? You’re using a file cabinet, you say? Well, use something, but if you can afford it, a number of companies make storage containers for media. They also make cabinets that can withstand fire. Spend the money; you’ll be glad you did. Doing a restore is so much less stressful when you can find the volume with no problem. Remember, though, that fireproof does not mean heat-proof. These types of media safes are meant to withstand brief fires that are quickly extinguished by a sprinkler system. If a fire burns for a long time right next to the container or raises the temperature in the room significantly, the volumes may be no good anyway. (This is another good reason why you also must store volumes off-site.)

Have the most well-organized person in your office design your media storage system. Here’s an idea. Ask your best administrative person to take a look at your storage system and compare it to his filing cabinet. Explain that you want an honest evaluation.

Once you have organized the media that you are storing on-site, it’s time to consider off-site storage. There are two ways to store your data off-site:

The latter can be expensive, but not as expensive as some people think. It is also much easier to use during a disaster, and you can’t lose tapes if there aren’t any tapes to lose. That is, of course, what off-site storage is meant to prepare you for—the destruction of your media and/or the building that holds it. If you have a complete set of backups in another location, you will be able to recover from even the worst local disaster.

It is important to test every type of restore. If you are testing filesystem backups, make sure you:

If you are testing database restores, make sure you:

As I said earlier, sit around one day with some really pessimistic people, and ask them to dream up scenarios for you to test. Test your ability to recover from each of these scenarios on a regular basis. What works this month might not work next month. The only thing that is guaranteed to remain constant is change. One suggestion is to create a list of recovery procedures and randomly test a subset of them every month. Management changes, hardware changes, networks change, and OS and database versions change. Every change you know about should make you want to perform a test of the affected area.

I don’t care how good your backups are; they can always be better. You could spend every waking hour tweaking and improving every piece of your backup program and know everything there is to know about backups, and they could still be better. My backups will never be good enough. There’s always a new bell or whistle on some other backup package, a bigger or smarter jukebox, a faster backup drive, or some scenario I thought of that I’m not covering. You must realize, however, that every change you make has a potential for data loss. A common thread that you will find in this book is that every time the human being enters into the equation, things can go wrong. You may be the best shell or Perl hacker in the world, and you will still make mistakes.

Baroque needed fixing. Come on! One constant tempo? And the harpsichord? It had to go. Thankfully, Bartolomeo Cristofori invented the piano in 1709 and gave us an instrument that could play loud (fortissimo) and quiet (piano)—hence its name. Shortly after that, the Classical period started, and music started changing in tempo and feeling.

But if it’s not broken, don’t fix it! You’ve heard it before, but given the risk that each change makes, it goes double in the backup world. As you read this book or some magazine, or talk to other administrators, you will undoubtedly come up with a list of things that you wish you were doing. Concentrate on the holes, or scenarios that your backup and recovery plan just does not cover. Worry about the fact that none of your volumes are stored off-site before you think about working on that cool menu program you’ve been wanting to write for your restores. Make sure you’re covering all the bases before you start redecorating the stands. Before you consider making a new change, ask yourself if something else is more important and if the change is really necessary and worth the risk.

One of the reasons that backups are unpopular is that people are worried that they might get fired if they do them wrong. People do get in trouble when restores don’t go right, but following the suggestions in this section will help you protect yourself from “recovery failure fallout.”

This final section has absolutely nothing to do with backups. It has to do with politics, budgeting, money, and cost justifications. I know that sometimes it sounds as if I think that backups get no respect. Maybe you work at Utopia, Inc., where the first thing they think about is backups. The rest of us, on the other hand, have to fight for every volume, drive, and piece of software that we buy in order to accomplish this increasingly difficult task of getting it all on a backup volume.

Getting the money you need to accomplish your task can sometimes be very difficult. Once a million dollar computer is rolled in the door and uncrated, how do you tell the appropriate department that the small, standalone backup drive that came with it just isn’t going to cut it? Do you know how many hoops they went through to spend a million dollars on one machine? You want them to spend how much more?