Chapter 6. Emergency tools: Building a system recovery device
This chapter covers
- Recovering broken Linux systems
- Controlling resources with Linux live-boot drives
- Recovering data from damaged storage media
- Manipulating an inaccessible file system
Don’t try to convince yourself otherwise: along with all the good stuff, you’re going to have bad days with Linux. Sometimes you’re going to forget the syntax for commands (which is why you should always keep a copy of this book within reach). You (or the users you support) are going to mistype commands and permanently destroy documents. Or, you’re going to experience that sinking feeling when you realize that some important piece of hardware or software has failed. (That’s gratitude after everything you did for it all those years.) Being properly backed up, as the last couple of chapters demonstrated, means that you can walk away from a nonfunctioning operating system (OS) or computer and rebuild it all somewhere else. But that’s always going to be Plan B. Plan A is to recover.
This chapter is where I’ll introduce you to key tools from the Linux recovery suite. You’ll learn how you can use a live-boot device to load a fresh copy of Linux, mount the drive that’s giving you trouble, and either update corrupted configuration files so you can boot normally again or rescue whatever data can be recovered before repurposing or destroying the damaged drive. And you’ll see how the files of a non-functional system can be brought to life and run within their own virtual environment, so you can do things like changing a user’s forgotten password.
There’s a lot that can go wrong when you try to throw hardware and software into a single box and expect them to play nicely together. I’m going to focus on catastrophic events like these:
- Your computer boots, the hard drive is working, but Linux doesn’t load.
- Your computer boots (as far as you can tell, anyway), but you’re not quite sure whether the hard drive is fully functional.
- Everything works, but a software problem or lost password prevents you from logging in to Linux.
The specific problem you’re facing will determine your plan of action to get yourself back in business. Figure 6.1 illustrates some diagnostic and recovery options, most of which I’ll discuss later in this chapter.
Figure 6.1. Common system problems along with diagnostic and solution considerations that we’ll explore in this chapter
6.1. Working in recovery/rescue mode
Linux not letting you log in normally? Perhaps the boot process unexpectedly stops before displaying the login screen. You’ll want some basic system administration tools.
But wait: if Linux won’t load, how are you going to launch those tools? Well, even if Linux won’t load all the way to a normal command prompt, often it’ll get you to the GRUB menu. From there (as figure 6.2 shows), you can use the up and down arrow keys and then Enter to select a Linux kernel running in recovery mode. That, as you’ll soon see, will open up a whole bag of tricks.
Figure 6.2. The GRUB Advanced Options menu of an Ubuntu installation showing links to both current and older kernel versions, along with options for launching in recovery mode
Before you’ll be able to take full advantage of those tools, however, you’ll first need to understand what GRUB is and how it works. In the next several sections, I’m going to explain what GRUB does, how it can be used to access recovery tools, and how you can use those tools to get yourself out of some nasty jams.
6.1.1. The GRUB bootloader
What’s GRUB? It’s the GNU GRand Unified Bootloader. OK, what’s a bootloader? It’s the code an OS uses to bring itself to life when it’s powered on. Figure 6.3 illustrates the process.
Figure 6.3. The key steps in the boot process of a Linux computer
When a computer powers up, firmware instructions embedded in the basic system hardware identify the network, storage, and memory resources that are available. This was done through the BIOS system on older computers and, more recently, using UEFI (both of which you briefly met back in chapter 4).
Once the system finds a hard-drive partition containing a Master Boot Record (MBR), it loads the contents into active memory. On Linux systems, the MBR partition contains a number of files that, when run, present one or more loadable kernel image boot configurations. You can choose to load any of those configurations from the GRUB bootloader menu.
Note
Often, GRUB will load its default image automatically without asking for your opinion, unless a previous session fails to load. If you want to force the GRUB menu to appear, press the right Shift key as your computer boots.
6.1.2. Using recovery mode on Ubuntu
As you can see from figure 6.4, once Ubuntu is loaded in recovery mode, you’ll be shown a menu of tools that address some common boot-time problems. It’s worth trying each in turn, in case one of those ends up addressing your root problem. The clean option, for instance, removes unused files if you suspect the trouble stems from a full disk. And the dpkg option attempts to fix any broken APT-based software packages that might be gumming things up. (The dpkg tool might require that you first enable networking.)
Figure 6.4. The Ubuntu Recovery menu with links to some basic diagnostic and repair tools, along with the option of opening a shell session as root
The root option opens a root command-line shell session for you where you’ll have Bash at your disposal. In general, using a simple shell session for recovery rather than a full GUI desktop makes a lot of sense. That’s because the fewer complicated services you’ve got running, the more likely it is that you’ll be able to at least get your system going.
Once you do manage to get a working command prompt, you can start poking around to see if you can identify and fix the problem. At the very least, you’ll look mighty cool doing that.
6.1.3. Using rescue mode on CentOS
The GRUB menu for CentOS offers a rescue kernel at boot time rather than a recovery mode. This kernel doesn’t include a tool menu like Ubuntu’s, but it will similarly drop you into a single-user shell session as root. Figure 6.5 shows the rescue boot option in CentOS GRUB.
Figure 6.5. CentOS Linux offers a rescue kernel for booting directly to a single-user shell session for troubleshooting a damaged system.
Once you choose the rescue selection on your CentOS machine, what’s next? You’ll get to meet some useful tools through the rest of this chapter. But first, why not take a page from the Ubuntu playbook and manually apply some of the tools from the automated Recovery menu? “Easier said than done,” I hear you say. “But just how do those tools work?” Over next page or so, I’m going to show you how to put some of your Bash and scripting skills to work figuring some of that out.
6.1.4. Finding command-line rescue tools
Got an Ubuntu machine running (this one won’t work on Debian)? If there’s code running the menu, it must already exist somewhere within the Ubuntu file system. Go take a look for yourself. Use locate to find it:
$ locate recovery-mode /lib/recovery-mode /lib/recovery-mode/l10n.sh 1 /lib/recovery-mode/options /lib/recovery-mode/recovery-menu 2 /lib/recovery-mode/options/apt-snapshots /lib/recovery-mode/options/clean 3 /lib/recovery-mode/options/dpkg /lib/recovery-mode/options/failsafeX /lib/recovery-mode/options/fsck /lib/recovery-mode/options/grub /lib/recovery-mode/options/network /lib/recovery-mode/options/root /lib/recovery-mode/options/system-summary
- 1 The 110n.sh script sets appropriate environment variables for the menu.
- 2 The recovery-menu script file
- 3 The script for reducing disk usage
If you navigate over to the /lib/recovery-mode/ directory, you’ll see that the recovery-menu file is the script that displays the menu interface you saw back in figure 6.3. The options/ directory contains files for executing each of the menu items. For example, fsck will check and fix, if possible, any broken file systems.
Because you’re now an accomplished Bash scripting expert, why not take a look at each of the scripts in the options/ directory to see if you can figure out how they work? Here are the contents of the fsck script to get you going. Note the way the script is nicely documented (using the # character) to help you understand what’s going on:
$ cat /lib/recovery-mode/options/fsck #!/bin/sh . /lib/recovery-mode/l10n.sh 1 if [ "$1" = "test" ]; then echo $(eval_gettext "Check all file systems") exit 0 fi # Actual code is in recovery-menu itself 2 exit 0
- 1 The 110n.sh script is called to set the environment.
- 2 This comment tells you where the real action takes place.
Here are a couple of things you can try on your own. Manually run the clean script on a Ubuntu machine. What happened? Then try carefully editing the /lib/recovery-mode/recovery-menu script (make a backup copy first). Change something simple, like the menu title or one of the script descriptions. Then reboot your machine and, from the GRUB menu, go into Recovery Mode to see what the recovery environment looks like. With some variations and exceptions, you should be able to put those scripts to good use elsewhere, including on CentOS.
6.2. Building a live-boot recovery drive
As you probably already know, those .ISO OS images you employed for your VirtualBox VMs back in chapter 2 can also be written to a CD or USB drive and used to boot a live session of the OS. Such live-boot devices let you load fully functioning Linux sessions without having to install anything to a hard drive. Many people use such drives to confirm that a particular Linux distribution will run happily on their hardware before trying to install it. Others will run live sessions as a secure way to maintain their privacy while engaged in sensitive activities like online banking.
It turns out that those live boot drives are also a fantastic tool for system rescue and recovery. Remember our second disaster scenario from earlier in this chapter?
Your computer boots (as far as you can tell, anyway), but you’re not quite sure whether the hard drive is fully functional.
Plugging a live-boot drive into a troubled computer and launching Linux with all of its administration tools can help you figure out what’s really going on—and give you the tools to fix it. I’m going to show you how to create a live-boot USB drive and then how to use it. But first, a quick look at some of the most useful distro images currently available.
6.2.1. System rescue images
If you happen to already have a DVD or USB drive with a full Linux system (like Ubuntu) lying around anyway, then that’ll be your simplest solution because most of the software you’ll need will come preinstalled. Assuming you’ve got a network connection, you can always install other packages during a live session. Otherwise, the following sections describe some specialty images.
Boot-Repair
If it’s an Ubuntu system you’re trying to rescue, then you’ll want to try out Boot-Repair. A small and fast Boot-Repair live-boot checks your GRUB settings and, if necessary, rebuilds them. It can, as shown in figure 6.6, also perform other useful administration tasks. This one has saved the day for me more times than I can remember.
Figure 6.6. The Boot-Repair desktop with the System Tools menu visible
Boot-Repair can also be installed on an already-launched live session. The website (https://help.ubuntu.com/community/Boot-Repair) has great instructions.
GParted Live
If your problem relates to corrupted partitions that can damage your data or prevent you from successfully booting, the GParted Live image brings the full-featured and powerful GParted partition editor to a live CD or USB. Like Boot-Repair, GParted can also be installed and used from any normal Linux session. If you know your problem is partition-related, however, the full live-boot version can be the quickest and most direct way to fix it. Unlike Boot-Repair, GParted is built to be used on about any Linux distribution, rather than just Ubuntu.
SystemRescueCd
Another alternative, SystemRescueCd, is a lightweight image built on the Gentoo Linux distribution. As it comes with loads of useful system recovery tools, SystemRescueCd makes a great USB to carry around with you. (The CD is also great, although it doesn’t fit so easily into back pockets, and sitting down safely can be a challenge.)
6.2.2. Writing live-boot images to USB drives
Before you can write any .ISO image, you’ll need to download it. I think I’m safe assuming that you’re OK finding the right page on the website of the distro you’re after to get that done. But you might not know that large downloads can sometimes be corrupted during their journeys across the internet. Worse, they can be switched with malware by intrusion or man-in-the-middle attackers. That happened to some people downloading Mint Linux a while back.[1]
See Kelly Fiveash, “Linux Mint Hit by Malware Infection on Its Website, Forum After Hack Attack,” Ars Technica, February 22, 2016, http://mng.bz/irPi.
Besides checking to make sure the website you’re on is properly encrypted (your browser should show some kind of lock icon in the URL bar), your best defense is to generate a hash on the downloaded file and compare it to the hash provided by the website. (Hashes were discussed back in chapter 2.)
The hashes will sometimes be displayed right on the image download page, but some distros make them a bit harder to find. Canonical (the fine people who give us Ubuntu) has not exactly bathed itself in glory on this one. There is a single page (https://help.ubuntu.com/community/UbuntuHashes) with links to all its hashes. Finding that page from the download site isn’t simple. You’re usually better off using a reliable internet search engine. Figure 6.7 shows you what the hashes for Ubuntu 17.04 will look like.
Figure 6.7. SHA256 hashes for the various images of Ubuntu 17.04, currently available for download
Once you find the published hashes for your image, you’ll compute the hash for the file you’ve downloaded and compare the two values. This example generates the SHA256 hash for the SystemRescueCd image from the same directory to which it was downloaded:
$ cd Downloads $ ls | grep systemrescue systemrescuecd-x86-5.0.2.iso $ sha256sum systemrescuecd-x86-5.0.2.iso a2abdaf5750b09886cedcc5233d91ad3d1083e10380e555c7ca508 49befbf487 systemrescuecd-x86-5.0.2.iso
With a reliable image of one kind or another safely downloaded, it’s time to create that recovery tools live-boot drive. If both your current computer and the live-boot USB you’re creating will run Debian, Ubuntu, or a derivative, the simplest approach is to use the Ubuntu Startup Disk Creator.
The Creator tool is available from the regular GUI menus and, as you can see from figure 6.8, is straightforward. Select an .ISO from somewhere on your hard drive and a target USB (or CD) where you’d like the image written. The Creator will take care of the rest.
Figure 6.8. Ubuntu’s Startup Disk Creator with an Ubuntu Server 16.04 image and target USB drive selected
If, on the other hand, you need a live-boot device built from a different distro (you’re trying to rescue a CentOS machine and want to use CentOS tools to do it), then you’ll need to dust off your old friend dd. If you want a CD or DVD-based live bootable, dd running on any Linux host is up to the job. But if the image will be written to a USB drive and if you happen to be working on an Ubuntu host, you’ll first need to modify the image by adding an MBR to the .ISO archive so that BIOS and UEFI firmware will know what to do with it. Figure 6.9 represents how the process looks.
Figure 6.9. The steps necessary for writing a working live-boot USB image
On Ubuntu hosts, you use isohybrid for that image modification. The apt package containing isohybrid is called syslinux-utils. Once that’s installed, move to the directory containing the downloaded image and run isohybrid using the image filename as the sole argument:
# apt update # apt install syslinux-utils $ cd ~/Downloads $ isohybrid systemrescuecd-x86-5.0.2.iso
The next steps should work no matter what distribution you’re on. Carefully identify the system designation for your target device. As you’ll remember, df lists all the currently recognized file systems along with their designations:
$ df -h Filesystem Size Used Avail Use% Mounted on udev 3.5G 0 3.5G 0% /dev tmpfs 724M 1.5M 722M 1% /run /dev/sda2 910G 183G 681G 22% / 1 tmpfs 3.6G 214M 3.4G 6% /dev/shm tmpfs 5.0M 4.0K 5.0M 1% /run/lock tmpfs 3.6G 0 3.6G 0% /sys/fs/cgroup /dev/sda1 511M 3.4M 508M 1% /boot/efi tmpfs 724M 92K 724M 1% /run/user/1000 /dev/sdb1 15G 16K 15G 1% /media/myname/KINGSTON 2
- 1 This is my root file system. I definitely don’t want to overwrite that!
- 2 The file system on my removable USB drive
In this example, there’s a file system mounted on my Kingston USB device called /dev/sdb1. This tells me that the device itself is known as /dev/sdb.
If you plan to write the image to an optical drive like a CD or DVD, then you get its designation through lsblk, which stands for list block devices. The drive itself must be writable. Here my DVD drive is known as sr0:
$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 931.5G 0 disksda1 8:1 0 512M 0 part /boot/efi
sda3 8:3 0 7.2G 0 part [SWAP] sdb 8:16 1 14.4G 0 disk
- 1 The writable DVD drive
The moment of truth. No more putting it off. It’s time to unleash dd on your poor, defenseless file system and write your image to your USB.
First unmount the drive itself so dd can get full access. Then write the archive. In this example, I used the systemrescuecd-x86-5.0.2.iso image and wrote it to the drive at /dev/sdb.
Careful! Typing sda instead of sdb (in this particular case) will irretrievably overwrite your host file system and ruin your day, not to mention your life. Naturally, you should also make sure there’s nothing important on the USB drive, as that will definitely disappear:
# umount /dev/sdb
# dd bs=4M if=systemrescuecd-x86-5.0.2.iso
of=/dev/sdb && sync 1
- 1 The added sync command ensures that all cached data is immediately written to the target disk.
It might take some time for dd to finish writing the image to your USB device, but, when it’s done, you should be able to plug the drive into a computer, power it up, and enter a live session. This is assuming that your computer is configured to boot from a USB drive. If it isn’t, you can force it to boot to a selected device this one time by entering the Boot menu during the boot process. Each PC manufacturer designates its own keys for such things (often displaying a list of keystroke options during the startup process). But pressing one of F1, F9, or F12 early in the boot sequence will often work.
You could also enter the setup utility (either BIOS or UEFI) to permanently set the order used to boot devices. Accessing the setup utility might also happen through a range of keys: I’ve seen F2, F10, Del, and Enter.
6.3. Putting your live-boot drive to work
You can do a lot of things with this Linux-in-a-pocket drive that you’ve created. The following sections describe a few common scenarios and the ways you might address them.
6.3.1. Testing system memory
If you’ve experienced sudden and unexpected system crashes, a likely culprit is your physical memory (RAM). Like all hardware, RAM will eventually fail. The problem is that you can’t properly test your RAM for errors while it’s being used by a running OS. Instead, you’ve got to catch it before the OS loads. Fortunately, as the proud owner of a Linux live-boot drive, you’ve already got everything you’ll need for this job. As figure 6.10 shows, one of the menu items that will be displayed after booting to an Ubuntu drive is Test Memory.
Figure 6.10. The main menu of an Ubuntu installation process showing Test Memory
Selecting Test Memory (using the up and down arrow keys and pressing Enter) takes you to the Memtest86+ program (figure 6.11). That program runs multiple scans of your RAM, displaying any errors it finds. Honestly, I’m not sure that the tool will ever stop on its own; I’ve certainly never reached that point. But if it returns no errors after running for at least a few full passes, you’re memory is probably not the cause of your trouble. Which can be either good news or bad news...after all, you still don’t know what’s causing your crashes.
Note
Besides Test Memory, the Rescue a Broken System option in the main menu of an Ubuntu installation process will give you a working Bash shell.
Figure 6.11. The Memtest86+ tool shows the location and kind of any errors in your RAM memory. This system is clean (so far).
CentOS installation disks include their own link to Memtest86+ in the Troubleshooting menu (figure 6.12).
Figure 6.12. The Troubleshooting menu linked to from the main menu on a CentOS installation disk
6.3.2. Damaged partitions
A partition is really metadata pointing to the location on a physical disk occupied by a file system. They’re fragile things, those partitions, and it doesn’t seem to take much to upset them. If disk data is somehow corrupted and the exact addresses of a partition’s start and end points are changed or lost, then the file system on the partition will become unreachable. And if a file system is unreachable, then the data on the file system is as good as gone.
Can’t access your partition? It’s time to boot up your new SystemRescue drive. SystemRescue is a lightweight package, so don’t expect all the GUI miracles of a full-featured distro. Things like laptop touch pads and automatic WiFi connectivity might not work the way you expect. But it’s a quick way to get some powerful rescue tools into play. As you can see in figure 6.13, the default boot option will open a special command-line shell.
Figure 6.13. The Boot Options menu displayed during SystemRescue startup. Note the detailed explanations for each option at the bottom of the page.
As you gear up toward rescuing your stricken partition (or its data), SystemRescue provides some useful orientation information along with the command line, including networking and text-editing basics (figure 6.14). Worry not: both Vim and Nano are available from the command line. And typing startx will load a fast and light GUI desktop.
Figure 6.14. The SystemRescue shell. Note the text editors available by default (including Nano) and also that typing startx launches a graphical desktop.
If you’re going to need network access to download or install more tools, perhaps, or to transfer data, you’ll type net-setup at the command prompt, select the correct interface, and indicate whether your network is wired or wireless. If it’s wireless, you’ll enter your WiFi router’s SSID and its encryption key (which will probably use ASCII rather than hex characters). In most cases, you’ll want to let DHCP auto-detect your network.
With a damaged partition to worry about, your primary concern right now will be recovery. If there’s any chance that the physical disk itself might be failing, then your first task must be to secure its data. For that, I’d use dd at the SystemRescue command prompt to create a perfect copy of the partition in its current state and save it to a healthy disk of equal or greater capacity. After running lsblk to confirm the designations of both partitions, the copy operation might look something like this, where the failing disk is /dev/sda and the empty partition on your target drive is /dev/sdc1:
# dd if=/dev/sda of=/dev/sdc1
With that done, you’re free to see whether you can save the original copy. Type testdisk. You’ll be asked how you’d like session events logged, which disk you want to recover, and which partition type to expect to find. More often than not, TestDisk will guess the right one and highlight it as the default.
You’ll then find yourself facing a screen that looks like the one in figure 6.15, where you can ask TestDisk to analyze your disk, looking for existing partitions. When you’ve discovered and appropriately marked your damaged partitions, you can write the changes to disk, and you should be able to successfully boot once again.
Figure 6.15. The TestDisk Partition Repair page, where partitions discovered through Analyze can then be edited and recovered using the other tools
Wait...that’s it? What about all the fine print? Does “discovered and appropriately marked your damaged partitions” cover the complex process in a way that’s at all useful? Nope. But, then, why are you reading this chapter? I’d say it’s either because you’re trying to prepare for some future catastrophe or because the future is now and you’re looking at your broken partition.
If it’s the future that worries you, then following along with me as I work through the narrowly focused details of one or two hypothetical solutions won’t help all that much. Besides, I’m not going to advise you to purposely corrupt a real partition so you can try out TestDisk for real. And if you’ve got your own real-world problem to deal with, then it’s highly unlikely that any example I happen to choose will be useful to you. Practically, the best I can do is let you know that such software exists and teach you how to get it running. Oh, and there’s some pretty good documentation available at https://cgsecurity.org/testdisk.pdf.
6.3.3. Recovering files from a damaged file system
If it turns out that you can’t fully rescue your disk, perhaps you managed to revive your partitions just enough to allow access to the files but not enough to reliably boot from the disk. Then your new priority is to save as many important files as you can.
The simplest approach is to use the regular Linux file system management tools from any kind of live-boot session. If you’re not sure about the partition’s designation, running lsblk will, as always, get you up to speed.
It’s possible that your partition won’t yet be accessible as a file system because it hasn’t been mounted. If, in fact, it doesn’t show up among the results of a df command, then you’ll have to mount it, meaning assign it to a location within a file system where it can be accessed. You can fix that one quickly enough by creating a new directory where your partition will be mounted and then using the mount command. I chose to create my temporary mount point directory in /run/, but any unused location you happen to find (like /media/ or /mnt/) will do as well. Let’s assume, as before, that the partition is called /dev/sdc1:
# mkdir /run/temp-directory # mount /dev/sdc1 /run/temp-directory
From here on in, any healthy files from the stricken file system will be available to copy or edit from within temp-directory. They’ll probably also automatically show up in any GUI desktop file management tool you might want to use.
Recovery using ddrescue
Didn’t work? Time to pull out the heavy guns. The data-recovery tool, ddrescue, copies files between file systems. But there’s something else it does at the same time that should put a smile on your face: it analyzes your files and attempts to repair any that are broken. No tool is guaranteed to fix everything that’s gone sour, but by reputation, ddrescue is as good as you’re going to get.
If it’s not already installed on the live boot you’re using, ddrescue comes as part of SystemRescue by default. Grab it from your repository. Then identify the troubled partition (/dev/sdc1 in this example), the partition where you want the image saved, and the name and location of a log file where a record of operation events can be written. Having a log file allows ddrescue to continue a stopped operation rather than starting again from scratch. My example uses an external drive (of greater capacity than the source drive) mounted to a directory called /run/usb-mount:
# apt install gddrescue 1 # ddrescue -d /dev/sdc1 /run/usb-mount/sdc1-backup.img /run/usb-mount/sdc1-backup.logfile 2
- 1 Note: for CentOS, the command would be yum install ddrescue.
- 2 Tells ddrescue to ignore the kernel cache and access the disk directly
You can test the rescue by using dd to write the backup image to a new, empty drive (called /dev/sdd/ in my example) and then booting the system to the new drive:
# dd if=backup.img of=/dev/sdd
Even if you find that you still can’t boot the partition, it may now be possible to use it to reliably access important individual files. Either way, you’re ahead of where you would have been.
File recovery using PhotoRec
Another tool that can help you grab files off a damaged drive is PhotoRec. One look at the interface tells you that it must have come from the same place as TestDisk. And, in fact, both were created and are maintained by Christophe Grenier of CGSecurity. To load the program, type
# photorec
Once you identify the file system that needs rescuing, which file types you want to include, and the location to which you’d like the files saved, the files are saved to numbered directories using the recup_dir.? prefix:
$ ls recup_dir.12 f1092680.elf f0668624.png f0853304.xml f0859464.xml f0867192.txt f0977016.gz f0982184.xml [...]
Something to keep in mind about PhotoRec is that the files are all given new numbered filenames while retaining their original file extensions. This can sometimes make finding individual files difficult, but it sure beats losing them altogether.
6.4. Password recovery: Mounting a file system using chroot
You know that the passwords chosen by the people you support are probably not strong enough to protect your infrastructure against a serious attack. And even the few exceptions to the rule are probably being reused on multiple servers and accounts. You beg and nag and beg and nag, but it seems like a losing battle. All is not entirely lost.
The problem of keeping track of sufficiently complex passwords can be largely solved by using a good password vault like KeePass2 or LastPass. And the problem of overusing passwords can be at least blunted by implementing a single sign-on solution such as Kerberos. Still, uh-ohs are always going to happen.
What happens to the one or two users who care enough to dream up good, strong passwords for each server they access? Every now and then they forget a password. That won’t be a problem if there’s another admin with sudo power who can log in to the server and run passwd to create a new password for the user:
# passwd username [sudo] password for yourname: Enter new UNIX password: Retype new UNIX password: passwd: password updated successfully
But if your unlucky and forgetful user was the only admin with an account on that machine, you’ve got trouble. Except that you don’t. The grandfather of all Linux virtualization, chroot, is going to save your day. Here’s one way that it might work.
Use a live-boot drive to power up the server that’s got you locked out, run lsblk to determine the designation of your root partition on the server’s hard disk, and mount the root partition to a temporary directory:
# mkdir /run/mountdir/ # mount /dev/sdb1 /run/mountdir/
Then whisper the magic words, and you’re in:
# chroot /run/mountdir/ root@ubuntu:/#
That’s all it takes. At this point, you’re free to run commands as though you were working on a running version of the physical hard drive. Use passwd to give your admin a new password to replace the lost one. And, after typing exit to shut down your chroot session, reboot the machine (without the live-boot USB). Everything should now be fine.
Encrypting the data on your storage drives using a tool like ecryptfs or dm-crypt makes it a great deal less likely that your data will be compromised. But on the other hand, many of the rescue and recovery operations discussed in this chapter won’t work on an encrypted volume.
Striking a balance between security and accessibility isn’t an exact science. Many admins leave local servers and desktop workstations unencrypted because those are at least protected by locked office doors, but they insist that mobile devices be encrypted.
Summary
- Linux recovery modes provide access to administration tools useful for repairing systems that won’t boot normally.
- Live-boot drives allow you to boot your choice of Linux distros independently of the file systems on a computer’s physical drives.
- Purpose-build distros like SystemRescueCd are lightweight versions of Linux that come preinstalled with a full range of rescue tools.
- Damaged partitions can sometimes be restored using tools like TestDisk.
- Data from damaged partitions can sometimes be recovered using tools like ddrescue and PhotoRec.
- File systems can be mounted and administered using a virtual process called chroot.
Key terms
- GRUB is a bootloader that manages the images to be used in the Linux boot process.
- A hash (checksum) is a cryptographically generated value that can be checked against a master copy to confirm an image’s authenticity.
- A partition’s Master Boot Record (MBR) for a CD/DVD will be different than for a USB, requiring special attention for creating live-boot USBs.
- The tool chroot opens virtual root shells within mounted file systems.
Security best practices
- Always confirm that downloaded images are authentic by checking their hashes. And avoid downloading from unencrypted (HTTP rather than HTTPS) websites.
- Carefully consider whether or not to encrypt the data at rest on your system drives, balancing accessibility against security.
- Enforce the use of password vaults and single sign-on services for users of your infrastructure.
Command-line review
- sha256sum systemrescuecd-x86-5.0.2.iso calculates the SHA256 checksum of a .ISO file.
- isohybrid systemrescuecd-x86-5.0.2.iso adds a USB-friendly MBR to a live-boot image.
- dd bs=4M if=systemrescuecd-x86-5.0.2.iso of=/dev/sdb && sync writes a live-boot image to an empty drive.
- mount /dev/sdc1 /run/temp-directory mounts a partition to a directory on the live file system.
- ddrescue -d /dev/sdc1 /run/usb-mount/sdc1-backup.img /run/usb-mount/ sdc1-backup.logfile saves files on a damaged partition to an image named sdc1-backup.img and writes events to a log file.
- chroot /run/mountdir/ opens a root shell on a file system.
Test yourself
Which of the following will allow you to access a damaged Linux machine’s own recovery mode?
- SystemRecovery
- GRUB
- Hitting CRTL+Right Shift during boot
- chkdsk
Which user account is used by default for a Linux recovery mode shell session?
- root
- The initial account created during installation
- admin
- guest
What is the goal of the Clean option in the Ubuntu Recovery Mode menu?
- Find and remove viruses
- Remove unused program files
- Remove unused user accounts
- Clean up a session and shut down
Booting a Linux live-boot device will do which of the following?
- Launch a system recovery environment
- Launch a file recovery environment
- Launch a Linux session using host machine partitions
- Launch a Linux session using only live-boot partitions
What tool do you need in order to create a live-boot image that can be used to boot from a USB device?
- ddrescue
- GRUB
- isohybrid
- ecryptfs
The sha256sum program is used for what purpose?
- Opening a virtual shell session on a mounted file system
- Managing Linux images at system startup
- Generating hashes to authenticate file integrity
- Managing file recovery from damaged drives
Which of the following will write a SystemRescue image to the second block device on your system?
- dd if=systemrescuecd-x86-5.0.2.iso of=/dev/sdb
- dd if=/dev/sdb of=systemrescuecd-x86-5.0.2.iso
- dd if=/dev/sdc of=systemrescuecd-x86-5.0.2.iso
- dd if=systemrescuecd-x86-5.0.2.iso of=/dev/sdb2
What does the chroot command do?
- Creates a live-boot image that can be used to boot from a USB device
- Opens a virtual shell session on a mounted file system
- Generates hashes to authenticate file integrity
- Removes unused program files
Answer key
b
a
b
d
c
c
a
b