In the event that a number of OSDs have failed, and you are unable to recover them via the ceph-object-store tool, your cluster will most likely be in a state where most, if not all, RBD images are inaccessible. However, there is still a chance that you may be able to recover RBD data from the disks in your Ceph cluster. There are tools that can search through the OSD data structure, find the object files relating to RBDs, and then assemble these objects back into a disk image, resembling the original RBD image.
In this section, we will focus on a tool by Lennart Bader to recover a test RBD image from our test Ceph cluster. The tool allows the recovery of RBD images from the contents of Ceph OSDs, without any requirement that the OSD is in a running or usable state. It should be noted that if the OSD has been corrupted due to an underlying file system corruption, the contents of the RBD image may still be corrupt. The RBD recovery tool can be found in the following github repository:
https://gitlab.lbader.de/kryptur/ceph-recovery
Before we start, make sure you have a small test RBD with a valid file system created on your Ceph cluster. Due to the size of the disks in the test environment that we created in Chapter 2, Deploying Ceph, it is recommended that the RBD is only a gigabyte in size.
We will perform the recovery on one of the monitor nodes, but in practice, this recovery procedure can be done from any node that can access the Ceph OSD disks. To access the disks, we need to make sure that the recovery server has sufficient space to recover the data.
In this example, we will mount the remote OSDs contents via sshfs, which allows you to mount remote directories over ssh. However in real life, there is nothing to stop you from physically inserting disks into another server or whatever method is required. The tool only requires to see the OSDs data directories:
- First, we need to clone the Ceph recovery tool from the Git repository.
git clone https://gitlab.lbader.de/kryptur/ceph-recovery.git
- Also, make sure you have sshfs installed:
sudo apt-get install sshfs
- Change into the cloned tool directory, and create the empty directories for each of the OSDs:
cd ceph-recovery
sudo mkdir osds
sudo mkdir osds/ceph-0
sudo mkdir osds/ceph-1
sudo mkdir osds/ceph-2
Now, mount each remote OSD to the directories that we have just created. Note that you need to make sure your OSD directories match your actual test cluster:
sudo sshfs vagrant@osd1:/var/lib/ceph/osd/ceph-0 osds/ceph-0
sudo sshfs vagrant@osd2:/var/lib/ceph/osd/ceph-2 osds/ceph-2
sudo sshfs vagrant@osd3:/var/lib/ceph/osd/ceph-1 osds/ceph-1
Now, we can use the tool to scan the OSD directories and compile a list of the RBDs that are available. The only parameter needed for this command is the location where the OSDs are mounted. In this case, it is in a directory called osds. The results will be listed in the VM directory:
sudo ./collect_files.sh osds
If we look inside the VM directory, we can see that the tool has found our test RBD image. Now that we have located the image, the next step is to assemble various objects located on the OSDs. The three parameters for this command are the name of the RBD image found in the previous step, the size of the image, and the destination for the recovered image file. The size of the image is specified in bytes, and it is important that it is at least as big as the original image; it can be bigger, but the RBD will not recover if the size is smaller:
sudo ./assemble.sh vms/test.id 1073741824 .
The RBD will now be recovered from the mounted OSD contents to the specified image file. Depending on the size of the image, it may take a while, and a progress bar will show you its progress.
Once completed, we can run a file system called fsck on the image to make sure that it has been recovered correctly. In this case, the RBD was formatted with ext4, so we can use the e2fsck tool to check the image:
sudo e2fsck test.raw
Excellent, the image file is clean, which means that there is now a very high chance that all our data has been recovered successfully.
Now, we can finally mount the image as a loop-back device to access our data. If the command returns no output, then we have successfully mounted it:
sudo mount -o loop test.raw /mnt
You can see that the image is successfully mounted as a loop device:
This concludes the process for recovering RBD images from dead Ceph OSDs.