In the previous chapter, you learned how to troubleshoot common Ceph problems, which, although they may be affecting the operation of the cluster, weren't likely to cause a total outage or data loss. This chapter will cover more serious scenarios, where the Ceph cluster is down or unresponsive. It will also cover various techniques to recover from data loss. It is to be understood that these techniques are more than capable of causing severe data loss themselves and should only be attempted as a last resort. If you have a support contract with your Ceph vendor or have a relationship with Red Hat, it is highly advisable to consult them first before carrying out any of the recovery techniques listed in this chapter.

In this chapter, we will cover the following topics:

• Avoiding data loss
• Using RBD mirroring to provide highly available block storage
• Investigating asserts
• Rebuilding monitor databases from OSDs
• Extracting PGs from a dead OSD
• Examining data from an offline Bluestore OSD
• Recovering from lost objects or inactive PGs
• Recovering from a failed CephFS filesystem
• Rebuilding an RBD from dead OSDs