In the previous chapter, you learned how to troubleshoot common Ceph problems, which, although may be affecting the operation of the cluster, weren't likely to cause a total outage or data loss. This chapter will cover more serious scenarios where the Ceph cluster is down or unresponsive. It will also cover various techniques to recover from data loss. It is to be understood that these techniques are more than capable of causing severe data loss themselves and should only be attempted as a last resort. If you have a support contract with your Ceph vendor or have a relationship with Red Hat, it is highly advisable to consult them first before carrying out any of the recovery techniques listed in this chapter.

In this chapter, you will learn the following:

• How to avoid data loss
• How to use RBD mirroring to provide highly available block storage
• How to investigate asserts
• How to rebuild monitor dbs from OSDs
• How to extract PGs from a dead OSD
• How to recover from lost objects or inactive PGs
• How to rebuild a RBD from dead OSDs