If we lose the entire cluster, we can't do anything other than get it back up and running as soon as possible.

It's important to have monitoring and a proper process in place to understand what needs to be done, when and by who should this ever happen.

Recovering when the entire cluster goes down is essentially restoring from backups and setting up new shards, which is complicated and will take time.

Dry testing this on staging environment is also advisable, and so is investing in regular backups via MongoDB Ops Manager or any other backup solution.