By default, every replica in RethinkDB is created as a voting replica. That means those replicas will take part in the failover process to perform the selection of the next primary replica. You can also change this option using the reconfigure() command.
Automatic failover requires at least three server clusters with three replicas for table. Two server clusters will not be covered under the automatic failover process and the system may go down during the failure of any RethinkDB instance.
In such cases-where RethinkDB cannot perform failover-you need to do it manually using the reconfigure() command, by passing the emergency repair mode key.
Upon running emergency repair mode, each of the shards is first examined and classified into three categories:If more than half of the total shards are available, it will return a healthy status
- Repairable: In the case of repairable, the shard is not healthy but has one replica, which can be used
- Beyond repair: In the case of beyond repair, the shard has no available replica and cannot be used
For each and every shard that can be repaired, RethinkDB will first change all the offline replicas into non-voting replicas. If there is no voting replica available, RethinkDB will choose one non-voting replica and forcefully convert it into a voting replica.
You can specify two options along with the emergency repair option:
- unsafe_rollback: This will leave those shards that are beyond repair during the failover process
- unsafe_rollback_or_erase: This will delete those shards that are beyond repair and create one on the available server that holds another shard for that table
Here is the command reference:
r.table(users).reconfigure( {emergencyRepair: "unsafe_rollback_or_erase"} ).run(conn, callback);
Please note that emergency failover handling is very critical and should be done by a skilled person, or else you may end up losing all your data.