All servers in a replica set maintain regular communication with every other member via a heartbeat. The heartbeat is a small packet sent regularly to verify that all members are operating normally.
Secondary members also communicate with the primary to get the latest updates from the oplog and apply them to their own data.
Schematically, we can see how this works:
When the primary member goes down, all secondaries will miss a heartbeat or more. They will be waiting up until settings.electionTimeoutMillis time passes (default 10 seconds) and then the secondaries start one or more rounds of elections to find the new primary.
For a server to be elected as primary from the secondaries it must have two properties:
- Belong in a group of voters that have 50%+1 of the votes
- Be the most up-to-date secondary in this group
In the simple example of three servers with one vote each, once we lose the primary, the other two servers will each have one vote, so in total two-thirds, and as such the one with the most up-to-date oplog will be elected primary.
Now consider a more complex setup:
- Seven servers (one primary, six secondaries)
- One vote each
We lose the primary and the six remaining servers have network connectivity issues resulting in a network partition:
Partition North:
- Three servers (one vote each)
Partition South:
- Three servers (one vote each)
Each partition doesn't have any knowledge of what happened to the rest of the servers. Now when they hold elections no partition can establish majority as they have three out of seven votes. No primary will get elected from either partition.
This problem can be overcome by having, for example, one server having three votes.
Now our overall cluster setup looks as follows:
- Server #1 - one vote
- Server #2 - one vote
- Server #3 - one vote
- Server #4 - one vote
- Server #5 - one vote
- Server #6 - one vote
- Server #7 - three votes
After losing Server #1, now our partitions look as follows:
Partition North:
- Server #2 - one vote
- Server #3 - one vote
- Server #4 - one vote
Partition South:
- Server #5 - one vote
- Server #6 - one vote
- Server #7 - three votes
Partition South has three servers with a total of five out of nine votes. The secondary among servers #5 ,#6, and #7 that is most up to date according to its oplog entries will be elected as primary.