Since the first chapter, we’ve been using a standalone server, a single mongod server. It’s an easy way to get started but a dangerous way to run in production: what if your server crashes or becomes unavailable? Your database will at least be unavailable for a little while. If there are problems with the hardware, you may have to move your data to another machine. In the worst case, disk or network issues could leave you with corrupt or inaccessible data.

Replication is a way of keeping identical copies of your data on multiple servers and is recommended for all production deployments. Replication keeps your application running and your data safe, even if something happens to one or more of your servers.

With MongoDB, you set up replication by creating a replica set. A replica set is a group of servers with one primary, the server taking client requests, and multiple secondaries, servers that keep copies of the primary’s data. If the primary crashes, the secondaries can elect a new primary from amongst themselves.

If you are using replication and a server goes down, you can still access your data from the other servers in the set. If the data on a server is damaged or inaccessible, you can make a new copy of the data from one the other members of the set.

This chapter introduces replica sets and covers how to set up replication on your system.

This section will get you started quickly by setting up a three-member replica set on your local machine. This setup is obviously not suitable for production, but it’s a nice way to familiarize yourself with replication and play around with configuration.

Start up a mongo shell with the --nodb option, which allows you to start a shell that is not connected to any mongod:

$ mongo --nodb

Create a replica set by running the following command:

> replicaSet = new ReplSetTest({"nodes" : 3})

This tells the shell to create a new replica set with three servers: one primary and two secondaries. However, it doesn’t actually start the mongod servers until you run the following two commands:

> // starts three mongod processes
> replicaSet.startSet()
>
> // configures replication
> replicaSet.initiate()

You should now have three mongod processes running locally on ports 31000, 31001, and 31002. They will all be dumping their logs into the current shell, which is very noisy, so put this shell aside and open up a new one.

In the second shell, connect to the mongod running on port 31000:

> conn1 = new Mongo("localhost:31000")
connection to localhost:31000
testReplSet:PRIMARY>
testReplSet:PRIMARY> primaryDB = conn1.getDB("test")
test

Notice that, when you connect to a replica set member, the prompt changes to testReplSet:PRIMARY>. "PRIMARY" is the state of the member and "testReplSet" is an identifier for this set. You’ll learn how to choose your own identifier later; testReplSet is the default name ReplSetTest uses.

Examples from now on will just use > for the prompt instead of testReplSet:PRIMARY> to keep things more readable.

Use your connection to the primary to run the isMaster command. This will show you the status of the set:

> primaryDB.isMaster()
{
    "setName" : "testReplSet",
    "ismaster" : true,
    "secondary" : false,
    "hosts" : [
        "wooster:31000",
        "wooster:31002",
        "wooster:31001"
    ],
    "primary" : "wooster:31000",
    "me" : "wooster:31000",
    "maxBsonObjectSize" : 16777216,
    "localTime" : ISODate("2012-09-28T15:48:11.025Z"),
    "ok" : 1
}

There are a bunch of fields in the output from isMaster, but the important ones indicate that you can see that this node is primary (the "ismaster" : true field) and that there is a list of hosts in the set.

Now that you’re connected to the primary, let’s try doing some writes and see what happens. First, insert 1,000 documents:

> for (i=0; i<1000; i++) { primaryDB.coll.insert({count: i}) }
>
> // make sure the docs are there
> primaryDB.coll.count()
1000

Now check one of the secondaries and verify that they have a copy of all of these documents. Connect to either of the secondaries:

> conn2 = new Mongo("localhost:31001")
connection to localhost:31001
> secondaryDB = conn2.getDB("test")
test

Secondaries may fall behind the primary (or lag) and not have the most current writes, so secondaries will refuse read requests by default to prevent applications from accidentally reading stale data. Thus, if you attempt to query a secondary, you’ll get an error that it’s not primary:

> secondaryDB.coll.find()
error: { "$err" : "not master and slaveok=false", "code" : 13435 }

This is to protect your application from accidentally connecting to a secondary and reading stale data. To allow queries on the secondary, we set an “I’m okay with reading from secondaries” flag, like so:

> conn2.setSlaveOk()

Note that slaveOk is set on the connection (conn2), not the database (secondaryDB).

Now you’re all set to read from this member. Query it normally:

> secondaryDB.coll.find()
{ "_id" : ObjectId("5037cac65f3257931833902b"), "count" : 0 }
{ "_id" : ObjectId("5037cac65f3257931833902c"), "count" : 1 }
{ "_id" : ObjectId("5037cac65f3257931833902d"), "count" : 2 }
...
{ "_id" : ObjectId("5037cac65f3257931833903c"), "count" : 17 }
{ "_id" : ObjectId("5037cac65f3257931833903d"), "count" : 18 }
{ "_id" : ObjectId("5037cac65f3257931833903e"), "count" : 19 }
Type "it" for more
>
> secondaryDB.coll.count()
1000

You can see that all of our documents are there.

Now, try to write to a secondary:

> secondaryDB.coll.insert({"count" : 1001})
> secondaryDB.runCommand({"getLastError" : 1})
{
    "err" : "not master",
    "code" : 10058,
    "n" : 0,
    "lastOp" : Timestamp(0, 0),
    "connectionId" : 5,
    "ok" : 1
}

You can see that the secondary does not accept the write. The secondary will only perform writes that it gets through replication, not from clients.

There is one other interesting feature that you should try out: automatic failover. If the primary goes down, one of the secondaries will automatically be elected primary. To try this out, stop the primary:

> primaryDB.adminCommand({"shutdown" : 1})

Run isMaster on the secondary to see who has become the new primary:

> secondaryDB.isMaster()

It should look something like this:

{
    "setName" : "testReplSet",
    "ismaster" : true,
    "secondary" : false,
    "hosts" : [
        "wooster:31001",
        "wooster:31000",
        "wooster:31002"
    ],
    "primary" : "wooster:31001",
    "me" : "wooster:31001",
    "maxBsonObjectSize" : 16777216,
    "localTime" : ISODate("2012-09-28T16:52:07.975Z"),
    "ok" : 1
}

Your primary may be the other server; whichever secondary noticed that the primary was down first will be elected. Now you can send writes to the new primary.

isMaster is a very old command, predating replica sets to when MongoDB only supported master-slave replication. Thus, it does not use the replica set terminology consistently: it still calls the primary a “master.” You can generally think of “master” as equivalent to “primary” and “slave” as equivalent to “secondary.”

When you’re done working with the set, shut down the servers from your first shell. This shell will be full of log output from the members of the set, so hit Enter a few times to get back to a prompt. To shutdown the set, run:

> replicaSet.stopSet()

Congratulations! You just set up, used, and tore down replication.

There are a few key concepts to remember:

Now that you understand the basics, the rest of this chapter focuses on configuring a replica set under more realistic circumstances. Remember that you can always go back to ReplSetTest if you want to quickly try out a configuration or option.

For actual deployments, you’ll need to set up replication across multiple machines. This section takes you through setting up a real replica set that could be used by your application.

Let’s say that you already have a standalone mongod on server-1:27017 with some data on it. (If you do not have any pre-existing data, this will work the same way, just with an empty data directory.) The first thing you need to do is choose a name for your set. Any string whatsoever will do, so long as it’s UTF-8.

Once you have a name for your replica set, restart server-1 with the --replSet name option. For example:

$ mongod --replSet spock -f mongod.conf --fork

Now start up two more mongod servers with the replSet option and the same identifier (spock): these will be the other members of the set:

$ ssh server-2
server-2$ mongod --replSet spock -f mongod.conf --fork
server-2$ exit
$
$ ssh server-3
server-3$ mongod --replSet spock -f mongod.conf --fork
server-3$ exit

Each of the other members should have an empty data directory, even if the first member had data. They will automatically clone the first member’s data to their machines once they have been added to the set.

For each member, add the replSet option to its mongod.conf file so that it will be used on startup from now on.

Once you’ve started the mongods, you should have three mongods running on three separate servers. However, each mongod does not yet know that the others exist. To tell them about one another, you have to create a configuration that lists each of the members and send this configuration to server-1. It will take care of propagating it to the other members.

First we’ll create the configuration. In the shell, create a document that looks like this:

> config = {
    "_id" : "spock",
    "members" : [
        {"_id" : 0, "host" : "server-1:27017"},
        {"_id" : 1, "host" : "server-2:27017"},
        {"_id" : 2, "host" : "server-3:27017"}
    ]
}

There are several important parts of config. The config’s "_id" is the name of the set that you passed in on the command line (in this example, "spock"). Make sure that this name matches exactly.

The next part of the document is an array of members of the set. Each of these needs two fields: a unique "_id" that is an integer and a hostname (replace the hostnames with whatever your servers are called).

This config object is your replica set configuration, so now you have to send it to a member of the set. To do so, connect to the server with data on it (server-1:27017) and initiate the set with this configuration:

> // connect to server-1
> db = (new Mongo("server-1:27017")).getDB("test")
>
> // initiate replica set
> rs.initiate(config)
{
    "info" : "Config now saved locally.  Should come online in about a minute.",
    "ok" : 1
}

server-1 will parse the configuration and send messages to the other members, alerting them of the new configuration. Once they have all loaded the configuration, they will elect a primary and start handling reads and writes.

If you are starting a brand-new set, you can send the configuration to any member in the set. If you are starting with data on one of the members, you must send the configuration to the member with data. You cannot initiate a set with data on more than one member.

> db.adminCommand({"replSetInitiate" : config})

Replica set configurations can be changed at any time: members can be added, removed, or modified. There are shell helpers for some common operations; for example, to add a new member to the set, you can use rs.add:

> rs.add("server-4:27017")

Similarly, you can remove members;

> rs.remove("server-1:27017")
Fri Sep 28 16:44:46 DBClientCursor::init call() failed
Fri Sep 28 16:44:46 query failed : admin.$cmd { replSetReconfig: { 
    _id: "testReplSet", version: 2, members: [ { _id: 0, host: "ubuntu:31000" }, 
   { _id: 2, host: "ubuntu:31002" } ] } } to: localhost:31000
Fri Sep 28 16:44:46 Error: error doing query: 
    failed src/mongo/shell/collection.js:155
Fri Sep 28 16:44:46 trying reconnect to localhost:31000
Fri Sep 28 16:44:46 reconnect localhost:31000 ok

Note that when you remove a member (or do almost any configuration change other than adding a member), you will get a big, ugly error about not being able to connect to the database in the shell. This is okay; it actually means the reconfiguration succeeded! When you reconfigure a set, the primary closes all connections as the last step in the reconfiguration process. Thus, the shell will briefly be disconnected but will automatically reconnect on your next operation.

The reason that the primary closes all connections is that it briefly steps down whenever you reconfigure the set. It should step up again immediately, but be aware that your set will not have a primary for a moment or two after reconfiguring.

You can check that a reconfiguration succeeded by run rs.config() in the shell. It will print the current configuration:

> rs.config()
{
    "_id" : "testReplSet",
    "version" : 2,
    "members" : [
        {
            "_id" : 1,
            "host" : "server-2:27017"
        },
        {
            "_id" : 2,
            "host" : "server-3:27017"
        },
        {
            "_id" : 3,
            "host" : "server-4:27017"
        }
    ]
}

Each time you change the configuration, the "version" field will increase. It starts at version 1.

You can also modify existing members, not just add and remove them. To make modifications, create the configuration document that you want in the shell and call rs.reconfig. For example, suppose we have a configuration such as the one shown here:

> rs.config()
{
    "_id" : "testReplSet",
    "version" : 2,
    "members" : [
        {
            "_id" : 0,
            "host" : "server-1:27017"
        },
        {
            "_id" : 1,
            "host" : "10.1.1.123:27017"
        },
        {
            "_id" : 2,
            "host" : "server-3:27017"
        }
    ]
}

Someone accidentally added member 1 by IP, instead of its hostname. To change that, first we load the current configuration in the shell and then we change the relevant fields:

> var config = rs.config()
> config.members[1].host = "server-2:27017"

Now that the config document is correct, we need to send it to the database using the rs.reconfig helper:

> rs.reconfig(config)

rs.reconfig is often more useful that rs.add and rs.remove for complex operations, such as modifying members’ configuration or adding/removing multiple members at once. You can use it to make any legal configuration change you need: simply create the config document that represents your desired configuration and pass it to rs.reconfig.

To plan out your set, there are certain replica set concepts that you must be familiar with. The next chapter goes into more detail about these, but the most important is that replica sets are all about majorities: you need a majority of members to elect a primary, a primary can only stay primary so long as it can reach a majority, and a write is safe when it’s been replicated to a majority. This majority is defined to be “more than half of all members in the set,” as shown in Table 9-1.

Note that it doesn’t matter how many members are down or unavailable, as majority is based on the set’s configuration.

For example, suppose that we have a five-member set and three members go down, as shown in Figure 9-1. There are still two members up. These two members cannot reach a majority of the set (at least three members), so they cannot elect a primary. If one of them were primary, it would step down as soon as it noticed that it could not reach a majority. After a few seconds, your set would consist of two secondaries and three unreachable members.

Figure 9-1. With a minority of the set available, all members will be secondaries

Many users find this frustrating: why can’t the two remaining members elect a primary? The problem is that it’s possible that the other three members didn’t go down, and that it was the network that went down, as shown in Figure 9-2. In this case, the three members on the left will elect a primary, since they can reach a majority of the set (three members out of five).

In the case of a network partition, we do not want both sides of the partition to elect a primary: otherwise the set would have two primaries. Then both primaries would be writing to the data and the data sets would diverge. Requiring a majority to elect or stay primary is a neat way of avoiding ending up with more than one primary.

Figure 9-2. For the members, a network partition looks identical to servers on the other side of the partition going down

It is important to configure your set in such a way that you’ll usually be able to have one primary. For example, in the five-member set described above, if members 1, 2, and 3 are in one data center and members 4 and 5 are in another, there should almost always be a majority available in the first data center (it’s more likely to have a network break between data centers than within them).

One common setup that usually isn’t what you want is a two member set: one primary and one secondary. Suppose one member becomes unavailable: the other member cannot see it, as shown in Figure 9-3. In this situation, neither side of the network partition has a majority so you’ll end up with two secondaries. For this reason, this type of configuration is not generally recommended.

Figure 9-3. With an even number of members, neither side of a partition has a majority

There are a couple of configurations that are recommended:

More complex requirements might require different configurations, but you should keep in mind how your set will acquire a majority under adverse conditions.

All of these complexities would disappear if MongoDB supported having more than one primary. However, multimaster would bring its own host of complexities. With two primaries, you would have to handle conflicting writes (for example, someone updates a document on one primary and someone deletes it on another primary). There are two popular ways of handling conflicts in systems that support multiple writers: manual reconciliation or having the system arbitrarily pick a “winner.” Neither of these options is a very easy model for developers to code against, seeing that you can’t be sure that the data you’ve written won’t change out from under you. Thus, MongoDB chose to only support having a single primary. This makes development easier but can result in periods when the replica set is read-only.

Wed Jun 20 17:44:02 [rsMgr] replSet info electSelf 1
Wed Jun 20 17:44:02 [rsMgr] replSet couldn't elect self, only received -9999 votes

The replica sets we have set up so far have been fairly uniform in that every member has the same configuration as every other member. However, there are many situations when you don’t want members to be identical: you might want one member to preferentially be primary or make a member invisible to clients so that no read requests can be routed to it. These and many other configuration options can be specified in the member subdocuments of the replica set configuration. This section outlines the member options that you can set.

> rs.addArb("server-5:27017")

> rs.add({"_id" : 4, "host" : "server-5:27017", "arbiterOnly" : true})

> rs.add({"_id" : 4, "host" : "server-4:27017", "priority" : 1.5})

> rs.isMaster()
{
    ...
    "hosts" : [
        "server-1:27107",
        "server-2:27017",
        "server-3:27017"
    ],
    ...
}

> var config = rs.config()
> config.members[2].hidden = 0
0
> config.members[2].priority = 0
0
> rs.reconfig(config)

> rs.isMaster()
{
    ...
    "hosts" : [
        "server-1:27107",
        "server-2:27017"
    ],
    ...
}