Improving Scalability

For web applications in particular, scalability is a critical design requirement, especially those that rely heavily on backend databases. Replication can help you create more scalable applications in two ways:

• Improve redundancy: Replication can help you improve redundancy by enabling you to host an application in several data centers. In this approach, you ensure that there is a local copy of the data in each data center, so that the application can have high-speed access to it. Users can then be connected to the data center that is closest to them, minimizing latency.
• Improve performance: Replication can under certain circumstances, help you improve an application’s raw performance. This is particularly true for cases where you have a large web application with a predominantly read-based dataset, and you want to distribute queries to multiple database servers to increase parallelism. Or with query loads that have widely different working sets such as reporting or aggregation.

Improving Durability/Reliability

Replication is commonly used to help guard against hardware failure or database corruption and allows flexibility when performing backups or other potentially high-impact maintenance activity with little or no impact – as these tasks can be performed individually on members of the set without impacting the set as a whole. Some specific examples where people use replication in this manner include the following:

• When you want to have a duplicate of your database that is running at a delay. You may want to protect yourself against flaws in your application or provide a simple mechanism to provide trend information by highlighting the differences between the results of queries against both datasets. This can also provide a safety buffer for human errors and avoids the need to fully restore from backup.
• When you want a backup system in case of a failure. You may want to run a replica as a backup in cases where normal backup schemes would take too long to restore in the event of a system failure.
• When you want a redundant system for administrative purposes. You may want to run a replica so you can rotate between nodes for administrative tasks such as backups or upgrades.

Providing Isolation

There are some processes that, if run against the production database, would significantly impact that database’s performance or availability. You can use replication to create synchronized copies that isolate processes from the production database, for example:

• When you want to run reports or backups without impacting the performance of your production system: Maintaining a hidden secondary replica enables you to isolate queries from your reporting system and make sure that the end-of-month reports don’t delay or otherwise impact your normal operations.

What Is a Primary?

In replica set terms, a primary is the source of truth at a given moment for your replica set. It is the only node in the set to which data can be written and from which all other nodes replicate their data. A primary is elected by agreement of a majority of all voting members, known as a quorum.

Once a primary is elected, all secondaries will use it as the source of truth for their replication, and as such all writes must be directed to this member.

What Is a Secondary?

A secondary member is a data-carrying nonprimary member, which can (barring a few exceptions) theoretically become a primary. It is a node, which can be read from and which replicates its data from the primary of its set in as close to real-time as possible. By default, if you directly connect to a secondary without any read preference, you cannot perform read operations. This is done to reinforce that with any read to a nonprimary, if there is a delay in replication, you may be reading from older data. You can use the command rs.slaveOk()to set the current connection you are on to read from secondaries. Or, if you are using a driver, you can set a read preference, as we will discuss later in this chapter.

What Is an Arbiter?

An arbiter is a non–data-bearing node that is used to provide an additional vote to help maintain a majority for replica set elections. It does not cast a deciding vote or direct which node is the primary, but participates and can be a member of a quorum to decide a primary. Arbiters are best used to help avoid the “split brain” problem described earlier. Consider the diagram shown in Figure 11-2. With the addition of an arbiter to Site A, we can always have one side that can create a majority. This means in the event of the network break, we don’t wind up with two primaries! We can add further redundancy by having an arbiter in a third Site C. This way, if Site A goes down we can still form a majority from the nodes in Sites B and C. Using a third site like this we can always continue in the event we lose connectivity to any one site.

Creating a Replica Set

The best way to learn how to create a replica set is by looking at an example. In the example that follows, you will create a replica set called testset. This set will have three members (two active and one passive). Table 11-1 lists the members of this set.

Replica sets will allow you to use localhost as an identifier, but only when all machines are co-located on one server. This occurs as each member of a replica set must be able to contact all other MongoDB instances via hostname for replication to function.

Normally when working with replica sets we use hostnames; you can find the current hostname by using the hostname command, as in the following example:

$hostname
Pixl.local

In the examples that follow, replace the term [hostname] with whatever value is returned by running the hostname command on your own system.

Getting a Replica Set Member Up and Running

The first step is to get the first active member up and running. To do so, open a terminal window and type the following:

$ mkdir -p /db/active1/data
$ mongod --dbpath /db/active1/data --port 27021 --replSet testset

The --replSet option tells the instance the name of the replica set it is joining. This is the first member of the replica set, so you can give it the address of any other member, even if that member has not been started up yet. Only one member address is required, but you can also provide the names of other members by separating their addresses with commas, as shown in the following example:

$ mongod --dbpath /db/active1/data --port 27021 --replSet testset

To keep things simple, this example will rely on only one address. The next step is to get the other members up and running. Do so by opening two more terminal windows, and then type the following into the first window to get the second member up and running:

$ mkdir -p /db/active2/data
$ mongod --dbpath /db/active2/data --port 27022  --replSet testset

Next, type this into the second window to get the final (passive) member up and running:

$ mkdir -p /db/passive1/data
$ mongod --dbpath /db/passive1/data --port 27023 --replSet testset

At this point, you have three server instances running and communicating with each other; however, you do not quite have your replica set running, because you haven’t yet initialized the replica set and instructed each member about its role and responsibilities.

To do that, you need to attach to one of the servers and initialize the replica set. The following code selects the first server to attach to:

$mongo [hostname
]:27021

Next, you need to initialize the first member of this set to create its oplog and a default configuration document. You can see the MongoD instance suggesting that you need to do this in your logfiles:

Mon Jun  3 21:25:23.712 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Jun  3 21:25:23.712 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done

So run the rs.initiate command:

> rs.initiate()
{
     "info2" : "no configuration explicitly specified -- making one",
     "me" : "[hostname]:27021",
     "info" : "Config now saved locally. Should come online in about a minute.",
     "ok" : 1
}

Finally, you should check the status of the replica set to determine whether it has been set up correctly:

>rs.status()
{
     "set" : "testset",
     "date" : ISODate("2013-06-03T11:28:58Z"),
     "myState" : 1,
     "members" : [
          {
               "_id" : 0,
               "name" : "[hostname]:27021",
               "health" : 1,
               "state" : 1,
               "stateStr" : "PRIMARY",
               "uptime" : 264,
               "optime" : {
                    "t" : 1370258919,
                    "i" : 1
               },
               "optimeDate" : ISODate("2013-06-03T11:28:39Z"),
               "self" : true
          }
     ],
     "ok" : 1
}

The output here indicates that all is OK: you’ve successfully configured, set up, and initialized a new replica set. Remember that you should use the name of your own machine in place of [hostname] because neither “localhost” nor “127.0.0.1” will work.

Adding a Server to a Replica Set

Now that you have started your new replica set, you need to start adding members to it. Let’s start by adding your first secondary. You can do this by simply adding the rs.add() command and providing the hostname and port of this instance. To add that, connect to your primary and run the following:

$ mongo [hostname
]:27021
> rs.add("[hostname
]:27021")
{ "ok" : 1 }

You will need to wait a minute or two as this node brings itself online, creates its own oplog, and readies itself. You can monitor the progress with rs.status() while waiting till this node comes online as a secondary:

>use admin
>rs.status() {
     "set" : "testset",
     "date" : ISODate("2013-06-03T11:36:37Z"),
     "myState" : 1,
     "members" : [
          {
               "_id" : 0,
               "name" : "[hostname]:27021",
               "health" : 1,
               "state" : 1,
               "stateStr" : "PRIMARY",
               "uptime" : 723,
               "optime" : {
                    "t" : 1370259331,
                    "i" : 1
               },
               "optimeDate" : ISODate("2013-06-03T11:35:31Z"),
               "self" : true
          },
          {
               "_id" : 1,
               "name" : "[hostname]:27022",
               "health" : 1,
               "state" : 2,
               "stateStr" : "SECONDARY",
               "uptime" : 66,
               "optime" : {
                    "t" : 1370259331,
                    "i" : 1
               },
               "optimeDate" : ISODate("2013-06-03T11:35:31Z"),
               "lastHeartbeat" : ISODate("2013-06-03T11:36:35Z"),
               "lastHeartbeatRecv" : ISODate("2013-06-03T11:36:36Z"),
               "pingMs" : 0,
               "syncingTo" : "[hostname]:27021"
          }
     ],
     "ok" : 1
}

Now let’s make use of that third passive member. First add the member as usual with rs.add():

$ mongo [hostname]:27022
> rs.add("[hostname]:27022")
{ "ok" : 1 }

Now we need to make a copy of the config document and modify it. Run the following command to create a document called conf, which contains your current replica set configuration.

> conf = rs.conf()
{
     "_id" : "testset",
     "version" : 3,
     "members" : [
          {
               "_id" : 0,
               "host" : "[hostname]:27021"
          },
          {
               "_id" : 1,
               "host" : "[hostname]:27022"
          },
          {
               "_id" : 2,
               "host" : "[hostname]:27023"
          }
     ]
}

Now that your configuration document is loaded you need to modify it. We want to set the node to be hidden and have a priority of 0, so it will not be elected to be the primary. Notice that the document has a members array, which contains a document for each member of the replica set. You need to pick the member you wish to access using the array operator []. So to create a value of hidden : true for the third member, we need to update the array element at 2 (remember, arrays start at 0). Run the following:

> conf.members[2].hidden = true
true

Now we can set a priority value of 0 using the same commands:

> conf.members[2].priority = 0
0

You can output this config document by simply executing the name of the variable you placed the config document into:

> conf
{
     "_id" : "testset",
     "version" : 3,
     "members" : [
          {
               "_id" : 0,
               "host" : "[hostname]:27021"
          },
          {
               "_id" : 1,
               "host" : "[hostname]:27022"
          },
          {
               "_id" : 2,
               "host" : "[hostname]:27023",
               "hidden" : true,
               "priority" : 0
          }
     ]
}

As you can see, this member now has the hidden value set and a priority of 0. Now all we need to do is update the replica set configuration to use this document. We do this by issuing the rs.reconfig ( ) command with our new config document as an argument.

> rs.reconfig(conf)
Tue Jun  4 20:01:45.234 DBClientCursor::init call() failed
Tue Jun  4 20:01:45.235 trying reconnect to 127.0.0.1:27021
Tue Jun  4 20:01:45.235 reconnect 127.0.0.1:27021 ok
reconnected to server after rs command (which is normal)

Your whole replica set lost connectivity and then reconnected! This happens because of the reconfiguration. Any changes to a replica set will potentially cause a replica set to reconfigure itself and have a new election, which under most circumstances has the previous primary step back into its role. Now if we re-run the rs.conf() command you can see your new replica set configuration in action.

> rs.conf()
{
     "_id" : "testset",
     "version" : 4,
     "members" : [
          {
               "_id" : 0,
               "host" : "[hostname]:27021"
          },
          {
               "_id" : 1,
               "host" : "[hostname]:27022"
          },
          {
               "_id" : 2,
               "host" : "[hostname]:27023",
               "priority" : 0,
               "hidden" : true
          }
     ]
}

Notice that this replica set configuration’s version number has now increased. This happens automatically as part of the reconfiguration to ensure that any replica set members don’t have the wrong configuration document.

You should now have a fully configured and working three-member replica set with an active primary and a hidden “passive.”

Adding an Arbiter

Adding an arbiter to be a voting member of your replica set is an easy process. Let’s start by spinning up a new member.

$ mkdir -p /db/arbiter1/data
$ mongod --dbpath /db/ arbiter1/data --port 27024  --replSet testset –rest

Now that you have a new member created, it’s just a matter of adding the new arbiter with the rs.addArb() command:

>rs.addArb("[hostname]:27024")
{ "ok" : 1 }

And if you now run rs.status(), you will see your arbiter in the output:

{
     "_id" : 3,
     "name" : "Pixl.local:27024",
     "health" : 1,
     "state" : 7,
     "stateStr" : "ARBITER",
     "uptime" : 721,
     "lastHeartbeat" : ISODate("2013-06-07T11:21:01Z"),
     "lastHeartbeatRecv" : ISODate("2013-06-07T11:21:00Z"),
     "pingMs" : 0
}

You may have recognized a problem here; we have four nodes now. That’s an even number, and even is bad! If we continue to run like this, your MongoDB nodes will start logging the following:

[rsMgr] replSet total number of votes is even - add arbiter or give one member an extra vote

To fix that, we know that we need to have an odd number of members; so one potential solution is to add another arbiter as the log message suggests, but that’s not strictly elegant as we add extra complexity that isn’t required. The best solution is to stop one of the existing nodes from voting and being viewed as a member of the quorum. We can do this by setting votes to zero for our hidden secondary. We do this in the same manner as we did to set the hidden and priority values.

conf = rs.conf()
conf.members[2].votes = 0
rs.reconfig(conf)

And that’s it. We have now set our passive node to be truly passive: it will never become primary; it is viewed as part of the replica set by clients; and it cannot participate in elections or be counted toward a majority. To test this, you can try shutting down the passive node, and the arbiter and your other two nodes will continue on primary unchanged; whereas before, the primary would have stepped down, citing that it could not see a majority of nodes.

Replica Set Chaining

You have seen that normally a member of a replica set will attempt to sync data from the primary of that set. But that is not the only place replica set secondaries can sync from; they can also sync-from other secondaries. In this manner your secondaries can form a “sync-chain,” in which each one syncs the latest data from other secondaries in your replica set.

Managing Replica Sets

MongoDB provides a number of commands for managing the configuration and status of a replica set. Table 11-2 shows the available commands that you can use to create, manipulate, and inspect the state of a cluster in a replica set.

The following sections take a closer look at some of the more commonly used commands listed in Table 11-2, providing additional details about what they do and how to use them.

>rs.status()
> rs.status();
{
     "set" : "testset",
     "date" : ISODate("2013-06-04T10:57:24Z"),
     "myState" : 1,
     "members" : [
          {
               "_id" : 0,
               "name" : "[hostname]:27021",
               "health" : 1,
               "state" : 1,
               "stateStr" : "PRIMARY",
               "uptime" : 4131,
               "optime" : Timestamp(1370340105, 1),
               "optimeDate" : ISODate("2013-06-04T10:01:45Z"),
               "self" : true
          },
          {
               "_id" : 1,
               "name" : "[hostname]:27022",
               "health" : 1,
               "state" : 2,
               "stateStr" : "SECONDARY",
               "uptime" : 3339,
               "optime" : Timestamp(1370340105, 1),
               "optimeDate" : ISODate("2013-06-04T10:01:45Z"),
               "lastHeartbeat" : ISODate("2013-06-04T10:57:23Z"),
               "lastHeartbeatRecv" : ISODate("2013-06-04T10:57:23Z"),
               "pingMs" : 0,
               "syncingTo" : "[hostname]:27021"
          },
          {
               "_id" : 2,
               "name" : "[hostname]:27023",
               "health" : 1,
               "state" : 2,
               "stateStr" : "SECONDARY",
               "uptime" : 3339,
               "optime" : Timestamp(1370340105, 1),
               "optimeDate" : ISODate("2013-06-04T10:01:45Z"),
               "lastHeartbeat" : ISODate("2013-06-04T10:57:22Z"),
               "lastHeartbeatRecv" : ISODate("2013-06-04T10:57:23Z"),
               "pingMs" : 0,
               "syncingTo" : "[hostname]:27021"
          }
     ],
     "ok" : 1
}

> rs.stepDown()
> rs.status()
{
     "set" : "testset",
     "date" : ISODate("2013-06-04T11:19:01Z"),
     "myState" : 2,
     "members" : [
          {
               "_id" : 0,
               "name" : "[hostname]:27021",
               "health" : 1,
               "state" : 2,
               "stateStr" : "SECONDARY",
               "uptime" : 5428,
               "optime" : Timestamp(1370340105, 1),
               "optimeDate" : ISODate("2013-06-04T10:01:45Z"),
               "self" : true
          },
          {
               "_id" : 1,
               "name" : "[hostname]:27022",
               "health" : 1,
               "state" : 2,
               "stateStr" : "SECONDARY",
               "uptime" : 4636,
               "optime" : Timestamp(1370340105, 1),
               "optimeDate" : ISODate("2013-06-04T10:01:45Z"),
               "lastHeartbeat" : ISODate("2013-06-04T11:19:00Z"),
               "lastHeartbeatRecv" : ISODate("2013-06-04T11:19:00Z"),
               "pingMs" : 0,
               "syncingTo" : "[hostname]:27021"
          },
          {
               "_id" : 2,
               "name" : "[hostname]:27023",
               "health" : 1,
               "state" : 2,
               "stateStr" : "SECONDARY",
               "uptime" : 4636,
               "optime" : Timestamp(1370340105, 1),
               "optimeDate" : ISODate("2013-06-04T10:01:45Z"),
               "lastHeartbeat" : ISODate("2013-06-04T11:19:01Z"),
               "lastHeartbeatRecv" : ISODate("2013-06-04T11:19:00Z"),
               "pingMs" : 0,
               "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: [hostname]:27021",
               "syncingTo" : "[hostname]:27021"
          }
     ],
     "ok" : 1
}

>db.isMaster()
{
     "setName" : "testset",
     "ismaster" : true,
     "secondary" : false,
     "hosts" : [
          "[hostname]:27022",
          "[hostname]:27021"
     ],
     "primary" : "[hostname]:27022",
     "me" : "[hostname]:27022",
     "maxBsonObjectSize" : 16777216,
     "maxMessageSizeBytes" : 48000000,
     "localTime" : ISODate("2013-06-04T11:22:28.771Z"),
     "ok" : 1
}

Configuring the Options for Replica Set Members

The replica set functionality includes a number of options you can use to control the behavior of a replica set’s members. When you run the rs.initiate(replSetcfg) or rs.add(membercfg) options, you have to supply a configuration structure that describes the characteristics of a replica set’s members:

{
  _id : <setname>,
 
  members: [
    {
      _id : <ordinal>,
      host : <hostname[:port]>,
      [ priority: <priority>, ]
      [arbiterOnly : true, ]
      [ votes : <n>, ]
      [ hidden: true, ]
      [ tags: { document }, ]
      [ slaveDelay: <seconds>, ]
      [ buildIndexes: true, ]
    }
    , ...
  ],
 
  settings: {
    [ chainingAllowed : <boolean>, ]
    [ getLastErrorModes: <modes>, ]
    [ getLastErrorDefaults: <lasterrdefaults>, ]
  }
}

For rs.initiate(), you should supply the full configuration structure, as shown here. The topmost level of the configuration structure itself includes three levels: _id, members, and settings. The _id is the name of the replica set, as supplied with the --replSet command-line option when you create the replica set members. The members array consists of a set of structures that describe each member of the set; this is the member structure that you supply to the rs.add() command when adding an individual server to the set. Finally, the settings array contains options that apply to the entire replica set.

Connecting to a Replica Set from Your Application

Connecting to a replica set from PHP is similar to connecting to a single MongoDB instance. The only difference is that it can provide either a single replica set instance address or a list of replica set members; the connection library will work out which server is the primary server and direct queries to that machine, even if the primary server is not one of the members that you provide. For this reason it is often best to specify multiple members in your connection string regardless; this way, you remove the risk associated with trying to discover from only one member which may be offline. The following example shows how to connect to a replica set from a PHP application:

<?php
 
$m = new MongoClient("mongodb://localhost:27021,
        localhost:27022", array("replicaSet" => "testSet"));
...
?>

<?php
$m = new MongoClient("mongodb://localhost:27021,
        localhost:27022", array("replicaSet" => "testSet"));
$m->setReadPreference(MongoClient::RP_SECONDARY_PREFERRED, array());
...
?>

mongodb://localhost:27021,localhost:27022?readPreference=nearest

$col->insert($document,array("w" => 1));

conf=rs.conf()
conf.members[0].tags = {site : "a"}
conf.members[1].tags = {site : "b"}
conf.members[2].tags = {site : "a"}
rs.reconfigure(conf)

rs.conf()
{
     "_id" : "testset",
     "version" : 8,
     "members" : [
          {
               "_id" : 0,
               "host" : "Pixl.local:27021",
               "tags" : {
                    "site" : "a"
               }
          },
          {
               "_id" : 1,
               "host" : "Pixl.local:27022",
               "tags" : {
                    "site" : "b"
               }
          },
          {
               "_id" : 2,
               "host" : "Pixl.local:27023",
               "priority" : 0,
               "hidden" : true,
               "tags" : {
                    "site" : "a"
               }
          },
          {
               "_id" : 3,
               "host" : "Pixl.local:27024",
               "arbiterOnly" : true
          }
     ]
}

$m->setReadPreference(MongoClient::RP_NEAREST, array(    array('site' => 'a'),));

conf = rs.conf()
conf.settings. getLastErrorModes = { bothSites : { "site": 2 } } }
rs.reconfig(conf)

$col->insert($document, array("w" => "bothSites"));

conf = rs.conf()
conf.settings.getLastErrorDefaults = { bothSites : 1 } }
rs.reconfig(conf)