One question people often have is when to start sharding. There are a couple of signs that sharding might be a good idea:

In general, you should start with a nonsharded setup and convert it to a sharded one, if and when you need.

Suppose we have an existing collection of logs and we want to shard it. If we enable sharding and tell MongoDB to use "timestamp" as the shard key, we’ll have a single shard with all of our data. We can insert any data we’d like, and it will all go to that one shard.

Now, suppose we add a new shard. Once this shard is up and running, MongoDB will break up the collection into two pieces, called chunks. A chunk contains all of the documents for a range of values for the shard key, so one chunk would have documents with a timestamp value between -∞ and, say, June 26, 2003, and the other chunk would have timestamps between June 27, 2003 and ∞. One of these chunks would then be moved to the new shard.

If we get a new document with a timestamp value before June 27, 2003, we’ll add that to the first chunk; otherwise, we’ll add the document to the second chunk.

The distribution of inserts across shards is very dependent on which key we’re sharding on.

If we choose to shard on something like "timestamp", where the value is probably going to increase and not jump around a lot, we’ll be sending all of the inserts to one shard (the one with the [June 27, 2003, ∞] chunk). Notice that, if we add a new shard and it splits the data again, we’ll still be inserting on just one server. If we add a new shard, MongoDB might split [June 27, 2003, ∞] into [June 27, 2003, December 12, 2010) and [December 12, 2010, ∞]. We’ll always have a chunk that will be “some date through infinity,” which is where our inserts will be going. This isn’t good for a very high write load, but it will make queries on the shard key very efficient.

If we have a high write load and want to evenly distribute writes across multiple shards, we should pick a shard key that will jump around more. This could be a hash of the timestamp in the log example or a key like "logMessage", which won’t have any particular pattern to it.

Whether your shard key jumps around or increases steadily, it is important to choose a key that will vary somewhat. If, for example, we had a "logLevel" key that had only values "DEBUG", "WARN", or "ERROR", MongoDB won’t be able to break up your data into more than three chunks (because there are only three different values). If you have a key with very little variation and want to use it as a shard key anyway, you can do so by creating a compound shard key on that key and a key that varies more, like "logLevel" and "timestamp".

Determining which key to shard on and creating shard keys should be reminiscent of indexing, because the two concepts are similar. In fact, often your shard key will just be the index you use most often.

To the end user, a sharded setup should be indistinguishable from a nonsharded one. However, it can be useful, especially when setting up sharding, to understand how different queries will be done depending on the shard key chosen.

Suppose we have the collection described in the previous section, which is sharded on the "name" key and has three shards with names ranging from A to Z. Different queries will be executed in different ways:

If we insert a new document, mongos will send that document to the appropriate shard, based on the value of its "name" key.

First we need to start up our config server and mongos. The config server needs to be started first, because mongos uses it to get its configuration. The config server can be started like any other mongod process:

$ mkdir -p ~/dbs/config
$ ./mongod --dbpath ~/dbs/config --port 20000

A config server does not need much space or resources. (A generous estimate is 1KB of config server space per 200MB of actual data.)

Now you need a mongos process for your application to connect to. Routing servers don’t even need a data directory, but they need to know where the config server is:

$ ./mongos --port 30000 --configdb localhost:20000

Shard administration is always done through a mongos.

$ mkdir -p ~/dbs/shard1
$ ./mongod --dbpath ~/dbs/shard1 --port 10000

$ ./mongo localhost:30000/admin
MongoDB shell version: 1.6.0
url: localhost:30000/admin
connecting to localhost:30000/admin
type "help" for help
>

> db.runCommand({addshard : "localhost:10000", allowLocal : true})
{
    "added" : "localhost:10000",
    "ok" : true
}

MongoDB won’t just distribute every piece of data you’ve ever stored: you have to explicitly turn sharding on at both the database and collection levels. Let’s look at an example: we’ll shard the bar collection in the foo database on the "_id" key. First, we enable sharding for foo:

> db.runCommand({"enablesharding" : "foo"})

Sharding a database results in its collections being stored on different shards and is a prerequisite to sharding one of its collections.

Once you’ve enabled sharding on the database level, you can shard a collection by running the shardcollection command:

> db.runCommand({"shardcollection" : "foo.bar", "key" : {"_id" : 1}}) 

Now the collection will be sharded by the "_id" key. When we start adding data, it will automatically distribute itself across our shards based on the values of "_id".

Setting up multiple config servers is simple. As of this writing, you can have one config server (for development) or three config servers (for production).

Setting up multiple config servers is the same as setting up one; you just do it three times:

$ mkdir -p ~/dbs/config1 ~/dbs/config2 ~/dbs/config3
$ ./mongod --dbpath ~/dbs/config1 --port 20001
$ ./mongod --dbpath ~/dbs/config2 --port 20002
$ ./mongod --dbpath ~/dbs/config3 --port 20003

Then, when you start a mongos, you should connect it to all three config servers:

$ ./mongos --configdb localhost:20001,localhost:20002,localhost:20003

Config servers use two-phase commit, not the normal MongoDB asynchronous replication, to maintain separate copies of the cluster’s configuration. This ensures that they always have a consistent view of the cluster’s state. It also means that if a single config server is down, the cluster’s configuration information will go read-only. Clients are still able to do both reads and writes, but no rebalancing will happen until all of the config servers are back up.

You can also run as many mongos processes as you want. One recommended setup is to run a mongos process for every application server. That way, each application server can talk to mongos locally, and if the server goes down, no one will be trying to talk to a mongos that isn’t there.

In production, each shard should be a replica set. That way, an individual server can fail without bringing down the whole shard. To add a replica set as a shard, pass its name and a seed to the addshard command.

For example, say we have a replica set named "foo" containing a server at prod.example.com:27017 (among other servers). We could add this set to the cluster with the following:

> db.runCommand({"addshard" : "foo/prod.example.com:27017"})

If prod.example.com goes down, mongos will know that it is connected to a replica set and use the new primary for that set.

This may seem like an overwhelming number of machines: three config servers, at least two mongods per shard, and as many mongos processes as you want. However, not everything has to have its own machine. The main thing to avoid is putting an entire component on one machine. For example, avoid putting all three config servers, all of your mongos processes, or an entire replica set on one machine. However, a config server and mongos processes can happily share a box with a member of a replica set.

All of the code in the following sections assume that you are running a shell connected to a mongos process and have already run use config.

> db.shards.find()
{ "_id" : "shard0", "host" : "localhost:10000" }
{ "_id" : "shard1", "host" : "localhost:10001" }

> db.databases.find()
{ "_id" : "admin", "partitioned" : false, "primary" : "config" }
{ "_id" : "foo", "partitioned" : false, "primary" : "shard1" }
{ "_id" : "x", "partitioned" : false, "primary" : "shard0" }
{
    "_id" : "test",
    "partitioned" : true,
    "primary" : "shard0",
    "sharded" : {
        "test.foo" : {
            "key" : {"x" : 1},
            "unique" : false
        }
    }
}

> db.chunks.find()
{
    "_id" : "test.foo-x_MinKey",
    "lastmod" : { "t" : 1276636243000, "i" : 1 },
    "ns" : "test.foo",
    "min" : {
        "x" : { $minKey : 1 }
    },
    "max" : {
        "x" : { $maxKey : 1 }
    },
    "shard" : "shard0"
}

We’ve already covered some of the basic commands, such as adding chunks and enabling sharding on a collection. There are a couple more commands that are useful for administering a cluster.

> db.printShardingStatus()
--- Sharding Status ---
  sharding version: { "_id" : 1, "version" : 3 }
  shards:
      { "_id" : "shard0", "host" : "localhost:10000" }
      { "_id" : "shard1", "host" : "localhost:10001" }
  databases:
      { "_id" : "admin", "partitioned" : false, "primary" : "config" }
      { "_id" : "foo", "partitioned" : false, "primary" : "shard1" }
      { "_id" : "x", "partitioned" : false, "primary" : "shard0" }
      { "_id" : "test", "partitioned" : true, "primary" : "shard0",
          "sharded" : { "test.foo" : { "key" : { "x" : 1 }, "unique" : false } } }
              test.foo chunks:
                  { "x" : { $minKey : 1 } } -->> { "x" : { $maxKey : 1 } } on : shard0
                  { "t" : 1276636243000, "i" : 1 }

> db.runCommand({"removeshard" : "localhost:10000"});
{
    "started draining" : "localhost:10000", 
    "ok" : 1
}

> db.runCommand({"removeshard" : "localhost:10000"});
{ 
    "msg" : "already draining...",
    "remaining" : {
        "chunks" : 39,
        "dbs" : 2 
    },
    "ok" : 1 
}

> db.runCommand({"moveprimary" : "test", "to" : "localhost:10001"})
{
    "primary" : "localhost:10001",
    "ok" : 1
}