In replica sets, if we issue a createIndex() command, secondaries will begin creating the index after the primary is finished creating it. Similarly, in sharded environments, primaries will start building indexes and secondaries will start after the primary for each shard is finished.

A recommended approach to building indexes in replica sets is:

• Stop one secondary from the replica set
• Restart it as a standalone server in a different port
• Build the index from the shell as a standalone index
• Restart the secondary in the replica set
• Allow the secondary to catch up with the primary

We need to have a large enough oplog in the primary to make sure that the secondary will be able to catch up once it's reconnected. This is a manual process, involving several steps for each primary/secondary server.

This approach can be repeated for each secondary server in the replica set. Then for the primary server we can do either of these things:

1. Build the index in the background.
2. Step down the primary using rs.stepDown() and repeat the preceding process with the server as a secondary now.

Using approach number 2, when the primary steps down there will be a brief period when our cluster won't be taking any writes. Our application shouldn't timeout during this (usually less than) 30-60 second period.

Building an index in the background in the primary will build it in the background for the secondaries too. This may impact writes in our servers during index creation but on the plus side has exactly zero manual steps. It is always a good idea to have a staging environment that mirrors production and dry run operations that affect the live cluster in staging to avoid surprises.