A 10 G network is strongly recommended for building a Ceph cluster; while 1 G networking will work, the amount of latency will almost be unacceptable, and will limit you as to the size of the nodes you can deploy. Thought should also be given to recovery; in the event of a disk or node failure, large amounts of data will need to be moved around the cluster. Not only will a 1 G network be table to provide sufficient performance for this, but normal I/O traffic will be impacted. In the very worst case, this may lead to OSDs timing out, causing cluster instabilities.

As mentioned, one of the main benefits of 10 G networking is the lower latency. Quite often, a cluster will never push enough traffic to make full use of the 10 G bandwidth; however, the latency improvement will be realized no matter the load on the cluster. The round trip time for a 4k packet over a 10 G network might take around 90 us, whereas the same 4k packet over 1 G networking will take over 1 ms. As you will learn in the tuning section of this book, latency has a direct affect on the performance of a storage system, particularly when performing direct or synchronous I/O.

Lately, the next generation of networking hardware has become available, supporting speeds starting at 25 G and climbing to 100 G. If you are implementing a new network when deploying your Ceph cluster, it is highly recommended that you look into deploying this next-generation hardware.

If your OSD node will come equipped with dual NICs, you should carefully work on for a network design that allows you to use them active for both transmit and receive. It is wasteful to leave a 10 G link in a passive state, and will help to lower latency under load.