By default, RBD will be configured with a 128 KB readahead. If your workload mainly involves large sequential reading, you can get a significant boost in performance by increasing the readahead value. In kernels before 4.4, there was a limitation where readahead values larger than 2 MB were ignored. In most storage systems, this was not an issue, as the stripe sizes would have been smaller than 2 MB. As long as readahead is bigger than the stripe size, all the disks will be involved and performance will increase.

By default, a Ceph RBD is striped across 4 MB objects, and so an RBD has a chunk size of 4 MB and a stripe size of 4 MB * number of OSDS in the cluster. Therefore, with a readahead size smaller than 4 MB, most of the time, readahead will be doing very little to improve performance, and you will likely see that read's performance is struggling to exceed that of a single OSD.

In kernel 4.4 and above, you can set the readahead value much higher and experience read performance in the range of hundreds of MBs in a second.