Cassandra is all about speed—quick installation, fast reads, and fast writes. You have got your application optimized, minimized network trips by batching, and denormalized the data to get maximum information in one request, but still the performance is not what you read over the Web and in various blogs. You start to doubt if the claims actually measure. Hold! You may need to tune things up.
This chapter will discuss how to get a sense of a Cassandra cluster's capacity and have a performance number handy to back up your claims. It then dives into the various settings that affect read and write throughputs, a couple of JVM tuning parameters, and finally a short discussion on how scaling horizontally and vertically can improve the performance.
Before you start to claim the performance numbers of your Cassandra backend based on numbers that you have read elsewhere, it is important to perform your own stress testing. It is very easy to do that in Cassandra as it provides special tooling for stress testing. It is a good idea to customize the parameters of the stress test to represent a use case that is closer to what your application is going to do. And this will save a lot of heated discussion later due to discrepancies in the load test and the actual throughput that the software is able to pull out of the setting.
The load test tool is found under the Cassandra installation directory under tools as tools/bin/cassandra-stress. Here are some useful parameters:
-d, --nodes: CSV list of nodes to run the queries against-o, --operation: Operation to perform, default: insert-R, --replication-strategy: Replication strategy, def: Simple-O, --strategy-properties: in <dc>:<RF>,<dc1>:<RF1> format-U, --comparator: column comparator -e, --consistency-level: consistency level, default: ONE-c, --columns: Number of columns per key, default: 5-n, --num-keys: Number of keys, default: 1Mn-S, --column-size: Size of column values, default: 34 bytes-L, --enable-cql: Perform operations using CQL
Let's take an example application, say a chat application that stores each chat session in a row and each message in a separate column. Let's assume that each session has an average message transfer of 500 messages (500 columns), each message has 100 UTF8 characters (300 bytes/column), our servers are located in an Amazon EC2 environment but in a single region (Ec2Snitch, NetworkTopologyStrategy), the columns are sorted by time (the TimeUUIDType comparator), and that we want to read and write consistency level 2 (for 3 server configurations of ours, R + W > N).
We will load test this against a three servers setup which has all defaults and in which initial_token is equidistant. Each machine has the following specification (standard M1.large instances):
Memory: 7.5 GiB CPU: 2 Virtual cores with 2 ECU each, 64 bit Hard disc: 840 GiB (RAID0) OS: CentOS 5.4 Cassandra: version 1.1.11
Here is the test run command:
$ tools/bin/cassandra-stress -d 10.99.9.67,10.147.171.159,10.114.189.54 -o INSERT -R org.apache.cassandra.locator.NetworkTopologyStrategy -O us-east:2 -U TimeUUIDType -e QUORUM -c 5 -S 300 -y Standard -t 200 -n 1000000 -F 10000
It says insert 1 million records with 5 columns in a row, under 10,000 unique row keys in a standard column family using 200 threads, sorted by
TimeUUID, with consistency level as QUORUM. This will ensure that each row roughly has 500 columns. The test result shows that the inserts rate stays about 5000 inserts per second. There are a couple of blips, maybe due to a MemTable flush or a compaction in the process. This is shown in the following figure.
On the same setup, let's execute a read stress test:
$ tools/bin/cassandra-stress -d 10.99.9.67,10.147.171.159,10.114.189.54 -o READ -R org.apache.cassandra.locator.NetworkTopologyStrategy -O us-east:2 -U TimeUUIDType -e QUORUM -c 5 -S 300 -y Standard -t 200 -n 1000000 -F 10000
Figure 5.1: Result of write stress test
With these parameters, the reads stay just below 8000 reads per second. The reads stay rather stable. The statistics for the read performance is described in the following figure.
This is the most basic stress test that gives an overall idea of what can be achieved from a Cassandra setup. There may be multiple intrinsic (bad choices, high I/O due to frequent MemTable flushes, and so on) or extrinsic (locking in application code, poor inter-node connectivity, and so on) configurations that can differ the performance of your setup from what is mentioned here. But you should run this test anyway to get the baseline statistics of your system.
Not all test cases can be stress tested using Cassandra's built-in stress tool. Based on what your application does, you may need to write your own stress test that represents your application's model behavior.
Another load test mechanism that is worth looking at is Yahoo! Cloud Serving Benchmark (YCSB). You can read more on YCSB at https://github.com/brianfrankcooper/YCSB/wiki/Getting-Started. It is a framework with which you can add a database interface as a plugin and run a load against it. Cassandra is one of the many databases that it supports out of the box. Details of this tool are out of the scope of this book, but you can easily learn how to use it from its excellent documentation on GitHub.
Figure 5.2: Result of read stress test