There are two sets of files that Cassandra writes to as part of handling update operations: the commit log and the datafile. Their different purposes need to be considered in order to understand how to treat them during configuration.

The commit log can be thought of as short-term storage. As Cassandra receives updates, every write value is written immediately to the commit log in the form of raw sequential file appends. If you shut down the database or it crashes unexpectedly, the commit log can ensure that data is not lost. That’s because the next time you start the node, the commit log gets replayed. In fact, that’s the only time the commit log is read; clients never read from it. But the normal write operation to the commit log blocks, so it would damage performance to require clients to wait for the write to finish.

The datafile represents the Sorted String Tables (SSTables). Unlike the commit log, data is written to this file asynchronously. The SSTables are periodically merged during major compactions to free up space. To do this, Cassandra will merge keys, combine columns, and delete tombstones.

Read operations can refer to the in-memory cache and in this case don’t need to go directly to the datafiles on disk. If you can allow Cassandra a few gigabytes of memory, you can improve performance dramatically when the row cache and the key cache are hit.

The commit logs are periodically removed, following a successful flush of all their appended data to the dedicated datafiles. For this reason, the commit logs will not grow to anywhere near the size of the datafiles, so the disks don’t need to be as large; this is something to consider during hardware selection. For example, if Cassandra runs a flush, you’ll see something in the server logs like this:

INFO 18:26:11,497 Enqueuing flush of Memtable-LocationInfo@26830618(52 bytes, 2 
operations)
INFO 18:26:11,497 Writing Memtable-LocationInfo@26830618(52 bytes, 2 operations)
INFO 18:26:11,732 Completed flushing /var/lib/cassandra/data/system/
LocationInfo-2-Data.db
INFO 18:26:11,732 Discarding obsolete commit log:
  CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1278894011530.log)

Then, if you check the commit log directory, that file has been deleted.

By default, the commit log and the datafile are stored in the following locations:

<CommitLogDirectory>/var/lib/cassandra/commitlog</CommitLogDirectory>

<DataFileDirectories>
  <DataFileDirectory>/var/lib/cassandra/data</DataFileDirectory>
</DataFileDirectories>

You can change these values to store the datafiles or commit log in different locations. You can specify multiple datafile directories if you wish.

For testing, you might not see a need to change these locations. However, it’s recommended that you store the datafiles and the commit logs on separate hard disks for maximum performance. Cassandra, like many databases, is particularly dependent on the speed of the hard disk and the speed of the CPUs (it’s best to have four or eight cores, to take advantage of Cassandra’s highly concurrent construction). So make sure for QA and production environments to get the fastest disks you can, and get at least two separate ones so that the commit logfiles and the datafiles are not competing for I/O time. It’s more important to have several processing cores than one or two very fast ones.

The reply timeout is a setting that indicates how long Cassandra will wait for other nodes to respond before deciding that the request is a failure. This is a common setting in relational databases and messaging systems. This value is set by the RpcTimeoutInMillis element (rpc_timeout_in_ms in YAML). By default, this is 5,000, or five seconds.

You can set the value for how large the commit log is allowed to grow before it stops appending new writes to a file and creates a new one. This is similar to setting log rotation on Log4J.

This value is set with the CommitLogRotationThresholdInMB element (commitlog_rotation_threshold_in_mb in YAML). By default, the value is 128MB.

Another setting related to commit logs is the sync operation, represented by the commitlog_sync element. There are two possible settings for this: periodic and batch. periodic is the default, and it means that the server will make writes durable only at specified intervals. When the server is set to make writes durable periodically, you can potentially lose the data that has not yet been synced to disk from the write-behind cache.

In order to guarantee durability for your Cassandra cluster, you may want to examine this setting.

If your commit log is set to batch, it will block until the write is synced to disk (Cassandra will not acknowledge write operations until the commit log has been completely synced to disk). This clearly will have a negative impact on performance.

You can change the value of the configuration attribute from periodic to batch to specify that Cassandra must flush to disk before it acknowledges a write. Changing this value will require taking some performance metrics, as there is a necessary trade-off here: forcing Cassandra to write more immediately constrains its freedom to manage its own resources. If you do set commitlog_sync to batch, you need to provide a suitable value for CommitLogSyncBatchWindowInMS, where MS is the number of milliseconds between each sync effort. Moreover, this is not generally needed in a multinode cluster when using write replication, because replication by definition means that the write isn’t acknowledged until another node has it.

If you decide to use batch mode, you will probably want to split the commit log onto a separate device to mitigate the performance impact. It’s a good idea to split it out onto a separate disk from the SSTables (data) anyway, even if you don’t do this.

Each column family has a single memtable associated with it. There are a few settings around the treatment of memtables. The size that the memtable can grow to before it is flushed to disk as an SSTable is specified with the MemtableSizeInMB element (binary_memtable_throughput_in_mb in YAML). Note that this value is based on the size of the memtable itself in memory, and not heap usage, which will be larger because of the overhead associated with column indexing.

You’ll want to balance this setting with MemtableObjectCountInMillions, which sets a threshold for the number of column values that will be stored in a memtable before it is flushed.

A related configurable setting is memtable_throughput_in_mb. This refers to the maximum number of columns that will be stored in a single memtable before the memtable is flushed to disk as an SSTable. The default value is 0.3, which is approximately 333,000 columns.

You can also configure how long to keep memtables in memory after they’ve been flushed to disk. This value can be set with the memtable_flush_after_mins element. When the flush is performed, it will write to a flush buffer, and you can configure the size of that buffer with flush_data_buffer_size_in_mb.

Another element related to tuning the memtables is memtable_flush_writers. This setting, which is 1 by default, indicates the number of threads used to write out the memtables when it becomes necessary. If you have a very large heap, it can improve performance to set this count higher, as these threads are blocked during disk I/O.

Cassandra differs from many data stores in that it offers much faster write performance than read performance. There are two settings related to how many threads can perform read and write operations: concurrent_reads and concurrent_writes. In general, the defaults provided by Cassandra out of the box are very good. But you might want to update the concurrent_reads setting immediately before you start your server. That’s because the concurrent_reads setting is optimal at two threads per processor core. By default, this setting is 8, assuming a four-core box. If that’s what you have, you’re in business. If you have an eight-core box, tune it up to 16.

The concurrent_writes setting behaves somewhat differently. This should match the number of clients that will write concurrently to the server. If Cassandra is backing a web application server, you can tune this setting from its default of 32 to match the number of threads the application server has available to connect to Cassandra. It is common in Java application servers such as WebLogic to prefer database connection pools no larger than 20 or 30, but if you’re using several application servers in a cluster, you’ll need to factor that in as well.

There are several settings related to caching, both within Cassandra and at the operating system level. Caches can use considerable memory, and it’s a good idea to tune them carefully once you understand your usage patterns.

There are two primary caches built into Cassandra: a row cache and a key cache. The row cache caches complete rows (all of their columns), so it is a superset of the key cache. If you are using a row cache for a given column family, you will not need to use a key cache on it as well.

Your caching strategy should therefore be tuned in accordance with a few factors:

The keys_cached setting indicates the number of key locations—not key values—that will be saved in memory. This can be specified as a fractional value (a number between 0 and 1) or as an integer. If you use a fraction, you’re indicating a percentage of keys to cache, and an integer value indicates an absolute number of keys whose locations will be cached.

This setting will consume considerable memory, but can be a good trade-off if your locations are not hot already.

The purpose of disk_access_mode is to enable memory mapped files so that the operating system can cache reads, thus reducing the load on Cassandra’s internal caches. This sounds great, but in practice, disk_access_mode is one of the less-useful settings, and at this point doesn’t work exactly as was originally envisioned. This may be improved in the future, but it is just as likely that the setting will be removed. Certainly feel free to play around with it, but you might not see much difference.

You can also populate the row cache when the server starts up. To do this, use the preload_row_cache element. The default setting for this is false, but you will want to set it to true to improve performance. The cost is that bootstrapping can take longer if there is considerable data in the column family to preload.

The rows_cached setting specifies the number of rows that will be cached. By default, this value is set to 0, meaning that no rows will be cached, so it’s a good idea to turn this on. If you use a fraction, you’re indicating a percentage of everything to cache, and an integer value indicates an absolute number of rows whose locations will be cached. You’ll want to use this setting carefully, however, as this can easily get out of hand. If your column family gets far more reads than writes, then setting this number very high will needlessly consume considerable server resources. If your column family has a lower ratio of reads to writes, but has rows with lots of data in them (hundreds of columns), then you’ll need to do some math before setting this number very high. And unless you have certain rows that get hit a lot and others that get hit very little, you’re not going to see much of a boost here.

The buffer sizes represent the memory allocation when performing certain operations. The following is a quick overview of these settings:

Cassandra ships with a popular utility called py_stress that you can use to run a stress test on your Cassandra cluster. To run py_stress, navigate to the <cassandra-home>/contrib directory. You might want to check out the README.txt file, as it will have the list of dependencies required to run the tool.

There are a few steps we need to go through before we can run the tool. First, make sure that you still have the default keyspace (Keyspace1) in your configuration and have it loaded, as the tool will run against the default column family definitions.

Then, you need to build the Python Thrift interface, which may require a few steps.

No module named thrift.transport

eben@morpheus$ python
Python 2.6.5 (r265:79063, Apr 16 2010, 13:09:56) 
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information

$ sudo apt-get install -y libboost-dev libevent-dev python-dev automake pkg-config 
libtool flex bison

$ ./bootstrap.sh
$ ./bjam

$ ./bootstrap.sh
$ ./configure
$ ./make
$ ./make install
$ cd lib/py
$ ./make

$ thrift -version
Thrift version 0.2.0-exported

$ export PYTHONPATH=/usr/lib/python2.6/site-packages/

$ ant gen-thrift-py

$ stress.py -d 192.168.1.5

eben@morpheus$ ./stress.py -d 192.168.1.5 -o insert
total,interval_op_rate,avg_latency,elapsed_time
196499,19649,0.0024959407711,10
370589,17409,0.00282591440216,20
510076,13948,0.00295883878841,30
640813,13073,0.00438663874102,40
798070,15725,0.00312562838215,50
950489,15241,0.0029109908417,60
1000000,4951,0.00444872583334,70

eben@morpheus$ ./stress.py -d 192.168.1.5 -o insert -t 10
total,interval_op_rate,avg_latency,elapsed_time
219217,21921,0.000410911544945,10
427199,20798,0.000430060066223,20
629062,20186,0.000443717396772,30
832964,20390,0.000437958271074,40
1000000,16703,0.000463042383339,50

$ ./stress.py -d 192.168.1.5 -o read
total,interval_op_rate,avg_latency,elapsed_time
103960,10396,0.00478858081549,10
225999,12203,0.00406984714627,20
355129,12913,0.00384438665076,30
485728,13059,0.00379976526221,40
617036,13130,0.00378045491559,50
749154,13211,0.00375620621777,60
880605,13145,0.00377542658007,70
1000000,11939,0.00374060139004,80

Cassandra allows you to configure a variety of options for how the server should start up, how much Java memory should be allocated, and so forth. In this section we look at how to tune the startup.

If you’re using Windows, the startup script is called cassandra.bat, and on Linux it’s cassandra.sh. You can start the server by simply executing this file, which sets several defaults. But there’s another file in the bin directory that allows you to configure a variety of settings related to how Cassandra starts. This file is called cassandra.in.sh and it separates certain options, such as the JVM settings, into a different file to make it easier to update.

JVM_OPTS=" 
        -da 
        -Xms1024M 
        -Xmx1024M 
        -XX:+UseParallelGC 
        -XX:+CMSParallelRemarkEnabled 
        -XX:SurvivorRatio=4 
        -XX:MaxTenuringThreshold=0 

       -XX:CMSInitiatingOccupancyFraction=88 
       -XX:+HeapDumpOnOutOfMemoryError 
       -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -verbose:gc 

In this chapter we looked at the settings available in Cassandra to aid in performance tuning, including caching settings, memory settings, and hardware concerns. We also set up and used the Python stress test tool to write and then read one million rows of data.

If you’re somewhat new to Linux systems and you want to run Cassandra on Linux (which is recommended), you may want to check out Jonathan Ellis’s blog entry on using a variety of Linux performance monitoring tools to help you understand the performance of your underlying platform so that you can troubleshoot in the right place. You can find that entry at http://spyced.blogspot.com/2010/01/linux-performance-basics.html.

Ultimately, you want to have lots of memory available, and as many processors and cores as you can afford.