Last, but not least, Cassandra log is a good tool for monitoring what is going on inside Cassandra. However, monitoring the log file is an extremely non-scalable option. Therefore, if you are starting with fewer than five Cassandra machines, you may consider occasionally looking into their log files. The most common use of the Cassandra log is to perform the postmortem for a failure when you do not have any other monitoring and reporting mechanism in place.

The location of the log4j log can be found from Cassandra's conf/log4j-server.properties file:

# This has been altered during installation
log4j.appender.R.File=/mnt/cassandra-logs/system.log

As long as you view this file filled with lines, starting with INFO, you may think the system has been behaving alright. Lines with WARN may or may not be interesting.

For example, it is ok to have some WARN messages in the system, as follows:

INFO [MemoryMeter:1] 2013-05-24 12:31:39,099 Memtable.java (line 213) CFS(Keyspace='Keyspace1', ColumnFamily='Standard1') liveRatio is 1.0 (just-counted was 1.0).calculation took 3ms for 325 columns
WARN [MemoryMeter:1] 2013-05-24 12:31:39,135 Memtable.java (line 197) setting live ratio to minimum of 1.0 instead of 0.8662032831217121

However, some warnings, as shown in the following lines, may be a definite sign of danger and should be fixed to avoid a catastrophe (it may not be a big deal if you have RF and CL set properly):

# May be a future crash due to lack of disk space
WARN [CompactionExecutor:45] 2013-05-24 17:34:03,709 CompactionTask.java (line 82) insufficient space to compact all requested files SSTableReader(path='/mnt/cassandra- data/data/Keyspace1/Standard1/Keyspace1-Standard1-hf-34- Data.db'), SSTableReader(path='/mnt/cassandra- data/data/Keyspace1/Standard1/Keyspace1-Standard1-hf-35- Data.db'), SSTableReader(path='/mnt/cassandra- data/data/Keyspace1/Standard1/Keyspace1-Standard1-hf-28- Data.db')

An ERROR message as shown in the following lines should always be attended to. It usually answers the buts and whys of Cassandra behavior, points out an incorrect configuration, and tells you about what to do next and what to change in the read/write pattern:

# oh snap! Dave was right about horizontal scaling.
ERROR [FlushWriter:7] 2013-05-24 17:35:21,617 AbstractCassandraDaemon.java (line 132) Exception in thread Thread[FlushWriter:7,5,main]
java.lang.RuntimeException: Insufficient disk space to flush 71502048 bytes
[–- snip --]
WARN [CompactionExecutor:46] 2013-05-24 17:35:26,871 FileUtils.java (line 116) Failed closing IndexWriter(/mnt/cassandra-data/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-hf-42)
java.io.IOException: No space left on device

Cassandra logs are great. If you find yourself looking too frequently into them, you probably need a better monitoring mechanism. Also, if you feel like getting blinded by an over-abundance of information, you may change the log a level up from INFO to WARN in the log4j-server.properties file. It is suggested you should not turn on the DEBUG mode, unless you are editing the Cassandra code base and debugging the changes. It may lead to lots of I/O activity and affect the performance.

# GC logging options -- uncomment to enable
# JVM_OPTS="$JVM_OPTS -XX:+PrintGCDetails"
# JVM_OPTS="$JVM_OPTS -XX:+PrintGCDateStamps"
# JVM_OPTS="$JVM_OPTS -XX:+PrintHeapAtGC"
# JVM_OPTS="$JVM_OPTS -XX:+PrintTenuringDistribution"
# JVM_OPTS="$JVM_OPTS -XX:+PrintGCApplicationStoppedTime"
# JVM_OPTS="$JVM_OPTS -XX:+PrintPromotionFailure"
# JVM_OPTS="$JVM_OPTS -XX:PrintFLSStatistics=1"
# JVM_OPTS="$JVM_OPTS -Xloggc:/var/log/cassandra/gc-`date +%s`.log"

# If you are using JDK 6u34 7u2 or later you can enable GC log rotation
# don't stick the date in the log name if rotation is on.
# JVM_OPTS="$JVM_OPTS -Xloggc:/var/log/cassandra/gc.log"
# JVM_OPTS="$JVM_OPTS -XX:+UseGCLogFileRotation"
# JVM_OPTS="$JVM_OPTS -XX:NumberOfGCLogFiles=10"
# JVM_OPTS="$JVM_OPTS -XX:GCLogFileSize=10M"