Last, but not least, Cassandra log is a good tool for monitoring what is going on inside Cassandra. However, monitoring the logfile is an extremely non-scalable option. So, if you are starting with anything fewer than five Cassandra machines, you may consider occasionally looking into their logfiles. The most common use of the Cassandra log is to perform the postmortem for a failure when you do not have any other monitoring and reporting mechanism in place.
The location of the log4j log can be found out from Cassandra's conf/log4j-server.properties file:
# This has been altered during installation
log4j.appender.R.File=/mnt/cassandra-logs/system.logAs long as you view this file filled with lines, starting with INFO, you may think the system has been behaving alright. Lines with WARN may or may not be interesting.
For example, it is OK to have some WARNs in the system as shown in the following system:
INFO [MemoryMeter:1] 2013-05-24 12:31:39,099 Memtable.java (line 213) CFS(Keyspace='Keyspace1', ColumnFamily='Standard1') liveRatio is 1.0 (just-counted was 1.0).calculation took 3ms for 325 columns WARN [MemoryMeter:1] 2013-05-24 12:31:39,135 Memtable.java (line 197) setting live ratio to minimum of 1.0 instead of 0.8662032831217121
But, some warnings may be a definite sign of danger and should be fixed to avoid a catastrophe (it may not be a big deal if you have RF and CL set properly).
# May be a future crash due to lack of disk space
WARN [CompactionExecutor:45] 2013-05-24 17:34:03,709 CompactionTask.java (line 82) insufficient space to compact all requested files SSTableReader(path='/mnt/cassandra-data/data/Keyspace1/Standard1/Keyspace1-Standard1-hf-34-Data.db'), SSTableReader(path='/mnt/cassandra-data/data/Keyspace1/Standard1/Keyspace1-Standard1-hf-35-Data.db'), SSTableReader(path='/mnt/cassandra-data/data/Keyspace1/Standard1/Keyspace1-Standard1-hf-28-Data.db') An ERROR should always be attended to. It usually answers buts and whys about Cassandra behavior, points out incorrect configuration, tells you about what to do next, and what to change in the read/write pattern.
# oh snap! Dave was right about horizontal scaling.
ERROR [FlushWriter:7] 2013-05-24 17:35:21,617 AbstractCassandraDaemon.java (line 132) Exception in thread Thread[FlushWriter:7,5,main]
java.lang.RuntimeException: Insufficient disk space to flush 71502048 bytes
[–- snip --]
WARN [CompactionExecutor:46] 2013-05-24 17:35:26,871 FileUtils.java (line 116) Failed closing IndexWriter(/mnt/cassandra-data/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-hf-42)
java.io.IOException: No space left on device Cassandra logs are great. If you find yourself looking too frequently into it, you probably need a better monitoring mechanism. Also, if you feel like getting blinded by the abundance of information, you may change the log a level up from INFO to WARN in the log4j-server.properties file. It is suggested not to turn on the DEBUG mode, unless you are editing the Cassandra code base and debugging the changes. It may lead to lots of I/O activity and may affect the performance.
JVM options provide a nice way to monitor Java applications. It is not specific to Cassandra. One may set various -XX options as part of arguments when starting the Java application. In Cassandra, these options can be enabled by uncommenting the lines with –XX. The following is the list of the options:
# GC logging options -- uncomment to enable # JVM_OPTS="$JVM_OPTS -XX:+PrintGCDetails" # JVM_OPTS="$JVM_OPTS -XX:+PrintGCDateStamps" # JVM_OPTS="$JVM_OPTS -XX:+PrintHeapAtGC" # JVM_OPTS="$JVM_OPTS -XX:+PrintTenuringDistribution" # JVM_OPTS="$JVM_OPTS -XX:+PrintGCApplicationStoppedTime" # JVM_OPTS="$JVM_OPTS -XX:+PrintPromotionFailure" # JVM_OPTS="$JVM_OPTS -XX:PrintFLSStatistics=1" # JVM_OPTS="$JVM_OPTS -Xloggc:/var/log/cassandra/gc-'date +%s'.log" # If you are using JDK 6u34 7u2 or later you can enable GC log rotation # don't stick the date in the log name if rotation is on. # JVM_OPTS="$JVM_OPTS -Xloggc:/var/log/cassandra/gc.log" # JVM_OPTS="$JVM_OPTS -XX:+UseGCLogFileRotation" # JVM_OPTS="$JVM_OPTS -XX:NumberOfGCLogFiles=10" # JVM_OPTS="$JVM_OPTS -XX:GCLogFileSize=10M"
It is pretty obvious what these options do. In case you wanted to enable some of these options, the following table shows what they mean:
The rest of the options are for log location, log size, and rolling log counts.