Storage Service MBean

Because Cassandra is a database, it’s essentially a very sophisticated storage program; therefore, Cassandra’s storage engine as implemented in the org.apache.cassandra.service.StorageService is an essential focus of monitoring and management. The corresponding MBean for this service is the StorageServiceMBean, which provides many useful attributes and operations.

The MBean exposes identifying information about the node, including its host ID, the cluster name and partitioner in use. It also allows you to inspect the node’s OperationMode, which reports normal if everything is going smoothly (other possible states are leaving, joining, decommissioned, and client). These attributes are used by nodetool commands such as describecluster and info

You can also view the current set of live nodes, as well as the set of unreachable nodes in the cluster. If any nodes are unreachable, Cassandra will tell you their IP addresses in the UnreachableNodes attribute.

To get an understanding of how much data is stored on each node, you can use the getLoadMapWithPort() method, which will return to you a Java Map with keys of IP addresses with values of their corresponding storage loads. You can also use the effectiveOwnershipWithPort(String keyspace) operation to access the percentage of the data in a keyspace owned by each node. This information is used in the nodetool ring command.

If you’re looking for which nodes own a certain partition key, you can use the getNaturalEndpointsWithPort() operation. Pass it the keyspace name, table name and the partition key for which you want to find the endpoint value, and it will return a list of IP addresses (with port number) that are responsible for storing this key.

You can also use the describeRingWithPortJMX() operation to get a list of token ranges in the cluster including their ownership by nodes in the cluster. This is used by the nodetool describering operation.

There are many standard maintenance operations that the StorageServiceMBean affords you, including resumeBootstrap(), joinRing(), flush(), truncate(), repairAsync(), cleanup(), scrub(), drain(), removeNode(), decommission(), and operations to start and stop gossip, and the native transport. We’ll dig into the nodetool commands that correspond to these operations in Chapter 12.

If you want to change Cassandra’s log level at runtime without interrupting service (as you’ll see in “Logging”), you can invoke the setLoggingLevel(String classQualifier, String level) method.

Storage Proxy MBean

As you learned in Chapter 6, the org.apache.cassandra.service.StorageProxy provides a layer on top of the StorageService to handle client requests and inter-node communications. The StorageProxyMBean provides the ability to check and set timeout values for various operations including read and write. Along with many other attributes exposed by Cassandra’s MBeans, these timeout values would originally be specified in the cassandra.yaml file. Setting these attributes takes effect only until the next time the node is restarted, whereupon they’ll be initialized to the values in the configuration file.

This MBean also provides access to hinted handoff settings such as the maximum time window for storing hints. Hinted handoff statistics include getTotalHints() and getHintsInProgress(). You can disable hints for nodes in a particular data center with the disableHintsForDC() operation.

You can also turn this node’s participation in hinted handoff on or off via setHintedHandoffEnabled(), or check the current status via getHintedHandoffEnabled(). These are used by nodetool’s enablehandoff, disablehandoff, and statushandoff commands, respectively.

Hints Service MBean

In addition to the hinted handoff operations on the StorageServiceMBean, Cassandra provides more hinted handoff controls via the org.apache.cassandra.hints.HintsServiceMBean. The MBean exposes the ability to pause and resume hint delivery You can delete hints that are stored up for a specific node with deleteAllHintsForEndpoint().

Additionally, you can pause and resume hint delivery to all nodes with pauseDispatch() and resumeDispatch(). You can delete stored hints for all nodes with the deleteAllHints() operation, or for a specific node with deleteAllHintsForEndpoint(). These are used by nodetool’s pausehandoff, resumehandoff, and truncatehints commands.

Column Family Store MBean

Cassandra registers an instance of the org.apache.cassandra.db.ColumnFamilyStoreMBean for each table stored in the node under org.apache.cassandra.db > Tables (this is a legacy name: tables were known as column families in early versions of Cassandra).

The ColumnFamilyStoreMBean provides access to the compaction and compression settings for each table. This allows you to temporarily override these settings on a specific node. The values will be reset to those configured on the table schema when the node is restarted.

The MBean also exposes a lot of information about the node’s storage of data for this table on disk. The getSSTableCountPerLevel() operation provides a list of how many SStables are in each tier. The estimateKeys() operation provides an estimate of the number of partitions stored on this node. Taken together, this information can give you some insight as to whether invoking the forceMajorCompaction() operation for this table might help free space on this node and increase read performance.

There is also a trueSnapshotsSize() operation that allows you to determine the size of SSTable shapshots which are no longer active. A large value here indicates that you should consider deleting these snapshots, possibly after making an archive copy.

Because Cassandra stores indexes as tables, there is also a ColumnFamilyStoreMBean instance for each indexed column, available under org.apache.cassandra.db > IndexTables (previously IndexColumnFamilies), with the same attributes and operations.

Commit Log MBean

The org.apache.cassandra.db.commitlog.CommitLogMBean exposes attributes and operations that allow you to learn about the current state of commit logs. The CommitLogMBean also exposes the recover() operation which can be used to restore database state from archived commit log files.

The default settings that control commit log recovery are specified in the conf/commitlog_archiving.properties file, but can be overridden via the MBean. You’ll learn more about data recovery in Chapter 12.

Compaction Manager MBean

We’ve already taken a peek inside the source of the org.apache.cassandra.db.compaction.CompactionManagerMBean to see how it interacts with JMX, but we didn’t really talk about its purpose. This MBean allows us to get statistics about compactions performed in the past, and the ability to force compaction of specific SSTable files we identify by calling the forceUserDefinedCompaction method of the CompactionManager class. This MBean is leveraged by nodetool commands including compact, compactionhistory, and compactionstats.

Cache Service MBean

The org.apache.cassandra.service.CacheServiceMBean provides access to Cassandra’s key cache, row cache, and counter cache under the domain org.apache.cassandra.db > Caches. The information available for each cache includes the maximum size and time duration to cache items, and the ability to invalidate each cache.

Gossiper MBean

The org.apache.cassandra.gms.GossiperMBean provides access to the work of the Gossiper.

We’ve already discussed how the StorageServiceMBean reports which nodes are unreachable. Based on that list, you can call the getEndpointDowntime() operation on the GossiperMBean to determine how long a given node has been down. The downtime is measured from the perspective of the node whose MBean we’re inspecting, and the value resets when the node comes back online. Cassandra uses this operation internally to know how long it can wait to discard hints.

The getCurrentGenerationNumber() operation returns the generation number associated with a specific node. The generation number is included in gossip messages exchanged between nodes and is used to distinguish the current state of a node from the state prior to a restart. The generation number remains the same while the node is alive and is incremented each time the node restarts. It’s maintained by the Gossiper using a timestamp.

The assassinateEndpoint() operation attempts to remove a node from the ring by telling the other nodes that the node has been permanently removed, similar to the concept of “character assassination” in human gossip. Assassinating a node is a maintenance step of last resort when a node cannot be removed from the cluster normally. This operation is used by the nodetool assassinate command.

Failure Detector MBean

The org.apache.cassandra.gms.FailureDetectorMBean provides attributes describing the states and Phi scores of other nodes, as well as the Phi conviction threshold.

Snitch MBeans

Cassandra provides two MBeans to monitor and configure behavior of the snitch. The org.apache.cassandra.locator.EndpointSnitchInfoMBean provides the name of the rack and data center for a given host, as well as the name of the snitch in use.

If you’re using the DynamicEndpointSnitch, the org.apache.cassandra.locator.DynamicEndpointSnitchMBean is registered. This MBean exposes the ability to reset the badness threshold used by the snitch for marking nodes as offline, as well as allowing you to see the scores for various nodes.

Stream Manager MBean

The org.apache.cassandra.streaming.StreamManagerMBean allows us to see the SSTable streaming activities that occur between a node and its peers. There are two basic ideas here: a stream source and a stream destination. Each node can stream its data to another node in order to perform load balancing, and the StreamManager class supports these operations. The StreamManagerMBean gives necessary view into the data that is moving between nodes in the cluster.

The StreamManagerMBean supports two modes of operation. The getCurrentStreams() operation provides a snapshot of the current incoming and outgoing streams, and the MBean also publishes notifications associated with stream state changes, such as initialization, completion or failure. You can subscribe to these notifications in your JMX client in order to watch the streaming operations as they occur.

So in conjunction with the StorageServiceMBean, if you’re concerned that a node is not receiving data as it should, or that a node is unbalanced or even down, these two MBeans working together can give you very rich insight into exactly what’s happening in your cluster.

Messaging Service MBean

As you learned in Chapter 6 the org.apache.cassandra.net.MessagingService manages messages to and from other nodes other than streaming. The MessagingServiceMBean exposes attributes including data about pending and dropped messages, as well as operations to manage backpressure.

Thread Pool MBeans

Cassandra’s thread pools are implemented via the JMXEnabledThreadPoolExecutor and JMXConfigurableThreadPoolExecutor classes in the org.apache.cassandra.concurrent package. The first class The MBeans for each stage implement the JMXEnabledThreadPoolExecutorMBean and JMXConfigurableThreadPoolExecutorMBean interfaces, respectively, which allow you to view and configure the number of core threads in each thread pool as well as the maximum number of threads. The MBeans for each type of thread pool appear under the org.apache.cassandra.internal domain to JMX clients.

Garbage Collection MBeans

The JVM’s garbage collection processing can impact tail latencies if not tuned properly, so it’s important to monitor its performance, as we’ll see in Chapter 13. The GCInspectorMXBean appears in the org.apache.cassandra.service domain. It exposes the operation getAndResetStats() which retrieves and then resets garbage collection metrics that Cassandra collects on its JVM, which is used by the nodetool gcstats command. It also exposes attributes that control the thresholds at which INFO and WARN logging messages are generated for long GC pauses.

Security MBeans

The org.apache.cassandra.auth domain defines the AuthCacheMBean, which exposes operations used to configure how Cassandra caches client authentication records. We’ll discuss this MBean in Chapter 14.

Metrics MBeans

The ability to access metrics related to application performance, health, and key activities has become an essential tool for maintaining web-scale applications. Fortunately, Cassandra collects a wide range of metrics on its own activities to help us understand the behavior. JMX supports several different styles of metrics including counters, gauges, meters, histograms and timers.

To make its metrics accessible via JMX, Cassandra uses the Dropwizard Metrics open source Java library. Cassandra uses the org.apache.cassandra.metrics.CassandraMetricsRegistry to register its metrics with the Dropwizard Metrics library, which in turn exposes them as MBeans in the org.apache.cassandra.metrics domain. You’ll see below in “Metrics” a summary of the specific metrics that Cassandra reports and learn how these can be exposed to metrics aggregation frameworks.

describecluster

The describecluster command prints out basic information about the cluster, including the name, snitch, and partitioner. For example, here’s a portion of the output when run against the cluster created for the Reservation Service using ccm:

$ ccm node1 nodetool describecluster

Cluster Information:
	Name: reservation_service
	Snitch: org.apache.cassandra.locator.SimpleSnitch
	DynamicEndPointSnitch: enabled
	Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
	Schema versions:
		2b88dbfd-6e40-3ef1-af11-d88b6dff2c3b: [127.0.0.4, 127.0.0.3, 127.0.0.2, 127.0.0.1]
...

We’ve shortened the output a bit for brevity. The “Schema versions” portion of the output is especially important for identifying any disagreements in table definitions, or “schema,” between nodes. While Cassandra propagates schema changes through a cluster, any differences are typically resolved quickly, so any lingering schema differences usually correspond a node that is down or unreachable that needs to be restarted, which you should be able to confirm via the summary statistics on nodes that are also printed out.

status

A more direct way to identify the nodes in your cluster and what state they’re in, is to use the status command:

$ ccm node1 nodetool status

Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address    Load        Tokens  Owns (effective)  Host ID                               Rack
UN  127.0.0.1  251.77 KiB  256     48.7%             d23716cb-a6e2-423a-a387-f26f5b623ae7  rack1
UN  127.0.0.2  250.28 KiB  256     50.0%             635f2ab7-e81a-423b-a566-674d8010c819  rack1
UN  127.0.0.3  250.47 KiB  256     53.6%             a1cd5663-02a3-4038-8c34-0ce6de397da7  rack1
UN  127.0.0.4  403.46 KiB  256     47.7%             b493769e-3dce-47d7-bae6-1bed956cf560  rack1

The status is organized by data center and rack. Each node’s status is identified by a two-character code, in which the first character indicates whether the node is up (currently available and ready for queries) or down, and the second character indicates the state or operational mode of the node. The load column represents the byte count of the data each node is holding. The owns column indidates the effective percentage of the token range owned by the node, taking replication into account.

info

The info command tells nodetool to connect with a single node and get basic data about its current state. Just pass it the address of the node you want info for:

$ ccm node2 nodetool info

ID                     : 635f2ab7-e81a-423b-a566-674d8010c819
Gossip active          : true
Native Transport active: true
Load                   : 250.28 KiB
Generation No          : 1574894395
Uptime (seconds)       : 146423
Heap Memory (MB)       : 191.69 / 495.00
Off Heap Memory (MB)   : 0.00
Data Center            : datacenter1
Rack                   : rack1
Exceptions             : 0
Key Cache              : entries 10, size 896 bytes, capacity 24 MiB, 32 hits, 44 requests, 0.727 recent hit rate, 14400 save period in seconds
Row Cache              : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
Counter Cache          : entries 0, size 0 bytes, capacity 12 MiB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds
Chunk Cache            : entries 16, size 256 KiB, capacity 91 MiB, 772 misses, 841 requests, 0.082 recent hit rate, NaN microseconds miss latency
Percent Repaired       : 100.0%
Token                  : (invoke with -T/--tokens to see all 256 tokens)

The information reported includes the memory and disk usage (“Load”) of the node and the status of various services offered by Cassandra. You can also check the status of individual services by the nodetool commands statusgossip, statusbinary, and statushandoff (note that handoff status is not part of info).

ring

To determine what nodes are in your ring and what state they’re in, use the ring command on nodetool, like this:

$ Datacenter: datacenter1
==========
Address    Rack        Status State   Load            Owns                Token
                                                                          9218490134647118760
127.0.0.1  rack1       Up     Normal  251.77 KiB      48.73%              -9166983985142207552
127.0.0.4  rack1       Up     Normal  403.46 KiB      47.68%              -9159867377343852899
127.0.0.2  rack1       Up     Normal  250.28 KiB      49.99%              -9159653278489176223
127.0.0.1  rack1       Up     Normal  251.77 KiB      48.73%              -9159520114055706114
...

This output is organized in terms of vnodes. Here we see the IP addresses of all the nodes in the ring. In this case, we have three nodes, all of which are up (currently available and ready for queries). The load column represents the byte count of the data each node is holding. The output of the describering command is similar but is organized around token ranges.

Other useful status commands provided by nodetool include:

• The getLoggingLevels and setLoggingLevels commands allow dynamic configuration of logging levels, using the Logback JMXConfiguratorMBean we discussed previously.

• The gossipinfo command prints the information this node disseminates about itself and has obtained from other nodes via gossip, while failuredetector provides the Phi failure detection value calculated for other nodes.

• The version command prints the version of Cassandra this node is running.

Using tpstats

The tpstats tool gives us information on the thread pools that Cassandra maintains. Cassandra is highly concurrent, and optimized for multiprocessor/multicore machines, so understanding the behavior and health of the thread pools is important to good Cassandra maintenance.

To find statistics on the thread pools, execute nodetool with the tpstats command:

$ bin/nodetool tpstats
ccm node1 nodetool tpstats

Pool Name                         Active   Pending      Completed   Blocked  All time blocked
ReadStage                              0         0            399         0                 0
MiscStage                              0         0              0         0                 0
CompactionExecutor                     0         0          95541         0                 0
MutationStage                          0         0              0         0                 0
...

Message type           Dropped                  Latency waiting in queue (micros)
                                             50%               95%               99%               Max
READ_RSP                     0              0.00              0.00              0.00              0.00
RANGE_REQ                    0              0.00              0.00              0.00              0.00
PING_REQ                     0              0.00              0.00              0.00              0.00
_SAMPLE                      0              0.00              0.00              0.00              0.00

The top portion of the output presents data on tasks in each of Cassandra’s thread pools. You can see directly how many operations are in what stage, and whether they are active, pending, or completed. For example, by reviewing the number of active tasks in the MutationStage, you can learn how many writes are in progress.

The bottom portion of the output indicates the number of dropped messages for the node. Dropped messages are an indicator of Cassandra’s load shedding implementation, which each node uses to defend itself when it receives more requests than it can handle. For example, internode messages that are received by a node but not processed within the rpc_timeout are dropped, rather than processed, as the coordinator node will no longer be waiting for a response.

Seeing lots of zeros in the output for blocked tasks and dropped messages means that you either have very little activity on the server or that Cassandra is doing an exceptional job of keeping up with the load. Lots of non-zero values is indicative of situations where Cassandra is having a hard time keeping up, and may indicate a need for some of the techniques described in Chapter 13.

Using tablestats

To see overview statistics for keyspaces and tables, you can use the tablestats command. You may also recognize this command from its previous name cfstats. Here is sample output on the reservations_by_confirmation table:

$ ccm node1 nodetool tablestats reservation.reservations_by_confirmation

Total number of tables: 43
----------------
Keyspace : reservation
	Read Count: 0
	Read Latency: NaN ms
	Write Count: 0
	Write Latency: NaN ms
	Pending Flushes: 0
		Table: reservations_by_confirmation
		SSTable count: 0
		Old SSTable count: 0
		Space used (live): 0
		Space used (total): 0
		Space used by snapshots (total): 0
		Off heap memory used (total): 0
		SSTable Compression Ratio: -1.0
		Number of partitions (estimate): 0
		Memtable cell count: 0
		Memtable data size: 0
		Memtable off heap memory used: 0
		Memtable switch count: 0
		Local read count: 0
		Local read latency: NaN ms
		Local write count: 0
		Local write latency: NaN ms
		Pending flushes: 0
		Percent repaired: 100.0
		Bloom filter false positives: 0
		Bloom filter false ratio: 0.00000
		Bloom filter space used: 0
		Bloom filter off heap memory used: 0
		Index summary off heap memory used: 0
		Compression metadata off heap memory used: 0
		Compacted partition minimum bytes: 0
		Compacted partition maximum bytes: 0
		Compacted partition mean bytes: 0
		Average live cells per slice (last five minutes): NaN
		Maximum live cells per slice (last five minutes): 0
		Average tombstones per slice (last five minutes): NaN
		Maximum tombstones per slice (last five minutes): 0
		Dropped Mutations: 0

We can see the read and write latency and total number of reads and writes. We can also see detailed information about Cassandra’s internal structures for the table, including memtables, Bloom filters and SSTables. You can get statistics for all the tables in a keyspace by specifying just the keyspace name, or specify no arguments to get statistics for all tables in the cluster.