During prolonged testing or after the application has gone to production, we need to monitor its performance to make sure the application continues to meet the performance objectives. There may be infrastructure or operational issues impacting the performance or availability of the application, or occasional spikes in latency or dips in throughput. Generally, monitoring alleviates such risk by generating a continuous feedback stream.
Roughly there are three kinds of components used to build a monitoring stack. A collector sends the numbers from each host that needs to be monitored. The collector gets host information and the performance numbers and sends them to an aggregator. An aggregator receives the data sent by the collector and persists them until asked by a visualizer on behalf of the user.
The project metrics-clojure (https://github.com/sjl/metrics-clojure) is a Clojure wrapper over the Metrics (https://github.com/dropwizard/metrics) Java framework, which acts as a collector. Statsd is a well-known aggregator that does not persist data by itself but passes it on to a variety of servers. One of the popular visualizer projects is Graphite that stores the data as well as produces graphs for requested periods. There are several other alternatives to these, notably Riemann (http://riemann.io/) that is written in Clojure and Ruby. Riemann is an event processing-based aggregator.
One of the popular performance monitoring approaches that has emerged in recent times is via logs. The idea is simple—the application emits metrics data as logs, which are shipped from the individual machine to a central log aggregation service. Then, those metrics data are aggregated for each time window and further moved for archival and visualization.
As a high-level example of such a monitoring system, you may like to use Logstash-forwarder (https://github.com/elastic/logstash-forwarder) to grab the application logs from the local filesystem and ship to Logstash (https://www.elastic.co/products/logstash), where it forwards the metrics logs to StatsD (https://github.com/etsy/statsd) for metrics aggregation or to Riemann (http://riemann.io/) for events analysis and monitoring alerts. StatsD and/or Riemann can forward the metrics data to Graphite (http://graphite.wikidot.com/) for archival and graphing of the time-series metrics data. Often, people want to plug in a non-default time-series data store (such as InfluxDB: https://influxdb.com/) or a visualization layer (such as Grafana: http://grafana.org/) with Graphite.
A detailed discussion on this topic is out of the scope of this text, but I think exploring this area would serve you well.
If you develop web software using Ring (https://github.com/ring-clojure/ring) then you may find the Ring extension of the metrics-clojure library useful: http://metrics-clojure.readthedocs.org/en/latest/ring.html —this tracks a number of useful metrics that can be queried in JSON format and integrated with visualization via the web browser.
To emit a continuous stream of metrics data from the web layer, Server-Sent Events (SSE) may be a good idea due to its low overhead. Both http-kit (http://www.http-kit.org/) and Aleph (http://aleph.io/), which work with Ring, support SSE today.
Both Oracle JDK and OpenJDK provide two GUI tools called
JConsole (executable name jconsole) and JVisualVM (executable name jvisualvm) that we can use to introspect into running JVMs for instrumentation data. There are also some command-line tools (http://docs.oracle.com/javase/8/docs/technotes/tools/) in the JDK to peek into the inner details of the running JVMs.
A common way to introspect a running Clojure application is to have an nREPL (https://github.com/clojure/tools.nrepl) service running so that we can connect to it later using an nREPL client. Interactive introspection over nREPL using the Emacs editor (embedded nREPL client) is popular among some, whereas others prefer to script an nREPL client to carry out tasks.
The JVM has a built-in mechanism to introspect managed resources via the extensible Java Management Extensions (JMX) API. It provides a way for application maintainers to expose manageable resources as "MBeans". Clojure has an easy-to-use contrib library called java.jmx (https://github.com/clojure/java.jmx) to access JMX. There is a decent amount of open source tooling for visualization of JVM instrumentation data via JMX, such as jmxtrans and jmxetric, which integrate with Ganglia and Graphite.
Getting quick memory stats of the JVM is pretty easy using Clojure:
(let [^Runtime r (Runtime/getRuntime)] (println "Maximum memory" (.maxMemory r)) (println "Total memory" (.totalMemory r)) (println "Free memory" (.freeMemory r))) Output: Maximum memory 704643072 Total memory 291373056 Free memory 160529752