Elasticsearch exposes a rich set of APIs, known as stats APIs, to monitor Elasticsearch at the cluster, node, and indices levels. Some of these APIs are _cluster/stats, _nodes/stats, and myindex/stats. These APIs provide state/monitoring information in real time, and the statistics that are presented in these APIs are point-in-time and in .json format. As an administrator/developer, when working with Elasticsearch, you will be interested in both real-time statistics as well as historical statistics, which would help you in understanding/analyzing the behavior (health or performance) of a cluster better.

Also, reading through a set of numbers for a period of time (say, for example, to find out the JVM utilization over time) would be very difficult. Rather, a UI that pictorially represents these numbers as graphs would be very useful for visualizing and analyzing the current and past trends/behaviors (health or performance) of the Elasticsearch cluster. This is where the monitoring feature of X-Pack comes in handy. 

The X-Pack monitoring components allow you to easily monitor the Elastic Stack (Elasticsearch, Kibana, and Logstash) from Kibana. X-Pack consists of a monitoring agent that runs on each of the instances (Elasticsearch, Kibana, and Logstash) and periodically collects and indexes the health and performance metrics. These can then be easily visualized using the Monitoring UI component of Kibana. The Monitoring UI of Kibana comes with predefined dashboards that let you easily visualize and analyze real-time and past performance data.

By default, the metrics collected by X-Pack are indexed within the cluster you are monitoring. However, in production, it is strongly recommended to have a separate, dedicated cluster to store these metrics. A dedicated cluster for monitoring has the following benefits:

• Allows you to monitor multiple clusters from a central location
• Reduces the load and storage on your production clusters since the metrics are stored in a dedicated monitoring cluster
• There is access to Monitoring, even when some clusters are unhealthy or down
• Separate security levels from Monitoring and Production Cluster can be enforced:

As we mentioned previously, the metrics collected by X-Pack are indexed within the cluster you are monitoring. If a dedicated monitoring cluster is set up, then we need to configure where to send/ship the metrics to in the monitored instances. This can be configured in the elasticsearch.yml file of each node, as shown in the following code: 

xpack.monitoring.exporters:
    id1:
    type: http
    host: ["http://dedicated_monitoring_cluster:port"]