A Ceph storage administrator will perform most of the cluster monitoring using CLI via commands provided by the Ceph interface. Ceph also provides a rich interface for admin APIs that can be used natively to monitor the entire Ceph cluster. There are a couple of open source projects that make use of Ceph's REST admin API and represent the monitoring results in a GUI dashboard where you can have a quick look at your entire cluster. We will now take a look at such open source projects and their installation procedures.
Kraken is an open source Ceph dashboard written in Python for stats and monitoring of a Ceph cluster, initially developed by Donald Talton, and later joined by David Moreau Simard.
Donald is the owner of Merrymack, an IT consulting company. He is an expert engineer with over 20 years of experience and has worked for companies such as Apollo Group, Wells Fargo, PayPal, and Cisco. During his tenure in PayPal and Cisco, he focused primarily on OpenStack and Ceph. During the time he was working for Cisco, he kickstarted the development of Kraken. Fortunately, Donald is one of the technical reviewers of this book, too.
David Simard started his career in 2006 when he was studying in college; he started working for iWeb, a web-hosting company, as a temporary employee during his summer holidays; later, when his job got converted to a full-time position, he had no other option than to leave his studies and continue working on amazing stuff at iWeb. It has now been 8 years since he started working with iWeb as an IT architecture specialist. He deals with cloud storage, cloud computing, and other interesting fields.
There are a few key reasons behind the development of Kraken. The first is that when it was conceived, Ceph's Calamari tool was only available to commercial customers of Inktank. Donald believes that it's necessary to have a good open source dashboard to monitor the Ceph cluster and its components from a single window; this will lead to better management and speed up the adoption of Ceph as a whole. He took this as a challenge and kicked off the development of Kraken. Donald decided to use ceph-rest-api to extract all the necessary cluster data for monitoring and reporting. To converge all this cluster data in a presentable dashboard format, Donald used several other tools such as Python, Django, humanize, and python-cephclient.
The roadmap for Kraken has been divided into several milestones. Currently, Kraken is at its first milestone, which consists of the following feature sets:
- Cluster data usage
- The MON status
- The OSD status
- The PG status
- Better user interface
- Support for multiple MONs
The next development stage of Kraken will be the inclusion of operational changes for OSD, live CRUSH map configuration, Ceph user authentication, pool operations, block device management, and system metrics such as CPU and memory usage. You can follow the Kraken roadmap on the GitHub page at https://github.com/krakendash/krakendash or on Kraken's readme file. Kraken is fully open source and follows BSD-licensing. Developers who want to contribute to Kraken can send a pull request to Donald and can contact him at <[email protected]>.
The building blocks of Kraken consist of several open source projects such as:
- Python 2.7 or later: This is required for libraries such as collections.
- ceph-rest-api: This is included with Ceph binaries.
- Django 1.6.2 or later: This is the core framework for Kraken.
- humanize 0.5 or later: This is required for data-free display conversion.
- python-cephclient 0.1.0.4 or later: Recently used in Kraken, it is the client wrapper for ceph-rest-api. In the previous releases, Kraken made use of ceph-rest-api directly without any wrapper. This has been written by David to wrap ceph-rest-api, which gives us additional capabilities, such as being able to support multiple clusters in the future.
- djangorestframework 2.3.12 or later: This is used for some custom API additions that might or might not stay in Kraken.
- django-filter 0.7 or later: This is required by Django.
In this section, we will learn how to deploy Kraken to monitor your Ceph cluster. It is a lightweight application that requires a significantly lower amount of system resources. In this deployment, we will use the ceph-node1 machine; you can use any Ceph cluster node that has access to a Ceph cluster. Follow the listed steps:
- Install dependencies for Kraken, such as python-pip, screen, and the Firefox browser, using the following command. If you have any other browser, you can skip installing Firefox packages. Python-pip is a package manager used to install Python packages, which is required to install dependencies. Kraken will use separate screen sessions to initiate subprocess required for dashboards; these screens will be provided by the screen package:
# yum install python-pip screen firefox - Install the required development libraries:
# yum install gcc python-devel libxml2-devel.x86_64 libxslt-devel.x86_64 - Create a directory for Kraken:
# mkdir /kraken - Clone the Kraken repository from GitHub:
# git clone https://github.com/krakendash/krakendash - Use the Python package manager to install the required packages for Kraken, such as Django, python-cephclient, djangorestframework, markdown, and humanize:
# cd krakendash # pip install -r requirements.txt
- Once these packages are installed, execute
api.shanddjango.sh, which will invoke the ceph-rest-api and django python dashboards, respectively. These scripts will execute in independent screen environments; you can use screen commands to manage these sessions. Press Ctrl + D to detach screen sessions and move them to the background:# cp ../krakendash/contrib/*.sh . # ./api.sh # ./django.sh
- You can check the screen sessions using the
pscommand and reattach to the screen session using the-rcommand:# ps -ef | grep -i screenThe output is shown in the following screenshot:
- Finally, when
api.shanddjango.share running, open your web browser and navigate tohttp://localhost:8000/; you should be able to view your Ceph cluster status on the Kraken dashboard:
The ceph-dash is another free open source dashboard/monitoring API for a Ceph cluster, which has been developed by Christian Eichelmann, who is working fulltime for 1&1 Internet AG, Germany as a senior software developer. Christian started the development of this project at the time when there were very few open source dashboards available for Ceph. Moreover, the other dashboards that were available had complex architectures and did not work well with large clusters. So, Christian focused on developing a simple REST API-based dashboard that allows cluster monitoring via simple REST calls that should work well with large Ceph clusters.
The ceph-dash tool has been designed with a keep-it-simple approach to provide an overall Ceph cluster health status via a RESTful JSON API as well as web GUI. It is a lightweighted application that does not have any dependencies on ceph-rest-api. It is a pure Python wsgi application that talks to the cluster directly via librados. Currently, ceph-dash provides a clean and simple web GUI that is able to show the following information about the Ceph cluster:
- The overall cluster status with detailed problem description
- Support for multiple monitors and the status of every monitor
- The OSD status with count for IN, OUT, and unhealthy OSD
- Graphical storage capacity visualization
- The current throughput, including writes/second, reads/second, and operations/second
- Graphical placement group status visualization
- Cluster recovery state
In continuation to this, ceph-dash also provides REST endpoint that generates all the cluster information in JSON format, which can further be used in various creative ways. Since ceph-dash is an open source project, anyone can contribute by sending pull requests to Christian via https://github.com/Crapworks/ceph-dash.
If you run ceph-dash tests/development purposes, you can run it independently. For production usage, it is strongly recommended to deploy the application on a wsgi capable web server (Apache, nginx, and so on). The ceph-dash tool uses Flask microframework and ceph-python bindings to connect directly to the Ceph cluster. The access to Ceph cluster by ceph-dash is purely read only and doesn't require any write privileges. The ceph-dash tool uses the ceph status command via the Python Rados class. The returned JSON output is then either exposed via the REST API or via a web GUI that refreshes itself every 5 seconds.
In this section, we will learn how to deploy ceph-dash for a Ceph cluster. Proceed with the following steps:
- The ceph-dash tool must be installed on a machine that has access to a Ceph cluster. Since it is not resource hungry, you can designate any of your monitor machines for it.
- Create a directory for ceph-dash and clone its repository from GitHub:
# mkdir /ceph-dash # git clone https://github.com/Crapworks/ceph-dash.git
- Install python-pip:
# yum install python-pip - Install Jinja2 package:
# easy_install Jinja2 - Once the installation is complete, you are good to launch the ceph-dash GUI. To start ceph-dash, execute:
# ./ceph-dash.pyThe output is shown in the following screenshot:
- Open the web browser, point it to
http://localhost:5000/, and start monitoring your cluster using ceph-dash:
Calamari is the management platform for Ceph, an attractive dashboard to monitor and manage your Ceph cluster. It was initially developed by Inktank as a proprietary software that ships with the Inktank Ceph Enterprise product for their customers. Just after the acquisition of Inktank by Red Hat, it was open sourced on May 30, 2014 by Red Hat. Calamari has several great feature sets, and its future roadmap is quite impressive. Calamari has two parts and each part has its own repositories.
Fronted is the browser-based graphical user interface that is majorly implemented in JavaScript. The fronted part makes use of Calamari's REST API and is constructed in a modular approach so that each component of the fronted can be updated or can undergo maintenance independently. The Calamari frontend has been open sourced with an MIT license. You can find the repository at https://github.com/ceph/calamari-clients.
Calamari backend is the core part of the platform, which is written in Python. It also makes use of other components such as SaltStack, ZeroRPC, graphite, djangorestframework, Django, and gevent, and provides a new REST API for integration with Ceph and other systems. Calamari has been reinvented in its new version, where it uses the new Calamari REST API to interact with Ceph clusters. The pervious release of Calamari uses the Ceph REST API, which is a bit restrictive for this purpose. The Calamari backend has been open sourced with the LGPL2+ license; you can find the repository at https://github.com/ceph/calamari.
Calamari has a good documentation available at http://calamari.readthedocs.org. Whether you are a Calamari operator, a developer working on Calamari, or a developer using the Calamari REST API, this documentation is a good source of information to get you started with Calamari. Like Ceph, Calamari has also been developed upstream; you can get involved with Calamari on IRC irc://irc.oftc.net/ceph by registering to the mailing list <[email protected]> or sending pull requests on Calamari GitHub accounts at https://github.com/ceph/calamari and https://github.com/ceph/calamari-clients.
If you want to install Calamari and are curious to see what it looks like, you can follow my blog about Calamari's step-by-step installation at http://karan-mj.blogspot.fi/2014/09/ceph-calamari-survival-guide.html.