Splunk has a rich SDK that allows you to create custom dashboards with extended functionality. The aim of this chapter is to show you how to create a custom dashboard using data stored in Hadoop. We are going to visualize data containing aggregated customer data records on Google Maps. The goal is to show a customer activity heatmap.
Splunk has various SDKs for different languages and platforms. We are going to talk about them shortly and won't cover deep application development. Our goal is to get the result as rapidly as possible and using few application development skills. We can always turn to custom development, and our goal is to get quick insights as soon as possible.
There is a nice portal—https://splunkbase.splunk.com—where you can find hundreds of published applications. Extending Splunk with a SDK is not something special; it's a recommended approach to get extended functionality. This book has a chapter describing integration with MongoDB. The MongoDB integration application is taken from the splunkbase portal. We encourage you to look through published apps before starting to develop your own.
You can use several languages to create custom applications for Splunk:
- Python
- Java
- JavaScript
- PHP
- Ruby
- C#
Great, we have a set of popular languages; it shouldn't be a problem to find guys who knows a least one of the listed languages.
Splunk SDK could be used for these purposes:
- Integrating with third-party software
- Logging directly to Splunk
- Running search queries and getting results to display in your custom application
- Building custom UIs and other features
Splunk provides a REST API for its services. Generally, it means that you can interact with Splunk components using curl/wget or any other tool or language library that can send a correct HTTP request. Splunk provides so-called endpoints for services. You can call Splunk using a GET request:
search/jobs/export
To stream search results. Any Splunk component has its own endpoint. The Splunk SDK for the languages listed earlier simplifies interaction with the service endpoints. Each language SDK provides bindings for API endpoints and a level of abstraction over HTTP calls. This approach is common; for example, Cloudera manager API provides a Java and Python SDK. These SDKs are just wrappers for REST services of Cloudera manager. The good point is that you can integrate with Splunk and you don't have to be tightly coupled with Ruby or C#, for example. You can even call services using C++.
We are going to cover these topics in order to develop a custom dashboard with heatmaps based on aggregated customer data records collected in Milano, Italy:
- Learn how raw data looks like and what properties it has
- Create a data sample using Pig to make development iteration shorter
- Query our data using Splunk's query language to see if data presented correctly and the query returns the expected results
- Meet the Splunk JS SDK API to extend basic functionality in Splunk and visualize the heatmap results
Application development using Splunk SDK would make a separate book, which is why we only touch on it briefly here. Let's summarize the good points:
- Splunk provides access from various popular languages.
- Splunk services have endpoints and we can reach them using the REST API with the help of
curl/wgetor any other tool that sends HTTP requests. - SDK wraps the REST API internally to simplify interaction with Splunk services. It would be the best choice for production implementation.
Now it's time to move on to dashboard visualization using the Splunk JS stack. This should help us to reach our goal: getting results using minimal application development skills.