The app key-value store is a feature provided by Splunk Enterprise to manage and maintain the state of the application. Using an app key-value store, users can save and retrieve data from Splunk apps.
The app key-value store feature is only available in the 64-bit distribution of Splunk Enterprise. It is not available in the 32-bit version of Splunk. It uses the 8191 port by default, but it can be configured from Server.conf located at $SPLUNK_HOMEetcsystemlocal by modifying the [kvstore] code block.
The following are some of the uses of a key-value store:
- It can be used to manage the app state of the user interface by storing the session/application state information
- It creates a checkpoint of the uploaded data in case of modular inputs
- It enlists the environment variable used, accessed, or modified by users
- It is the metadata storage of the user
- It caches results from search queries
The key-value store saves data in the collections of the key-value pair. The key-value store files are located on the search heads. The following are the various components of the key-value store:
- Collections: Collections are containers for data storage similar to a database table.
- Records: Records store the entry of data in the collection.
- Fields: Fields contain the value of data in the JSON format file. Fields correspond to the key name similar to columns in the database table.
- _key: This is the reserved field that contains a unique ID for each record. It is an autogenerated field that is not explicitly specified.
- _user: This is also a reserved field that is used to map the user ID of each record.
- Accelerations: This is used to improve search performance that contains the accelerated fields.
Let's take a look at how to create a key-value store collections via a config file. To use a key-value store, we need to create a key-value store collection using the following steps:
- Create a
collections.conffile in the application'sdefaultorlocaldirectory, as follows$SPLUNK_HOMEetcappsAPPNAMEdefaultcollections.confor$SPLUNK_HOMEetcappsAPPNAMElocalcollections.conf. - Modify
collections.confby specifying the name of the collection and optionally, the schema of the data. Listed in the following sublist is the description of the parameters which need to be configured incollections.conffile:[collection_name]: This is the collection nameenforceTypes: This is set toTrueorFalseto enforce the data types of values when inserting them into the collection.field.name: This is an optional field. The available data types are string, time, Boolean, and number. If the data type is not set explicitly, then it is set to JSON.
Any change in collections.conf needs a restart of the Splunk instance to apply the changes on the search heads. Refer to the following example for better understanding:
[AndroidCollections] #collection_name
The screenshot that follows shows a code snippet of the sample JSON data:
The following screenshot is the code snippet of the enforce data type for the preceding JSON data:
The following screenshot shows the sample code snippet for hierarchical JSON data:
The following screenshot shows how a data type can be enforced on hierarchical data using a dot (.) notation:
The Splunk REST API can be used to create, read, delete, update, and manage key-value store data and collections. The Splunk REST API accesses Splunk via the management port (by default, 8089). The following are the REST endpoints for the key-value store:
storage/collections/config:GET: This fetches a list of collections in a specific appPOST: This creates a new collection in a specific app
storage/collections/config/{collection}:GET: This fetches information about a specific collectionDELETE: This deletes a collectionPOST: This updates a collection
storage/collections/data/{collection}:GET: This fetches records from a specific collectionPOST: This inserts a new record into a specific collectionDELETE: This deletes all records from a specific collection
storage/collections/data/{collection}/{id}:GET: This fetches records in a collection by a key IDPOST: This updates records in a collection by a key IDDELETE: This deletes a record in a collection by a key ID
storage/collections/data/{collection}/batch_save:
There are various notations used in the following examples, such as username, password, IPAddress, and others. Users need to replace them with their own corresponding values to execute the examples. The following are the examples:
- Fetching a list of collections for an android app:
curl -k -u username:password https://IPAddress:8089/servicesNS/nobody/android/storage/collections/config
- Creating a new collection called AndroidCollections in the android app:
curl -k -u username:password -d name= AndroidCollections https://IPAddress:8089/servicesNS/nobody/android/storage/ collections/config
- Defining a collection schema:
curl -k -u username:password https://IPAddress:8089/servicesNS/nobody/android/storage/ collections/config/ AndroidCollections -d field.Devicename = string -d field.DeviceID = number -d field.DeviceInfo.DeviceBuild = string -d field.DeviceInfo.DeviceAndroidVersion = string
- Adding data of the hierarchical JSON format to a collection:
curl -k -u username:password https://IPAddress:8089/servicesNS/nobody/android/storage/ collections/config/ AndroidCollections -H 'Content-Type: application/json' -d '{ "Devicename" : "Test Device", "DeviceID" : 9661, "DeviceInfo" : { "DeviceBuild" : "Test build 9661C", "DeviceAndroidVersion" : "Marshmallow 6.0", "DeviceIMEI" : 12345678909876, "DeviceMAC" : "AA:BB:CC:DD:EE:FF" }} ' - Getting all data from the collection:
curl -k -u username:password https://IPAddress:8089/servicesNS/nobody/android/storage/ collections/config/ AndroidCollections
- Getting a specific range of records from collections, for example, records from
10to15:curl -k -u username:password https://IPAddress:8089/servicesNS/nobody/android/storage/ collections/config/ AndroidCollections?sort=Devicename&skip=10&limit=5
- Getting a record of a specific key ID:
curl -k -u username:password https://IPAddress:8089/servicesNS/nobody/android/storage/ collections/config/ AndroidCollections/KEYID
Where the key ID is the unique
_keyof collections for which the record is to be fetched. - Deleting the record of the specific key ID:
curl -k -u username:password –X DELETE https://IPAddress:8089/servicesNS/nobody/android/storage/ collections/config/ AndroidCollections/KEYID
- Deleting all records of the AndroidCollections collection:
curl -k -u username:password –X DELETE https://IPAddress:8089/servicesNS/nobody/android/storage/ collections/config/ AndroidCollections
In case of a distributed environment, the key-value store can be replicated to a large number of search heads by enabling replication. By default, the key-value store is not replicated to indexers in distributed deployment of Splunk.
To enable replication, the collections.conf file is to be modified and we need to add replicate = true to the file.