In this first recipe, you will create a data model for our web access logs. You will use Splunk's Data Model Editor to do this and define a number of object types, and add constraints and attributes.
To step through this recipe, you will need a running Splunk Enterprise server, with the sample data loaded from Chapter 1, Play Time – Getting Data In. You should also have completed the recipes from the earlier chapters. You should be familiar with navigating the Splunk user interface.
Follow the steps in this recipe to create the Web Access data model:
- Log in to your Splunk server.
- Select the Operational Intelligence application.
- Select the Settings menu item at the top-right corner of the screen, and then select Data models.
- An empty list of data models to manage will load. Click on the New Data Model button located at the top-right corner of the screen.
- A pop-up box will be displayed. Enter
Web Accessin the Title field and as you type, the ID will automatically populate. Ensure the selected app is Operational Intelligence and then click on the Create button.
- This will create an empty data model, and the Data Model Editor will be displayed. Next, create an object type. Select the Add Object drop-down and select Root Event.
- The Add Event Object page will be displayed. Enter
All Web Accessin the Object Name field, and in the Constraints box, enter the search syntaxindex=main sourcetype=access_combined. Once these have been entered, click on the Preview button and a few web access log events will be displayed in the preview area. Following this, click on Save to save the event object type.
- After saving the object type, the Data Model Editor will be displayed with the newly created All Web Access object displayed. Some inherited attributes will be seen on the right-hand side. You will now add a few more. Click on the Add Attribute drop-down and select Auto-Extracted.
- A pop-up box will be displayed with all the fields that Splunk has already extracted for the web access logs. You can select specific fields for the model, but to keep things simple, let's select them all by checking the checkbox next to the Field column heading at the top of the field list. Note that all the fields will then get checked. Once complete, click on Save and all these fields will now become attributes for the data model object.
- You will now add an evaluated attribute to add a status category. Click on the Add Attribute drop-down again, but this time select Eval Expression.
- The Add Attributes with an Eval Expression screen will be displayed. In the Eval Expression box, enter the following search syntax to match the status code in the events to a descriptive category:
case(like(status,"1%"),"Informational", like(status,"2%"),"Success", like(status,"3%"),"Redirect", like(status,"4%"),"Client Error", like(status,"5%"),"Server Error")
- Enter
status_categoryin the Field Name field and leave all the other fields with the defaults. Then click on the Preview button. You should see the newstatus_categoryfield populated in the preview results. Click on Save to save this newly evaluated attribute.
- You will now use this newly created
status_categoryobject attribute to create a couple of child object types for error and success events. From the Data Model Editor, select the Add Object drop-down again, but this time select Child.
- The Add Child Object screen will be displayed. Enter
Successin the Object Name field and in the Additional Constraints box, enter the search syntaxstatus_category="Success". Click on Preview to confirm that the results are displayed, and then click on Save to save the new child object type.
- After saving the object, you will be back in the Data Model Editor and the new
Successchild object will be seen underneath the root level All Web Access event object. Click on the All Web Access Event Object, and then repeat steps 13 and 14, adding another child object type namedErrorwith Additional Constraints ofstatus_category="Client Error" OR status_category="Server Error". - Once complete, you should see two child objects, named Success and Error, underneath the root event object named All Web Access.
Our initial data model is now complete—congrats!
In this recipe, you started off by creating a new data model for our web access dataset. After the initial data model was created, you added a root-level event object type, named All Web Access to sit at the top of the object hierarchy. This event object allows for simple constraints and you created an object constraint that constrained the object to only web access logs. Following this, you added object attributes to the object, consisting of all the autoextracted fields that Splunk already knew about, in addition to an evaluated expression attribute to categorize the various status codes in the event data. You then used this newly created status_category evaluated attribute to create child object types for Success and Error events.
Behind the scenes, Splunk is essentially creating a Splunk search to report on the dataset that is being modeled. The constraint provided actually tells Splunk what data to look at, and the attributes are basically the fields within the data that Splunk will search. The Success and Error child object types inherit all the attributes and constraints from their parent All Access Logs object and act as further filters for the backend search that Splunk creates.
Once the data models are built, they provide the search time mapping needed by the Pivot tool for reporting. However, it is also possible to view the underlying dataset mapped by data models and their objects using the Splunk search interface.
The Splunk
datamodel command allows for the searching of the dataset mapped by data models and their associated objects directly from the Splunk search interface. In order to be used, the command must be the first command used in your Splunk search.
Navigate to the search bar in the Operational Intelligence application and enter the following search to see a list of all the data models in the application:
| datamodel
If we have multiple data models, you can filter to just the Web Access data model, by entering the following search:
| datamodel Web_Access
The data returned is in JSON format and you are able to expand objectNameList to see all the objects within the data model. To see data related to the All Web Access object, enter the following search:
| datamodel Web_Access All_Web_Access
Here you can actually see the underlying search that has been written by Splunk for the data model object—it is pretty large! It might be large but it is not that complicated. There are actually a lot of renames and evals taking up most of the search, which give a rather intimidating appearance.
To search the data in this object, enter the following search:
| datamodel Web_Access All_Web_Access search
Note how all the data is now displayed and all the object attributes you declared in the object type are represented as fields in the field sidebar. All these fields begin with the name of the object, in this case, All_Web_Access.fieldname, which is not terribly useful, so we can remove this by entering the following search:
| datamodel Web_Access All_Web_Access search | rename All_Web_Access.* AS *
Now we can basically use this data as we would use any Splunk search. So, to see all the error events by status_category, you would enter the following search:
| datamodel Web_Access All_Web_Access search | rename All_Web_Access.* AS * | search is_Error=1 | stats count by status_category
So why would you want to take this search approach over searching the data directly? Well, you can actually use data models to do a bit of heavy lifting for you in terms of correlating datasets and adding additional calculated fields, and so on. Therefore, there may be times when this approach to searching the data does indeed make a lot of sense. Additionally, even when data model acceleration is turned on, the datamodel command does not currently use the acceleration. However, the pivot command (more on this later) does use acceleration, but is not as useful.
Note
For more information on the datamodel command, you should review the search reference at http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Datamodel.