Big data analytics is a very popular trend. As a result, most business users want to discover their big data using intuitive and user-friendly tools because exploring data stored in Hadoop or any NoSQL data stores is a challenging task. Fortunately, Hunk does away with all the complexity obstructing analysts and business users. Moreover, it gives additional features that allow us to handle big data in just several mouse clicks. This is possible with Hunk knowledge objects.
In the previous chapter, we created virtual indexes based on web logs for the international fashion retailer Unicorn Fashion. We created some queries and reports via Search Processing Language (SPL). Moreover, we created a web operation dashboard and learnt how to create alerts.
In this chapter, we will explore Hunk knowledge objects, which will help us to achieve better results with less effort. Moreover, we will become familiar with pivots and data models, in order to learn how to work with Hunk with the traditional Business Intelligence (BI) tool.
Hunk has the same capabilities as Splunk; as a result we can create various knowledge objects that can help us explore big data and make it more user-friendly.
To work with knowledge objects, go to the KNOWLEDGE menu under Settings:
There are various knowledge objects available in Hunk. We encountered SPL, reports, dashboards, and alerts in the previous chapter. Let's expand our knowledge of Hunk and explore additional knowledge objects.
Tip
For more information about knowledge objects, see: http://docs.splunk.com/Documentation/Splunk/latest/Knowledge/WhatisSplunkknowledge.
Field aliases help us to normalize data over several sources. We can create multiple aliases in one field.
Let's create a new alias using the following steps:
- Go to Settings | Fields | Field aliases.
- Click on Add new.
- Enter the Name as
Web Browser, type the sourcetype asaccess_combined, create this alias under Field aliases:useragent=web_browser, and click Save—as shown in the following screenshot:
- Change sharing permissions to This app only (search).
- Go to search and run:
index="digital_analytics". - Then look at the fields. There is a new field—
web_browser:
Moreover, we can create the same alias web_browser for any other data source. For example, we could have other logs, where instead of useragent we could just have agent. In this case, we can create a new alias that will map agent as web_browser. As a result, we create one alias for two different fields from various data sources.
A calculated field acts as a shortcut for performing repetitive, long, or complex transformations using the eval command.
For example, say we want to monitor bandwidth usage in megabytes but we have all our data in bytes. Let's create a new field to convert bytes to megabytes:
- Go to Settings | Fields | Calculated fields.
- Click on Add new.
- Type the sourcetype as
access_combined, the Name asbandwidth, and Eval expression asbytes/1024/1024. Click on Save:
- Change the sharing permissions for the new field to This app only (search).
- Go to search and run the new query in order to measure the bandwidth across countries and find the most popular countries:
index="digital_analytics" | iplocation clientip |stats sum(bandwidth) by Country | sort – sum(bandwidth)
As a result, we got the top countries and their bandwidth in megabytes and used the new calculated field in a search like any other extracted field.
Field extractions are a special utility that helps us create custom fields. It generates a regular expression that pulls those fields from similar events. We can extract fields that are static and often needed in searches using Interactive Field Extractor (IFX). It is a very useful tool that:
- Has graphical UI
- Generates regex for us
- Ensures extracted fields persist as a knowledge object
- Is reusable in multiple searches
Let's try to extract new fields from our digital data set:
- Run a search using the following query:
index="digital_analytics" - Select Extract Fields from the Event actions menu as shown in the following screenshot:
- The new window will appear; we can highlight one or more values in the sample event to create fields. In our case, we want to extract just the name of the browser without the version or any other information. We should highlight the name of browser, give the name, and click on Add Extraction:
- The next step is validation. We can take a quick look at how Hunk extracted the browser name from other events:
- Click Next, change the app permission as usual for the app, and click on Finish.
- We can check the result; just run a new query in the search app:
index="digital_analytics" | stats count by browser_name
Moreover, there is another way to extract fields during the search using the rex and erex commands.
Tip
You can learn more about the rex and erex commands, with examples, at: http://docs.splunk.com/Documentation/Splunk/6.2.2/SearchReference/Erex and http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Rex.
Tags are like nicknames that you create for related field/value pairs. They can make our data more understandable and less ambiguous. It is possible to create several tags for any field/value combination.
Let's create tags for our data set:
- Run a search with the following query:
index="digital_analytics" - Click on the arrow for event details. Then, under action click on the down arrow and select Edit Tags for the action field:
- Name the tag Checkout and click on Save.
- Let's check our new tag. Run a new query:
index="digital_analytics" tag="Checkout"
We get the following result with our new tag:
An event type is a method of categorizing events based on a search; in other words, we can create a group of events based on common values. Let's look at the following example in order to better understand how this works. We can create a new event to:
- Categorize specific key/value pairs
- Classify search strings
- Tag event types to organize data in categories
- Identify fields we can report on
For example, say the sales team wants to track monthly online sales. They want to easily identify purchases that are categorized by item. Let's create a new event type for coats:
- Run a search using the following query:
index="digital_analytics" action=purchase productName=COATS - Click on Save As | Event Type:
- Type the name as
Purchase Coats. In addition, we can create a new tag and choose the color and priority. Then click on Save. - Go to Settings | Event Type and change permissions for our event type as usual.
- We can check this by running a new query:
index="digital_analytics" action=purchase. - There will be a new and interesting field:
eventype. As a result, we can group our events in custom groups using tags and event types.
Workflow actions launch from fields and events in our search results in order to interact with external resources or narrow our search. The possible actions are:
- GET: This is used to pass information to an external web resource
- POST: This is used to send field values to an external resource
- Search: This uses field values to perform a secondary search
For example, organizations often need to track ongoing attempts by external sources trying to log in with invalid credentials. We can use a GET workflow action that will open a new browser window with information about the source IP address.
Tip
For more information about workflow actions in the Splunk knowledgebase with detailed explanations and examples, see: http://docs.splunk.com/Documentation/Splunk/latest/Knowledge/CreateworkflowactionsinSplunkWeb.
Macros are useful when we frequently run searches with a similar search syntax. It can be a full search string or a portion of a search that can be reused in multiple places. In addition, macros allow us to define one or more arguments within the search segment.
Let's create macros with an argument:
- Go to Settings | Advanced search | Search macros.
- Click Add new and type the name as
activitybycategory(2). - Enter the search string:
index="digital_analytics" action=$action1$ AND productName=$Name1$ | stats count by product_name - In the Arguments field type these arguments:
action1, Name1. We should get the following:
- Let's try to run our macros. Type in a search and run it:
activitybycategory(purchase,COATS) - As a result, we get a result that is similar to running an ordinary search:
index="digital_analytics" action=purchase AND productName=COATS | stats count by productName
A data model is a hierarchically structured data set that generates searches and drives a pivot. (A pivot is an interface in which we can create reports based on data models. Soon we will explore pivots more closely.) In other words, data models provide a more meaningful representation of underlying raw machine data.
Data models are designed to make it easy to share and reuse domain knowledge. The idea is that admins or powerusers create data models for non-technical users, who interact with data via a user-friendly pivot UI.
Let's create a data model for our digital data set.
- Go to Settings | Data Models.
- Click on New Data Model.
- In the Title type
Unicorn Fashion Digital Analyticsand click on Create. A new data model will be created. - Click Add Object and choose Root Event. There are four types of objects:
- Event objects—a set of events
- Transaction objects—transactions and groups of events
- Search objects— the result of an arbitrary search
- Child objects—a subset of the dataset connected by their parent object
- Type in the Object Name as
Digital Dataand Constraints asindex=digital_analytics sourcetype=access_combined. Click on Save:
- Moreover, we can add child events in order to create predefined events with additional constraints.
We successfully added a root event and now we can add fields that Hunk can extract automatically. Let's do it:
- Select the Digital Data object.
- Click Add Attribute and select Auto-Extracted. A new window will come up displaying all auto-extracted fields. Check the checkbox to the left of the Field column header in order to select all extracted fields. In addition, we can easily rename some fields.
- Change the clientip type to IPV4.
- Click on Save:
There are four types of attribute in Hunk:
- String: With this, field values are recognized as alpha-numeric.
- Number: With this, field values are recognized as numeric.
- Boolean: With this, field values are recognized as true/false or 1/0.
- IPV4: With this, field values are recognized as IP addresses. This field type is useful for geo data because Hunk can easily extract geo data from the IP address.
Moreover, there are also four types of attribute flag:
- Required: Only events that contain this field are returned in the pivot
- Optional: This field doesn't have to appear in every event
- Hidden: This field is not displayed to pivot users when they select an object in the pivot
- Hidden & Required: only events that contain this field are returned, and the fields are hidden from use in the pivot
In order to add GeoIP attributes we should have a latitude and longitude lookup table or GeoIP mapping fields.
Let's add GeoIP attributes:
- Click on Add Attribute and choose GeoIP.
- Rename the lon and lat fields as
longitudeandlatituderespectively. Click on Save:
As a result new fields will be added to our data model.
Hunk offers us other methods of adding attributes, such as:
- Eval expression: This allows us to define a new field using an
evalexpressionTip
For more information about the
evalexpression, see: http://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Addanevalexpressionattribute. - Lookup: This allows us to use existing lookup definitions to add fields to our event object
Tip
For more information about lookup in data models, see: http://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Addalookupattribute.
- Regular expression: This allows us to define a new field using a regular expression
Tip
For more information about regular expressions in data models, see: http://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Addaregularexpressionattribute.