We are going to explore Apache web logs taken from the online store. These logs are taken from the Apache web server and uploaded to HDFS. You'll see how to read Apache logs out of the box. The name of the store is unicorn fashion. Here is an example log line:
135.51.156.129 - - [02/Dec/2013:13:52:29] "POST /product.screen?productName=SHORTS&JSESSIONID=CA10MO9AZ5USANA4955 HTTP 1.1" 200 2334 "http://www.yahoo.com" "Opera/9.01 (Windows NT 5.1; U; en)" 167
It's a normal Apache access combined log. We can build reports, dashboards, and alerts on top of this data. You will:
- Learn the basics of SPL to create queries
- Learn visualization abilities
- Drill-down from the aggregated report to the underlying detailed data
- Check the job details used to prepare report data
- Create alerts and see a simple alert use-case
- Create a dashboard presenting web analytics reports on a single page
- Create a virtual index
You know already how to create a virtual index; we provide a screenshot with an index configuration:
Let's try to create some reports in order to meet basic functionality of Hunk and Search Processing Language (SPL).
Let's get the top five browsers used by online store visitors. We need to start the Explore data wizard:
- Go to Virtual indexes:
- Click on Explore Data.
- Pick the Hadoop provider and our
digital_analyticsvirtual index:
- Select a file and click on Next:
- Select Web | Access combined as a type for logs:
Your screen should look like this:
- Complete the context settings:
- Choose search in App Context and select App under Sharing Context.
- Review the settings and click on Finish. Now we are ready to create our first dashboard.
- Open
http://quickstart.cloudera:8000/en-US/app/launcher/homeand click on Search & Reporting:
- Use a query to get the top five browsers:
index="digital_analytics" | top 5 useragentThe search interface in front of you should look like this:
- Save the report by selecting Save As | Report from the drop-down list:
- The settings for the report to be saved are as follows:
- The report will have visualizations in the Column and Time Range Picker controls.
- Here is the final result. Go to the Reports page:
- Select the report named Report top 5 browsers that we have just created:
Let's create one more report. We are going to display the sources of site traffic. Go back to the query search and use the following expression to get referrers:
index="digital_analytics" referer != *unicorn*| top referer percentfield=percent
You should read the expression in this way:
- Use the
digital_analyticsindex. - Exclude lines where the referrer field contains the
unicornsubstring. - Group by the referrer field value, count those lines having the same referrer value, and order counts in descending order.
Note
You can read more about
topcommand here: http://docs.splunk.com/Documentation/Splunk/6.2.2/SearchReference/Top. - Check your search result page:
- Check the job statistics.
- Click on Job and select Inspect Job from the drop-down list:
You'll see a nice job report providing insights into why it's so slow or extremely fast. There is also a link to a log file. You might need it later if you encounter errors:
- Select a visualization for the report. Click on the Visualization tab and select a pie chart from the Drilldown option:
- You can click on the pie and get detailed information. Click on any sector:
- Hunk will automatically create a query for you to display detailed data:
- Save the top referrer report:
Let's perform naive analytics to count the errors occurring on our site.
- See the statuses logged by the Apache server.
index="digital_analytics" | chart count by statusUse bars to visualize the result:
We see various status codes. Let's pay special attention to code=500, which indicates an error on the server side.
- Calculate the error ratio using the
evalexpression.index="digital_analytics" | eval errorRatio = if (status ==500, "ERROR", "ELSE") | timechart count by errorRatio | sort –errorRatioThe idea of the expression is to:
- Use the
digital_analyticsindex. - Calculate the field with the name
errorRatio. If thestatusfield in the index has the value500, thenerrorRatio = «ERROR»; otherwise, theerrorRatiofield gets the value«ELSE». - Count
errorRatiooccurrences over time and sort by the count oferrorRatio:
- Use the
- Save the report:
Hunk can issue alerts when a condition is met. Let's configure an alert when the error count threshold is reached.
- Use a query to count the status with code
=500, which signifies an error on the server side:index="digital_analytics" status=500 | stats count - This query returns the error count. Select Save As | Alert:
It's hard to emulate scheduled activity right now; let's pick the most simple case and see how it works generally.
The following screenshot shows the settings for saving a new alert:
- We've chosen Per-Result to get an alert each time the report returns something.
- Set the alert to be displayed by selecting Activity | Triggered Alerts:
The following screenshot is an overview of the created alert:
- Go to Activity | Triggered Alerts and confirm that the alert has been published:
Now it's time to see how dashboards work. Let's find regions where visitors get problems (status = 500) while using our online store:
index="digital_analytics" status=500 | iplocation clientip | geostats latfield=lat longfield=lon count by Country
You should see a map showing country errors:
Now let's save this as a dashboard. Click on Save As and select Dashboard Panel from the drop-down menu:
The following screenshot shows the values for fields in the Save form:
You should get a new dashboard with a single panel and our report on it. We have several previously created reports. Let's add them to the newly created dashboard using separate panels. Click on Edit | Edit Panels:
Select Add new panel | New from report and add one of our reports:
