Like the fact that you only need a drop of blood to test for the amount of sugar and sodium levels in your blood, you often only need a small amount of data from large datasets to make conclusions to build accurate searches. When developing and testing in Splunk, event sampling can be particularly useful against large datasets:

Event sampling uses a sample ratio value that reduces the number of results. If a typical search result returns 1,000 events, a 1:10 event sampling ratio will return 100 events. As you can see from the previous screenshot, these ratios can significantly cut the amount of data searched, and can range from a fairly large ratio (which can be set using the Custom... setting) to one as small as 1:100,000 (or even smaller, again using the Custom... setting).

This is not suitable for searches for which you need accurate counts. This is, however, perfect when you are testing your searches as they will return significantly faster. Much of the time you will spend in Splunk is taken up with trying and retrying queries using SPL. If you have to deal with a large amount of data in each execution of a search, then your productivity will be negatively impacted. Consider using event sampling to reduce the time it takes to create useful searches.

The following steps indicate the steps you should take in this process:

• Do a quick search to ensure that the correct event data is present
• Look over the characteristics of the events and determine how you want to analyze them
• Set your event sampling for the level you find useful and efficient for this stage in the process
• Test your search commands against the resulting subset of data
• Keep going through this process until you have a search that you are happy with

When you are done, make sure to reset event sampling to No Event Sampling before saving your search query to a dashboard, otherwise the search will generate incomplete results.