It is very important to keep track of Splunk's health status. Splunk Enterprise keeps logging various important information which can be helpful in the various stages of Splunk usage. Splunk's log and Splunk Enterprise can be used together to keep track of Splunk's health and various other important measures related to Splunk Enterprise. The Splunk logs can be useful in troubleshooting, system maintenance and tuning, and so on.
The following activities can be tracked by using Splunk's inbuilt logging mechanism:
- Resource utilization and Splunk license usage
- Data indexing, searching, analytics-related information, warnings, and errors
- User activities and application usage information
- Splunk component performance-related information
Splunk logging ranges from a wide variety of sources like audit log, kvstore log, conf log, crash log, license log, splunkd log, and many more.
Of all the sources for Splunk logs, one of the most important and useful logs is splunkd.log. This log file has information of data input/output, errors, warnings, debugging messages, and so on. splunkd also contains log messages generated by scripted/modular inputs. splunkd (Splunk daemon) is the service which runs the Splunk server and hence the name of the log file.
The splunkd.log file can be found at $SPLUNK_HOME$varlogSplunk. The maximum size of a single splunkd.log file is 25MB and only the five most recent files are retained in the file system. The log messages of the splunkd.log file can also be accessed by the Splunk Web console via index=_internal. The Splunk Web console can also be used to access the log messages of remote forwarders and indexers from the search head in a distributed environment.
The logging of the splunkd.log file can be configured to the required log level as per the requirement by modifying the log.cfg file located as $SPLUNK_HOME$etc.
The search log can be found on the indexer and search head, which keeps logs related to search queries run on Splunk. The search.log file can be found at $SPLUNK_HOME$var
unsplunkdispatchsearch_id. The search.log file is generated as per the searches and hence each search will have its own log file.
This log file contains complete information regarding the respective search, along with errors and warnings. Similar to splunkd.log configuration, search.log can also be configured by modifying changes in log-searchprocess.cfg located at $SPLUNK_HOME$etc.
Apart from splunkd.log and search.log, there are various other important log files, like scheduler.log, which can be used to debug scheduling related issues, and Splunk utility logs, which keep track of license usage database validations.
Splunk logs provide operational information about performance, warnings, and errors. The recent log files can be accessed through the file system, whereas the historical file can be accessed from the Splunk Web console via a Splunk CLI query. Thus, Splunk can be very useful in various scenarios of usage, development, and deployment of Splunk Enterprise.