Solr's logging facility provides a wealth of information, from basic performance statistics, to what queries are being run, to any exceptions encountered by Solr. The log files should be one of the first places you look when you want to investigate any issues with your Solr deployment. There are two types of logs:
- The HTTP server request style logs, which record the individual web requests made to Solr.
- The Solr application logging that uses SLF4J (Simple Logging Framework for Java, a logging façade), which uses the built-in Java JDK logging facility to log the internal operations of Solr.
The HTTP server request logs record the requests that come in and are defined by the Servlet container in which Solr is deployed. For example, the default configuration for managing the server logs in Jetty is defined in jetty.xml:
<Ref id="RequestLog">
<Set name="requestLog">
<New id="RequestLogImpl" class="org.mortbay.jetty.NCSARequestLog">
<Arg><SystemProperty name="jetty.logs" default="./logs"/>/yyyy_mm_dd.request.log</Arg>
<Set name="retainDays">90</Set>
<Set name="append">true</Set>
<Set name="extended">false</Set>
<Set name="LogTimeZone">GMT</Set>
</New>
</Set>
</Ref>The log directory is created in the subdirectory of the Jetty directory. If you have multiple drives and want to store your data separately from your application directory, then you can specify a different directory. Depending on how much traffic you get, you should adjust the number of days to preserve the log files.
We recommend you keep the log files for as long as possible by archiving them. The search request data in these files is some of the best data available to help you improve the relevancy of your search results. By using web analytics tools such as the open source AWStats package to parse your request logs, you can quickly visualize how often different queries are run, and what search terms are frequently being used. This leads to a better understanding of what your users are searching for.
Tailing the HTTP logs is one of the best ways to keep an eye on a deployed Solr. You'll see each request as it comes in and can gain a feel for what types of transactions are being performed, whether it is frequent indexing of new data, or different types of searches being performed. A pause in the logging will quickly highlight garbage collection issues!
The request time data will let you quickly see performance issues. Here is a sample of some requests being logged. You can see that the first request is a POST to the /solr/update URL from a browser running locally (127.0.0.1) with the date. The request was successful, with a 200 HTTP status code being recorded. The POST took 149 milliseconds. The second line shows a request for the admin page being made, which also was successful and took a slow 3,816 milliseconds, primarily because in Jetty, the JSP page is compiled the first time it is requested.
The last line shows a search for dell being made to the /solr/select URL. You can see that up to 10 results were requested and that it was successfully executed in 378 milliseconds.
On a faster machine with more memory and a properly "warmed" Solr cache, you can expect a few tens of milliseconds result time. Unfortunately, you don't get to see the number of results returned, as this log only records the request.
127.0.0.1 - - [25/02/2015:22:57:14 +0000] "POST /solr/update HTTP/1.1" 200 149 127.0.0.1 - - [25/02/2015:22:57:33 +0000] "GET /solr/admin/ HTTP/1.1" 200 3816 127.0.0.1 - - [25/02/2015:22:57:33 +0000] "GET /solr/admin/solr-admin.css HTTP/1.1" 200 3846 127.0.0.1 - - [25/02/2015:22:57:33 +0000] "GET /solr/admin/favicon.ico HTTP/1.1" 200 1146 127.0.0.1 - - [25/02/2015:22:57:33 +0000] "GET /solr/admin/solr_small.png HTTP/1.1" 200 7926 127.0.0.1 - - [25/02/2015:22:57:33 +0000] "GET /solr/admin/favicon.ico HTTP/1.1" 200 1146 127.0.0.1 - - [25/02/2015:22:57:36 +0000] "GET /solr/select/?q=dell%0D%0A&version=2.2&start=0&rows=10&indent=on HTTP/1.1" 200 378
While you may not see things quite the same way Neo did in the movie The Matrix, you will get a good gut feeling of how Solr is performing!
Tip
AWStats is a full-featured open source request log file analyzer under the GPL license and is available from http://awstats.sourceforge.net.
Logging events is a crucial part of any enterprise system. Veteran Java programmers know that the history of Java and logging is complicated, resulting in a fragmented landscape. However, logging in Solr has long been a fraught situation, with various approaches to logging, from Java's built-in logging (also known as JDK logging) to Log4J competing with each other.
As of Version 1.4, Solr standardized on using the Simple Logging Facade for Java (SLF4J) package, which logs to another target logging package selected at runtime instead of at compile time, but to do this, you have to change the logging JAR files inside the WAR file, a nonobvious process! So in Solr 4.3, all the logging files were removed from the Solr WAR file. Instead, you are to provide your own logging files for your chosen implementation. If you use the default Jetty-based Solr package, you'll see that the SLF4J, Log4J, and related jars are in ./lib/ext/. If you want to change what logging you use, change the jar files there, without repackaging Solr. There is more information on the wiki at https://wiki.apache.org/solr/SolrLogging; however, hopefully, logging is a solved topic for the Solr community from here on out. Logging has been, as Erik Hatcher in a post to the solr-dev mailing list memorably called it: a JARmageddon.
By default, Solr sends its logging messages to the standard error stream:
0 [main] INFO org.eclipse.jetty.server.Server – jetty-8.1.10.v20130312
Obviously, in a production environment, Solr will be running as a service, and you will want the messages to be recorded to a log file instead. Here is an example of setting up logging to a file using JDK logging. Create a logging.properties file (an example is at examples/11/logging.properties) with the following content:
# Default global logging level: .level = INFO # Write to a file: handlers = java.util.logging.ConsoleHandler, java.util.logging.FileHandler # Write log messages in human readable format: java.util.logging.FileHandler.formatter = java.util.logging.SimpleFormatter java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter # Log to the logs subdirectory, with log files named solrxxx.log java.util.logging.FileHandler.pattern = ./logs/solr_log-%g.log java.util.logging.FileHandler.append = true java.util.logging.FileHandler.count = 10 java.util.logging.FileHandler.limit = 10000000 #Roughly 10MB
When you start Solr, you need to pass the location of the logging.properties file:
>>java -Djava.util.logging.config.file=logging.properties -jar start.jar
By specifying two log handlers, you can send the output to the console as well as log files. The FileHandler logging is configured to create up to 10 separate logs, each with 10 MB of information. The log files are appended, so that you can restart Solr and not lose previous logging information. Note, if you are running Solr as a service, it is probably going to redirect the STDERR output from ConsoleHandler to a log file as well. In that case, you will want to remove java.util.ConsoleHandler from the list of handlers. Another option is to reduce how much is considered as output by specifying java.util.logging.ConsoleHandler.level = WARNING.
Regardless of which logging solution you go with, you don't want to make the startup arguments for Solr more complex. You can leverage Jetty's configuration to specify the system properties during startup. Edit jetty.xml and add the following stanza to the outermost <Configure id="Server" class="org.mortbay.jetty.Server"/> element:
<Call class="java.lang.System" name="setProperty"> <Arg>log4j.configuration</Arg> <Arg>file:/Users/epugh/log4j.properties</Arg> </Call>
This is also how you can configure other system properties that you might pass in via –D parameters.
Sometimes you need more information than you are typically logging to debug a specific issue, so Solr provides an admin interface to change the logging verbosity of the components in Solr.
While you can't change the overall setup of your logging strategy, such as the appenders or file rollover strategies at runtime, you can change the level of detail to log without restarting Solr. If you change a component like org.apache.solr.core.SolrCore to FINE level of logging, then make a search request to see more detailed information. One thing to remember is that these customizations are NOT retained through restarts of Solr. If you find that you are reapplying log configuration changes after every restart, then you should change your default logging setup to specify custom logging detail levels.
Tip
Even if you adjust the logging levels here to something more detailed, you probably still won't see the messages in the console. By default, ConsoleHandler has an INFO level threshold. You can lower it with this in your logging.properties: java.util.logging.ConsoleHandler.level = FINE.
One of the challenges with logging is that you need to log enough details to troubleshoot issues, but not so much that your log files become ridiculously large and you can't winnow through the information to find what you are looking for.
Tools have arisen to manage those log files and make actionable decisions on the information stored within. Splunk and Loggly are commercial options; however, recently, LucidWorks has paired LogStash for collecting logs, Kibana for visualization, and Solr for storing, and powering the interface has become available. You can download the tool, and the open source components, from http://www.lucidworks.com/lucidworks-silk/.