A configuration-cache reload was only one of the things available in the runtime section. Let's look at the remaining options in there:
housekeeper_execute Execute the housekeeper
log_level_increase=target Increase log level, affects all processes if target is not specified
log_level_decrease=target Decrease log level, affects all processes if target is not specified
Log level control targets: pid
Process identifier process-type All processes of specified type (for example, poller)
process-type,N Process type and number (e.g., poller,3)
As discussed in Chapter 20, Zabbix Maintenance, the internal housekeeper is first run 30 minutes after the server or proxy startup. The housekeeper_execute runtime option allows us to run it at will:
# zabbix_server --runtime-control housekeeper_execute
Even more interesting is the ability to change the log level for a running process. This feature first appeared in Zabbix 2.4, and it made debugging much, much easier. Zabbix daemons are usually started and just work—until we have to change something. While we cannot tell any of the daemons to reread their configuration file, there are a few more options that allow us to control some aspects of a running daemon. As briefly mentioned in Chapter 20, Zabbix Maintenance, the DebugLevel parameter allows us to set the log level when the daemon starts, with the default being 3. Log level 4 adds all the SQL queries, and log level 5 also adds the received content from web monitoring and VMware monitoring.
For the uninitiated, anything above level 3 can be very surprising and intimidating. Even a very small Zabbix server can easily log dozens of megabytes in a few minutes at log level 4. As some problems might not appear immediately, you might have to run it for hours or days at log level 4 or 5. Imagine dealing with gigabytes of logs you are not familiar with. The ability to set the log level for a running process allows us to increase the log level during a problem situation and lower it later, without requiring a daemon restart.
Even better, when using the runtime log level feature, we can select which exact components should have their log level changed. Individual processes can be identified by either their system PID or by the process number inside Zabbix. Specifying processes by the system PID could be done like this:
# zabbix_server --runtime-control log_level_increase=1313
Specifying an individual Zabbix process is done by choosing the process type and then passing the process number:
# zabbix_server --runtime-control log_level_increase=trapper,3
A fairly useful and common approach is changing the log level for all processes of a certain type—for example, we don't know which trapper will receive the connection that causes the problem, so we could easily increase the log level for all trappers by omitting the process number:
# zabbix_server --runtime-control log_level_increase=trapper
And if no parameter is passed to this runtime option, it will affect all Zabbix processes:
# zabbix_server --runtime-control log_level_increase
When processes are told to change their log level, they log an entry about it and then change the log level:
21975:20151231:190556.881 log level has been increased to 4 (debug)
Note that there is no way to query the current log level or set a specific level. If you are not sure about the current log level of all the processes, there are two ways to sort it out:
Restart the daemon
Decrease or increase the log level 5 times so that it's guaranteed to be at 0 or 5, then set the desired level. As a simple test of the options we just explored, increase the log level for all pollers:
# zabbix_server --runtime-control log_level_increase=poller
Open a tail on the Zabbix server logfile:
# tail -f /tmp/zabbix_server.log
Notice the amount of data that just 5 poller processes on a tiny Zabbix server can generate. Then decrease the log level:
# zabbix_server --runtime-control log_level_decrease=poller