The simple thing to do is to avoid extensive resource usage by your script by placing the script on the agent side. Zabbix provides an alternative method, and the script should instead be on the server side and load the Zabbix server; it can be offloaded to the agent side with UserParameter.
UserParameter is defined on the agent configuration file. Once it is configured, it is treated in the same way as all the other Zabbix agent items by simply using the key specified in the parameter option. To define a user parameter, you need to add something similar to the following to the agent configuration file:
UserParameter=<key>,<shell command>
Here, key must be unique and the shell command represents the command to execute. The command can be specified here inline and doesn't need to be a script, as shown in the following example:
UserParameter=process.number, ps -e |wc -l
In this example, the process.number key will retrieve the total number of processes on your server.
With the same kind of approach, you can check the number of users currently connected with the following code:
UserParameter=process.number, who |wc -l
It is easy to understand that using this method you are going to define a large number of entries inside the agent configuration file. This is not the right approach because it is better to keep the configuration file simple.
Zabbix provides an interesting UserParamenter feature to avoid the proliferation of those items on the agent side—the flexible user parameter. This feature is enabled with an entry of this kind:
UserParameter=key[*],<shell command>
Here, key still needs to be unique, and the [*] term defines that this key accepts the parameters. The content between the square brackets is parsed and substituted with $1...$9; please note that $0 refers to the command itself. An example of UserParameter can be the following:
UserParameter=oraping[*],tnsping $1 | tail -n1
This command will execute tnsping to your SID, passing it as $1. You can apply the same method in the process to count specified users as follows:
UserParameter=process.number[*], ps -e |grep ^$1 | wc -l
Then, if we want to move to the agent side for the first script that returns the number of open files for a defined user, the configuration will be the following:
UserParameter=lsof.sh[*],/usr/local/bin/lsof.sh $1
Once this has been added, you only need to restart the agent. On the server side, you need to switch the item Type to Zabbix agent and save it. The following screenshot depicts this discussion:
With the same method, you can configure the check_ora.sh script to check the database with the following code:
UserParameter=check_ora.sk[*],check_ora.sh –i $1 –q $2
On the Zabbix server side, you need to create an item of the Zabbix agent type or the Zabbix agent (active) type, and on the key you need to specify:
check_ora.sk[<databasename> <query_to_execute>]
There are methods to test whether your UserParameter is working fine and the agent is able to recognize it. The first one is with zabbix_get; for example, in the case of losf.sh from the Zabbix server, we can use the following:
# zabbix_get -s 127.0.0.1 -p 10050 -k lsof.sh["postgres"] 2116
The response is the result of the operation. Alternatively, we can log on to the monitored host and run the following command:
#/usr/sbin/zabbix_agentd -t lsof.sh["postgres"] lsof.sh[postgres][/usr/local/bin/lsof.sh postgres] [t|2201]
Again, this will display the output and the script that is called.
With UserParameter, you moved the script from the server side to the agent side. The workload introduced by the script is now on the agent side, and you avoided resource stealing on the server side. Another point to consider is that this approach divides the workload between multiple servers. Obviously, each agent will monitor the database present on its hosts.
The UserParameter parameters are really flexible. To enable them on the agent side, you need to change the configuration file and restart the agent. Also, here you need to be sure that the returned value is properly set; if it isn't properly set, it will be discarded.
Now, between the cons, you need to consider the observer effect (discussed in Chapter 1, Deploying Zabbix) introduced with this kind of monitoring. You need to keep things as lightweight as possible, especially because the agent runs on the same server that provides the service.
The usage of UserParameter implies that you need to distribute the scripts and the relative updates across all your servers. In this example, where you want to monitor Oracle, you need to consider how many different versions of operating systems and software you need to handle. It is possible that in time, you will need to handle a myriad of different flavors of your scripts and software. This myriad of scripts, versions, and so on will force you to have centralized deployment, that is, all the versions of the scripts are stored in a centralized repository. In addition, you need to take care of the workload added by your scripts and, if they don't handle all the possible exceptions well, this can be a really complex scenario to manage.
UserParamenter is really good, flexible, and sometimes indispensable to solve some monitoring requirements, but is not designed for massive monitoring against the same host. For all these reasons, it is time to explore another way to massively monitor the items that Zabbix doesn't support natively.
The following are certain very important points about external scripts and UserParamenter:
- All pieces of input are passed as parameters to the script and should properly be properly sanitized within the script to prevent command injection.
- All values are returned via
STDOUTand should be in the format of the expected return type. Returning nothing will cause the Zabbix server to flag this item as unsupported. - Make sure that all scripts terminate in a short period of time.
- Make sure that scripts do not share or lock any resources, or have any other side effects, to prevent race conditions or incorrect interactions from incurring multiple executions.