Now, let's move on to the items on the Zabbix side. To test the most simple thing first, we will try to send values from the Zabbix server. Navigate to Configuration | Hosts, click on A test host in the Name column, and click on Add in the SNMP interfaces section. Click on the Update button at the bottom, and then click on Items next to A test host. Click on Create item and enter these values:

• Name: SNMP trap tests
• Type: SNMP trap
• Key: snmptrap[test]
• Type of information: Character

When you're done, it should look like this:

This item will collect all traps that this host gets, if the traps contain the test string. We have the trap daemon configured to place traps in a file, and we have the item to place these traps in. What's left is telling the Zabbix server where to get the traps. Open zabbix_server.conf and modify the StartSNMPTrapper parameter:

StartSNMPTrapper=1 

There is a special process in Zabbix that reads traps from a temporary file. This process is not started by default, so we changed that part of the configuration. Take a look at the parameter just preceding this one:

SNMPTrapperFile=/tmp/zabbix_traps.tmp 

Notice how it matches the file in the Perl script. A change in the script should be matched by a change in this configuration file and vice versa. At this time, we will not change the location of this temporary file.

After these changes have been made, restart the Zabbix server daemon. Now, we are ready to test this item. Let's send a trap by executing the following from the Zabbix server:

$ snmptrap -Ci -v 2c -c public localhost "" "NET-SNMP-MIB::netSnmpExperimental" NET-SNMP-MIB::netSnmpExperimental s "test"  

This slightly non-optimal Net-SNMP syntax will attempt to send an SNMP trap to localhost using the public community and some nonsense OID. It will also wait for a response to verify that snmptrapd has received the trap successfully—this is achieved by the -Ci flag. It uses the default port, 162, so make sure the port is open in your firewall configuration on the Zabbix server to receive traps.

If the command is successful, it will finish without any output. If it fails with the snmpinform: Timeout error message, then several things could have gone wrong. As well as double-checking that UDP port 162 is open for incoming data, verify that the community in the /etc/snmp/snmptrapd.conf file matches the one used in the snmptrap command and that the snmptrapd daemon is actually running.

If everything goes well, we should be able to see this item with a value on the latest data page:

Now, let's send a different trap. Still on the Zabbix server, run this:

$ snmptrap -Ci -v 2c -c public localhost "" "NET-SNMP-MIB::netSnmpExperimental" NET-SNMP-MIB::netSnmpExperimental s "some other trap"

This trap will not appear in the item we created. What happened to it? As the value that we sent did not contain the test string, this value did not match the one in the item. By default, such traps are logged in the server log file. If we check the log file, it should have something similar to the following:

9872:20160318:232004.319 unmatched trap received from "127.0.0.1": 23:20:02 2016/03/18 PDU INFO: 
 requestid                      253195749 
 messageid                      0 
 transactionid                  5 
 version                        1 
 notificationtype               INFORM 
 community                      public 
 receivedfrom                   UDP: [127.0.0.1]:54031→[127.0.0.1]:162 
 errorindex                     0 
 errorstatus                    0 
VARBINDS: 
 DISMAN-EVENT-MIB::sysUpTimeInstance type=67 value=Timeticks: (2725311) 7:34:13.11 
 SNMPv2-MIB::snmpTrapOID.0      type=6  value=OID: NET-SNMP-MIB::netSnmpExperimental 
 NET-SNMP-MIB::netSnmpExperimental type=4  value=STRING: "some other trap" 

This is not so easy to trigger on, or even see in, the frontend at all. We will improve the situation and tell Zabbix to handle such unmatched traps for this host by placing them in a special item:

1. Navigate to Configuration | Hosts, click on Items next to A test host, click on Create item, and then fill in these values:

• •  Name: SNMP trap fallback
  •  Type: SNMP trap
  •  Key: snmptrap.fallback
  •  Type of information: Character

2. When you're done, click on the Add button at the bottom

The key we used here, snmptrap.fallback, is a special one. Any trap that does not match any of the snmptrap[] items will be placed here. Retry sending our previously unmatched trap:

$ snmptrap -Ci -v 2c -c public localhost "" "NET-SNMP-MIB::netSnmpExperimental" NET-SNMP-MIB::netSnmpExperimental s "some other trap"

Let's check the latest data page again:

The fallback item got the value this time. To see what the value looks like, let's click on the History link next to one of these items:

It contains quite a lot of information, but it also looks a bit strange, almost as if the value was cut. It turns out that, with this method, the trap information that is recorded in the database is quite verbose and the character information type does not offer enough space for it—this type is limited to 255 characters. We cannot even see the string we sent in the trap that matched or failed to match the filter. Let's try to fix this with the mass update functionality again:

1. Go to Configuration | Hosts
2. Click on Items next to A test host
3. Mark the checkboxes next to both SNMP trap items and click on the Mass update button
4. In the resulting form, mark the checkbox next to Type of information and choose Text
5. Click on the Update button

This should have fixed it, but we don't know that for sure yet. Let's verify—send both of these traps again:

$ snmptrap -Ci -v 2c -c public localhost "" "NET-SNMP-MIB::netSnmpExperimental" NET-SNMP-MIB::netSnmpExperimental s "test"
$ snmptrap -Ci -v 2c -c public localhost "" "NET-SNMP-MIB::netSnmpExperimental" NET-SNMP-MIB::netSnmpExperimental s "some other trap"  

If we look at the history of one of these items now, we will see that this change has indeed helped, and much more information is displayed—including the custom string we used for distributing these values across items:

The first item we created, with the snmptrap[test] key, can actually have a regular expression as the item parameter. This allows us to perform more advanced filtering, such as getting a link up and down traps in a single item. If a trap matches expressions from multiple items, it would get copied to all of those items.