Chapter 7. Beacons
By default, beacon modules run on a fast, one-second interval and emit Salt events. Their main function is to import external events on the Salt bus. For this reason, they pair quite well with the Salt reactor (introduced in Chapter 9). For example, the inotify beacon can watch a file or directory and emit an event if any files are modified, then the reactor can kick off a state run to put the file back into the desired state.
There are many beacon modules that cover a wide variety of tasks. To name just a few: btmp and wtmp watch user logins; sh watches user shell activity; diskusage, load, memusage, and network_info regularly poll relevant system stats and emit events if a threshold has been crossed; status can emit regular events containing current system stats; log and journald can watch log files for certain patterns and emit an event with matching log entries; haproxy and service can monitor services and emit events if the service is down or overloaded. Like other Salt modules, new beacon modules are easy to write if you have a custom need.
Configuration
Beacon configuration lives in the Salt minion configuration file under a beacons key. As usual, make sure any dependencies for the module are also installed on the minion, and restart the minion daemon after making changes to the config file.
Beacons can be equally used to ensure that processes are alive, and restart them otherwise. Considering that a number of proxy minion processes are executed on a server which is managed using the regular Minion, we can use the salt_proxy beacon to keep them alive.
Note
Remember: the proxy minions manage the network devices, while the regular minion manages the server where the proxy processes run.
Consider the following beacon configuration, which mantains the alive status of the proxy processes managing our devices from previous examples in just a few simple lines:
beacons:salt_proxy:-device1:{}-device2:{}-device3:{}
After restarting the minion process, we can observe events with the following structure on the Salt bus:
salt/beacon/minion1/salt_proxy/{"_stamp":"2017-08-25T10:17:20.227887","id":"minion1","device1":"Proxy device1 is already running"}
minion1 is the ID of the minion that manages the server where the proxy processes are executed.
In case a proxy process dies, the salt_proxy beacon will restart it, as seen from the event bus:
salt/beacon/minion1/salt_proxy/{"_stamp":"2017-08-25T10:17:31.503653","id":"minion1","device1":"Proxy device1 was started"}salt/minion/device1/start{"_stamp":"2017-08-25T10:17:42.676464","cmd":"_minion_event","data":"Minion device1 started at [...]","id":"device1","pretag":null,"tag":"salt/minion/device1/start"}
A good approach to monitor the health of the minion server is using the status beacon, which will emit the system load average every 10 seconds:
beacons:status:-interval:10# seconds-loadavg:-all
The event takes the following format:
salt/beacon/minion1/status/2017-08-11T09:28:28.233194{"_stamp":"2017-08-11T09:28:28.240186","data":{"loadavg":{"1-min":0.01,"15-min":0.05,"5-min":0.03}},"id":"minion1"}
Tip
The status beacon—together with others such as diskusage, memusage, network_info or network_settings—can also be enabled when managing network gear that permits installing the Salt minion directly on the platform, in order to monitor their health from Salt, and eventually automate reactions.
See the documentation for each beacon module for how to configure it and when and what it will emit. The events can be seen on the master using the state.event Runner, and the reactor (see Chapter 9) can be configured to match on beacon event tags.
Troubleshooting
The best way to troubleshoot beacon modules is to start the minion daemon in the foreground with trace-level logging: salt-minion -l trace. Look for log entries to see if the module is loaded successfully, and then watch for log entries that appear for each interval tick to make sure the beacon is running.