Chapter 6. The Salt Event Bus
Salt is a heavily asynchronous tool that comes with all the simplicity, reliability, and difficulty inherent in asynchronicity. Minions can request data and files from the master, send job returns or other data, refresh authentication, and more and all of this can happen at any time. Sometimes this is in response to user actions on either the minion or master; other times it is a scheduled job or routine maintenance.
Salt makes use of an event bus internally to send messages from minion to master and from one process to another. As with most things in Salt, this event bus is also exposed to users to extend, read from, and write to.
Event Tags and Data
A Salt event consists of two things: an event tag and event data. Event tags are a slash-delimited string where each slash provides a level of namespacing. Event data is a dictionary that contains a timestamp plus any arbitrary data useful for that type of event.
For example, when a minion finishes processing a job and sends the job return data back to the master, an event is fired that looks similar to the one shown in Example 6-1.
Example 6-1. salt/job/20170706170156951774/ret/device1
{"_stamp":"2017-07-06T23:01:57.098368","...snip...":null}
All Salt events start with salt/, all Salt events start with salt/job/, and so on, with each slash delimiting a new level of namespacing. The event bus in Salt can get quite busy so event tags are often matched-on to filter unwanted event types. (Event tags that do not contain slashes are legacy tags and should be ignored.)
Any user-generated events should define a unique namespace specific to that user or specific to a particular team or workflow.
Consume Salt Events
Salt events are easily viewed and processed from a variety of sources.
Reactor
Most common is to use Salt’s reactor system to match certain events and initiate another Salt action in response. This is described in detail in Chapter 9.
HTTP Stream
salt-api via the rest_cherrypy module can expose Salt’s event bus as an HTTP
stream for easy access from external tools or even external systems.
Raw ZeroMQ
And finally, for the adventurous, Salt’s event bus is simply a ZeroMQ pub/sub socket that can be consumed by the ZeroMQ bindings for any supported programming language.
Event Types
There are many types of events for different parts of Salt or for various actions. In addition, custom event types are fully supported.
Job Events
A new job event is created when the master generates a job ID and before it broadcasts the new job to listening minions (Example 6-2).
Example 6-2. salt/job/20170706170156951774/new
{"_stamp":"2017-07-06T23:01:56.953760","arg":[],"fun":"test.ping","jid":"20170706170156951774","minions":["device1"],"tgt":"*","tgt_type":"glob","user":"shouse"}
Notable fields include the following:
minions-
An array of minion IDs that the master expects to receive returns from for the target specified in this job. This is only an educated guess produced by matching the target against minion grains that have been cached on the master—the minions themselves perform the final matching.
user-
The person who initiated the command. This value will be the local username or eauth username, and will show if the command was run via sudo. Very useful for keeping long-term audit records of who ran what, where, and when.
Separate job return events are created for each minion when each minion completes and job and delivers the result back to the master (Example 6-3).
Example 6-3. salt/job/20170706170156951774/ret/device1
{"_stamp":"2017-07-06T23:01:57.098368","cmd":"_return","fun":"test.ping","fun_args":[],"id":"device1","jid":"20170706170156951774","retcode":0,"return":true,"success":true}
Notable fields include the following:
success-
A Boolean value for whether Salt detected any internal errors while running the job. Consult along with “retcode” to determine overall success.
retcode-
A return code describing any module-level or system-level errors while running the job. Any non-zero number indicates failure; consult along with “success” to determine overall success.
Authentication Events
Authentication events occur when a minion performs an authentication handshake with the master, or when an unauthenticated minion polls the master to learn if its key has been accepted yet. The master will periodically rotate the AES key it uses to encrypt broadcasts which also triggers a re-auth handshake (Example 6-4).
Example 6-4. salt/auth
{"_stamp":"2017-07-06T23:00:33.076164","act":"accept","id":"device1","pub":"-----BEGIN PUBLIC KEY-----[...snip...]-----END PUBLIC KEY-----","result":true}
The act field gives the current status of this minion’s key. Possible values are accept if the key has been accepted, pend if the key has not yet been accepted, and reject if the key has been rejected. Useful to automate accepting the key for a new minion, perhaps along with some custom validation logic.
Minion Start Events
Minion start events are triggered each time a minion establishes a connection with a master. The minion is ready to receive jobs from the master once this event is fired, so it is a good event to use to perform an initialization action such as running a highstate to bring the minion up to the latest configured state (Example 6-5).
Example 6-5. salt/minion/device1/start
{"_stamp":"2017-07-06T23:00:54.252873","cmd":"_minion_event","data":"Minion device1 started at Thu Jul 6 17:00:54 2017","id":"device1","pretag":null,"tag":"salt/minion/device1/start"}
Key Events
Key events are triggered in response to changes to minion keys via the
salt-key CLI utility as well as functions in the key Wheel module.
Authentication events and minion start events are better choices for most operations that send a command to the minion, because a minion may not be available to respond at the time this event is fired (Example 6-6).
Example 6-6. salt/key
{"_stamp":"2017-07-07T01:27:44.656330","act":"accept","id":"device2","result":true}
The act field gives the action taken for the given key, such as accept if it was newly accepted or delete if it was deleted.
Presence Events
Presence is a system to determine which minions are currently connected to a master by querying only the local system and not sending executions to minions (such as with test.ping).
It works by consulting the /proc/net/tcp table and cross-referencing any IPs connected to Salt’s publish port with the IP addresses in the local cache of minion grains. It is useful as a lightweight check only and not for sensitive operations since it does not make use of Salt’s authentication system. And it requires that the minion connect to the master using the same IP address it sees locally (i.e., not through a NAT).
The master can generate events that contain minion headcount information. This system can be enabled by adding presence_events: True to the master config file.
Example 6-7 generates a list of all currently connected minions, generated every 30 seconds.
Example 6-7. salt/presence/present
{"_stamp":"2017-07-07T01:38:41.569652","present":["device1"]}
Example 6-8 generates two lists of minion IDs that show changes in the connected minion headcount since the last poll interval. This is useful for detecting and responding to accidentally disconnected minions within a small window of time.
Example 6-8. salt/presence/change
{"_stamp":"2017-07-07T01:36:43.357588","new":[],"lost":["device1"]}
State Events
State events cause each minion to emit progress events during a state run as it completes each individual state function in the state tree, as seen in Example 6-9. They are another opt-in event type and can be enabled by adding state_events: True to the Salt master config file.
Example 6-9. salt/job/20170706200340461543/prog/device1/0
{"_stamp":"2017-07-07T02:03:40.892345","cmd":"_minion_event","data":{"len":4,"ret":{"__id__":"stage_one","__run_num__":0,"changes":{},"comment":"Success!","duration":0.75,"name":"stage_one","result":true,"start_time":"20:03:40.877168"}},"id":"device1","jid":"20170706200340461543","tag":"salt/job/20170706200340461543/prog/device1/0"}
Notable fields include the following:
data.len-
The total number of states in the current state run.
data.ret.__run_num__-
A counter (zero-indexed) that keeps track of the order that each state function was run. Progress through the entire state run can be calculated as a percentage via
(data.ret.__run_num__ + 1) / data.len * 100.
This chapter introduced the Salt event bus. This constitutes the basic knoweledge for the event-driven infrastructure. In the following chapters you will learn how to interact directly with the bus, how to generate events, and how to consume them and trigger actions.