Salt belongs to a class of configuration management systems intended to make administrating a large number of machines easier. It does so by applying the same rules to different machines, making sure that any differences in their configuration are intentional.
It is written in Python and, more importantly, extensible in Python. For example, wherever a YAML file is used, Salt allows a Python file that defines a dictionary.
The salt package includes the client/server code. It depends on pyzmq, which, in turn, relies on the libzmq C library.
The salt-ssh package only includes the local and SSH-based client. Because of that, it does not depend on the libzmq library. When only local/SSH support is needed, it is better to install salt-ssh.
Other than this distinction, the two packages are identical.
10.1 Salt Basics
Calculates the desired configuration
Calculates how the system differs from the desired configuration
Issues commands to bring the system to the desired configuration
Some extensions to Salt go beyond the operating system concept to configure some SaaS products into the desired configuration; for example, there is support for Amazon Web Services, PagerDuty, or some DNS services (those supported by libcloud).
Since, in a typical environment, not all operating systems need to be configured the same way, Salt allows detecting properties of systems and specifying which configurations apply to which systems. Salt uses them at runtime to decide the complete desired state and enforce it.
Locally: Run a local command that takes the desired steps.
SSH: The server will ssh into clients and run commands that take the desired steps.
Native protocol: Clients connect to the server and take whatever steps the server instructs them.
Using the ssh mode removes the need to install a dedicated client on the remote hosts since, in most configurations, an SSH server is already installed. However, Salts native protocol for managing remote hosts has several advantages.
It allows the clients to connect to the server, thus simplifying discovery. All you need for discovery is just for clients to the server. It also scales better. Finally, it allows you to control which Python modules are installed in the remote client, which is sometimes essential for Salt extensions.
If some Salt configuration requires an extension that needs a custom module, you can take a hybrid approach. Use the SSH-based configuration to bring a host to the point where it knows where the server is and how to connect to it, and then specify how to bring that host to the desired configuration.
This means there are two parts to the server: one that uses SSH to bring up the system to a basic configuration which, among other things, has a Salt client, with the second part waiting for the client to connect to send the rest of the configuration.
This has the advantage of solving the secret bootstrapping problem. You verify the client hosts SSH key using a different mechanism, and when connecting to it via Salt, inject the Salt secret to allow the host to connect to it.
When you choose the hybrid approach, there needs to be a way to find all machines. When using some cloud infrastructure, it is possible to do this using API queries; however, you need to make it accessible to Salt if you get this information.
In ideal circumstances, all parameters are identical for the machines. The user is the SSH user. The sudo boolean states whether sudo is needed, which is almost always true. The only exception is if it is an administrative user (usually root). Since it is a best practice to avoid SSH as root, this is set to True in most environments.
The priv field is a path to the private key. Alternatively, it can be agent-forwarding to use SSH agent. This is often a good idea since it presents an extra barrier to key leakage.
The roster can go anywhere, but Salt looks for it in /etc/salt/roster by default. Putting this file in a different location is subtle. salt-ssh finds its configuration, by default, from /etc/salt/master. Since the usual reason to put the roster elsewhere is to avoid touching the /etc/salt directory, you usually need to configure an explicit master configuration file using the -c option.
If you put in the value. In config_dir, it looks in the current directory for a master file. You can set the roster_file field in the master file to a local path (for example, roster) to make sure the entire configuration is local and locally accessible. This can help if a version control system is managing things.
After defining the roster, it is useful to check that the Salt system is functioning. It is also possible to run the commands to test Salt locally.
For testing locally, use salt-call --local instead of salt '*'. This needs privileged access, so it should probably be done in a VM or a container.
They are all supposed to return True. This command fails if machines are unreachable, SSH credentials are wrong, or other common configuration problems.
Because this command does not affect the remote machines, it is a good idea to run it first before starting to perform any changes. This ensures that the system is correctly configured.
Several other test functions are used for more sophisticated checks of the system.
The test.false command intentionally fails, which is useful to see what failures look like. For example, when running Salt via a higher-level abstraction, such as a continuous deployment system, this can be useful to see visible failures (for example, send appropriate notifications).
The test.collatz and test.fib functions perform heavy computations and return the time it took and a result. It is used to test performance; for example, this might be useful if machines dynamically tune CPU speed according to available power or external temperature. You want to test whether this is the cause of performance problems.
Since the combination of the shell parsing rules and the Salt parsing rules can be, at times, hard to predict, this is a useful command to be able to debug those combinations and figure out what things are over- or under-quoted.
Instead of '*' you can target a specific machine by logical name. This is often useful when seeing a problem with a specific machine. It allows a quick feedback mechanism when trying various fixes (for example, changing firewall settings or SSH private keys).
This causes all connected machines to create a /src directory. More sophisticated commands are possible, and it is possible to only target specific machines.
The technical term for the desired state in Salt is highstate, which is shortened from high-level state. It describes the goal of the state. The name is a frequent cause of confusion because it seems to be the opposite of a low state, which is described almost nowhere.
The low states, or the low-level states, are the steps Salt takes to get to the goal. Since the compilation of the goal to the low-level states is done internally, nothing in the user-facing documentation talks about a low state, thus leading to confusion.
Again, both do the same thing: figure out the desired state for all machines and then issue commands to reach it.
The state is described in SLS files. These files are usually in the YAML format and describe the desired state.
The usual way to configure is one file top.sls, which describes which other files apply to which machines. The top.sls name is used by default as the top-level file.
This example would have all machines apply the configuration from core.sls (presumably, making sure the basic packages are installed, the right users are configured, etc.), from monitoring.sls (presumably, making sure that tools that monitor the machine are installed and running), and kubelet.sls, defining how to install and configure the kubelet.
Indeed, much of the time, Salt configures machines for workload orchestration tools such as Kubernetes or Docker Swarm.
10.2 Salt Concepts
Salt introduces quite a bit of terminology and quite a few concepts.
A minion is the Salt agent. Even in the agentless SSH-based communication, there is still a minion. The first thing that Salt does is send over code for a minion and then start it.
A Salt master sends commands to minions.
This ensures the procps package (which includes the ps command among others) is installed.
Most Salt states are written to be idempotent to have no effect if they are already in effect. For example, if the package is already installed, Salt does nothing.
Salt modules are different from Python modules. Internally, they do correspond to modules, but only some modules.
Unlike states, modules run things. This means that there is no guarantee, or even attempt at, idempotence.
Often, a Salt state wraps a module with some logic to decide whether it needs to run the module; for example, before installing a package, pkg.installed checks if the package is already installed.
A pillar is a way of attaching parameters to specific minions, which different states can then reuse.
If a pillar filters out some minions, then these minions are guaranteed to never be exposed to the values in the pillar. This means that pillars are ideal for storing secrets since they are not sent to the wrong minions.
For better protection of secrets, it is possible to use gpg to encrypt secrets in pillars. Since gpg is based on asymmetric encryption, it is possible to advertise the public key; for example, in the same source control repository that holds the states and pillars.
This means anyone can add secrets to the configuration, but the private key is needed, on the master, to apply those configurations.
Since GPG is flexible, it is possible to target the encryption to several keys. As a best practice, it is best to load the keys into a gpg-agent. When the master needs the secrets, it uses gpg, which communicates with the gpg-agent.
This means the private keys are never exposed to the Salt master directly.
Having explicit readable names helps you make sure you depend on the right state. Note that even though Extract precedes Copy, it still waits for the copying to complete.
In general, inverting the relationship does not improve things. However, this can sometimes be used to minimize or localize changes to files in a shared repository.
There are other relationships possible, and all of them can be inverted. onchanges specifies that the state should only be reapplied if another state has caused actual changes. onfail specifies that the state should only be reapplied if another state application fails. This can be useful to set alerts or make sure that the system goes back to a known state.
A few more esoteric relationships are possible, like watch and prereq, which are more specialized.
Minions generate keys when using the built-in Salt communication rather than the SSH method. Those keys need to be accepted or rejected. One way to do so is to use the salt-key command.
As mentioned, one way of bootstrapping the trust is to use SSH. Using Salt to transfer over parsed output from running salt-key -F master to the minion, and then set it in the minion’s configuration under the master_finger field.
Similarly, run remotely salt-call key.finger --local on the minion (for example, with salt 'minion' cmd. run) and compare it to the pending key before accepting. This can be automated and leads to a verified chain.
There are other ways to bootstrap the trust, depending on what primitives are available. If, for example, hardware key management (HKM) devices are available, they can be used to sign the minions and the master’s keys.
Trusted Platform Modules (TPM) can also mutually assure trust. Both mechanisms are beyond the current scope.
Grains (as in, a grain of salt) parameterize a system. They differ from pillars in that the minion decides on the grain, and that configuration is stored and modified on the minions.
Some grains, such as fqdn, are autodetected on the minions. It is also possible to define other grains in the minion configuration file.
It is possible to push grains from the master. It is also possible to grab grains from other sources when bootstrapping the minion. For example, it is possible to set the UserData as a grain on AWS.
Salt environments are directory hierarchies that each define a separate top file. Minions can be assigned to an environment, or an environment can be selected when applying the highstate using salt '*' state.highstate saltenv=....
file_roots is a list of directories that function like a path. When looking for a file, Salt searches file_roots in order. It can be configured on a per-environment basis and are the primary thing distinguishing environments.
10.3 Salt Formats
So far, the example SLS files were YAML files. However, Salt interprets YAML files as Jinja templates of YAML files. This is useful for customizing fields based on grains or pillars.
For example, the name of the package containing the things you need to build Python packages differs between CentOS and Debian.
It is important to notice that the Jinja processing step is completely ignorant of the YAML formatting. It treats the file as plain text, does the formatting, and then Salt uses the YAML parser on the result.
This means that Jinja can make an invalid file only in some cases. Indeed, you embedded such a bug in the preceding example. If the OS is neither CentOS nor Debian, the result would be an incorrectly indented YAML file, which fails to parse in strange ways.
This raises an exception at the right point if a machine is added to the roster that is not one of the supported distributions. Otherwise, the YAML would be incorrectly formatted. In that case, the symptom would be that Salt would complain a parse error the YAML file, making it harder to troubleshoot the issue.
Such care is important whenever doing something non-trivial with Jinja because the two layers, the Jinja interpolation, and the YAML parsing, are not aware of each other: Jinja does not know it is supposed to produce YAML, and the YAML parser does not know what the Jinja source looked like.
Jinja supports filtering to process values. Some filters are built into Jinja, but Salt extends them with a custom list.
Among the interesting filters is YAML_ENCODE. Sometimes you need to have a value in the .sls file, which is YAML itself; for example, the content of a YAML configuration file that you need to be copied over.
Embedding YAML in YAML is often unpleasant; special care must be given to proper escaping. With YAML_ENCODE, it can encode values written in the native YAML.
For a similar reason, JSON_ENCODE_DICT and JSON_ENCODE_LIST are useful for systems that take JSON as input. The list of custom filters is long, and this is one of the frequent things that changes from release to release. The canonical documentation is on the Salt documentation site, docs.saltstack.com, under Jinja → Filters.
Until now, we referred to SLS files as files that are processed by Jinja and then YAML; however, this is inaccurate. It is the default processing, but it can override the processing with a special instruction.
Salt only cares that the final result is a YAML-like (or, equivalently in our case, JSON-like) data structure: a dictionary containing recursively dictionaries, lists, strings, and numbers.
Converting the text into such a data structure is called rendering in Salt parlance. This is opposed to common usage, where rendering means transforming to text and parsing means transforming from text, so it is important to note when reading Salt documentation.
A thing that can do rendering is a renderer. It is possible to write a custom renderer, but the most interesting is the py renderer among the built-in renderers.
Let’s indicate that a file should be parsed with the py renderer with #!py at the top.
In that case, the file is interpreted as a Python file. Salt looks for a run function, runs it, and treats the return value as the state.
When running, __grains__ and __pillar__ contain the grain and pillar data.
Since the py renderer is not a combination of two unrelated parsers, mistakes are sometimes easier to diagnose.
In this case, the result is a NameError pinpointing the erroneous line and the missing name.
The trade-off is that reading it in YAML form is more straightforward if the configuration is big and mostly static.
10.4 Salt Extensions
Since Salt is written in Python, it is fully extensible in Python. The easiest way to extend Salt for new things is to put files in the file_roots directory on the Salt master. Unfortunately, there is no package manager for Salt extensions yet. Those files automatically get synchronized to the minions, either when running state.apply or explicitly running saltutil.sync_state. The latter is useful if you want to test, for example, a dry run of the state without causing any changes but with the modified modules.
10.4.1 States
State modules go under the root directory for the environment. If you want to share State modules between environments, it is possible to make a custom root and share that root between the right environments.
The name of the function maps to the name of the state in the SLS state file. If you put this code in mean.py, the appropriate way to address this state would be mean.enforce_no_mean_files.
The right way to find files or do anything in a Salt state extension is to call Salt executors. In most non-toy examples, this means writing a matching pair: a Salt executor extension and a Salt state extension.
Finally, you return a dictionary with the same changes as those documented in the test mode but with a comment indicating that these have already run.
This is the typical structure of a state module: one (or more) functions that accept a name (and possibly more arguments) and then return a result. The structure of checking if changes are needed and whether you are in test mode, and then performing the changes is also typical.
10.4.2 Execution
For historical reasons, execution modules go in the file roots _modules subdirectory. Similar to execution modules, they are also synchronized when state.highstate is applied and when explicitly synchronized via saltutil. sync_all.
Note that Salt is usable in execution modules as well. However, while it can cross-call other execution modules (in this example, file) it cannot cross-call into state modules.
Execution modules are often simpler than state modules, as in this example. In this toy example, the execution module barely does anything except coordinate calls to other execution modules.
This is not completely atypical, however. Salt has so much logic for managing machines that all an execution module often has to do is coordinate calls to other execution modules.
10.4.3 Utility
When writing several execution or state modules, sometimes there is common code that can be factored out.
This code can sit in utility modules under the root file _utils directory. It is available as the __utils__ dictionary.
In this case, you could have put the function as a regular function in the module. Putting it in a utility module was used to show how to call functions in utility modules.
10.4.4 Extra Third-Party Dependencies
Sometimes it is useful to have third-party dependencies, especially when writing new state and execution modules. This is straightforward to do when installing a minion. You just make sure to install the minion in a virtual environment with those third-party dependencies.
When using Salt with SSH, this is significantly less trivial. In that case, it is sometimes best to bootstrap from SSH to a real minion. One way to achieve that is to have a persistent state in the SSH minion directory and have the installation of the minion set a grain of completely_disable in the SSH minion. This would ensure that the SSH configuration does not cross-talk with the regular minion configuration.
10.5 Summary
Salt is a Python-based configuration management system. For non-trivial configurations, it is possible to express the desired system configuration using Python, which can sometimes be more efficient than templating YAML files. It is also possible to extend it with Python to define new primitives.