Time is relative.
My future is your present.
These two lines started this chapter in the first edition of this book, and as expected, lots of things have changed since then. Most of the features previously seen as future are already present and some of them are well consolidated. And, as with the first edition, some of the features we'll review here will be not the future, but the present of some of the readers of this book.
While writing this second edition, Puppet 4 has become a reality and we may not be seeing so many revolutionary changes and experimental features as we saw with the latest releases. The roadmap to the next Puppet releases aims more at improvements in existing features and to continue working on performance and on a more optimized codebase. These changes include full code rewrites to other languages, and changes in network protocols.
Puppet also has to respond to how the computing world is changing; everything is in the cloud now and containers are present everywhere, also, other operating systems as Microsoft Windows require the same level of automation as the ones found in Linux and Puppet aspires to be a reference in these platforms too.
In this chapter, we are going to explore the following topics:
- Changes in serialization formats, MessagePack and the replacement of PSON by JSON.
- Direct Puppet and cached mode, to avoid unnecessary Puppet runs by improved changes detection and improvements in protocol.
- File sync, the new solution to deploy code in Puppet enterprise.
- Other expected changes as Puppet CA reimplementation, some features for better package management support, or the future of Puppet faces.
- Beyond Puppet 4. Even if it's probably too soon to talk about this, we will explore what seems to be in the roadmap to Puppet 5.
Puppet server and clients exchange a remarkable amount of data: facts, catalogs, and reports. All this data has to be serialized, that is, converted into a format that can be stored to a file, managed in a memory buffer, and sent on a network connection.
There are different serialization formats: Puppet, during its years of existence, has used XML-RPC, YAML, and PSON (a custom variation of JSON that allows inclusion of binary objects), the latter being the currently preferred choice.
PSON has some problems: it was for some time pure JSON, but then it evolved separately to be adapted to Puppet convenience, one of the main differences is that JSON is restricted to UTF-8, while PSON accepts any encoding. This change was introduced to allow binary data to be sent as the content of files, but it also introduced the problem of losing control over the encoding Puppet code has to support.
Currently, another protocol supported by Puppet and maintained by the community is MessagePack. It's a binary format that is more compact and efficient (some tests suggest that it can be 50 percent more compact than JSON), so it reduces both the quantity of data to transmit over the wire and the computational resources needed to manage it.
A simple test shows how an array such as ['one' , 'two' , 'three'] can look in different formats. Expressed in the YAML format, this becomes (24 bytes, newlines included):
--- - one - two - three
In the JSON format, this becomes (21 bytes):
["one","two","three"]
In MessagePack, this becomes (15 bytes (a string such as x93 represents a single byte in hexadecimal notation):
x93xA3onexA3twoxA5three
If we consider that Puppet is constantly making serialization and deserialization of data, it's clear that the introduction of such a format may deliver great performance benefits.
To enable MessagePack serialization, we need to install the msgpack gem on both clients and server:
gem install msgpack
Remember to use the appropriate gem command depending on your installation. And set it, in the [main] section of their puppet.conf:
preferred_serialization_format = msgpack
Clients where msgpack is not supported can keep the default PSON format and continue to operate normally. But although on the way to Puppet 3 and 4 MessagePack looked like the right solution, as it uses binary representation, it couldn't help with the encoding problems.
On the other hand, the well-known and universally supported JSON format requires everything to be valid Unicode (PSON is represented with 8-bit ASCII), which can help a lot with this problem. It's not as fast or compact as MessagePack, but it has multiple libraries that can be tested to see which one provides a better performance for Puppet needs.
Puppet also uses YAML to store some data and reports on server and clients; the migration plan to JSON also includes the deprecation of these files. For the migration to new formats, Puppet will keep backwards compatibility with PSON by now, but decoding and encoding everything as JSON from version 5.