What we have seen up to now are more or less standard and mainstream Puppet documentation and usage patterns. I have surely forgotten valuable alternatives and I may have been subjective on some solutions, but they are all common and existing ones, nothing has been invented.

In this section, I'm going to discuss something that is not mainstream, has not been validated in the field, and is definitely a personal idea on a possible approach to higher abstraction modules.

It's not completely new or revolutionary, I'd rather call it evolutionary, in the line of established patterns like parameterized classes, growing usage of PuppetDB, roles and profiles, with a particular focus on reusability.

I call stack here a module that has classes with parameters, files, and templates that allow the configuration of a complete application stack, either on a single all-in-one node or on separated nodes.

It is supposed to be used by all the nodes that concur to define our application stack, each one activating the single components we want to be installed locally.

The components are managed by normal application modules, whose classes and definitions are declared inside the stack module according to a some what opinionated logic that reflects the stack's target.

In my opinion there's an important difference between application (or component) modules and stack (higher abstraction) modules.

Application modules are supposed to be like reusable libraries; they shouldn't force a specific configuration unless strictly necessary for the module to work. They should not be opinionated and should expose alternative reusability options (for example, different ways to manage configuration files, without forcing only a settings or file-based approach).

Stack modules have to provide a working setup, they need templates and resources to make all the stuff work together. They are inherently opinionated since they provide a specific solution, but they can present customization options that allow reusability in similar setups.

The stack's classes expose parameters that allow:

Let's look at a sample stack::logstash class that manages a logging infrastructure based on LogStash (a log collector and parsing tool), ElasticSearch (a search engine), and Kibana (a web frontend for ElasticSearch). This is obviously an opinionated setup, even if it is quite common for LogStash.

The class can have parameters like:

class stack::logstash (
  $syslog_install                   = false,
  $syslog_config_template     = 'stack/logstash/syslog.conf.erb',
  $syslog_config_hash               = { },
  $syslog_server                    = false,
  $syslog_files                     = '*.*',
  $syslog_server_port               = '5544',

  $elasticsearch_install            = false,
  $elasticsearch_config_template    = 'stack/logstash/elasticsearch.yml.erb',
  $elasticsearch_config_hash        = { },
  $elasticsearch_protocol           = 'http',
  $elasticsearch_server             = '',
  $elasticsearch_server_port        = '9200',
  $elasticsearch_cluster_name       = 'logs',
  $elasticsearch_java_heap_size     = '1024',
  $elasticsearch_version            = '1.0.1',

  $logstash_install                 = false,
  $logstash_config_template  = 'stack/logstash/logstash.conf.erb',
  $logstash_config_hash             = { },

  $kibana_install                   = false,
  $kibana_config_template           = undef,
  $kibana_config_hash               = { },
) {

You can see some of the reusability oriented parameters we have discussed in Chapter 5, Using and Writing Reusable Modules, the class' users can provide:

For each of the managed components, there's a Boolean parameter that defines if such a component has to be installed (elasticsearch_install, logstash_install …).

The implementation is quite straightforward, if these variables are true, the relevant classes are declared with parameters computed in the stack class:

  if $elasticsearch_install {
    class { 'elasticsearch':
      version       => $elasticsearch_version,
      java_opts     => $elasticsearch_java_opts,
      template      => $elasticsearch_config_template,
    }
  }

  if $syslog_server and $syslog_install {
    if $syslog_config_template {
      rsyslog::config { 'logstash_stack':
        content  => template($syslog_config_template),
      }
    }
    class { '::rsyslog':
      syslog_server => $syslog_server,
    }
  }

The resources used for each component can be different and have different parameters, defined according to the stack class' logic and the modules used.

It's up to the stack's author as to the choice of which vendors to use for the application modules and how many features, reusability options, and how much flexibility to expose to the stack's users as class parameters.

The stack class(es) are supposed to be the only entry point for users' parameters and they are the places where resources, classes, and definitions are declared.

The stack's variables, which are then used to configure the application modules, can be set via parameters or calculated and derived according to the required logic.

This is a relevant point to underline: the stack works at a higher abstraction layer, and can manipulate and manage how interconnected resources are configured.

At the stack level you can define, for example, how many ElasticSearch servers are available, what are the LogStash indexers and how to configure them in a coherent way.

You can also query PuppetDB in order to set variables based on your dynamic infrastructure data.

In this example the query_nodes function from the puppetdbquery module (we have seen it in Chapter 3, Introducing PuppetDB) is used to fetch hostnames and IP addresses of the nodes where the stack class has installed ElasticSearch. The value retrieved from PuppetDB is used if there isn't an explicit $elasticsearch_server parameter set by users:

  $real_elasticsearch_server = $elasticsearch_server ? {
    ''      => query_nodes('Class[elasticsearch]',ipaddress),
    default => $elasticsearch_server,
  }

In this case the stack manages configurations via a file-based approach, so it uses templates to configure applications.

The stack class has to provide default templates, which should be possible to override, where the stack's variables are used. For example, stack/logstash/syslog.conf.erb can be something like:

<%= scope.lookupvar('stack::logstash::syslog_files') %> @@<%= scope.lookupvar('stack::logstash::syslog_server') %>:<%= scope.lookupvar('stack::logstash::syslog_server_port') %>

Here the scope.lookupvar() function is used to get variables by their fully qualified name so that they can be consistently used in our classes and templates.

When using Hiera, the general stack's parameters can be set in common.yaml:

---
  stack::logstash::syslog_server: '10.42.42.15'
  stack::logstash::elasticsearch_server: '10.42.42.151'
  stack::logstash::syslog_install: true

Specific settings or install Booleans can be specified in role-related files such as el.yaml:

---
  stack::logstash::elasticsearch_install: true
  stack::logstash::elasticsearch_java_opts: '-Xmx1g -Xms512m'

Compared to profiles, as commonly described, such stacks have the following differences: