Part 3. Beyond the basics

In Part 2, we worked through the core features, from source and matching directives to filtering, routing, and log event transformation and manipulation. We also saw some common sources and targets, from log files to Elasticsearch, MongoDB, and Slack. With this, we have enough knowledge to develop monitoring solutions to address many needs. But eventually, we will find ourselves needing to look beyond the basics.

We have made references to Cloud Native, Docker, and Kubernetes throughout the book but have not invested too much in the specifics of configuring Fluentd into these environments. This is mainly because it is worth appreciating that Fluentd is more than just a utility for Kubernetes. Before we specifically address logging with Docker and Kubernetes, we should first handle how Fluentd can scale, as this will inform aspects of how we can support containerization.

When we look at Docker and Kubernetes, we will address how Fluentd supports containerized applications and how we capture the log events from these technologies and the challenges they can bring.

Finally, we take on the challenge of what to do when existing plugins can’t help us deal with esoteric or archaic applications or platforms with their custom ways of exposing data to be logged. Perhaps it is an application with an overly complex data structure, or maybe we need a custom parser to process it efficiently instead of using a regular expression. Maybe the only way to get log events is to call an application API. Whatever the problem, we need to develop our own plugins. So we’ll build a custom plugin to understand how to address such a problem and reveal the heart of Fluentd’s extensibility.