Fluentd official documentation

https://docs.fluentd.org/

The official GitHub documentation for Fluentd

Slack

https://slack.fluentd.org

Fluentd community on Slack

Stack Overflow for Fluentd

https://stackoverflow.com/ questions/tagged/fluentd

Stack Overflow’s content relating to Fluentd

Fluentular

https://fluentular.herokuapp.com/

Utility to help validate regular expressions in Fluentd configuration.

Grok parser

https://mng.bz/nYJa

This uses a Grok-based approach to pulling details from a log entry. It includes multiline support.

Microsoft’s Visual Studio Code

https://code.visualstudio.com/

Free IDE that can support a wealth of languages and syntaxes through the use of a plugin framework.

Multiformat parser

https://mng.bz/vo17

Attempts to use different format patterns in the defined order to get a match.

VS Code—Fluentd plugin

https://github.com/msysyamamoto/ vscode-fluentd

Search in Visual Studio Code for msysyamamoto.vscode-fluentd.

A website version of the VS Code plugin

https://regexper.com/

Provides a visual representation of a regular expression. Helps resolve correct groups, etc.

Logging best practices from Logz.io.

https://logz.io/blog/logging-best-practices/

Loggly guide to logging—covers multiple language perspectives.

https://www.loggly.com/ultimate-guide/

Loggly’s The Pragmatic Logging Handbook.

https://mng.bz/4j1w

National Institute of Standards and Technology (NIST) Guide to Computer Security Log Management.

https://mng.bz/QW8G

This defines the standard for Syslog; in addition to helping you better understand Syslog, it also contains some good ideas for logging practices.

https://tools.ietf.org/html/rfc5424

Apache HTTP logs

https://httpd.apache.org/docs/2.4/logs.html

Common Event Format (CEF)

https://mng.bz/XW5v

Graylog Extended Log Format (GELF)

https://mng.bz/y4QB

Nginx logging

https://mng.bz/M2BW

Syslog

https://tools.ietf.org/html/rfc5424

Systemd Journal

https://systemd.io/JOURNAL_FILE_FORMAT/

W3C Extended Log File Format (ELF)

www.w3.org/TR/WD-logfile.html

WinLoG (Windows native logging)

https://mng.bz/aD17

Explanation of Global Interpreter Lock

https://thoughtbot.com/blog/untangling-ruby-threads

Ruby uses a Global Interpreter Lock, which impacts how threading is managed and how to tune Fluentd.

Gem spec

https://mng.bz/g4BV

Defines the details of the gemspec file needed to package custom-developed plugins.

Gemspec vs. Gemfile

https://mng.bz/5Kwa

Explains the difference between the Gemspec and Gemfile.

Minitest—Ruby Unit Testing framework

https://github.com/seattlerb/minitest

Another popular unit testing framework. This resource includes information that helps differentiate it from the other commonly used frameworks.

Rake

https://github.com/ruby/rake

Provides a build process that can be incorporated into a CI/CD pipeline.

rbenv

https://github.com/rbenv/rbenv

This open source tool makes it easier to ensure that different applications can be worked on with different Ruby versions.

RDoc

https://github.com/ruby/rdoc

The standard Ruby-Doc generation. If you need to generate documentation but don't want to use the extensions from Yard, then RDoc is standard.

RSpec Ruby Unit Testing framework

https://rspec.info/

RSpec is an alternative to Ruby unit testing, which supports a test-driven development (TDD) approach to development.

RuboCop

https://docs.rubocop.org/rubocop/ installation.html

Lint tool for Ruby. Worthwhile if developing custom plugins.

Ruby API

https://rubyapi.org/

Ruby documentation tool.

RubyGems

https://rubygems.org/

Catalog of RubyGems along with the RubyGems manager.

RubyGuides

www.rubyguides.com

A set of online guides to implement specific capabilities with Ruby.

Ruby in Twenty Minutes

www.ruby-lang.org/en/documentation/ quickstart/

An excellent introduction to Ruby using Hello World.

Ruby in Practice

www.manning.com/books/ruby-in-practice

Ruby in Practice is a second book from Manning Publications taking a different approach to teaching Ruby development.

RubyInstaller

https://rubyinstaller.org/

Windows installer for Ruby.

Ruby Language Specification and Documentation

www.ruby-lang.org/en/documentation/

Includes summary pages of how Ruby differs from other languages such as Java.

ruby-lint

https://rubygems.org/gems/ruby-lint/

When developing custom plugins, lint tooling will help keep the code tidy and, importantly, help spot any potential errors.

Ruby threading

https://thoughtbot.com/blog/untangling-ruby -threads

A more detailed look at Ruby’s threading.

Ruby unit test framework

https://test-unit.github.io/

Fluentd provides additional supporting resources to make it easy for plugins to be tested using the test-unit tool.

test-unit

https://github.com/test-unit/test-unit

An xUnit-based unit testing solution for Ruby.

The Well-Grounded Rubyist, third edition

www.manning.com/books/the-well-grounded -rubyist-third-edition

Manning bible on Ruby development.

VSCode Ruby plugin

https://marketplace.visualstudio.com/items?itemName=wingrunr21.vscode-ruby

Syntax-aware highlighting for Ruby in Microsoft’s Visual Studio Code.

YARD

https://yardoc.org/

RubyDoc-compliant documentation generator, but also supports a tagging notation that helps provide more comprehensive metadata.

Docker

www.docker.com

Home of the Docker ecosystem.

Docker Hub

https://hub.docker.com/

Repository of Docker images, including Fluentd images and other images used by this book.

Docker plugin for Visual Studio Code

https://mng.bz/6ZDA

Plugin to help understand Docker files and the file syntax.

Home of Kubernetes

https://kubernetes.io

The official Kubernetes website.

Kubernetes log (klog)

https://github.com/kubernetes/klog

The implementation and documentation of the Kubernetes klog component. Klog is being adopted by Kubernetes as the default native logger.

logrotate

https://github.com/logrotate/logrotate

Logrotate is an open source tool for handling log rotation. Depending on how Kubernetes is set up, this is deployed to manage Kubernetes log rotation.

Kubernetes in Action, Second Edition

https://mng.bz/oa1p

Manning’s definitive guide to Kubernetes.

Kubernetes secrets

https://mng.bz/nYW2

Kubernetes’s approach to sharing with pods and container credentials.

Open Containers

https://opencontainers.org/

A standard supported by Docker and several other container development organizations to standardize how Kubernetes interacts with containers.

Alternate Containers to Docker

https://containerd.io/

https://cri-o.io/

Several alternate initiatives to develop containerization, with input from organizations like Intel, IBM, Red Hat, and other Linux vendors such as SUSE, to name a few.

Deployment tools

https://rancher.com

https://helm.sh

Deploying pods and configurations to Kubernetes can be complicated. Several solutions have been developed, such as Helm and Rancher, representing the leading solutions in this space.

Elasticsearch Stack

www.elastic.co/elastic-stack

Details for the ELK stack (Elasticsearch, Logstash, Kibana). While Logstash can be seen as competition to Fluentd, Fluentd is often used with Elasticsearch and Kibana.

Elasticvue

https://elasticvue.com

A UI tool for looking at the contents of Elasticsearch.

Fluentd Elasticsearch plugin

https://docs.fluentd.org/output/ elasticsearch

A plugin that allows you to integrate Elasticsearch.

Elasticsearch plugin source

https://mng.bz/von4

Repository for the Fluentd Elasticsearch plugin.

Manning, Elasticsearch in Action

www.manning.com/books/ elasticsearch-in-action

Manning’s guide to using Elasticsearch.

Home of Redis

https://redis.io/

Provides the basic documentation and the downloads for Redis. In addition to the commercial offerings, we will use Redis in our custom plugin development.

Manning, Redis in Action

www.manning.com/books/redis-in-action

While Redis in Action goes beyond what we need to know for caching, this guide to Redis is helpful.

Redis RubyGem

https://github.com/redis/redis-rb

This is the Ruby Redis library for connecting and using Redis operations. We will use this when building our custom plugin.

Certificate Authority

https://jamielinux.com/docs/ openssl-certificate-authority/

A look at establishing your own certificate authority.

Introduction to TLS

www.internetsociety.org/ deploy360/tls/basics/

An Introduction to TLS.

Let’s Encrypt

https://letsencrypt.org/

Let’s Encrypt is a service developed by the Linux Foundation that provides free certificates with a short shelf life. The service includes some excellent automation to automate recertification.

OWASP TLS Cheat Sheet

https://cheatsheetseries.owasp.org/ cheatsheets/ Transport_Layer_Protection _Cheat_Sheet.html

Provides practical, helpful information about TLS and how it works from the Open Web Application Security Project.

Self-Signed Certificate

https://dzone.com/articles/ creating-self-signed-certificate

Explains how to create a self-signed certificate.

SSL and TLS

www.hostingadvice.com/how-to/tls-vs-ssl/

This site explains the differences between SSL and TLS.

Vault

www.vaultproject.io

Vault is a tool for managing secrets, including details such as usernames and passwords. It includes the means to safely distribute information from a master node.

OpenSSL

www.openssl.org

Open source full-featured toolkit for the Transport Layer Security (TLS) and Secure Sockets Layer (SSL) protocols. Adopted by many products, including Fluentd.

Browserling Tools

www.browserling.com/tools

A set of online tools to help with format conversion, etc.

Chocolatey

https://chocolatey.org/

Chocolatey is a package manager for Windows. It provides a user experience with more Linux-like package managers such as rpm, yum, and others.

Clang

https://clang.llvm.org/

Libraries to support compiling the platform-native elements.

GCC

https://gcc.gnu.org/

The GCC compiler tools, needed if you build the OS native features for the source.

GraalVM

www.graalvm.org/

Next-generation multi-language virtual machine, supporting Java and other languages. It is capable of creating platform-native binaries.

Linux Package Manager Summary

https://mng.bz/4jDj

Provides information about the different package managers and the commands to use depending on your package manager.

NTP (Network Time Protocol)

https://doc.ntp.org/

The official site for the NTP definition; includes software and white papers on the subject.

Quarkus

https://quarkus.io/

A Java-based stack using OpenJDK capable of creating Kubernetes native solutions.

Semantic Versioning

https://semver.org/

A standard form for versioning artifacts.

N/A

PHP

https://www.php-fig.org/ psr/psr-3/

The PHP community has developed various standards (PSRs), including logging frameworks (PSR-3). PSR defines an interface standard for logging, so any framework supporting this will be compatible. A wide range of PHP frameworks supports this specification, from Zend to Drupal. Frameworks such as Magneto provide PSR-compliant logging that supports communication to Fluentd.

Django

Python

https://docs.djangoproject.com/ en/3.1/

Django is a web application framework that leverages native logging and provides some additions. This means the Fluentd extension can be incorporated.

Language native

Java

https://mng.bz/QWPv

This is part of the Java language; however, it isn’t the definitive logging solution. Many still prefer using other open source solutions.

Language native

C#—Language Native Library

https://mng.bz/XWNa

This is the C# native framework provided by Microsoft. It supports JSON configuration file controls and injection of loggers driven from the configuration.

Language native

VB.Net—Language Native Library

https://mng.bz/y4Qd

This covers the Microsoft provided capabilities for logging in Visual Basic using .Net.

Language native

Ruby—Language Native Library

https://docs.ruby-lang.org/ en/2.4.0/Logger.html

Ruby provides a native logging class with logging levels. The output is limited in choice. The language basics could be extended to support more destinations.

Language native

Python—Language Native Library

https://docs.python.org/ 3/library/logging.html

The Python native language features form the foundation for several logging frameworks that extend the capabilities, including a number of the Python frameworks that bring value with the addition of appenders. This includes the Fluentd Python library.

Language native

Go—Native Logging

https://golang.org/ pkg/log/

This is Go’s native logging package. Compared to many other logging mechanisms, this is quite simple and best seen as a set of helper methods for sending log events in a more structured way to stderr.

lgr

R

https://mng.bz/M2BB

A logging framework for the R language, drawing its design principles from Log4J. The number of appenders provided is smaller and focused on databases (understandable, given that this is a data analytics-focused language).

Log4cplus

C++

https://sourceforge.net/p/ log4cplus/wiki/Home/

Based on the Log4J framework.

Log4Cxx

C++

https://logging.apache.org/ log4cxx/latest _stable/

Port of Log4J as part of the Apache Log4J family.

Log4J2

Java

https://logging.apache.org/ log4j/2.x/index.html

Apache also provides several JVM-related language facades for Kotlin and Scala.

Log4Net

C# & VB and other languages supported on .Net framework

https://logging.apache.org/ log4net/

Log4Net is the port of the Log4J framework by the Apache Foundation, which developed the original framework.

Logrus

Go

https://github.com/ Sirupsen/logrus

Logrus is now in a maintenance-only state, as the developers feel it has reached its extensibility limits without breaking compatibility. However, it is referenced as it is commonly used.

Monolog

PHP

https://github.com/ Seldaek/monolog

Monolog is just a logging framework supporting the PSR-3 specification. It includes a formatter for Fluentd, which can be combined with a handler for socket-level communication to Fluentd.

NLog

C# & VB and other languages supported on the .Net framework

https://nlog-project.org/

HTTP appender included in the framework extension.

Pino

Node JS

https://getpino.io/

Pino lends itself to being integrated into various Node.JS frameworks, such as Express. It uses the concept of transports to send logs to other systems, including some native cloud vendor solutions. It does support sockets and HTTP endpoints, so these could be used to communicate with Fluentd.

Serilog

C# & VB and other languages supported on the .Net framework

https://serilog.net/

Serilog is an open source framework that promotes stronger structured logging. It provides a wide range of destinations for output, including Fluentd.

SLF4J

Java

https://www.slf4j.org/

Simple Logging Facade for Java (SLF4J) is not a framework, but an abstraction layer so that different logging frameworks can be used using the same foundations; for example, Logback and Log4J2.

Twisted

Python

https://twistedmatrix.com/trac/

This is another Python framework with an event-driven model. It provides a logging mechanism that integrates into the broader ecosystem. From there, Twisted can be used to send log events to a small set of native publishers or to send log events via Python’s native logging capability.

Winston JS

Node JS

https://github.com/ winstonjs/winston

Winston has many characteristics of a logging framework, with features such as log levels and appenders (called transports in Winston). The range of transports is limited compared to some, but a framework for building your own is provided.

DLA Piper Data Protection

www.dlapiperdataprotection.com

DLA Piper is a global law firm that has developed and maintained a website that provides good insight into individual country positioning on data protection.

GDPR

• https://gdpr-info.eu/

• https://ico.org.uk/for-organisations/guide-to-data-protection/

• https://oag.ca.gov/privacy/ccpa

GDPR (General Data Protection Regulation) was developed by the European Union to strengthen personal data protection. All the EU countries have ratified this legislation and some national legislation that builds upon it. A wide range of countries and US states have developed their own derivative legislation; for example, California’s California Consumer Privacy Act (CCPA).

HIPAA

www.hhs.gov/hipaa/index.html

Health Insurance Portability and Accountability Act (HIPAA) covers the detailing of health care–related information.

ISO/IEC 27001

www.iso.org/isoiec-27001-information-security.html

Many organizations look for ISO/IEC 27001 compliance; while not a legislative-driven set of rules, it is a best-practice set of standards.

PCI DSS (Payment Card Industry Data Security Standard)

www.pcisecuritystandards.org/

A standard adopted by all payment card operators defining a range of requirements from infrastructure to development practices.

Sarbanes-Oxley Act (SOX)

www.govinfo.gov/content/pkg/STATUTE-116/pdf/STATUTE-116-Pg745.pdf

The Sarbanes-Oxley Act (SOX) was developed in the United States to address corporate legal reporting. But as a by-product, it set down practices for security, including the handling of data. There are other national variants such as J-SOX (Japan), C-SOX (Canada), and TC-SOX (Turkey).

United Nations Conference on Trade and Development (UNCTAD)

https://mng.bz/aD1m

UNCTAD provides a similar resource to DLA Piper but is focused on e-commerce considerations.

Converting different time representations

www.epochconverter.com/

Converts timestamps to and from their second or millisecond epoch representations as used by Linux/Unix systems and languages such as Java.

Examples of documented error codes

• HTTP codes: https://datatracker.ietf.org/ doc/html/rfc7231#section-6.1

• Email codes: https://www.rfc-editor.org/ rfc/rfc5248.html

• WebLogic server: https://docs.oracle.com/ cd/E24329_01/doc.1211/ e26117/chapter_bea_messages.htm#sthref7

An example of good error code documentation. Covering HTTP and email from IETF and WebLogic Application Server.

ISO 8601 Date Time Standard

www.w3.org/TR/NOTE-datetime

This describes the different industry-standard ways to define date and time.

ITIL (Information Technology Infrastructure Library)

www.axelos.com/ best-practice-solutions/itil

An industry-standard set of recommended practices and processes for things like ITSM (IT service management)

Payment Card Industry (PCI) Security Standards Council (SSC)

https://www.pcisecuritystandards.org/

The organization that defines PCI DSS (Data Security Standard)—the security standards for processing payment card data.

Regular expression development

www.regular-expressions.info/

This website gives a detailed insight into the use of regular expressions.

N-tier architectures

https://stackify.com/ n-tier-architecture/

https://livebook.manning.com/book/the-cloud-at-your-service/chapter-6/point-12033-14-14-1

Provides an explanation to N-tier architectures and the value proposition they make.

TCP and UDP Network protocols

www.vpnmentor.com/blog/tcp-vs-udp/

www.cs.dartmouth.edu/ ~campbell/cs60/socketprogramming.html

Explanation of TCP and UDP and their differences.

Fluentd-supplied logging libraries

https://github.com/fluent/fluent-logger-java

Fluentd provided a library for direct logging from Java.

https://github.com/fluent/fluent-logger-ruby

Fluentd provided a library for direct logging from Ruby.

https://github.com/fluent/fluent-logger-python

Fluentd provided a library for direct logging from Python.

https://github.com/fluent/fluent-logger-perl

Fluentd provided a library for direct logging from Perl.

https://github.com/fluent/fluent-logger-php

Fluentd provided a library for direct logging from PHP.

https://github.com/fluent/fluent-logger-node

Fluentd provided a library for direct logging from NodeJS.

https://github.com/fluent/fluent-logger-scala

Fluentd provided a library for direct logging from Scala.

https://github.com/fluent/fluent-logger-golang

Fluentd provided a library for direct logging from Go.

https://github.com/fluent/fluent-logger-erlang

Fluentd provided a library for direct logging from Erlang.

https://github.com/fluent/fluent-logger-ocaml

Fluentd provided a library for direct logging from OCaml.

msgpack

https://msgpack.org/

Compression library implemented in multiple languages that Fluentd can utilize. Used when the forward plugin is sending or receiving events.

Core Kubernetes

https://www.manning.com/books/core -kubernetes

Fluentd is often used within the context of Kubernetes and is just one aspect of a Kubernetes setup. This book and Kubernetes in Action will cover much of what you’ll need.

Design Patterns

https://refactoring.guru/design -patterns/catalog

The details of the core patterns first described by the Gang of Four (GoF) and their book Design Patterns: Elements of Reusable Object-Oriented Software. The Gang of Four are Erich Gamma, John Vlissides, Richard Helm, and Ralph Johnson. Here we have provided a link to a brief guide to each of the patterns.

Docker in Action

https://mng.bz/g4Bv

Docker is the typical tech to implement a container. When we do not need the sophistication of Kubernetes, we’ll use Docker more directly. This book covers the core of the building and running containers.

Effective Unit Testing

www.manning.com/books/effective-unit-testing

In chapter 9, when we implemented our custom plugin, we looked at unit testing. Ideally, the unit testing built for production use is extensive. This book, while focused on Java, will provide insight into the best practices.

Elasticsearch in Action

www.manning.com/books/elasticsearch-in-action

This book can be handy when working with Elasticsearch and Fluentd.

Groovy in Action, Second Edition

https://mng.bz/en1V

Our logging simulator was built using Groovy, making it extremely easy to bolt on enhancements to effectively simulate different sources. If you are interested in digging further into Groovy, this is the book that will help.

Kubernetes in Action

https://mng.bz/p2PK

This book covers a lot of details about Kubernetes and the way it works, giving more insight into container orchestration.

MongoDB in Action, Second Edition

https://mng.bz/OGxw

We use MongoDB as an output target. This will provide all the information likely to be needed relating to MongoDB.

Operations Anti-Patterns, DevOps Solutions

https://mng.bz/Yg1z

Getting your operational processes and logging is an integral part of achieving a DevOps way of working. This book looks at the potential pitfalls (or anti-patterns) you could end up facing.

Redis in Action

www.manning.com/books/redis-in-action

We used Redis as part of our example of building a custom plugin. This book will give more insight into Redis.

Ruby in Practice

www.manning.com/books/ruby-in -practice

More practical guidance on working with Ruby.

Software Telemetry

www.manning.com/books/software-telemetry

Software Telemetry is about the idea of getting metrics, logging, and certain types of business application state data and using them to provide you with a health perspective. Fluentd can be a crucial part of a telemetry solution.

The Well-Grounded Rubyist, Third Edition

https://mng.bz/GGyD

If you’re considering getting under the hood of Fluentd or developing your own custom plugins, then this is a great read.

Securing DevOps

www.manning.com/books/securing -devops

Looks at the needs and techniques for securing cloud environments.

YAML

https://yaml.org/

The official site for YAML includes details of its syntax.