contents
Part 1. From zero to “Hello World”
1.1 Elevator pitch for Fluentd
Fluentd compared to middleware
1.3 Evolving ideas
Three pillars of observability
1.4 Log unification
Unifying logs vs. log analytics
1.5 Software stacks
Comparing Fluentd and Logstash
The relationship between Fluentd and Fluent Bit
The relationship between Logstash and Beats
1.6 Log routing as a vehicle for security
Relationship to major cloud vendors PaaS/IaaS
1.9 Where can Fluentd and Fluent Bit be used?
1.11 Plugins
1.12 How Fluentd can be used to make operational tasks easier
Rapid operational consolidation
2 Concepts, architecture, and deployment of Fluentd
2.1 Architecture and core concepts
Fluent configuration execution order
Putting timing requirements into action
Deploying Fluentd for the book’s examples
Deployment considerations for Fluentd
2.3 Bringing Fluentd to life with “Hello World”
2.4 “Hello World” with Fluent Bit
Alternate Fluent Bit startup options
Fluent Bit configuration file comparison
Fluent Bit configuration file in detail
Putting the dummy plugin into action
2.5 Fluentd deployment with Kubernetes and containers
2.6 Using Fluentd UI
Part 2. Fluentd in depth
3 Using Fluentd to capture log events
3.1 Dry running to check a configuration
Putting validating Fluentd configuration into action
Putting the adaption of a Fluentd configuration to Fluent Bit into action
Rereading and resuming reading of log files
Configuration considerations for tracking position
Wildcards in the path attribute
Controlling the impact of wildcards in filenames
Replacing wildcards with delimited lists in action
3.3 Self-monitoring
3.4 Imposing structure on log events
Applying a Regex parser to a complex log
Putting parser configuration into action
4 Using Fluentd to output log events
Chunks and Controlling Buffering
Putting configuring buffering size settings into action
Putting JSON formatter configuration into action
4.3 Sending log events to MongoDB
Deploying MongoDB Fluentd plugin
Configuring the Mongo output plugin for Fluentd
Putting MongoDB connection configuration strings into action
Actionable log events through service invocation
Actionable through user interaction tools
4.5 Slack to demonstrate the social output
Handling tokens and credentials more carefully
Externalizing Slack configuration attributes in action
4.6 The right tool for the right job
5.1 Reaching multiple outputs by copying
5.2 Configuration reuse and extension through inclusion
Place holding with null output
Putting inclusions with a MongoDB output into action
5.3 Injecting context into log events
Putting tag naming conventions into action
Putting dynamic tagging with extract into action
5.5 Tag plugins
5.6 Labels: Taking tags to a new level
Using a stdout filter to see what is happening
Illustrating label and tag routing
Putting a common pipeline into action
All is well events do not need to be distributed
Spotting the needle in a haystack
Easier to process meaning downstream
Record when we have reacted to a log event
6.3 Applying filters and parsers
Changing log events with the record_transformer plugin
Filter parser vs. record transformer
6.4 Demonstrating change impact with stdout in action
A solution demonstrating change impact with stdout in action
6.6 Deriving new data values with the record_transformer
Putting the incorporation of calculations into a log event transformation into action
6.7 Generating simple Fluentd metrics
Putting log event counting into action
Part 3. Beyond the basics
7.1 Threading and processes to scale with workers
Controlling output plugin threads
Memory management optimization
7.2 Scaling and moving workloads
Fan-in/log aggregation and consolidation
Fan-out and workload distribution
Putting a high-availability comparison into action
7.3 Fluentd scaling in containers vs. native and virtual environments
Kubernetes worker node configuration
7.4 Securing traffic between Fluentd nodes
Certificate and private key storage
Security is more than certificates
Putting certification into action
8 Driving logs with Docker and Kubernetes
8.1 Fluentd out of the box from Docker Hub
Getting set up for Docker log drivers
Docker drivers via the command line
A quick check of network connections
Switching to driver configuration through a configuration file
8.3 Kubernetes components logging and the use of Fluentd
Kubernetes components and structured logging
Kubernetes default log retention and log rotation
8.4 Demonstrating logging with Kubernetes
Understanding how Fluentd DaemonSets are put together
8.5 Getting a peek at host logs
8.6 Configuring a Kubernetes logging DaemonSet
Getting the Fluentd configuration ready to be used
Creating our Kubernetes deployment configuration
Putting the implementation of a Fluentd for Kubernetes into action
8.7 Kubernetes configuration in action
8.8 More Kubernetes monitoring and logging to watch for
9.2 What is Redis, and why build a plugin with the Redis list capability?
Redis list over RedisTimeSeries
9.3 Illustrating our objective using Redis CLI
9.6 Implementing the plugin core
How configuration attributes work
Getting the plugin to work with our Fluentd installation
Putting additional configuration validation into action
Implementing the Redis output logic
Putting the testing of synchronous output into action
9.7 Implementing the Redis input plugin
Testing input and output plugin execution
9.8 Extending output with buffering
Improving our scenario by putting maintainability into action
9.9 Unit testing
9.10 Putting the development of unit tests into action
Complete metadata aka manifest
Rerun without the plugin paths
9.12 Extending to be an enterprise-class solution
Part 4. Good logging practices and frameworks to maximize log value
10.1 Audit events vs. log events
10.2 Log levels and severities
Extending or creating your own log levels
10.3 Clear language
10.4 Human and machine-readable
10.5 Context is key
A practical checklist for capturing context
10.6 Error codes
Codes can be for more than errors
10.7 Too little logging or too much?
Putting making log entries ready for application shipping into action
10.9 Use frameworks if you can
10.10 Development practices
Using standard exceptions and error structures
String construction as a reason not to log
11.1 Value of logging frameworks
11.2 Typical structure of a logging framework
11.3 Appender structures
11.4 Logging framework landscape
11.5 Choosing a framework
Putting optimizing application logging into action
11.6 Fluentd’s own logging and appenders
11.7 Illustrations of an application logging directly to Fluentd
Python with logging framework: Using the Fluentd library
Invoking Fluentd appender directly
Illustration with only Python’s logging
Illustration without Python’s logging or Fluentd library
Porting the Fluentd calls to another language into action
Using generic appenders: The takeaways
Appendix A. Installation of additional tools and services
Appendix B. Processing times and dates, regular expressions, and other configuration values
Appendix C. Plugins summary
Appendix D. Real-world use case
Appendix E. Useful resources