Filebeat modules

Filebeat modules simplify the process of collecting, parsing, and visualizing logs of common formats. 

A module is made up of one or more filesets. A fileset is made up of the following:

  • Filebeat input configurations that contain the default paths needed to look out for logs. It also provides configuration for combining multiline events when needed.
  • An Elasticsearch Ingest pipeline definition to parse and enrich logs. 
  • Elasticsearch templates, which define the field definitions so that appropriate mappings are set to the fields of the events.
  • Sample Kibana dashboards, which can be used for visualizing logs.
Filebeat modules require the Elasticsearch Ingest node. The version of Elasticsearch should be greater that 5.2.

Some of the modules that are shipped with Filebeat are as follows:

  • Apache module
  • Auditd module
  • Elasticsearch module
  • Haproxy module
  • IIS module
  • Kafka module
  • MongoDB module
  • MySQL module
  • Nginx module
  • PostgreSQL module
  • Redis module

The modules.d directory contains the default configurations for all the modules that are available in Filebeat. Any configuration that's specific to a module is stored in a .yml file, with the name of the file being the name of the module. For example, the configuration related to the redis module would be stored in the redis.yml file.

Since each module comes with the default configuration, make the appropriate changes in the module configuration file.

The basic configuration for the redis module is as follows:

#redis.yml
- module: redis
# Main logs
log:
enabled: true

# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths: ["/var/log/redis/redis-server.log*"]

# Slow logs, retrieved via the Redis API (SLOWLOG)
slowlog:
enabled: true

# The Redis hosts to connect to.
#var.hosts: ["localhost:6379"]

# Optional, the password to use when connecting to Redis.
#var.password:

To enable modules, execute the modules enable command, passing one or more module names:

Windows:
E:filebeat-7.0.0-windows-x86_64>filebeat.exe modules enable redis mysql

Linux:
[locationOfFileBeat]$./filebeat modules enable redis mysql
If a module is disabled, then in the modules.d directory the configuration related to the module will be stored with .disabled extension.

To disable modules, execute the modules disable command, passing one or more module names to it. For example:

Windows:
E:filebeat-7.0.0-windows-x86_64>filebeat.exe modules disable redis mysql

Linux:
[locationOfFileBeat]$./filebeat modules disable redis mysql

Once the module is enabled, to load the recommended index template for writing to Elasticsearch, and to deploy sample dashboards for visualizing data in Kibana, execute the setup command, as follows:

Windows:
E:filebeat-7.0.0-windows-x86_64>filebeat.exe -e setup

Linux:
[locationOfFileBeat]$./filebeat -e setup

The -e flag specifies logging the output to stdout. Once the modules are enabled and the setup command is run, to load index templates and sample dashboards, start Filebeat as usual so that it can start shipping logs to Elasticsearch.

The setup command has to be executed while installating or upgrading Filebeat, or after a new module is enabled.

Most modules have dependency plugins such as ingest-geoip and ingest-user-agent, which need to be installed on Elasticsearch prior to setting up the modules, otherwise the setup will fail.

Rather than enabling the modules by passing them as command-line parameters, you can enable the modules in the filebeat.yml configuration file itself, and start Filebeat as usual:

filebeat.modules:
- module: nginx
- module: mysql

Each of the modules has associated filesets which contain certain variables that can be overridden either using the configuration file or by passing it as a command-line parameter using the -M flag when running Filebeat.

For the configuration file, use the following code:

filebeat.modules:
- module: nginx
  access:
    var.paths: ["C:
gnixaccess.log*"]

For the command line, use the following code:

Windows:
E:filebeat-7.0.0-windows-x86_64>filebeat.exe -e -modules=nginx -M "nginx.access.var.paths=[C: gnixaccess.log*]"

Linux:
[locationOfFileBeat]$./filebeat -e -modules=nginx -M "nginx.access.var.paths=[var gnixaccess.log*]"
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.135.246.72