The file plugin is used to stream events from file(s) line by line. It works in a similar fashion to the tail -0f LinuxUnix command. For each file, it keeps track of any changes in the file, and the last location from where the file was read, only sends the data since it was last read. It also automatically detects file rotation. This plugin also provides the option to read the file from the beginning of the file.

The file plugin keeps account of the current position in each file. It does so by recording the current position in a separate file named sincedb. This makes it possible as well as convenient to stop and restart Logstash and have it pick up where it left off without missing the lines that were added to the file while Logstash was stopped.

The location of the sincedb file is set to <path.data>/plugins/inputs/file by default, which can be overridden by providing the file path for the sincedb_path plugin parameter. The only required parameter for this plugin is the path parameter, which accepts one or more files to read from.

Let's take some example configurations to understand this plugin better:

#sample configuration 1
#simple1.conf

input
{ file{ 
    path => "/usr/local/logfiles/*"
 }
}
 output 
{ 
 stdout { 
 codec => rubydebug 
 }
}

The preceding configuration specifies the streaming of all the new entries (that is, tailing the files) to the files found under the /usr/local/logfiles/ location:

#sample configuration 2 
#simple2.conf
input
{
    file{
        path => ["D:esapp*","D:eslogs*.txt"]
        start_position => "beginning"    
        exclude => ["*.csv] 
        discover_interval => "10s"
        type => "applogs" 
    }
}

output
{ 
  stdout { 
    codec => rubydebug 
    }
} 

The preceding configuration specifies the streaming of all the log entries/lines in the files found under the D:esapp* location, and only files of the .txt type. Files found under the D:eslogs*.txt location, starting from the beginning (specified by the start_position => "beginning" parameter), and while looking for files, it excludes files of the .csv type (specified by the exclude => ["*.csv] parameter, which takes an array of values). Every line that's streamed would be stored in the message field by default. The preceding configuration also specified to add a new additional field type with the applogs value (specified by the type => "applogs" parameter). Adding additional fields would be helpful while transforming events in filter plugins or identifying the events in the output. The discover_interval parameter is used to define how often the path will be expanded to search for new files that are created inside the location specified in the path parameter.