As we already know, Logstash has a rich set of plugins for different types of inputs, outputs and filters, which can read, parse, and filter data as per our needs. We will utilize the file input plugin to read the source file.
A file input plugin streams events from the input file, and each event is assumed as a single line. It automatically detects file rotation and handles it. It maintains the location where it left reading, and will automatically detect the new data if configured correctly. It reads files in a similar manner:
tail -0f
In general, a file input plugin configuration will look as follows:
input {
file {
path => #String (path of the files) (required)
start_position => #String (optional, default "end")
tags => #array (optional)
type => #string (optional)
}
}path: Thepathfield is the only required field infileinput plugin, which represents the path of the file from where input events have to be processed.start_position: This defines from where Logstash starts reading input files. Values can be"beginning"or"end". The default value is"end"which caters to the needs of reading live streams. If we need to read some historic data, it can be set to"beginning".tags:tagsrepresents any number of strings as an array that can be utilized later to filter and process events based on tags assigned to them.type: Thetypefield can be used to mark a specific type of events, which helps to filter and search them later. Type is added to the document that is stored in Elasticsearch, and can later be viewed in Kibana under the_typefield. For example, we can assign type as"error_logs"or"info_logs".
Let's configure Logstash for our input dataset:
input{
file{
path =>"/opt/logstash/input/GOOG.csv"
start_position =>"beginning"
}
}We will provide the path of the CSV file in the path attribute, and as our dataset is historic, we will use start_position as "beginning".