Logstash can be installed where you are comfortable sending your network log to. The installation steps are available at https://www.elastic.co/guide/en/logstash/current/installing-logstash.html. By default, you can put the Logstash configuration file under /etc/logstash/conf.d/. The file is in the input-filter-output format (https://www.elastic.co/guide/en/logstash/current/advanced-pipeline.html). In the following example, we specified the input as a network log file, with a placeholder for filtering input, and the output as both printing out message to the console as well as having the output exported toward our AWS Elasticsearch Service instance:
input {
file {
type => "network_log"
path => "path to your network log file"
}
}
filter {
if [type] == "network_log" {
}
}
output {
stdout { codec => rubydebug }
elasticsearch {
index => "logstash_network_log-%{+YYYY.MM.dd}"
hosts => ["http://<instance>.<region>.es.amazonaws.com"]
}
}
Now let's look at what more we can do with Python and Logstash.