Prior to Elasticsearch 5.0, if we wanted to preprocess documents before indexing them to Elasticsearch, then the only way was to make use of Logstash or preprocess them programmatically/manually and then index them to Elasticsearch. Elasticsearch lacked the ability to preprocess/transform the documents, and it just indexed the documents as they were. However, the introduction of a feature called ingest node in Elasticsearch 5.x onward provided a lightweight solution for preprocessing and enriching documents within Elasticsearch itself before they are indexed.

If an Elasticsearch node is implemented with the default configuration, by default, it would be master, data, and ingest enabled (that is, it would act as a master node, data node, and ingest node). To disable ingest on a node, configure the following setting in the elasticsearch.yml file:

node.ingest: false

The ingest node can be used to preprocess documents before the actual indexing is performed on the document. This preprocessing is performed via an ingest node that intercepts bulk and index requests, applies the transformations to the data, and then passes the documents back to the index or bulk APIs. With the release of the new ingest feature, Elasticsearch has taken out the filter part of Logstash so that we can do our processing of raw logs and enrichment within Elasticsearch.

To preprocess a document before indexing, we must define the pipeline (which contains sequences of steps known as processors for transforming an incoming document). To use a pipeline, we simply specify the pipeline parameter on an index or bulk request to tell the ingest node which pipeline to use:

POST my_index/my_type?pipeline=my_pipeline_id
{
  "key": "value"
}