When using a system that manages logs, it is very common to use rolling files for your log entries. Taking this idea, we can have indices that are similar to rolling files.
We can define some conditions to be checked and leave it to Elasticsearch to automatically roll new indices and refer via an alias to only a virtual index.
You need an up-and-running Elasticsearch installation, as used in the Downloading and installing Elasticsearch recipe in Chapter 2, Downloading and Setup.
To execute curl via the command line, you need to install curl for your operative system.
To enable a rolling index, we need an index with an alias that only points to it. For example, to set a log rolling index we follow these steps:
- We need an index with a
logs_writealias that only points to it:curl -XPUT 'http://127.0.0.1:9200/mylogs-000001' -d ' { "aliases": { "logs_write": {} } }'The result will be an acknowledgement:
{"acknowledged":true} - We can add the rolling to the
logs_writealias:curl -XPOST 'http://127.0.0.1:9200/logs_write/_rollover? pretty'-d ' { "conditions": { "max_age": "7d", "max_docs": 100000 }, "settings": { "index.number_of_shards": 3 } }'The result will be as follows:
{ "old_index" : "mylogs-000001", "new_index" : "mylogs-000001", "rolled_over" : false, "dry_run" : false, "conditions" : { "[max_docs: 100000]" : false, "[max_age: 7d]" : false } } - In case your alias doesn't point to a single index, a similar error is returned:
{ "error" : { "root_cause" : [ { "type" : "illegal_argument_exception", "reason" : "source alias maps to multiple indices" } ], "type" : "illegal_argument_exception", "reason" : "source alias maps to multiple indices" }, "status" : 400 }
The rolling index is a special alias that manages the auto-creation of new indices when one of the conditions is matched.
This is a very convenient functionality because it is completely managed by Elasticsearch, reducing a lot of user custom backend code.
The information of creating the new index is taken from the source, but you can also apply custom settings on index creation.
The name convention is automatically managed by Elasticsearch, automatically incrementing the numeric part of the index name (by default, it uses six ending digits).