Data enrichment on ingest

Another option is to move the decision making about which hosts belong to which applications further upstream to the time of ingest. One way to do this would be to direct your data through Logstash, and use a filter plugin to add additional fields to the data based upon a lookup against an asset list (file, database, and so on). Consult the Logstash documentation at https://www.elastic.co/guide/en/logstash/current/lookup-enrichment.html, which shows you how to dynamically enrich the indexed documents with additional fields to provide context.

For example, you could have Logstash add an application_name field and dynamically populate the value of this field with the appropriate name of the application, for example (truncated JSON here):

   "host": "wasinv2.acme.com", 
"application_name": "invoice_processing",

Alternatively, you could do the following:

   "host": "www3.acme.com", 
"application_name": "online_purchases",

Once the value of this field is set and inserted into the indexed documents, then you would use the application_name field, along with the ability of filtering the query for the ML job (as previously described) to limit your data analysis to the pertinent application of interest. The addition of the data enrichment step may seem like a little more up-front effort, but it should pay dividends in the long term as it will be easier to maintain as asset names change or evolve, since the first method requires hardcoding the asset names into the searches of the ML jobs.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
18.189.195.91