Home Page Icon
Home Page
Table of Contents for
Table of Contents
Close
Table of Contents
by Saurabh Chhajed
Learning ELK Stack
Learning ELK Stack
Table of Contents
Learning ELK Stack
Credits
About the Author
About the Reviewers
www.PacktPub.com
Support files, eBooks, discount offers, and more
Why subscribe?
Free access for Packt account holders
Preface
What this book covers
What you need for this book
Who this book is for
Conventions
Reader feedback
Customer support
Downloading the example code
Downloading the color images of this book
Errata
Piracy
Questions
1. Introduction to ELK Stack
The need for log analysis
Issue debugging
Performance analysis
Security analysis
Predictive analysis
Internet of things and logging
Challenges in log analysis
Non-consistent log format
Tomcat logs
Apache access logs – combined log format
IIS logs
Variety of time formats
Decentralized logs
Expert knowledge requirement
The ELK Stack
Elasticsearch
Logstash
Kibana
ELK data pipeline
ELK Stack installation
Installing Elasticsearch
Running Elasticsearch
Elasticsearch configuration
Network Address
Paths
The cluster name
The node name
Elasticsearch plugins
Installing Logstash
Running Logstash
Logstash with file input
Logstash with Elasticsearch output
Configuring Logstash
Installing Logstash forwarder
Logstash plugins
Input plugin
Filters plugin
Output plugin
Installing Kibana
Configuring Kibana
Running Kibana
Kibana interface
Discover
Visualize
Dashboard
Settings
Summary
2. Building Your First Data Pipeline with ELK
Input dataset
Data format for input dataset
Configuring Logstash input
Filtering and processing input
Putting data to Elasticsearch
Visualizing with Kibana
Running Kibana
Kibana visualizations
Building a line chart
Building a bar chart
Building a Metric
Building a data table
Summary
3. Collect, Parse and Transform Data with Logstash
Configuring Logstash
Logstash plugins
Listing all plugins in Logstash
Data types for plugin properties
Array
Boolean
Codec
Hash
String
Comments
Field references
Logstash conditionals
Types of Logstash plugins
Input plugins
file
Configuration options
add_field
codec
delimiter
exclude
path
sincedb_path
sincedb_write_interval
start_position
tags
type
stdin
Configuration options
add_field
codec
tags
type
twitter
Configuration options
add_field
codec
consumer_key
consumer_secret
full_tweet
keywords
oauth_token
oauth_token_secret
tags
type
lumberjack
Configuration options
add_field
codec
host
port
ssl_certificate
ssl_key
ssl_key_passphrase
tags
type
redis
Configuration options
add_field
codec
data_type
host
key
password
port
Output plugins
csv
Configuration options
codec
csv_options
fields
gzip
path
file
Configuration options
email
Configuration options
attachments
body
cc
from
to
htmlbody
replyto
subject
elasticsearch
Configuration options
ganglia
Configuration options
metric
unit
value
jira
Configuration options
kafka
Configuration options
topic_id
lumberjack
Configuration options
hosts
port
ssl_certificate
redis
Configuration options
rabbitmq
stdout
mongodb
Configuration options
collection
database
uri
Filter plugins
csv
Configuration options
date
Configuration options
drop
Configuration options
geoip
Configuration options
source
grok
Custom grok patterns
mutate
Configuration options
sleep
Codec plugins
json
line
multiline
plain
rubydebug
Summary
4. Creating Custom Logstash Plugins
Logstash plugin management
Plugin lifecycle management
Installing a plugin
Updating a plugin
Uninstalling a plugin
Structure of a Logstash plugin
Required dependencies
Class declaration
Configuration name
Configuration options setting
Plugin methods
Input plugin
Filter plugin
Output plugin
Codec plugin
Writing a Logstash filter plugin
Building the plugin
Summary
5. Why Do We Need Elasticsearch in ELK?
Why Elasticsearch?
Elasticsearch basic concepts
Index
Document
Field
Type
Mapping
Shard
Primary shard and replica shard
Cluster
Node
Exploring the Elasticsearch API
Listing all available indices
Listing all nodes in a cluster
Checking the health of the cluster
Health status of the cluster
Creating an index
Retrieving the document
Deleting documents
Deleting an index
Elasticsearch Query DSL
Elasticsearch plugins
Bigdesk plugin
Elastic-Hammer plugin
Head plugin
Summary
6. Finding Insights with Kibana
Kibana 4 features
Search highlights
Elasticsearch aggregations
Scripted fields
Dynamic dashboards
Kibana interface
Discover page
Time filter
Quick time filter
Relative time filter
Absolute time filter
Kibana Auto-refresh setting
Querying and searching data
Freetext search
AND
OR
NOT
Groupings
Wildcard searches
Field searches
Range searches
Special characters escaping
New search
Saving the search
Loading a search
Field searches using field list
Summary
7. Kibana – Visualization and Dashboard
Visualize page
Creating a visualization
Visualization types
Metrics and buckets aggregations
Buckets
Date Histogram
Histogram
Range
Date Range
Terms
Metrics
Count
Average, Sum, Min, and Max
Unique Count
Advanced options
Visualizations
Area chart
Data table
Line chart
Markdown widget
Metric
Pie chart
Tile map
Vertical bar chart
Dashboard page
Building a new dashboard
Saving and loading a dashboard
Sharing a dashboard
Summary
8. Putting It All Together
Input dataset
Configuring Logstash input
Grok pattern for access logs
Visualizing with Kibana
Running Kibana
Searching on the Discover page
Visualizations – charts
Building a Line chart
Building an Area chart
Building a Bar chart
Building a Markdown
Dashboard page
Summary
9. ELK Stack in Production
Prevention of data loss
Data protection
System scalability
Data retention
ELK Stack implementations
ELK Stack at LinkedIn
Problem statement
Criteria for solution
Solution
Kafka at LinkedIn
Operational challenges
Logging using Kafka at LinkedIn
ELK at SCA
How is ELK used in SCA?
How is it helping in analytics?
ELK for monitoring at SCA
ELK at Cliffhanger Solutions
Kibana demo – Packetbeat dashboard
Summary
10. Expanding Horizons with ELK
Elasticsearch plugins and utilities
Curator for index management
Curator commands
Curator installation
Shield for security
Shield installation
Adding users and roles
Using Kibana4 on shield protected Elasticsearch
Marvel to monitor
Marvel installation
Marvel dashboards
ELK roadmap
Elasticsearch roadmap
Logstash roadmap
Event persistence capability
End-to-end message acknowledgement
Logstash monitoring and management API
Kibana roadmap
Summary
Index
Search in book...
Toggle Font Controls
Playlists
Add To
Create new playlist
Name your new playlist
Playlist description (optional)
Cancel
Create playlist
Sign In
Email address
Password
Forgot Password?
Create account
Login
or
Continue with Facebook
Continue with Google
Sign Up
Full Name
Email address
Confirm Email Address
Password
Login
Create account
or
Continue with Facebook
Continue with Google
Prev
Previous Chapter
Cover
Next
Next Chapter
Learning ELK Stack
Table of Contents
Learning ELK Stack
Credits
About the Author
About the Reviewers
www.PacktPub.com
Support files, eBooks, discount offers, and more
Why subscribe?
Free access for Packt account holders
Preface
What this book covers
What you need for this book
Who this book is for
Conventions
Reader feedback
Customer support
Downloading the example code
Downloading the color images of this book
Errata
Piracy
Questions
1. Introduction to ELK Stack
The need for log analysis
Issue debugging
Performance analysis
Security analysis
Predictive analysis
Internet of things and logging
Challenges in log analysis
Non-consistent log format
Tomcat logs
Apache access logs – combined log format
IIS logs
Variety of time formats
Decentralized logs
Expert knowledge requirement
The ELK Stack
Elasticsearch
Logstash
Kibana
ELK data pipeline
ELK Stack installation
Installing Elasticsearch
Running Elasticsearch
Elasticsearch configuration
Network Address
Paths
The cluster name
The node name
Elasticsearch plugins
Installing Logstash
Running Logstash
Logstash with file input
Logstash with Elasticsearch output
Configuring Logstash
Installing Logstash forwarder
Logstash plugins
Input plugin
Filters plugin
Output plugin
Installing Kibana
Configuring Kibana
Running Kibana
Kibana interface
Discover
Visualize
Dashboard
Settings
Summary
2. Building Your First Data Pipeline with ELK
Input dataset
Data format for input dataset
Configuring Logstash input
Filtering and processing input
Putting data to Elasticsearch
Visualizing with Kibana
Running Kibana
Kibana visualizations
Building a line chart
Building a bar chart
Building a Metric
Building a data table
Summary
3. Collect, Parse and Transform Data with Logstash
Configuring Logstash
Logstash plugins
Listing all plugins in Logstash
Data types for plugin properties
Array
Boolean
Codec
Hash
String
Comments
Field references
Logstash conditionals
Types of Logstash plugins
Input plugins
file
Configuration options
add_field
codec
delimiter
exclude
path
sincedb_path
sincedb_write_interval
start_position
tags
type
stdin
Configuration options
add_field
codec
tags
type
twitter
Configuration options
add_field
codec
consumer_key
consumer_secret
full_tweet
keywords
oauth_token
oauth_token_secret
tags
type
lumberjack
Configuration options
add_field
codec
host
port
ssl_certificate
ssl_key
ssl_key_passphrase
tags
type
redis
Configuration options
add_field
codec
data_type
host
key
password
port
Output plugins
csv
Configuration options
codec
csv_options
fields
gzip
path
file
Configuration options
email
Configuration options
attachments
body
cc
from
to
htmlbody
replyto
subject
elasticsearch
Configuration options
ganglia
Configuration options
metric
unit
value
jira
Configuration options
kafka
Configuration options
topic_id
lumberjack
Configuration options
hosts
port
ssl_certificate
redis
Configuration options
rabbitmq
stdout
mongodb
Configuration options
collection
database
uri
Filter plugins
csv
Configuration options
date
Configuration options
drop
Configuration options
geoip
Configuration options
source
grok
Custom grok patterns
mutate
Configuration options
sleep
Codec plugins
json
line
multiline
plain
rubydebug
Summary
4. Creating Custom Logstash Plugins
Logstash plugin management
Plugin lifecycle management
Installing a plugin
Updating a plugin
Uninstalling a plugin
Structure of a Logstash plugin
Required dependencies
Class declaration
Configuration name
Configuration options setting
Plugin methods
Input plugin
Filter plugin
Output plugin
Codec plugin
Writing a Logstash filter plugin
Building the plugin
Summary
5. Why Do We Need Elasticsearch in ELK?
Why Elasticsearch?
Elasticsearch basic concepts
Index
Document
Field
Type
Mapping
Shard
Primary shard and replica shard
Cluster
Node
Exploring the Elasticsearch API
Listing all available indices
Listing all nodes in a cluster
Checking the health of the cluster
Health status of the cluster
Creating an index
Retrieving the document
Deleting documents
Deleting an index
Elasticsearch Query DSL
Elasticsearch plugins
Bigdesk plugin
Elastic-Hammer plugin
Head plugin
Summary
6. Finding Insights with Kibana
Kibana 4 features
Search highlights
Elasticsearch aggregations
Scripted fields
Dynamic dashboards
Kibana interface
Discover page
Time filter
Quick time filter
Relative time filter
Absolute time filter
Kibana Auto-refresh setting
Querying and searching data
Freetext search
AND
OR
NOT
Groupings
Wildcard searches
Field searches
Range searches
Special characters escaping
New search
Saving the search
Loading a search
Field searches using field list
Summary
7. Kibana – Visualization and Dashboard
Visualize page
Creating a visualization
Visualization types
Metrics and buckets aggregations
Buckets
Date Histogram
Histogram
Range
Date Range
Terms
Metrics
Count
Average, Sum, Min, and Max
Unique Count
Advanced options
Visualizations
Area chart
Data table
Line chart
Markdown widget
Metric
Pie chart
Tile map
Vertical bar chart
Dashboard page
Building a new dashboard
Saving and loading a dashboard
Sharing a dashboard
Summary
8. Putting It All Together
Input dataset
Configuring Logstash input
Grok pattern for access logs
Visualizing with Kibana
Running Kibana
Searching on the Discover page
Visualizations – charts
Building a Line chart
Building an Area chart
Building a Bar chart
Building a Markdown
Dashboard page
Summary
9. ELK Stack in Production
Prevention of data loss
Data protection
System scalability
Data retention
ELK Stack implementations
ELK Stack at LinkedIn
Problem statement
Criteria for solution
Solution
Kafka at LinkedIn
Operational challenges
Logging using Kafka at LinkedIn
ELK at SCA
How is ELK used in SCA?
How is it helping in analytics?
ELK for monitoring at SCA
ELK at Cliffhanger Solutions
Kibana demo – Packetbeat dashboard
Summary
10. Expanding Horizons with ELK
Elasticsearch plugins and utilities
Curator for index management
Curator commands
Curator installation
Shield for security
Shield installation
Adding users and roles
Using Kibana4 on shield protected Elasticsearch
Marvel to monitor
Marvel installation
Marvel dashboards
ELK roadmap
Elasticsearch roadmap
Logstash roadmap
Event persistence capability
End-to-end message acknowledgement
Logstash monitoring and management API
Kibana roadmap
Summary
Index
Add Highlight
No Comment
..................Content has been hidden....................
You can't read the all page of ebook, please click
here
login for view all page.
Day Mode
Cloud Mode
Night Mode
Reset
