Beginning Elastic Stack

Vishal Sharma

New Delhi, Delhi, India

Any source code or other supplementary materials referenced by the author in this text are available to readers at www.apress.com . For detailed information about how to locate your book’s source code, go to www.apress.com/source-code/ . Readers can also access source code at SpringerLink in the Supplementary Material section for each chapter.

ISBN 978-1-4842-1693-4

e-ISBN 978-1-4842-1694-1

DOI 10.1007/978-1-4842-1694-1

Library of Congress Control Number: 2016961231

This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed.

Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with every occurrence of a trademarked name, logo, or image, we use the names, logos, and images only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights.

While the advice and information in this book are believed to be true and accurate at the date of publication, neither the author nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The Publisher makes no warranty, express or implied, with respect to the material contained herein.

Printed on acid-free paper

Distributed to the book trade worldwide by Springer Science+Business Media New York, 233 Spring Street, 6th Floor, New York, NY 10013. Phone 1-800-SPRINGER, fax (201) 348-4505, e-mail [email protected], or visit www.springer.com. Apress Media, LLC is a California LLC and the sole member (owner) is Springer Science + Business Media Finance Inc (SSBM Finance Inc). SSBM Finance Inc is a Delaware corporation.

Introduction

Back in 2005 when I have started my career as a server administrator in a startup, I had just 2 servers to manage. That was an easy job with just few websites running on both the servers. However, in next few years I had more than 10 servers running with different application and services. So I had to check the logs of every server and it was like spending more than a half day every day. Slowly the number of servers increased so I have configured few scripts to send me some important log information of each server, but again in just next 1 year I had 50+ servers to manage and it was crazy checking logs and I was worried, as there were all kinds of attacks happening on the servers. It was a huge task to read logs and troubleshooting the issues for each server, all I wanted was to have a centralized log server. I googled and found Logstash, as I was learning more about Logstash I came across with Elasticsearch & Kibana as well and it was a wow moment for me.

I have configured the ELK setup and started working on it. The whole experience was amazing, I was able to configure all the service logs and application log to a centralized server and also was able to define the parameters I wanted. The setup helped me to quickly search through the logs and find out the issues. Using the plugins, I have configured alerts as well.

There is a good community support and the product is keep evolving even to this date. The book Beginning Elastic Stack covers everything to configure a centralized log server quickly and effectively. In the book I have also covered Elastic Stack setup with Puppet and Foreman, which will help the server administrators not to just having ELK Stack configured quickly and easily but also having a system managing servers using Puppet.

Acknowledgments

I would like to thank a few special people:

I would first like to thank my Mom, Mrs. Rama Pachauri, without her continuous support and love I never would have been able to finish the book.
A special thanks to my Dad, Mr. V.C Pachauri for letting me do whatever I want to, and for providing me the much needed support always.
I can’t thank enough to my wife Shweta for giving me the much needed confidence and the courage to complete the book. There was a rough time when I was caught between my business and completing the book and Shweta was always there with suggestions and support.
I’d like to thank my friends Gaurav Mahajan, Hemant Gaba & Yakesh Arora for being pillars of my life over the years.
A big thank you to Panos for helping me out with the Technical Review and suggestions which helped me to include more technical things.
I want to thank Thomas d’Otreppe (Author Aircrack-ng) for doing Technical Review of my book. Man you are my hero, thank you so much for your help.
Thanks to Louise from Apress for giving me an opportunity and for helping me out with everything.
Nancy from Apress thanks for the push I needed.

Contents at a Glance

About the Author
About the Technical Reviewer
Acknowledgments
Introduction
Chapter 1: Getting Started with Logstash
Chapter 2: Getting Started with Elasticsearch
Chapter 3: Getting Started with Kibana
Chapter 4: Working with Remote Servers
Chapter 5: Configuring Logstash for Services and System Logs
Chapter 6: Graphite Monitoring and Graphs
Chapter 7: Configuring Elasticsearch Watcher
Chapter 8: Securing the ELK Stack with Shield
Chapter 9: Logstash Plug-ins
Chapter 10: Managing the ELK Stack with Puppet and Foreman
Index

About the Author and About the Technical Reviewer

About the Author

Vishal Sharma is a developer and entrepreneur with more than ten years’ experience working with various GNU/Linux server distributions and open source tools. As well as Logstash, he enjoys exploring server and web application security, to stay ahead of hackers and spammers and protect clients’ data.

About the Technical Reviewer

Panayiotis Gotysis has been working with systems and system administration since the moment he understood the power and magic of the CLI. For the past 12 years, he has specialized in architecting systems for redundancy, high availability and security, with an emphasis on virtualization and storage technologies.

In the last 3 years, working for the Greek Research Network ( https://www.grnet.gr ), he has moved into the DevOps mindset, seeking for configuration management, automation, and orchestration. Puppet and the ELK stack form the staple of the Greek Research Network’s operations as their service portfolio expands and their cloud offering, based on Ganeti ( http://www.ganeti.org/ ) and Synnefo ( https://www.synnefo.org/ ) provides virtualized resources to the academic and research institutions in Greece.

His current interests lie in architecting disaster recovery solutions, working with Fabric ( http://www.fabfile.org/ ) and using these tools to perform capacity management.

When not on a keyboard, he likes to improve his amateur photography skills and roll d20s with his role playing group.

Panayiotis can be reached at [email protected] .

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.

Table of Contents for Frontmatter

Create new playlist

Sign In

Sign Up