Oftentimes, you will need to search a large volume of data to solve issues quickly or get business insights. The ability to search your application data will help you to access detailed information and analyze it from different views. To search for data with low latency and high throughput, you need to have search engines as your technology choice.
Elasticsearch is one of the most popular search engine platforms and is built on top of the Apache Lucene library. Apache Lucene is a free and open source software library that is the foundation of many popular search engines. The ELK (short for Elasticsearch, LogStash, and Kibaba) stack is easy to use for the automatic discovery of large-scale data and to index it for searching. Because of its property, there are multiple tools that have been developed around Elasticsearch for visualization and analysis—for example, LogStash works with Elasticsearch to collect, transform, and analyze a large amount of an application’s log data. Kibana has an in-built connector with Elasticsearch that provides a simple solution for creating dashboards and analysis on indexed data.
Elasticsearch can be deployed in virtual machines and scale horizontally to increase capacity by adding new nodes to the cluster. The public cloud AWS provides the managed service Amazon Elasticsearch Service (Amazon ES), which makes it cost effective and simple to scale and manage the Elasticsearch cluster in the cloud.
In this section, you learned about the various database technologies and what they are used for. Your applications can use a combination of all database technologies in order for their different components to achieve optimal performance. For complex transactions, you need to use a relation OLTP database, and to store and process unstructured or semistructured data, you need to use a nonrelational NoSQL database. You should use the NoSQL database where very low latency is required over multiple geographical regions and where you need to handle complex queries at the application layer, such as in a gaming application. If you need to perform any large-scale analytics on structured data, use a data warehouse OLAP database.
Let's look at another critical component of your architecture, which is networking. Networking is the backbone of the entire application and establishes communication between the servers and the outside world. Let's learn about networking as regards application performance.