146 | Big Data Simplied
• Square: Kafka as bus to move all system events to various Square data centres (logs, custom
events, metrics and so on). Outputs to Splunk, Graphite and Esper-like alerting systems.
• Spotify, Uber, Tumbler, Goldman Sachs, PayPal, Box, Cisco, CloudFflare, Datadog,
LucidWorks, Mailchimp, Netflix, etc.
6.2.4 Kafka Use Cases
Kafka gets actually used for data stream processing, website activity tracking, metrics collec-
tion and monitoring, log aggregation, real-time analytics, ingesting data into Spark, ingesting
data into Hadoop, replay messages, error recovery, and guaranteed distributed commit log for
in-memory computing (microservices).
• Message: Kafka has replaced the traditional message brokers by providing advantages like no
other. It has now become one of the most popular messaging systems for large-scale message
processing applications.
• Metrics: Kafka finds good application for operational monitoring data. To produce central-
ized feeds of operational data and statistics are aggregated from distributed applications.
• Event Sourcing: In this style of application, the changes in design state are logged as a
time-ordered sequence of records. Kafka is an excellent backend for applications of event
sourcing as it supports very large stored log data.
• Commit Log: For a distributed system, Kafka can act as a sort of external commit-log.
Thereplication of data between nodes and re-syncing the failed nodes to restore their data is
the function of this log.
• Tracking of Website Activity: Initially, the use case for Kafka was intended on building
a tracking pipeline as a set of real time publish subscribe feeds. Huge number of activity
messages are generated for each user page view as activity tracking is of very high volume.
Nowadays, the most important thing is to capture the user (buying or searching a product)
behaviour. Here, Kafka is playing a great role to compute those complex analytics and it helps
to increase the productivity of an e-commerce organization (Ref. Figure 6.13).
FIGURE 6.13 KAFKA use case
Apache KAFKA
Website user logs
from web server
Custom apps user
logs
Microservices
Twitter/Social media
NoSQL database
Oracle RDBMS/
datawarehouse
Monitoring
User analytics
M06 Big Data Simplified XXXX 01.indd 146 5/17/2019 2:49:17 PM
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.