Real-time mode

Real-time data comes from the data source(s) like IoT sensors, streaming applications like clickstream, social media, etc. The typical real-time analytics scenarios built on this data are social media analytics, vehicle maintenance alerts, generating SOS alerts for passengers, or generating alarms depending on the temperature in scenarios like mining and oil exploration, etc. Generally, in these scenarios data is not only processed in real time, but it’s also stored in the data storage layer to perform batch processing. This helps to extract information like trends and volumetric data, and generate feedback to improve the system and improve ML models’ accuracy, etc.

For real-time data ingestion, since the velocity of incoming data will be high, the solution should have a queuing mechanism to avoid losing any of the event data. Moreover, it should have the capability of connecting to millions of devices at the same time, scale infinitely to handle the load, and process the data in real-time. These factors can enable the solution to have the capabilities to perform real-time operations (e.g, in oil exploration or mining scenarios, if the alert of a specific parameter like rise in temperature or some leakage is reported in real time, it can help to avert accidents). These are scenarios where any delay is not affordable; otherwise the solution will lose its purpose.

Batch mode analysis acts as a feedback system, and it can also help to integrate this data with the data coming from other LOB applications to build 360-degree scenarios. In this section, the discussion is on the technologies shown in Figure 4-3.

There are various advantages to using PaaS services on Azure. Their setup/installation, high availability, monitoring, version upgrades, security patches, and scalability are managed by Microsoft. Every service has multiple tiers and SLAs. An appropriate tier can be chosen, depending on the performance and availability requirement.

Before getting into the details of these services, let’s briefly discuss minute yet important concepts of events and messages. These are commonly used terms and their meaning and applicability is summarized in Table 4-1.

Events – Events are lightweight notifications of a condition or a state change. The publisher of these events has no expectation about how the event should be handled. However, the consumer of the event decides how this even should be consumed. Events are of two types: discrete and continuous series of values.

Discrete events – These events report state change. The consumer of the event only needs to know that something happened. For example, an event notifies consumers that an order was placed. It may have general information about the order, but it may not have details of the order. The consumer may just need to send an email to the customer with confirmation of the order placed. Discrete events are ideal for serverless solutions that need to scale.

Series events – These events report conditions that are time-ordered and interrelated. The consumers of these events need the sequenced series of events to analyze what happened. This generally refers to scenarios of data coming from telemetry from IoT devices or distributed streams of data from social media applications.

In this chapter, the focus is on data coming from telemetry or distributed data streams, and that falls under the series events category. Now, let’s discuss real-time data ingestion services in the next section.

Apache Kafka on HDInsight Cluster

Apache Kafka is one of the most popular solutions for ingesting streaming data. However, not only ingesting streaming data but integrating multiple disparate applications within the organization can also be done through Apache Kafka. In short, anything to do with events and messages can be managed with Apache Kafka—for example, collecting application logs, activity tracking, and gathering data from websites or data from millions of IoT devices.

To cater to the requirement for processing real-time data, this solution is highly efficient, and it can process millions of messages per second. In fact, this platform was built by LinkedIn for purposes like improving user experience or analytics like page views, which content was popular, or recommendation engines. This platform was later donated to the open source community. There are many large companies like Netflix, Twitter, Airbnb, Uber, etc. that use this solution.

Producer – The producer is the source that produces the message. The producer can be a website generating streams of clickstream data or a fleet truck sharing its location or any IoT sensors producing data streams. The producer just connects to a broker and sends the event without worrying about how it will be processed.

Broker – The broker is the server that is part of the Kafka cluster. In other words, a collection of brokers forms a Kafka cluster. Its primary role is to pick up the event from the producer and deliver it to the consumers. Kafka brokers work in leader and worker topology, and there is at least one leader and two in-sync replicas. When an Apache Kafka on HDInsight cluster is built, there are by default two master nodes where the Hadoop ecosystem runs and three mandatory worker nodes (broker nodes) that perform the task.

Topic – The topic, like a table in RDBMS, is where the stream of data is written inside apache Kafka brokers. Topics are further divided into partitions, and each message in the partition will be assigned an offset. Offset in a partition becomes the pointer for a consumer to read and process the data.

ZooKeeper – ZooKeeper is another component in Apache Kafka, which manages all the brokers starting from replication, availability, and fault tolerance, etc. ZooKeeper has separate servers to manage the Apache Kafka ecosystem; it also needs an odd number of servers, with a minimum of three. Kafka can’t run without ZooKeeper. ZooKeeper even works in leader and follower mode, where at least one node will act as a leader and at least two will be followers.

Consumer – The consumer is the destination application that processes the event data. In the case of real-time analytics, the same event can be consumed by multiple consumers. In the case of Lambda architecture, when both real-time and batch mode processing is needed, the same event is both passed to stream processing services like Azure Stream analytics, Apache Storm and Spark streaming, etc. and written to Azure Data Lake or Blob storage at the same time for processing later.

Now that we have built some background on Apache Kafka, let’s discuss a little bit about how to set up an Apache Kafka on HDInsight cluster. Since detailed documentation is already available on the Microsoft website, we will discuss it on a very basic level.

The cost of the infrastructure is calculated on a per hour basis. With Azure, an Apache Kafka cluster can be up and running within minutes, whereas these steps generally take weeks in an on-premises setup.

Generally, these inputs must be available before the sizing of the cluster. In the case of inputs not being available, Microsoft Azure provides flexibility to scale on demand—it provides flexibility to the architects to start small in the beginning and gradually scale, as needed.

All the preceding information is important to decide which VM will meet the requirement and how many VMs are required to withstand the incoming data streams.

Step 1 - Total throughput needed would be message rate × message size × replica count.

That is 1,000 × 10 × 3 = 30,000 = 30 MB per second approx.

Step 2 – Total storage required for this solution is total message size × retention time.

That is 30 MB/s × 6 hours (6 × 60 × 60) = 30 × 21,600 = 6.4 TB approx.

Ideal sizing of D3 V2 VM can attach seven 1 TB disks, and a total of 1 VMs will be needed.

After all this exercise and creation of Apache Kafka on HDInsight cluster, it’s time for some action. In this chapter, the discussion is only around setting up producers and ensuring the data streams are landing in Apache Kafka. The coming chapters will talk about various options to land this data either in Spark or a storage layer, and how to apply ML for sentiment analytics.

Since an Apache Kafka cluster has been created already, let’s configure the producer to pick up live stream data as follows.

Exercise: Create an Apache Kafka Cluster in Azure HDInsight for Data Ingestion and Analysis

Since this chapter is about data ingestion, all the exercises are discussed till data ingestion. However, all the remaining phases like event processing and dashboards are discussed in the coming chapters.

Now, let’s learn how to create topics and produce events. Since the Apache Kafka cluster has been created in the previous section already, let’s log into the cluster to complete the exercise.

After the preceding configurations are set up, Kafka topics can be created and the events can be produced/consumed.

Kafka comes with an inbuilt utility script named kafka-topics.sh, which can be used to create/delete/list topics. It also provides utility scripts like kafka-console-producer.sh, which can be used to write records to the topics and kafka-console-consumer.sh, which can be used to consume those records from the topic.

Let’s run these scripts to manage the Kafka topics:

1. 1.
   To create a topic test, run the following command:

   /usr/hdp/current/kafka-broker/bin/kafka-topics.sh ​--create --replication-factor 3 --partitions 8 --topic test --zookeeper $KAFKAZKHOSTS

   A Kafka topic with replication factor 3 and 8 partitions is created. The worker nodes created are 4; therefore, we can replicate the topics to less than or equal to 4. There are 8 partitions created, which means that the consumer group should have a maximum 8 consumers.

    
2. 2.
   To list the topic as shown in Figure 4-7:

   /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --list ​--zookeeper $KAFKAZKHOSTS
    

This was a basic exercise to give a hands-on experience on Apache Kafka cluster. Now, let’s discuss the next option—Event Hub—as follows.

Event Hub

Event Hub is a native Azure service for ingesting real-time data streams. This is a turnkey solution based on a serverless offering on Microsoft Azure. There is no need to manage any infrastructure; throughput units can be selected based on the estimated incoming messages. The entire underlying infrastructure is managed by Microsoft. It means there is no need to upgrade or patch, or back up the event service. It reduces maintenance overhead from the infrastructure teams. Refer to Figure 4-9; Event Hub consists of the following components to make it run:

1. 1.

   Throughput unit

    
2. 2.

   Cluster unit

    
3. 3.

   Namespace

    
4. 4.

   Partitions

    
5. 5.

   Event producers

    
6. 6.

   Event receivers

    

Cluster Unit (CU) – If the requirement is larger than 20 TU—that is, 20 MBs per second or 20,000 messages per second—the option is to opt for an event hub with dedicated capacity. For dedicated capacity, the measure of throughout is cluster units, and 1 CU = 100 TUs.

Namespace – Namespace is a logical collection of event hubs. Throughout units or cluster units are mentioned when the namespace is created. Throughput units/cluster units allocated at the time of namespace creation are shared among all the event hubs.

Partitions – Partition is the place where the streaming messages are written by the event producers and are read by the consumers. While creating event hubs, the number of partitions is mentioned. The general recommendation is to create a number of partitions equal to throughput units. The number of partitions should be decided based on the downstream applications.

Event producers – Event producers are the entities that produce and send events to event hubs. Some examples of event producers are social media platforms, clickstreams, or application event logs. Event producers write the event to partitions, which are further processed as needed. Event producers can use standard AMQP or HTTP(S) protocols to send data to Event Hub.

Event receivers – Event receivers are the entities that consume the events from Event Hub partitions. These are generally applications that process the event data. This can be Apache Spark steaming clusters, Storm, or stream analytics service or even Blob storage. Consumer groups are created for downstream applications. As shown in Figure 4-9, based on the need, consumers can either be part of existing consumer groups or create a new consumer group.

Why Event Hubs?

Moreover, 1 CU = 100 TU and 1 TU is:

Ingress: Up to 1 MB per second or 1,000 events per second (whichever comes first).

Egress: Up to 2 MB per second or 4096 events per second.

This means with dedicated hosts, millions of messages per seconds can be easily processed without any maintenance overheads.

Moreover, for parallel real-time, and batch processing, features like event hub capture are helpful. With event hub capture, incoming streaming data can be sent directly to Azure Blob or Azure Data Lake Storage. This feature is like Apache Kafka Connect or Amazon Firehose. For a hands-on experience on Azure Event Hub, let’s go through the following exercise.

Exercise: Ingesting Real-Time Twitter Data Using Event Hub

Today, organizations spend lots of time and money on performing social media analytics to build marketing strategy or improve their service offerings and products. Sentiment analysis is performed on these high-volume Twitter streams of the trending topics. This helps to understand and determine public responses toward the latest products and ideas.

In this section, let’s discuss how to build a real-time analytics solution that can take continuous streaming data from Twitter into an event hub.

This entire exercise can also be referred to on this link:

https://docs.microsoft.com/en-us/azure/stream-analytics/stream-analytics-twitter-sentiment-analysis-trends.

After the event hub is created, a Twitter client application needs to be created and configured.

It shows the number of incoming requests, messages, and throughput of the event hub. As discussed earlier, event hubs can fetch data from social media, clickstream, or other custom data streaming applications. However, there is another large segment, which generates stream data through IoT devices. Let’s discuss IoT Hub, a serverless service available on Azure to connect with these IoT devices and fetch real-time IoT streams.

IoT Hub – IoT Hub is a service available on Microsoft Azure for processing data coming particularly from IoT devices. IoT devices are widely used in manufacturing, smart cities, connected cars, or anywhere where real-time data is needed from the hardware devices. The real purpose of IoT is to enhance user experience, detect anomalies, or improve the efficiency of operations without human intervention.

Let’s take the example of connected cars: companies use IoT devices to understand the efficiency of the machinery working inside the car and even enhance the user experience. Today, insurance companies use IoT devices to build the driver profile based on their driving behavior and charge them for the insurance accordingly. Even the steering wheel of the car can have sensors like a heartbeat monitor, and this data can be sent to the cloud to offer health alerts, etc.

One caveat with IoT devices is the need to be connected to the Internet to constantly send and receive data from the target cloud server. Consider places like mines, manufacturing units, or where the real-time response is needed and Internet connectivity is low but real-time response is required. Therefore, IoT devices should have the capability to process data on the devices themselves and generate alerts in real-time. Edge devices are IoT devices with a capability to process data locally to a certain extent, generate real-time alerts, and send the output to the cloud whenever there is connectivity.

To connect to IoT devices, Microsoft offers Azure IoT Hub, which are like event hubs but with added features. Both IoT hubs and event hubs are for ingesting real-time data. However, IoT hubs are designed specifically for ingesting data from IoT devices, and event hubs are designed for ingesting telemetry and event data streams. IoT hubs support bidirectional communication between device and the cloud, which helps to do much more with the devices. However, event hubs are unidirectional and just receive the data for further processing. One of the biggest challenges with IoT devices today is the security of data. IoT Hub provides various capabilities like secure communication protocols; integration with security tools on Azure detect and react to the security breaches on IoT devices.

The following chart summarizes the difference between event hubs and IoT hubs tiers.

To find an updated version of this chart, please check the weblink https://docs.microsoft.com/en-us/azure/iot-hub/iot-hub-compare-event-hubs.

IoT Devices – IoT devices must be created to connect to the actual IoT devices. This helps to create two-way communication between IoT Hub and IoT devices.

Partitions – The concept of partitions between Event Hub and IoT Hub is the same: stream data is written to partitions. The maximum number of partitions in the IoT hubs can be 32, and the number varies based on the tier. The number of partitions can’t be changed after the IoT hub is created.

Register IoT devices

The next level is to configure the IoT devices with the IoT Hub.

In order to simulate an IoT device, C#-based simulator code has been created, which is available at:

https://github.com/Apress/data-lakes-analytics-on-ms-azure/tree/master/AdvanceAnalyticsOnDataLake/Simulator/AdvanceAnalyticsOnDataLake.

Once the devices are configured and the input data ingestion has started, this data can be queried. IoT Hub provides an SQL-like language to fetch information like device twins, message routing, jobs, and module twines.

These queries can be executed in Query explorer, as shown in Figure 4-14.

With this exercise, real-time data stream ingestion comes to an end. In the next section, the discussion is on batch data ingestion.

Batch Data Ingestion

IoT hubs and event hubs deal with real-time event streams; however, batch mode data processing is done on periodic intervals. Data coming from various data sources—real-time or batch mode—is stored in a storage solution like Azure Data Lake Storage, Blob storage, or even relational/nonrelations databases. A typical architecture pattern for solutions today is as shown in Figure 4-15 (also refer to Figure 4-1).

To detect patterns or build 360-degree analytics solutions, batch mode processing plays a key role. Real-time processing is required for solutions where mission critical or time-bound response is needed. Examples like fraud detection or predictive maintenance are scenarios where if real-time response is not given, opportunity to avoid an event is lost and can cause financial loss to the business. In fact, batch mode processing acts as a feedback loop that helps to enhance the response for real-time processing; for example, detecting patterns of fraudulent transactions can happen only with batch mode processing, which improves the accuracy of the machine learning model. When the newly trained ML models are used in real time, they can detect frauds/events more effectively.

Azure Data Factory

Azure Data Factory (ADF) is a Microsoft native cloud solution built for ETL/ELT scenariosc. It’s a fully managed PaaS offering from Azure and is a serverless solution. It means data engineers can build data ingestion solution without worrying about the infrastructure requirements. It provides more than 90 data connectors for connecting to multiple on-premises and cloud data sources. Moreover, it provides a code-free user interface where developers can drag and drop activities to create and monitor data pipeline. Data pipeline is another widely used jargon in data analytics solutions.

Data pipeline is the automation of a series of steps that data transverses to yield required output. As we know, the data analytics solution has four major stages: Data Ingestion, Data Storage, Prep and Train, Model and Serve. Solutions like ADF provide the platform to build data pipelines and infuse data into these stages in an orderly fashion.

SQL Server Integration Services and Informatica are the other solutions that have become famous for on-premises environments. These solutions are now cloud ready. Azure Data Factory provides seamless execution of SSIS packages using ADF. Moreover, Informatica is an Azure-certified solution available on Azure Marketplace.

Azure Data Factory provides the following features:

Connect and collect – In any data processing tool, the first step is to connect and fetch data from the data sources. These data sources can be databases, file systems, FTP/SFTP web services, or SaaS based systems. It’s important that the tool understands the underneath protocol, version, and configurations required to connect to each source. Once the sources are connected, the next step is to fetch data and put that in the centralized location for further processing.

Usually enterprises build or buy data management tools or create custom services that can integrate with these data sources and perform processing. This can be an expensive and time-consuming activity. With the frequent version changes in the source data engines, it’s hard to keep pace with regular version upgrades and compatibility checks. Moreover, these custom applications and services often lack features like alerting, monitoring, and other controls that a fully managed PaaS service provides.

All the aforementioned drawbacks are well taken care of in ADF; it provides templates like Copy Activity through which one can easily connect to some 90+ data sources, fetch data, and store in a centralized data store or data lake. For example, data can be collected from an on-premises SQL Server, stored in Azure Data Lake Storage, and further processed in Azure Databricks.

Transform and enrich – Azure Data Factory provides features like data flows through which collected data can be easily processed and transformed in the desirable format. Internally, data flows execute on Spark-based distributed in-memory processing, which makes them highly efficient. A developer who is developing data flows need not understand Spark programming for the processing.

Though in case developers prefer to write and execute custom code for transformations, ADF provides external activities that can then execute the custom code on Hadoop, Spark, machine learning, and Data Lake Analytics.

In case there are already predefined SSIS packages available, those can also be executed on the Azure platform with minimal configuration changes.

CI/CD and publish – Azure DevOps and GitHub can be easily integrated with Azure Data factory for a full continuous integration and development. It helps in incrementally developing, sharing, reviewing, and testing a process before pushing it to production. By maintaining a code repository, forking branches for multilevel development becomes quick and easy.

Monitor – Once the data pipeline building activity is completed, the developer can debug the flow for initial testing and monitoring, once confirmed data flows can be scheduled. For scheduling, ADF provides triggers that can be executed at a specified time or on occurrence of any external event. The external event could be addition or deletion of a file/folder in Azure Data Lake Storage. The developer can also execute pipelines in real time using the trigger row option.

Once executed, the pipelines can be monitored with the status of all the activities included in that pipeline. The performance details are also shared in the monitoring.

It’s important to understand the components of the Azure Data Factory toolbox. The following are major components that are used to build pipelines:

1. 1.

   Datasets

    
2. 2.

   Activities

    
3. 3.

   Linked service

    
4. 4.

   Integration runtime

    
5. 5.

   Pipeline

    
6. 6.

   Data flows

    

Datasets – A dataset is a named reference to the data, which will be used for input or output of an activity. There are more than 90 connectors supported by ADF to create the dataset. The data can be brought from open source DBs, SaaS applications, or any public cloud storage, etc.

Activities – Actions to be performed on the data are called activities. Activities take inputs, transform input data, and produce outputs. The following activities are available on ADF:

1. 1.

   Move and Transform

    
2. 2.

   Azure Data Explorer

    
3. 3.

   Azure Function

    
4. 4.

   Batch Service

    
5. 5.

   Databricks

    
6. 6.

   Data Lake Analytics

    
7. 7.

   General

    
8. 8.

   HDInsight

    
9. 9.

   Iteration and Conditionals

    
10. 10.

    Machine Learning

     

Linked service – Linked service is like a connection string to the data set. There are two types of linked services

1. 1.

   Data Linked Services: It’s the connection to various data sources that ADF can connect with. Currently ADF can connect with more than 90 data sources.

    
2. 2.

   Compute Linked Services: It’s the connection to various compute resources where the activities mentioned in the above section.

    

Integration runtime – Integration runtime is the infrastructure used by ADF to run the operations like data movement, activity dispatch, data flows, and SSIS package execution. It acts as a bridge between activity and linked service.

Pipeline – It’s the logical group of activities to be performed on the data. It’s a workflow that comes into existence by infusing datasets, linked services, and activities.

The relationship between these components can be seen in Figure 4-16.

Data Flows - Data flow is a way to build data pipelines without coding. It’s a GUI-based method to create linked services, data sets, and activities and build a complete pipeline. Data flows can be used to create pipelines that are less complex and smaller in size.

Exercise: Incrementally Load Data from On-premises SQL Server to Azure Blob Storage

In this exercise, we will load the historic, new, and changed data from a table in an SQL Server on-premises instance to Blob storage in an Azure cloud.

A high-level solution diagram looks as shown in Figure 4-17.

Go to Connections ➤ Linked Service ➤ New ➤ Select SQL Server ➤ click continue ➤ provide the required details like name, choose integration runtime, provide server name, database name, username, and password ➤ click create.

When this pipeline is successfully completed, the data from on-premises SQL server will be inserted into Blob storage.

SQL Server Integration Services – SQL Server Integration Services is an ETL solution and comes as a part of the SQL Server installation package. It’s been famous for building ETL of structured data. There are many organizations that have invested heavily in this technology; when they move to cloud, they can move to Azure Data Factory or may prefer to use SSIS packages only.

The first option is straightforward— it just needs installation of the SSIS component on Azure VM and connection to the SQL database for the SSIS catalog. However, to run SSIS packages using Azure Data Factory, SSIS integration runtime needs to be configured.

Integration runtimes provide the compute infrastructure used by Azure Data Factory for data flow, data movement, activity dispatch, and SSIS package execution.

Azure-SSIS IR – It’s the compute instance that can an SSIS package migrated from on-prem or VM installation on a public cloud. It’s a fully managed compute that can be scaled up or out depending on the requirement. The SSIS package can be run on this compute with minimal to no change.