Now, let us look at how we can load data in bulk into the Hadoop layer and see the support provided by Hadoop ecosystem in achieving this.
- Preface
- Introduction to Data
- Comprehensive Concepts of a Data Lake
- What is a Data Lake?
- How does a Data Lake help enterprises?
- How Data Lake works?
- Differences between Data Lake and Data Warehouse
- Approaches to building a Data Lake
- Lambda Architecture-driven Data Lake
- Data ingestion layer - ingest for processing and storage
- Batch layer - batch processing of ingested data
- Speed layer - near real time data processing
- Data storage layer - store all data
- Serving layer - data delivery and exports
- Data acquisition layer - get data from source systems
- Messaging Layer - guaranteed data delivery
- Exploring the Data Ingestion Layer
- Exploring the Lambda layer
- Summary
- Lambda Architecture as a Pattern for Data Lake
- What is Lambda Architecture?
- History of Lambda Architecture
- Principles of Lambda Architecture
- Components of a Lambda Architecture
- Complete working of a Lambda Architecture
- Advantages of Lambda Architecture
- Disadvantages of Lambda Architectures
- Technology overview for Lambda Architecture
- Applied lambda
- Working examples of Lambda Architecture
- Kappa architecture
- Summary
- Applied Lambda for Data Lake
- Knowing Hadoop distributions
- Selection factors for a big data stack for enterprises
- Batch layer for data processing
- Serving layer
- Summary
- Data Acquisition of Batch Data using Apache Sqoop
- Context in data lake - data acquisition
- Why Apache Sqoop
- Workings of Sqoop
- Sqoop connectors
- Sqoop support for HDFS
- Sqoop working example
- When to use Sqoop
- When not to use Sqoop
- Real-time Sqooping: a possibility?
- Other options
- Summary
- Data Acquisition of Stream Data using Apache Flume
- Context in Data Lake: data acquisition
- Why Flume?
- Flume architecture principles
- The Flume Architecture
- Flume event - Stream Data
- Flume agent
- Flume source
- Flume Channel
- Flume sink
- Flume configuration
- Flume transaction management
- Other flume components
- Context Routing
- Flume working example
- When to use Flume
- When not to use Flume
- Other options
- Summary
- Messaging Layer using Apache Kafka
- Context in Data Lake- messaging layer
- Why Apache Kafka
- Kafka architecture
- Other Kafka components
- Kafka programming interface
- Producer and consumer reliability
- Kafka security
- Kafka as message-oriented middleware
- Scale-out architecture with Kafka
- Kafka connect
- Kafka working example
- When to use Kafka
- When not to use Kafka
- Other options
- Summary
- Data Processing using Apache Flink
- Context in a Data Lake - Data Ingestion Layer
- Why Apache Flink?
- Working of Flink
- Flink API's
- Flink working example
- When to use Flink
- When not to use Flink
- Other options
- Summary
- Data Store Using Apache Hadoop
- Indexed Data Store using Elasticsearch
- Context in Data Lake: data storage and lambda speed layer
- What is Elasticsearch?
- Why Elasticsearch
- Working of Elasticsearch
- Elastic Stack
- Elastic Cloud
- Elasticsearch DSL (Query DSL)
- Nodes in Elasticsearch
- Elasticsearch and relational database
- Elasticsearch ecosystem
- Elasticsearch deployment options
- Clients for Elasticsearch
- Elasticsearch for fast streaming layer
- Elasticsearch as a data source
- Elasticsearch for content indexing
- Elasticsearch and Hadoop
- Elasticsearch working example
- Installation
- Creating and Deleting Indexes
- Indexing Documents
- Getting Indexed Document
- Searching Documents
- Updating Documents
- Deleting a document
- Elasticsearch in purview of SCV use case
- Data preparation
- Customer data import into Hive using Sqoop
- Data acquisition via Flume into Kafka channel
- Data ingestion via Flink to HDFS and Elasticsearch
- Writing into Elasticsearch
- Flink deployment
- Parquet data visualization as Hive tables
- Data indexing from Hive
- Query data from ES (customer, address, and contacts)
- When to use Elasticsearch
- When not to use Elasticsearch
- Other options
- Summary
- Data Lake Components Working Together
- Where we stand with Data Lake
- Core architecture principles of Data Lake
- Challenges faced by enterprise Data Lake
- Expectations from Data Lake
- Data Lake for other activities
- Knowing more about data storage
- Knowing more about Data processing
- Thoughts on data security
- Thoughts on data encryption
- Metadata management and governance
- Thoughts on Data Auditing
- Thoughts on data traceability
- Knowing more about Serving Layer
- Summary
- Data Lake Use Case Suggestions
- Establishing cybersecurity practices in an enterprise
- Know the customers dealing with your enterprise
- Bring efficiency in warehouse management
- Developing a brand and marketing of the enterprise
- Achieve a higher degree of personalization with customers
- Bringing IoT data analysis at your fingertips
- More practical and useful data archival
- Compliment the existing data warehouse infrastructure
- Achieving telecom security and regulatory compliance
- Summary
Example - Bulk Data Load
-
No Comment
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.