For the bulk/batch transfer of data from RDBMS to the Hadoop filesystem there aren't many options in the open source world. However, there are possible choices whereby we could transfer data from RDBMS to Hadoop, and this section tries to give you the reader some possible options so that, according to enterprise demands, they can be evaluated and brought into the data lake as technologies if found suitable.
- Preface
- Introduction to Data
- Comprehensive Concepts of a Data Lake
- What is a Data Lake?
- How does a Data Lake help enterprises?
- How Data Lake works?
- Differences between Data Lake and Data Warehouse
- Approaches to building a Data Lake
- Lambda Architecture-driven Data Lake
- Data ingestion layer - ingest for processing and storage
- Batch layer - batch processing of ingested data
- Speed layer - near real time data processing
- Data storage layer - store all data
- Serving layer - data delivery and exports
- Data acquisition layer - get data from source systems
- Messaging Layer - guaranteed data delivery
- Exploring the Data Ingestion Layer
- Exploring the Lambda layer
- Summary
- Lambda Architecture as a Pattern for Data Lake
- What is Lambda Architecture?
- History of Lambda Architecture
- Principles of Lambda Architecture
- Components of a Lambda Architecture
- Complete working of a Lambda Architecture
- Advantages of Lambda Architecture
- Disadvantages of Lambda Architectures
- Technology overview for Lambda Architecture
- Applied lambda
- Working examples of Lambda Architecture
- Kappa architecture
- Summary
- Applied Lambda for Data Lake
- Knowing Hadoop distributions
- Selection factors for a big data stack for enterprises
- Batch layer for data processing
- Serving layer
- Summary
- Data Acquisition of Batch Data using Apache Sqoop
- Context in data lake - data acquisition
- Why Apache Sqoop
- Workings of Sqoop
- Sqoop connectors
- Sqoop support for HDFS
- Sqoop working example
- When to use Sqoop
- When not to use Sqoop
- Real-time Sqooping: a possibility?
- Other options
- Summary
- Data Acquisition of Stream Data using Apache Flume
- Context in Data Lake: data acquisition
- Why Flume?
- Flume architecture principles
- The Flume Architecture
- Flume event - Stream Data
- Flume agent
- Flume source
- Flume Channel
- Flume sink
- Flume configuration
- Flume transaction management
- Other flume components
- Context Routing
- Flume working example
- When to use Flume
- When not to use Flume
- Other options
- Summary
- Messaging Layer using Apache Kafka
- Context in Data Lake- messaging layer
- Why Apache Kafka
- Kafka architecture
- Other Kafka components
- Kafka programming interface
- Producer and consumer reliability
- Kafka security
- Kafka as message-oriented middleware
- Scale-out architecture with Kafka
- Kafka connect
- Kafka working example
- When to use Kafka
- When not to use Kafka
- Other options
- Summary
- Data Processing using Apache Flink
- Context in a Data Lake - Data Ingestion Layer
- Why Apache Flink?
- Working of Flink
- Flink API's
- Flink working example
- When to use Flink
- When not to use Flink
- Other options
- Summary
- Data Store Using Apache Hadoop
- Indexed Data Store using Elasticsearch
- Context in Data Lake: data storage and lambda speed layer
- What is Elasticsearch?
- Why Elasticsearch
- Working of Elasticsearch
- Elastic Stack
- Elastic Cloud
- Elasticsearch DSL (Query DSL)
- Nodes in Elasticsearch
- Elasticsearch and relational database
- Elasticsearch ecosystem
- Elasticsearch deployment options
- Clients for Elasticsearch
- Elasticsearch for fast streaming layer
- Elasticsearch as a data source
- Elasticsearch for content indexing
- Elasticsearch and Hadoop
- Elasticsearch working example
- Installation
- Creating and Deleting Indexes
- Indexing Documents
- Getting Indexed Document
- Searching Documents
- Updating Documents
- Deleting a document
- Elasticsearch in purview of SCV use case
- Data preparation
- Customer data import into Hive using Sqoop
- Data acquisition via Flume into Kafka channel
- Data ingestion via Flink to HDFS and Elasticsearch
- Writing into Elasticsearch
- Flink deployment
- Parquet data visualization as Hive tables
- Data indexing from Hive
- Query data from ES (customer, address, and contacts)
- When to use Elasticsearch
- When not to use Elasticsearch
- Other options
- Summary
- Data Lake Components Working Together
- Where we stand with Data Lake
- Core architecture principles of Data Lake
- Challenges faced by enterprise Data Lake
- Expectations from Data Lake
- Data Lake for other activities
- Knowing more about data storage
- Knowing more about Data processing
- Thoughts on data security
- Thoughts on data encryption
- Metadata management and governance
- Thoughts on Data Auditing
- Thoughts on data traceability
- Knowing more about Serving Layer
- Summary
- Data Lake Use Case Suggestions
- Establishing cybersecurity practices in an enterprise
- Know the customers dealing with your enterprise
- Bring efficiency in warehouse management
- Developing a brand and marketing of the enterprise
- Achieve a higher degree of personalization with customers
- Bringing IoT data analysis at your fingertips
- More practical and useful data archival
- Compliment the existing data warehouse infrastructure
- Achieving telecom security and regulatory compliance
- Summary
Other options
-
No Comment
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.