Cover image for Data Management at Scale

Data Management at Scale

Author Piethein Strengholt
Release Date: 2020/11/01
ISBN: 9781492054771
Topic:
0%
8
Chapters
0-1
Hours read
0k
Total Words
    Start Reading Now    
       Add to Wishlist       
View table of contents

Book Description

The amount of data generated is growing tremendously in size and complexity. As trends in data management and integration, such as cloud, API management, microservices, open data, software as a service (SaaS), and new software delivery models, continue to evolve rapidly, data warehouses and data lakes are no longer scalable.

With this practical book, you’ll learn how to migrate your enterprise from a complex and tightly coupled data landscape to a new data management architecture that’s more flexible, distributed, and scalable. Ready for the modern world of data consumption, this architecture can be introduced incrementally without a large up-front investment. Author Piethein Strengholt provides blueprints, principles, observations, best practices, and patterns to get you up to speed.

In three parts, this book helps you:

  • Examine data management trends and difficulties, including technological developments and regulatory and privacy requirements that puzzle enterprises
  • Go deep into this innovative new architecture and learn how the pieces fit together
  • Explore data governance and security, business intelligence, and analytics
  • Understand data management, self-service data marketplaces, and the importance of metadata
    •

    Table of Contents

    1. 1. Introducing the DIAL Architecture: Organizing Data at Scale
      1. Lessons learned
        1. Applications are specific and have unique context
        2. There’s no escape from the data integration dilemma
        3. Applications play the roles of data providers and data consumers
      2. Key theoretical considerations
        1. Object-oriented programming principles
        2. Domain-Driven Design
        3. Business Architecture
      3. Communication and integration patterns
        1. Point-to-point
        2. Silos
        3. Hub and Spoke model
      4. Digital Integration & Access Layer (DIAL)
        1. Golden Sources and Integrated Data Stores
        2. Data Delivery Contracts & Data Sharing Agreements
        3. Eliminating the siloed approach
        4. Domain-Driven Design on an Enterprise Scale
        5. Read-optimized data
        6. Integration layer as a holistic picture
        7. Metadata and the Target Operating Model
      5. Summary
    2. 2. Managing Vast Amounts of Data: The Read-Only Data Stores Layer
      1. Command and Query Responsibility Segregation (CQRS)
        1. What is CQRS?
        2. CQRS at scale
      2. Scalable, foundational, and domain-agnostic components
        1. RDS tiers
        2. Data Quality
        3. Metadata
        4. Design variations
        5. Data Replication
        6. Data Access layer
        7. File Manipulation Service
        8. Consumption Service
        9. Delivery Notification Service
        10. De-identification Service
        11. Historical data service
        12. Data ingestion
      3. Populating RDSs on demand
      4. RDS direct usage
      5. Summary
    3. 3. API Management: The API Layer
      1. What is Service Orientation Architecture?
        1. How did SOA emerge and how is it being used today?
        2. Enterprise Application Integration
        3. Service Orchestration
        4. Service Choreography
        5. Business Services and Internal Services
        6. Canonical data model
        7. Similarities SOA with the Enterprise Data Warehousing Architecture
      2. Modern view on SOA
        1. API Gateway
        2. Responsibility Model
        3. New role of the ESB
        4. Service Contracts
        5. Service discovery
      3. Microservices
        1. The Role of the API Gateway within Microservices
        2. Functions
        3. Service mesh
        4. Logical boundaries of the domain
        5. Microservices within the SOA reference architecture
      4. Ecosystem communication
      5. Channel communication
        1. GraphQL
      6. Metadata
      7. Combination with the RDS Layer
      8. Summary
    4. 4. Event and Respond Management: The Streaming Layer
      1. The asynchronous event model makes the difference
      2. What do event-driven architectures look like?
        1. Event producers
        2. Event consumers
        3. Event platform
      3. A gentle introduction to Apache Kafka
        1. Event Sourcing and Command Sourcing
        2. Distributed Event Data
        3. Summary
      4. The Streaming Layer within DIAL
        1. Governance Model
        2. Business events
        3. Streaming Consumption patterns
        4. Application state movement
        5. Playing the role of an RDS
        6. Using streaming to populate RDSs
        7. Control and policy guiding the teams in the usage
      5. Streaming as the operational backbone
      6. Guarantees and consistency
        1. Consistency level
        2. At Least Once, Exactly Once, and At Most Once processing
        3. Message order
        4. Dead letter queue
        5. Streaming interoperability
      7. Metadata
      8. Summary
    3.142.119.241