DataEngineering Hub

This cluster covers the engineering side of data — pipelines, streaming, modeling, transformation, and the catalog layer that turns raw data into something usable. The focus is the operational and architectural patterns; modeling and analysis are adjacent topics.

Strategy and Lifecycle

• Data Maturity Lifecycle — A structural roadmap from fragmented silos to Data Mesh.
• Shift Left Data Engineering — Moving data quality upstream via contracts.

Streaming Platforms

The event-log substrate and its alternatives.

• Apache Kafka Fundamentals — Partition mechanics, durability trade-offs, and production failure modes
• Kafka Alternatives: Streaming Platforms Compared — Pulsar, Redpanda, WarpStream, NATS, RabbitMQ, and the decision rules
• Apache Pulsar Fundamentals — BookKeeper segments, multi-tenancy, tiered storage
• Redpanda, WarpStream, and Kafka-Compatible Engines — Faster disks vs no disks under the Kafka protocol
• NATS and JetStream — Lightweight pub/sub, request-reply, and edge messaging
• RabbitMQ vs Kafka: Queues vs Logs — The fundamental distinction that decides the choice
• Schema Registry and Evolution — Avro/Protobuf compatibility across producers and consumers
• Change Data Capture — Debezium and log-based database streaming

Stream Processing

Turning event streams into answers.

• Stream Processing Engines: Flink, Kafka Streams, Spark — The compute layer compared, plus SQL-native streaming databases
• Event Time, Watermarks, and Windowing — The semantics every engine implements
• Exactly-Once and Delivery Guarantees — What the guarantees promise, and effectively-once design
• Apache Spark Fundamentals — Catalyst, shuffles, and data-skew strategies

Pipeline design

• DataPipelineDesign — Sources, transforms, sinks; idempotency and observability
• EtlVsElt — When transform belongs early vs. late
• MapReduceParadigm — The paradigm that defined the batch era
• DbtAndAnalyticsEngineering — dbt as transformation tool, the analytics-engineering role

Vertical-Specific Pipelines

• Fintech Data Ingestion Blueprint — Ingesting, normalizing, and storing third-party financial data

Modeling

• Data Modeling Fundamentals — Star, snowflake, dimensional, the fact-and-dim mental model
• NoSQL Database Types — When and why to move beyond relational
• Jsonb In Postgresql — Handling semi-structured data in a relational engine
• Master Data Management — MDM as the discipline; tools as the implementation

Catalogs and metadata

• Data Catalog Tools — DataHub, Amundsen, Atlan; what they actually do
• Data Lake Architecture — Organizing massive unstructured datasets

Adjacent clusters

• Cloud Platforms Hub — Where pipelines and warehouses run
• DevOps and SRE Hub — Operating data pipelines
• Distributed Systems Hub — The consistency and consensus theory beneath streaming