Data Lakehouse Architecture: Open Table Formats, ACID Transactions, and Metadata Engines

The Data Lakehouse architecture represents the modern convergence of data lakes and enterprise data warehouses. By implementing transactional metadata management layers directly on top of open, cost-effective cloud object storage (such as AWS S3, Google Cloud Storage, or Azure Data Lake Storage), the Lakehouse provides ACID transaction guarantees, schema evolution, versioned time-travel, and high-performance analytical query processing without requiring redundant data movement into proprietary warehouse silos.

1. Architectural Evolution: Warehouse vs. Lake vs. Lakehouse

+-------------------------------------------------------------------------------+
|                       ANALYTICAL STORAGE ARCHITECTURAL EVOLUTION              |
+-------------------------------------------------------------------------------+
| Traditional Warehouse (EDW)   Data Lake (Hadoop / S3)   Modern Lakehouse      |
| - Proprietary storage formats - Open Parquet/ORC files  - Open Parquet files  |
| - Strong ACID & high speed    - Cheap object storage    - Open Table Formats  |
| - High cost, rigid schemas    - No ACID, dirty reads    - ACID, Time-Travel   |
| - Compute & storage coupled   - "Data Swamp" risk       - Decoupled Engines   |
+-------------------------------+-------------------------+---------------------+

The Two-Tier Architecture Bottleneck

Historically, organizations maintained two distinct systems:

  1. An unstructured/semi-structured Data Lake for raw ingestion and machine learning.
  2. A structured Data Warehouse for SQL business intelligence and reporting.

This two-tier design required continuous ETL data duplication, induced synchronization lag, broke data governance lineage, and caused high cloud egress and compute costs. The Lakehouse replaces this duality with a single, open storage tier accessible simultaneously by SQL engines (Trino, DuckDB, Snowflake), streaming frameworks (Apache Flink, Spark), and AI pipelines (PyTorch, Ray).

2. Open Table Formats: Apache Iceberg, Delta Lake, and Apache Hudi

Open table formats abstract collections of immutable Parquet or ORC data files into structured database tables.

+-------------------------------------------------------------------------------+
|               APACHE ICEBERG METADATA HIERARCHY ARCHITECTURE                  |
+-------------------------------------------------------------------------------+
| [ Iceberg Catalog (REST, Hive Metastore, AWS Glue, DynamoDB, JDBC) ]          |
|    | (Atomic Pointer Swap: points to current Metadata JSON)                   |
|    v                                                                          |
| [ Table Metadata JSON: v3.metadata.json (Schema, Partition Spec, Snapshots) ] |
|    |                                                                          |
|    v                                                                          |
| [ Manifest List File: snap-827391.avro (Snapshots of Manifest Files) ]        |
|    | (Contains Partition Summaries and Bounds for Fast Pruning)               |
|    +-----------------------------+-----------------------------+              |
|    |                             |                             |              |
|    v                             v                             v              |
| [ Manifest File 1 (.avro) ]  [ Manifest File 2 (.avro) ]   [ Manifest 3 ]     |
| (Data file paths & column    (Data file paths & column     (Data file paths)  |
|  min/max statistics)          min/max statistics)                             |
|    |                             |                                            |
|    v                             v                                            |
| [ Parquet Data Files ]       [ Parquet Data Files ]                           |
+-------------------------------------------------------------------------------+

Metadata Pruning and Partition Evolution

In traditional Hive-style directory partitioning (/date=2026-06-21/region=US/), modifying partitioning schemes required rewriting terabytes of data.

In Apache Iceberg, partition specifications are decoupled from physical storage layouts:

  1. Partition Evolution: Changing a partition spec (e.g., from days(ts) to hours(ts)) creates a new partition spec ID in the metadata JSON. Existing files remain untouched, while new writes use the updated layout.
  2. Hidden Partitioning: Users query natural timestamps WHERE event_time >= '2026-06-01' without explicitly referencing synthetic partition columns (event_date), preventing accidental full-table scans.
  3. Min/Max Column Statistics: Manifest files store lower and upper bounds for every column in each data file. The query engine prunes non-matching files during query planning at the coordinator level without contacting object storage.

3. ACID Transactions and Concurrency Control

Lakehouses enforce ACID guarantees on object stores that offer only eventual or read-after-write consistency.

Optimistic Concurrency Control (OCC)

When two transactions write simultaneously:

  1. Snapshot Read: Both transactions read the current snapshot S_0.
  2. Independent Write: Both write new data files and stage new manifest files.
  3. Commit via Atomic Compare-and-Swap (CAS):
    • Transaction 1 attempts to update the catalog pointer from S_0 \to S_1. It succeeds.
    • Transaction 2 attempts to update S_0 \to S_2. The CAS operation fails because the catalog is now at S_1.
    • Conflict Resolution: If Transaction 2 modified partitions disjoint from Transaction 1, it rebases against S_1 and commits S_2 without rewriting data files. Otherwise, it fails with a serialization conflict.
Row-Level Mutations: Copy-on-Write (COW) vs. Merge-on-Read (MOR):
+-------------------------------+-----------------------------------------------+
| Strategy                      | Mechanism and Trade-Offs                      |
+-------------------------------+-----------------------------------------------+
| Copy-on-Write (COW)           | Modifying 1 row rewrites the entire Parquet   |
|                               | file. High write amplification; optimal for   |
|                               | read-heavy analytical workloads.              |
+-------------------------------+-----------------------------------------------+
| Merge-on-Read (MOR)           | Writes modified rows to small Positional or   |
| (with Deletion Vectors)       | Equality Delete files. Fast real-time writes; |
|                               | query engine merges deletes at read time.     |
+-------------------------------+-----------------------------------------------+

4. The Medallion Multi-Hop Architecture

The Medallion Architecture structures data quality and transformation stages across three logical tiers:

Medallion Architecture Processing Flow:
Raw Sources (Kafka, CDC Logs, APIs, Files)
            |
            v
+-----------------------+
|  BRONZE (Raw Landing) | ---> Raw, immutable, append-only historical record
+-----------------------+      Preserves full schema fidelity and raw JSON payloads
            |
            v  [ Cleaning, Deduplication, Schema Enforcement, Typing ]
+-----------------------+
|  SILVER (Enriched)    | ---> Conformed, cleansed, joined enterprise datasets
+-----------------------+      Optimized for feature engineering & ad-hoc analysis
            |
            v  [ Business Aggregations, Metric Calculations, Dimensional Modeling ]
+-----------------------+
|  GOLD (Business Hub)  | ---> Star/Snowflake schema data marts and metrics
+-----------------------+      Sub-second analytical BI reporting (Tableau, PowerBI)

5. Storage Maintenance and Compaction Strategies

Unmanaged continuous streaming ingestion generates millions of tiny files (the "small file problem"), which degrades query performance due to object store metadata API latency.

Automated Maintenance Operations

  1. Bin-Packing / Compaction: Periodically merges small Parquet files (e.g., 5–10 MB) into optimal 128\,\text{MB} - 512\,\text{MB} columnar chunks.
  2. Z-Ordering & Space-Filling Curves: Multi-dimensional clustering algorithm that organizes data along a Hilbert or Peano space-filling curve, optimizing data skipping across multiple query filter columns simultaneously.
  3. Snapshot Expiration & Orphan File Deletion: Prunes expired metadata snapshots and removes unreferenced orphan data files, reclaiming cloud storage capacity.

6. Table Formats Feature Comparison

+---------------------------+-------------------+--------------------+------------------------+
| Feature                   | Apache Iceberg    | Delta Lake         | Apache Hudi            |
+---------------------------+-------------------+--------------------+------------------------+
| Primary Origin            | Netflix / Apache  | Databricks / Linux | Uber / Apache          |
| Multi-Engine Governance   | Universal (Trino, | Strong (Universal  | Strong (Presto, Spark, |
|                           | Spark, Snowflake) | via UniForm)       | Flink)                 |
| Metadata Format           | Avro Manifests    | JSON Delta Log     | Timeline + Avro Log    |
| Partition Evolution       | Fully Supported   | Metadata partition | Physical partition     |
| Row-Level Deletions       | Positional Deletes| Deletion Vectors   | Merge-on-Read Log      |
| Time-Travel Querying      | Snapshot / Time   | Version / Timestamp| Timestamp commit query |
+---------------------------+-------------------+--------------------+------------------------+

References

  1. Armbrust, M., et al. (2021). Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics. Proceedings of CIDR.
  2. Apache Software Foundation. (2024). Apache Iceberg Table Spec Version 2 & 3. Apache Iceberg Documentation.
  3. Zaharia, M., et al. (2020). Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores. Proceedings of the VLDB Endowment, 13(12), 3411–3424.
  4. Kleppmann, M. (2017). Designing Data-Intensive Applications. O'Reilly Media.
  5. Shvachko, K., et al. (2010). The Hadoop Distributed File System. IEEE MSST.