Batch vs. Stream Processing: Architectural Patterns

The choice between batch and stream processing defines the latency, consistency guarantees, and operational complexity of a modern data platform. While batch processing optimizes for throughput and historical completeness across bounded datasets, stream processing optimizes for low-latency insight on unbounded continuous data. Modern architectures, from Lambda to Kappa and beyond, attempt to unify these paradigms to serve diverse business needs.

In this deep dive, we will explore the underlying principles, mathematical models, architectural patterns, and real-world applications of both paradigms. We will also dissect the complexities of state management, exactly-once semantics (EOS), and the true financial cost of implementing these distributed systems.

1. Batch Processing: The Throughput Specialist

Batch processing operates on high-latency, large-volume data blocks. It is characterized by bounded datasets where the "end" of the data is strictly known. Processing is scheduled periodically (e.g., nightly, hourly) and executes transformations over the entire dataset or a specific time partition.

1.1 Core Strengths and Weaknesses

Strengths: High efficiency via parallelization (e.g., MapReduce, Apache Spark), reliable error recovery (simply re-running a failed job), deep historical analysis, and generally lower infrastructure costs.
Weaknesses: High latency (minutes to hours). The business must wait for the next batch cycle to see the results of incoming data.

1.2 The Mathematics of Batch Throughput

In a batch processing system, efficiency is driven by parallel execution across distributed worker nodes. We can model the expected time to complete a batch job using Amdahl’s Law and parallel execution metrics.

Let $N$ be the total number of records, $P$ be the number of parallel worker nodes, and $\lambda$ be the processing rate per node. The total batch execution time $T_{batch}$ can be approximated as:

T_{batch} = T_{setup} + \frac{N}{P \cdot \lambda} + T_{shuffle} + T_{commit}

Where $T_{shuffle}$ is the time taken to exchange data across the network (e.g., a distributed GROUP BY operation). Batch processing is highly efficient because $T_{setup}$ and $T_{commit}$ are amortized over a massive $N$ .

1.3 Real-World Application: Financial Reconciliation

Consider a global bank processing daily credit card settlements. The bank receives millions of transaction records throughout the day. At midnight, a batch job aggregates these transactions, calculates fees, and reconciles balances across merchant accounts.

Because financial reconciliation requires absolute correctness and involves complex joins against massive historical tables, batch processing is ideal. If a node fails, the job can be safely retried. The cost of running this nightly batch cluster might be around $50K per year, whereas attempting to maintain a globally consistent, real-time reconciliation engine could exceed $1.5M in engineering and infrastructure costs without providing significant additional business value.

2. Stream Processing: The Latency Specialist

Stream processing operates on unbounded data streams, processing events continuously as they arrive. Modern frameworks like Apache Flink, Apache Kafka Streams, and Spark Structured Streaming treat data as a never-ending flow.

2.1 Core Strengths and Weaknesses

Strengths: Sub-second latency, real-time alerting, continuous state updates, and immediate reaction to changing conditions.
Weaknesses: immense complexity in handling late-arriving data, distributed state management, and achieving exactly-once processing guarantees.

2.2 Queuing Theory in Stream Processing

Stream processing latency can be modeled using queueing theory (e.g., an M/M/1 or M/M/k queue). If events arrive at a rate $\lambda$ and a node processes them at a rate $\mu$ , the average number of events in the queue $L_q$ and the wait time $W_q$ increase exponentially as $\lambda$ approaches $\mu$ .

W_q = \frac{\lambda}{\mu(\mu - \lambda)}

If a sudden burst of traffic causes $\lambda > \mu$ , the buffer will grow indefinitely until memory is exhausted, causing the system to crash or drop packets. This makes backpressure—the ability of a consumer to signal to the producer to slow down—a critical requirement in streaming architectures.

2.3 Real-World Application: Fraud Detection

In e-commerce, identifying credit card fraud requires immediate action. A batch process that runs every hour is useless if the fraudulent transaction has already been approved and the physical goods shipped.

Using a stream processor, each transaction event is immediately scored against a machine learning model. The system maintains a sliding window of the user's past behavior. If the user makes three high-value purchases in different geographic locations within 5 minutes, the stream processor triggers an alert and blocks the card in milliseconds. While a robust Kafka and Flink infrastructure might cost $120K annually, it can prevent $3M in fraudulent chargebacks, making the ROI highly compelling.

3. Distributed Architectures: Lambda vs. Kappa

Historically, organizations struggled to choose between the latency of streaming and the reliability of batch processing. This led to architectural patterns designed to harness the best of both worlds.

3.1 Lambda Architecture

The Lambda architecture, popularized by Nathan Marz, attempts to provide both low-latency views and highly accurate historical views by running two parallel pipelines:

Batch Layer: The immutable "source of truth." It stores all raw data (e.g., in HDFS or S3) and periodically computes complex, highly accurate views using engines like Hadoop or Spark.
Speed Layer: Processes recent data in real-time (e.g., via Storm or Flink) to provide low-latency views. It compensates for the lag of the batch layer but prioritizes speed over perfect accuracy.
Serving Layer: Merges results from both layers to answer end-user queries.

Criticism: The main drawback of the Lambda architecture is logic duplication. Data engineers must write, test, and maintain the exact same business logic in two fundamentally different systems (e.g., SQL/Scala for the batch layer, and Java for the speed layer). This leads to divergence, subtle bugs, and increased maintenance costs.

3.2 Kappa Architecture

Proposed by Jay Kreps (co-creator of Apache Kafka), the Kappa architecture simplifies the system by removing the batch layer entirely. Everything is treated as an immutable stream.

Stream Layer: A single pipeline (e.g., Apache Flink) handles both real-time processing and historical processing.
Re-processing: To "re-run" a job (e.g., when business logic changes), the system simply provisions a new stream processing job that replays the historical data from the beginning of a high-retention log like Kafka, writing to a new output table. Once the new job catches up to real-time, the read traffic is switched over.

Advantage: A single, unified code base for all data processing. Disadvantage: Storing years of historical data in a message broker like Kafka can be prohibitively expensive compared to cheap object storage like AWS S3.

3.3 The Modern Compromise: Data Lakehouses

Today, the industry is moving towards Data Lakehouses (using formats like Apache Iceberg, Delta Lake, or Apache Hudi). These formats provide ACID transactions and stream-like consumption over cheap object storage, allowing engines like Spark and Flink to treat the same underlying storage as both a batch table and a streaming log, effectively blending Lambda and Kappa.

4. The "Exactly-Once" Semantics Challenge

In a distributed network, servers crash, networks partition, and packets drop. To guarantee fault tolerance, systems rely on retries, which introduces the risk of processing the same event multiple times.

4.1 Delivery Guarantees

At-Most-Once: Events are sent once. If a failure occurs, the event is lost. (Low latency, high risk).
At-Least-Once: Events are guaranteed to be processed. If an acknowledgment is lost, the producer retries. Requires idempotency to avoid duplicate data.
Exactly-Once Semantics (EOS): The holy grail. Events are processed and their side effects are reflected in the final state exactly once, regardless of failures.

4.2 Mechanisms for Exactly-Once

Achieving EOS requires complex coordination between the source, the stream processor, and the sink.

Distributed Snapshotting (Checkpointing): Engines like Apache Flink implement variations of the Chandy-Lamport algorithm. Flink periodically injects "checkpoint barriers" into the data stream. When an operator receives a barrier, it snapshots its local state to durable storage (e.g., S3). On failure, the entire graph is rolled back to the last successful checkpoint, and the data stream is rewound.
Transactional Writes: Even if the processor is exactly-once, the sink (database) might receive duplicate writes during a failure/rollback cycle. To prevent this, systems use Two-Phase Commit (2PC). The data is written to the database but kept uncommitted until the stream processor confirms the checkpoint is successful.
Idempotency: A simpler alternative to 2PC is designing the downstream system to be idempotent. Mathematically, an operation $f$ is idempotent if:

f(f(x)) = f(x)

For example, instead of streaming an UPDATE balance = balance - 10 (which is dangerous if duplicated), the stream processor calculates the final state and streams an UPSERT balance = 90 WHERE event_id = 123. No matter how many times the UPSERT is applied, the final balance remains correct.

5. Watermarking, Windowing, and Time Domains

In batch processing, time is simple: you process data from yesterday. In stream processing, data often arrives out of order due to network delays or mobile devices dropping offline. We must distinguish between:

Event Time: When the event actually occurred on the device.
Processing Time: When the stream processor received the event.

5.1 Windowing Strategies

Because streams are unbounded, we must slice them into finite blocks called windows to perform aggregations (e.g., "count of clicks").

Tumbling Windows: Fixed-size, non-overlapping time intervals (e.g., every 5 minutes).
Sliding Windows: Overlapping time intervals (e.g., the last 5 minutes, updated every 1 minute).
Session Windows: Dynamic windows based on user activity, separated by a defined period of inactivity.

5.2 Watermarks

If we are aggregating by Event Time, how long do we wait for late data before closing a window and emitting the result? A Watermark is a heuristic mechanism. A watermark of timestamp $W$ declares: "The system assumes all events with a timestamp $T < W$ have arrived."

If a tumbling window closes at 12:05, the system will wait until the watermark passes 12:05 before emitting the final count. Any data arriving after the watermark is considered "late" and can either be dropped or trigger an update to the previously emitted result.

6. Conclusion: Navigating the Trade-offs

Choosing between batch and streaming is ultimately a business decision driven by latency requirements and budget constraints.

Default to Batch: If the business SLA allows for hourly or daily latency, batch processing is vastly superior. It is easier to debug, cheaper to scale, and heavily optimized for large historical joins.
Embrace Streaming Selectively: Use stream processing only where sub-second latency translates directly to revenue or risk mitigation (e.g., algorithmic trading, fraud detection, real-time inventory management). Be prepared to invest heavily in engineering talent and complex infrastructure.
Converge Architecture: Look toward modern Data Lakehouse architectures that allow you to write standard SQL that can be executed incrementally as a stream or globally as a batch, minimizing code duplication while controlling costs.