Batch vs. Stream Processing: Architectural Patterns

The choice between batch and stream processing defines the latency and consistency guarantees of a data platform. While batch processing optimizes for throughput and historical completeness, stream processing optimizes for low-latency insight. Modern architectures attempt to unify these paradigms.

1. Batch Processing: The Throughput Specialist

Batch processing operates on high-latency, large-volume data blocks. It is characterized by bounded datasets where the "end" of the data is known.

• Strengths: High efficiency via parallelization, reliable error recovery (re-running a job), and deep historical analysis.
• Weaknesses: High latency (minutes to hours).

2. Stream Processing: The Latency Specialist

Stream processing operates on unbounded data streams, processing events as they arrive.

• Strengths: Sub-second latency, real-time alerting, and continuous state updates.
• Weaknesses: Complexity in handling late-arriving data and state management.

3. Distributed Architectures: Lambda vs. Kappa

3.1 Lambda Architecture

The Lambda architecture attempts to provide both low-latency views and highly accurate historical views by running two separate pipelines:

1. Batch Layer: The "source of truth." It stores all raw data and periodically computes complex, accurate views (e.g., via Spark/MapReduce).
2. Speed Layer: Processes recent data in real-time to provide low-latency views (e.g., via Flink/Storm). It compensates for the lag of the batch layer.
3. Serving Layer: Merges results from both layers to answer queries.

Criticism: The main drawback is logic duplication. Developers must write and maintain the same transformation logic in two different systems (e.g., Java for the speed layer and SQL/Scala for the batch layer).

3.2 Kappa Architecture

Proposed by Jay Kreps, the Kappa architecture simplifies the system by removing the batch layer entirely. Everything is treated as a stream.

1. Stream Layer: A single pipeline (e.g., Apache Flink or Kafka Streams) handles both real-time processing and historical re-processing.
2. Re-processing: To "re-run" a batch job, the system simply re-plays historical data from the beginning of the stream (using a high-retention log like Kafka).

Advantage: Single code base for all data processing.

4. The "Exactly-Once" Semantics Challenge

Achieving exactly-once semantics (EOS) in distributed systems is difficult due to network failures and retries.

4.1 At-Least-Once vs. Exactly-Once

• At-Least-Once: Events are guaranteed to be processed, but may be processed multiple times due to retries. Requires idempotency at the sink to avoid duplicate data.
• Exactly-Once: Events are processed and reflected in the final state exactly once, even if failures occur.

4.2 Mechanisms for Exactly-Once

1. Checkpointing (Flink): Periodic snapshots of the entire distributed state. On failure, the system rolls back to the last successful checkpoint.
2. Transactional Writes: Using two-phase commit (2PC) between the stream processor and the sink (e.g., Kafka's transactional producer).
3. Idempotency: Designing the processing logic such that f(x) = f(f(x)). For example, using UPSERT into a database based on a unique event ID rather than INSERT.

5. Watermarking and Windowing

In streaming, data often arrives out of order.

• Windowing: Grouping events by time (Sliding, Tumbling, or Session windows).
• Watermarks: A mechanism to signal that no more events with a timestamp earlier than T are expected. It allows the system to close a window and emit results despite the unbounded nature of the stream.

Summary: When to Use Which?

• Use Batch: For financial reconciliation, historical reporting, and large-scale model training.
• Use Stream: For fraud detection, real-time dashboards, and sensor monitoring.
• Use Kappa: When you want to simplify operations and your stream processor can handle the historical volume.