The log stores events; a stream processor turns them into answers — aggregations, joins, enrichments, and materialized state that updates as data arrives. The engines differ on three axes that matter: execution model (true streaming vs micro-batch), deployment shape (cluster vs library vs database), and how seriously they take event time and state.
Flink processes record-at-a-time with first-class managed state: keyed state lives in an embedded store (RocksDB for large state), checkpoints snapshot the entire pipeline consistently via distributed barrier snapshotting, and recovery restores exactly where processing stood. Event-time semantics, watermarks, and the richest windowing model in the field are native, not bolted on (the semantics page covers the concepts). End-to-end exactly-once works with transactional sinks (Kafka, Iceberg) via two-phase commit hooked to checkpoints. Flink SQL has matured into a serious interface — much production Flink is now SQL, not DataStream code.
Costs: it is a distributed cluster to operate (JobManager/TaskManagers, checkpoint storage, savepoint management, upgrade choreography), and tuning large state is a genuine skill. Managed offerings (Confluent, AWS, Ververica, Decodable) exist precisely because self-hosting Flink is the heaviest option here.
Fit: the default when requirements include large keyed state, event-time correctness under disorder, low latency, and strong guarantees — fraud detection, sessionization, streaming joins, CDC-driven materialization.
Kafka Streams is just a JVM library: your stream processor is an ordinary application — deploy it like any service, scale by running more instances (partitions rebalance automatically), no separate cluster. State is local (RocksDB) and backed by Kafka changelog topics; exactly-once comes from Kafka transactions; interactive queries expose state directly from the app. The constraint is symmetrical: Kafka in, Kafka out, JVM only, and per-instance state that must fit the partition-assignment model.
Fit: Kafka-centric microservices that are stream processors — enrichment, per-key aggregation, joining a stream to a table — where introducing a Flink cluster would double the operational estate for one team's needs. (ksqlDB, its SQL veneer, has faded — Flink SQL won that lane; new SQL-first work should look there or at the streaming databases below.)
Spark processes streams as micro-batches on the Spark engine: the same DataFrame/SQL code serves batch and streaming, the scheduler amortizes work per batch, and latency floors sit at hundreds of milliseconds to seconds. Watermarks and stateful operations exist but are less expressive than Flink's; the compelling case is organizational — if Spark already runs your batch platform (Databricks especially), extending it to near-real-time via one API and one ops model beats adding a second engine. Real-time SLAs below the micro-batch floor are the signal to step off.
The newest class inverts the interface: instead of writing pipelines, you CREATE MATERIALIZED VIEW over streams and the engine keeps it incrementally correct — RisingWave (Postgres-compatible, S3-backed state, open source) and Materialize (differential dataflow lineage) lead it, with Arroyo-style lightweight engines nearby. They collapse the pipeline-plus-serving-store pattern into one system queryable like a database. Trade-offs: younger operational track records, and complex logic can outgrow SQL. Fit: real-time dashboards, features, and joins where the deliverable is an always-fresh table rather than an event-driven program.
| Situation | Engine |
|---|---|
| Heavy state, event-time correctness, strong guarantees | Flink |
| Kafka-native service-level processing, no new cluster | Kafka Streams |
| Existing Spark/Databricks platform, seconds-latency OK | Spark Structured Streaming |
| The output is a live table; SQL suffices | RisingWave / Materialize class |
| Simple stateless transforms/filtering | Often just consumers — don't buy an engine for a filter() |