Event Time, Watermarks, and Windowing

Streaming's hardest problems are not throughput but time: events arrive out of order, delayed, and duplicated, yet the questions we ask ("how many orders in the 10:00–10:05 window?") are about when things happened, not when they showed up. This page is the semantics course beneath every stream processor — the concepts are engine-independent and transfer verbatim between Flink, Kafka Streams, Spark, and the streaming databases.

Event time vs processing time

Event time is the timestamp carried by the event (when the order was placed); processing time is the clock on the machine handling it (when your operator saw it). The gap between them — skew — is unbounded in real systems: mobile clients buffer offline, producers retry, partitions lag, a stalled consumer replays an hour in a minute. Processing-time results are therefore non-reproducible: rerun the same data tomorrow and every window changes. The streaming discipline in one sentence: compute in event time, monitor in processing time. Reprocessing history (the log's replay superpower) only yields correct results if your logic was event-time-based to begin with.

Windows: how infinite streams become finite answers

Tumbling — fixed, non-overlapping (every 5 minutes): counts and sums.
Sliding/hopping — fixed length, overlapping starts (5-minute window every minute): smoothed trends; each event lands in several windows.
Session — gap-defined: a window per burst of activity, closed after inactivity (user sessions, device episodes). Data-driven boundaries, hence merge logic under the hood.
Global + triggers — the general escape hatch: one window, custom firing policy.

The open question every window scheme faces: when may a window close? Waiting forever is complete but useless; closing immediately is fast but wrong. Watermarks are the mechanism that answers it.

Watermarks: the completeness heuristic

A watermark at event-time $T$ is the system's claim: "expect no further events with timestamps ≤ $T$ ." Watermarks flow through the pipeline with the data; when the watermark passes a window's end, the window fires. They are generated at the sources — typically as observed-max-timestamp minus a bounded-delay allowance (e.g., 30 seconds) — and multi-input operators take the minimum of their inputs' watermarks, so one slow or idle partition holds back the whole pipeline's clock (the classic "my windows never fire" bug: an idle source; the fix: idleness timeouts).

The watermark is a bet, not a fact. Set the allowed delay small and you fire fast but declare stragglers late; set it large and results are complete but slow. That latency-vs-completeness dial is irreducible — the streaming version of a conclusion every distributed-systems course reaches: you cannot know a distributed input is complete; you can only decide when to stop waiting.

Late data: policy, not accident

Events arriving behind the watermark need an explicit decision:

Drop — acceptable when a bounded error is fine (dashboards).
Allowed lateness + updates — keep windows open in state past their firing; late arrivals re-fire the window with a correction. Downstream must handle retractions/updates, not just appends — the moment your pipeline's outputs become mutable, which propagates real design consequences into sinks.
Side-output — route late events to a separate stream for reconciliation jobs.

Production systems commonly pair fast approximate streaming results with periodic batch correction — honest about the bet the watermark made. Choose per use-case: billing tolerates latency but not loss; alerting tolerates loss but not latency.

The mental model that unifies it

Every streaming computation answers four questions (the Dataflow/Beam model): What are you computing (the aggregation)? Where in event time (the window)? When in processing time do you emit (triggers/watermarks)? How do refinements relate (discard/accumulate/retract)? Engines differ in syntax and defaults; the four questions — and the watermark bet — are the invariant core worth actually understanding.