Event time is not wall-clock time: reading the Dataflow paper

The paper separates questions that engines often conflate

Akidau and colleagues organise the Dataflow model around the result being computed, its event-time grouping, the processing-time moment when output appears, and the relationship between successive outputs. Their 2015 paper argues that unbounded data should not be treated as a batch that will eventually become complete. My reading is that this is primarily a semantic contribution: it gives an application a vocabulary for choosing between timeliness, completeness and resource cost without allowing the execution engine's scheduling behaviour to decide the meaning of the answer.

That separation is valuable even for a small service with no dedicated stream processor. If a dashboard reports a minute's activity, someone must decide whether the minute refers to when the activity occurred or when a server received the message. Someone must also decide whether the number may change later. Without explicit decisions, retries, deployments and network interruptions become accidental business rules. I would first describe the desired answers in ordinary examples, then choose an engine and configuration that implement them. A faster engine cannot correct an unspecified definition of the metric.

References: [1] Akidau et al.: The Dataflow Model (2015)

Two clocks produce two legitimate, different answers

Consider two hypothetical sensor readings. The first has event time 12:00:15, value 4 and arrival time 12:00:17. The second has event time 12:00:40, value 7 and arrival time 12:03:10 after a disconnected device reconnects. Grouping by the device's event minute puts both in the 12:00 window for a total of 11. Grouping by server arrival minute produces 4 in the 12:00 window and 7 in the 12:03 window. Neither arithmetic operation is broken. They answer different questions about the same records.

The event-time result better describes activity during the observed minute, assuming the device timestamp is meaningful. The arrival-time result better describes ingestion load during each server minute. Mixing them creates a metric that changes its meaning under failure. I would therefore retain both timestamps and name the aggregation accordingly. I would also retain the raw timestamp and any normalisation decision. A device reporting tomorrow's date should not silently advance the entire pipeline's notion of progress. Timestamp validation is a separate input-quality policy, not a property delivered automatically by event-time windowing.

A watermark carries an assumption about progress

A watermark allows a pipeline to reason about how far event-time processing has progressed. Flink's documentation explains watermark propagation and the constraint imposed by multiple input streams. A slow input can hold back a downstream operator; excluding an idle input requires an explicit idleness policy. I do not read a watermark as a magical observation that every possible old event has arrived. Its strength depends on the source's progress information and the assumptions used to generate it. A heuristic based on recent timestamps remains a heuristic.

In the sensor example, a rule that subtracts two minutes from the largest observed event timestamp may appear to tolerate ordinary delay. It can still fail when one device uploads a day of buffered readings or another emits a timestamp far in the future. A global maximum is particularly dangerous if unrelated devices share the same progress estimate. I would test the watermark generator using disconnected, idle and misclocked sources, then document which late arrivals remain possible. That statement is more useful than a configuration value presented without the behaviour that justifies it.

References: [3] Apache Flink: Timely Stream Processing

A revised answer needs a stable identity

Apache Beam exposes triggers, accumulation behaviour and allowed lateness as distinct controls. That distinction makes an important application question unavoidable: when a late record changes a window, is the new output a replacement total or an additional contribution? A downstream consumer that adds every cumulative result will double count. One that overwrites with every incremental result will lose contributions. The pipeline and sink must agree on the interpretation, including what happens when an output is retried. Correct aggregation inside a worker is not sufficient.

For the invented minute, an early result of 4 followed by a corrected result of 11 can be represented as two revisions of the same window. A sink can store a key comprising sensor group, window start and metric version, then accept only increasing revision numbers. Alternatively, a changelog can carry a retraction of 4 followed by insertion of 11, if the sink supports that contract. I would not use arrival order alone to choose the winner. Retries can deliver an older revision after a newer one, and that should not make the dashboard move backwards.

key = (sensor_group, window_start, metric_version)
first_output  = { key, revision: 1, total: 4 }
late_output   = { key, revision: 2, total: 11 }
sink_update   = replace only when incoming.revision > stored.revision

References: [2] Apache Beam Programming Guide: Watermarks, triggers and late data

Completeness consumes a state budget

Keeping windows open longer has a storage cost even when the aggregation is a simple sum. Assume, for illustration, one million active keys and a per-key, per-window state cost of 160 bytes. Retaining ten windows requires 1.6 billion bytes before indexing, checkpoint copies and runtime overhead. Doubling the retention horizon roughly doubles this particular state component, although real workloads may have uneven key activity and additional buffers. These assumptions are deliberately explicit because a statement such as a few extra minutes of lateness can hide a substantial operational commitment.

A timeout that discards state is therefore also a correctness boundary. After it expires, the system needs a defined treatment for an older event: reject it, route it to correction processing, or rebuild the affected result from retained input. The appropriate choice depends on the product. A live operational display might tolerate a bounded approximation while a monthly report requires later reconciliation. I would keep those views distinct rather than force one retention policy to serve incompatible purposes. The same source events can support a fast provisional projection and a slower audited one.

The model does not validate the meaning of a timestamp

My main reservation is that precise temporal machinery can create unjustified confidence in imprecise source data. A device clock might be wrong, a business event might be timestamped when it was entered rather than when it occurred, and a correction might intentionally refer to an earlier period. The engine can faithfully compute the declared model while the model misrepresents the domain. I would distinguish event occurrence, source recording, ingestion and processing wherever those moments affect interpretation. Collapsing them into one field named timestamp makes later investigation unnecessarily difficult.

The counterargument is that most systems do not need this degree of temporal sophistication. If the real question is how much work reached a service during the last minute, processing time may be exactly right. A periodic batch may also meet the latency requirement with less operational complexity. The Dataflow paper should not be read as a requirement to deploy streaming infrastructure. I read it as a demand to separate semantics from execution. Once those semantics are explicit, an ordinary scheduled query may be the most appropriate implementation.

Review the late-data policy as part of the product

I would review a streaming feature with a small set of adversarial timelines before reviewing its throughput. Deliver a record after the nominal window closes. Duplicate it. Pause one source while others advance. Restart the job from a checkpoint and deliver an older output revision after a newer one. For each timeline, write down what the reader sees and whether that result is provisional, corrected or final under the stated policy. This makes the user-visible contract inspectable without requiring everyone to understand the runtime's internal scheduling.

The paper's enduring value for me is the permission to make refinement ordinary. A useful result can arrive early and become more complete later, provided the system identifies the result, communicates its status and preserves the rules for updating it. The difficult engineering decisions remain at the boundaries: trustworthy timestamps, affordable retained state, compatible sink semantics and a response to data outside the chosen horizon. Event time is a way to describe those decisions precisely. It does not remove the responsibility to decide what an answer means when the world arrives out of order.