Choosing between batch and streaming: required latency, event time and late data

What it is

A batch job reads a bounded input, usually one scheduling interval, after the interval has ended, and writes a result that can be recomputed from the same input at any time. A streaming job reads an unbounded input, which the Dataflow documentation (cited) describes as a collection with potentially infinitely many elements per key, so that grouping by key alone is impossible and aggregation needs windows, watermarks and triggers; a watermark is the threshold at which the system expects all of the data in a window to have arrived, and data arriving later with a timestamp inside the window is late data. The Flink documentation (cited) separates two notions of time: processing time, the clock of the machine running the operator, which is simplest and lowest-latency but not deterministic because results depend on arrival speed and outages; and event time, the time each event occurred on its producing device, which gives consistent results even for out-of-order events or when reprocessing history, at the cost of waiting for stragglers, and only for a finite time, since some elements can be arbitrarily delayed.

Why it matters

The difference is not speed but semantics. A batch total for Monday is defined by the rows present when the job ran; a streaming total for Monday is a sequence of provisional results that may still change when late events arrive, and the design must say whether they are dropped, emitted as corrections or ignored. Teams that adopt streaming for a dashboard refreshed hourly pay for state, checkpoints and an always-on job without anyone consuming the latency they bought.

How to apply

• Start from the consumer: name the action taken on the result and how soon after the event it must happen. Only a reaction within seconds to a minute justifies a streaming path; a report can wait for the interval to close.
• If streaming is justified, use event time, define the allowed lateness explicitly and route late events to a side output that a batch correction consumes.
• Keep a batch reprocessing path even for streamed results: it is the reference for correctness, the recovery tool after a bug, and the way to backfill history.
• Consider micro-batches (minutes) before a continuous engine; they keep batch semantics with lower latency.

Pitfalls

Stream joins hold state for the join window on both sides; unbounded windows mean unbounded memory. Exactly-once output requires idempotent or transactional sinks, not just an engine setting. Two answers, streamed and batched, for the same metric must be reconciled or consumers will trust neither.