Sujet : data-pipelines
-
How far back should a scheduled pipeline reprocess for late-arriving events, and how have teams chosen the window?
Open question: stream engines admit that some events can be arbitrarily delayed, and batch schedulers run each interval once after it closes; a common compromise re-runs the last N intervals on every run, but N is usually a guess. What evidence has been used to size N, and what happened to the events that arrived later still?
-
ETL versus ELT: where the transformation runs and what that changes
ETL transforms data in a separate engine before loading it into the target; ELT loads raw data first and transforms it with the target store's own processing. The choice decides where compute is paid for, what raw data lands in the warehouse, and how easily a transformation can be rerun.
-
Idempotent data pipelines: partition overwrite, safe reruns and backfills without double counting
A pipeline task should produce the same output whenever it is rerun for the same data interval: read a fixed partition of input, replace rather than append the corresponding partition of output, and upsert by key where replacement is impossible. Backfills then become ordinary reruns over a range of intervals instead of a source of duplicated rows.
-
Choosing between batch and streaming: required latency, event time and late data
Batch processes a bounded input after its interval closes and is reproducible by construction; streaming processes an unbounded input as it arrives and must reason about event time, watermarks and late data to give stable answers. Pick streaming only when a consumer acts within seconds of an event; otherwise the batch path is simpler, and it is needed for reprocessing anyway.
Lisible par machine : JSON