Тема: streaming
-
How far back should a scheduled pipeline reprocess for late-arriving events, and how have teams chosen the window?
Open question: stream engines admit that some events can be arbitrarily delayed, and batch schedulers run each interval once after it closes; a common compromise re-runs the last N intervals on every run, but N is usually a guess. What evidence has been used to size N, and what happened to the events that arrived later still?
-
JSON Lines: one value per line for logs, datasets and streamed responses
JSON Lines (also called NDJSON) puts one complete JSON value per line, UTF-8 without a byte order mark and newline-terminated; files can be appended, split, grepped and compressed, and a truncated stream loses only its last line. RFC 7464 JSON text sequences add a record-separator byte for recovery. Use either instead of one large JSON array whenever records are produced or consumed one at a time.
-
Deduplication strategies for records: exact rows, keep-latest by key and bounded windows
Decide first what counts as a duplicate: identical rows, several versions of one key, or messages redelivered within a window. Exact duplicates fall to DISTINCT; versions need a keep-latest rule with an explicit ordering; redelivery is deduplicated on an idempotency key within a bounded time or state window, as message queues and stream engines do.
-
Choosing between batch and streaming: required latency, event time and late data
Batch processes a bounded input after its interval closes and is reproducible by construction; streaming processes an unbounded input as it arrives and must reason about event time, watermarks and late data to give stable answers. Pick streaming only when a consumer acts within seconds of an event; otherwise the batch path is simpler, and it is needed for reprocessing anyway.
Машиночитаемо: JSON