Sujet : streaming
-
Jusqu'où en arrière un pipeline planifié doit-il retraiter les événements arrivés en retard, et comment les équipes ont-elles choisi la fenêtre ?
Question ouverte : les moteurs de flux admettent que certains événements peuvent être retardés arbitrairement, et les ordonnanceurs par lots exécutent chaque intervalle une seule fois après sa clôture ; un compromis courant réexécute les N derniers intervalles à chaque exécution, mais N relève souvent de l'estimation. Quelles preuves ont servi à dimensionner N, et qu'est-il advenu des événements arrivés encore plus tard ?
-
JSON Lines: one value per line for logs, datasets and streamed responses
JSON Lines (also called NDJSON) puts one complete JSON value per line, UTF-8 without a byte order mark and newline-terminated; files can be appended, split, grepped and compressed, and a truncated stream loses only its last line. RFC 7464 JSON text sequences add a record-separator byte for recovery. Use either instead of one large JSON array whenever records are produced or consumed one at a time.
-
Deduplication strategies for records: exact rows, keep-latest by key and bounded windows
Decide first what counts as a duplicate: identical rows, several versions of one key, or messages redelivered within a window. Exact duplicates fall to DISTINCT; versions need a keep-latest rule with an explicit ordering; redelivery is deduplicated on an idempotency key within a bounded time or state window, as message queues and stream engines do.
-
Choosing between batch and streaming: required latency, event time and late data
Batch processes a bounded input after its interval closes and is reproducible by construction; streaming processes an unbounded input as it arrives and must reason about event time, watermarks and late data to give stable answers. Pick streaming only when a consumer acts within seconds of an event; otherwise the batch path is simpler, and it is needed for reprocessing anyway.
Lisible par machine : JSON