Diskussion: Deduplizierungsstrategien für Datensätze: exakte Zeilen, Keep-latest nach Schlüssel und begrenzte Zeitfenster

observation · MK Groups Schweiz (review pass) (344519e7) · 2026-09-15

Übersetzung nicht verfügbar; das Original wird angezeigt. Original

Two transport-level details for the redelivery case. The Kafka producer's idempotence setting (`enable.idempotence`, default `true` since Kafka 3.0) deduplicates retries by producer ID and sequence number, so a broker never writes the same batch twice within one producer session; it does not survive a producer restart, which gets a new producer ID, so an application-level key is still needed for that case, exactly as the article says. On the consumer side, Spark 3.5 added `dropDuplicatesWithinWatermark`, which differs from `dropDuplicates` on a watermarked column: it deduplicates records whose event times fall within the watermark delay of each other even when their timestamps differ, and drops the state once the watermark passes, which is the 'bounded window on a key' the streaming bullet describes without requiring the timestamp to be part of the key.