テーマ： data-quality
-
Pipelines that reject unexpected source schema changes at ingestion detect upstream changes sooner but fail more often than pipelines that coerce
Hypothesis: a pipeline that fails an ingestion task when the source schema differs from the declared one (in the way Avro's schema resolution signals an error when a reader field has no default and the writer lacks it) finds upstream changes within one run but fails on harmless changes too, while a coercing pipeline runs on and lets some changes reach consumers unnoticed; a proposed comparison on the same sources, with no result claimed.
-
Freshness and row-count checks on raw source tables catch most pipeline incidents earlier than column-level tests downstream
Hypothesis: in a warehouse with layered models, the majority of incidents that end up visible to report consumers first show as a stale or under-sized raw source load, so freshness and volume checks at the source layer detect them earlier than not-null, uniqueness and accepted-value tests on downstream models; a proposed comparison over recorded incidents.
-
Data quality checks: freshness, volume, nulls and uniqueness as a minimum test set
Four cheap checks catch most broken loads: the source was updated recently enough (freshness), the interval delivered a plausible number of rows (volume), keys and required measures are not null, and the declared grain is unique. Express each as a query that returns failing rows, run it after loading and before publishing, and separate warnings from blocking errors.
-
State units beside every measurement
Make measurement records interpretable by keeping units, aggregation and population beside each value.
-
Convert timestamps to UTC at the boundary
Reject ambiguous timestamps, convert aware instants to UTC and preserve the original timezone when local scheduling semantics matter.
-
Datenqualitätsprüfungen: Aktualität, Menge, Nullwerte und Eindeutigkeit als Mindestsatz
Vier billige Prüfungen fangen die meisten kaputten Ladeläufe: Ist die Quelle frisch genug, kam eine plausible Zeilenzahl, sind Schlüssel und Kennzahlen gefüllt, ist die erklärte Körnung eindeutig? Jede Prüfung als Abfrage formulieren, die fehlerhafte Zeilen liefert, nach dem Laden und vor dem Veröffentlichen ausführen, Warnung und Blockade trennen.
-
Canonicalize URLs without changing meaning
Use an application-defined comparison key while retaining the exact request URL when normalization might alter routing or signatures.
-
Deduplication strategies for records: exact rows, keep-latest by key and bounded windows
Decide first what counts as a duplicate: identical rows, several versions of one key, or messages redelivered within a window. Exact duplicates fall to DISTINCT; versions need a keep-latest rule with an explicit ordering; redelivery is deduplicated on an idempotency key within a bounded time or state window, as message queues and stream engines do.
機械可読： JSON