Topic: data-quality
-
Pipelines that reject unexpected source schema changes at ingestion detect upstream changes sooner but fail more often than pipelines that coerce
Hypothesis: a pipeline that fails an ingestion task when the source schema differs from the declared one (in the way Avro's schema resolution signals an error when a reader field has no default and the writer lacks it) finds upstream changes within one run but fails on harmless changes too, while a coercing pipeline runs on and lets some changes reach consumers unnoticed; a proposed comparison on the same sources, with no result claimed.
-
Freshness and row-count checks on raw source tables catch most pipeline incidents earlier than column-level tests downstream
Hypothesis: in a warehouse with layered models, the majority of incidents that end up visible to report consumers first show as a stale or under-sized raw source load, so freshness and volume checks at the source layer detect them earlier than not-null, uniqueness and accepted-value tests on downstream models; a proposed comparison over recorded incidents.
-
Data quality checks: freshness, volume, nulls and uniqueness as a minimum test set
Four cheap checks catch most broken loads: the source was updated recently enough (freshness), the interval delivered a plausible number of rows (volume), keys and required measures are not null, and the declared grain is unique. Express each as a query that returns failing rows, run it after loading and before publishing, and separate warnings from blocking errors.
-
State units beside every measurement
Make measurement records interpretable by keeping units, aggregation and population beside each value.
-
Convert timestamps to UTC at the boundary
Reject ambiguous timestamps, convert aware instants to UTC and preserve the original timezone when local scheduling semantics matter.
-
Data quality checks: freshness, volume, nulls and uniqueness as a minimum set
Four cheap checks catch most broken load runs: is the source fresh enough, did a plausible number of rows arrive, are keys and measures populated, and is the declared grain unique? Write each check as a query that returns bad rows, run it after loading and before publishing, and separate warnings from blocking failures.
-
Confident Learning: auditing label quality with targeted review
Use out-of-sample class probabilities to rank suspected label issues, review them against annotation rules and preserve the distinction between model disagreement and verified error.
-
Canonicalize URLs without changing meaning
Use an application-defined comparison key while retaining the exact request URL when normalization might alter routing or signatures.
-
Deduplication strategies for records: exact rows, keep-latest by key and bounded windows
Decide first what counts as a duplicate: identical rows, several versions of one key, or messages redelivered within a window. Exact duplicates fall to DISTINCT; versions need a keep-latest rule with an explicit ordering; redelivery is deduplicated on an idempotency key within a bounded time or state window, as message queues and stream engines do.
Machine-readable: JSON