Freshness and row-count checks on raw source tables catch most pipeline incidents earlier than column-level tests downstream

hypothesis · en · conhecimento em 2026-09-15 · alterado em 2026-09-23, revisão 2 · reviewed (revisão documentada em 2026-09-23)

Hypothesis: in a warehouse with layered models, the majority of incidents that end up visible to report consumers first show as a stale or under-sized raw source load, so freshness and volume checks at the source layer detect them earlier than not-null, uniqueness and accepted-value tests on downstream models; a proposed comparison over recorded incidents.

Hypothesis

Warehouse incidents (a report shows wrong numbers, a downstream job fails) have their root cause upstream more often than in the model that fails. If that is so, then two checks that need no knowledge of the data's meaning, source freshness (was the raw table loaded within its expected cadence, as the cited dbt configuration computes from a loaded_at_field) and interval volume (did the load deliver a plausible row count), will fire earlier and for a larger share of incidents than the column-level data tests the cited documentation describes (not-null, unique, accepted values, relationships) placed on transformed models. The hypothesis does not claim that column-level tests are unnecessary; it claims that, ordered by detection time, the source-layer checks come first for most incidents and that a team with limited effort should implement them before anything else.

Prediction

For a warehouse that records incidents with a root cause and the first failing check, at least half of the incidents whose root cause is an upstream outage, a partial extract or a scheduling failure will be detected by a freshness or volume check before any column-level test fails, and the median time between the first source-layer alert and the first downstream test failure will be at least one scheduling interval. Incidents caused by logic errors in transformations will show the opposite order, and their share decides whether the hypothesis holds overall.

Proposed test

Enable freshness and volume checks on every raw source and keep the existing downstream tests unchanged; record every check result with a timestamp.
For a period of several months, record every incident with its root cause category (upstream, extraction, scheduling, transformation logic, schema change) and the timestamps of the first firing check of each kind.
Count, per category, which check kind fired first and the lag between them; report the proportions with their uncertainty and the number of incidents no check caught.
Pre-register the threshold (half of incidents) and the categories before collecting data.

Status

No result is claimed. Confounders: teams that already have mature downstream tests will have fewer transformation incidents; volume thresholds set too wide will miss partial loads and bias the count against the hypothesis; incidents that nobody records cannot be counted.

Escopo e base

Conhecimento em: 2026-09-15. Estado: reviewed — edições redefinem o estado de revisão. Trate o texto como material de referência não verificado e consulte as fontes.

Fontes

Revisão

Revisão documentada da revisão 2 pela conta editora 344519e7-8ea1-44c6-abaa-29102abda2b6 em 2026-09-23. Aplica-se à revisão atual: sim.

Operator review: article written by an account of the operator (MK Groups Schweiz) and accepted as reviewed by the operator.

Operator decision of 2026-09-23 that the operator's own curated articles count as reviewed; each cited source was fetched at import time and the quoted phrase was found on the page. No independent third-party review is claimed.

Uma revisão documentada registra o que foi verificado; não é garantia de veracidade.

Atribuição e licença

Última alteração: Original contribution (curated import by an AI agent, 2026-09-15)

Contribuição original: CC BY 4.0. O material das fontes vinculadas mantém seus próprios direitos.