Pipelines that reject unexpected source schema changes at ingestion detect upstream changes sooner but fail more often than pipelines that coerce

Hypothesis

When a source system changes the shape of what it emits, a pipeline can do one of two things at ingestion. It can compare the incoming schema with the declared one and fail the task on any difference it cannot resolve; the Avro specification's schema resolution rules work this way, signalling an error, for example, when the reader's record schema has a field without a default that the writer's schema lacks. Or it can coerce: add missing columns as null, drop unknown ones, cast where possible, and keep running. Strict mode is commonly called fragile and permissive mode dangerous, with little evidence about the size of the trade. The hypothesis is that both intuitions are right and that the sizes matter: strict pipelines detect upstream schema changes within one scheduled run and almost never let a change through unnoticed, at the cost of several times more failed runs, most of them for changes that would have been harmless; coercing pipelines fail less and let a meaningful number of changes reach consumers unnoticed, with detection times of days rather than hours.

Prediction

Over the same sources and period, the strict pipeline shows a shorter median time from an upstream schema change to its first detection (a failed run) than the coercing pipeline (a consumer complaint, a data-quality alert or a later audit), more failed runs, a higher share of failures judged harmless afterwards, and fewer schema changes that reached a consumer table without a recorded detection. If the coercing pipeline detects changes about as fast, or the strict pipeline's extra failures are mostly for harmful changes, the hypothesis is false in that part.

Proposed test

1. Split one pipeline's sources into two groups with comparable change frequency, and run one group strict and the other coercing for at least six months.
2. Keep a schema change register: every upstream change, the date it took effect, how it was first detected, by whom, after how long, and whether it was later judged harmful (wrong values, dropped data, broken consumer) or harmless.
3. Count failed runs per group and classify each failure as schema-related or not; for schema-related failures, record whether the change was harmful.
4. Compare detection time distributions, failure counts and the number of undetected or late-detected harmful changes; report per source, since one noisy source can dominate.

Status

No result is claimed. The register in step 2 is the hard part, because the coercing pipeline's undetected changes are by definition not noticed at the time; a periodic schema audit against the source is needed to find them, and the audit interval bounds the measurable detection time.