Schema evolution with Avro and Parquet: reader and writer schemas, merged files and compatibility modes

What it is

Readers must cope with records written under another schema version. The Avro specification (cited) resolves the writer's schema, which travels with the data, against the reader's schema, which the application expects. Record fields are matched by name and order does not matter; a writer field absent from the reader is ignored; a reader field absent from the writer takes the reader's default, and without a default an error is signalled. Promotions go one way (int to long, float or double; long to float or double; float to double); string and bytes are interchangeable. Parquet has no resolution step: each file carries its schema, and the Spark documentation (cited) describes merging the schemas of files that differ but are mutually compatible, off by default as a relatively expensive operation and enabled per read with mergeSchema. The Confluent Schema Registry documentation (cited) names the compatibility modes a registry can enforce: backward (consumers using the new schema can read data produced with the last schema), forward (old consumers read new data), full (both), and transitive variants checked against all earlier versions; backward is the default because it lets consumers rewind to the beginning of a topic.

Why it matters

A field renamed in the producer is, to a reader whose schema carries no alias for the old name, one field deleted and one added (Avro has field aliases, but the specification makes their use optional); a field added without a default breaks every consumer of old data; a type changed from int to string breaks resolution. These failures appear at read time, in consumers the schema author may not know.

How to apply

• Add fields as optional with a default (in Avro, a union with null, default null); never remove a field consumers still read.
• Treat a rename as add-then-remove over two releases, writing both fields in between, unless every reader honours aliases.
• Keep Parquet column names stable and rely on names, not positions; enable schema merging only for reads that need it, and register the current schema in the table catalogue.
• Choose the compatibility mode from the deployment order: backward when consumers deploy first, forward when producers do, full for either.
• For an incompatible change, create a new subject, topic or table version and write both as long as old readers exist.

Pitfalls

An Avro default must be a value permitted for the field's schema; for a union it must match one of the union's branches. An enum symbol unknown to the reader is an error unless the reader's enum has a default. Parquet files from different tools may disagree on logical types (timestamps, decimals) even when physical types match.

Which compatibility mode enforces which rule

A registry enforces only the direction its mode names. BACKWARD (the Confluent default) checks that a new consumer schema reads old data; it permits deleting fields and adding fields with defaults, so it will register a producer schema that removes a field old consumers still read. FORWARD checks that old consumers read new data; it permits adding fields and deleting only optional ones. FULL checks both, and the transitive variants check against every earlier version instead of the last. Choose the mode from who deploys first: consumers first, BACKWARD; producers first, FORWARD; unknown or many consumers, FULL. The rule 'never remove a field consumers still read' is enforced by the registry only under FORWARD or FULL.