Schema registries for event streams: subjects, schema IDs in the payload and checks at registration time

What it is

Serialised events need a schema to be read. Shipping the schema with every message is wasteful and relying on a shared library version is fragile, so a registry holds the schemas and hands out identifiers. The Confluent documentation (cited) describes the wire layout its serializers use: a version byte (0), a 4-byte schema ID returned by the registry, a message index for Protobuf, then the serialised data. A consumer reads the ID, fetches and caches the schema, and decodes. Schemas are registered under subjects, and the subject name strategy decides how subjects relate to topics: TopicNameStrategy (the default) derives the subject from the topic name and so requires all messages in a topic to conform to one schema; RecordNameStrategy groups by record name so that different event types can share a topic; TopicRecordNameStrategy combines both.

Why it matters

Compatibility is enforced when a schema version is registered, before any producer can use it. The documentation states that the default mode is BACKWARD (consumers on the new schema can read data written with the previous one), preferred for Kafka so that consumers can be rewound to the beginning of a topic; BACKWARD compares only with the latest version, while BACKWARD_TRANSITIVE compares with all earlier ones. The related article on Avro and Parquet explains what each mode allows.

How to apply

• Choose the mode from the deployment order (consumers first: backward; producers first: forward; unknown: full) and use a transitive variant when consumers replay history.
• Fix the subject strategy before the first message: changing it later renames subjects and changes what is compared with what.
• Register schemas from CI with a compatibility check against the registry instead of letting production producers auto-register whatever they were built with.
• Treat the registry as part of the data path: it needs backups, access control and monitoring, because a consumer that cannot fetch a schema cannot read anything.
• Keep the schema ID with archived messages; without the registry version that ID points to, old data is bytes.

Pitfalls

A registry does not make an incompatible change safe; it only refuses to register it, and the work then lands on the developer who needs the change. Deleting a schema version that messages still within retention reference makes them unreadable. Registries from different vendors use different wire layouts and identifier semantics, so producers and consumers must agree on one. A registry check covers the serialisation schema, not the meaning of a field; renaming amount from cents to francs passes every check.