Apache Kafka / Intermediate: contracts, failures and observability / Lesson 7

Serialization and schema evolution

Keep byte-level encoding, event meaning and reader compatibility in one reviewed contract.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Explain serializer and deserializer responsibilities.
Review additive changes, missing values and incompatible meaning changes.

The broker stores bytes with record metadata

A producer serializer converts a key or value into bytes, and a consumer deserializer interprets those bytes. Kafka's ability to transport a record does not establish agreement about units, required fields or schema versions. JSON, Avro and Protobuf represent different contracts and tooling choices; a schema registry is an additional component, not an automatic property of every Kafka topic. Record the encoding and schema identity explicitly. The key serializer also affects partitioning, so changing key encoding can move logically identical identifiers to different partitions even if the visible text appears unchanged in an application debugger.

Validate meaning after decoding

The offline decoder accepts a declared version-one event and checks that weight_g is an integer rather than merely accepting any valid JSON value. Boolean values need special attention in Python because bool is an int subclass; the exact type check deliberately excludes them. This is a small synthetic contract, not a complete schema library. A negative weight or unknown business identifier would require additional domain rules. Keep transport decoding failures distinct from business validation failures, because their recovery paths differ: malformed bytes may require quarantine, whereas a recognized but disallowed transition might require a correction event.

python Offline versioned payload validation

import json
event = json.loads('{"version":1,"weight_g":1250}')
assert event['version'] == 1
assert type(event['weight_g']) is int
assert event['weight_g'] >= 0
print(event['weight_g'])

Expected output

The assertions validate the declared fixture and reject a boolean if used for weight.

Additive is not automatically compatible

Adding an optional field can be compatible when old readers ignore it and new readers define a suitable default for its absence. Adding a field that silently changes interpretation of an existing number is different. If weight switches from grams to kilograms while retaining the same name and integer shape, syntax checks pass but business results become wrong. Renaming a field, changing enum membership and replacing a scalar with an object also deserve explicit reader-writer tests. Compatibility depends on the serialization format and configured registry policy when present, so do not call a JSON change universally backward compatible.

Test old and new readers against fixtures

The second example defines a new reader that supplies a default depot only when the field is absent. It demonstrates one deliberate evolution rule using both an older and a newer fixture. The fallback unknown is a business decision that must remain distinguishable from a real depot identifier. A null field is not necessarily the same as an absent field, and silently coercing both can hide upstream errors. Keep representative historical records in compatibility tests, including non-ASCII text, maximum identifiers and tombstones. Tests should compare meaning and routing assumptions as well as whether decoding raises an exception.

python An explicit missing-field default

def depot_of(event):
    return event.get('depot', 'unknown')

assert depot_of({'version': 1}) == 'unknown'
assert depot_of({'version': 2, 'depot': 'D2'}) == 'D2'
print(depot_of({'version': 1}))

Expected output

unknown

Only absence selects unknown; an explicit None would remain None under this contract.

Roll out readers before relying on new meaning

A safe rollout plans old-reader/new-writer and new-reader/old-writer overlap explicitly. Deploy readers that tolerate the intended additive field before writers require it, then observe actual version distribution. Retention means an old event can reappear during replay long after the newest producer rollout. Removing support solely because live traffic looks current can break recovery. For genuinely incompatible meaning, a new event type or topic with a reviewed migration may be clearer than a hidden reinterpretation. No registry, connector or Kafka client is installed here; only local JSON contract checks are performed.

Common errors and fixes

A valid JSON payload carries the wrong weight unit.

Why it happens: Syntax validation replaced semantic versioning.
How to fix it: Specify units in the contract and migrate incompatible meanings explicitly.
Prevent it next time: Test meaningful values across reader and writer versions.

A key serializer change breaks entity-local order.

Why it happens: Partitioning depends on serialized key bytes.
How to fix it: Review routing compatibility before changing key encoding.
Prevent it next time: Retain serialized-key fixtures in compatibility reviews.

Replay fails on an old version absent from live traffic.

Why it happens: Reader support was removed before retained history expired.
How to fix it: Restore reviewed historical decoding or migrate retained data through a controlled plan.
Prevent it next time: Tie schema retirement to replay and recovery requirements.

Check your understanding

1. Is adding required depot to every new event safe for all existing readers and stored records?

Show worked solution

No universal answer follows from the word adding. Existing readers may reject unknown fields, and new readers may reject old records without depot. Define the serialization compatibility rule, add a meaningful missing-field policy where appropriate, and test both reader-writer directions with retained fixtures before rollout.

2. Why does an integer weight_g test use type(value) is int rather than isinstance(value, int)?