Apache Kafka / Basic: records, partitions and reader progress / Lesson 1

Events, topics and the KRaft architecture

Locate an event inside Kafka and separate record storage, client behavior and cluster metadata.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Explain the responsibilities of producers, brokers, controllers and consumers.
Choose an event identity and distinguish append-only history from current state.

An event records a fact

An event says that something happened: a synthetic parcel was accepted at a depot, for example. Its identity, occurrence time, business key and payload have different purposes. The identity distinguishes this occurrence from a retry; the key determines related processing; the timestamp describes a chosen clock; the payload carries the fact. Kafka stores serialized records, so those business meanings remain an application agreement. Describe who emits the fact and whether corrections arrive as new events. A record saying parcel-created is easier to replay safely when it describes a completed decision rather than implicitly asking every reader to perform an irreversible action.

Inspect one synthetic record

The JSON fixture separates the event identifier from the parcel identifier. Two different facts about parcel P17 keep the same business key while receiving different event identifiers. A retransmission of this exact fact retains evt-101. Headers and payload fields are application choices, not proof that the broker validates a schema. The fixture is a readable representation of bytes a producer could serialize; it is not Kafka's wire format. Readers should reject unsupported event versions deliberately and avoid exposing personal addresses in keys, because keys often appear in logs, metrics labels and operational investigations.

json A synthetic business event

{"event_id":"evt-101","event_version":1,"key":"P17","type":"parcel-created","status":"accepted"}

The key groups parcel history; event_id identifies one occurrence. JSON validity does not establish a Kafka schema contract.

Separate the data plane from metadata

A broker serves partition records to producers and consumers. KRaft controllers maintain cluster metadata through a controller quorum: topic definitions, partition leadership and broker registrations belong to this control plane. Replicating metadata does not mean every controller stores every application record. A partition leader handles its partition's traffic while follower replicas copy its log. Deployment designs can combine roles for small demonstrations, but production role placement requires an explicit availability and capacity design. This course selects Apache Kafka 4.1 documentation; Kafka 4.x uses KRaft and old ZooKeeper-based setup instructions do not describe this baseline.

Model history and a materialized view separately

The offline example derives a current parcel status by visiting two events in their declared order. Both events remain in the history even though the map retains only the newest status for the key. This distinction explains why Kafka can support several independent readers with different projections. One service might derive delivery progress while another counts depot arrivals. The Python dictionary is not a broker, a distributed transaction or a complete event store. Its small result illustrates the projection rule only. A real replay also needs version handling, duplicate handling, ordering assumptions and policies for events that contradict previous facts.

python Offline projection of declared event order

events = [('P17', 'accepted'), ('P17', 'sorted')]
current = {}
for key, status in events:
    current[key] = status
assert len(events) == 2
assert current == {'P17': 'sorted'}
print(current['P17'])

Expected output

sorted

Two facts produce one current-state entry; no Kafka service is involved.

Write down the operating boundary

Kafka is useful when retained event history and independent consumers solve a concrete problem. It does not automatically replace an authoritative database, guarantee the success of an external email, or remove the need to design contracts. Start with one bounded synthetic flow and define the acceptable delay, retention period and recovery requirement. Keep client versions, broker versions and enabled features in the same review record. This material runs no broker, formats no storage and contacts no configured endpoint. Configuration examples later in the course remain illustrations so their expected behavior is always conditional on the stated environment.

Common errors and fixes

A controller is treated as a copy of all topic data.

Why it happens: Metadata replication and partition replication were confused.
How to fix it: Draw the controller quorum separately from each partition replica set.
Prevent it next time: Review data-plane and control-plane failure cases independently.

A retry receives a new event identifier and is counted twice.

Why it happens: Request attempt identity replaced business occurrence identity.
How to fix it: Retain the original event identifier across retransmission of that fact.
Prevent it next time: Document where event identifiers are created and persisted.

A Kafka 4.1 design includes ZooKeeper as a required component.

Why it happens: An older setup guide was applied without version review.
How to fix it: Use the selected KRaft documentation and assess legacy migration separately.
Prevent it next time: Record the exact documentation and runtime baseline.

Check your understanding

1. Should parcel-created and parcel-sorted for P17 have the same event identifier?

Show worked solution

No. They describe different occurrences and need distinct identifiers, while the business key can remain P17. A retry of parcel-created keeps its own original identifier. Consumers can then group facts by parcel and recognize retransmission without confusing a legitimate later status change with a duplicate.

2. Why does a successful local projection not prove recovery after losing a broker?