Apache Kafka / Basic: records, partitions and reader progress / Lesson 2

Partitions, keys and ordering

Design the scope of ordering and see how partition count changes affect routing assumptions.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Explain partition-local offsets and ordering.
Choose a key that matches the required business sequence.

Order belongs to a partition

A topic contains partitions, and each partition is an ordered append-only log. Offset 12 in partition zero is unrelated to offset 12 in partition one; neither number establishes which business event occurred first across the two. When parcel updates must be processed in sequence, the producer and consumer design must keep the relevant key on a compatible partitioning path and preserve processing order. A timestamp is useful context but is not a universal ordering oracle. Clocks can disagree, two facts can share a timestamp, and late arrival can append an older business occurrence after a newer one.

Use topic, partition and offset together

The fixture contains two records at offset zero, each in a different partition. A record coordinate therefore needs topic, partition and offset rather than an offset alone. These coordinates identify positions within a retained log; they do not replace a business event identifier across copied topics or replayed environments. The assertion counts two distinct coordinates even though the offsets match. This small example can guide log formatting: include the full coordinate when diagnosing one reader's progress, and separately include the application identifier when comparing the same fact across a producer, processing service and external destination.

python Partition-local coordinates

coordinates = [('parcels', 0, 0), ('parcels', 1, 0)]
assert len(set(coordinates)) == 2
print(len(set(coordinates)))

Expected output

Equal offset numbers do not mean equal record positions.

Choose locality from business invariants

Partitioning by parcel identifier supports parcel-local sequencing, while partitioning by depot supports depot-local sequencing and can concentrate a busy depot's traffic. Neither is universally correct. Identify the operation that needs serialization and the largest plausible hot key. Adding consumers cannot divide one traditional consumer-group partition among several active owners. Conversely, using an unrelated random key may distribute load while breaking the sequence the application depends on. Null-key routing is client behavior and can use batching strategies rather than stable business grouping. Do not depend on an undocumented hash calculation across languages or client versions.

Understand a partition-count change

The arithmetic example deliberately uses a declared training hash value of five. With three partitions it selects partition two; with four it selects partition one. This is not Kafka's actual partitioner implementation. It isolates the important design fact: a mapping that includes partition count can move a key when that count changes. Existing records remain in their original partitions, so old and new records can temporarily be read concurrently by different owners. A reviewed repartitioning or new-topic migration must preserve the required business ordering rather than assuming a partition increase rewrites historical placement.

python A declared arithmetic routing model

training_hash = 5
assert training_hash % 3 == 2
assert training_hash % 4 == 1
print(training_hash % 3, training_hash % 4)

Expected output

2 1

This illustrates count-dependent movement and does not implement Kafka hashing.

Separate broker order from application completion

Receiving A before B does not ensure an asynchronous worker finishes A before B. If both modify one parcel, parallel execution can store the older state last. Preserve partition order through synchronous processing, key-aware scheduling or a downstream version check appropriate to the business rule. Committing past unfinished work introduces a second risk because a replacement reader may skip it. Decide both execution ordering and acknowledgment ordering. The traditional consumer-group model used here assigns each partition to at most one active consumer in that group; Kafka 4.1 share groups have different semantics and remain outside these examples.

Common errors and fixes

Two updates to one parcel complete in reverse order.

Why it happens: Asynchronous processing discarded partition order.
How to fix it: Serialize the key or enforce an appropriate version rule at the destination.
Prevent it next time: Test overlapping work on the same key.

A partition increase is assumed to preserve all key routing.

Why it happens: The key mapping depended on the previous partition count.
How to fix it: Design a migration that accounts for historical and new placement.
Prevent it next time: Review ordering before changing topic topology.

An offset is logged without topic or partition.

Why it happens: A partition-local position was treated as globally unique.
How to fix it: Log the full record coordinate and business event identifier.
Prevent it next time: Standardize diagnostic context in consumer code.

Check your understanding

1. A topic has four partitions and seven traditional group consumers. How many can own partitions simultaneously?

Show worked solution

At most four can actively own those four partitions within that group. The remaining consumers may be idle for this topic. This does not predict balanced processing time: one hot partition can dominate. More owners require a different reviewed partitioning design, and changing partition count can affect key routing.

2. Can timestamps merge two partition histories into a guaranteed business sequence?