Apache Kafka / Basic: records, partitions and reader progress / Lesson 4

Consumer groups and polling

Use independent groups and bounded polling while keeping assignment and processing responsibilities clear.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Explain how group identity changes delivery and progress.
Design a poll loop that accounts for slow work and reassignment.

A group is one logical reader

Traditional consumers sharing a group identifier cooperate to consume topic partitions. A second group reads the same retained history independently, with its own assignment and committed positions. This makes an audit projection and a dispatch projection independent without producing each event twice. Accidentally sharing their group identifier instead splits partitions between incompatible applications. Consumer instances are temporary workers; the group identifier represents a longer-lived processing purpose. Choose it deliberately and include environment boundaries so a development reader cannot unintentionally participate in a production group's assignment. Merely changing a process hostname does not create an independent logical reader.

Reason about bounded ownership

The local fixture assigns three partitions to two workers and verifies that each partition has one owner. It represents a possible outcome, not the implementation of a Kafka assignor or a promise of equal work. Partition zero might receive far more records than partitions one and two combined. When membership changes, assignment changes and a worker must stop treating revoked partitions as its responsibility. The classic and newer consumer rebalance protocols expose different configuration and coordination details. This course describes traditional group semantics and labels protocol-sensitive properties rather than assuming a legacy heartbeat setting controls every group mode.

python Offline ownership invariant

assignment = {'worker-a': [0, 2], 'worker-b': [1]}
owned = [p for values in assignment.values() for p in values]
assert sorted(owned) == [0, 1, 2]
assert len(owned) == len(set(owned))
print(len(owned))

Expected output

This checks only one supplied assignment, without running group coordination.

Polling is part of the liveness contract

A consumer loop must return to poll within the applicable processing interval. A large batch followed by slow synchronous processing can exceed max.poll.interval.ms even if the process still exists. Increasing a timeout hides the symptom only when the work has a justified finite upper bound. Reduce records per poll, shorten work, pause assigned partitions under the client rules, or use a carefully bounded worker design that keeps consumer API ownership correct. KafkaConsumer is not generally thread-safe; placing calls from arbitrary executor threads is not a valid shortcut. Polling, processing and committing must share a coherent ownership model.

Read a processing-limit fragment

The properties choose a named projection and cap the number of records returned by one poll. max.poll.records limits returned records rather than guaranteeing a fixed amount of broker fetch memory or a fixed processing duration. A hundred tiny validation operations and a hundred slow remote calls have very different costs. Disabling automatic commits gives the application responsibility for safe progress recording; it does not automatically implement it. Document the worst plausible processing path and what happens when ownership is revoked while work remains. The fragment supplies no endpoint and is never executed against a broker.

properties Illustrative consumer processing limits

group.id=lab-parcel-projection-v1
enable.auto.commit=false
max.poll.records=100

Manual commits and a poll cap require application logic; they do not prove correct acknowledgment.

Prepare for reassignment without improvising

A rebalance is an expected coordination event when members join, leave or fail under group rules. Before losing a partition, account for completed work and avoid committing beyond unfinished records. Cancellation may need cooperation from downstream work; a revoked assignment does not magically cancel an HTTP request already issued. For external effects, use event identity and a destination-side duplicate policy. Observe rebalance frequency alongside processing duration and membership churn. Adding consumers can cause temporary disruption and does not solve a single hot key by itself. Treat stable ownership and bounded work as part of application correctness, not merely throughput tuning.

Common errors and fixes

Two applications each miss some partitions.

Why it happens: They accidentally share one group identifier.
How to fix it: Give distinct logical projections distinct group identities after reviewing replay behavior.
Prevent it next time: Keep group identity in the application contract.

A busy consumer repeatedly loses assignment.

Why it happens: Processing delays exceed the applicable poll interval.
How to fix it: Bound work per poll and review the chosen protocol settings.
Prevent it next time: Measure processing duration and rebalance frequency together.

Concurrent threads race on one KafkaConsumer.

Why it happens: Worker parallelism ignored client thread-safety rules.
How to fix it: Keep consumer API calls on their designated owner and coordinate worker results.
Prevent it next time: Review cancellation, commits and reassignment as one design.

Check your understanding

1. Audit and dispatch must each receive every retained parcel event. Should they share a group?

Show worked solution

Use different group identities because they are independent logical readers. Instances implementing audit can share the audit group to divide its partitions, and dispatch instances can share another. Confirm starting-offset policy before first use, because an independent group does not itself guarantee access to history already removed by retention.

2. Why might max.poll.records=100 still permit a processing timeout?