Apache Kafka / Intermediate: contracts, failures and observability / Lesson 12

Lag, throughput and operational evidence

Interpret consumer lag alongside rates, partition skew and business freshness.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Distinguish offset distance from elapsed business delay.
Build a bounded diagnosis from correlated metrics.

Lag measures one kind of distance

A common lag calculation subtracts a group's committed position from a partition's available end position. It is useful evidence of backlog, but its interpretation depends on the metric source, isolation level and whether it uses committed or fetched position. Offset distance is not automatically a count of pending business actions because gaps, filtering and transactional records can affect the relationship. Nor is it a duration: ten thousand records may represent seconds of recent activity or hours of old traffic. Name the exact lag definition on the dashboard and keep a separate business-freshness measure where user-visible delay matters.

Estimate drain time with explicit assumptions

The synthetic model has an arrival rate of eighty records per second, a processing rate of one hundred and a backlog of two hundred. With constant rates and no other constraints, net drain is twenty per second and catch-up takes ten seconds. If arrival exceeds processing, dividing backlog by processing rate misleadingly predicts recovery while the queue grows. The model assumes comparable record work and no partition bottleneck, retries or destination saturation. Real recovery estimates need rate distributions and sustained capacity evidence in a separately authorized environment, not a single arithmetic result presented as an operational guarantee.

python Offline constant-rate catch-up model

backlog = 200
arrival_per_s = 80
processed_per_s = 100
net_drain = processed_per_s - arrival_per_s
assert net_drain > 0
assert backlog / net_drain == 10
print(backlog / net_drain)

Expected output

10.0

Ten seconds follows only from the declared constant-rate assumptions.

An aggregate can hide the critical partition

One hot partition can accumulate lag while the other partitions remain nearly idle. More consumers in the same traditional group cannot split that partition into several active owners. Inspect per-partition input and completion rates, processing duration, key distribution and destination behavior. A hot key may reflect a legitimate large customer, a mistakenly constant key or a change in serialized routing. These causes require different fixes. Repartitioning affects ordering and migration, so do not prescribe a larger partition count before confirming the business sequence and whether the bottleneck is actually in the broker or downstream work.

Compare measurements on the same interval

The second fixture compares two committed-position snapshots and their corresponding partition ends. The committed position advances by thirty while the end advances by fifty, so lag rises by twenty even though the consumer is making progress. A rising lag line is therefore not proof the process is stopped. Keep timestamps and sampling windows aligned; comparing a fresh end offset with an old commit snapshot produces misleading spikes. Metrics can also reset on process restart. Preserve deployment markers and metric-source identity so a discontinuity is not interpreted as sudden miraculous catch-up or a new data-loss incident.

python Aligned synthetic lag snapshots

before = {'end': 100, 'committed': 60}
after = {'end': 150, 'committed': 90}
lag_before = before['end'] - before['committed']
lag_after = after['end'] - after['committed']
assert (lag_before, lag_after) == (40, 60)
print(lag_after - lag_before)

Expected output

Lag rises by twenty while committed progress advances by thirty.

Connect signals to a user consequence

A useful alert states the affected projection, partition scope, duration and expected business consequence. Correlate lag with poll duration, rebalance events, send failures, under-replicated partitions and destination latency. Check whether input volume changed before attributing every rise to slower consumption. Choose an intervention that addresses the observed cause and compare the same signals afterward. Do not reset offsets simply to make a lag graph green; that can skip required work while hiding the symptom. These examples read only synthetic numbers and perform no monitoring query, administrative command, service restart or load test.

Common errors and fixes

A lag number is reported as seconds without conversion evidence.

Why it happens: Offset distance was confused with event age.
How to fix it: Measure freshness from an agreed clock and state the metric definition.
Prevent it next time: Label units and position source on dashboards.

More consumers do not drain one hot partition.

Why it happens: Partition ownership limits and key skew were ignored.
How to fix it: Diagnose per-partition work and review ordering-aware remedies.
Prevent it next time: Track skew alongside aggregate capacity.

An offset reset clears the alert but loses required work.

Why it happens: Progress was changed to improve a graph rather than recovery.
How to fix it: Restore a reviewed recovery plan and reconcile missing coverage.
Prevent it next time: Require a business justification for every replay or skip.

Check your understanding

1. Backlog is 200, arrival 110/s and processing 100/s. Is recovery time two seconds?

Show worked solution

No. Net backlog grows by ten per second under those assumptions, so there is no finite catch-up time while they continue. Dividing by processing alone ignores new arrivals. Reduce incoming work, increase sustainable effective completion capacity or revise the workload through a reviewed business decision.

2. Why can low committed lag coexist with a stale customer dashboard?