Rising lag while consumers remain active

Understand the likely causes

• Incoming work may exceed sustainable completion, one partition may dominate, or frequent reassignment may interrupt work. A mismatched metric source can also exaggerate apparent backlog.

Confirm what the alert subtracts

Record whether the alert uses committed, fetched or another consumer position, and which end position it compares against. Capture both values and their timestamps for the same partition. An old commit sample paired with a fresh end sample can manufacture a spike. Transactional isolation can introduce a distinction between the physical log end and stable readable progress. Do not compare a dashboard labeled records behind with a business delay measured in minutes as if they had the same unit. Establish the affected group and environment before interpreting any trend.

Compare arrival and completion over one interval

If an aligned interval receives five hundred records and completes four hundred, backlog grows even though the reader is active. Inspect unique business completion separately when records can be filtered or duplicated. Use several representative intervals rather than extrapolating from a single quiet second. A large commit gap can also mean processing finished but progress recording failed, so correlate application completion with commit outcomes. This distinction matters because replay after a crash may repeat effects that already happened. Preserve original event identities while diagnosing the accounting mismatch.

Locate skew and the expensive operation

Compare partitions rather than only group totals. One partition with a constant accidental key suggests a routing defect; one with a legitimate large tenant suggests a workload-design question. Check processing duration and downstream waits for that partition before changing broker settings. Rebalance frequency and poll-interval violations can explain repeated abandoned batches. Additional group members cannot divide one traditional partition among simultaneous owners. A justified remedy may bound remote calls, reduce work per poll or revise an ordering-aware design, but the observed cause must choose the intervention.

Check whether the proposed capacity can catch up

Use a bounded arithmetic model to test the proposal before any authorized operational experiment. With one thousand pending records, arrivals of sixty per second and completions of eighty, net drain is twenty and the simplified catch-up estimate is fifty seconds. The result excludes skew, retries and changing rates. If completion is only fifty, no finite recovery follows while arrivals stay at sixty. This guide does not query a service or run a benchmark; the calculation makes the hidden steady-rate assumption explicit so the estimate can be challenged.

pending = 1000
arrivals = 60
completed = 80
net = completed - arrivals
assert net > 0
assert pending / net == 50
print(pending / net)

50.0

Keep it from recurring

• Keep metric definitions, processing budgets and key-distribution expectations with the projection design. Alert on sustained business freshness and repeated reassignment, and review changes using the same time windows and units.