Apache Kafka / Intermediate: contracts, failures and observability / Lesson 9

Replication, ISR and write availability

Evaluate acknowledged writes against replica health and make durability-availability tradeoffs explicit.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Explain leaders, followers and the in-sync replica set.
Reason about acks=all with min.insync.replicas.

Replication protects a declared failure model

Each partition has a leader and configured replicas. Followers replicate the leader's log, and the in-sync replica set reflects replicas considered sufficiently caught up under Kafka's rules. Replication factor describes intended copies, while ISR describes a changing health-related set; they are not interchangeable counts. Placing all replicas in one failure domain leaves correlated loss exposure even with a large factor. Controller quorum health is also separate from partition replica health. Before choosing a durability claim, specify whether the design considers one broker loss, one rack loss, prolonged disconnection or complete loss of the original cluster.

A minimum rejects some degraded writes

With acks=all, min.insync.replicas can require a minimum ISR size for successful writes. A replication factor of three and minimum of two permits continued writing with two suitable in-sync replicas but rejects writes when only one remains under the applicable rules. This is an availability tradeoff chosen to preserve a stronger acknowledged-copy condition. It is not a guarantee against every timing and storage failure. Lowering the minimum during an incident changes the contract and should not be disguised as routine tuning. Diagnose why replicas fell behind before deciding whether any degraded operation is acceptable.

Model the minimum without claiming a cluster test

The Boolean example compares a synthetic ISR count with a declared minimum. It shows that two meets the threshold and one does not. This is only the threshold portion of a write decision: it excludes leadership changes, request timing, disk errors and replica acknowledgment behavior. A real client may see a retriable error, a timeout or an uncertain outcome depending on when the failure occurs. Distinguish a write rejected before append from a response lost after acceptance. Recovery of the business request still needs stable event identity rather than blindly counting every timeout as a lost record.

python Offline minimum-ISR threshold

minimum = 2
def meets_minimum(isr_count):
    return isr_count >= minimum
assert meets_minimum(2)
assert not meets_minimum(1)
print(meets_minimum(2), meets_minimum(1))

Expected output

True False

The model checks a number only and does not emulate append or acknowledgment timing.

Keep producer and topic settings linked

The fragment names min.insync.replicas=2 as a topic-level choice for a separately declared three-replica training design. It does not create the topic, configure placement or verify the actual ISR. A producer using acks=1 has a different acknowledgment contract, so merely setting the topic minimum is not a substitute for reviewing producer behavior. Also distinguish unclean leader election from normal leadership changes among suitable replicas. Selecting a stale replica can restore service while exposing acknowledged-data loss; the exact feature and eligibility behavior must be reviewed against the deployed version and configured policies.

properties Illustrative topic property

min.insync.replicas=2

Assume three replicas and acks=all elsewhere; this line alone creates no durability guarantee.

Read replication evidence over time

Track under-replicated and under-minimum-ISR partitions together with offline partitions, request errors and disk or network symptoms. A cluster-wide average can hide one critical partition whose replicas are unhealthy. Correlate the first ISR shrink with deployment changes, storage pressure and follower fetch behavior; a later consumer lag rise may be a consequence rather than the cause. Do not restart every node or lower safeguards merely because a dashboard is red. Build a bounded incident timeline and preserve the original settings so a reviewed intervention can be evaluated against its intended failure model and reversed when appropriate.

Common errors and fixes

Three configured replicas are reported as three healthy copies.

Why it happens: Replication factor was substituted for current ISR evidence.
How to fix it: Inspect per-partition replica state and ISR history.
Prevent it next time: Expose intended and healthy replica counts separately.

A degraded incident is fixed by lowering the minimum without review.

Why it happens: Write availability was prioritized without recording changed loss exposure.
How to fix it: Assess the failure model and restore replica health through an authorized plan.
Prevent it next time: Treat safeguard changes as explicit contract changes.

One rack failure removes every replica.

Why it happens: Copies were not separated across relevant failure domains.
How to fix it: Review placement and independent recovery requirements.
Prevent it next time: Include correlated failures in design reviews.

Check your understanding

1. With three replicas, acks=all and minimum two, is writing with only one ISR member an intended success?

Show worked solution

No. The minimum is intended to reject that degraded write condition. The rejection preserves the chosen acknowledgment requirement at a cost to availability. A real failure response still depends on request timing, so client recovery must distinguish a clear rejection from an uncertain timeout and retain the original event identity.

2. Why is a healthy controller quorum insufficient evidence that every topic can serve reads?