Apache Kafka / Advanced: atomic processing and recovery design / Lesson 18

Recovery, upgrades and cross-cluster replay

Build a recovery plan that distinguishes retained data, metadata compatibility and destination correctness.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Explain recovery point and recovery time with measurable evidence.
Review version transitions and cross-cluster identity mappings.

Define what must be recovered

A recovery point objective states acceptable missing business history, while a recovery time objective states how long the service can remain below its required behavior. A replicated live cluster is not automatically an independent backup against operator mistakes, logical corruption or complete domain loss. Identify source topics, configurations, schemas, application identities and destination state needed to restore the service. Restoring record bytes without compatible readers or credentials may leave the application unavailable. Conversely, a healthy process with skipped business events is not a successful recovery. Define a small observable acceptance scenario before choosing the mechanism.

Software version and finalized features are different

Kafka upgrades require a version-specific plan covering broker software, client compatibility and metadata or feature levels. Rolling out binaries and finalizing a feature version can have different rollback implications. Kafka 4.x removed ZooKeeper mode, so legacy clusters require a supported migration before entering that family. The 4.1 documentation also describes dynamic controller membership and its feature boundary; do not copy those instructions into an older quorum. This course selects 4.1 for teaching rather than claiming it is the newest or currently supported release. Review the exact official upgrade path and service restrictions before any separately authorized operation.

Coordinates change when records are copied

A record copied into another cluster can receive a different destination offset and topic name. Its source topic-partition-offset is therefore not automatically a valid destination seek coordinate. Cross-cluster tools can provide offset translation under their supported policies, but it must be verified for the exact topology and failover plan. Preserve business event identity so duplicates across original and mirrored paths remain recognizable. Looping replication paths and simultaneous writers can introduce additional identity and ordering problems. A secondary cluster being reachable does not establish complete data coverage or that consumers can resume at the correct business boundary.

Make an explicit source-to-destination map

The fixture maps one declared source coordinate to one different destination coordinate and retains the event identifier. It proves only that application representation can keep these identities separate; it does not implement MirrorMaker offset translation or certify failover. A second arithmetic example calculates a recovery interval from a detection, restoration and verification budget. Verification is part of recovery time, not optional work after success has already been declared. Include replay drain and destination reconciliation when they are required to restore the user-visible service. A plan that stops when brokers start may substantially understate the real outage.

python Offline cross-cluster coordinate fixture

record = {'event_id': 'evt-101', 'source': ('parcels', 0, 17), 'destination': ('mirror.parcels', 0, 24)}
assert record['source'] != record['destination']
assert record['event_id'] == 'evt-101'
print(record['source'][2], record['destination'][2])

Expected output

17 24

The map is supplied synthetic data, not a replication or offset-translation test.

python Offline recovery-budget arithmetic

detect_min = 2
restore_min = 8
verify_min = 3
recovery_min = detect_min + restore_min + verify_min
assert recovery_min == 13
print(recovery_min)

Expected output

The declared recovery budget includes verification and does not predict a real outage.

Verify coverage and behavior before declaring success

A recovery review should compare unique business events, expected projections, retained ranges and explicitly accepted gaps. Include an event before the incident boundary, one at the boundary and one produced after recovery. Confirm duplicate-safe handling when original traffic resumes, then check the documented rollback or cutback conditions. Keep evidence about versions and feature levels with the result. None of these procedures is executed here: no broker restart, storage format, migration, mirror job or administrative command is run. Only the isolated coordinate and timing calculations are tested, leaving live recovery readiness for an independently authorized environment.

Common errors and fixes

Source offsets are used directly in a copied destination topic.

Why it happens: Coordinates were assumed stable across clusters.
How to fix it: Use a verified supported translation and reconcile business coverage.
Prevent it next time: Keep event identity separate from physical log positions.

An upgrade rollback is assumed possible after every feature change.

Why it happens: Binary rollout and finalized feature boundaries were conflated.
How to fix it: Review the exact version-specific rollback constraints before finalization.
Prevent it next time: Record software, metadata and feature levels separately.

Recovery is declared before destination reconciliation.

Why it happens: Process availability replaced business acceptance.
How to fix it: Verify coverage, duplicate handling and expected projections.
Prevent it next time: Include acceptance verification in the recovery time budget.

Check your understanding

1. Does a second live cluster automatically provide a zero-loss recovery point?

Show worked solution

No. Replication delay, excluded topics, source failures and translation policy can leave gaps. Measure the supported mirrored coverage and reconcile unique events at the intended cutover boundary. The recovery objective must state acceptable missing history and evidence rather than deriving a zero-loss claim from the existence of another endpoint.

2. Why should a recovery exercise retain its exact feature-level evidence?