Apache Kafka / Advanced: atomic processing and recovery design / Lesson 13

Transactions, isolation and fencing

Link Kafka output with consumed offsets while stating the exact boundary of atomicity.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Explain transactional identifiers, commit visibility and producer fencing.
Distinguish Kafka-to-Kafka atomicity from external effects.

One transaction can cover Kafka output and progress

A consume-transform-produce flow can write output records and the consumed group's next offsets in one Kafka transaction. A commit makes that Kafka work visible under the transaction protocol; an abort leaves it unavailable to read_committed consumers. This links output and progress so replay does not expose a half-completed Kafka transformation. It does not include an arbitrary email, HTTP call or database transaction. State this boundary every time exactly-once processing is discussed. External side effects still require their own destination protocol, even when the surrounding Kafka transaction is perfectly configured and the processor restarts safely.

Assign identities to logical producer slots

transactional.id identifies a transactional producer across restarts and enables fencing of an older producer incarnation. Active instances must not accidentally share one transactional identifier, while a replacement for the same logical slot needs a coherent identity strategy. Randomly changing the identifier on every start loses part of the intended restart relationship. The properties fragment shows the consumer and producer settings in separate labeled blocks because transactional.id belongs to the producer, not the consumer. Initialization, begin, send, offset attachment, commit and abort still require proper client API handling. The fragment is not an executable transaction program.

properties Illustrative separate consumer and producer settings

# Consumer properties
enable.auto.commit=false
isolation.level=read_committed
# Producer properties
transactional.id=lab-parcel-slot-0

The comments divide two client configurations; application transaction APIs are still required.

Read committed does not mean read every latest byte

A read_committed consumer excludes aborted transactional records and observes visibility constrained by ongoing transactions. An open transaction can hold back the last stable offset even while the physical log end advances. This can look like stalled reading when the actual issue is a long-lived or failing transactional producer. Nontransactional records have their own visibility under these rules; do not assume every record in the topic was transactionally produced. Monitor transaction duration and failure outcomes rather than treating isolation as a performance toggle. Lowering isolation to hide a delay changes what records downstream applications may observe.

Separate committed and aborted fixture records

The offline list labels three synthetic output records by transaction outcome and selects only committed outcomes. It illustrates the business expectation of excluding aborted output without implementing Kafka's log markers, last stable offset or coordinator protocol. The two committed records survive, while the aborted one does not enter the result. A complete client test would also exercise crashes during commit, transaction timeout and reassignment. Those runtime behaviors are deliberately untested here. Keep the model's claim limited to filtering the supplied labels so a reader does not mistake a Python list comprehension for transaction certification.

python Offline outcome-label model

records = [('tx-a', 'committed', 'P17'), ('tx-b', 'aborted', 'P18'), ('tx-c', 'committed', 'P19')]
visible = [value for _, outcome, value in records if outcome == 'committed']
assert visible == ['P17', 'P19']
print(visible)

Expected output

['P17', 'P19']

This filters declared outcomes and does not emulate a Kafka coordinator or isolation implementation.

Handle fencing as ownership loss

A fenced producer should not keep attempting writes as though it still owns the transactional slot. Separate fatal producer state from errors for which aborting the current transaction and retrying is supported. Follow the exact client API documentation; catching every exception and calling commit again is not a reliable recovery strategy. Bound transaction work so timeouts and rebalances are understandable. Kafka 4.0 introduced a newer transaction protocol with version-dependent behavior, so review client and broker compatibility together before operational adoption. This lesson changes no feature levels, starts no producer and claims no live transactional test.

Common errors and fixes

Two active producers repeatedly fence one another.

Why it happens: They share a transactional identifier intended for one logical slot.
How to fix it: Correct instance-to-slot identity assignment and stop the stale owner.
Prevent it next time: Test restart and concurrent deployment identity rules.

An external charge repeats despite Kafka transactions.

Why it happens: The charge was outside the Kafka atomic boundary.
How to fix it: Use the destination’s duplicate-safe protocol and original event identity.
Prevent it next time: State the exact systems included in every atomicity claim.

read_committed appears behind a growing log end.

Why it happens: An open transaction may constrain stable visibility.
How to fix it: Correlate transaction state and producer duration before changing isolation.
Prevent it next time: Monitor transaction duration and terminal outcomes.

Check your understanding

1. Can an email sent between beginTransaction and commitTransaction be rolled back by abortTransaction?

Show worked solution

No. Kafka does not control that external provider through the producer transaction. The email can remain sent even when Kafka output is aborted. Use an output intent consumed by a duplicate-safe delivery service or another reviewed protocol, and preserve business identity across retries and uncertain provider responses.

2. Why is a unique random transactional identifier at every restart not automatically a good identity policy?