Apache Kafka / Basic: records, partitions and reader progress / Lesson 3

Producers, batching and acknowledgments

Interpret producer completion and tune a bounded latency-throughput tradeoff without inventing durability guarantees.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Separate local enqueue, broker acknowledgment and downstream processing.
Explain batching, compression and delivery deadlines.

Follow the producer lifecycle

A producer serializes a record, selects its partition, buffers it and sends batches to partition leaders. An asynchronous send returning a future is not evidence that the broker has accepted the record; the completion result must be observed. Even a successful broker acknowledgment does not say a consumer has completed a business action. Make those milestones visible separately in application metrics and error handling. Reusing a producer usually lets its connections and batches work as intended; creating one for every record defeats that design. Client-library thread-safety and lifecycle rules still need review for the chosen implementation.

Read acknowledgment settings as a contract

The properties fragment requests acknowledgments from all current in-sync replicas and enables producer idempotence. It is intentionally incomplete: no endpoint, credentials or runnable Java application is supplied. The min.insync.replicas topic setting and replication arrangement determine which degraded situations reject a write. The word all refers to the current in-sync replica set, not every configured replica regardless of health. If that set shrinks and the minimum is too small, the practical failure tolerance also shrinks. Treat these properties as part of a jointly reviewed producer-topic contract rather than copying a magic durable configuration.

properties Illustrative Java producer properties

acks=all
enable.idempotence=true
linger.ms=5
compression.type=lz4

This fragment needs a compatible client and a separately reviewed broker/topic contract; it was not run.

Budget waiting and compression together

A batch groups records for the same partition so network and storage overhead is shared. batch.size bounds an allocation target, while linger.ms allows waiting for additional records; neither means every batch reaches that size. Compression works on batches and depends on record similarity and codec cost. Kafka 4.0 changed the default linger.ms from zero to five milliseconds, so older intuition about defaults can be wrong for this baseline. Choose a latency budget first and measure representative distributions in an authorized environment. Average throughput alone can conceal occasional long delivery delays or a skewed hot partition.

Reason about a delivery budget

The offline calculation allocates a synthetic forty-millisecond application budget between batching wait, network-and-broker allowance and remaining slack. It does not predict real broker latency. Its purpose is to expose an impossible budget before tuning: an extra batching delay consumes the same finite allowance as retries and scheduling. delivery.timeout.ms limits the overall time a record can spend awaiting a success or failure result, while other settings constrain individual requests or metadata-related blocking. Check documented constraints among settings. A timeout can leave the business outcome uncertain, so application recovery must use event identity rather than assuming nothing was written.

python Offline latency-budget arithmetic

budget_ms = 40
batching_ms = 5
network_broker_ms = 20
slack_ms = budget_ms - batching_ms - network_broker_ms
assert slack_ms == 15
print(slack_ms)

Expected output

Fifteen milliseconds remain under these assumptions; this is not a latency measurement.

Observe asynchronous failure and close deliberately

A callback or future handler should capture the full outcome without blocking the producer's progress with slow unrelated work. Decide whether a terminal send failure rejects the originating request, places the original event in an approved durable retry path, or raises an operational signal. Logging and discarding an error silently changes the delivery contract. A flush waits for buffered sends under client rules; it does not repair previously ignored failures or confirm external processing. Controlled shutdown needs a finite deadline and an accounting of unfinished sends. No real producer is created or benchmarked in these examples.

Common errors and fixes

send returns and the application reports delivery success immediately.

Why it happens: Local enqueue was confused with completion.
How to fix it: Observe the future or callback and propagate terminal failure.
Prevent it next time: Track enqueue and acknowledgment as separate milestones.

acks=all is called protection against losing every replica.

Why it happens: Acknowledgment scope was confused with independent recovery.
How to fix it: Review replica placement, in-sync minimums and recovery copies.
Prevent it next time: State the actual failure scenarios the design tolerates.

Larger batches breach the request latency objective.

Why it happens: Throughput tuning omitted queueing and retry budgets.
How to fix it: Measure end-to-end latency and choose bounded batching settings.
Prevent it next time: Review tail latency as well as average throughput.

Check your understanding

1. Does a producer acknowledgment prove the parcel dashboard has updated?

Show worked solution

No. It concerns acceptance under the producer and broker contract. The dashboard has its own consumer position, processing time and destination state. If the user needs read-after-write confirmation, design a separate observable business acknowledgment or status query instead of treating the send future as dashboard completion.

2. The forty-millisecond model changes batching wait from five to twenty milliseconds. What slack remains?