Software Architecture / Interactions: distribution, failure, and operation / Lesson 8

Design asynchronous work with explicit completion and pressure

Use queues to decouple timing while accounting for durability, duplicate delivery, backlog, and user-visible outcomes.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Distinguish accepted work from completed work and define acknowledgement boundaries.
Reason about backlog growth and bounded admission without a live broker.

A queue changes the conversation with the user

A synchronous call returns after a result is available within the request deadline. An asynchronous command can return before business completion, so the system needs a durable work identity and a way to report progress or final failure. Accepted must not be labeled confirmed unless the business definition truly permits it. For a workshop report export, a job record and later download may be suitable. For the last seat, delaying the allocation decision may require a pending state and a different user promise.

Treat acknowledgement as a designed boundary

A producer needs to know when the broker has durably accepted a message according to the selected broker configuration. A consumer must decide whether acknowledgement follows successful processing, a durable handoff, or another explicit event. A crash before acknowledgement can cause redelivery; a crash after an early acknowledgement can lose work. These windows differ by product and transaction model. Assume duplicate delivery is possible unless a precisely scoped stronger guarantee is documented and verified. Business effects still need deduplication or an idempotent state transition.

json An illustrative work envelope

{"message_id":"m-51","type":"ReportRequested","schema_version":1,"job_id":"job-9","requested_at":"2026-10-28T10:00:00Z"}

The envelope distinguishes delivery identity from the business job identity. Timestamp and schema version support interpretation, but they provide neither ordering nor durability by themselves.

A buffer cannot fix a permanent capacity deficit

When arrival rate exceeds sustainable completion rate, backlog grows even if the API responds quickly. Queue length alone is incomplete: oldest-message age and expected completion time can better express user impact. The offline arithmetic below uses fictional constant rates to expose this relationship. Real systems have bursts, retries, variable service times, and capacity limits, so this model is a planning aid rather than a forecast. Increasing worker concurrency may merely move contention to a database or external provider that cannot absorb more parallel requests.

python A bounded backlog calculation

arrival_per_minute = 12
completion_per_minute = 9
minutes = 5
backlog = max(0, (arrival_per_minute - completion_per_minute) * minutes)
print(f"backlog={backlog}")
print(f"net_growth_per_minute={arrival_per_minute - completion_per_minute}")

Expected output

backlog=15
net_growth_per_minute=3

The fictional empty queue accumulates fifteen items over five minutes, growing by three per minute. No broker, worker, load generator, or external service is started.

Bound admission, retries, and poison-message handling

Backpressure means reducing or refusing incoming work when downstream capacity cannot safely keep up. Define queue limits, expiration, tenant fairness, and an honest rejection response. Bounded retries need a terminal outcome and an owned investigation path for persistent failures. A dead-letter queue is storage for unresolved messages, not a repair policy; replaying everything blindly can repeat harmful effects or overload the same dependency. Record the original identity and failure reason, then require a compatible fix and controlled replay conditions.

Specify ordering only where the business needs it

Parallel consumers can complete messages out of arrival order. If a workshop update must follow its creation, consider per-workshop sequencing, version checks, or state transitions that reject stale events. Global ordering is often more expensive than the actual requirement. Explain how expired jobs, cancelled work, and user polling interact with final states. Verify duplicate, delayed, and out-of-order examples in an isolated implementation before claiming reliability. This lesson models messages and rates only; it does not establish guarantees for any particular broker or deployment.

Common errors and fixes

An accepted export is shown as completed.

Why it happens: Transport acceptance was confused with business completion.
How to fix it: Expose a durable job state and final result.
Prevent it next time: Define user-facing states before adding the queue.

More workers make the external provider fail faster.

Why it happens: Concurrency ignored the downstream capacity limit.
How to fix it: Bound concurrency and admission against measured dependency capacity.
Prevent it next time: Observe age, completion rate, and dependency saturation together.

A dead-letter queue grows without ownership.

Why it happens: Failure storage was mistaken for resolution.
How to fix it: Assign diagnosis, retention, correction, and replay responsibilities.
Prevent it next time: Test a persistent poison message and its terminal path.

Check your understanding

1. At 20 jobs per minute arriving and 15 completing, what happens after ten minutes from empty?

Show worked solution

Under the stated constant-rate model, fifty jobs accumulate. A quick enqueue response hides this delay rather than removing it. Investigate workload reduction, increased sustainable downstream capacity, or explicit admission limits. The arithmetic does not determine a safe worker count because processing time and dependency contention are unspecified.

2. A consumer crashes after sending mail but before acknowledgement. What design issue follows?