Azure / Advanced: secure and recoverable architecture / Lesson 13

Messaging, delivery, and durable business identity

Choose Service Bus, Event Grid, or Event Hubs from the communication contract. Separate acceptance, delivery, processing, settlement, ordering, and idempotent effects.

About 4 min readContent reviewed 2026-10-25

On this page

Lesson goals

Distinguish commands, notifications, and event streams.
Reason about locks, retries, and duplicate effects.
Model stable operation identity without a broker.

A queue, event router, and stream are different contracts

Azure Service Bus supports brokered enterprise messaging with queues and topics under its documented delivery and settlement model. Azure Event Grid supports event-routing scenarios, while Azure Event Hubs is designed for high-throughput event streams and consumer patterns. Choose based on command ownership, fan-out, retention, replay, ordering, latency, and processing requirements rather than treating all three as interchangeable queues. Features vary by service option and client support. A notification that a document changed differs from a command to charge an account, and a retained telemetry stream differs from a single work item requiring an acknowledgement.

State the sender and consumer obligations

The model distinguishes a durable work command, a change notification, and a replayable event stream. It records why a candidate service might fit without creating namespaces, topics, subscriptions, or consumers. The real design still needs message schema, identity, size limits, retention, dead-letter behavior, and access scope. Define whether ordering is needed per business key, session, partition, or across the entire workload; do not infer a global order from one local sequence. A producer’s accepted send means something different from successful processing and an external business effect. Observe each promised stage explicitly.

json A communication-contract decision model

{
  "model": "messaging-decision-only",
  "patterns": [
    {
      "need": "owned-work-command",
      "candidate": "Azure Service Bus"
    },
    {
      "need": "resource-change-notification",
      "candidate": "Azure Event Grid"
    },
    {
      "need": "partitioned-replayable-stream",
      "candidate": "Azure Event Hubs"
    }
  ]
}

Expected output

candidate services require separate delivery, retention, and ordering verification

The model neither sends a message nor validates a service tier, SDK, or delivery guarantee.

Processing can finish before acknowledgement is recorded

With Service Bus receive-lock processing, settlement and lock lifetime are part of the delivery contract. A handler can commit its effect and fail before completion is recorded, allowing another attempt. Lock renewal and timeouts reduce some risks but do not make an arbitrary remote effect exactly once. Duplicate detection at a broker, where enabled and supported, also has a defined scope and window and is not a substitute for application idempotency. Keep operation identity stable across retries. A new random identity on every attempt defeats deduplication even when the payload is otherwise identical.

An in-memory receipt is an explanation, not a durable guarantee

The local model accepts two synthetic business identities and skips a repeated first identity, producing fifteen units rather than twenty-five. It demonstrates desired arithmetic only. In production, a receipt and local state change need durable coordination, often involving a uniqueness constraint and transaction. A process-local set cannot coordinate multiple workers or survive a restart. External side effects need a provider idempotency or delivery protocol, and an outbox can help coordinate publication with committed local data where the design requires it. Replay must preserve the original identity and relevant authorization context rather than creating a new business operation.

python Offline repeated-effect arithmetic

events = [{"id": "evt-a", "units": 10}, {"id": "evt-a", "units": 10}, {"id": "evt-b", "units": 5}]
seen = set()
total = 0
for event in events:
    if event["id"] not in seen:
        total += event["units"]
        seen.add(event["id"])
print(total, len(seen))

Expected output

15 2

The set is ephemeral and single-process. No broker, queue lock, durable transaction, or external effect is tested.

Failure handling includes poison data and safe replay

Define bounded attempts, backoff, dead-letter handling, operator ownership, and the conditions under which replay is permitted. A dead-letter queue needs investigation and retention rules, not an assumption that moving messages back always fixes them. Protect payload privacy and correlate accepted work with completed outcomes. Monitor queue delay and failures alongside HTTP status. This course runs no messaging SDK, namespace, send, receive, lock, replay, or administrative operation. Future authorized testing should include duplicate delivery, failure before and after the effect, lock expiry, malformed input, and a missing target under the actual chosen service contract.

Common errors and fixes

A successful send is reported as completed business work

Why it happens: Broker acceptance and consumer effects are conflated.
How to fix it: Track accepted, processed, and effect-completed outcomes separately.
Prevent it next time: Preserve stable operation identity across those stages.

A process-local set is called distributed idempotency

Why it happens: Durability and multiple consumers are omitted.
How to fix it: Design durable receipt/effect coordination and remote protocols.
Prevent it next time: Test restart and concurrent attempts in a future controlled fixture.

Dead-letter replay generates new business identities

Why it happens: Replay is treated as a new operation instead of recovery.
How to fix it: Preserve identity and review replay authorization and side effects.
Prevent it next time: Document replay ownership and valid recovery conditions.

Check your understanding

1. Why can retrying after a lost acknowledgement duplicate a charge?

Show worked solution

The first attempt may already have committed the external effect even though completion was not recorded. Preserve a stable business identity and use the provider’s appropriate idempotency protocol plus durable local coordination. A broker lock or retry counter alone cannot make that remote operation exactly once.

2. What remains untested by the fifteen-unit local model?