Course outline

ASP.NET Core / Advanced: Verification and Reliable Operations / Lesson 17

Reliability, Effects, and Retries

Define deadlines, idempotency, commit boundaries, durable work, and recovery before treating retries as a reliability feature.

About 5 min readContent reviewed
On this page

Lesson goals

  • Separate transient failure from uncertain committed outcome.
  • Design bounded retry and idempotency policy.
  • Connect request, persistence, messaging, and shutdown contracts.

A timeout does not prove that an effect failed

A client or server can lose a response after an operation commits. Retrying an unsafe create/update can duplicate a business effect even when the transport reports failure. Define operation identity, permitted retries, and how clients discover status or reconcile an unknown outcome. An HTTP status and a database commit are different boundaries.

Not every failure is transient. Invalid input, forbidden access, concurrency conflict, overload, downstream timeout, and internal defect need different treatment. A blanket retry around an entire handler can repeat validation side effects or external calls and worsen an outage. Choose the retried unit deliberately.

An idempotency key needs scope and payload meaning

The first complete program models sequential synthetic effects keyed by actor and operation key. Reusing the same key/payload returns a replay outcome; reusing it with a different amount returns conflict. It checks a small domain rule only. The in-process dictionary is not durable, distributed, thread-safe transaction storage and is not a production exactly-once implementation.

A real design needs atomic creation/effect recording, retention/expiry, caller scope, payload identity, replay response, and failure recovery. Authorization must still apply; an idempotency header is not proof of ownership. Preserve the result of an already committed operation rather than pretending cancellation rolled it back.

csharp Check a sequential synthetic idempotency rule
using System;
using System.Collections.Generic;

var ledger = new Dictionary<(string Actor, string Key), int>();
int total = 0;
int first = Apply("actor-a", "operation-7", 10);
int replay = Apply("actor-a", "operation-7", 10);
int conflict = Apply("actor-a", "operation-7", 15);
Console.WriteLine($"first={first} replay={replay} conflict={conflict} total={total}");

int Apply(string actor, string key, int amount)
{
    if (ledger.TryGetValue((actor, key), out int existing)) return existing == amount ? 200 : 409;
    ledger[(actor, key)] = amount;
    total += amount;
    return 201;
}
Expected output
first=201 replay=200 conflict=409 total=10

The first effect adds ten, replay adds nothing, and changed payload conflicts; no persistence or concurrency guarantee is tested.

Retries consume the same deadline and capacity budget

Choose a bounded attempt count, backoff/jitter, per-attempt timeout, total deadline, and cancellation behavior appropriate to the operation. A retry can amplify load on an unhealthy dependency. Circuit breaking, concurrency limits, and fallback have different purposes and require an explicit correctness contract. A fallback returning misleading success can be worse than an honest failure.

The second program calculates a synthetic worst-case sequential attempt budget. It does not configure an HTTP resilience package or time an actual request. Framework/client-library resilience integrations have versioned packages and policies; untested registration fragments must not be reported as verified runtime behavior.

csharp Calculate an invented sequential retry envelope
using System;

int attempts = 3, attemptMilliseconds = 1000, backoffMilliseconds = 500, budgetMilliseconds = 3000;
int modeled = checked(attempts * attemptMilliseconds + (attempts - 1) * backoffMilliseconds);
Console.WriteLine($"worstMs={modeled} budgetMs={budgetMilliseconds} fits={modeled <= budgetMilliseconds}");
Expected output
worstMs=4000 budgetMs=3000 fits=False

Three attempts plus two backoffs require 4000ms in the model, exceeding 3000ms; no operation waits or retries.

A request and a message do not share one transaction automatically

If an operation updates data and publishes a message, a failure between them can leave an inconsistent state. An outbox/inbox or another justified protocol can connect durable records with eventual delivery and idempotent consumption. The database/provider and broker semantics must support the intended boundaries; naming a class Outbox does not provide them.

BackgroundService and an in-memory channel can coordinate process work but do not inherently survive restart. Define accepted, persisted, running, completed, and failed states and what 202 means. A durable-job claim needs the intended storage/queue system and recovery tests, which are outside this course’s local examples.

Reliability includes shutdown and data recovery

Plan stop-admission, cancellation, task joining, draining, dependency disposal, and failure reporting under a bounded shutdown policy. Forced termination can still interrupt work, so correctness must not depend on every finally running before a machine disappears. Persist or reconcile required state at its proper commit boundary.

Define recovery objectives, restore evidence, schema/artifact compatibility, and user-facing validation. A health endpoint or passing handler unit test does not prove backup restoration or regional recovery. Record assumptions and tested scope. This lesson uses only local finite models and starts no server, durable queue, database, or external dependency.

Common errors and fixes

A timeout triggers an unconditional repeat of a committed action.

Why it happens
Unknown outcome was treated as known failure.
How to fix it
Use operation identity and a documented reconciliation/idempotency policy.
Prevent it next time
Test response-loss and duplicate-attempt cases.

Retries exceed the request deadline and overload the dependency.

Why it happens
Attempt, backoff, and total budgets are designed independently.
How to fix it
Bound the whole operation and admission under cancellation.
Prevent it next time
Calculate and test the worst-case policy envelope.

An in-memory worker is called durable delivery.

Why it happens
Process lifetime and persisted work were conflated.
How to fix it
Use the required durable protocol and test recovery separately.
Prevent it next time
Describe 202 and restart behavior honestly.

Check your understanding

1. A POST changes data, but the caller times out before receiving the result. Why is retrying only safe under an explicit contract?

Show worked solution

The first attempt may have committed. A retry needs a scoped operation key, payload/result identity, durable atomic recording, authorization, and retention/replay policy appropriate to the domain. Transport failure alone does not establish database rollback. Test crashes and lost responses around the commit/response boundary with the intended provider/queue when validating a real implementation.

2. What is missing from a retry arithmetic model that reports fits=True?

Show worked solution

It does not measure actual latency distribution, cancellation cooperation, downstream concurrency, jitter, resource exhaustion, or side-effect correctness. The model only compares declared durations. Use it to expose inconsistent budgets, then validate the supported runtime policy and operation semantics in an authorized isolated setup. A timing calculation is not a resilience guarantee.

Key takeaways

  • Unknown outcomes, retries, and committed effects need an explicit protocol.
  • Bound attempts, total time, queues, and dependencies together.
  • Durability and recovery require evidence beyond process-local tasks and health signals.

Official references