CI/CD / Integrity, coordination, and improvement / Lesson 15

Concurrency, stale releases, and idempotent retries

Prevent overlapping adoption and reason about an operation whose completion is uncertain.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Separate mutual exclusion from release ordering.
Model duplicate and stale requests without real side effects.

A lock does not establish freshness

Two Parcel candidates can finish testing in a different order from their source revisions. Candidate r9 might reach the environment first, followed by a slower r8 job. Serializing adoption prevents overlap but can still install the older candidate last. Define both mutual exclusion and ordering or freshness policy. A resource lock answers who may act now; it does not decide whether the waiting request remains appropriate. Use a target-specific coordination boundary and avoid a global lock that unnecessarily blocks unrelated environments.

Choose queue behavior intentionally

Some systems keep every waiting release; others replace pending work or prioritize newer candidates. Neither policy is universally right. A migration requiring every transition differs from a stateless service where intermediate candidates can be skipped. Platform concurrency and resource-group behavior also differ, so do not assume a lock provides FIFO ordering. Document which candidates may be superseded and how cancellation interacts with external work already accepted. The illustrative sequence below shows why serialization alone can still regress an environment.

text A stale-release race

r8 starts tests
r9 starts and finishes tests
r9 adopts under target lock
r8 later acquires lock -> must recheck freshness

This timeline is synthetic. The final check must consult an authoritative release policy or target state in a real system; source labels should not be sorted as strings to infer chronology.

Repeating a request needs a stable identity

An operation is idempotent under its declared contract when repeating it has the same intended effect as applying it once. A client retry after a timeout may occur after the target already accepted the first request. Use a stable operation identity and target-side deduplication where supported, then inspect the recorded outcome. A random new request identifier on every attempt defeats deduplication. Do not claim that a shell script is safe to repeat merely because it often exits zero twice.

Reject duplicates and older generations

The complete in-memory model below uses integer generations as a declared ordering mechanism. It records request identities, rejects an older candidate, and avoids repeating a successful request. This is intentionally simpler than distributed coordination: no database transaction, concurrent worker, fencing token, or crash recovery is implemented. A real controller must make acceptance and state changes atomic under its platform’s guarantees. The model teaches the needed invariants without pretending a Python dictionary solves a distributed race.

python A sequential request-state model

current = 9
seen = set()
def adopt(request_id, generation):
    global current
    if request_id in seen:
        return "duplicate"
    if generation < current:
        return "stale"
    current = generation
    seen.add(request_id)
    return "accepted"
print(adopt("request-a", 8))
print(adopt("request-b", 10))
print(adopt("request-b", 10))

Expected output

stale
accepted
duplicate

The output follows declared integer ordering and sequential calls. This program does not provide distributed locking or a production idempotency implementation.

Observe before repeating uncertain work

When a promotion times out, distinguish request rejection, accepted-but-running, completed-but-unreported, and unknown state. Read authoritative target evidence through the approved path before retrying. Canceling a CI job may stop the client without stopping an external rollout. Cleanup also needs ownership: one canceled job must not remove a resource now used by another request. State transitions, durable operation records, and narrowly scoped compensation make recovery explainable. A blind rerun can turn one ambiguous outcome into two conflicting changes.

Common errors and fixes

An older candidate overwrites a newer one despite a lock.

Why it happens: Mutual exclusion was mistaken for freshness policy.
How to fix it: Recheck eligibility against authoritative state after acquiring coordination.
Prevent it next time: Document ordering, supersession, and required intermediate transitions.

Every retry uses a new operation identifier.

Why it happens: The target cannot recognize repeated intent.
How to fix it: Retain a stable request identity and inspect its recorded result.
Prevent it next time: Design retry identity and deduplication together.

Canceling the CI job is assumed to cancel external adoption.

Why it happens: Client lifetime was confused with target-side operation lifetime.
How to fix it: Observe the target operation and use its supported cancellation process if authorized.
Prevent it next time: Define cancellation and cleanup semantics before release.

Check your understanding

1. A stateless release can skip intermediate candidates. Should every queued candidate still deploy?

Show worked solution

Not necessarily. A reviewed newest-eligible policy may supersede waiting candidates, provided required evidence and compatibility hold. Verify the platform’s actual pending and cancellation semantics. Do not apply that shortcut to migrations that require ordered transitions.

2. A target accepted request-b but the response was lost. What should a retry do?