Jenkins / Persistent state and controlled delivery / Lesson 14

Durability, timeout scope, and retry safety

Reason about interruption and repeatability without treating resumption as a universal transaction.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Distinguish Pipeline persistence from agent and external-operation state.
Calculate bounded retry time and review timeout placement.

Persistence supports a particular recovery promise

Pipeline durability settings trade persistence work against resilience characteristics under documented modes and infrastructure assumptions. They do not make every plugin step resumable or reconstruct a lost workspace automatically. Parcel needs to know which state belongs to the controller continuation, agent process, temporary files, artifact storage, and external target. A recovered orchestration record can coexist with an incomplete or already completed outside action. Review durability as a service recovery decision with the owner rather than adopting a universal performance setting from another installation.

Timeout placement changes what is bounded

A timeout around retry bounds the whole group of attempts; a timeout inside retry can give each attempt a fresh budget. Declarative top-level and stage-level timeout options also differ in relation to agent allocation under documented behavior. Queue and provisioning delay can therefore matter differently. State the intended budget explicitly and leave room for diagnostics and cleanup. A timeout interrupts Pipeline execution according to its contract, but it is not proof that an external request was never accepted or that every child operation was reversed.

Show a bounded orchestration shape

The fragment below illustrates an outer timeout and two attempts around replaceable log output. It does not test a service, simulate an actual failure, or justify retrying arbitrary commands. A real retry body must be safe to repeat and its error class must be understood. Pipeline retry conditions can target certain infrastructure or nonresumable failures where supported, but those require the installed step implementation and a reviewed scope. Never hide a deterministic test defect by rerunning until one attempt passes.

groovy A bounded retry-shape illustration

timeout(time: 2, unit: "MINUTES") {
    retry(2) {
        echo "Illustrative replaceable observation"
    }
}

No waiting, failure, timeout, or Jenkins step occurs here. The body has no side effect beyond a hypothetical log line and is not a template for retrying deployment or database work.

Calculate declared attempt and wait costs

The complete local model computes three bounded attempts plus two declared backoff waits. It performs no sleep or retry itself. The result excludes queueing, agent startup, transfer, and cancellation latency, so it is not an end-to-end recovery-time guarantee. A production budget should reflect actual step behavior and timeout nesting. Retain every attempt outcome and reason. A final successful attempt is relevant evidence, but repeated earlier failures may reveal a race, unstable dependency, resource issue, or test defect that still needs an owner.

python A synthetic retry-budget calculation

attempts = 3
limit_per_attempt = 12
backoffs = [2, 4]
assert len(backoffs) == attempts - 1
print(attempts * limit_per_attempt + sum(backoffs))
print("arithmetic only")

Expected output

42
arithmetic only

Three twelve-second attempt limits and six seconds of waits total forty-two seconds in the declared model. No actual timing, agent, or service is measured.

Observe uncertain completion before repeating

When a request times out after submission, classify absent, running, completed, failed, or unknown state through the target owner’s approved observation path. Reuse operation identity under the target’s deduplication contract when repeating the same intent. Do not repeat adoption because report publication failed afterward. A lost agent may require restoring inputs and redoing safe computation, while authoritative data changes need their own recovery process. Local models can rehearse these decisions; only a separately authorized environment exercise can establish actual restart, plugin, and target behavior.

Common errors and fixes

Pipeline resumption is assumed to restore every agent file.

Why it happens: Controller continuation and workspace lifecycles were conflated.
How to fix it: Retain required inputs and outputs through explicit storage and recovery contracts.
Prevent it next time: Document ownership and persistence for each state layer.

Retrying a whole stage duplicates a completed external action.

Why it happens: The retry unit includes side effects with uncertain completion.
How to fix it: Observe the original request and resume only the safe incomplete phase.
Prevent it next time: Define request identity and repeatability before adding retries.

A per-attempt timeout is mistaken for one total deadline.

Why it happens: Timeout nesting was not considered.
How to fix it: Place and calculate the budget according to intended scope.
Prevent it next time: Review allocation, attempts, waits, and cleanup together.

Check your understanding

1. An outer two-minute timeout wraps multiple retries. Does every attempt receive two minutes?

Show worked solution

No. The outer scope bounds the combined body according to the step’s interruption semantics. If each attempt needs a separate limit, that is another nesting contract with another total budget. State which promise the job intends.

2. A resumed run finds the target already adopted the candidate but the report is missing. What should be repeated?