Apache Airflow / Basic: definitions, intervals and safe task execution / Lesson 6

Retries, idempotency and failure states

Design tasks that can repeat safely and interpret orchestration states without confusing them with external outcomes.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Distinguish a retry attempt from a new business effect.
Explain success, skipped, upstream_failed and retry-related states.

A retry repeats task execution

Airflow can execute task logic again after failure, timeout or an interrupted worker. A retry policy therefore exposes any non-idempotent side effect inside that logic. Appending the same interval rows twice can duplicate a report even when the task eventually turns green. Prefer an operation that replaces the declared partition or enforces a stable uniqueness rule appropriate to the destination. An attempt number is execution evidence, not a fresh business identifier. Keep source inputs stable across retries so repeating the task computes the same intended result rather than silently switching to a newer latest dataset.

A timeout can leave an uncertain external outcome

If a remote system accepts an operation but its response is lost, the task can fail while the business effect already exists. Retrying with a new identifier can apply it twice. Destination-supported idempotency keys, atomic replacement or a reconciliation protocol are needed at that boundary. Airflow's metadata transaction does not make arbitrary external APIs atomic with task success. Likewise, marking a task successful manually does not create a missing report. Draw the sequence from input read through destination acknowledgment to task-state recording, then place a crash between each step to see which evidence remains available.

Read states as orchestration facts

success means the task execution completed without a reported failure under its operator contract. skipped indicates a control-flow decision rather than a successful calculation, and upstream_failed means a prerequisite failure prevented normal eligibility under the trigger rule. up_for_retry records a failure with retry capacity remaining; deferred and up_for_reschedule describe specific waiting mechanisms. These distinctions matter when a join or notification task decides what to do next. A green DAG run can depend on leaf-task states and chosen trigger rules, so inspect whether a permissive final task could hide a failed business stage instead of treating the run color as complete validation.

Model replacement instead of repeated append

The local dictionary writes the same business partition twice with the same complete result. It retains one partition entry, illustrating a replacement contract rather than a database implementation. The append comparison produces four rows from two repetitions of a two-row input. Neither model establishes atomic persistence, concurrency safety or a remote storage guarantee. The destination must provide the actual supported operation, and conflicting reruns may require version checks or locking appropriate to that system. Still, this small contrast makes the duplicate mechanism visible before a team blames the scheduler for effects caused by task logic.

python Offline repeated partition-replacement model

rows = [('P1', 200), ('P2', 300)]
partitions = {}
for _ in range(2):
    partitions['2026-10-01'] = list(rows)
assert len(partitions) == 1
assert len(partitions['2026-10-01']) == 2
print(len(partitions['2026-10-01']))

Expected output

One in-memory partition retains two rows; durable atomicity is not modeled.

python Offline duplicate-append comparison

rows = [('P1', 200), ('P2', 300)]
appended = []
for _ in range(2):
    appended.extend(rows)
assert len(appended) == 4
print(len(appended))

Expected output

Repeated append duplicates rows under this contract even though each individual append succeeds.

Retry only failures that can plausibly recover

A bounded policy names attempt count, delay and execution timeout, then classifies permanent invalid input separately from temporary dependency failure. More retries do not repair an unsupported schema or a missing authorization grant. Airflow provides documented exceptions for intentional skip or fail-without-further-retry behavior; use them only when the business meaning is explicit. Avoid swallowing every exception and returning an empty success result, because that erases failure evidence. This lesson executes only local state and replacement models. It clears no tasks, changes no run state and sends no external request or notification.

Common errors and fixes

A retry creates duplicate destination rows.

Why it happens: The task appends without a stable partition or uniqueness rule.
How to fix it: Use destination-supported replacement or idempotent application.
Prevent it next time: Test the same interval twice and inspect final business state.

A failed external call is assumed never to have happened.

Why it happens: Missing acknowledgment was treated as proof of nonexecution.
How to fix it: Reconcile with the original operation identity before creating a new effect.
Prevent it next time: Document uncertain-result handling for every side effect.

An exception is swallowed and an empty report is marked successful.

Why it happens: Error handling replaced failure with a valid-looking default.
How to fix it: Propagate or classify the failure and preserve diagnostic context.
Prevent it next time: Distinguish valid empty input from unavailable input.

Check your understanding

1. Why should an operation key usually remain stable across retries of one interval?

Show worked solution

It identifies one intended business result, allowing the destination to recognize replay rather than treat every attempt as new work. Attempt identity remains useful for logs but should not defeat deduplication. Include operation type and tenant or partition scope where required, and detect conflicting payloads under the same key.

2. Does skipped mean a report was calculated correctly and found empty?