Apache Airflow / Intermediate: runtime work, control flow and resource budgets / Lesson 9

Sensors, deferral and timeouts

Wait for explicit external conditions without confusing waiting capacity with business completion.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Compare poke, reschedule and deferrable waiting models.
Distinguish execution timeout from the overall wait budget.

Wait for a precise readiness condition

A sensor checks whether a condition is satisfied, such as the arrival of a complete interval manifest. The condition must be stronger than an arbitrary file existing if a producer can still be writing it. Define how completion is signaled, which interval is expected and what a timeout means for the business process. A successful wait does not validate every row or guarantee the object remains available forever. Separate readiness, integrity and processing checks. Avoid a permanent sleep loop inside an ordinary Python task when a documented waiting operator can express the condition with bounded resource and timeout behavior.

Resource ownership depends on the waiting mode

A poke-mode sensor generally keeps its worker slot while repeatedly checking. Reschedule mode releases the worker between checks and returns later through the scheduler. Deferrable operators can suspend execution and have a triggerer monitor an asynchronous condition before the task resumes. These mechanisms are not interchangeable flags for every operator; the provider must support the selected behavior and the deployment must run the necessary component. Deferring does not move the heavy computation into the triggerer. Triggers must remain suitable for shared asynchronous execution rather than blocking it with long synchronous work or carrying large mutable business state.

Name each clock and retry relationship

execution_timeout bounds one task execution attempt under its operator contract. A sensor's timeout can bound the overall wait according to documented mode-specific behavior; retrying an individual failed check does not necessarily reset that total wait budget. Deferral has its own supported timeout handling and resume path. Review the specific operator rather than assuming all these fields are synonyms. Airflow 3.1 also introduces experimental Deadline Alerts as a replacement direction for the removed Airflow 2 SLA feature; an alerting deadline is not the same thing as terminating task execution. Keep notification, attempt timeout and data-readiness deadline conceptually separate.

Compare slot occupancy with elapsed waiting

The offline model imagines six readiness checks at ten-second spacing, each consuming one second of actual checking. A slot held across the whole minute uses a different capacity budget from six seconds of checking. This arithmetic is an illustration, not a scheduler simulation: exact reschedule timing, startup overhead and triggerer behavior are excluded. The intended lesson is to estimate both elapsed wait and occupied execution capacity. Releasing a worker does not make the source arrive sooner, and thousands of short rescheduled checks can still create scheduling pressure. Choose a reasonable check interval from the readiness requirement and platform evidence.

python Offline wait-versus-check occupancy

checks = 6
check_cost_s = 1
elapsed_wait_s = 60
active_check_s = checks * check_cost_s
assert active_check_s == 6
assert elapsed_wait_s > active_check_s
print(elapsed_wait_s, active_check_s)

Expected output

60 6

Sixty elapsed seconds differ from six check seconds; real scheduler overhead is not modeled.

Confirm support before selecting deferrable behavior

The configuration sketch states an intended wait policy without claiming that an arbitrary class accepts these fields. It is a design record to compare with the selected provider's documented constructor and deployment support. A deferred task that never resumes may indicate a triggerer problem, a failed trigger or a condition that never becomes true; a queued task suggests a different boundary. Preserve exact states and timestamps before changing settings. No sensor, triggerer or external readiness check runs here. The only executed example computes synthetic occupancy, leaving operator behavior for a separately authorized compatible Airflow environment.

json Synthetic waiting-policy design record

{"condition":"complete-manifest-for-declared-interval","overall_wait_seconds":3600,"check_interval_seconds":30,"preferred_mode":"provider-supported-deferral"}

This is not an Airflow operator configuration schema; translate it only through the chosen provider’s documented API.

Common errors and fixes

Many waiting tasks consume all worker slots.

Why it happens: Poke-style occupancy was not included in capacity planning.
How to fix it: Review supported reschedule or deferrable behavior and its prerequisites.
Prevent it next time: Track waiting duration and occupied capacity separately.

A deferred task remains suspended indefinitely.

Why it happens: The triggerer, trigger condition or timeout path may be unhealthy or unspecified.
How to fix it: Inspect exact deferred-state evidence and the provider’s resume contract.
Prevent it next time: Test trigger failure and bounded wait outcomes.

A readiness check succeeds on an incomplete file.

Why it happens: Existence was treated as completion.
How to fix it: Use a producer completion marker or validated manifest contract.
Prevent it next time: Separate availability from integrity and business validation.

Check your understanding

1. Does releasing a worker slot during a wait guarantee faster input arrival?

Show worked solution

No. It changes platform resource occupancy, not the source’s readiness. The workflow still needs an overall deadline and an explicit condition. Choose polling or deferral based on supported operator behavior, scheduling cost and capacity, then verify that failure and timeout paths produce the intended business status.

2. Why should a Deadline Alert not be described as execution_timeout?