Apache Airflow / Advanced: historical processing, deployment and recovery evidence / Lesson 14

Executors, workers and deployment boundaries

Trace task dispatch and execution while keeping locality, runtime compatibility and external capacity separate.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Compare local, queued-worker and per-task-container execution patterns.
Explain why queued state is not proof that task code began.

An executor connects a decision to execution

The scheduler decides that a task instance is eligible, and the selected executor arranges for it to run. Executor logic belongs to the scheduler process; it is not necessarily a separate service called executor that should be started independently. The execution target can still be local processes, persistent remote workers or task-specific containers depending on the implementation. queued means work has entered dispatch-related handling rather than proving its Python body is running. Preserve the transition timestamps and executor evidence when diagnosing a delay. A missing task log may indicate the task never reached execution, not that its business function silently did nothing.

Choose a pattern from workload and isolation needs

Local execution shares a host resource context with scheduling and can be appropriate for a bounded single-machine design. Queued workers can amortize startup cost but require queue routing, compatible worker environments and a capacity model for shared processes. Per-task containers can offer separate runtime packaging with startup and infrastructure costs. No pattern automatically grants business transactionality or access isolation beyond its actual deployment controls. A worker's filesystem and installed libraries are not guaranteed to match the DAG processor's environment. Record where providers and business dependencies must exist, and review their compatibility before claiming a definition that parses will also execute.

Compare declared queue and worker coverage

The offline fixture says a task is routed to reports while available workers subscribe only to default and imports. The set comparison identifies missing routing coverage without inspecting or starting a real queue. A compatible worker may exist elsewhere, so the supplied inventory is an explicit assumption rather than a service discovery claim. Diagnose the exact executor-specific path before changing queue names. Some executors do not use the same queue semantics, and a valid routing label still does not prove free CPU, memory or external permissions. Keep routing availability and runtime resource availability as separate questions.

python Offline routing-coverage check

task_queue = 'reports'
worker_queues = {'default', 'imports'}
covered = task_queue in worker_queues
assert not covered
print(covered)

Expected output

False

The supplied worker inventory lacks reports; no real executor or message queue is queried.

Describe the execution image without provisioning it

The JSON record names synthetic code and runtime versions, required provider families and the artifact contract. It is documentation, not a container manifest or an instruction to install dependencies. A real deployment should build and verify an immutable runtime through its approved process and distribute compatible DAG definitions or bundles. Credentials belong to supported runtime lookup rather than the image's source code. If an operator delegates computation to another service, that service has its own runtime and completion contract. Do not confuse successfully starting a task container with successfully completing the remote job it submitted.

json Synthetic worker runtime contract

{"code_version":"lab-report-v3","runtime_version":"lab-runtime-2","required_providers":["standard"],"artifact_contract":"immutable-manifest-v1","credential_source":"approved-runtime-lookup"}

These labels document assumptions and are not tested package versions, image names or installation instructions.

Separate dispatch failure from task failure

A task can wait because no suitable worker accepts it, fail at process startup due to a missing dependency, or fail later because its business input is invalid. Each boundary calls for different evidence and remediation. Collect scheduler/executor status, worker startup logs and task logs in order, with the same run and attempt identity. Increasing retries on invalid code does not create a missing package or correct routing. This material installs no executor, starts no worker and contacts no queue or cluster. Only the bounded routing comparison and a JSON contract are validated locally, leaving actual scheduling and runtime compatibility untested.

Common errors and fixes

A queued task is assumed to have executed its first line.

Why it happens: Dispatch state was confused with running state.
How to fix it: Trace executor acceptance, worker startup and task logs separately.
Prevent it next time: Keep transition timestamps and attempt identity in incident evidence.

A DAG parses but a worker cannot import its provider.

Why it happens: Definition and execution runtimes differ.
How to fix it: Review the approved immutable runtime and provider compatibility.
Prevent it next time: Validate both parse and execution environments before deployment.

A task routes to a queue no worker serves.

Why it happens: Routing labels and worker subscriptions diverged.
How to fix it: Correct the reviewed executor-specific routing contract.
Prevent it next time: Include queue coverage in deployment acceptance.

Check your understanding

1. Does a per-task container automatically make an external database write atomic with task success?

Show worked solution

No. Runtime isolation and distributed transactionality are different properties. The task still needs a destination-supported idempotent or transactional protocol and reconciliation for uncertain outcomes. A separate container can improve packaging and process isolation while leaving the effect-to-state crash window unchanged.

2. Why can adding more default-queue workers fail to help a reports-queue task?