Apache Airflow / Basic: definitions, intervals and safe task execution / Lesson 1

Orchestration, DAGs and architecture

Separate workflow definitions, scheduled runs and execution components before authoring a pipeline.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Explain the difference between a DAG, a DAG run and a task instance.
Locate authoring, scheduling, execution and state responsibilities.

Orchestrate bounded work with dependencies

Airflow represents a workflow as tasks connected by dependencies. A synthetic daily parcel report might validate an input manifest, summarize its rows and publish a report pointer. Airflow decides when task instances become eligible and records their execution state; it does not automatically validate the business correctness of the report. The work inside a task can be Python or an operation delegated to another system. Define a finite unit of work and an observable output before drawing arrows. A permanently running event-consumer loop is a different execution pattern from a bounded report task and should not be disguised as an ordinary daily batch.

A definition is not one execution

The DAG definition names tasks, dependencies, schedule and policies. A DAG run is one occurrence of that workflow, and a task instance is one task within a particular run, potentially with retries and a mapping index. The same task identifier can therefore appear in many runs without representing the same work. Include DAG identity, run identity and task identity when diagnosing an incident. A task function's source code is reusable logic, while its task instance state is runtime evidence. Editing the definition does not retroactively prove that older task instances used the same inputs, code or provider versions.

Inspect a synthetic execution identity

The JSON fixture identifies one non-mapped validation task in a synthetic run. try_number distinguishes an execution attempt from the logical task instance; it should not become the business partition identifier used for durable outputs. The run identifier is a label rather than proof that a schedule actually fired. A manually triggered run can process a declared interval too, but only if the task input contract supports it. Keep the business date separate from the machine's present date. This separation makes later backfills and retries reviewable without inferring the intended data range from a worker log timestamp.

json Synthetic task-instance context

{"dag_id":"lab_parcel_report","run_id":"lab-run-2026-10-01","task_id":"validate","map_index":-1,"try_number":1}

The identifiers distinguish workflow, run, task and attempt; they do not prove an Airflow execution occurred.

Follow the control path without starting services

DAG processing discovers and serializes definitions; scheduling decides eligible work; an executor connects scheduling decisions to execution; workers run tasks under the selected deployment model. Airflow 3 separates task execution through its public Task SDK and execution-facing interfaces rather than permitting task code to manipulate the metadata database directly. The API server supports application interfaces, while the metadata database stores orchestration state. A triggerer is relevant when deferrable operators are used. Deployment details depend on executor and provider choices, so a single-machine sketch is not a universal production architecture or an excuse to couple tasks through one local filesystem.

Check the declared dependency direction

The offline example records validate before summarize and summarize before publish. It verifies only the supplied order and uniqueness of stage names; it is not a complete cycle detector or an Airflow scheduler. Its purpose is to make the business dependency explicit before adopting framework syntax. If validation fails, publishing a report as though validation succeeded would violate the workflow contract even if a permissive trigger rule allowed it. This course selects Airflow 3.1.0 documentation and runs no API server, scheduler, executor or metadata database. Framework examples remain non-runnable until a separately authorized compatible environment is available.

python Offline dependency-order check

stages = ['validate', 'summarize', 'publish']
assert len(stages) == len(set(stages))
assert stages.index('validate') < stages.index('summarize') < stages.index('publish')
print('validate -> summarize -> publish')

Expected output

validate -> summarize -> publish

The declared sequence passes; no scheduling or task-state transition is simulated.

Common errors and fixes

A task identifier alone is used to identify an incident.

Why it happens: Runs and attempts were collapsed into the reusable definition.
How to fix it: Collect DAG, run, task, mapping index and attempt context.
Prevent it next time: Standardize execution identity in logs and artifacts.

A downstream task expects a file left on another worker.

Why it happens: Distributed execution was treated as one shared process.
How to fix it: Use a reviewed shared artifact location and a small reference contract.
Prevent it next time: Test task isolation and rerun assumptions.

A successful task is called a correct report without validation.

Why it happens: Execution state replaced business acceptance.
How to fix it: Define and check report invariants before publication.
Prevent it next time: Keep acceptance evidence with the task output.

Check your understanding

1. Are validate in two different run identifiers the same task instance?

Show worked solution

No. They share a task definition but belong to different workflow occurrences, potentially processing different intervals and artifacts. Logs must identify the run as well as the task. A retry belongs to the relevant logical instance and changes attempt evidence rather than changing the business meaning of the interval.

2. Why is the sequence assertion not evidence that Airflow can recover a worker failure?