Free to read ยท Three levels
Apache Airflow: workflow contracts, scheduling and reliable recovery
Eighteen developed lessons connect DAG authoring, interval semantics, task communication and retry safety to executors, observability and recovery evidence.
Start learningIn this course
Study synthetic parcel reports, bounded partition manifests and explicit publication contracts. Six lessons at each level develop concrete scheduling, failure and recovery decisions through explained examples and worked exercises.
The course separates reusable DAG definitions, run context and external business effects. Local Python checks establish small deterministic properties while platform examples remain clearly distinguished from unperformed orchestration tests.
What you will learn
- Define explicit task, interval, artifact and publication contracts.
- Reason through mapping, branching, waiting, concurrency and duplicate-safe retries.
- Evaluate executor, security, observability, versioning and recovery boundaries with specific evidence.
Your learning path
Three levels
Basic: definitions, intervals and safe task execution
Build precise DAG, context, communication and retry contracts.
- Orchestration, DAGs and architecture
- TaskFlow, dependencies and execution context
- DAG runs, data intervals and catchup
- Operators, connections and templates
- XCom and task data boundaries
- Retries, idempotency and failure states
Intermediate: runtime work, control flow and resource budgets
Design bounded mapping, joins, waiting, asset updates and admission policies.
- Dynamic task mapping and task groups
- Branching, trigger rules and joins
- Sensors, deferral and timeouts
- Assets, events and data-aware scheduling
- Pools, concurrency and backpressure
- Testing DAG contracts and safe local logic
Advanced: historical processing, deployment and recovery evidence
Review backfills, executor boundaries, trust, observability and versioned recovery.
- Backfills, partitions and late data
- Executors, workers and deployment boundaries
- Secrets, authentication and access
- Observability, logs and run diagnosis
- Performance, parse time and DAG distribution
- Upgrade compatibility and recovery design
Who this is for
- Developers authoring batch workflows and repeatable data pipelines.
- Engineers diagnosing scheduling, queueing, runtime and historical-processing incidents.
Before you start
- Basic Python, JSON and familiarity with files, application calls and data partitions.
- No Airflow installation, metadata database, broker, cloud account or credentials are needed for the offline examples.