Apache Airflow / Intermediate: runtime work, control flow and resource budgets / Lesson 7

Dynamic task mapping and task groups

Expand bounded runtime work deliberately and keep grouping, identity and aggregation semantics clear.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Distinguish runtime mapping from a Python loop during DAG construction.
Explain expand, partial and the cost of mapped result collection.

The task count can depend on upstream output

Dynamic task mapping lets an upstream task return a bounded list or dictionary that determines mapped instances at runtime. A Python loop in the DAG file instead constructs known graph nodes while the file is parsed. The distinction matters when discovering input partitions requires external access: that discovery belongs in a task rather than repeated module import. Runtime flexibility is not unlimited cardinality. Define the maximum expected number of work items, the empty-input policy and stable item identifiers. A million tiny mapped tasks can overwhelm orchestration overhead even when each business operation is cheap, so choose a useful unit of work.

Separate mapped inputs from constant arguments

The illustrative DAG maps process over two synthetic depot identifiers while keeping multiplier constant through partial. expand takes keyword arguments, and multiple independent argument lists form combinations rather than automatically pairing corresponding elements. If the requirement is pairwise work, construct or zip the intended records using the documented APIs. The map order is not a business sequencing guarantee. A mapped task should derive destination identity from a stable item key and interval, not from an assumed presentation order in the user interface. This framework example is not imported or executed, and no partitions are actually scheduled.

python Illustrative bounded TaskFlow mapping

from datetime import datetime, timezone
from airflow.sdk import dag, task

@dag(schedule=None, start_date=datetime(2026, 1, 1, tzinfo=timezone.utc), catchup=False)
def lab_depot_map():
    @task
    def process(depot, multiplier):
        return {'depot': depot, 'units': multiplier * 2}
    process.partial(multiplier=3).expand(depot=['D1', 'D2'])

lab_depot_map()

The declared inputs create two mapped instances with six units each; no Airflow runtime is exercised.

A task group organizes rather than isolates execution

Task groups provide logical organization and namespacing in the graph. They do not create one shared transaction, one worker or a shared local directory for all contained tasks. Mapping a group can express a repeated subworkflow with documented dependency behavior, but its internal tasks still have separate states and failure boundaries. Stable group and task identifiers help preserve diagnostic continuity across definition changes. Avoid using visual nesting to imply that data is copied or permissions are inherited automatically. Document which item key flows through each stage and how one failed item affects aggregation or downstream publication.

Count combinations before expansion

The offline product has two depots and three dates, producing six work items. It contrasts with a two-pair zip that stops at the shorter input. These are plain Python semantics used to expose cardinality, not an implementation of the Airflow scheduler. A mistaken product can multiply cost and repeat outputs for unintended combinations. Verify the expected item set explicitly before introducing mapping, and reject duplicate logical keys when they would collide at the destination. For a true partition matrix, the product may be exactly correct; the lesson is to choose it intentionally rather than infer pairing from list positions.

python Offline product-versus-pair cardinality

from itertools import product
depots = ['D1', 'D2']
dates = ['2026-10-01', '2026-10-02', '2026-10-03']
combinations = list(product(depots, dates))
pairs = list(zip(depots, dates))
assert len(combinations) == 6
assert len(pairs) == 2
print(len(combinations), len(pairs))

Expected output

6 2

Six combinations differ from two pairs; choose the intended work set explicitly.

Collect mapped results with a resource budget

Mapped outputs can be represented by a lazy sequence rather than an ordinary eagerly materialized list. Converting all results to a list loads them and can create a large XCom or memory burden. Prefer compact per-item summaries and an aggregation strategy that fits the declared scale. An empty map can be skipped under documented behavior, so a final task's trigger rule must handle that meaning deliberately. Limits on mapping and task concurrency are deployment settings to review, not values to raise blindly after a failure. This course tests local cardinality only and performs no mapped Airflow run or configuration mutation.

Common errors and fixes

Mapping unexpectedly creates many combinations.

Why it happens: Independent lists were assumed to pair by position.
How to fix it: Construct the intended pairwise records or use documented zipping.
Prevent it next time: Assert work-item cardinality and identities before expansion.

A task group is expected to share one local output directory.

Why it happens: Visual grouping was mistaken for execution locality.
How to fix it: Use explicit artifact references accessible to each task.
Prevent it next time: Review worker isolation independently of graph nesting.

A reduce task exhausts memory collecting every result.

Why it happens: Lazy outputs were materialized without a size budget.
How to fix it: Aggregate compact summaries or redesign the work granularity.
Prevent it next time: Bound map size and result payloads together.

Check your understanding

1. Two depots and three dates are passed as independent mapped lists. Is two work items the safe expectation?

Show worked solution

No. Independent expansion can create a six-combination product. If two specific depot-date pairs are intended, represent those records explicitly and check which date is unused or why. A cardinality test catches this design error before scheduling creates unintended work or colliding destination outputs.

2. Why should map_index not automatically become the business artifact key?