Apache Airflow / Basic: definitions, intervals and safe task execution / Lesson 3

DAG runs, data intervals and catchup

Choose a schedule and interval contract that remains correct when a run starts late or is replayed.

About 5 min readContent reviewed 2026-10-28

On this page

Lesson goals

Distinguish logical date, data interval and actual start time.
Explain catchup without assuming tasks automatically partition their inputs.

A run processes a declared interval

For an interval-based daily schedule, a run normally represents a completed data interval rather than whatever day the worker happens to start. The logical date identifies the run's scheduling context and is not a synonym for actual execution time. A delayed October first report can start on October third while still needing October first inputs. Use data_interval_start and data_interval_end according to the selected timetable and make the interval half-open where the business contract requires it. Different timetables, manual runs and asset-triggered runs can expose different semantics, so do not infer every interval from a date-shaped run label.

Test exact interval membership offline

The local example defines one UTC day and checks records exactly at its start, just before its end and exactly at the next day's start. The last record belongs to the next interval, preventing double counting when adjacent reports are combined. Explicit timezone-aware values avoid relying on the worker machine's local timezone. This arithmetic does not implement Airflow's timetable or daylight-saving behavior. Calendar-based reporting in a named local timezone needs its own boundary tests, particularly around offset changes. Preserve the chosen business timezone in the dataset contract rather than silently converting a calendar requirement into twenty-four-hour arithmetic.

python Offline half-open UTC interval check

from datetime import datetime, timedelta, timezone
start = datetime(2026, 10, 1, tzinfo=timezone.utc)
end = start + timedelta(days=1)
points = [start, end - timedelta(seconds=1), end]
inside = [point for point in points if start <= point < end]
assert len(inside) == 2
print(len(inside))

Expected output

The end boundary is excluded; no Airflow timetable is executed.

Start dates establish scheduling eligibility

A start_date is a scheduling boundary, not a command to begin running immediately at that wall-clock moment. For common interval schedules, eligibility follows completion of the relevant interval and other constraints. catchup determines whether missed scheduled intervals are created under the scheduling rules; it does not rewrite task code to read historical partitions. A task that always queries latest data can produce the wrong result for every catchup run. Choose a fixed, timezone-aware start boundary and explicit catchup policy. Avoid a moving now-based start date that changes during repeated parsing and makes the intended first interval difficult to reason about.

Read an explicit UTC daily definition

The non-runnable definition explicitly selects CronDataIntervalTimetable with a UTC daily boundary and catchup disabled. Airflow 3 cron defaults can use trigger-based timetables, so interval behavior is chosen rather than assumed. This means it is not asking the scheduler to create every missed historical daily run; it does not prohibit an independently authorized manual or backfill operation. The empty body is represented by a small task that returns a label so the example is a complete illustrative DAG rather than an unexplained ellipsis. Its output label is not a data partition contract by itself. A production report would obtain the declared interval through runtime context and use that interval consistently in source selection and destination identity.

python Illustrative explicit daily DAG policy

from datetime import datetime, timezone
from airflow.sdk import dag, task
from airflow.timetables.interval import CronDataIntervalTimetable

@dag(schedule=CronDataIntervalTimetable('0 0 * * *', timezone='UTC'), start_date=datetime(2026, 10, 1, tzinfo=timezone.utc), catchup=False)
def lab_daily_report():
    @task
    def identify():
        return 'interval-bound-report'
    identify()

lab_daily_report()

The schedule is declarative and unexecuted; the returned label does not select source rows.

Separate a late run from late-arriving data

A run delayed by capacity and a source record arriving after its intended interval closes are different cases. The first may be solved by executing the same deterministic partition later; the second requires a correction, watermark or reprocessing policy appropriate to the source. Airflow scheduling alone cannot decide whether a daily report is complete enough for business publication. Record expected input availability and how a late correction replaces or amends output. Verify adjacent intervals have no unintended overlap or gap. This course performs only local boundary calculations and does not create scheduled runs, change a timetable or trigger catchup.

Common errors and fixes

A delayed run reads today instead of its assigned interval.

Why it happens: Wall-clock now replaced the data interval.
How to fix it: Pass explicit interval boundaries into source selection and output identity.
Prevent it next time: Test a historical run executed on a later date.

Catchup creates runs that all write the same latest partition.

Why it happens: The task was not interval-aware.
How to fix it: Make each supported interval select and replace its own business partition.
Prevent it next time: Review task partition logic before enabling catchup.

A boundary record is counted twice in adjacent reports.

Why it happens: Both ends of both intervals were inclusive.
How to fix it: Use the documented half-open selection rule or another explicit nonoverlap rule.
Prevent it next time: Keep exact-boundary fixtures in tests.

Check your understanding

1. A report for October first starts on October third. Which clock should select its source partition?

Show worked solution

Use the report’s declared data interval, not the actual worker start date. The source partition and output identity should remain tied to the intended business range. If the source has late corrections, apply the documented correction policy rather than silently broadening the interval to whatever data is newest.

2. Does catchup=False prove historical reprocessing can never happen?