Apache Airflow / Basic: definitions, intervals and safe task execution / Lesson 5

XCom and task data boundaries

Pass small execution-scoped references while keeping large artifacts and durable evidence under explicit storage contracts.

About 5 min readContent reviewed 2026-10-28

On this page

Lesson goals

Explain XCom scope and automatic TaskFlow return values.
Design a small artifact manifest and validate its integrity fields.

Task communication is not shared memory

Tasks are isolated execution units and may run on different workers. XCom carries small serializable messages associated with task execution, while a Python variable in one worker is not automatically visible to another. A TaskFlow return value commonly becomes an XCom return_value, and passing its output reference creates the intended data dependency. XCom is not a general-purpose large-data transport or a replacement for a durable business database. Keep message size and serialization requirements explicit. A custom backend can change storage behavior, but it does not remove the need to reason about lifecycle, permissions and the size of values loaded by consumers.

Pass a manifest rather than the entire dataset

A useful handoff message can contain an artifact URI, a business interval, a row count and a content digest. The dataset lives in a reviewed shared store accessible to the relevant tasks; the small manifest tells the reader what to fetch and verify. The synthetic JSON example uses an invalid-domain URI and creates no remote object. A path alone is weak evidence because its contents can change after publication. Immutable or versioned artifact identity and a checksum help distinguish an expected object from a later replacement. Choose a lifecycle long enough for supported retries and recovery, not merely the next task's normal start time.

json Synthetic small artifact-reference contract

{"uri":"https://artifacts.training.invalid/reports/2026-10-01/v1.csv","interval":"2026-10-01","row_count":2,"format":"csv-utf8"}

The reserved invalid URI is descriptive only; a real contract should add versioned integrity evidence.

Expect retry-related communication changes

An unsuccessful task can have its XComs cleared on retry under Airflow's documented behavior. Do not rely on a failed attempt's partial message as the authoritative checkpoint for an external side effect. Publish a manifest only after the artifact satisfies its completion contract, and retain durable completion evidence in the storage system when required. XCom scope also differs from Variables, which represent shared configuration rather than one task-run handoff. Reusing a global Variable as a per-run output pointer can make overlapping runs overwrite one another. Keep interval and run identity in the handoff contract instead of assuming only one run is active.

Verify a small deterministic artifact offline

The Python example hashes exact UTF-8 bytes for a two-row fixture and builds a compact manifest. It checks row count and digest length without writing to a shared filesystem or contacting object storage. A digest verifies byte identity only when compared with trusted expected evidence; it does not prove the source rows are true or complete. Keep newline, encoding and serialization rules stable so the producer and reader hash the same bytes. If compression or encryption changes the stored representation, decide whether the manifest describes stored bytes, decoded content or both and label that distinction explicitly.

python Offline artifact-manifest calculation

import hashlib
content = 'parcel_id,weight_g\nP1,200\nP2,300\n'.encode('utf-8')
manifest = {'row_count': 2, 'sha256': hashlib.sha256(content).hexdigest()}
assert manifest['row_count'] == 2
assert len(manifest['sha256']) == 64
print(manifest['row_count'], len(manifest['sha256']))

Expected output

2 64

The digest describes exact supplied bytes and does not establish source completeness or remote persistence.

Review the backend as part of deployment

The default XCom behavior and an object-storage-backed configuration can have different operational costs and retention paths. Installing a backend or changing its threshold is a deployment decision requiring compatibility and permission review, not a reason to push arbitrarily large frames between tasks. Avoid accessing XCom database models directly from task code in Airflow 3; use the public context and Task SDK mechanisms. During diagnosis, identify the producing task instance, key and run rather than searching one global name. This material validates local JSON and hashing only and does not create XCom records, configure a backend or contact an Airflow instance.

Common errors and fixes

A large dataframe overwhelms task communication.

Why it happens: XCom was treated as an unrestricted dataset transport.
How to fix it: Store the artifact under a reviewed contract and pass a small reference.
Prevent it next time: Set explicit message size and serialization expectations.

Overlapping runs consume the wrong global output pointer.

Why it happens: A shared Variable replaced execution-scoped communication.
How to fix it: Use run-aware handoff values and immutable artifact identities.
Prevent it next time: Test concurrent intervals and retry behavior.

A failed attempt’s partial message is missing on retry.

Why it happens: XCom lifecycle was mistaken for durable external completion evidence.
How to fix it: Record durable completion at the artifact boundary and publish only valid manifests.
Prevent it next time: Review unsuccessful-attempt cleanup semantics.

Check your understanding

1. Does a SHA-256 digest prove a report contains every expected parcel?

Show worked solution

No. It identifies the supplied byte content when compared with trusted evidence. Completeness requires independent source coverage and business checks, such as expected partitions and reconciled counts. Preserve both integrity and completeness evidence because an incomplete but unchanged file can have a perfectly valid digest.

2. Why can a local temporary filename be an unsafe downstream handoff?