Apache Airflow / Advanced: historical processing, deployment and recovery evidence / Lesson 16

Observability, logs and run diagnosis

Correlate task identity, state transitions and business freshness without equating orchestration success with complete output.

About 5 min readContent reviewed 2026-10-28

On this page

Lesson goals

Build a bounded incident timeline from run, task and attempt evidence.
Distinguish scheduling delay, execution duration and business data age.

Start with the exact execution identity

An incident report should identify DAG, run, task, map index and attempt before comparing logs. The same task definition can have overlapping runs and several failed attempts, so a timestamp and task name alone can join unrelated evidence. Capture state transitions such as scheduled, queued, running and terminal outcome with their source timestamps. Preserve the code or bundle version and relevant runtime labels. These facts help distinguish a task that never started from one that completed an external effect but failed while reporting success. Sanitize identifiers when they contain sensitive business information, and avoid using raw payloads as high-cardinality metric labels.

Break delay into meaningful intervals

Time waiting for dependencies differs from time queued for execution, task runtime and delay until downstream publication. A short successful task can still contribute to a stale report if input readiness or admission took hours. Business freshness should reference the intended data boundary and accepted output, not only a worker heartbeat. The offline calculation separates queued-to-start wait from start-to-finish runtime using supplied integer seconds. It assumes one clock and ordered timestamps; production evidence may need timezone and clock-skew review. Keep metric units explicit, and compare aligned intervals instead of adding a fresh scheduler value to an old task sample.

python Offline delay decomposition

queued_s, started_s, finished_s = 100, 145, 157
queue_wait_s = started_s - queued_s
runtime_s = finished_s - started_s
assert queue_wait_s == 45
assert runtime_s == 12
print(queue_wait_s, runtime_s)

Expected output

45 12

The supplied timeline has 45 seconds waiting and 12 running; it measures no real task.

Missing logs and failed work are different diagnoses

Task log storage and serving depend on the configured handler and deployment. A task can have run while its log is unavailable through the interface because a worker is gone, remote upload failed or a retrieval permission differs. Conversely, no task log may be expected if dispatch never reached execution. Inspect task-state and worker evidence before inferring business behavior from a blank log panel. Remote logging configuration has its own storage permissions and retention, separate from report artifacts. A deleted local log should not erase the only evidence needed for the supported incident horizon, so review retrieval and retention together.

Keep a compact sanitized event record

The JSON fixture records a failed attempt with a failure category and a business partition while omitting payloads and credentials. A category is useful for aggregation, but preserve the exact sanitized exception class in detailed evidence so a schema failure is distinguishable from an unavailable dependency. Repeated retries should remain linked to the same logical operation. Notifications and callbacks have their own execution and delivery outcomes; failure to receive an alert is not proof no task failed. Airflow 3.1 Deadline Alerts are experimental, so their specific callback support and timing semantics must be reviewed rather than copied from newer versions.

json Synthetic sanitized diagnostic event

{"dag_id":"lab_parcel_report","run_id":"lab-run-17","task_id":"publish","map_index":-1,"try_number":2,"state":"failed","failure_category":"temporary-dependency","business_partition":"2026-10-01"}

The record preserves execution and business identity without credentials or raw input rows.

Change the component supported by the timeline

If wait grows while runtime remains stable, examine admission and dispatch before rewriting transformation code. If runtime grows with downstream latency, investigate that dependency and the task's bounded timeout policy. If output freshness fails despite successful tasks, inspect interval selection, publication and acceptance checks. Preserve the original evidence before any authorized rerun because new attempts can make the incident harder to reconstruct. This course queries no monitoring system, reads no production logs and sends no notification. Its local timing arithmetic and JSON fixture validate representation only; actual log serving, scheduler metrics and callback delivery remain untested.

Common errors and fixes

Logs from two runs are merged into one incident timeline.

Why it happens: The investigation used only task name and rough time.
How to fix it: Correlate complete run, mapping and attempt identities.
Prevent it next time: Standardize structured execution context in diagnostics.

A blank log panel is treated as proof the task did not run.

Why it happens: Log retrieval was confused with execution state.
How to fix it: Inspect state transitions, worker evidence and logging backend separately.
Prevent it next time: Test log retrieval after worker removal in an authorized environment.

Low runtime is reported as low report latency.

Why it happens: Readiness, queueing and publication delays were excluded.
How to fix it: Measure the business freshness boundary and decompose its stages.
Prevent it next time: Label each duration and its start/end clocks.

Check your understanding

1. Queue wait is 45 seconds and runtime 12. Does optimizing runtime by two seconds remove most delay?

Show worked solution

No. The supplied 57-second interval is dominated by queue wait. A two-second runtime reduction helps only a small part of that interval. Investigate the admission or dispatch evidence first, while also checking whether input readiness and final publication add further delay outside these timestamps.

2. Does a delivered failure notification prove the destination report was never published?