Data Engineering / Intermediate: change, time and coordination / Lesson 11
Orchestration, dependencies and retries
Model task ordering and retry state without treating a scheduler as an atomic database. Bind each attempt to an identified data interval and accepted artifacts.
On this page
Lesson goals
- Explain DAG dependencies and task/data-state distinctions.
- Choose bounded retry and restart behavior by error class.
A scheduler coordinates work, not every data invariant
An orchestrator can schedule tasks, enforce dependencies and record attempts, but it does not automatically make their external effects atomic or idempotent. A successful upstream task should expose an identified accepted artifact rather than a local temporary path another worker cannot see. Distinguish logical data interval, run identity, task attempt and output version. Airflow is one concrete implementation with versioned APIs and scheduling semantics; this lesson uses offline models rather than running Airflow or assuming every orchestrator handles retries and data intervals identically.
Check a small dependency graph for readiness
The local model allows publication only when extraction, transformation and validation have their declared accepted states. A task status is supplied synthetic evidence, not a verified dataset check. Real systems need artifact and quality-state binding so a green task from another interval cannot satisfy this run’s dependency. Also detect cycles and missing dependency names in the chosen scheduler. A graph can be acyclic but semantically wrong if publication bypasses validation or waits for a task that does not produce the required data.
required = {'extract', 'transform', 'validate'}
state = {'extract': 'accepted', 'transform': 'accepted', 'validate': 'failed'}
assert not all(state.get(task) == 'accepted' for task in required)
state['validate'] = 'accepted'
assert all(state.get(task) == 'accepted' for task in required)Publication is blocked while validation is failed. Changing metadata alone does not prove a real validation result.
Retry transient failures without hiding deterministic defects
A temporary unavailable dependency and a malformed schema need different handling. Repeatedly retrying deterministic bad input can consume capacity without changing the outcome. Bound attempts, elapsed time and concurrency, preserve the original error and make a terminal failure visible. A lost response can mean the effect already succeeded, so retries must use the task’s idempotency and reconciliation contract. Backoff and jitter are load-management tools, not proof that repetition is safe. Do not place non-idempotent notifications or charges casually inside a retried transformation.
Make a bounded retry decision explicit
The toy policy retries only a declared transient condition while attempts remain. A schema violation fails immediately. Three is a teaching budget, not a recommended production number. Real policies need service deadlines, side-effect classification, source retention and owner escalation. The function does not sleep, contact a dependency or implement an orchestrator. Its purpose is to make the allowed decision auditable and expose the difference between an exhausted transient case and a deterministic input defect, both of which should remain visible in run evidence.
def retry_decision(error_kind, attempt, max_attempts=3):
if error_kind != 'transient':
return 'fail'
return 'retry' if attempt < max_attempts else 'fail'
assert retry_decision('transient', 1) == 'retry'
assert retry_decision('transient', 3) == 'fail'
assert retry_decision('schema', 1) == 'fail'Only supplied labels and counters are evaluated. No real retry, wait or platform operation occurs.
Reconcile task restarts with logical data state
A cleared or restarted task can rerun against data already published, so task-state manipulation needs the same stable interval and idempotency rules as ordinary retries. Passing small artifact references between workers differs from moving entire datasets through scheduler metadata. Keep secrets outside code and task logs. Define how partial outputs are hidden, reused or cleaned after failure, and how downstream tasks detect accepted replacement versions. An operator pressing retry is a control action, not evidence that the source, sink and checkpoint are now consistent.
Common errors and fixes
Retries create duplicate target effects
- Why it happens
- Task retry policy ignored sink idempotency.
- How to fix it
- Bind retry to stable operation identity and accepted publication.
- Prevent it next time
- Test lost responses and repeated attempts.
A task cannot find its upstream local file
- Why it happens
- Worker-local storage was treated as a shared durable artifact.
- How to fix it
- Use the approved shared artifact/reference contract.
- Prevent it next time
- Test different-worker execution and restart.
A green task from another interval unblocks publication
- Why it happens
- Dependency state was not bound to data/run identity.
- How to fix it
- Verify accepted artifact and interval alignment.
- Prevent it next time
- Include cross-run identity checks.
Check your understanding
1. Should a schema error receive unlimited exponential-backoff retries?
Show worked solution
No. A deterministic incompatible input usually remains incompatible and wastes capacity. Preserve the rejected evidence, fail visibly and route the issue to the appropriate producer/contract owner. Retry only a classified recoverable condition under an explicit attempt/time budget and an idempotent effect protocol.
2. Why is a scheduler success flag weaker than a published dataset version?
Show worked solution
The flag records task execution state, while consumers need a specific accepted data population, schema and artifact boundary. A task can succeed with stale input or a partial output. Bind the flag’s evidence to those dataset properties and keep publication separate from mere process termination.
Key takeaways
- Task coordination does not replace atomicity or idempotency.
- Bind every attempt and dependency to accepted data identity.