MLOps / Pipelines and controlled delivery / Lesson 7

Pipeline DAGs, caching, and idempotency

Represent true dependencies and define what retries or cache reuse may repeat safely.

About 5 min readContent reviewed 2026-10-28

On this page

Lesson goals

Distinguish a dependency graph from a sequence of commands.
Construct cache identity from every relevant declared input.
Separate repeated computation from repeated external effects.

A pipeline graph declares dependencies

A directed acyclic graph connects tasks through the artifacts and conditions they require. A transform depends on validated data, a trainer depends on the resulting representation, and a release package may require an accepted evaluation. The first fixture computes one valid ordering for a small chain. Independent branches can have several valid orders, so an implementation should test dependency constraints rather than assume an arbitrary total order is essential. A graph edge does not establish that the input content is correct or that every hidden dependency was declared. Review the actual input and output contract of each component.

Component boundaries make reuse interpretable

Define a task’s input artifacts, configuration, code identity, environment assumptions, output artifacts, status, and failure policy. Prefer an inspectable result boundary over a step that modifies several shared locations without a clear record. A task reading current time, a mutable path, or an external table can produce another result under apparently identical parameters. Either make that changing state an explicit input or declare that reuse is unsupported. The graph fixture is scheduling mathematics only: its task names do not run a validator, trainer, evaluator, or packaging service.

python Deterministic offline standard-library fixture

from graphlib import TopologicalSorter
graph = {"validate":(), "transform":("validate",), "train":("transform",),
         "evaluate":("train",), "package":("evaluate",)}
order = list(TopologicalSorter(graph).static_order())
print(order)
print("evaluation before package", order.index("evaluate") < order.index("package"))

Expected output

['validate', 'transform', 'train', 'evaluate', 'package']
evaluation before package True

A topological order respects declared dependencies; this program does not execute those tasks or prove that their undeclared inputs are complete.

A cache key represents a computation claim

The second fixture hashes explicit values and a code revision before reusing a total. Identical declared inputs reuse the result, while changed data or code do not. Real component identity can additionally depend on schema, preprocessing state, dependency environment, seed, and selected execution settings. A friendly task name or mutable dataset path is not enough. Cache reuse is valid only when the key captures the inputs that determine the result and the cached artifact remains valid and available. A cache hit means a declared result was reused, not that new validation occurred or a fresh measurement was taken.

python Deterministic offline standard-library fixture

import hashlib, json
cache = {}
def total(values, code_revision):
    raw = json.dumps({"values":values,"code":code_revision}, sort_keys=True).encode()
    key = hashlib.sha256(raw).hexdigest()
    hit = key in cache
    if not hit: cache[key] = sum(values)
    return cache[key], hit
print(total([1,3], "r1"))
print(total([1,3], "r1"))
print(total([1,3], "r2"))
print(total([1,4], "r1"))

Expected output

(4, False)
(4, True)
(4, False)
(5, False)

Both explicit data and code revision enter the in-memory cache identity; a code change causes recomputation even when its result happens to match.

Retries need an effect and ownership contract

A pure deterministic calculation can usually be repeated without changing external state. A task that sends a notification, appends predictions, or changes a selection alias has an effect that may occur again after a timeout. Idempotency means repeating a defined operation has the same intended effect under its stated identity rules; it is not the same as deterministic arithmetic. Use a request or output identity and an appropriate completion protocol in a real implementation. Do not claim exactly-once effects merely because a scheduler retries tasks or because a cache exists. This course performs no such external actions.

Publish completed artifacts at a clear boundary

Separate incomplete work from validated outputs so a downstream consumer cannot mistake a partially written artifact for success. Record attempt identity, final status, and the output content that a consumer should use. A retry can start from a clean boundary or resume a supported checkpoint, but those are different procedures. Test cache invalidation, missing cached artifacts, partial failure, and the intended retry semantics with tiny fixtures before introducing a platform. The supplied programs execute only graph ordering and an in-memory sum cache using Python’s standard library; no orchestration engine, file store, model, service, or distributed guarantee is tested.

Common errors and fixes

A cache returns a result after training code changed

Why it happens: The key omitted a relevant code or environment input.
How to fix it: Include all declared result-determining identities or disable reuse.
Prevent it next time: Test invalidation with a changed input whose numeric result can stay equal.

A retried step duplicates an external action

Why it happens: A computation retry policy was mistaken for effect idempotency.
How to fix it: Define operation identity and completion behavior for the owned effect.
Prevent it next time: Test timeout and repeat cases separately from pure calculation.

A downstream task consumes an incomplete artifact

Why it happens: Publication did not distinguish partial work from completed validated output.
How to fix it: Expose a clear completion and artifact identity boundary.
Prevent it next time: Keep failure-at-publication cases in the component contract tests.

Check your understanding

1. Why does the cache fixture recompute for code r2 even though the total remains four?

Show worked solution

A code revision is part of the declared computation identity. Equal output in one fixture does not prove all behavior is unchanged. Recomputing prevents the cache from assuming semantic equivalence that was never established, while a separately justified compatibility policy could define a different explicit reuse rule.

2. Two independent validation branches both precede training. Must they always appear in the same order?