MLOps / Observation, recovery, and lifecycle decisions / Lesson 18

Cost budgets and complete lifecycle review

Evaluate resource use and readiness against the whole supported decision process.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Define resource denominators and include failed or fallback work.
Assess competing constraints rather than optimize one metric alone.
Assemble traceable lifecycle evidence and retirement responsibilities.

Resource cost follows the supported workload

Training, feature preparation, evaluation, storage, inference, telemetry, fallback, and incident response all consume resources. Define the workload and denominator before comparing efficiency: per submitted request, eligible request, supported prediction, or completed decision are different quantities. A candidate that rejects difficult inputs can appear cheap per processed example while failing the intended coverage. Include failed attempts and retries where they belong to the boundary. State whether values are analytical estimates or measurements from an actual environment, and identify hardware, software, shapes, time window, and omitted categories for any measured claim.

A favorable metric does not satisfy every constraint

The first fixture uses invented compute units, supported-result counts, error values, and fixed acceptance bounds. One candidate meets the resource bound but misses the error requirement; another meets the error requirement but exceeds the resource bound. Neither passes the whole declared contract. The per-supported-result calculation supplies another view, not a reason to ignore the other conditions. Thresholds must follow the task and evidence rather than a universal cheapest-model rule. A more complex system can also create maintenance and recovery costs that are not represented by one inference measurement.

python Deterministic offline standard-library fixture

candidates = [{"id":"a","error":2.0,"compute_units":50,"supported_results":90},
              {"id":"b","error":1.8,"compute_units":80,"supported_results":100}]
for row in candidates:
    per_result = row["compute_units"]/row["supported_results"]
    accepted = row["error"] <= 1.9 and row["compute_units"] <= 70
    print(row["id"], f"units_per_supported_result={per_result:.3f}", "all bounds", accepted)

Expected output

a units_per_supported_result=0.556 all bounds False
b units_per_supported_result=0.800 all bounds False

Invented compute units and thresholds show a multi-constraint decision; they are not prices, measured efficiency, or universal acceptance limits.

Budgets need controlled adaptation

A measured bottleneck can motivate a smaller model, batching, caching, feature simplification, or a revised delivery schedule. Each change can alter latency, freshness, coverage, numerical behavior, or quality, so reevaluate the complete contract. Cache reuse saves computation only when its identity and validity assumptions hold. Resource bounds should prevent noisy retraining triggers or oversized requests from creating uncontrolled work. If no candidate meets the requirements, a documented decision to retain the baseline, revise the task with authority, or stop the project can be more appropriate than silently weakening a gate.

Readiness is a linked evidence package

The second fixture combines matching evaluation identity, maturity, consumer compatibility, budget, and recovery. It fails on compatibility even though the remaining checks pass. A real review also preserves the task owner, data and split contracts, code and transform identities, provenance, observation definitions, fallback, and known limitations. Avoid treating a checklist as complete proof of every property; it summarizes specific evidence whose quality still matters. Verify the inputs behind each check and which consumer version will actually use the package. Registration, selection, delivery, and outcome observation remain separate lifecycle events.

python Deterministic offline standard-library fixture

release = {"package":"model-2", "report_package":"model-2",
           "mature_evaluation":True, "consumer_compatible":False,
           "budget_supported":True, "recovery_ready":True}
checks = {"matching report":release["package"] == release["report_package"],
          "mature evaluation":release["mature_evaluation"],
          "consumer compatibility":release["consumer_compatible"],
          "budget":release["budget_supported"], "recovery":release["recovery_ready"]}
print("gaps", [name for name,passed in checks.items() if not passed])
print("declared release ready", all(checks.values()))

Expected output

gaps ['consumer compatibility']
declared release ready False

One incompatible consumer blocks this declared checklist despite matching evaluation and resource evidence; the script performs no release action.

Ownership continues through revision and retirement

Define what triggers reassessment, how delayed outcome evidence updates the record, which dependencies recovery requires, and when an unsupported model is retired. Retention should preserve necessary audit and recovery artifacts while following the applicable data-handling contract. Removing an unused artifact can still affect an undeclared consumer, so identify dependencies before retirement. The course’s final review should leave a concrete decision and evidence boundary rather than a claim that automation eliminates judgment. These fixtures calculate synthetic resource ratios and readiness Booleans only; no prices, capacity benchmark, platform budget, deployment, or organizational approval is created or changed.

Common errors and fixes

Efficiency improves by excluding unsupported requests from all counts

Why it happens: The denominator hid lost coverage and failed work.
How to fix it: Report submitted, eligible, supported, fallback, and failed populations explicitly.
Prevent it next time: Check resource ratios alongside coverage and quality.

A resource optimization bypasses model evaluation

Why it happens: A faster implementation was assumed behaviorally equivalent.
How to fix it: Reassess preprocessing, timing, numerical outputs, and decision quality.
Prevent it next time: Keep known-input and acceptance checks with performance evidence.

A checklist pass is treated as permanent model approval

Why it happens: Evidence limits and ongoing ownership were ignored.
How to fix it: Record the specific readiness scope and reassessment conditions.
Prevent it next time: Maintain monitoring, recovery, and retirement responsibilities.

Check your understanding

1. Why does neither candidate pass the first fixture despite each having one advantage?

Show worked solution

Candidate a uses fifty units, within the bound of seventy, but error two exceeds 1.9. Candidate b has error 1.8, within the quality bound, but uses eighty units. The conjunction fails for both. A decision must address the actual constraints or explicitly revise authorized requirements; it cannot select only a favorable metric.

2. What should be reviewed before retiring an apparently unused model artifact?