MLOps / Observation, recovery, and lifecycle decisions / Lesson 15

Retraining triggers and feedback loops

Control when a new candidate is built and examine how existing decisions shape future training evidence.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Separate a trigger, training request, accepted candidate, and activation.
Coalesce repeated events under a defined request identity.
Identify selective labels and decision-induced feedback.

A trigger proposes work rather than approves a model

A schedule, new eligible data, a revised target, an incident investigation, or a measured performance change can motivate a training request. None guarantees that another fit will improve the decision process. Define prerequisites such as label maturity, data quality, minimum useful change, resource budget, and a valid evaluation plan. Keep the current baseline available. A newly produced artifact must still pass the ordinary acceptance and compatibility process. Continuous training means controlled repeated candidate construction under a declared policy, not unconditional replacement whenever an event arrives.

Request identity prevents accidental duplicate work

The first fixture receives repeated data-change and schedule events but coalesces them under one snapshot-and-recipe identity. That is a chosen policy: a deliberate independent replication could include a seed or replication id and therefore be a different request. Record why a request was created, skipped, already in progress, or superseded. A real scheduler also needs concurrency, persistence, failure, and cancellation semantics; a set in memory provides none of those distributed guarantees. The fixture teaches the identity decision only and performs no training, scheduling, service call, or external effect.

python Deterministic offline standard-library fixture

events = [{"reason":"data-change", "snapshot":"data-2"},
          {"reason":"data-change", "snapshot":"data-2"},
          {"reason":"schedule", "snapshot":"data-2"}]
requested = set()
for event in events:
    identity = (event["snapshot"], "recipe-1")
    new_request = identity not in requested
    requested.add(identity)
    print(event["reason"], "new training request", new_request)
print("unique requests", len(requested))

Expected output

data-change new training request True
data-change new training request False
schedule new training request False
unique requests 1

The declared identity coalesces these events into one synthetic request for a snapshot and recipe; it does not train or schedule anything.

The active policy can change which labels appear

When a model selects which cases receive review, measurement, or intervention, its outputs can affect future observed targets. The second fixture selects only high-scoring cases for review. In a real process, labels for the unreviewed cases may be absent, so training only on reviewed cases can reinforce the current policy’s blind spots. The supplied fixture lists all synthetic outcomes to expose the selection mechanism; it does not suggest that unknown outcomes are observable. Document the action and eligibility process alongside each label and assess whether the available evidence supports the intended target population.

python Deterministic offline standard-library fixture

population = [{"id":"a","score":0.9,"outcome":1},
              {"id":"b","score":0.8,"outcome":0},
              {"id":"c","score":0.2,"outcome":1},
              {"id":"d","score":0.1,"outcome":0}]
reviewed = [row for row in population if row["score"] >= 0.5]
print("reviewed ids", [row["id"] for row in reviewed])
print(f"observed coverage={len(reviewed)/len(population):.2f}")
print("unreviewed outcomes identifiable from observed labels alone", False)

Expected output

reviewed ids ['a', 'b']
observed coverage=0.50
unreviewed outcomes identifiable from observed labels alone False

All outcomes are supplied only to make the synthetic selection mechanism visible; a real selected-label process would not reveal the unreviewed outcomes automatically.

Feedback changes require an explicit learning design

A model can influence user behavior, source data, or another model that later feeds its inputs. Distinguish a genuine world change from a distribution the system helped create. Correcting selective labels or estimating alternative-policy outcomes can require additional data collection, experimental design, or justified causal assumptions; a generic weighting formula is not a universal repair. Also inspect delayed and revised labels, duplicate examples, and cross-run contamination. Preserve an assessment cohort and selection history that remain meaningful under the new data policy. More frequent fitting cannot remove an unexamined feedback mechanism.

Close the loop with acceptance and observed outcomes

Compare the candidate with the appropriate baseline on compatible mature evidence, including coverage, important slices, resource changes, and fallback. Record whether the request produced a rejected, accepted, activated, or abandoned artifact and why. After activation, continue the defined observation and incident process rather than treating retraining as completion of responsibility. Keep request rates and resource bounds so noisy events cannot create uncontrolled work. The programs here model request coalescing and synthetic label selection only; no model, pipeline engine, queue, feedback correction method, or causal performance claim is implemented.

Common errors and fixes

Every trigger immediately replaces the active model

Why it happens: Candidate construction was confused with acceptance and activation.
How to fix it: Route triggers through training prerequisites and the normal release evidence.
Prevent it next time: Test rejected and insufficient-evidence outcomes.

Several alerts create identical concurrent fits

Why it happens: No request identity or coalescing policy was defined.
How to fix it: Specify snapshot, recipe, replication, and relevant state identity.
Prevent it next time: Test repeated and superseded events under the intended policy.

Reviewed cases are treated as the full population

Why it happens: The current model’s selection mechanism was ignored.
How to fix it: Document label eligibility and assess unsupported or missing outcomes.
Prevent it next time: Keep action, review, and label provenance with each cohort.

Check your understanding

1. Should a new independent replication of recipe-1 on data-2 always be coalesced with the fixture request?

Show worked solution

Not if the experiment intentionally requires another independent draw. Include the relevant replication or random-state identity in the request contract. The fixture coalesces because its declared question is one candidate build for that snapshot and recipe, not because all repeated work is inherently redundant.

2. Why can retraining on only high-score reviewed cases worsen a blind spot?