MLOps / Observation, recovery, and lifecycle decisions / Lesson 13

Service and model observability

Observe delivery health, supported coverage, and predictive outcomes as separate properties.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Define metric populations, time windows, and units.
Separate operational success from model coverage and quality.
Use bounded telemetry dimensions and actionable diagnostic context.

Observability starts with questions and boundaries

Ask whether eligible requests are completed on time, whether supported inputs reach the expected model version, whether fallbacks increase, and whether mature labeled outcomes meet the task’s quality requirements. These are different questions and can fail independently. A healthy process can return invalid predictions, while a strong model can be inaccessible behind a failing delivery path. Define what counts as eligible, successful, late, failed, deferred, or fallback before building a dashboard. Preserve the measurement window and denominator so a change in population cannot masquerade as an improvement in reliability.

Latency distributions need conventions and counts

The first fixture compares a mean with a nearest-rank tail statistic from four synthetic values. It exposes how one long response can be hidden by an average and states the quantile convention explicitly. Four observations cannot characterize a production tail reliably. Real histograms, summaries, and aggregations have different approximation and combination properties, so choose a representation according to the question and tool semantics. Include queue and feature work if they belong to the service boundary. Do not average per-instance percentiles and call that a fleet percentile without a justified method.

python Deterministic offline standard-library fixture

from math import ceil
latencies = [10,20,30,100]
ordered = sorted(latencies)
p95 = ordered[ceil(0.95*len(ordered))-1]
print(f"mean_ms={sum(ordered)/len(ordered):.1f}")
print("nearest-rank p95_ms", p95)
print("observations", len(ordered))

Expected output

mean_ms=40.0
nearest-rank p95_ms 100
observations 4

The fixture states its nearest-rank convention and tiny count; it does not estimate a production tail or run a latency benchmark.

Coverage and operational success are not accuracy

The second fixture counts a fallback as a service success under its declared policy, while only a direct model result contributes to model coverage. That can be legitimate if the fallback is an accepted response, but the policy and its outcome limits must be visible. A rising fallback rate can preserve availability while changing decision quality or burdening human review. Predictive accuracy requires appropriate target evidence and timing; request success cannot substitute for it. Report eligible counts, model and fallback versions, unsupported input reasons, and relevant slices under the data-handling policy.

python Deterministic offline standard-library fixture

events = [{"eligible":True,"status":"predicted"},
          {"eligible":True,"status":"fallback"},
          {"eligible":True,"status":"failed"},
          {"eligible":False,"status":"invalid"}]
eligible = [event for event in events if event["eligible"]]
operational_success = sum(event["status"] in {"predicted","fallback"} for event in eligible)
model_predictions = sum(event["status"] == "predicted" for event in eligible)
print(f"declared service success={operational_success/len(eligible):.2f}")
print(f"model coverage={model_predictions/len(eligible):.2f}")
print("eligible count", len(eligible))

Expected output

declared service success=0.67
model coverage=0.33
eligible count 3

The explicit fixture counts fallback as an operational success while distinguishing it from a model prediction; other tasks can define another documented policy.

Telemetry dimensions have a cost and disclosure contract

A bounded status or model-family label supports aggregation; an unrestricted request id or raw feature value as a metric label can create enormous cardinality and expose unnecessary data. Use appropriate logs or traces with controlled access and retention for detailed correlation, keeping secrets and private payloads out of routine telemetry. Link observations to the exact resolved artifact where needed, but design version labeling and retention deliberately rather than assuming every new dimension is free. Instrument the evaluator and feature path as well as the endpoint so the first failing boundary can be investigated.

Alerts should lead to an owned response

Choose alert conditions and observation windows that correspond to a meaningful risk or service objective, with a documented first diagnostic action and fallback. A single noisy slice or a missing signal needs interpretation rather than automatic retraining. Monitor telemetry delivery itself: no recorded errors can mean no traffic or a broken collector. Test metric logic with small known fixtures and then validate collection and operational behavior in the actual supported system. This course computes fixed lists and counts only; it does not deploy monitoring, scrape metrics, generate traffic, run tracing, or claim measured production reliability.

Common errors and fixes

Availability is reported as predictive accuracy

Why it happens: Successful responses were counted without target outcomes.
How to fix it: Measure quality on aligned mature labels separately from delivery status.
Prevent it next time: Keep service, coverage, fallback, and model-quality definitions distinct.

The average looks healthy while some requests are very slow

Why it happens: Only a mean was tracked or the timing boundary was too narrow.
How to fix it: Inspect an appropriate distribution with counts and complete scope.
Prevent it next time: Test tail conventions and aggregation behavior explicitly.

Metrics create one time series per request

Why it happens: Unbounded identifiers were used as metric labels.
How to fix it: Use bounded aggregation dimensions and a controlled detail channel.
Prevent it next time: Review cardinality, disclosure, retention, and correlation needs together.

Check your understanding

1. Why are service success 0.67 and model coverage 0.33 both correct in the fixture?

Show worked solution

There are three eligible events. One model response and one fallback satisfy the declared service-success policy, giving two thirds. Only the model response counts toward coverage, giving one third. These are different measurements, and neither is predictive accuracy because no target outcomes were supplied.

2. A dashboard has zero errors after an exporter fails. What should be checked?