CI/CD / Promotion and environment contracts / Lesson 12

Release observability and feedback

Connect a candidate to runtime signals without confusing telemetry with causation.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Correlate release identity with metrics, logs, and traces.
Design bounded verification with an explicit missing-data state.

Start with a user-visible question

A release verification plan should ask whether the intended behavior works for the intended users, not merely whether a process exists. For Parcel, label creation success and latency matter more than a green deployment command. Process readiness remains useful but answers a narrower question. Choose a small set of signals that cover critical journeys and failure modes. Declare the observation window and who investigates a violation. An unlimited watch that nobody owns is not a release gate; it is deferred uncertainty.

Use complementary evidence

Metrics summarize rates and distributions, logs retain events, and traces follow work across components. Each has limits: aggregate metrics can hide one tenant’s failure, logs can omit requests, and sampled traces may miss rare problems. Connect signals through stable service and release attributes where appropriate. Avoid sensitive payloads and uncontrolled high-cardinality labels such as every customer address. A release identifier is useful when its cardinality and retention are managed. Keep technical health and business outcomes related without pretending they are interchangeable.

Identify the adopted version

If old and new instances coexist, an overall error rate may hide a regression in the new candidate. Distinguish candidate-specific results and compare equivalent populations when possible. Also record configuration and environment because the same package can behave differently across them. The following event is synthetic and intentionally contains no user address or secret. It illustrates a release annotation that can link operational evidence back to a candidate, not a required OpenTelemetry schema or a complete logging standard.

json A minimal synthetic release-linked event

{"service":"parcel","candidate":"candidate-r8","configuration":"production-c8","event":"label_created","outcome":"success"}

The fields are an application teaching convention. Actual OpenTelemetry resource and log attributes should follow the selected semantic conventions and supported instrumentation version. This record is not sent to a collector.

Treat missing telemetry as unknown

The complete local example requires all declared signals to be present before calling a release observed. A zero error count with zero requests is not a healthy error rate. Missing data can mean no traffic, delayed export, a broken query, or failed instrumentation. Keep those states visible and choose an approved alternate verification path when appropriate. Never auto-expand a rollout because the monitoring backend returned an empty result. Observability infrastructure is part of the evidence chain and can fail independently of the application.

python Avoid a false healthy conclusion

samples = {"requests": 0, "errors": 0}
if samples["requests"] == 0:
    print("unknown: no request evidence")
else:
    print(samples["errors"] / samples["requests"])

Expected output

unknown: no request evidence

The model distinguishes zero observed requests from a zero error fraction. It neither queries monitoring nor establishes a universal minimum sample size.

Correlation guides investigation

An error increase after release is a strong reason to investigate and may justify stopping exposure, but temporal correlation alone does not identify the cause. Check concurrent configuration changes, dependency incidents, traffic shifts, and telemetry changes. Preserve the candidate and timeline rather than repeatedly redeploying until the graph looks better. Use a bounded comparison and explain uncertainty. After recovery, feed the discovered failure mode into tests, release design, or instrumentation so the next pipeline produces more relevant evidence instead of simply adding another dashboard.

Common errors and fixes

No errors are interpreted as healthy despite no requests.

Why it happens: Missing observations were converted into a zero error rate.
How to fix it: Report unknown and obtain appropriate request evidence.
Prevent it next time: Define missing-data handling in the verification contract.

A release tag is added as an unbounded user-specific metric label.

Why it happens: Correlation needs were implemented with uncontrolled cardinality.
How to fix it: Use managed release attributes and keep sensitive identifiers out of metrics.
Prevent it next time: Review label cardinality and retention before instrumentation rollout.

A post-release incident is blamed on the package without investigation.

Why it happens: Time correlation was treated as proof of causation.
How to fix it: Compare configuration, traffic, dependencies, and candidate-specific evidence.
Prevent it next time: Retain a precise release timeline and independent signals.

Check your understanding

1. Overall success is stable, but the new candidate serves only one percent of traffic. What can be hidden?

Show worked solution

A large regression in that small population can disappear inside the aggregate. Compare candidate-specific and relevant cohort signals with adequate observations, while accounting for population differences. Stable aggregate health is useful but insufficient evidence for the new path.

2. The metrics exporter fails during rollout. Should the canary expand?