Observability / Advanced: bias, trust, and evidence engineering / Lesson 17

Telemetry contract tests and migrations

Protect signal meaning through SDK, convention, collector, and query changes with deterministic fixtures and explicit evidence boundaries.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Choose meaningful assertions for telemetry behavior.
Separate schema compatibility from semantic equivalence.
Design a staged consumer migration.

Test the meaning that users depend on

A telemetry contract includes event population, observation boundary, units, identity, permitted attributes, timestamp meaning, and failure behavior. Tests should protect those semantics rather than merely snapshot every incidental field. A completed-update event after a failed commit is a meaningful regression; a harmless change in attribute ordering may not be. Create synthetic success, failure, timeout, duplicate, missing, and forbidden-field cases. Decide which layer can answer each claim: a pure classifier can test event semantics, while SDK integration or collector configuration requires separate implementation evidence. Keep the expected result independent enough to catch the intended defect.

A renamed field may change more than spelling

Semantic conventions and instrumentation packages evolve under their documented stability rules. A new field name may accompany a changed meaning, requirement level, or unit. Do not infer compatibility solely from a successful parse or a schema label. Review actual versions, release notes, and supported migration mechanisms. If old and new streams coexist, consumers must avoid double-counting them. A migration can preserve meaning through an explicit mapping only when the source and target definitions truly match. Historical data remains under its original contract unless a verified transformation changes it with an auditable boundary.

Validate a narrow unit migration explicitly

The offline fixture converts a synthetic millisecond field into seconds while preserving an operation name. It checks a known value and rejects an input that supplies both old and new duration fields, avoiding an ambiguous choice. This is a pure dictionary migration, not an OpenTelemetry schema translator. Real migrations need missing-field, type, range, identity, and consumer checks as well. State whether a transformation changes only representation or also the measured population. A neat mapping function cannot prove equivalence when the old event was observed before commit and the new event after commit.

python Offline unit-migration contract fixture

def migrate(record):
    if "duration_ms" in record and "duration_seconds" in record:
        raise ValueError("ambiguous-duration-fields")
    return {"operation": record["operation"], "duration_seconds": record["duration_ms"] / 1000}
converted = migrate({"operation": "parcel.store", "duration_ms": 250})
assert converted["duration_seconds"] == 0.25
print(converted["operation"], converted["duration_seconds"])
try:
    migrate({"operation": "parcel.store", "duration_ms": 250, "duration_seconds": 0.25})
except ValueError as error:
    print(error)

Expected output

parcel.store 0.25
ambiguous-duration-fields

The function supports one stated legacy shape only. It proves the synthetic conversion and ambiguity guard, not SDK or collector schema compatibility.

Migrate producers and consumers deliberately

The editorial plan below reviews old and new definitions, tests consumers, limits overlap, and preserves rollback assumptions. Update dashboards, alerts, recording rules, exports, and documentation together where they rely on the changed contract. A producer rollout can be mixed across instances, so consider the interval when both versions are present. Do not remove the old query path before its data and consumers are accounted for. Compatibility is a tested property of the chosen versions and mapping, not a promise that every OpenTelemetry component or backend can transform every schema automatically.

yaml Illustrative telemetry migration plan

migration_review: parcel-duration-contract
compare: [population, boundary, unit, identity, stability-status]
fixtures: [known-value, missing-field, ambiguous-fields, forbidden-field]
consumer_updates: [queries, dashboards, alerts, exports]
overlap_policy: explicit-no-double-counting
rollback_limit: preserve-version-and-mapping-evidence

This record is a review plan, not an automatic migration tool. It changes no instrumentation, collector, stored data, or query.

Report local, integration, and operational evidence separately

An isolated unit fixture can run safely without external services and gives precise evidence for its input-output rule. An integration test may verify serialization, context activation, or processor syntax for supported versions. Operational evidence is needed for throughput, retention enforcement, recovery, and access behavior in an actual environment. Keep those claims separate and retain failures and untested conditions. This lesson executes only the pure migration fixture; it installs no SDK or test package, starts no collector, and validates no live query. Reliable telemetry testing protects meaningful contracts while avoiding a false sense of completeness.

Common errors and fixes

A parser pass is called semantic compatibility

Why it happens: Only field shape was tested.
How to fix it: Compare population, boundary, units, and identity.
Prevent it next time: Include known-value and failure-path contract fixtures.

Old and new streams double a total

Why it happens: Overlap was not handled during rollout.
How to fix it: Define one compatible counting path and migrate consumers.
Prevent it next time: Test mixed-version intervals.

A local fixture is advertised as end-to-end proof

Why it happens: Evidence boundaries were omitted.
How to fix it: State precisely which code and inputs ran.
Prevent it next time: Separate model, integration, and operational results.

Check your understanding

1. A metric changes from attempted writes to committed writes but keeps the same name. Is it a harmless rename?

Show worked solution

No. The population and completion boundary changed even though spelling did not. Version the semantic transition, update consumers, and avoid comparing histories as if they measured the same event set. Add fixtures for failed and unknown commits so the new meaning remains explicit.

2. Why reject records containing both old and new duration fields in the toy migration?