Observability / Intermediate: instrumentation and interpretation / Lesson 7
Instrumentation boundaries and ownership
Combine framework instrumentation with deliberate business observations while avoiding duplicate spans, ambiguous completion events, and hidden SDK assumptions.
On this page
Lesson goals
- Compare automatic and code-based coverage.
- Place observations at meaningful state transitions.
- Review instrumentation ownership and compatibility.
Automatic coverage has a defined scope
Automatic or zero-code instrumentation can observe supported framework, runtime, and library boundaries without adding every observation manually. It does not automatically know that a parcel scan became a durable tracking update or that a business rule rejected a state transition. Code-based instrumentation can add those meanings, but it requires maintenance and careful placement. Review the exact runtime, library version, supported hooks, and semantic-convention behavior. Treat the two approaches as complementary. An instrumented HTTP request is evidence about that request boundary; it is not a universal record of every business step performed behind it.
Observe the state transition you claim
Choose observation points that distinguish accepted, attempted, committed, published, and visible states. A metric incremented before persistence may measure attempts, even if its name says completed updates. In the synthetic state table, a storage failure follows an accepted scan, so the completed count must remain zero. Include failure and cancellation paths in the design review. Do not emit the same completion event from both a shared library and its caller unless the populations are intentionally different and clearly named. Duplicate instrumentation can inflate counts while looking reassuringly detailed.
operation: parcel-scan-processing
state_sequence: [accepted, storage-attempted, storage-failed]
accepted_event: emit-on-acceptance
completed_event: emit-only-after-required-commit
expected_completed_count: 0
owner_boundary: application-business-transition
The record demonstrates a completion boundary after the required commit. It is not application code or a tracing SDK configuration.
Test semantic events with a pure model
The offline function always records acceptance for an eligible synthetic scan, then records completion only when the modeled persistence result succeeds. The two calls produce one accepted-only path and one accepted-and-completed path. This checks the intended classification without simulating storage. A real implementation also needs to consider exceptions, process crashes, duplicate delivery, and partial outcomes. Separate a business event contract from the mechanism used to export telemetry; an exporter failure should not silently change the business state or redefine whether processing actually completed.
def observations(persisted):
events = ["scan.accepted"]
if persisted:
events.append("scan.completed")
return events
assert observations(False) == ["scan.accepted"]
assert observations(True) == ["scan.accepted", "scan.completed"]
print(observations(False))
print(observations(True))
['scan.accepted']
['scan.accepted', 'scan.completed']The boolean is a synthetic outcome, not a storage response. Only the emitted event sequence is validated.
Identify the instrumenting library separately
OpenTelemetry instrumentation scope identifies the library or component producing telemetry, while resource identity describes the entity being observed. Keeping those concepts separate helps explain why two libraries emit similar operations or why an instrumentation upgrade changes attributes. Record compatible versions and ownership of custom observations. A shared instrumentation wrapper can enforce safe names and fields, but an overly rigid wrapper can hide necessary signal semantics. Review both the business contract and the actual SDK behavior. Never assume every language implementation supports identical features or defaults just because the conceptual API looks similar.
Ship instrumentation as behavior that needs review
Instrumentation can add CPU, allocation, synchronization, network work, and privacy exposure. Review hot paths and bounded attribute values before adding detail, then obtain appropriate isolated performance evidence when needed. Use error handling that preserves the applicationโs intended behavior without hiding telemetry failures from operators. Track semantic changes and update queries, dashboards, and tests together. This course executes a pure event-classification fixture only; it installs no instrumentation agent, attaches to no process, and makes no runtime-specific overhead or context-coverage claim. Those require the actual supported environment and explicit evidence.
Common errors and fixes
Completed updates include failed writes
- Why it happens
- Instrumentation ran before the required commit.
- How to fix it
- Move or rename the observation to match the state boundary.
- Prevent it next time
- Test failure and partial-completion semantics.
Every dependency call appears twice
- Why it happens
- Automatic and manual instrumentation overlap.
- How to fix it
- Inspect scope and ownership before removing duplicate coverage.
- Prevent it next time
- Review integration changes with representative traces.
An SDK upgrade silently breaks queries
- Why it happens
- Attribute and convention changes were not tracked.
- How to fix it
- Review supported versions and migrate consumers deliberately.
- Prevent it next time
- Pin and document the instrumentation contract.
Check your understanding
1. An HTTP library already emits a client span. Should business code always create another identical span?
Show worked solution
No. First identify the diagnostic question and existing coverage. A distinct business operation may deserve its own span, but duplicating the same transport boundary can distort interpretation and cost. Keep scopes and names meaningful and verify integration behavior in the actual supported runtime.
2. What should a custom completed-update metric mean after an acknowledgement but before durable storage?
Show worked solution
It should not claim durable completion at that point. Either observe acceptance with a truthful name or emit completion only after the required state transition. Include unknown and partial outcomes in the contract, and ensure telemetry export behavior does not redefine the business result.
Key takeaways
- Automatic coverage does not replace business semantics.
- Observation placement determines truthful event meaning.
- Version instrumentation and consumers as one contract.