Observability / Advanced: bias, trust, and evidence engineering / Lesson 14
Asynchronous context and batch relationships
Model producer, queue, consumer, and batch evidence without pretending that one HTTP-shaped parent tree captures every asynchronous relationship.
On this page
Lesson goals
- Separate enqueue acknowledgement from processing completion.
- Compare parent relationships and span links.
- Review context lifetime, retries, and baggage trust.
Asynchronous work has several completion points
An ingestion API can acknowledge a parcel event before a consumer updates the tracking view. The producer request duration, queue residence time, consumer processing duration, and user-visible freshness are separate measurements. A fast enqueue acknowledgement cannot establish timely processing. Draw each state transition and ownership boundary, including retry and dead-letter paths where they exist. Decide which outcome the user promise depends on. Preserve event and acceptance timestamps with their clock limitations. The tracing structure should help explain those transitions rather than force all work into the lifetime of the original HTTP request.
Use relationships that match the operation
A span can have one parent context and can also carry links to other relevant span contexts under the tracing model. Links are useful when one batch consumes work from several producers or when a new operation relates to earlier work without fitting simple nesting. Exact messaging conventions and instrumentation behavior depend on the supported library and convention version. Do not invent a second parent field or merge unrelated trace identifiers to make a picture look connected. The editorial batch record illustrates two input contexts related to one consumer operation without claiming an executable messaging schema.
operation: parcel-batch-consume
parent_context: explicitly-chosen-by-supported-instrumentation
linked_input_contexts: [synthetic-producer-a, synthetic-producer-b]
separate_measures: [queue-residence, processing-duration, update-freshness]
claim_limit: links-express-relationship-not-exact-wall-clock-causation
The named contexts are synthetic labels, not valid serialized span contexts. Actual links use the tracing API and supported messaging conventions.
Separate queue time from processing time
The offline fixture uses one synthetic clock to compare two messages. The first waits eight seconds and processes for two; the second waits one and processes for five. A processing-duration view alone would suggest the second is slower, while its total modeled residence is shorter. Real event systems can have clock skew, redelivery, and missing timestamps. Keep those limitations explicit and do not compute negative queue time into a reassuring number. Define how each attempt and the eventual user-visible operation relate so retries do not inflate independent-operation counts.
messages = [("a", 0, 8, 10), ("b", 0, 1, 6)]
for name, enqueued, started, finished in messages:
print(name, "queue", started - enqueued, "processing", finished - started, "total", finished - enqueued)
a queue 8 processing 2 total 10
b queue 1 processing 5 total 6All timestamps share a synthetic clock. The script models intervals only and neither connects to a broker nor verifies delivery guarantees.
Propagate context deliberately across tasks
Execution context may not automatically follow a newly created thread, task, callback, or message. Use the runtime’s supported capture, activation, injection, and extraction mechanisms, and restore prior context correctly so concurrent work does not inherit another operation’s identity. Test the actual integration rather than assuming synchronous examples prove async behavior. Baggage can carry selected application context, but it is not automatically a span attribute or a trusted authorization claim. Allowlist safe fields and review where they propagate, especially across external services or long-lived messages.
Keep attempts, duplicates, and retained evidence distinct
A redelivered message can produce a new processing attempt for the same logical event. Record attempt meaning without exposing raw customer identifiers or assuming a new attempt is a new satisfied user. Sampling and retention may keep some attempts but omit others, so a trace search cannot establish exactly-once behavior. Business idempotency and data reconciliation need their own verified contracts. This lesson executes only synthetic timing arithmetic and presents an editorial relationship model; no broker, worker, async runtime integration, OpenTelemetry API, or message-delivery mechanism is exercised.
Common errors and fixes
The enqueue span is used as end-to-end freshness
- Why it happens
- Acknowledgement and processing completion were conflated.
- How to fix it
- Measure queue and consumer boundaries separately.
- Prevent it next time
- Document each completion promise.
Concurrent tasks share the wrong trace identity
- Why it happens
- Context activation or restoration leaked across work.
- How to fix it
- Use supported scoped context handling and integration tests.
- Prevent it next time
- Include concurrent and callback cases in verification.
A batch invents several parent fields
- Why it happens
- The relationship model was forced into a tree.
- How to fix it
- Use supported parent and link semantics.
- Prevent it next time
- Review messaging conventions and version assumptions.
Check your understanding
1. A consumer processes a batch from five producer requests. Must it choose one and erase the rest?
Show worked solution
No. The actual instrumentation can use the supported parent relationship and links to represent relevant input contexts. Follow the messaging conventions and library behavior for the operation type. Keep batch processing, per-message outcomes, and queue timing clear; links alone do not establish ordering or successful processing.
2. Can a complete-looking trace prove a message was processed exactly once?
Show worked solution
No. Instrumentation, sampling, export, and retention can omit attempts, and trace identity is not a durable business deduplication mechanism. Verify idempotency and state invariants through the application’s authoritative evidence. Use traces to investigate execution, with their coverage limitations stated.
Key takeaways
- Async completion has several distinct boundaries.
- Links can express relationships beyond one parent tree.
- Context propagation and business deduplication solve different problems.