Observability / Basic: signals and measurement meaning / Lesson 6
Collection pipelines and visible data loss
Map telemetry from instrumentation to query and distinguish source silence, intentional filtering, buffering, export failure, and backend visibility.
On this page
Lesson goals
- Explain receiver, processor, and exporter responsibilities.
- Reconcile synthetic stage counts with explicit loss.
- Define freshness and independent pipeline health.
Draw every transformation boundary
Telemetry passes through several boundaries before a query can display it: instrumentation, SDK aggregation or buffering, transport, collection, processing, export, storage, and query selection. Not every deployment has every stage, and some stages are repeated. A collector can receive signals, apply configured processors, and export them through configured pipelines. Component availability and configuration syntax depend on the actual distribution and version. Treat a pipeline diagram as a model to verify, not evidence that every signal follows that path. Record where filtering, sampling, attribute removal, and temporality conversion can change the data.
Define what each stage accepts and emits
For each boundary, record signal type, accepted format, destination, expected transformations, failure handling, and ownership. A receiver being declared in a configuration does not by itself prove that it participates in an active service pipeline. A successful transport acknowledgement may mean accepted into a queue rather than durably stored and queryable. The editorial record below separates those states and names the required observations. It is intentionally not a runnable collector configuration: no endpoints, credentials, storage paths, or environment-specific component versions are supplied.
pipeline_review: parcel-telemetry
stages: [accepted, processed, queued, exported, queryable]
intentional_drop_policy: bounded-and-counted
retry_policy: finite-and-observable
freshness_measure: source-event-to-query-visibility
health_owner: telemetry-platform-team
The record distinguishes transport progress from query visibility. It is an editorial contract rather than OpenTelemetry Collector YAML.
Count intentional and unexpected loss separately
The offline example starts with one thousand accepted records, removes one hundred intentionally under policy, and observes twenty export failures from the remaining set. Eight hundred eighty reach the modeled destination. Reporting only an eighty-eight-percent delivery fraction would conceal the difference between expected filtering and unexpected failure. Real asynchronous pipelines also have in-flight and delayed records, so stage counts may not balance within one instant. Align windows and account for queue changes before declaring loss. Counter resets, retries, and duplicate exports can further complicate reconciliation.
accepted = 1000
policy_dropped = 100
export_failed = 20
exported = accepted - policy_dropped - export_failed
assert accepted == policy_dropped + export_failed + exported
print("exported", exported)
print("unexpected_loss_fraction", f"{export_failed / (accepted - policy_dropped):.2%}")
exported 880
unexpected_loss_fraction 2.22%The unexpected fraction uses the 900 records eligible after intentional filtering. The model assumes no queued or duplicated records.
Recognize buffering and backpressure limits
Buffering can absorb a short interruption, but finite memory or persistent storage eventually fills when arrivals exceed successful exports. Retry behavior can delay visibility and consume resources, while a restart may lose in-memory work depending on implementation. Define what happens when buffers are full and how that condition becomes visible. A healthy application can become harder to diagnose when its telemetry pipeline is overloaded. Avoid making business requests wait indefinitely for telemetry export. Performance and durability tradeoffs require the actual SDK and collector behavior, not a universal queue-size recommendation.
Observe the observer with a separate question
Pipeline health asks whether expected telemetry is complete and timely enough to support decisions. Inspect accepted and rejected volume, queue pressure, export errors, processing failures, and data age under the actual component’s documented semantics. Keep a way to recognize when the normal query path itself is blind. Do not interpret an empty chart as zero service errors until collection and selection are checked. This course uses only local count arithmetic and editorial designs; it does not run a collector, open a port, send telemetry, fill a queue, or verify backend durability.
Common errors and fixes
An acknowledgement is reported as durable storage
- Why it happens
- Transport acceptance and backend persistence were conflated.
- How to fix it
- Identify the acknowledged boundary and verify later stages separately.
- Prevent it next time
- Document acceptance states and durability limitations.
Missing data is blamed on instrumentation immediately
- Why it happens
- Filtering, buffering, export, and query stages were skipped.
- How to fix it
- Trace the signal through each owned boundary.
- Prevent it next time
- Maintain stage-level counts and freshness evidence.
A buffer is treated as unlimited outage protection
- Why it happens
- Finite resources and sustained backlog were ignored.
- How to fix it
- Define overflow behavior and bounded recovery expectations.
- Prevent it next time
- Review capacity and failure semantics for the actual version.
Check your understanding
1. Accepted records rise, export volume falls, and the queue grows. Has all missing data been lost?
Show worked solution
Not necessarily. Some records may still be buffered or retrying, and the observed interval may end before they export. Compare queue change, failure and drop counters, timestamps, and later visibility using compatible windows. Do not declare durable safety either; inspect the implementation’s persistence and overflow behavior.
2. Why should intentional filtering be counted?
Show worked solution
It makes the data contract auditable and distinguishes expected reduction from a broken pipeline. A new filter can otherwise look like an application traffic drop. Record the policy, bounded reasons, affected signal, and revision so downstream consumers understand which population remains and why.
Key takeaways
- Pipeline acceptance has several distinct boundaries.
- Account for policy drops, failures, and in-flight work.
- Telemetry freshness and completeness need explicit ownership.