Observability / Basic: signals and measurement meaning / Lesson 1

Observability starts with questions

Choose telemetry from an explicit diagnostic question and explain the difference between observing a signal, forming a hypothesis, and establishing a cause.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Connect a user symptom to a system boundary.
Select complementary evidence for a question.
State uncertainty and instrumentation limitations.

Begin with a decision someone needs to make

Observability helps people infer internal system behavior from available evidence. Start with a concrete question: why do parcel-tracking updates reach users late? A chart of CPU usage cannot answer it alone. Draw the path from an accepted scan event through processing, storage, and the tracking response. Identify where delay can accumulate and what observation would distinguish those locations. The useful output is a better decision, such as investigating ingestion lag rather than tuning the response renderer. Buying a telemetry backend does not automatically create the context or instrumentation needed to answer an unfamiliar question.

Use each signal for its strengths

Metrics summarize many observations into a time series or distribution. Logs preserve selected event details, and traces relate units of work across a request path. A rise in update age can identify population impact; a structured event can explain why one update was rejected; a trace can reveal the time spent waiting for a storage call. These views complement rather than replace each other. Their clocks, sampling, retention, and populations can differ. A missing trace is not proof that an operation never happened, and a plausible log message is not proof that all users saw the same behavior.

Write a small evidence contract

The editorial record below connects one question with its evidence and blind spot. Record units, observation boundary, eligible population, and data freshness beside the signal. For update freshness, time since a parcel event occurred differs from time since the service received it; confusing those timestamps can send the investigation to the wrong team. Keep event identifiers in suitable restricted event evidence rather than as unbounded metric labels. A small explicit contract allows future responders to understand what the measurement can and cannot establish without interviewing its original author.

yaml Illustrative evidence contract

question: why-tracking-updates-are-late
metric: accepted-event-processing-delay-seconds
event_evidence: bounded-rejection-reason
trace_boundary: ingest-to-store-operation
blind_spot: delay-before-service-acceptance
owner: parcel-observability-team

The record is an editorial design, not OpenTelemetry configuration. It makes upstream delay an explicit blind spot instead of claiming end-to-end coverage.

Keep measured intervals distinct

The offline example separates event-to-arrival delay from arrival-to-processing delay using synthetic times on one clock. The total is thirty seconds, but twenty-five occurred before service arrival. If the processing team measured only its own five seconds, both statements could be correct. Real distributed clocks require additional care; subtracting timestamps from different clocks can introduce skew. Preserve timestamp meaning and source. This arithmetic is a model for reasoning about boundaries, not a trace collector, a clock synchronization test, or a measurement of a running parcel system.

python Offline delay-boundary model

event_time = 100
arrival_time = 125
processed_time = 130
print("upstream_delay", arrival_time - event_time)
print("processing_delay", processed_time - arrival_time)
print("total_delay", processed_time - event_time)

Expected output

upstream_delay 25
processing_delay 5
total_delay 30

All values are synthetic seconds on one clock. The result localizes the modeled delay but establishes no real network or service cause.

Improve the questions and the instruments together

When evidence cannot distinguish plausible explanations, add the narrowest safe observation that would help. Do not respond by logging every payload or labeling every customer. Review whether the new detail changes a decision and whether its cost and privacy exposure are justified. During an incident, label hypotheses and uncertainty so correlation is not reported as causation. Afterward, preserve a small regression fixture for important event semantics. Observability matures through a feedback loop between real questions, instrumentation, interpretation, and engineering changes, with explicit ownership of the telemetry path itself.

Common errors and fixes

A dashboard exists but cannot answer the incident question

Why it happens: Signals were selected before decisions and boundaries.
How to fix it: Write the question and map the relevant path.
Prevent it next time: Review every panel against a diagnostic purpose.

A processing metric disproves an upstream complaint

Why it happens: Different intervals were compared as if identical.
How to fix it: Align timestamp meaning and observation boundaries.
Prevent it next time: Document blind spots and data freshness.

A correlated release is declared the cause

Why it happens: Temporal association replaced a tested mechanism.
How to fix it: State a hypothesis and seek distinguishing evidence.
Prevent it next time: Keep facts and interpretations separate.

Check your understanding

1. Users see old parcel states but the tracking API is fast. Which question comes next?

Show worked solution

Ask whether the latest accepted event has been processed and whether newer events reached the service at all. Fast response latency measures serving behavior, not data freshness. Compare event time, acceptance time, and processing time with explicit clock limitations before assigning responsibility.

2. Why not collect all event payloads to avoid future blind spots?