DevOps / Intermediate: controlled delivery systems / Lesson 10

Observability and useful telemetry

Connect user symptoms to traces, metrics, and logs while controlling telemetry volume, identity, and sensitive data.

About 4 min readContent reviewed 2026-10-25

On this page

Lesson goals

Connect metrics, logs, and traces to user questions.
Control context, cardinality, and sensitive fields.
Design actionable alerts and bounded telemetry pipelines.

Begin with questions about behavior

Observability helps a team investigate a system through its emitted signals and context. Instrumentation should answer practical questions: which user operation failed, where time was spent, and what changed near the incident. A dashboard full of infrastructure numbers may miss a failed business journey. Define request and operation boundaries before selecting a backend. OpenTelemetry supplies instrumentation, context propagation, and signal collection interfaces; it is not by itself a complete storage, alerting, or incident response product. Verify the supported SDK and exporter versions for the language and backend actually selected.

Use complementary signals

Metrics summarize measurements over populations and time; logs record discrete events; traces connect spans within an operation across boundaries. A trace can show an upstream wait while a metric shows whether that wait affects many users. The first example is a synthetic structured event containing a route template and revision, with no credential or user payload. Keep field names and units consistent. Do not attach a unique request identifier as a metric label simply because it is useful in logs; that can create an unbounded number of time series and overwhelm the monitoring system.

json Synthetic structured diagnostic event

{"event":"request_completed","service":"booking-api","route":"/bookings/:id","status_class":"5xx","duration_ms":240,"revision":"lab-build-42","request_id":"lab-request-a"}

The route template supports bounded aggregation; request_id belongs in correlation rather than a metric label. All identifiers are synthetic and no actual request is recorded.

Propagate context without granting trust

A trace or request identifier connects observations but does not authenticate a caller. Validate incoming context according to the protocol and do not put credentials, email addresses, or arbitrary large values into baggage. At trust boundaries, decide which fields may be propagated and who can see them. Sampling reduces collection cost but changes which investigations are possible; a missing trace does not prove no request occurred. Resource attributes should identify a service and deployment consistently, while avoiding values that change per request. Plan correlation between deployed revision, configuration version, and incident time.

Control collector behavior

A telemetry pipeline receives, processes, and exports signals. The second example sketches a Collector pipeline using established component names, but it is a non-runnable fragment because receivers, exporter endpoints, authentication, and resource policy depend on the chosen distribution and environment. Bound queues and memory, batch deliberately, and define what happens when the backend is unavailable. A full telemetry queue must not consume all application resources or silently retain secrets. Test redaction and failure behavior in isolation, and review high-cardinality fields before enabling broad instrumentation.

yaml OpenTelemetry Collector pipeline fragment

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp]

The fragment shows stage order only. Component definitions, memory settings, endpoints, TLS, and credentials are deliberately absent, so it must not be used as a deployable configuration.

Alert on actionable user impact

Choose alerts that identify sustained user impact or imminent resource exhaustion and include an owner, severity, and diagnostic entry point. Avoid paging on every isolated log line or every harmless retry. Compare error and latency signals with deployment events and dependency health, then test the proposed explanation. Retain enough data for the response window under a clear access and deletion policy. Instrumentation is part of the application’s operational contract and deserves code review, performance budgeting, and security review. More emitted data is not automatically more understanding; useful context and reliable interpretation matter more.

Common errors and fixes

Metrics cardinality grows without limit

Why it happens: Unique IDs or complete URLs became labels.
How to fix it: Use bounded route and status dimensions.
Prevent it next time: Review label budgets and monitor series growth.

A trace ID is trusted as identity

Why it happens: Diagnostic context was confused with authentication.
How to fix it: Authenticate and authorize through the security boundary independently.
Prevent it next time: Treat incoming telemetry context as untrusted input.

Telemetry failure degrades the application

Why it happens: Export queues and memory had no failure policy.
How to fix it: Bound buffering and choose documented loss or retry behavior.
Prevent it next time: Test backend unavailability in an isolated environment.

Check your understanding

1. Where should a unique booking ID appear: a latency metric label or a controlled diagnostic record?

Show worked solution

Use a bounded route template for the metric. Include an approved identifier only in access-controlled logs or traces when needed and permitted. This preserves aggregation while avoiding a series per booking and limiting sensitive correlation exposure.

2. No trace exists for a failed request. Can the incident be closed as a client mistake?