SRE / Basic: promises and measurements / Lesson 5
Telemetry for user impact and diagnosis
Connect traffic, errors, latency, and saturation to a user-facing symptom, using bounded labels and complementary evidence rather than a wall of charts.
On this page
Lesson goals
- Separate symptoms from possible causes.
- Choose metrics, logs, and traces by question.
- Preserve units, scope, and telemetry health.
Ask four concrete questions
Traffic asks how much work the service handles; errors ask how much work fails; latency asks how long outcomes take; saturation asks how close a constrained resource is to its limit. For Library reservations, requests per second, bad eligible fraction, confirmation delay, and worker-queue occupancy form a useful starting set. These signals are related but not interchangeable. Low CPU does not prove spare capacity when every worker waits on a database connection. Define the resource and boundary behind each saturation measure before deciding that a service needs more machines.
Choose the evidence for the question
Metrics summarize populations and support trend or threshold decisions. Logs describe selected discrete events, and traces connect spans of a distributed operation. A metric can reveal that reservations slowed after a release, a trace can localize time spent waiting for storage, and a bounded structured log can explain a rejected input. Sampling means a missing trace is not proof that an operation never occurred. Correlation identifies a place to investigate, not causation by itself. Preserve a safe request correlation identifier where useful, while avoiding secrets and unrestricted user content in telemetry.
Bound dimensions deliberately
A label dimension creates separate time series for its values. Combining region, route family, and outcome can remain manageable when each value set is bounded; adding request IDs or email addresses can create explosive cardinality and privacy exposure. The offline model multiplies the declared dimension counts to estimate a simple upper bound. Real series counts also depend on metric families, histogram buckets, and lifecycle churn. A low count is not automatically good: removing every useful dimension can hide a regional failure. Keep diagnostic usefulness and cost visible in the instrumentation review.
dimensions = {"region": 3, "route_family": 8, "outcome": 4}
series = 1
for count in dimensions.values():
series *= count
print(series)
print(series * 100_000)
96
9600000The bounded labels permit 96 combinations. Adding 100,000 distinct request IDs could create 9,600,000 combinations before metric-family overhead.
Build a decision-oriented view
Start a service dashboard with the user SLI, current impact, demand, and relevant saturation, then link deeper dependency views. Include units, aggregation, time range, freshness, and release annotations. The editorial record below ties each panel to a question instead of selecting attractive graphs first. A percentage without its denominator can mislead during low traffic; an aggregate without location can conceal a partial outage. Keep a visible distinction between zero, missing, and stale observations. A dashboard is useful when an unfamiliar responder can explain the situation and choose a next observation.
view: reservation-triage
panels:
- question: are-eligible-users-failing
measure: bad-fraction-and-eligible-count
- question: where-is-work-waiting
measure: queue-age-and-in-flight-count
metadata: [utc-range, units, freshness, release-id]
Each panel has a diagnostic purpose. This YAML is an editorial design record, not a Grafana dashboard definition.
Observe the observer
Collection failures, dropped samples, and stale rule evaluations can undermine every downstream decision. Track the telemetry path independently and make coverage limitations visible during incidents. A sudden fall in errors deserves suspicion if request counts and emitter health disappear simultaneously. Retain only the detail needed for an operational question and apply access and retention controls appropriate to that data. Before expanding instrumentation, identify who will use it, which decision it changes, and how it will be verified. More telemetry can increase costs and risk without improving the ability to recognize or explain user harm.
Common errors and fixes
Low CPU is used to dismiss overload
- Why it happens
- The constrained resource was a connection pool.
- How to fix it
- Inspect waiting and capacity at the actual bottleneck.
- Prevent it next time
- Define saturation for every important constrained resource.
Telemetry cost grows after adding correlation IDs
- Why it happens
- Unbounded values became metric labels.
- How to fix it
- Keep bounded dimensions and move safe IDs to suitable event evidence.
- Prevent it next time
- Review cardinality and privacy before instrumentation changes.
A quiet dashboard hides an emitter failure
- Why it happens
- Missing data was displayed as zero.
- How to fix it
- Expose freshness and collection health.
- Prevent it next time
- Test absence as a distinct condition.
Check your understanding
1. Errors rise only in one region while aggregate success remains acceptable. What should the view show?
Show worked solution
Keep the service-wide view but add a bounded regional breakdown with eligible counts and timestamps. Compare traffic mix and release exposure before attributing the problem. Escalate according to actual regional user impact rather than dismissing it because the global average is healthy.
2. A trace shows a slow database span. What has been proved?
Show worked solution
The sampled operation spent recorded time in that span under the instrumentation semantics. It suggests a boundary for investigation, but does not establish a database-engine defect or represent all requests. Compare population metrics, connection waiting, deployment timing, and additional safe observations.
Key takeaways
- Use each signal to answer a named question.
- Bound labels while retaining useful failure slices.
- Telemetry absence is an operational condition.