Observability / Editorial solution guide
Dashboards disagree about the same service
Reconcile population, weighting, time, units, histogram semantics, and display transformations before deciding which chart represents the user question.
Read-only knowledge base ยท No public comments
Recognize the symptoms
Two panels show different success percentages or latency values for apparently the same service. An alert fires while the overview is green, or a detail view appears to contradict a service total. Both queries can be syntactically valid and still answer different measurement questions.
Understand the likely causes
- Hidden filters, unequal traffic weighting, different intervals, stale points, mixed units, incompatible histogram buckets, label joins, sampling, or panel reductions can change the result. A service rename or semantic migration may also split history. Visual similarity and a shared metric name do not establish equivalent definitions.
Write both measurement definitions side by side
Record source, environment, service identity, variables, operation and outcome filters, interval, evaluation time, units, and displayed reduction for each panel. Preserve the exact query revision without including credentials. State the intended user question before choosing a preferred answer. A latency percentile, average, and threshold fraction are different quantities even when all panels say response time. An overview may intentionally use a long accounting window while an alert examines recent harm, so disagreement is not automatically a defect.
Check the population arithmetic
The offline fixture uses two cohorts with very different traffic. Averaging their success percentages reports ninety percent, while combining counts reports ninety-eight percent. The difference comes entirely from weighting, not missing records. Validate the intended numerator and denominator and check that their selectors align. Do not solve a mismatch by averaging the two displayed answers. If dimensions are joined with metadata, inspect key uniqueness and unmatched records, because a one-to-many match can duplicate a count while the final percentage still looks plausible.
cohorts = [(8, 10), (90, 90)]
mean_percentage = sum(good / total for good, total in cohorts) / len(cohorts)
combined = sum(good for good, total in cohorts) / sum(total for good, total in cohorts)
print("mean_percentage", f"{mean_percentage:.0%}")
print("combined", f"{combined:.0%}")
mean_percentage 90%
combined 98%The same 98 good events out of 100 eligible events yield 98% under event weighting. Equal cohort weighting answers a different question.
Align time and freshness before comparing
Compare UTC intervals, query step, rate window, evaluation delay, point age, and late-data behavior. A last-value stat can display a stale point while a time series shows a gap. A longer rate window can smooth a short incident, and a rolling window can improve when old failures leave. Check timestamp meaning when logs use observation time and metrics use event time. Keep no traffic, zero, missing, and stale distinct. Do not claim that one panel disproves a user report until its observation boundary actually covers the reported experience.
Inspect units, buckets, and rendering
Verify whether durations are stored in seconds or milliseconds and whether the panel only changes the display label. For histograms, check cumulative versus disjoint buckets, compatible boundaries, and percentile estimation. Do not average per-instance percentiles to repair a service percentile. Inspect transformations and null mappings after the query because the rendered number can differ from the raw result. Choose the correction supported by the intended contract, document semantic migrations, and update every consumer that relied on the old interpretation rather than making one chart agree cosmetically.
Verify the fix
Use small independent fixtures for uneven traffic, no traffic, missing or stale inputs, known durations, and known histogram observations. Compare raw query output with rendering under the same scope. Explain any intentional difference between overview, alert, and diagnostic views. State that this guide executed only the local ratio fixture and did not query or alter a dashboard platform.
Keep it from recurring
- Maintain versioned query contracts with units, weighting, scope, time, and missing-data rules. Review dashboard variables and transformations alongside expressions. Keep consumer migration notes for identity or schema changes, and require a named owner so future disagreement leads to evidence comparison rather than arbitrary preference for the greener chart.