Grafana / Intermediate: transformations, configuration, and alerts / Lesson 10

Explore and cross-signal investigation

Use ad hoc query and inspection workflows to narrow a question while preserving source identity, time, access, and the different populations of metrics, logs, and traces.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Separate exploratory queries from saved dashboard contracts.
Preserve scope across signal links.
Distinguish query evidence from a causal explanation.

Explore is a questioning workflow

Explore supports ad hoc investigation through configured data-source integrations, with capabilities depending on the plugin and version. It can help compare a panel’s assumptions with a simpler query, but it does not bypass backend authorization or create missing telemetry. Record the source UID, expression, variables, time range, and result shape when preserving evidence. Query history and saved material can contain sensitive identifiers or query text, so review the actual retention and access behavior. An experimental query becoming useful does not automatically update the dashboard or alert that originally raised the question.

Navigation does not make populations identical

Metrics summarize observations, logs retain selected events, and traces relate sampled operations under their own policies. Supported exemplars, derived fields, or configured links can navigate between them, but availability and behavior depend on the integration. A linked trace may explain one slow departure lookup without representing every metric event. A missing log can reflect filtering or retention rather than no operation. Compare scope and collection coverage before generalizing. A shared label or identifier aids correlation but is not a trusted authorization claim or proof that two events caused each other.

Preserve a bounded investigation record

The editorial record names an intended transit question and the scope that must travel between sources. It is not an Explore URL, API request, or stored query. Preserve only safe references and necessary metadata rather than copying private payloads into a public report. A panel’s visible time range may differ from a linked view because of overrides, variable interpolation, or destination defaults. Verify those details explicitly. If a reader lacks authorized access to one signal, do not broaden permissions merely to complete a visual narrative; record the evidence limitation.

yaml Editorial cross-signal investigation record

investigation: transit-delay-lab
question: why-route-r1-delay-increased
carry_scope: [service, environment, route, absolute-time-range]
evidence: [metric-population, selected-events, sampled-traces]
known_limits: [sampling, retention, source-access]
record_policy: safe-references-without-private-payloads

The record defines scope and evidence limitations only. It creates no Explore session, correlation link, query history, or backend access.

Test scope transfer with fixed metadata

The offline comparison checks whether two synthetic query contexts agree on service, environment, route, and absolute time bounds. It finds that the destination end time differs. This proves only a mismatch in the supplied dictionaries, not that a real link or query is wrong. A complete investigation also needs units, timestamp meaning, sampling, and source freshness. Keep the difference visible rather than silently adjusting the destination to make two charts agree. A consistent scope is necessary for comparison but does not establish causality or equivalent signal completeness.

python Offline query-scope comparison

metric_scope = {"service":"transit", "environment":"lab", "route":"r1", "from":100, "to":160}
log_scope = {"service":"transit", "environment":"lab", "route":"r1", "from":100, "to":170}
mismatches = sorted(key for key in metric_scope if metric_scope[key] != log_scope.get(key))
print("scope_mismatches", mismatches)

Expected output

scope_mismatches ['to']

The fixed integers are synthetic time bounds. No URL, source, query inspector, Explore session, or live telemetry is accessed.

Keep facts, hypotheses, and display artifacts separate

A release annotation preceding delay is a hypothesis cue, not proof of cause. A slow span can include connection waiting or retries depending on instrumentation. A query inspector can expose requests and response data, but sharing its output requires privacy review and cannot prove source correctness by itself. Use a bounded next question that distinguishes explanations, then record what supports or weakens the hypothesis. This lesson runs only local dictionary comparison. No query, trace lookup, log search, inspector export, data-source action, or actual incident communication is performed.

Common errors and fixes

A linked trace is generalized to all departures

Why it happens: A selected example replaced population evidence.
How to fix it: Compare sampling and metric scope.
Prevent it next time: Document signal coverage and limits.

Two views disagree because the link dropped time scope

Why it happens: Destination defaults changed the observation set.
How to fix it: Carry and verify safe absolute context.
Prevent it next time: Test navigation with synthetic boundaries.

Inspector output is pasted into a public report

Why it happens: Diagnostic output was assumed free of sensitive data.
How to fix it: Minimize and review before authorized sharing.
Prevent it next time: Use synthetic fixtures and safe references in learning material.

Check your understanding

1. A metric spike and a slow trace occur after a release. What further evidence is needed before declaring a cause?

Show worked solution

Align service, environment, route, time, and instrumentation definitions; check sampling and collection health; and identify a mechanism that predicts the observed delay. Compare alternatives such as changed query scope or source timestamps. Several correlated charts still do not establish a causal chain by themselves.

2. What should be preserved when a useful Explore query becomes part of a dashboard change?