Grafana / Advanced: access, dependencies, and evidence / Lesson 18

Semantic tests and dashboard change review

Protect a dashboard’s measurement and access contracts through known-value fixtures, explicit dependency inventories, scoped review, and honest validation claims.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Separate JSON parsing from semantic and runtime tests.
Review source, frame, transform, display, and access changes together.
State tested tools and unexecuted integration boundaries precisely.

Test the decision the dashboard supports

A dashboard test should protect meaning: selected population, units, time, reducer, missing-data policy, navigation, and access assumptions. A field-order change may be harmless, while replacing a weighted ratio with a mean can reverse a decision without breaking JSON syntax. Write the expected behavior before inspecting a screenshot. Include normal, zero, missing, stale, extreme, renamed, and duplicate-key cases appropriate to the view. Do not snapshot every generated layout property and call that measurement assurance. A small fixture that catches an actual semantic defect is more useful than a large one that mirrors the current implementation.

Make the intended result independently understandable

The editorial JSON names one source, route, unit, reducer, and expected fixture result. It is not a Grafana dashboard schema or a unit-test runner format. A reviewer can explain why two synthetic route-r1 observations of four and eight seconds yield six. That independent arithmetic is a narrow useful contract. It does not prove that a real query returns those observations or that Grafana renders the configured unit. Keep the input fixture separate from runtime evidence so later source or transformation changes cannot quietly redefine the expected answer.

json Editorial dashboard semantic contract

{"view":"transit-lab-mean-delay","source_uid":"transit-lab-prom","route":"r1","unit":"seconds","reducer":"observed-mean","expected_fixture_value":6.0}

The fields define a synthetic review case, not Grafana API or dashboard fields. No resource is created and no actual source response is asserted.

Run only the local claim that can be checked

The Python fixture parses fixed JSON observations, checks their route population and unit, and verifies the known mean. Its complete tested boundary is those local values. It imports no Grafana client, sends no query, and checks no renderer or authentication. A valid JSON parser result would be weaker than the arithmetic and population assertions shown here. A real authorized integration should additionally check source resolution, plugin response, frame shape, transformations, selected fields, time context, and user access. Report which of those layers actually ran rather than treating one successful test as evidence for all of them.

python Offline semantic-fixture validation

import json
fixture = json.loads('{"unit":"seconds","rows":[{"route":"r1","delay":4},{"route":"r1","delay":8}]}')
assert fixture["unit"] == "seconds"
assert all(row["route"] == "r1" for row in fixture["rows"])
mean = sum(row["delay"] for row in fixture["rows"]) / len(fixture["rows"])
assert mean == 6.0
print("fixture_mean_seconds", mean)

Expected output

fixture_mean_seconds 6.0

The fixed JSON is the entire tested system. No Grafana schema, plugin, source, query, transformation engine, renderer, or account is exercised.

Review the complete change across dependencies

A change can alter source UID, query language, variable defaults, time overrides, transformations, field options, links, alert rules, or permissions. Record the user-visible trigger and before-and-after behavior, then inspect semantic diffs rather than only serialized noise. Keep the selected Grafana, edition, plugin, and backend versions with the evidence. A rollout or upgrade needs approved configuration, source mappings, ownership, and recovery assumptions under the actual management mechanism. Do not infer that restoring one dashboard JSON also reverses data-source credentials, plugin migrations, notification policies, or backend state.

An honest handoff includes limitations and ownership

Report exact local checks, expected and observed outputs, untested layers, official references, assumptions, and remaining decisions. An image can support layout review while a local fixture supports arithmetic; neither proves a live platform’s security or delivery. Keep unresolved source and access questions owned instead of hiding them behind a green screen. This course ends with synthetic evidence and version-aware contracts. It deploys nothing, changes no account or service, and executes no external query, alert, receiver, or cloud action. Further integration evidence belongs to a separately authorized environment and actual selected capabilities.

Common errors and fixes

A screenshot is called end-to-end validation

Why it happens: Appearance replaced query and access evidence.
How to fix it: State what the image and each fixture establish.
Prevent it next time: Separate layout, semantics, integration, and operational claims.

A test passes after its expected value is changed to match a defect

Why it happens: The contract was replaced with current behavior.
How to fix it: Restore an independently justified expectation.
Prevent it next time: Review the mechanism each fixture protects.

Restoring dashboard JSON is called complete rollback

Why it happens: Related resources and migrations were omitted.
How to fix it: Review the actual management and recovery scope.
Prevent it next time: Inventory sources, plugins, alerts, and permissions.

Check your understanding

1. Which cases would protect a transit mean-delay stat from misleading changes?

Show worked solution

Include known route observations, unequal populations, wrong units, missing and stale values, renamed fields, duplicate join keys, and changed time scope. Assert the expected population and reducer as well as the number. Use the actual authorized integration for Grafana behavior and keep the local model claim narrow.

2. What must a final report say when only Python and JSON checks ran?