Prometheus / Advanced: scale, persistence, and safe validation / Lesson 18

Rule testing and controlled measurement changes

Create mechanism-focused rule fixtures and review a measurement change from source semantics through queries, labels, timing, consumers, and rollback evidence.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Distinguish syntax checks from rule semantic tests.
Specify exact expected labels, annotations, and times.
Plan versioned changes without claiming unexecuted engine results.

Choose evidence for the claim being made

A configuration syntax check can detect malformed structure without proving correct denominator, matching, or alert timing. Rule unit tests can provide synthetic series and expected query or alert results to the appropriate promtool version without running a production server. They still do not establish live scraping, source correctness, receiver delivery, capacity, or access control. Separate those evidence layers in the review. Start from the failure mechanism: counter reset hidden by aggregation, duplicate metadata, missing labels, an empty denominator, or unstable alert identity. A small targeted case is more useful than a large fixture with no clear protected contract.

Read an explicit alert expectation

The non-runnable test fragment refers to the exact training rule from the alert-lifecycle lesson. It supplies zero-valued up samples every minute and expects the two-minute for condition to be firing at the named evaluation time, with exact labels and annotation. This is a declared test expectation, not a test result. The referenced rule file is not created by the public course, and no promtool binary is invoked. Keep file identity and tool version aligned in a real authorized isolated test. If the rule changes, update the expectation through review rather than making it pass by weakening assertions.

yaml Illustrative Prometheus v3.5.0 rule-test fragment

rule_files: [warehouse.rules.yml]
evaluation_interval: 1m
tests:
  - interval: 1m
    input_series:
      - series: 'up{job="warehouse-lab",instance="worker-a"}'
        values: '0 0 0'
    alert_rule_test:
      - eval_time: 2m
        alertname: WarehouseTargetScrapeFailed
        exp_alerts:
          - exp_labels:
              job: warehouse-lab
              instance: worker-a
              severity: training
            exp_annotations:
              summary: Synthetic warehouse collection failure

This fixture depends on the unchanged training rule file and remains unexecuted. Its expected firing state is an assertion to verify, not a claimed engine result.

Validate a narrow output-shape contract honestly

The offline Python example checks that a synthetic recorded result contains operation-level rates and does not retain instance. It also confirms a known aggregate value. That is useful evidence for the supplied dictionary contract only; it neither parses PromQL nor proves the rule-test fragment passes. Keep expected values independently understandable, and include negative cases that would catch the intended mistake. Do not create tests that merely copy the implementation without exposing a failure. A semantic test should help a reviewer explain which user or operational decision the protected measurement supports.

python Offline output-shape contract check

recorded = {"operation": "lookup", "rate_per_second": 5.0}
assert recorded["operation"] == "lookup"
assert recorded["rate_per_second"] == 2.0 + 3.0
assert "instance" not in recorded
print("operation-rate-contract-passed")

Expected output

operation-rate-contract-passed

The fixed dictionary is the entire tested system. No rule file, promtool process, Prometheus engine, or monitoring data is read.

Review the complete measurement change

A source or rule migration can change units, eligibility, labels, counter lifetimes, bucket boundaries, query timing, or annotations. Compare old and new contracts and identify every consumer: recordings, alerts, dashboards, remote paths, and incident guides. Mixed-version intervals can double-count or omit data if selectors are patched casually. Plan compatibility, limited exposure, verification, and rollback assumptions without mutating historical evidence silently. A dashboard becoming green after a label rename is not proof that users improved. State the actual changed behavior and the decision it enables.

Report exact validation and remaining uncertainty

Keep tool versions, fixture inputs, expected outputs, observed results, and untested boundaries in the handoff. If only Python arithmetic ran, say so; do not call configuration or rule-engine checks successful by association. When an authorized isolated engine is available, run appropriate documented checks and preserve failures rather than editing evidence after the fact. Operational rollout, service reload, receiver tests, and data lifecycle actions require their own approved plan. This course ends with reviewed contracts and offline evidence, not a claim that a production monitoring platform has been deployed or validated.

Common errors and fixes

A syntax pass is advertised as correct service measurement

Why it happens: Parsing was confused with semantic evidence.
How to fix it: Test known values, labels, populations, and timing.
Prevent it next time: Map checks to concrete failure mechanisms.

The expected result is weakened until a broken rule passes

Why it happens: Tests copied implementation behavior instead of protecting a contract.
How to fix it: Restore an independently justified expectation.
Prevent it next time: Review why each fixture should fail under a specific defect.

A local model is called a tested monitoring deployment

Why it happens: Evidence boundaries were omitted.
How to fix it: Report exactly which code and tool versions ran.
Prevent it next time: Separate model, engine, integration, and operational claims.

Check your understanding

1. Which fixtures would protect the error-ratio lesson’s contract?

Show worked solution

Include unequal instance traffic, no errors with eligible traffic, no eligible traffic, missing source, counter reset, and changed outcome labels. Assert both value interpretation and output grouping. Use the appropriate isolated engine for actual PromQL semantics, and keep any Python arithmetic claim limited to the modeled counts.

2. What must a final validation note say if no Prometheus or Alertmanager binary ran?