Prometheus / Intermediate: PromQL and rule contracts / Lesson 12

Discovery and relabeling stages

Distinguish target selection before scraping from metric relabeling after collection, and review internal labels, identity changes, and dropped evidence deliberately.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Place target and metric relabeling at the correct stages.
Explain temporary discovery labels and final identity.
Test keep, drop, missing-label, and collision cases.

The stage determines what can be saved

Target relabeling operates on candidate targets before scraping and can change address-related labels or keep and drop targets. Metric relabeling operates on collected samples before ingestion under the documented rules. Dropping metrics afterward does not avoid the original network scrape or all source work. It also does not automatically affect generated collection series such as up. Keep these boundaries explicit when discussing cost or privacy. A rule applied at the wrong stage can look syntactically plausible while doing none of the intended work or unexpectedly removing the entire target population.

Discovery metadata is not permanent identity by default

Discovery mechanisms can supply temporary __meta_ labels. Internal labels beginning with double underscores have special lifecycle and interpretation, so needed metadata must be deliberately mapped into suitable final labels under the actual configuration rules. Avoid copying every discovered field into stored identity. Raw annotations, identifiers, and dynamic values can create large cardinality or disclose sensitive context. Review job, instance, address, scheme, and path roles before changing them. A relabeling rule should explain both the source authority and the intended final identity, not merely make a target appear in a list.

Read keep and drop at their declared boundaries

The non-runnable job fragment keeps only targets with an existing service label equal to warehouse-api, then drops a synthetic debug metric family after scraping. A missing service label will not satisfy that keep rule. The configuration assumes an earlier authorized discovery or static-label stage supplies service; no such environment is accessed here. Regex behavior and replacement semantics follow the pinned documentation. Review the consequence before applying a rule because a broad regex or missing source label can erase evidence. The fragment contains no credentials, target address, or executable collection instruction.

yaml Illustrative two-stage relabeling fragment

relabel_configs:
  - source_labels: [service]
    regex: warehouse-api
    action: keep
metric_relabel_configs:
  - source_labels: [__name__]
    regex: warehouse_debug_.*
    action: drop

This is an incomplete scrape-job fragment. The first rule selects targets before collection; the second filters collected metric samples, without testing either stage.

Model stage effects with separate collections

The offline fixture filters a declared target list and a separate synthetic metric-name list. Only worker-a remains eligible, while the debug family is removed from the metric list. It deliberately does not model regex anchoring, internal labels, or Prometheus-generated series. Its purpose is to show that target selection and sample selection act on different objects. Test actual configuration syntax and behavior only with the correct authorized tool and isolated inputs. A Python list result cannot prove that a running server will reload a rule or that historical data will be removed.

python Offline relabeling-stage model

targets = [("worker-a", "warehouse-api"), ("worker-b", "other-service")]
metric_names = ["warehouse_requests_total", "warehouse_debug_payload_size"]
kept_targets = [name for name, service in targets if service == "warehouse-api"]
kept_metrics = [name for name in metric_names if not name.startswith("warehouse_debug_")]
print("targets", kept_targets)
print("metrics", kept_metrics)

Expected output

targets ['worker-a']
metrics ['warehouse_requests_total']

The model uses fixed lists and prefix matching only. It implements no discovery, Prometheus relabeling engine, scrape, or configuration reload.

Check identity collisions and consumer effects

Removing a distinguishing label can make previously different samples share one final identity, causing ingestion or interpretation problems rather than a valid aggregation. Renaming is not equivalent to summing. Review expected targets, final labels, dropped sample counts, and downstream selectors together. Document policy revisions and keep privacy minimization at the source where possible, because post-scrape removal cannot undo earlier exposure. This lesson runs only the fixed-list model and makes no claim about discovery permissions, relabeling validation, storage savings, or a live monitoring configuration.

Common errors and fixes

Metric dropping is expected to eliminate source scrape cost

Why it happens: Post-collection filtering was confused with target selection.
How to fix it: Place the intended policy at the correct stage.
Prevent it next time: Document which resource cost a rule can affect.

A keep rule removes every target

Why it happens: The expected source label is missing or mismatched.
How to fix it: Inspect candidate-label assumptions and test absent values.
Prevent it next time: Keep representative discovery fixtures.

Deleting a label is treated as aggregation

Why it happens: Distinct samples collide under one identity.
How to fix it: Review final uniqueness and aggregate explicitly where appropriate.
Prevent it next time: Test collision and consumer-selector cases.

Check your understanding

1. Can metric_relabel_configs guarantee that a sensitive field never left the application?

Show worked solution

No. It acts after collection, so the field may already have existed in source memory, transport, or logs. Minimize at instrumentation and review the complete data path. A late filter can be an additional defense but must not be described as source-level prevention.

2. Why should relabeling tests include missing source labels?