Prometheus / Intermediate: PromQL and rule contracts / Lesson 8

Vector matching and metadata enrichment

Explain how binary operations pair series by labels, and review uniqueness before adding group modifiers or metadata to a query.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Distinguish one-to-one and many-to-one matching.
Interpret on, ignoring, and group_left deliberately.
Test missing and duplicate metadata keys.

A binary operation needs a pairing contract

When an arithmetic operation combines instant vectors, series must match under the operation’s label rules. Unmatched elements can disappear from the result rather than becoming zero. on names the labels used for matching, while ignoring excludes named labels from that decision. Changing these lists changes the population relationship, not merely presentation. PromQL matching is not a general relational join language, and operator families have different rules. Read the pinned documentation for arithmetic, comparison, and set operations rather than assuming that one modifier works identically for every operator.

Require one metadata record for each key

The illustrative expression multiplies request rates by an information metric whose synthetic value is one, carrying team from the right side. Several operation or outcome series on the left may match one metadata series per job and instance on the right, so group_left declares that relationship. It does not make duplicate right-side keys safe. The team label is descriptive context, not an authorization claim. Ensure metadata freshness and ownership match the intended target identity before interpreting enriched results. A similarly named service in another environment is not a valid match.

promql Illustrative many-to-one metadata expression

rate(warehouse_requests_total{job="warehouse-lab"}[5m])
  * on (job, instance) group_left (team)
    warehouse_target_info{job="warehouse-lab"}

The unexecuted example assumes exactly one right-side series per matching key and information values of one. It does not verify those assumptions.

Model duplicate rejection explicitly

The offline fixture builds a metadata index and refuses duplicate keys before attempting enrichment. One successful lookup prints the synthetic team, while a duplicate produces a specific error. This models a uniqueness precondition, not PromQL’s complete matching implementation or exact error text. Real queries can have missing keys, stale metadata, and extra dimensions that change uniqueness over time. Test these cases before using enriched data in alerts. Do not choose an arbitrary first record merely to silence an error; that can attach incorrect ownership to otherwise valid service evidence.

python Offline metadata-uniqueness fixture

def index_metadata(rows):
    result = {}
    for key, team in rows:
        if key in result:
            raise ValueError("duplicate-metadata-key")
        result[key] = team
    return result
metadata = index_metadata([(("warehouse-lab", "worker-a"), "stock-team")])
print(metadata[("warehouse-lab", "worker-a")])
try:
    index_metadata([("a", "team-one"), ("a", "team-two")])
except ValueError as error:
    print(error)

Expected output

stock-team
duplicate-metadata-key

The dictionary model checks one declared key space. It performs no PromQL join, backend query, target discovery, or ownership authorization.

Investigate disappearance before broadening matching

If a binary expression returns fewer series than its inputs, compare the complete label sets, chosen matching labels, and time scope. A missing environment key or changed instance label can prevent pairing. Removing more labels may create cross-environment matches or many-to-many ambiguity. Numeric equality of values does not establish identity. Keep a narrow source selector and documented unique key rather than using group modifiers as trial-and-error fixes. Metadata enrichment should retain the intended business population; it should not silently drop unowned targets from a service-health result.

Review matching as an evolving schema contract

New labels, target renaming, deployment overlap, and information-series duplication can break a previously valid relationship. Version the producer and consumer assumptions and verify uniqueness at relevant evaluation times. Carry only necessary bounded metadata into result labels, because enrichment can also increase cardinality and expose sensitive associations. This lesson runs only a dictionary uniqueness check. No arithmetic matching engine or monitoring data is executed, and the PromQL fragment has no claimed numerical result. A useful review explains why the relationship is valid rather than merely noting that an expression once returned data.

Common errors and fixes

A binary query silently loses valid targets

Why it happens: Matching keys or metadata coverage differ.
How to fix it: Compare labels and time scope before adding modifiers.
Prevent it next time: Test unmatched keys explicitly.

group_left is added to fix duplicate metadata

Why it happens: The required unique side is still not unique.
How to fix it: Repair the metadata identity or intended relationship.
Prevent it next time: Assert key uniqueness in fixtures.

Enrichment attaches another environment’s team

Why it happens: The key omitted a required scope boundary.
How to fix it: Use trusted scoped identity and narrow selectors.
Prevent it next time: Review authorization separately from descriptive labels.

Check your understanding

1. Three request series match two metadata series under the same key. Does group_left make the relationship valid?

Show worked solution

No. Many-to-one arithmetic matching still requires the designated one side to be unique. Identify why metadata duplicates exist, such as overlapping instances or an incomplete key. Do not arbitrarily discard a record without reviewing which identity and ownership are correct.

2. Why is team metadata unsuitable as the sole authorization boundary?