Prometheus / Basic: collection and time-series meaning / Lesson 2

Time series, labels, and identity

Define a series by its complete label set and explain why changing dimensions creates new history rather than editing an existing series in place.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Identify metric name, labels, timestamps, and values.
Distinguish identity from display naming.
Choose bounded dimensions that preserve useful comparisons.

The complete label set identifies the series

Prometheus stores streams of timestamped samples identified by metric name and labels. The name is represented internally through the special __name__ label. Two samples with the same name but different region or outcome values belong to different series. Reordering labels does not create a meaningful new identity; changing a label value does. A query result can also create derived series that were never stored under that identity. Keep source identity and derived aggregation separate. This model explains why a deployment that adds a changing attribute can multiply series even when request volume remains unchanged.

Read exposition as an identity declaration

The exposition fragment contains two synthetic request counters, distinguished by outcome while sharing job-independent business dimensions. The values are cumulative observations, not rates. The example uses conservative ASCII names for portability within the fixed baseline, without claiming that newer naming capabilities do not exist. Label values require the exposition format’s escaping rules; generating text by unchecked string concatenation can corrupt a scrape or expose data. Prefer supported client libraries in real applications. Do not include a timestamp casually: explicit timestamps change ingestion and staleness considerations and need a deliberate documented reason.

text Synthetic two-series exposition fragment

warehouse_requests_total{operation="lookup",outcome="ok"} 120
warehouse_requests_total{operation="lookup",outcome="error"} 3

The label values separate successful and failed outcomes. This incomplete teaching fragment is not scraped or checked by a Prometheus parser.

Model identity without depending on order

The offline model canonicalizes a synthetic label dictionary by sorting its items. Two dictionaries with the same pairs compare equal after canonicalization, while changing the outcome makes a different identity. This is a teaching model, not Prometheus’s internal fingerprint algorithm or a complete validation of legal names. It demonstrates that display order is irrelevant to semantic identity. Real label handling also has reserved names, generated target labels, and exporter interactions. Review which stage supplies each label before assuming the application’s exposed set is the final stored set.

python Offline label-identity model

def identity(labels):
    return tuple(sorted(labels.items()))
a = {"__name__": "warehouse_requests_total", "outcome": "ok"}
b = {"outcome": "ok", "__name__": "warehouse_requests_total"}
c = {"__name__": "warehouse_requests_total", "outcome": "error"}
print(identity(a) == identity(b))
print(identity(a) == identity(c))

Expected output

True
False

The function compares fixed label pairs only. It does not hash Prometheus series, parse exposition, or implement reserved-label rules.

Choose dimensions for a question

Operation family, bounded outcome class, and region can expose a partial service failure. Raw URL, request ID, user email, and arbitrary error text can create unbounded value sets and privacy risk. Every useful dimension still multiplies combinations, and rapidly replaced instances create churn over time. Removing all labels is not automatically correct because it can hide the affected population. Write the allowed values, expected churn, diagnostic purpose, and owner for each dimension. Prefer route templates rather than raw identifiers when the question concerns operation behavior across users.

Treat label changes as contract changes

A renamed service, normalized outcome, or removed dimension can split or merge the meaning of historical queries. A graph that stops selecting old series may look like recovery even when the underlying behavior did not improve. Update queries, recording rules, alerts, and dashboards with an explicit transition plan. Check duplicate identities after relabeling and aggregation. Retain only the historical compatibility required by a documented use, rather than keeping every old label forever. This lesson tests a pure identity comparison; no real series are inspected, renamed, deleted, ingested, or queried.

Common errors and fixes

Changing label order is treated as a new service

Why it happens: Display representation was confused with identity.
How to fix it: Compare complete label pairs independent of order.
Prevent it next time: Document identity separately from formatting.

Every request creates a new series

Why it happens: A per-request value became a label.
How to fix it: Use a bounded classification and appropriate event evidence.
Prevent it next time: Review cardinality and privacy before adding dimensions.

A rename looks like a sudden reliability improvement

Why it happens: The query no longer selects the old population.
How to fix it: Review historical selectors and migration boundaries.
Prevent it next time: Version label contracts with all consumers.

Check your understanding

1. Two samples have identical labels except instance. Are they one series?

Show worked solution

No. Their complete label sets differ, so they are distinct series even if a dashboard later sums them. That aggregation produces a derived result with its own grouping. Verify whether instance separation is useful before removing it, and avoid merging incompatible observation populations.

2. Should a warehouse item identifier label be added for easier troubleshooting?