Prometheus / Basic: collection and time-series meaning / Lesson 1
The pull model and measurement boundaries
Follow a metric from an instrumented target through scraping, storage, and query evaluation while distinguishing monitoring evidence from the user operation itself.
On this page
Lesson goals
- Explain target, scrape, sample, and query roles.
- Separate process reachability from user success.
- State the fixed reference version and offline evidence limits.
Follow the sample rather than a product diagram
Prometheus commonly discovers targets, periodically retrieves their exposed metrics, stores time series, and evaluates PromQL queries or rules. Instrumentation decides what the exposed values mean; the server does not infer business correctness from a number. Our fictional warehouse API exposes accepted and failed request counts. An exporter may translate another system’s observations into an exposition endpoint, but translation adds its own boundary and possible failure. Draw source, endpoint, scraper, storage, and consumer separately. This makes it possible to ask whether a missing chart reflects absent application work, failed collection, or a query that selected the wrong series.
A successful scrape is one observation
A successful scrape means the monitoring system obtained an acceptable response at that collection boundary. It does not prove that warehouse users could authenticate, retrieve the correct inventory, or complete a write. Conversely, collection can fail while the application continues serving users. Keep service outcomes and collection health as distinct evidence. The editorial record names those boundaries without providing a network endpoint. A service team should own the business metric contract, while monitoring ownership includes collection and rule health. Shared responsibility still needs explicit contacts and escalation in the actual organization.
measurement_design: warehouse-api
source: application-instrumentation
collection_boundary: metrics-exposition-response
user_boundary: eligible-api-operation-outcome
separate_health: [collection, storage, rule-evaluation, user-success]
reference_baseline: Prometheus-v3.5.0
The map is not Prometheus configuration. It separates technical collection from user outcomes and supplies no address or credential.
Understand periodic observation
Scraping observes exposed state at discrete times. A gauge can change between observations without every intermediate value being retained; a counter can preserve accumulated increments if its process survives and the instrument is correct. The offline fixture observes a changing gauge at three selected instants. Its maximum observed value is lower than the maximum in the full synthetic sequence. This demonstrates a sampling boundary, not a defect in Prometheus. Choose collection cadence according to useful evidence, source cost, and system constraints, and do not claim that a shorter interval guarantees visibility of every transient event.
gauge_by_second = {0: 2, 1: 9, 2: 3, 3: 2, 4: 4}
scrape_times = [0, 2, 4]
observed = [gauge_by_second[t] for t in scrape_times]
print("observed", observed)
print("observed_max", max(observed))
print("full_sequence_max", max(gauge_by_second.values()))
observed [2, 3, 4]
observed_max 4
full_sequence_max 9Only a fixed dictionary is sampled. No endpoint is scraped and no Prometheus server runs; the result illustrates potentially unseen gauge peaks.
Separate data, queries, and notifications
Stored samples, a derived query result, a recording rule, an alert state, and a delivered notification are different objects. A rule can evaluate successfully but select no series, and a firing alert can fail to reach its intended recipient. Preserve metric definitions and ownership across these layers. A dashboard cannot repair an incorrect numerator merely by changing colors. During diagnosis, identify which layer’s claim is being tested before inspecting another layer. That discipline prevents a healthy query API from being presented as proof that every collector, rule, and user journey is functioning.
Use a fixed learning baseline honestly
This course pins syntax references to the official Prometheus v3.5.0 repository and Alertmanager v0.28.1 where relevant, without claiming either is the latest or suitable for deployment today. Confirm installed-version behavior and support before adapting any fragment. All PromQL, exposition, and configuration blocks are non-runnable teaching examples. Runnable Python fixtures test only explicitly modeled arithmetic or classification. They do not validate scrape negotiation, storage durability, query-engine semantics, or performance. Understanding those evidence limits is part of learning the system rather than a reason to confuse an illustration with an operational result.
Common errors and fixes
A green scrape is reported as a healthy user journey
- Why it happens
- Collection and business outcomes were conflated.
- How to fix it
- Measure eligible user operations separately.
- Prevent it next time
- Document both observation boundaries.
A short gauge spike never appears
- Why it happens
- The value changed between discrete observations.
- How to fix it
- Review whether a distribution or event-derived counter answers the question.
- Prevent it next time
- Match instrument and cadence to the diagnostic need.
A firing rule is called a delivered page
- Why it happens
- Evaluation and notification were treated as one step.
- How to fix it
- Verify each layer and its owner independently.
- Prevent it next time
- Keep rule and delivery evidence separate.
Check your understanding
1. The metrics endpoint responds, but users receive incorrect inventory. What has been established?
Show worked solution
Only collection reachability at that boundary has been established, subject to the scrape’s actual success criteria. Add or inspect user-outcome and correctness evidence. Do not change the meaning of scrape health to cover a promise it was never designed to measure.
2. Can the Python fixture prove that a one-second scrape interval is sufficient?
Show worked solution
No. It illustrates one fixed observation sequence and contains no source cost, scheduling delay, network, or real transient workload. Choose an interval from the actual measurement question and authorized evidence. A counter or histogram may preserve information that repeated gauge snapshots cannot.
Key takeaways
- Trace every claim to its observation boundary.
- Periodic scraping is not a record of every intermediate state.
- Fixed-version references and offline models have explicit limits.