Prometheus / Basic: collection and time-series meaning / Lesson 4

Jobs, targets, and scrape health

Interpret collection configuration and up without confusing discovered targets, attempted scrapes, successful exposition, and user-visible availability.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Explain job and instance collection identity.
Review interval, timeout, and target scope.
Distinguish failed collection from absent expected targets.

A target is a collection destination

A scrape target is an endpoint selected for collection, often discovered through a configured mechanism. A job groups targets with a common collection purpose; instance commonly identifies the target address unless relabeling changes it. These labels describe collection identity, not automatically the business service boundary. One application can expose several endpoints, and one exporter can represent several downstream systems. Record who owns the endpoint, the exporter’s observation boundary, and the expected target population. A target omitted by discovery is a different problem from a discovered target whose scrape fails.

Read a bounded configuration fragment

The non-runnable fragment declares a synthetic job with a fifteen-second interval and five-second timeout. The reserved .invalid hostname avoids suggesting a real target. It is not a complete deployment, security policy, or instruction to start a server. A timeout must fit the effective collection interval under the documented configuration rules, but choosing either number requires source cost and measurement needs. Collection time includes more than application computation. Review DNS, transport, exposition size, parsing, and authentication boundaries with the actual authorized environment before adapting a fragment.

yaml Illustrative Prometheus v3.5.0 scrape fragment

scrape_configs:
  - job_name: warehouse-lab
    scrape_interval: 15s
    scrape_timeout: 5s
    static_configs:
      - targets: ["warehouse.invalid:9100"]

This fragment is not loaded or contacted. The hostname is reserved for invalid examples, and transport/access configuration is intentionally outside this fragment.

Interpret up at its actual boundary

Prometheus generates up for scrape attempts: one for success and zero for failure under the scrape’s criteria. It also provides collection-related measurements such as duration and sample counts. A zero can reflect unreachable transport, timeout, invalid exposition, or another collection failure; it does not identify the cause by itself. An absent expected series can reflect discovery, relabeling, query scope, or staleness. Do not turn absence into zero without a justified population model. A successful exporter scrape may still contain a downstream failure metric, so review exporter semantics separately from endpoint health.

Compare expected and observed targets explicitly

The offline fixture compares a declared expected set with synthetic scrape results. It distinguishes one failed target from one absent target instead of merging both into a single health number. The expected inventory is an input assumption, not discovered infrastructure. In real monitoring, inventory freshness and authorized scope matter; an intentionally retired target should not remain a permanent false alarm. Preserve the reason for additions and removals, and compare complete identifying labels. This model does not resolve names, read configuration, contact endpoints, or reproduce Prometheus’s staleness handling.

python Offline expected-target comparison

expected = {"worker-a", "worker-b", "worker-c"}
observed_up = {"worker-a": 1, "worker-b": 0}
failed = sorted(name for name, value in observed_up.items() if value == 0)
absent = sorted(expected - observed_up.keys())
print("failed", failed)
print("absent", absent)

Expected output

failed ['worker-b']
absent ['worker-c']

worker-b has an explicit failed observation; worker-c has no observation in the synthetic map. Those are distinct evidence states.

Use collection evidence before changing limits

Inspect the failed boundary through approved read-only evidence: target selection, effective labels, timing, response compatibility, and collection errors. Increasing timeout blindly can delay failure recognition and consume more resources without fixing the problem. A shorter interval can amplify an expensive endpoint. Keep secrets out of screenshots and configuration excerpts, and never probe arbitrary hosts from a monitoring tutorial. This lesson executes only the local inventory comparison. No server, listener, target discovery, scrape, DNS lookup, or live configuration change is performed or claimed.

Common errors and fixes

up equals one, so downstream storage is declared healthy

Why it happens: Exporter endpoint health replaced downstream semantics.
How to fix it: Inspect the exporter’s documented failure signals and user outcomes.
Prevent it next time: Keep endpoint and represented-system boundaries separate.

A missing target is treated as an ordinary failed scrape

Why it happens: Expected inventory and discovery were not checked.
How to fix it: Compare authorized expected targets with observed identities.
Prevent it next time: Maintain inventory and retirement evidence.

Timeout is increased for every collection failure

Why it happens: The failed stage was never diagnosed.
How to fix it: Use timing and compatibility evidence to isolate the cause.
Prevent it next time: Review source cost before changing cadence or limits.

Check your understanding

1. One of ten expected targets has no up series. Can you conclude that its application is down?

Show worked solution

No. Check discovery, relabeling, query labels, retirement, and data age before inferring application state. The monitoring system may never have attempted that target. Even an explicit failed scrape establishes collection failure, not the complete user outcome.

2. Why is a five-second timeout not a universal best practice?