Prometheus / Editorial solution guide
up is zero or an expected target is absent
Distinguish discovery loss from a failed scrape and isolate timing, transport, exposition, and limit failures using bounded evidence.
Read-only knowledge base · No public comments
Recognize the symptoms
A synthetic warehouse target shows up equal to zero, or its expected series disappears while peers remain visible. Users may still reach the application. A recent exporter or label change can coincide with the symptom without proving it caused the failure.
Understand the likely causes
- A failed scrape can involve name resolution, transport, trust, authentication, timeout, incompatible exposition, or protective limits. An absent target can instead reflect discovery, target relabeling, retirement, query selection, or staleness. Increasing timeout cannot repair all these contracts and can obscure a source-cost problem.
Freeze expected identity and the observation window
Record job, instance, service scope, last known good time, first bad observation, source version, and effective configuration revision through approved evidence. Confirm whether the target is still expected and authorized before treating retirement as failure. Keep credentials and private addresses out of shared notes. This guide contacts no endpoint and does not run a server or discovery mechanism. A successful user request and a failed collection can coexist because they cross different routes, credentials, or exposition behavior.
Separate absent and explicitly failed evidence
The offline fixture compares declared expected identities with synthetic scrape outcomes. It reports a missing target separately from a present failed target. That is an inventory assumption, not an infrastructure observation. In a real review, check discovery and target relabeling before diagnosing a missing target’s transport. For a present failed scrape, use the recorded failure boundary and timing. Do not invent a zero-valued series for every missing host; an expected-inventory policy must establish which identities belong in that comparison.
expected = {"warehouse-a", "warehouse-b", "warehouse-c"}
up_values = {"warehouse-a": 1, "warehouse-b": 0}
for target in sorted(expected):
state = "absent" if target not in up_values else ("failed-scrape" if up_values[target] == 0 else "successful-scrape")
print(target, state)
warehouse-a successful-scrape
warehouse-b failed-scrape
warehouse-c absent
The target names and outcomes are fixed synthetic data. The script performs no DNS, HTTP, discovery, scrape, or actual up query.
Identify the failed contract before changing numbers
Compare the documented collection error with duration, response compatibility, and sample-limit evidence under the installed version. An HTTP response alone may still fail parsing or content negotiation. A successful exporter endpoint may separately report a failed downstream system. Review effective timeout versus interval and whether a release increased exposition cost. Use owner-approved read-only evidence rather than broad probing or credential experiments. A protective-limit failure should be diagnosed as coverage loss, not quietly accepted as a complete partial scrape.
Choose a narrow owned correction
Repair the actual discovery contract, authorized endpoint configuration, supported exposition, or reviewed limit according to evidence. Preserve the prior revision, intended effect, and rollback assumptions for real changes. Do not increase every timeout or remove every limit simultaneously. Verify collection separately from user outcomes, and retain a coverage warning while evidence remains unknown. This editorial guide supplies no operational command and performs no reload, scrape, network test, exporter change, or administrative action.
Verify the fix
Check the exact expected identities, fresh successful observations, exposition compatibility, and absence handling through an authorized isolated or read-only workflow. Confirm the underlying service promise independently. A green up value does not prove downstream correctness or complete history. Report whether only this classifier ran or the actual supported collection path was exercised.
Keep it from recurring
- Own target inventory and retirement, version relabeling and exposition contracts, and retain fixtures for missing labels, failed collection, limits, and no traffic. Monitor source cost and collection freshness separately. Review access and transport assumptions when exporter or platform versions change rather than relying on a formerly successful endpoint.