Prometheus / Advanced: scale, persistence, and safe validation / Lesson 16

Remote write, federation, and data copies

Compare pushing selected samples to a remote receiver with pulling selected series through federation, while keeping identity, lag, duplicate, and retention contracts explicit.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Distinguish remote write from federation.
Explain finite buffering and backlog recovery.
Review remote identity, access, and copy lifecycle.

Different transfer mechanisms answer different needs

Remote write sends samples to a compatible receiver according to its protocol and queue configuration. Federation lets another Prometheus scrape selected exposed series from a server, commonly for hierarchical or cross-service views. Federation is not a transparent copy of all historical raw samples, and remote write is not automatically a complete backup of local state. Review the exact protocol, selected series, timestamps, labels, and receiver behavior. A system diagram showing another box cannot establish completeness, deduplication, query consistency, or disaster recovery. Define the required use before choosing a transfer mechanism.

Read a remote destination as an additional data boundary

The non-runnable fragment names a reserved invalid receiver and filters a synthetic debug family for that remote path. It supplies no credentials or transport trust configuration. A write relabel policy can change what leaves this path without removing already stored local or remote history. Review the pinned ordering relative to external labels and other configuration stages. Do not assume a receiver supports every remote-write protocol or sample type. Authorization, destination ownership, sensitive fields, and retention require a real approved design, not inference from a syntactically plausible URL.

yaml Illustrative remote-write configuration fragment

remote_write:
  - url: https://metrics.invalid/api/v1/write
    write_relabel_configs:
      - source_labels: [__name__]
        regex: warehouse_debug_.*
        action: drop

The receiver hostname is reserved and never contacted. Filtering this path is not local deletion, verified privacy enforcement, or a tested protocol connection.

A healthy local query can coexist with remote lag

Buffers and retry mechanisms can absorb some receiver unavailability, but their capacity and retention are finite. A recovered receiver does not immediately imply that backlog has drained. The offline model adds thirty samples each step and permits different export capacities, ending with ten queued after three steps. It models no WAL, retries, loss, timestamps, or actual queue limits. Use it to separate incoming rate, successful export, and outstanding work. Real recovery requires enough useful export capacity above arrivals, plus compatible retention and receiver behavior, to reduce accumulated lag.

python Offline remote-backlog arithmetic

queued = 0
for step, export_capacity in enumerate([0, 20, 60], start=1):
    queued += 30
    exported = min(queued, export_capacity)
    queued -= exported
    print(step, "exported", exported, "queued", queued)

Expected output

1 exported 0 queued 30
2 exported 20 queued 40
3 exported 60 queued 10

The three-step model assumes no loss and no queue limit. It illustrates backlog accounting only and is not Prometheus remote-write implementation behavior.

Copies need deliberate labels and duplicate policy

Two monitoring replicas may send observations for the same target to one receiver. External labels and receiver-specific deduplication contracts determine whether those samples remain distinct, collide, or can be queried as one population. Do not simply sum replicas and call the result traffic. Federation label handling also needs review so source job and instance identities are not overwritten unintentionally. A label describing a tenant does not grant authorization. Keep trusted access enforcement separate from descriptive telemetry and test the chosen receiver’s semantics under an approved isolated plan.

Inventory copies and report the actual evidence

Local expiry, remote retention, secondary exports, and backups can follow different lifecycle rules. State which copy supports each query and recovery claim, and do not call deletion from one store global deletion. Keep transfer health, lag, dropped evidence, and receiver compatibility visible. This lesson executes only queue arithmetic and makes no HTTP call, scrape, remote write, federation request, cloud action, or monitoring change. It establishes no actual retention, transport security, deduplication, or backup guarantee. Those properties need the installed versions and authorized end-to-end evidence.

Common errors and fixes

Remote write is described as a full local backup

Why it happens: Samples were confused with all operational state and restore evidence.
How to fix it: Define recoverable artifacts and verify the intended recovery path.
Prevent it next time: Separate transfer and backup claims.

Two replicas double the reported request rate

Why it happens: Duplicate identity and receiver semantics were ignored.
How to fix it: Review external labels and the approved deduplication contract.
Prevent it next time: Test replica overlap before population aggregation.

Receiver recovery is called backlog recovery

Why it happens: Queued work and arrival rate were not considered.
How to fix it: Compare export capacity, lag, and outstanding samples.
Prevent it next time: Include outage and drain phases in isolated plans.

Check your understanding

1. Can federation be assumed to preserve every historical raw sample from the source?

Show worked solution

No. It exposes selected series for collection under federation semantics rather than copying the source’s entire historical store. Define the selected scope, timestamps, grouping, and label handling. Use another verified mechanism if the actual requirement is historical durability or recovery, and state its limitations.

2. After local retention expires, is the data necessarily gone everywhere?