Prometheus / Advanced: scale, persistence, and safe validation / Lesson 15

Local storage, retention, and recovery assumptions

Review time-series storage as persistent operational state with head data, blocks, write-ahead logging, retention, and verified recovery boundaries.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Distinguish head data, persisted blocks, and WAL roles.
Estimate sample volume without claiming disk capacity.
Explain retention limits and backup evidence separately.

Local storage is a stateful subsystem

Prometheus local storage organizes recent data and persisted time-series blocks, with a write-ahead log supporting recovery of recent writes under the documented implementation. Compaction and retention manage stored data over time. These mechanisms are not a substitute for an independently reviewed backup and recovery plan. A healthy process can still face storage exhaustion, filesystem failure, or unusable recovery artifacts. Treat storage paths and ownership as operational state, not disposable cache by default. This course reads no TSDB directory and runs no server, compaction, repair, snapshot, or administrative storage command.

Count samples before estimating bytes

The offline model assumes five thousand active series observed every fifteen seconds for a full day with no gaps. It produces 28.8 million samples. Actual disk use also depends on compression, indexes, chunks, metadata, churn, histograms, and workload behavior, so the sample count is not a capacity forecast. New series and retention can dominate beyond the simple active-count model. State decimal or binary units whenever converting bytes. Do not multiply a remembered bytes-per-sample number and present the result as a guaranteed storage requirement for a different deployment.

python Offline daily-sample-count model

active_series = 5000
interval_seconds = 15
seconds_per_day = 86400
samples_per_series = seconds_per_day // interval_seconds
print("samples_per_series", samples_per_series)
print("total_samples", active_series * samples_per_series)

Expected output

samples_per_series 5760
total_samples 28800000

The count assumes fixed active series and complete periodic observations. It estimates no compressed bytes and measures no disk, server, or actual scrape volume.

Read retention as a policy with implementation limits

The illustrative flags request a seven-day time policy and a two-gigabyte size policy under the pinned baseline. They are not a command line to run or a suitable capacity recommendation. When multiple retention constraints apply, deletion can occur according to the first relevant condition, and block granularity and background work affect timing. A size limit should not be assumed to cap every byte used by head data, WAL, temporary work, and filesystem overhead. Keep free-space headroom and recovery needs in the actual authorized design. Retention expiry in one local store does not prove deletion from remote copies or backups.

text Illustrative Prometheus v3.5.0 retention flags

--storage.tsdb.retention.time=7d
--storage.tsdb.retention.size=2GB

These are non-runnable flag fragments with synthetic values, not a server startup command. Their scope and storage-accounting limits require the pinned documentation.

WAL and replication are not restore evidence

A log or remote destination can reduce some loss scenarios while preserving a harmful write or missing other state. Recovery evidence should identify the source artifact, compatible version, metadata, permissions, restoration procedure, and verification of queryable data. A backup job success message alone cannot prove recovery time or completeness. Never treat deleting WAL or block files as a routine troubleshooting step; it can destroy evidence and data. Use the owning team’s approved recovery process in a real environment. This lesson provides conceptual review only and performs no backup, restore, deletion, or corruption experiment.

Separate ingestion, query, and storage pressure

A small incoming sample rate does not guarantee cheap queries, and sufficient free disk does not guarantee healthy evaluation latency. High churn, broad selectors, long ranges, and concurrent queries can stress different resources. Diagnose with authorized bounded evidence and version-specific semantics before changing retention or capacity. Document the user question supported by retained history and the consequences of shortening it. The only executed example is fixed sample arithmetic; no performance, durability, retention enforcement, storage repair, or operational recovery claim is made.

Common errors and fixes

The size retention flag is called a strict whole-directory cap

Why it happens: Head, WAL, and implementation accounting were ignored.
How to fix it: Review the pinned retention scope and leave justified headroom.
Prevent it next time: Separate modeled retention from actual disk monitoring.

WAL deletion is the first troubleshooting action

Why it happens: Persistent state was treated as disposable cache.
How to fix it: Stop and use the approved owner-led recovery process.
Prevent it next time: Preserve evidence and verified backup procedures.

Sample count is advertised as exact disk demand

Why it happens: Compression, indexes, churn, and overhead were omitted.
How to fix it: State model assumptions and obtain appropriate environment evidence.
Prevent it next time: Keep sample and byte estimates distinct.

Check your understanding

1. Does a successful restart prove the local history has a verified backup?

Show worked solution

No. Restart and WAL replay address a different scenario from restoring an independent usable artifact after loss. Verify the actual backup and restore path, versions, access, and queryable data through an authorized isolated exercise. Do not infer recovery objectives from a process starting successfully.

2. Why might a seven-day retention policy show less than seven days of useful data?