Docker / Build evidence and production operations / Lesson 18

Production operations, persistent data, and recovery

Create an operational contract covering ownership, recovery objectives, upgrades, and failure evidence rather than relying on container persistence alone.

About 4 min readContent reviewed 2026-10-25

On this page

Lesson goals

Define data ownership, backup consistency, RPO, and RTO.
Review release, capacity, monitoring, and restore responsibilities without deploying.

Every persistent resource needs an owner

A production plan identifies application owners, engine and host owners, storage owners, backup operators, and incident authority. Containers make process replacement convenient but do not assign these responsibilities. Inventory images, configuration, certificates, secrets, volumes, external databases, and registry dependencies. Decide which state is authoritative, which can be regenerated, and which must be restored together. A volume surviving a restart says nothing about whether anyone can recover the correct business state after loss.

Keep process and data lifecycles separate

The following is a teaching extension of the API model with a named data volume. The current minimal HTTP fixture does not write business records into /data, so no populated dataset or production-ready persistence is claimed. A stateful implementation would need a defined format, ownership, consistency, and recovery contract. An external volume declaration makes lifecycle ownership explicit but also requires the approved volume to exist; Compose should not invent the storage policy.

yaml Explicit externally owned storage

name: parcel-lab
services:
  api:
    image: parcel-api:release-a
    user: "10001:10001"
    volumes:
      - parcel-state:/data
    stop_grace_period: 20s
volumes:
  parcel-state:
    external: true
    name: parcel-state-reviewed

The logical key parcel-state refers to an already approved volume named parcel-state-reviewed. This is a complete model, not a command to create or remove data. Missing storage should trigger investigation of provisioning and environment identity, not automatic creation of an empty replacement that hides an incident.

A backup must meet an explicit recovery question

RPO states acceptable data loss and RTO states acceptable recovery duration. A backup schedule alone proves neither. Capture application-consistent state using the data system's supported procedure; a blind copy of changing database files can be unusable. Retain protected off-host copies, required encryption keys, and version information. Include credentials and certificates through secure recovery procedures without placing them into public image artifacts or unprotected backup manifests.

Verify which storage the running service actually uses

Before backup or incident work, compare the effective container mounts with the approved resource inventory. A wrong daemon context, project, or mount source can make existing data appear missing. Read only the fields necessary for identification and avoid publishing host paths or operational labels. Volume inspection is metadata evidence, not proof of file contents, application consistency, backup completeness, or successful restore. Never browse or modify Docker's internal storage directories as a shortcut.

bash Read effective mount and volume identity

docker container inspect parcel-api --format '{{json .Mounts}}'
docker volume inspect parcel-state-reviewed --format '{{json .Name}} {{json .Labels}}'

These queries require an authorized engine and actual object identities; the container name is illustrative. Metadata can include sensitive infrastructure context, so redact it. No data files are read or copied, no volume is deleted, and no restore claim follows merely from finding the expected name.

Rehearse recovery and upgrades before an incident

Restore into an isolated approved target, verify ownership and version compatibility, then test known records, invariants, and required application behavior. Measure the full elapsed recovery time, including obtaining artifacts and keys. Prevent restored schedulers or integrations from duplicating external actions during rehearsal. Engine upgrades need reviewed compatibility, capacity, maintenance, and rollback constraints. Container replacement, image rollback, and data restoration are different actions; an operational runbook should state who authorizes each and how success is verified.

Common errors and fixes

An empty replacement volume is mistaken for recovered data.

Why it happens: Missing expected storage was bypassed by provisioning a new empty resource.
How to fix it: Establish context and original storage identity before following the recovery plan.
Prevent it next time: Fail visibly on missing authoritative resources and retain inventory evidence.

A backup archive exists but the application cannot restore it.

Why it happens: Consistency, ownership, keys, version compatibility, or object coverage was never rehearsed.
How to fix it: Run an approved isolated restore and repair the documented missing prerequisite.
Prevent it next time: Schedule restore validation rather than checking only file existence.

An engine change is performed during an incident without a rollback plan.

Why it happens: Operational boundaries and compatibility risks were not separated from diagnosis.
How to fix it: Preserve evidence and follow the authorized maintenance or incident procedure.
Prevent it next time: Rehearse upgrades and document affected workloads and recovery constraints.

Check your understanding

1. Does an external volume declaration create a backup?

Show worked solution

No. It references storage whose lifecycle is managed outside this Compose model. That storage still needs a consistency method, retained copies, access protection, keys, and a tested restore. External ownership clarifies responsibility but does not itself provide durability, redundancy, or business recovery guarantees.

2. What evidence would support a thirty-minute RTO claim?