Docker / Build evidence and production operations / Lesson 16
Correlated observability and incident diagnosis
Connect deployment identity, process state, logs, metrics, and consumer symptoms before changing a running system.
On this page
Lesson goals
- Gather narrow read-only evidence with timestamps and bounded queries.
- Distinguish application, resource, storage, and network failures.
A useful incident starts with a timeline
Record what the consumer observed, when it began, which release was serving, and whether a replacement, configuration change, or dependency event preceded it. A container id is a process-instance identity, not a stable service name. Correlate request ids and deployment revisions without exposing personal payloads. A screenshot of one metric rarely establishes causation; the strongest diagnosis connects independent observations into a consistent sequence and states which alternatives remain untested.
Read bounded logs and preserve identity
Logs should describe meaningful events with timestamps, severity, and correlation context. stdout/stderr collection depends on the logging driver, rotation, and external shipping. A missing recent line can mean no event, a different container, buffering, rotation, or a collector issue. Query only the interval and amount needed, preserving the original failure before repeated restarts. Avoid printing environment variables, authorization headers, and raw customer data as a shortcut to diagnosis.
docker logs --timestamps --since 10m --tail 100 parcel-api
docker container inspect parcel-api --format '{{json .State}}'These queries inspect a known existing container through an authorized endpoint. Logs can contain sensitive application content even though the commands are read-only. A stopped container can still have inspectable state and logs; do not start a debugging process merely to learn its last exit reason. Actual lines and codes are not fabricated.
Metrics describe pressure, not the whole cause
CPU usage, memory, process count, network transfer, and block I/O help locate resource pressure. One snapshot can miss a short burst, and command-line memory presentation can differ from raw cgroup or API accounting. Compare trends and limits with request latency and health transitions. A process waiting on a remote service may use little CPU yet be severely unavailable. High resource use is evidence to investigate, not permission to increase every limit.
docker stats --no-stream parcel-api
docker events --since 10m --until 0s --filter container=parcel-api--no-stream makes a point-in-time resource view, while the bounded event interval requests recent lifecycle evidence. Event history is limited and is not a durable complete audit log. Correlate object identity and timestamps; a resource snapshot taken after recovery cannot prove the peak state during the incident.
Follow a diagnostic tree rather than a reset ritual
If the process exited, inspect actual command, exit state, OOM evidence, and startup logs. If it is running but unreachable, distinguish the caller namespace, DNS, listener, publication, and route. If writes fail, inspect mount mode, numeric ownership, identity mapping, and storage capacity. If latency rises, compare CPU throttling, memory pressure, dependency waits, and traffic changes. Each branch narrows a cause before any reviewed correction; restarting everything can destroy the evidence linking symptoms to that cause.
Make evidence usable across teams
Capture the exact context, project, image identity, container id, time zone, and selected diagnostics in a restricted incident record. Redact secrets while retaining stable correlation ids. Distinguish observed facts from hypotheses and list the verification expected after a fix. Long-lived monitoring should alert on consumer failure and stalled recovery as well as raw resource thresholds. A healthy process, a healthy probe, and a successful real operation are separate signals that should reinforce rather than replace one another.
Common errors and fixes
No errors are found because logs belong to a replacement container.
- Why it happens
- A stable service label was confused with the incident's specific process instance.
- How to fix it
- Correlate creation time, container id, release identity, and retained previous evidence.
- Prevent it next time
- Archive useful incident context before replacement removes access.
A low CPU snapshot is used to rule out earlier throttling.
- Why it happens
- Post-incident point measurements were mistaken for historical coverage.
- How to fix it
- Use time-series and event correlation for the relevant interval.
- Prevent it next time
- Retain enough temporal resolution for bursts and recovery.
A full inspect dump exposes secrets in a public report.
- Why it happens
- Diagnostic convenience included environment values and sensitive paths unnecessarily.
- How to fix it
- Remove exposure through incident handling and collect only narrow redacted fields.
- Prevent it next time
- Design a minimal approved diagnostic evidence template.
Check your understanding
1. Why inspect an exited container before trying an interactive shell?
Show worked solution
Its last state and retained logs may already explain the failure, while exec requires a running target and starts an additional process. A minimal image may not contain a shell at all. Begin with existing read-only evidence, then choose a separately authorized diagnostic method only if uncertainty remains.
2. Which observations help distinguish wrong publication from an application crash?
Show worked solution
A crash has exited or restarting state with relevant logs and exit evidence. A publication mismatch can coexist with a running process and successful internal HTTP health. Compare the real consumer URL, host mapping, container listener, and process state rather than letting one failed browser request decide the cause.
Key takeaways
- Correlate identity and time before interpreting logs and metrics.
- Read narrowly, preserve evidence, and verify the actual consumer operation.