Kubernetes / Intermediate: rollout, policy, and evidence / Lesson 12
Observability and diagnostic evidence
Correlate workload intent, status, events, metrics, and logs while preserving context and avoiding sensitive-data exposure.
On this page
Lesson goals
- Explain the model, assumptions, and boundaries of observability and diagnostic evidence.
- Interpret both examples and distinguish declared intent from established evidence.
- Diagnose the listed failures and justify the worked recovery decisions without executing cluster operations.
Ask a layer-specific question
Begin with the failing user outcome and locate the boundary whose evidence can distinguish causes. Desired replicas, ready Pods, node pressure, Service endpoints, and request latency measure different things. A green controller condition does not prove a business operation succeeded, while a client timeout does not identify Kubernetes as the cause. Record cluster context, namespace, object UID, container, node, revision, and time when relevant. Avoid joining records only by a reused Pod name. The first synthetic incident record separates a known user symptom from a scheduling hypothesis requiring more evidence.
impact: synthetic booking latency regression
known: one rollout cohort has fewer ready replicas
hypothesis: new requests conflict with placement capacity
needed: scheduling conditions and resource contract
not_proven: cluster-wide capacity exhaustion
The record states uncertainty instead of promoting a hypothesis into root cause. It neither queries metrics nor authorizes a resource change.
Use status and events as contextual evidence
Status is reported by components and may lag current reality. Conditions describe a resource-specific model; reasons and messages need interpretation against the observed generation and lifecycle. Events can explain scheduling, image, volume, and probe problems, but retention is limited and they are not a complete durable audit trail. Preserve the first useful evidence before replacement or repeated retries obscure it. Compare timestamps and object lifetimes. An old warning from a deleted Pod should not be treated as proof that its replacement still has the same failure. No events are collected from any cluster in this lesson.
Correlate logs without dumping secrets
Select the intended container, current or previous execution, and time window. A sidecar’s successful log line does not explain an application container’s exit. Structured logs should include safe correlation and release identifiers while omitting tokens, environment dumps, request bodies, and sensitive account data. Central collection needs explicit retention, access, and failure policy because node-local evidence can disappear. The second example is a public synthetic JSON event, not captured output. A request ID supports correlation and must never be used as an authenticated identity or an unbounded metric label.
{"event":"request_failed","service":"booking-api","route":"/bookings/:id","revision":"lab42","request_id":"lab-a","duration_ms":900,"error_code":"UPSTREAM_TIMEOUT"}
The route is a template, the identifiers are fictional, and no credential or user payload is present. An upstream timeout is evidence to investigate, not proof of a network-policy defect.
Separate resource symptoms and application causes
Metrics can show CPU throttling, memory pressure, restarts, queue growth, and latency, but each needs units, aggregation scope, and a workload hypothesis. A high restart count may accumulate across past failures; a memory peak does not alone prove a leak. Compare deployment and configuration changes with representative windows, and consider backend limits before adding replicas. Use traces when they clarify time across dependencies, with deliberate sampling and redaction. Audit logs answer API activity questions and can themselves contain sensitive object content depending on policy, so their collection and access require careful ownership.
Design safe diagnostic workflows
A runbook should start with bounded authorized read access, preserve evidence, classify the failing boundary, and name the next discriminating observation. Do not run arbitrary exec commands, port forwards, packet captures, or load tests as the default first step. Those operations have privileges and side effects beyond reading a status. Define who can authorize them in a real response. This lesson uses only synthetic records and static parsing. The outcome of diagnosis is a supported explanation and a scoped recovery decision, not merely a large collection of screenshots or a restart that makes the symptom temporarily disappear.
Common errors and fixes
An old event is applied to a replacement Pod
- Why it happens
- Evidence was correlated only by reused name.
- How to fix it
- Compare UID, timestamp, owner, and revision.
- Prevent it next time
- Record object lifetimes in incident evidence.
Logs expose configuration credentials
- Why it happens
- Full objects or environment were dumped.
- How to fix it
- Contain exposure and use safe allowlisted metadata.
- Prevent it next time
- Test redaction and diagnostic access policies.
A metric is presented as a complete cause
- Why it happens
- Aggregation and alternative hypotheses were ignored.
- How to fix it
- Correlate layer-specific evidence and test the explanation.
- Prevent it next time
- Document units, scope, and uncertainty.
Check your understanding
1. A Pod name matches yesterday’s warning but its UID differs. What should happen?
Show worked solution
Treat them as different lifetimes. Inspect current evidence for the new UID and retain the old record only as historical context. A repeated logical service name can link an investigation, but it cannot prove the same process or fault persisted.
2. CPU is low while request latency and a backend queue grow. Is adding application replicas justified?
Show worked solution
Not from that evidence alone. Inspect backend capacity, connection waits, request concurrency, and recent changes. More replicas can increase pressure on the constrained dependency. Choose a response based on the measured bottleneck and user impact rather than one idle resource.
Key takeaways
- Begin with the failing user outcome and locate the boundary whose evidence can distinguish causes.
- Select the intended container, current or previous execution, and time window.
- A runbook should start with bounded authorized read access, preserve evidence, classify the failing boundary, and name the next discriminating observation.