Software Architecture / Interactions: distribution, failure, and operation / Lesson 12

Design observability around questions and ownership

Use metrics, traces, logs, correlation, and alerts to explain workshop outcomes while controlling sensitive data and telemetry cost.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Choose telemetry signals for a concrete diagnostic or service question.
Define alert meaning, cardinality limits, and a useful operational handoff.

Instrument the user promise and its dependencies

Observability helps explain internal behavior through emitted evidence. Begin with questions: are valid reservation attempts succeeding, where is time spent, and how old is pending notification work? A graph of CPU alone cannot answer them. Instrument the business boundary and important dependencies with consistent outcome definitions. Distinguish a known capacity rejection from an unexpected failure and a pending workflow from a completed one. Telemetry should help evaluate the declared service promise rather than encourage teams to optimize a convenient infrastructure chart detached from user experience.

Use complementary signals rather than one universal log

Metrics summarize quantities over time, traces connect work across spans and boundaries, and logs record discrete events with context. None is automatically complete: sampling can omit traces, aggregation loses individual detail, and logs may lack timing relationships. Choose each signal for a question and preserve consistent names, units, and resource identity. The illustrative record links a reservation outcome to a trace without placing a learner email in the event. A correlation identifier is a locator, not proof of authorization and not a reason to copy sensitive payloads everywhere.

json A synthetic structured outcome event

{"event":"reservation.completed","outcome":"confirmed","reservation_ref":"r-12","trace_id":"11111111111111111111111111111111","duration_ms":42}

The fixture is valid JSON and uses invented identifiers and duration. It illustrates a telemetry schema only; no OpenTelemetry exporter or monitoring backend was run.

Keep high-cardinality identity out of metric dimensions

A metric series is distinguished by its label combination. Adding a reservation identifier, email address, or arbitrary URL can create a new series for nearly every event, increasing cost and making aggregation harder. Use bounded dimensions such as operation and outcome when they answer the question. Put detailed references in appropriately protected logs or traces where necessary. Bounded does not mean harmless: a region, tenant category, or error label can still reveal information or multiply combinations. Review dimensions and retention as part of the architecture contract.

An alert should request an owned decision

Alert on symptoms that matter and have a useful response. A rising pending-job age may require action even when every worker process is alive. Define the condition, evaluation window, severity rationale, owner, and a runbook that starts with safe observations. The sample policy intentionally leaves numeric thresholds to measured workload needs. Too many low-value alerts train responders to ignore the channel. Missing telemetry also needs a policy: silence can mean healthy operation or a broken collection path, and the interpretation must be explicit.

yaml An operational alert contract

alert: notification completion delay threatens the product target
evidence: oldest pending job age and completion rate
owner: notification workflow owner
first_checks: [provider errors, worker saturation, retry backlog]
missing_data: investigate collection health
threshold: derived from the agreed completion objective

This is an editorial policy example, not deployable alert configuration. The owner must define authorized actions and measured thresholds in the actual environment.

Preserve enough context to reproduce the reasoning

An operational handoff should include the affected promise, time window, deployed version, observed signal changes, relevant trace references, and uncertainty. Separate observation from hypothesis: a database span becoming slower supports investigation but does not alone prove the database is the root cause. Record what evidence would distinguish contention, network delay, or a changed query path. Verify instrumentation in isolated tests, including failed and cancelled operations, and check that exported data excludes secrets. This lesson validates documentation fixtures only and claims no live monitoring, tracing, or incident response execution.

Common errors and fixes

A success metric counts only requests that reached the last handler.

Why it happens: The denominator excludes earlier failures.
How to fix it: Define eligible attempts at the user-promise boundary.
Prevent it next time: Review numerator and denominator together.

A reservation identifier becomes a metric label.

Why it happens: Diagnostic detail was placed in an unbounded dimension.
How to fix it: Use bounded labels and protected trace or log references for detail.
Prevent it next time: Estimate label combinations before instrumentation changes.

An alert has no responder or next observation.

Why it happens: Notification was treated as resolution.
How to fix it: Assign an owner and a concise investigation path.
Prevent it next time: Review whether each alert leads to a useful decision.

Check your understanding

1. Workers are alive but the oldest notification is two hours old. Which claim is unsupported?

Show worked solution

Worker liveness does not establish workflow health or timely completion. Inspect completion rate, pending age, retries, provider responses, and resource saturation against the product objective. Avoid declaring a provider outage from age alone; several bottlenecks can produce the same symptom.

2. Why not log every request body to make diagnosis easier?