Google Cloud / Intermediate: Service Contracts and Operations / Lesson 12

Observability and Service Objectives

Turn telemetry into a bounded operational question with explicit population, latency, retention, and response ownership.

About 5 min readContent reviewed 2026-10-25

On this page

Lesson goals

Distinguish logs, metrics, traces, and audit evidence.
Define eligible and good events for an SLI.
Review alert behavior without sending notifications.

Each telemetry type answers a different question

Logs record events, metrics summarize measured time series, traces connect work across a request path, and audit logs provide documented administrative/data-access evidence. They overlap but are not substitutes. A missing application log is not proof that an API request never occurred; a high-level metric is not a transaction trace. Choose the signal from the diagnostic question.

Record project/resource scope, timestamps, service/revision identity, units, sampling, and retention. Telemetry can be delayed, excluded, sampled, or permission-restricted under its configuration. Cloud Audit Logs categories and enablement differ; do not assume every data read is automatically available in every log view. Protect sensitive payloads and credentials from collection.

Define good and eligible events before calculating

The first program computes a request-success SLI from invented records. The denominator includes all eligible synthetic requests, and the numerator follows the declared success rule. A production indicator must define exclusions, time windows, missing telemetry, and whether it represents edge responses, backend work, or complete user transactions. Different measurement points can yield different truths.

An SLO adds a target and compliance period to an SLI. A request-based objective differs from a windows-based one, especially under bursty traffic. Do not average unrelated percentages without their denominators or average percentiles and call the result a global percentile. Preserve the population and aggregation method.

python Calculate a synthetic request SLI

statuses = [200, 200, 503, 200, 500]
eligible = len(statuses)
good = sum(200 <= status < 400 for status in statuses)
if eligible == 0: raise ValueError("no eligible events")
print(f"good={good} eligible={eligible} sli={good / eligible:.0%}")

Expected output

good=3 eligible=5 sli=60%

The declared rule counts three of five requests as good; it is not a real service measurement.

Alert conditions have evaluation and delivery behavior

A metric-based alert policy includes alignment, aggregation, condition duration, missing-data handling, incident behavior, and notification routing. Crossing a threshold once in a local list is not evidence that a live incident would open or that a person would receive a message. Review the current product semantics and test the operational response through an authorized process.

Choose actionable symptoms and a named response owner. A page should connect to a runbook and decision, not merely report every fluctuating metric. Distinguish notification-channel delivery from the underlying condition. Silence can mean recovery, absent data, excluded resources, or a routing problem, so verify what the signal actually states.

A duration rule differs from a single spike

The second program checks an invented rule requiring three consecutive samples above a threshold. Only the final three qualify. This is a local sequence exercise, not an implementation of Cloud Monitoring alignment or retest semantics. The example makes duration explicit without configuring a policy, sending a notification, or reading a time series from an account.

For a real indicator, sample intervals, gaps, delayed points, resource grouping, and resets matter. Use synthetic cases for no data, a short spike, sustained failure, and recovery. Avoid universal thresholds: a meaningful latency or saturation limit depends on the application and its objectives.

python Check an invented consecutive-sample rule

samples = [2, 8, 3, 9, 10, 11]
threshold = 7
required = 3
run = 0
met = False
for value in samples:
    run = run + 1 if value > threshold else 0
    if run >= required: met = True
print(f"synthetic_duration_rule={met}")

Expected output

synthetic_duration_rule=True

The final9,10,11 satisfy the local three-sample rule; no Monitoring policy is evaluated.

Observability needs evidence quality and a response loop

Plan dashboards for current health, alerts for action, logs/traces for diagnosis, and audit evidence for accountability. Manage cardinality and retention deliberately so instrumentation does not create unbounded volume or expose user identifiers. A metric label per arbitrary request can be a poor design even when it is technically accepted.

Review access to telemetry and exported artifacts, test runbooks, and link changes to observed outcomes. A green dashboard does not prove backups restore or that every region is healthy. The course validates only static local calculations; real collection, alert evaluation, notification delivery, and operational response remain untested service behavior.

Common errors and fixes

An average of percentiles is reported as the global percentile.

Why it happens: Aggregation discarded the required distribution information.
How to fix it: Use the documented distribution aggregation or a correctly defined window metric.
Prevent it next time: State the indicator’s population and units.

No logs are treated as proof of no failures.

Why it happens: Collection, scope, access, and missing data were ignored.
How to fix it: Inspect telemetry configuration and corroborating signals.
Prevent it next time: Test no-data cases explicitly.

An alert has no actionable owner.

Why it happens: A threshold was configured without an operational contract.
How to fix it: Assign routing, response, runbook, and review responsibility.
Prevent it next time: Evaluate notification and response separately from condition logic.

Check your understanding

1. A low-traffic window has one failure in two requests, while a busy window has ten failures in one thousand. Why can equal-window averaging mislead?

Show worked solution

The windows carry different request exposure. If the question is overall request success, combine good and eligible request counts rather than averaging the two success percentages equally. A windows-based SLO can intentionally give each window weight, but that is a different indicator and must be named. State missing-data and window rules before interpreting the result.

2. What evidence is needed beyond a locally true duration rule before claiming an on-call alert works?