Observability / Basic: signals and measurement meaning / Lesson 3

Structured logs and event semantics

Design bounded event records with stable fields, timestamps, severity, and safe correlation while keeping sensitive payloads outside routine telemetry.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Distinguish event time from observed time.
Design stable machine-readable fields.
Review severity and redaction at the source.

Describe what happened in stable fields

A structured log is an event record whose fields have deliberate meanings. For a parcel scan rejection, a stable event name, bounded reason code, service identity, timestamp, and safe correlation context are more useful than one changing prose sentence. Keep the human message readable, but do not make a query depend on the exact wording of a sentence someone may translate or edit. Distinguish an attempted operation from a completed one. A log emitted before a storage write cannot prove that the state was durably saved, even if its message says processing succeeded.

Keep occurrence and observation separate

The time an event occurred can differ from the time a collector observed it. Buffered delivery, offline devices, clock skew, and retries can make those differences operationally important. Preserve both meanings when the data model supports them, and show which timestamp drives a query. Sorting delayed records by ingestion time may tell a different story from sorting by event time. A timestamp without timezone or source is difficult to compare safely. The illustrative JSON record uses synthetic UTC values and an explicit reason code; it is an event example rather than a complete OpenTelemetry export envelope.

json Synthetic structured event

{
  "event_name": "parcel.scan.rejected",
  "event_time": "2026-10-28T10:00:00Z",
  "observed_time": "2026-10-28T10:00:04Z",
  "severity": "WARN",
  "reason_code": "unsupported_state_transition",
  "service_name": "parcel-ingest"
}

The four-second difference describes this record’s observation delay only. The fields are illustrative and not a claim of a universal log schema.

Make severity support a response

Severity describes importance under an agreed event policy; it should not be assigned by how alarming a developer feels while writing the message. A rejected malformed input can be expected client behavior, while repeated valid events lost after acknowledgement can be a serious service problem. Log severity and incident severity are different classifications. Avoid emitting every retry at maximum severity and then paging on raw log count. Count outcomes and user impact separately. Preserve a bounded reason taxonomy so responders can distinguish expected validation from unexpected system failure without parsing arbitrary payload text.

Validate fields with a tiny fixture

The offline parser reads one fixed JSON event and checks required fields before extracting its bounded reason. It demonstrates a contract check without reading files, contacting a service, or handling real customer data. Real pipelines need policies for malformed and oversized records, encoding, duplicate events, and schema evolution. Do not silently discard every invalid record without counting the loss, because a formatting regression can erase the evidence needed for diagnosis. Keep a safe dead-letter or rejection workflow only where policy permits it and where sensitive content remains protected.

python Offline structured-event contract check

import json
raw = '{"event_name":"parcel.scan.rejected","severity":"WARN","reason_code":"unsupported_state_transition"}'
event = json.loads(raw)
required = {"event_name", "severity", "reason_code"}
assert required <= event.keys()
assert event["reason_code"] in {"unsupported_state_transition", "malformed_scan"}
print(event["event_name"], event["reason_code"])

Expected output

parcel.scan.rejected unsupported_state_transition

Only the fixed synthetic string is parsed. The assertions validate this small contract, not an OpenTelemetry SDK or backend ingestion path.

Minimize before export and retain deliberately

Never assume a diagnostic log is harmless. Request bodies, authentication headers, addresses, and raw parcel identifiers can reveal personal or secret information. Prefer an allowlist of necessary fields and safe synthetic examples in learning material. Redaction later in a collector is a useful defense but may occur after the source or transport already retained data. Apply access, retention, deletion, and incident-response rules to the telemetry system itself. Review logging changes as behavior changes: they can affect latency, storage cost, privacy, and the ability to investigate important failures.

Common errors and fixes

A wording edit breaks every query

Why it happens: Queries parsed free-form prose as a schema.
How to fix it: Use stable event fields and bounded reason codes.
Prevent it next time: Version and test event contracts.

A success log exists for missing data

Why it happens: The message was emitted before durable completion.
How to fix it: Align event meaning with the actual commit boundary.
Prevent it next time: Distinguish attempt, acknowledgement, and completion.

Redaction is applied only after export

Why it happens: Sensitive data already left the source.
How to fix it: Minimize at instrumentation and review the full path.
Prevent it next time: Use allowlisted fields and appropriate access controls.

Check your understanding

1. A scanner buffers events for an hour. Which time should a freshness query use?

Show worked solution

Define the question first. User-visible event freshness may need event occurrence time, while service processing delay uses arrival or acceptance time. Preserve both when appropriate, account for device clock reliability, and avoid blaming ingestion processing for time spent offline before the service could observe the event.

2. Rewrite a log that prints an entire failed request body.