Observability / Advanced: bias, trust, and evidence engineering / Lesson 18

Cross-signal investigation and evidence limits

Combine population metrics, selected events, traces, and pipeline-health evidence into a falsifiable investigation without hiding their different populations and clocks.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Frame a scoped cross-signal hypothesis.
Track evidence through selection and collection.
Separate mitigation, mechanism, and unresolved questions.

Keep the user question and scope fixed

Begin with a precise symptom and comparison: parcel updates accepted in one region became stale after a release, while acceptance volume remained steady. Fix service, environment, operation family, time range, and measurement definitions before navigating between tools. Record known blind spots and clock assumptions. This prevents a trace from a different region or a log search over another interval from becoming accidental evidence. A release annotation suggests a hypothesis, not a cause. Keep the immediate need to reduce user harm separate from the later need to explain the full mechanism.

Let each signal answer a bounded question

Metrics can establish how widespread a change is, traces can expose where selected operations spent time, and structured events can explain a bounded rejection or transition. Collector health can reveal that the apparent change came from missing or delayed telemetry instead. Compare those explanations explicitly. A long storage client span may include queueing or retries, so inspect its instrumentation boundary. A log message may precede commit, so inspect its event contract. Do not let a detailed example overrule population evidence merely because it tells a compelling story.

Build an evidence matrix with alternatives

The editorial matrix ties a hypothesis to predicted observations and an alternative instrumentation explanation. Record which observation would weaken the hypothesis as well as which supports it. A useful next check distinguishes alternatives with minimal risk and data exposure. If the necessary evidence is unavailable, state that uncertainty rather than filling it with a plausible cause. Preserve safe links and revisions, not copied private payloads. In a real incident, authorized responders may choose a mitigation before causal certainty, but they should still record its scope, expected effect, and verification separately.

yaml Illustrative cross-signal evidence matrix

hypothesis: consumer-wait-increased-after-release
supports: [queue-age-rises, compatible-processing-spans-show-wait]
weakens: [consumer-time-stable, source-events-arrive-late]
alternative: timestamp-or-instrumentation-contract-changed
next_check: same-scope-versioned-timing-evidence
claim_limit: correlation-does-not-establish-mechanism

The record is a falsifiable investigation plan. It executes no query, service action, trace collection, or operational experiment.

Explain missing examples before inferring behavior

The offline classifier distinguishes three reasons an expected trace is unavailable: policy selection, recording or export gap, and query-selection mismatch. The model uses known synthetic membership sets and cannot identify an actual pipeline fault. Its value is preserving the stages instead of treating every absence as proof that work never happened. Real investigations need supported SDK, collector, backend, and access evidence. Authorization can correctly exclude data from a reader; do not bypass it to make a trace appear. Missing or partial evidence limits the claim even when another signal remains healthy.

python Offline evidence-stage classifier

def locate(trace_id, selected, exported, visible):
    if trace_id not in selected:
        return "not-selected-by-policy"
    if trace_id not in exported:
        return "recording-or-export-gap"
    if trace_id not in visible:
        return "query-scope-or-visibility-gap"
    return "available-for-interpretation"
print(locate("lab-a", set(), set(), set()))
print(locate("lab-a", {"lab-a"}, set(), set()))
print(locate("lab-a", {"lab-a"}, {"lab-a"}, set()))

Expected output

not-selected-by-policy
recording-or-export-gap
query-scope-or-visibility-gap

The fixture classifies known membership only. It runs no sampler, exporter, access check, or backend query and does not infer semantic relevance.

Write a conclusion that survives scrutiny

State the observed impact, supported mechanism, confidence, mitigation result, residual correctness work, and unresolved questions. A restart that helped shows that reset state mattered, not which state or why it accumulated. Turn instrumentation blind spots into owned improvements with acceptance fixtures. Review whether extra telemetry would have changed a decision before expanding collection. This course’s capstone uses only a local stage classifier and an editorial matrix; no incident is declared, no person is notified, and no monitoring or application service is accessed. Good observability ends in an honest decision trail and a clearer system model.

Common errors and fixes

One slow trace is generalized to all users

Why it happens: A selected example replaced population evidence.
How to fix it: Compare compatible counts and distributions.
Prevent it next time: Document sampling and trace coverage.

Missing telemetry proves no operation occurred

Why it happens: Selection, recording, export, and query stages were conflated.
How to fix it: Locate the evidence gap stage by stage.
Prevent it next time: Preserve pipeline health and policy context.

Mitigation success becomes an unsupported root cause

Why it happens: Restoration and mechanism evidence were merged.
How to fix it: Record the result and investigate the causal mechanism separately.
Prevent it next time: Label confidence and unresolved questions.

Check your understanding

1. A slow trace and an error spike occur after a release. What would make the hypothesis stronger?

Show worked solution

Align service, region, operation, time, and instrumentation definitions; compare candidate and unaffected populations; identify a mechanism that predicts the observed wait or failure; and check alternatives such as changed sampling or timestamps. Temporal alignment alone remains correlation, even when several charts tell a similar story.

2. What should remain in the investigation record when the cause is uncertain?