GraphQL / Advanced: bounded and distributed execution / Lesson 17

Observability, deadlines and source resilience

Explain operation latency through stages and dependency chains while keeping traces bounded and private.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Separate parsing, validation, queueing, resolver and source duration.
Budget a shared deadline and reason about cancellation and retry amplification.

Observe one operation across its real stages

A slow GraphQL request may spend time waiting for a connection, parsing a large document, validating fragments, waiting on a source or serially resolving dependent services. HTTP duration alone cannot locate the cause. Record a safe operation identifier, implementation version, principal class, stage timings, source calls and result category without storing private variable values. High-cardinality raw documents and user IDs can overwhelm telemetry and expose data. Use reviewed document hashes and bounded labels, preserving a controlled request correlation mechanism for detailed investigation.

Do not add overlapping spans blindly

The trace note below separates sequential pre-execution work from two concurrent source spans. Their durations overlap, so adding every span would overstate wall-clock time. The operation’s critical path matters more than the sum of all resolver durations. Conversely, a parent span may already include child work, and adding both double-counts it. This JSON is a synthetic explanation record rather than output from an instrumentation package. It establishes no measured latency, success rate or service-level objective for a running deployment.

json Synthetic overlapping timing note

{
  "operation": "CourseCard",
  "preExecutionMs": 3,
  "concurrentSources": [{"name":"catalog","durationMs":12},{"name":"author","durationMs":8}],
  "serializationMs": 1,
  "illustrativeCriticalPathMs": 16
}

With the declared simple overlap, 3 + max(12,8) + 1 equals 16 ms. Real scheduling and unrecorded work require actual trace evidence.

Share the remaining budget across dependencies

Giving every downstream call a fresh long timeout can make a short client request consume resources far beyond its useful lifetime. Propagate a deadline or remaining budget through supported context and cancellation mechanisms. Reserve time for serialization and controlled cleanup. When the client disconnects, cancellation must reach the actual source task to save work; merely rejecting the top-level promise may leave a query running. Libraries and drivers differ in cancellation support, so document the limitation and verify it in a separately authorized integration rather than assuming a common signal guarantees interruption.

Calculate an explicit remaining source budget

The local arithmetic reserves ten milliseconds of a hypothetical hundred-millisecond budget for finalization. After thirty-five milliseconds of elapsed work, fifty-five remain for a source. If elapsed work already consumes the budget minus reserve, the source must not receive a negative timeout or an unlimited fallback. The example uses synthetic numbers to teach accounting, not to recommend a production timeout. Real choices require workload and source evidence. A deadline is a correctness and resource bound, while the service’s acceptable user latency remains a separate product requirement.

javascript Offline deadline accounting

function remaining(total, elapsed, reserve) {
  return Math.max(0, total - elapsed - reserve);
}
console.assert(remaining(100, 35, 10) === 55);
console.assert(remaining(100, 95, 10) === 0);
console.assert(remaining(100, 120, 10) === 0);

The arithmetic never returns a negative budget. It does not interrupt a task or configure a driver timeout.

Avoid multiplying retries through the graph

If a client, gateway and source adapter each retry independently, one visible operation can create many source attempts. Define a retry owner, remaining-deadline check, retryable category and bounded attempt budget. Reads and mutations need different outcome reasoning, especially after uncertain writes. Circuit breakers and fallback data require domain-specific policies; stale enrollment permission is not equivalent to a stale descriptive title. Verify source concurrency, error rate, partial-data frequency and cancellation cleanup together after a change. A lower average latency can hide worse tail failures or inappropriate fallback decisions.

Common errors and fixes

Resolver totals exceed request duration

Why it happens: Overlapping or nested spans were added as independent wall time.
How to fix it: Read parent/child relations and identify the critical path.
Prevent it next time: Document whether metrics are inclusive or exclusive.

Retries exhaust the shared source

Why it happens: Several layers each applied their own attempt policy.
How to fix it: Choose a retry owner and bound attempts by the shared deadline.
Prevent it next time: Count attempts per logical operation.

A timeout hides ongoing source work

Why it happens: The cancellation signal stopped only the caller wait.
How to fix it: Use supported source cancellation and verify cleanup.
Prevent it next time: Test disconnect and deadline resource release separately.

Check your understanding

1. Two source spans take 12 and 8 ms concurrently. Why is 20 ms not the source critical path?

Show worked solution

The spans overlap, so under the declared simple schedule the longer twelve-millisecond span determines the concurrent stage. Adding them measures accumulated work rather than elapsed wall time. Real traces may include queueing and dependency edges, so inspect their start/end times and parent relationships before calculating the critical path or choosing a remedy.

2. Why should a fallback for a title differ from one for access permission?