Generative AI / Advanced: evaluation, reliability, and accountability / Lesson 15

Latency, cost, and reliability budgets

Bound the complete request path and useful work rather than assuming shorter prompts, more parallelism, or one model setting solves operational limits.

About 5 min readContent reviewed 2026-10-26

On this page

Lesson goals

Explain the assumptions and trust boundaries of latency, cost, and reliability budgets.
Interpret both examples and distinguish offline fixture results from model or production evidence.
Diagnose the listed failures and justify the worked decisions while preserving task, data, permission, and uncertainty contracts.

Measure the whole operation

A user request can spend time in admission, retrieval, reranking, context assembly, generation, tool calls, validation, and human review. Generation latency is not automatically end-to-end latency. Time to first output and time to a complete verified result also answer different questions. The first offline example subtracts invented stage durations from a fictional deadline. It does not time a model, network, device, or service. A real system should record safe stage evidence and the useful completion contract rather than treating the arrival of the first fluent token as proof that the task is correct or finished.

python Offline total-deadline arithmetic

deadline_ms = 150
stages_ms = {"retrieval": 20, "generation": 80, "validation": 10}
remaining = deadline_ms - sum(stages_ms.values())
print(remaining)
print(50 <= remaining)
print("invented durations; no timing measurement")

Expected output

40
False
invented durations; no timing measurement

Forty fictional milliseconds remain, so another fifty-unit stage does not fit. These durations are authored fixtures and not model or network benchmarks.

Keep pricing assumptions separate from arithmetic

Inference cost can depend on input, output, caching, tool use, retrieval, hardware, provider terms, and the chosen runtime. Current prices and billing rules are unstable facts and are not quoted here. The second example uses Decimal and fictional input and output units with explicitly invented rates. It checks one arithmetic model only, not a real invoice or forecast. Cost per successful supported task can rise when retries, invalid outputs, or abstentions change the amount of useful work. Include storage, evaluation, telemetry, operational review, and redundancy when they belong to the accounting scope.

python Offline fictional cost model

from decimal import Decimal
input_units = Decimal("100")
output_units = Decimal("20")
fictional_input_rate = Decimal("0.001")
fictional_output_rate = Decimal("0.002")
total = input_units * fictional_input_rate + output_units * fictional_output_rate
print(f"{total:.3f}")
print("fictional accounting units; not provider prices")

Expected output

0.140
fictional accounting units; not provider prices

The invented total is 0.140 units. Actual tokenization, caching, tariffs, discounts, taxes, and provider billing behavior are not modeled.

Bound demand and repair loops

Set limits for admitted work, queue age, payload size, context, output, tool calls, retries, and overall deadline. Per-step timeouts do not automatically form a safe total budget. A repair loop can repeatedly generate invalid content while consuming resources and delaying every other caller. Use explicit stop, fallback, and review outcomes. More workers, parallel candidates, or agents can multiply downstream demand and sensitive exposure. A semaphore can limit active calls while an unbounded queue still exhausts memory. The right limit comes from the task and dependencies under authorized measurements, not a universal number in a course.

Optimize without changing the correctness contract

Caching, batching, smaller candidates, retrieval pruning, quantization, and serving-system changes can improve selected operating properties but may alter freshness, accuracy, privacy, and compatibility. A cache key must respect model bundle, prompt, evidence revision, user scope, and relevant policy, or be limited to genuinely shared public results. A faster unsupported answer is not a useful speedup. Serving research reports gains in particular hardware and workloads; it does not provide a universal latency or memory guarantee. No batching engine, accelerator, model server, or load experiment is installed or executed here.

Plan failure and uncertain outcomes

A service timeout may leave a generated candidate incomplete, a tool effect uncertain, or a queue item still pending. Classify those states separately and preserve operation identity for safe recovery. Retries must not repeat an irreversible effect merely because a model response was lost. A fallback must satisfy the user’s evidence and permission contract, or explicitly state its limitation. Test controlled local budget logic and separately authorize real performance or provider evaluation. Both examples are tiny offline calculations with no prices, credentials, model calls, or external state. Reliability means bounded useful behavior under a stated failure model, not unlimited retries until some text appears.

Common errors and fixes

First-token speed is called completed-task latency

Why it happens: Validation, tools, and final outcome were omitted.
How to fix it: Measure the actual user completion boundary.
Prevent it next time: Name stage and end-to-end metrics separately.

Repair retries consume the entire service

Why it happens: No overall admission and attempt budget existed.
How to fix it: Bound retries, queueing, and total deadline with explicit failure.
Prevent it next time: Test repeated malformed candidates and dependency outages.

A cache returns another user’s evidence

Why it happens: Access and version scope were absent from cache policy.
How to fix it: Constrain keys and sharing under the trusted data contract.
Prevent it next time: Test tenant, evidence-version, and model-bundle changes.

Check your understanding

1. An operation permits three thirty-second attempts under a forty-second overall deadline. What is wrong?

Show worked solution

The attempts cannot each consume their full budget while respecting the total. Allocate remaining time across work, backoff, validation, and cleanup, and stop when the operation budget is exhausted. A per-attempt timeout is not an overall guarantee.

2. Why can a smaller output limit reduce cost but harm the task?