MLOps / Pipelines and controlled delivery / Lesson 10

Batch and online delivery contracts

Preserve prediction identity, completion semantics, and latency meaning across delivery patterns.

About 5 min readContent reviewed 2026-10-28

On this page

Lesson goals

Distinguish scheduled batch outputs from request-response behavior.
Restore result identity under reordering and partial completion.
Include waiting and preprocessing in a declared latency budget.

Delivery mode follows the decision horizon

A scheduled batch can produce predictions for many entities before a later decision, while an online request may need an answer before a near-term deadline. Neither mode is automatically better. State input freshness, eligible population, completion time, output identity, and what happens when a result is late or absent. A streaming consumer adds its own ordering and repeat-delivery contract. A model accurate under a daily snapshot may not support decisions made before that snapshot exists. Connect the delivery schedule to the same prediction-time information boundary used during evaluation, rather than treating delivery as a wrapper added afterward.

Completion order does not define observation alignment

The first fixture deliberately reverses synthetic completion order and restores scores using stable ids. It checks uniqueness and coverage before reconstruction. A plain zip of original rows with returned scores would silently assign some predictions to the wrong entities. Real batch work also needs policies for duplicate requests, partial failures, retries, and outputs already published. Decide whether a batch is all-or-nothing or carries an explicit per-record status. A successful process exit does not prove that every eligible entity received one valid result, and a matching row count cannot detect a swapped identity.

python Deterministic offline standard-library fixture

rows = [{"id":"c", "value":3}, {"id":"a", "value":1}, {"id":"b", "value":2}]
completed = [{"id":row["id"], "score":2*row["value"]} for row in reversed(rows)]
assert len({row["id"] for row in rows}) == len(rows)
by_id = {row["id"]:row["score"] for row in completed}
assert set(by_id) == {row["id"] for row in rows}
restored = [by_id[row["id"]] for row in rows]
print("completion ids", [row["id"] for row in completed])
print("input-order scores", restored)

Expected output

completion ids ['b', 'a', 'c']
input-order scores [6, 2, 4]

Explicit identity restores a synthetic reordered result; this fixture creates no workers, queues, or asynchronous service.

A latency budget includes more than model computation

An online path can include parsing, validation, feature retrieval, preprocessing, queue waiting, batching, inference, post-processing, and response transfer. The second fixture adds hypothetical waiting and computation to compare completion with absolute deadlines. It does not measure a device or serving platform. Dynamic batching may improve throughput by sharing work while increasing waiting for some requests. Choose batch size and waiting limits from measured resource and decision requirements, not a universal tuning prescription. Report tail behavior and eligible request counts rather than only a favorable average or an isolated warm model call.

python Deterministic offline standard-library fixture

requests = [{"id":"a", "arrival_ms":0, "deadline_ms":7},
            {"id":"b", "arrival_ms":3, "deadline_ms":8}]
batch_start_ms, compute_ms = 5, 2
completion_ms = batch_start_ms + compute_ms
for request in requests:
    latency = completion_ms - request["arrival_ms"]
    print(request["id"], "latency_ms", latency, "within_deadline", completion_ms <= request["deadline_ms"])

Expected output

a latency_ms 7 within_deadline True
b latency_ms 4 within_deadline True

The fixture counts queue waiting plus a fixed hypothetical computation time; it is arithmetic, not an observed latency benchmark.

Failure and overload behavior are part of the API

Define timeouts, request-size bounds, unsupported input, stale features, and fallback when required work cannot finish. A timeout may occur after computation or publication, so an automatic retry can repeat an effect unless operation identity is handled deliberately. Backpressure or controlled rejection can preserve a service contract better than accepting unbounded work that cannot meet any deadline. These are design choices requiring an actual implementation and measurement; the lesson does not simulate a production queue. Keep error responses distinguishable from ordinary model scores so a caller cannot interpret failure as a valid zero.

Verify functional parity before measuring capacity

For supported inputs, compare batch and singleton behavior under the intended inference mode, preprocessing state, and version selection policy. Some models deliberately use context, so state permitted dependencies instead of assuming unconditional independence. Verify result identity, output shape, and decisions near thresholds as well as numeric tolerance. Then measure the complete relevant delivery path in its supported environment under a bounded workload. The examples here perform deterministic list reconstruction and deadline arithmetic only; no server, load test, queue, network request, framework model, or performance benchmark is run.

Common errors and fixes

Batch output has the right length but wrong entity scores

Why it happens: Completion order was assumed to match input order.
How to fix it: Join by stable request or observation identity and validate coverage.
Prevent it next time: Keep a reordered-result fixture with unequal values.

Latency claims count only a warm inference call

Why it happens: Waiting, feature work, validation, or transfer was omitted.
How to fix it: Define and measure the full relevant request boundary.
Prevent it next time: Report timing scope, counts, percentiles, and environment.

A failed request becomes a numeric zero

Why it happens: Error status and prediction value share an ambiguous representation.
How to fix it: Return a clear failure or documented fallback status.
Prevent it next time: Test timeout and unsupported-input paths separately from valid scores.

Check your understanding

1. Why does request a have greater latency than b in the deadline fixture despite one shared completion time?

Show worked solution

Request a arrives at time zero and finishes at seven, so its latency is seven. Request b arrives at three and also finishes at seven, so its latency is four. Waiting depends on arrival time. A single batch compute duration of two does not describe either complete request latency.

2. A batch returns two scores for three ids after one record fails. What must the consumer know?