FastAPI / Advanced: lifecycle and reliable service design / Lesson 16

Outbound clients and concurrency budgets

Own upstream clients, deadlines, failures, and bounded work without allowing a request fan-out to exhaust the application or its dependencies.

About 5 min readContent reviewed 2026-10-25

On this page

Lesson goals

Explain the assumptions and boundaries of outbound clients and concurrency budgets.
Interpret both complete examples and justify their observed outputs.
Diagnose the listed failures and apply the worked solutions while keeping identity, state, and lifecycle ownership explicit.

An upstream call creates another failure boundary

An API request may depend on another service with independent latency, authentication, rate limits, and side effects. A successful TCP or HTTP response can still contain invalid domain data. Define expected status, schema, deadlines, and retry safety before calling it. The first example uses HTTPX MockTransport, so the named reserved host is never resolved or contacted. It returns a synthetic success for one path and raises a controlled timeout for another. This tests error mapping and lifecycle ownership, not real timeout enforcement, network performance, TLS, or the upstream service’s availability.

python Complete mocked-client lifecycle and timeout check

from contextlib import asynccontextmanager
import httpx
from fastapi import FastAPI, HTTPException
from fastapi.testclient import TestClient
def upstream(request):
    if request.url.path == "/fail":
        raise httpx.ReadTimeout("synthetic timeout", request=request)
    return httpx.Response(200, json={"id": 1})
@asynccontextmanager
async def lifespan(app):
    async with httpx.AsyncClient(transport=httpx.MockTransport(upstream), base_url="https://upstream.example.invalid") as client:
        app.state.upstream = client
        yield
app = FastAPI(lifespan=lifespan)
@app.get("/proxy/{key}")
async def proxy(key: str):
    try:
        response = await app.state.upstream.get("/" + key)
    except httpx.TimeoutException:
        raise HTTPException(status_code=503, detail="Dependency timeout")
    return response.json()
with TestClient(app) as client:
    print(client.get("/proxy/item").json())
    print(client.get("/proxy/fail").status_code)

Expected output

{'id': 1}
503

MockTransport supplies success or a synthetic timeout entirely in process. The app maps that known failure to 503 and closes its client through lifespan.

Reuse clients under explicit lifespan ownership

A reusable async client can own a connection pool and configuration across requests within one application lifetime. Close it through lifespan, and do not create a fresh pool for every tiny helper call without reason. Per-request credentials and mutable headers need careful isolation even when the transport is shared. The example creates and closes the mocked client inside lifespan and accesses it only during that scope. In production, pool limits, acquisition waits, and connect or read timeout policy must match the dependency contract. A single timeout number is not automatically an overall deadline for all retries and processing stages.

Classify failure and preserve caller ownership

Translate a known dependency timeout into the documented public outcome without leaking endpoint credentials or internal traces. Unexpected failures should retain safe cause and request context for investigation. Cancellation should propagate through supporting layers so abandoned work does not accumulate. A timed-out side-effecting request may already have committed remotely; retries need idempotency or reconciliation rather than a blanket repeat. A fallback is also a contract: stale or incomplete data must be acceptable for the operation and caller. Do not turn every exception into an apparently successful empty result.

Bound active and waiting work

The second simulation limits four synthetic coroutines to two active operations and preserves result order. It demonstrates a local scheduling property only. A real fan-out needs bounds on input size, queued work, payload bytes, deadlines, and downstream calls as well. A semaphore placed after millions of tasks were created does not protect total memory. Keep a maximum consistent with database pools, external quotas, and user latency objectives. More workers or replicas can multiply total concurrency, so a per-process limit must not be mistaken for a global dependency budget.

python Complete bounded local-coroutine check

import asyncio
async def main():
    gate = asyncio.Semaphore(2)
    state = {"active": 0, "peak": 0}
    async def work(value):
        async with gate:
            state["active"] += 1
            state["peak"] = max(state["peak"], state["active"])
            try:
                await asyncio.sleep(0)
                return value * 2
            finally:
                state["active"] -= 1
    values = await asyncio.gather(*(work(value) for value in [1, 2, 3, 4]))
    print(values, state["peak"])
asyncio.run(main())

Expected output

[2, 4, 6, 8] 2

The deterministic synthetic run reaches two active tasks and preserves input order. No network, workload generator, database, or throughput measurement is involved.

Test with transport fakes and explicit proof limits

Use in-process or mock transports to test success, malformed responses, expected timeout classes, retry classification, and cleanup without any real endpoint. Keep the domain validation and error boundary active while controlling only the dependency. This course performs no network call, configured-service access, credential use, broker operation, or load test. The four-task coroutine model is not a capacity benchmark. Verify real integrations separately in an authorized isolated environment and report exactly which behavior was tested. Reliable outbound work combines resource ownership, bounded demand, and correct uncertain-outcome handling rather than simply awaiting a client method.

Common errors and fixes

Each request creates unbounded outbound clients

Why it happens: Pool ownership and lifecycle were undefined.
How to fix it: Use a reviewed application-scoped client and close it.
Prevent it next time: Test lifespan and per-request mutable state isolation.

Retry duplicates a remote write

Why it happens: Timeout was treated as proof of no effect.
How to fix it: Use operation identity and uncertain-result reconciliation.
Prevent it next time: Specify retryable operations before enabling automatic retries.

A per-process semaphore is called a global cap

Why it happens: Worker and replica multiplication was ignored.
How to fix it: Budget total dependency demand across the deployment.
Prevent it next time: Document active, queued, and distributed limits separately.

Check your understanding

1. Five workers each permit twenty outbound operations. What should a dependency budget consider?

Show worked solution

Potential active demand can reach one hundred before other clients or retries. A local cap is not a deployment-wide cap. Include replicas, queues, retries, acquisition waits, and upstream limits before choosing a safe policy.

2. What does a successful MockTransport test prove about TLS or actual timeout duration?