Elasticsearch / Advanced: resilient and recoverable retrieval / Lesson 14
Capacity, backpressure and query cost
Diagnose request pressure using scoped measurements and control offered work before raising protective limits.
On this page
Lesson goals
- Connect query shape, concurrency and memory pressure.
- Interpret rejection and breaker signals without universal tuning prescriptions.
One request shape can change the whole workload
A title lookup returning ten fields differs from a wide aggregation across every visit event. Deep offsets, many nested hits, high-cardinality buckets and expensive scripts can increase work independently of raw document count. Concurrent requests multiply their demand and can contend with ingestion, merges and recovery. Begin with a workload inventory: request classes, response-size budgets, arrival rate, concurrency and latency objectives. Separate server calculation from client fetch and network time. A single fast request on an idle cache does not demonstrate sustainable capacity when indexing and search operate together.
Bound the response contract deliberately
The query body returns a small catalog page, filters an exact status and limits source fields. track_total_hits:false deliberately forgoes total-hit counting, which must be acceptable to the UI. It is not a general optimization to apply when the product promises an exact count. A search timeout is also not a strict end-to-end cancellation guarantee or permission to ignore partial responses. The application must inspect timed_out and shard failures and decide whether an incomplete result is usable. This body illustrates a narrower product request rather than claiming a measured speedup.
{"size":10,"track_total_hits":false,"query":{"bool":{"filter":[{"term":{"status":"displayed"}}]}},"_source":["artifact_id","title"],"sort":[{"artifact_id":"asc"}]}A bounded catalog-response example. It intentionally supplies no exact total-hit count and assumes artifact_id is sortable keyword.
Rejections are feedback rather than lost evidence
Execution rejections and circuit breakers indicate that offered work exceeds a relevant acceptance or memory budget. Breakers track selected memory demands and do not account for every allocation, so they cannot guarantee immunity from out-of-memory failure. Raising a breaker limit can remove a protective boundary without adding capacity. Preserve the detailed error, request class and contemporaneous approved metrics. Reduce unnecessary bucket breadth, document transfer or concurrency before proposing infrastructure changes. Retry only eligible operations with bounded backoff and idempotent identity, because an immediate retry storm can turn transient pressure into prolonged unavailability.
A concurrency model is only a planning calculation
The local budget model multiplies an assumed eight-megabyte in-flight payload by four requests to obtain thirty-two megabytes of payload exposure. Real memory can be larger because parsing, query structures, responses and buffers add overhead. The calculation does not size a JVM or predict an Elasticsearch breaker. It simply makes offered-work multiplication visible. Change one assumption at a time and compare representative workload evidence in an authorized lab. Keep byte units consistent and distinguish compressed wire bytes, uncompressed JSON bytes and heap-resident structures. These are different quantities even for the same request.
{"assumed_payload_mib":8,"concurrent_requests":4,"payload_exposure_mib":32,"scope":"local planning model, not Elasticsearch configuration"}The multiplication 8 Γ 4 = 32 is validated locally. It excludes parser, response and engine memory overhead.
Tune from representative constraints
Inspect approved CPU, heap pressure, garbage collection, filesystem cache, disk, queueing and rejection evidence together. A CPU-bound script and a disk-bound recovery need different remedies. Heap is not all available memory: Lucene and vector workloads can depend heavily on filesystem cache. Many small shards add overhead, while excessively large shards can complicate recovery; choose from measured workload and resilience objectives rather than a fixed slogan. Self-managed JVM and node settings do not transfer directly to Serverless. This course executes no load generator, benchmark, cluster setting change or resource experiment and provides no universal heap percentage or shard-size prescription.
Common errors and fixes
circuit_breaking_exception
- Why it happens
- A tracked memory demand exceeds a protective threshold.
- How to fix it
- Inspect the request and pressure evidence before considering any reviewed capacity change.
- Prevent it next time
- Limit expensive request shapes and offered concurrency.
es_rejected_execution_exception
- Why it happens
- The offered execution work cannot be accepted within the current budget.
- How to fix it
- Apply backpressure and bounded retry to eligible idempotent work.
- Prevent it next time
- Monitor arrival rate, queueing and rejection as one workload.
A smaller source response is treated as authorization
- Why it happens
- Response shaping was confused with access control.
- How to fix it
- Enforce access through approved security and application boundaries.
- Prevent it next time
- Review field exposure separately from performance tuning.
Check your understanding
1. Does 32 MiB of modeled payload exposure mean the node needs exactly 32 MiB of heap?
Show worked solution
No. The model excludes parsing, engine structures, responses, buffers and other workloads. It illustrates concurrency multiplication only. Use representative approved measurements to establish memory and latency requirements, and keep protective limits in place while evaluating a narrower request contract.
2. Why can increasing retries reduce useful throughput?
Show worked solution
Retries add offered work while the original bottleneck remains. They can occupy queues and memory with repeated operations, delaying useful new work. Bound concurrency and retry budget, reconcile unknown outcomes, and repair permanent source errors instead of retrying them.
Key takeaways
- Protective rejections are evidence to reduce or reshape offered work.
- Capacity claims require representative measurements and a stated observation boundary.