Elasticsearch / Intermediate: models, pipelines and relevance evidence / Lesson 7
Aggregations, buckets and accuracy
Build catalog facets and summaries while preserving population, counting units and approximation boundaries.
On this page
Lesson goals
- Distinguish metric, bucket and pipeline calculations.
- Read top-term errors and choose a complete bucket traversal when needed.
An aggregation summarizes a chosen population
A materials facet for displayed artifacts should aggregate the same eligible population as the search unless the product deliberately defines broader navigation. The query normally restricts aggregation input, while post_filter can filter returned hits without applying the same restriction to aggregations. That difference is useful for some faceted interfaces but can produce apparently contradictory counts when introduced accidentally. Write the counted unit explicitly: documents, distinct artifacts, nested conservation records or values are not interchangeable. A multi-valued material field can place one document in more than one bucket, so bucket counts need not sum to the total hit count.
Use an exact field for categorical buckets
The search body requests no ordinary hits and groups displayed documents by keyword material. A sub-aggregation computes average age_years for known numeric values in each bucket. Missing values do not automatically count as zero in that average. A text field’s analyzed terms would create a different vocabulary, and enabling fielddata on a large text field can impose substantial memory cost. Prefer the intended keyword representation. size: 5 asks for top buckets rather than promising an exhaustive material inventory. The example is a query definition and supplies no invented distributed counts or measured timings.
{"size":0,"query":{"term":{"status":"displayed"}},"aggs":{"materials":{"terms":{"field":"material","size":5,"show_term_doc_count_error":true},"aggs":{"mean_age":{"avg":{"field":"age_years"}}}}}}Categorical material requires keyword mapping; age_years requires a numeric mapping. A missing age does not contribute to the average denominator.
Top terms are selected across shards
A terms aggregation gathers shard candidates and merges them. A globally important term can be incompletely represented if it was not among candidates on some shards. doc_count_error_upper_bound and sum_other_doc_count communicate aspects of this limitation under supported ordering semantics. Increasing shard_size can improve candidate coverage but consumes more memory and network work; changing sort order can change which error guarantees apply. Do not remove the error metadata from a report claiming exact counts. For audit requirements, choose an appropriate complete strategy and reconcile against authoritative data rather than treating top-five navigation as financial accounting.
Composite traversal has its own continuation
A composite aggregation can page through bucket keys in a defined order. Use the returned after_key for the next request rather than assuming it always equals the last displayed bucket key. Keep sources, query and consistency expectations stable. The body below begins a material-key traversal; it is not a continuation of the top-terms response. A changing index can still change bucket populations across requests, so define whether the application needs a consistent snapshot. Bounded page size controls one response, but a full high-cardinality traversal still has total resource and elapsed-time costs.
{"size":0,"aggs":{"materials":{"composite":{"size":20,"sources":[{"material":{"terms":{"field":"material"}}}]}}}}Starts an ordered bucket traversal. A later body uses the response after_key under composite.after, preserving the same sources.
Calendar buckets and distinct counts need labels
A date histogram’s calendar_interval follows calendar boundaries, while fixed_interval expresses a fixed duration. Time zones and daylight-saving transitions can therefore change bucket widths or membership. Label the business zone and whether empty periods are included. Cardinality aggregation is an approximate distinct-count tool with an accuracy and memory tradeoff, not a universal exact COUNT DISTINCT. Pipeline aggregations compute from other aggregation outputs and inherit their population and approximation limits. Validate a small fixture with missing values, multi-valued categories and boundary timestamps before trusting a dashboard’s totals or percentages.
Common errors and fixes
Facet counts do not match visible hits
- Why it happens
- post_filter or multi-valued fields changed the counted population or unit.
- How to fix it
- Document the intended facet population and compare each stage.
- Prevent it next time
- Test one multi-material document and an excluded status.
Top buckets are reported as a complete inventory
- Why it happens
- terms size limits and distributed candidate selection were ignored.
- How to fix it
- Preserve error metadata or use an appropriate complete traversal.
- Prevent it next time
- Label navigation counts separately from audited exact reports.
too_many_buckets_exception
- Why it happens
- The requested bucket structure exceeds the allowed resource boundary.
- How to fix it
- Reduce breadth and nesting or page a justified composite traversal.
- Prevent it next time
- Budget high-cardinality reports before exposing arbitrary aggregation controls.
Check your understanding
1. Can the sum of material bucket counts exceed the number of artifacts?
Show worked solution
Yes, when one artifact has multiple indexed material values and belongs to several buckets. Decide whether the chart counts documents per category or assigns each artifact one primary category. Do not repair the discrepancy by dividing counts without a business definition.
2. Which token should continue a composite aggregation?
Show worked solution
Use the returned after_key with the unchanged composite sources and query. Do not derive it from display labels or assume the last bucket is always equivalent. Track consistency and total traversal budget separately from the page size.
Key takeaways
- Bucket counts require a stated unit and population.
- Approximation and continuation metadata are part of the result.