Elasticsearch / Intermediate: models, pipelines and relevance evidence / Lesson 10

Templates, data streams and lifecycle

Separate current-state artifact documents from time-based visit events and manage future index contracts deliberately.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Explain component and composable index templates.
Choose a data stream and retention model from event semantics.

A mutable catalog differs from an event history

An artifact document describes current searchable state and may be replaced when curators edit its title. A visit event records one occurrence at a timestamp and is usually appended. A data stream is designed for time-series-oriented event data backed by managed indices, not simply a prettier name for every mutable catalog index. It requires an appropriate matching template and timestamp mapping. Updates or deletes of existing stream documents have specific backing-index targeting rules. Choose event identity, replay behavior and retention before selecting the storage abstraction, because those decisions control how repair and reconciliation work later.

Reuse a mapping component with a named scope

A component template is a reusable building block. It does not apply to indices merely because it exists; an index template must compose it. The synthetic component body defines timestamp and event identity for museum visit records. Keep field names and types stable across consumers, and avoid one global component whose casual edit changes unrelated future workloads. A template update governs new index creation, not a retrospective rewrite of existing backing indices. Historical mapping compatibility must therefore be reviewed across every index searched through a stream or alias.

json Illustrative technical example; logical results are reasoned, not engine-tested.

{"template":{"mappings":{"properties":{"@timestamp":{"type":"date"},"event_id":{"type":"keyword"},"artifact_id":{"type":"keyword"}}}},"_meta":{"purpose":"synthetic museum visit events"}}

Body for a fictional museum-visit-fields component template. It does not create a data stream or alter existing indices.

Template selection is a precedence decision

The composable template targets a deliberately narrow museum-visits-* pattern and enables a data stream. An explicit priority participates in selection among matching templates; it is not a universal recommended number. Built-in and integration templates can overlap broad names, so inspect and simulate the effective configuration before approval. Settings specified directly during supported index creation can override template values in documented ways. Record why the pattern and priority were chosen. A valid JSON body cannot establish that the named component exists or that a cloud deployment permits every included setting.

json Illustrative technical example; logical results are reasoned, not engine-tested.

{"index_patterns":["museum-visits-*"],"priority":250,"data_stream":{},"composed_of":["museum-visit-fields"],"_meta":{"owner":"synthetic museum analytics"}}

An illustrative index-template body requiring the named component. Priority 250 is a scenario value whose overlap must be reviewed.

Rollover changes the write target, not the event identity

Rollover creates a new backing write index under the supported lifecycle rules. A size or age threshold describes a maintenance condition, not a guarantee of exact event-time partition boundaries. Late-arriving events can be indexed into the current write index even when their event timestamp is old. Retention based on backing-index age can therefore differ from deleting every event exactly a fixed time after occurrence. Explain whether policy age is measured from creation, rollover or a separately configured origin according to the chosen mechanism. Keep legal holds and replay windows in the retention design.

Choose lifecycle controls for the deployment

Index lifecycle management and data stream lifecycle are related but distinct control surfaces with version and platform availability boundaries. Elastic Cloud Serverless manages infrastructure differently and does not expose every versioned-cluster ILM or node-tier setting. Review supported controls for the actual project rather than copying a self-managed policy. Before any destructive retention change, identify affected indices, protected source data, snapshot requirements and downstream consumers. Verify the policy with synthetic periods and late events in an independent lab. This course does not create templates, roll over indices, delete data or exercise a lifecycle engine.

Common errors and fixes

A component template has no effect

Why it happens: No selected index template composes it for the new target.
How to fix it: Inspect matching composable templates and their effective configuration.
Prevent it next time: Simulate narrow target names before enabling producers.

Old and new backing indices disagree on field type

Why it happens: A template change affected only newly created indices.
How to fix it: Inventory historical mappings and plan compatible search or rebuild.
Prevent it next time: Review stream-wide field compatibility in schema changes.

Retention removes events earlier than expected

Why it happens: Backing-index lifecycle age was confused with each event timestamp.
How to fix it: Reconcile the chosen age reference and late-arrival policy.
Prevent it next time: Test late events and protected retention windows before policy changes.

Check your understanding

1. Does editing a component template rebuild mappings in all existing indices?

Show worked solution

No. Templates primarily supply configuration for future index creation. Inspect old backing indices separately and plan a compatible migration if their representation must change. Verify the selected composable template and component order rather than assuming a component’s name makes it globally active.

2. Why can an old event be stored in the newest backing index?