Elasticsearch / Advanced: resilient and recoverable retrieval / Lesson 13

Shards, replicas and resilience evidence

Interpret cluster health as allocation evidence and connect redundancy to realistic failure domains.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Count primary and replica copies without confusing them with nodes.
Read allocation explanations before proposing operational repair.

A replica is another copy of one shard

An index with three primary shards and one replica per primary has six shard copies, not four shards and not necessarily six nodes. Elasticsearch avoids placing a primary and its replica on the same node, while wider zone resilience needs appropriate failure-domain design. More replicas can improve some search capacity and redundancy but add storage and write replication work. The primary-shard count is a structural planning choice; changing it later uses supported split, shrink or reindex approaches with prerequisites. Do not prescribe a universal shard count from document count alone without size, query and recovery evidence.

Green describes allocation rather than application correctness

Green means all primary and replica shard copies are allocated. Yellow means primaries are allocated but at least one replica is not; red means at least one primary is unassigned. The worst relevant index can determine broader health. A green cluster can still serve stale search data, unauthorized results or slow requests, so health is one operational dimension. The schematic response below represents three active primaries and three unassigned replicas. It could occur when allocation constraints prevent placing copies, but the response alone does not identify the cause.

json Illustrative technical example; logical results are reasoned, not engine-tested.

{"status":"yellow","active_primary_shards":3,"active_shards":3,"unassigned_shards":3}

A schematic health fragment consistent with three primaries and three missing replicas. It is not a captured cluster response.

Ask why a specific copy is unassigned

The allocation explain body targets replica shard zero of the synthetic index. It is read-only diagnostic intent and requires suitable privileges. Deciders can point to disk watermarks, allocation filters, awareness constraints, unavailable nodes or other conditions. Preserve the explanation and the relevant approved configuration before proposing a change. Removing a replica requirement may make health green while reducing protection, so a color change alone is not a satisfactory fix. Forced stale-primary allocation can accept data loss and is outside this editorial course; an incident owner must follow a tested recovery decision process.

json Illustrative technical example; logical results are reasoned, not engine-tested.

{"index":"museum-artifacts-v1","shard":0,"primary":false}

Body for POST /_cluster/allocation/explain in an authorized versioned-cluster lab. No allocation or cluster request is performed.

The elected master manages cluster metadata

Master-eligible nodes participate in cluster coordination, while data nodes hold shards according to their roles. A majority of the relevant voting configuration is required for elections; counting application-visible data nodes is not enough to prove coordination resilience. Network partitions and correlated zone failures must be part of the design. Avoid restarting multiple coordinating members as an improvised repair. In Elastic Cloud Serverless these infrastructure choices are automated differently, so the versioned-cluster node model is explanatory background rather than a supported project administration recipe. Document which platform owns each recovery action.

Resilience includes recovery time and independent backups

A surviving replica can restore availability after a node loss, but recovery and relocation consume disk, network and CPU. A topology that survives one failure only on paper may lack spare capacity to recover under normal traffic. Replica copies also reproduce accidental deletions and are not independent historical backups. Define a snapshot strategy and a rebuild path from authoritative source data where applicable. An authorized rehearsal should measure service impact, accepted-write reconciliation and restore readiness. This course runs no cluster, node failure, shard relocation or resilience experiment and claims no recovery duration.

Common errors and fixes

Yellow is treated as complete data loss

Why it happens: One or more replicas are unassigned while primaries remain allocated.
How to fix it: Inspect index-level health and specific allocation explanations.
Prevent it next time: Teach health colors with their exact allocation meaning.

Replica count is reduced solely to make health green

Why it happens: A status target replaced the required redundancy objective.
How to fix it: Repair the justified allocation constraint or approve a changed protection design explicitly.
Prevent it next time: Track resilience requirements separately from dashboard color.

Replica copies are treated as backups

Why it happens: Replication was confused with recoverable historical state.
How to fix it: Use supported snapshots and a tested restore or source-rebuild path.
Prevent it next time: Include accidental deletion in recovery scenarios.

Check your understanding

1. How many copies exist for three primaries and two replicas per primary?

Show worked solution

Nine copies: three multiplied by one primary plus two replicas. Placement still requires enough suitable nodes and failure domains. This arithmetic does not establish that allocation succeeds or that the cluster survives a particular zone failure; inspect the topology and voting design separately.

2. Can green health prove a catalog index contains every accepted source event?