Elasticsearch / Basic: document and query contracts / Lesson 5

Bulk ingestion and item accounting

Read NDJSON as a sequence of actions and handle partial success without duplicating accepted work.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Explain index, create, update and delete bulk actions.
Classify item errors and design bounded replay.

Bulk is a framed sequence, not a JSON array

The Bulk API accepts newline-delimited JSON. Each action metadata line determines whether a following source line is required. index and create require a document line, update requires an update body, and delete has no source line. The final line must end in a newline, and individual JSON records are not pretty-printed across several physical lines. This framing is independent of whether the source documents contain nested objects. A generic JSON-array serializer produces a different protocol and can fail even though a standalone JSON parser considers its output valid. Validate framing separately from field mappings.

Choose replacement or create-only behavior

The synthetic payload contains two create actions addressed to a versioned teaching index. Repeating it with the same identifiers should conflict for already existing documents instead of silently replacing them. An index action would allow replacement, which can be correct for a current-state projection but dangerous for immutable acquisition events. Define what a duplicate means before selecting the action. Data streams support append-oriented create behavior with their own update targeting rules, so ordinary-index examples should not be copied without checking that model. No bulk request is sent by this course.

ndjson Illustrative technical example; logical results are reasoned, not engine-tested.

{"create":{"_index":"museum-artifacts-v1","_id":"art-101"}}
{"artifact_id":"art-101","title":"Bronze compass"}
{"create":{"_index":"museum-artifacts-v1","_id":"art-102"}}
{"artifact_id":"art-102","title":"Paper navigation chart"}

Illustrative NDJSON for POST /_bulk. The terminal newline is part of framing; each metadata and source line is a complete JSON object.

An HTTP success can contain failed items

Bulk responses carry a top-level errors flag and an ordered item result for each action. A transport-level successful response does not establish that every document was indexed. Correlate each response item with its source record and retain the source identity plus rejection reason. The schematic response shows one created item and one mapping rejection. Retrying the entire batch can duplicate or overwrite successful work depending on the selected action and identifier policy. Count accepted, permanently rejected, retryable and unresolved items so their sum matches the submitted action count.

json Illustrative technical example; logical results are reasoned, not engine-tested.

{"errors":true,"items":[{"create":{"_id":"art-101","status":201}},{"create":{"_id":"art-102","status":400,"error":{"type":"mapper_parsing_exception","reason":"synthetic field-type conflict"}}}]}

A simplified response for reasoning about partial failure, not measured server output. Item order corresponds to action order.

Retry only what the outcome permits

A mapping error usually needs source or schema repair before replay. A version conflict can be an expected duplicate or a real competing update; classify it using the business key and revision contract. Rejected execution under pressure can justify bounded backoff with jitter and a retry budget, but endless immediate retries amplify overload. A connection loss can leave item outcomes unknown, requiring identity-based reconciliation. Keep batches bounded by both action count and byte size. There is no universal ideal batch size because document complexity, ingest processing, shard distribution and resource budgets differ.

A durable source makes repair possible

Keep a recoverable source stream or immutable batch manifest containing event identity, source revision and payload checksum as appropriate. A successful producer send is not enough to delete that source before downstream acknowledgment and reconciliation complete. Quarantine malformed records with a reason and an approved retention policy; a dead-letter collection is useful only if someone can inspect, correct and safely replay it. Protect customer data and credentials in diagnostic output. Treat bulk throughput as an application reliability problem involving backpressure and accounting, rather than a contest to maximize concurrent requests.

Common errors and fixes

illegal_argument_exception

Why it happens: Bulk framing or action/source pairing does not match the protocol.
How to fix it: Validate each NDJSON line and the final newline before sending.
Prevent it next time: Use a supported client helper or a tested local serializer.

mapper_parsing_exception

Why it happens: One document contains a value incompatible with the mapping.
How to fix it: Quarantine that item and repair the exact source contract.
Prevent it next time: Track per-item failures rather than only the HTTP status.

es_rejected_execution_exception

Why it happens: The system cannot accept the offered work within its current execution budget.
How to fix it: Reduce offered concurrency and use a bounded retry policy.
Prevent it next time: Monitor queueing, latency and rejection trends together.

Check your understanding

1. A response has HTTP 200 and errors:true. May the producer delete all source records?

Show worked solution

No. Inspect every item, persist accepted identities and retain rejected or unknown records for the documented repair path. Transport success only delivered the response. Reconcile action counts and avoid replaying successful create or additive update operations blindly.

2. Why should a batch limit consider bytes as well as document count?