Elasticsearch / Intermediate: models, pipelines and relevance evidence / Lesson 9
Ingest pipelines and failure paths
Normalize source records through ordered processors while preserving provenance and explicit rejected outcomes.
On this page
Lesson goals
- Explain processor ordering and simulation scope.
- Design missing-field, conversion and on_failure behavior.
Transformation order changes the output
A museum feed may send status as DISPLAYED and year as the string 1890. A pipeline can normalize status and convert year before mapping validation. Processors run in order, so a rename must precede a transformation referencing the new field. A pipeline does not replace the mapping or the producerโs business validation. Converting a syntactically numeric year does not prove it is a plausible acquisition date, and lowercasing a status does not prove it belongs to the approved vocabulary. Keep identity and source revision outside destructive cleanup so rejected records can be traced and replayed.
Keep the pipeline small and identifiable
The body defines two processors with explicit tags. Tags help correlate an error with a transformation rather than merely reporting that ingestion failed. A missing status or unconvertible year is not silently ignored here; the default failure behavior remains visible. Store the pipeline definition in version-controlled application material and identify which producer or template selects it. Changing a pipeline in place affects future processing, not historical documents that were already indexed. A reproducible catalog should record the transformation version when future interpretation depends on it.
{"version":1,"processors":[{"lowercase":{"field":"status","tag":"normalize_status"}},{"convert":{"field":"year","type":"integer","tag":"parse_year"}}]}Illustrative body for a named ingest pipeline. Both fields are required by this transformation contract.
Simulation validates a chosen path, not the deployment
The inline simulation body includes the same two transformations and a synthetic source. In an independent authorized lab, POST /_ingest/pipeline/_simulate would show status displayed and numeric year 1890. It would not prove that the live producer selects this pipeline, that the destination mapping accepts every transformed field or that bulk response items are all successful. Include missing fields, null values, already numeric years and invalid strings in the simulation set. This course parses the JSON locally only and does not execute the pipeline engine or claim a captured simulation response.
{"pipeline":{"processors":[{"lowercase":{"field":"status"}},{"convert":{"field":"year","type":"integer"}}]},"docs":[{"_source":{"artifact_id":"art-101","status":"DISPLAYED","year":"1890"}}]}Expected transformed source fields: status="displayed", year=1890. This expectation follows the documented processors.
An error marker does not automatically reject data
ignore_failure can continue after a processor fails, while on_failure can run defined recovery steps. Merely adding ingest_error in a handler may still allow a malformed record to be indexed. If the policy requires quarantine or rejection, design that outcome explicitly and verify its destination and accounting. Preserve the original field value when approved repair needs it, and avoid logging sensitive payloads by default. A handler should distinguish a tolerable optional field from a mandatory business fact; treating both identically can turn an obvious ingestion failure into a silent data-quality defect.
Ingest processing shares a resource budget
Expensive parsing, enrichment or scripts consume ingest resources before indexing completes. More concurrent bulk requests can increase queueing and rejection rather than throughput. Measure representative payloads and processor paths in an authorized lab, including failures that invoke recovery logic. Enrichment also needs a defined reference-data refresh process and a policy for unmatched keys. A successful enrichment lookup can still be stale relative to the source system. Keep pipeline changes, reference-data versions, rejected-item counts and end-to-end freshness together in operational review. Do not install processors, provision nodes or change a configured cluster merely to validate this editorial example.
Common errors and fixes
A processor reports a missing field
- Why it happens
- The source omitted a required field or an earlier rename changed its path.
- How to fix it
- Inspect processor order and the exact source at that step.
- Prevent it next time
- Test optional and required fields separately.
A failed conversion still produces indexed data
- Why it happens
- A failure handler or ignore policy allowed processing to continue.
- How to fix it
- Define rejection or quarantine explicitly and reconcile its output.
- Prevent it next time
- Review failure paths as carefully as successful transformations.
Historical documents retain old normalization
- Why it happens
- A pipeline edit applies to later processing, not existing indexed content.
- How to fix it
- Plan a versioned rebuild when historical consistency is required.
- Prevent it next time
- Record transformation versions and rebuild acceptance checks.
Check your understanding
1. Does successful inline simulation prove the producer uses the intended pipeline?
Show worked solution
No. It proves only the simulated processor path for the supplied documents if the engine actually runs it. Verify producer parameters, default or final pipeline selection, destination mapping and bulk item outcomes separately. Here only local JSON parsing is performed; no engine simulation is claimed.
2. Why can setting ingest_error be insufficient for a rejected-record policy?
Show worked solution
The document may continue to the normal index with the marker. Define whether the handler fails the request or routes to a controlled quarantine, and verify accepted plus rejected counts. Make repair and replay ownership explicit so the error collection does not become a permanent unexamined sink.
Key takeaways
- Ordered transformations and failure policies both define the ingestion contract.
- Simulation, producer selection and destination acceptance are different proofs.