Bash / Intermediate: reliable data and failure paths / Lesson 10

Read records without silently changing them

Specify delimiters, whitespace, backslashes, final-record behavior, and binary limits before selecting read, mapfile, or another parser.

About 4 min readContent reviewed 2026-10-25

On this page

Lesson goals

Use IFS= read -r for literal line-oriented records.
Handle an allowed final unterminated line explicitly.
Recognize formats that need a real structured parser.

A delimiter is part of the data format

A line-oriented format treats newline as a record separator, but not every filename or structured record obeys that assumption. Define encoding, allowed delimiters, empty records, and whether a final delimiter is required. CSV with quoted embedded newlines cannot be parsed reliably by splitting every line at commas. JSON needs its own parser, not a collection of shell substring tricks. Bash is useful for orchestration and simple declared text formats; choosing another parser is often the most reliable shell design. Reject unsupported input rather than silently changing its meaning.

Preserve whitespace and backslashes deliberately

read normally applies IFS splitting and can interpret backslashes. IFS= read -r line is the common literal-line form: the temporary empty IFS avoids trimming and field splitting for that read, and -r preserves backslashes. The record delimiter itself is not stored. A variable named line is just a destination and does not impose a format. Keep the read command as the loop condition so its EOF result controls termination. Input redirection can keep the loop in the intended shell; a pipeline may introduce a child environment.

bash Preserve spaces, a backslash, and an empty line

while IFS= read -r line; do
  printf '<%s>\n' "$line"
done < <(printf '%s\n' '  red apple  ' 'path\name' '')

Expected output

<  red apple  >
<path\name>
<>

The producer emits exactly three newline-terminated synthetic records. The loop prints them without trimming their payloads; the angle brackets expose boundaries.

Decide whether an unterminated last record is valid

read returns nonzero at EOF even when it assigned characters from a final line lacking a delimiter. A plain while read loop can therefore skip that payload. If the format permits it, a condition such as read ... || [[ -n $line ]] can process a nonempty final fragment. This does not invent an extra empty record at EOF. If a final delimiter is mandatory, report malformed input instead of quietly accepting it. The right behavior comes from the file contract, and tests should include both a terminated empty record and an unterminated nonempty one.

bash Accept an explicitly allowed final fragment

count=0
while IFS= read -r line || [[ -n $line ]]; do
  count=$((count + 1))
  printf '%s:<%s>\n' "$count" "$line"
done < <(printf 'first\nlast')

Expected output

1:<first>
2:<last>

The second record has no final newline. The alternate condition admits that nonempty fragment once; EOF does not produce a third record.

NUL-delimited streams still need a storage model

For filesystem records that can contain newlines, a NUL delimiter can preserve boundaries because Unix path components cannot contain NUL. Bash read supports an empty -d delimiter to stop at NUL on the tested baseline, and mapfile has related delimiter options in supporting versions. However, Bash variables themselves cannot store embedded NUL bytes. Consume one non-NUL payload at a time rather than capturing an entire binary stream in command substitution. Do not claim NUL handling makes arbitrary binary data safe in shell variables. Confirm tool support and define producer status handling too.

Choose streaming or bounded collection

mapfile can load text records into an indexed array conveniently, but loading all input trades simpler access for memory consumption. Streaming a loop can keep memory bounded, while a dedicated parser can validate a complex format more clearly. Decide how empty fields and duplicate keys should behave before assignment. Never source an input file to avoid writing a parser: that executes code in the caller. Preserve parse failures separately from EOF and distinguish an incomplete producer stream from a complete empty input. Test malformed records, unusual whitespace, and large valid inputs within safe local limits.

Common errors and fixes

Backslashes or edge spaces disappear

Why it happens: read used its default interpretation.
How to fix it: Use IFS= and -r for literal line payloads.
Prevent it next time: Test whitespace and backslash records.

The last nonempty line is skipped

Why it happens: EOF status ended the loop before processing an allowed fragment.
How to fix it: Handle the final fragment according to the format contract.
Prevent it next time: Include missing-final-delimiter tests.

Comma splitting corrupts quoted CSV

Why it happens: A structured grammar was treated as simple fields.
How to fix it: Use a parser that implements the format.
Prevent it next time: Declare supported formats and reject unsupported records.

Check your understanding

1. Why can a plain while read miss the characters it just assigned at EOF?

Show worked solution

The loop body runs only when the condition succeeds. EOF makes read nonzero even if an unterminated fragment was assigned. Add explicit fragment handling only if the format permits it; otherwise diagnose the missing delimiter.

2. Can Bash capture a NUL-delimited list intact in one scalar?