Bash / Advanced: channels, concurrency, and maintainable CLI design / Lesson 15

Bound concurrent work and coprocess protocols

Design finite worker batches and coprocess channels with explicit EOF, completion, and failure ownership instead of unbounded background launches.

About 4 min readContent reviewed 2026-10-25

On this page

Lesson goals

Separate concurrency limits from total work size.
Explain coprocess input/output descriptors and completion.
Prevent deadlocks and unobserved worker failures.

Concurrency is a resource contract

Launching one background process per record can exhaust process slots, descriptors, memory, or a downstream service. Choose a justified bound and define what happens when the queue is full. A simple batch barrier starts a small group and waits for all of it; a dynamic pool can start replacement work as jobs finish but needs more careful result tracking. Throughput is not the only acceptance condition: ordering, errors, cancellation, and memory also matter. This course uses tiny synthetic tasks and never performs service load tests or unbounded host work.

A batch barrier is simple and reviewable

A small array of owned PIDs can represent one batch. Capture every launch, wait each member, record nonzero results, and only then clear the batch state. The final partial batch must also be collected. This design can leave capacity idle while a slow member finishes, but the tradeoff may be preferable to an elaborate coordinator for small workloads. Avoid changing a parent counter inside background children and expecting it to aggregate automatically. Return each result through an explicit channel or update the parent after wait based on its saved task mapping.

bash Run a two-child barrier over four harmless tasks

pids=()
completed=0
for item in a b c d; do
  ( : "$item" ) &
  pids+=("$!")
  if (( ${#pids[@]} == 2 )); then
    for pid in "${pids[@]}"; do wait "$pid" || exit 1; completed=$((completed+1)); done
    pids=()
  fi
done
for pid in "${pids[@]}"; do wait "$pid" || exit 1; completed=$((completed+1)); done
printf 'completed=%s\n' "$completed"

Expected output

completed=4

Each child executes only a no-op with synthetic data. The parent counts verified completions and never has more than the declared batch size awaiting its barrier.

A coprocess is an asynchronous channel pair

Bash coproc creates asynchronous work with pipes for the parent to read output and write input. The named array exposes descriptors and a related variable exposes the child PID. Launch success does not equal coprocess completion. Preserve descriptors and identity while they are valid, because lifecycle changes can unset the convenience variables. Coprocesses are Bash extensions and require the platform’s pipe support. A protocol must define record boundaries, who writes first, how many records are expected, and which end closes to announce EOF. Treat the channel as a real protocol, not a magic function return.

Bound both directions and close unused ends

Pipes have finite buffering. If both sides write indefinitely before reading, each can block waiting for the other. A reader can also wait forever for EOF while another inherited write end remains open. Design a bounded request/response exchange or a streaming protocol with deliberate backpressure. Duplicate owned coprocess descriptors when needed for stable access, then close them explicitly. Handle unexpected EOF, malformed output, and nonzero completion independently. A successful read of one response does not prove the worker exited correctly, just as a zero worker status does not validate every response record.

bash Exchange one bounded synthetic record

coproc WORKER { IFS= read -r input; printf '<%s>\n' "$input"; }
worker_pid=$WORKER_PID
exec {reader}<&"${WORKER[0]}"
exec {writer}>&"${WORKER[1]}"
printf '%s\n' 'synthetic request' >&"$writer"
IFS= read -r reply <&"$reader"
exec {writer}>&-
exec {reader}<&-
wait "$worker_pid" || exit 1
printf 'reply=%s\n' "$reply"

Expected output

reply=<synthetic request>

The child waits for exactly one input record and returns one reply. Duplicated descriptors preserve access during child exit, and wait verifies completion before accepting the displayed response.

Use stronger tools when coordination dominates

A shell coordinator is reasonable when the task graph is small and failure rules remain visible. Complex retries, durable queues, distributed cancellation, or rich structured results may justify a task system or another language. Never increase complexity merely to exploit a newer wait flag without testing deployed support. Keep each worker’s input and output isolated, preserve all statuses, and test early failure plus cancellation in a safe fixture. A concurrency limit protects resources only if every launch path follows it and every child remains owned until completion or a documented transfer of responsibility.

Common errors and fixes

One background launch per unlimited record

Why it happens: Input size became an implicit process limit.
How to fix it: Add a justified bounded batch or pool.
Prevent it next time: Test resource bounds with finite synthetic inputs.

Both coprocess ends wait forever

Why it happens: Protocol ordering or EOF ownership is undefined.
How to fix it: Specify bounded exchanges and close unused ends.
Prevent it next time: Test missing response and early worker exit.

A response is accepted before wait

Why it happens: Data receipt was confused with worker completion.
How to fix it: Validate the response and collect final status separately.
Prevent it next time: Make both conditions part of success.

Check your understanding

1. What must a batch coordinator do with the last batch of one when its limit is two?

Show worked solution

Collect and validate that child too. The limit is a maximum concurrent group size, not a requirement to discard incomplete batches. Keep the remaining PID list and run the same status policy after the input loop ends.

2. Why can reading a valid coprocess line still leave a failed operation?