Go / Intermediate: Packages and Concurrent Work / Lesson 9

I/O Streams, Files, and Bounded Records

Process stream results in the correct order, enforce size policies, and make resource ownership explicit.

About 5 min readContent reviewed 2026-10-25

On this page

Lesson goals

Interpret n and err together in Reader contracts.
Choose bounded scanning or exact-length reads.
Define closing and complete-record validation policies.

A read can return data and an error together

io.Reader returns a byte count and an error. Callers must process n greater than zero before handling the accompanying error, because a reader may return data and io.EOF in the same call. Only the first n bytes of the supplied buffer are newly meaningful. Do not inspect unused capacity or stale bytes after a short read.

A read is not necessarily a complete application record. Network and file boundaries differ from protocol frames. Use io.ReadFull when a fixed byte count is required, or a parser that retains partial-record state. Distinguish a clean end before a record from truncation inside a required record.

Require exactly the documented bytes

The first program reads three bytes from a four-byte in-memory source. io.ReadFull either fills the requested slice or reports an error, returning the number transferred. The final byte remains in the reader for a later operation. This avoids the mistaken assumption that one arbitrary Reader.Read call must fill the entire buffer.

The fixed source makes the example reproducible and avoids external paths. For an untrusted frame length, validate a maximum before allocation and reading. A format length is not permission to allocate arbitrarily large memory. Include cumulative limits when a stream can contain many individually acceptable records.

go An exact-length in-memory read

package main

import (
	"fmt"
	"io"
	"strings"
)

func main() {
	input := strings.NewReader("ABCD")
	data := make([]byte, 3)
	n, err := io.ReadFull(input, data)
	if err != nil {
		fmt.Println(err)
		return
	}
	fmt.Printf("%d %q remaining=%d\n", n, string(data), input.Len())
}

Expected output

3 "ABC" remaining=1

ReadFull fills three bytes with ABC and leaves D unread.

Scanning needs an error and size policy

bufio.Scanner provides convenient token iteration and a configurable maximum token size. Configure its buffer before scanning when the application permits larger bounded records, and inspect Scanner.Err after Scan returns false. Otherwise a too-long record can be mistaken for ordinary end of input. Its default line split removes the line ending rather than preserving it in each token.

The second program uses small known lines and checks the final error. It counts logical records, including an empty line if one is present under the split policy. For formats needing precise delimiter retention or controlled recovery after oversize input, bufio.Reader or a custom bounded parser may be a better fit.

go A bounded line scanner

package main

import (
	"bufio"
	"fmt"
	"strings"
)

func main() {
	scanner := bufio.NewScanner(strings.NewReader("red\nblue\n"))
	scanner.Buffer(make([]byte, 64), 1024)
	count := 0
	for scanner.Scan() {
		count++
		fmt.Println(scanner.Text())
	}
	if err := scanner.Err(); err != nil {
		fmt.Println(err)
		return
	}
	fmt.Println("records", count)
}

Expected output

red
blue
records 2

Two tokens are emitted without their line endings, then successful completion is checked.

The opener usually owns the close

An operation opening an os.File should arrange closing after successful acquisition. A borrowed io.Reader does not automatically transfer closing responsibility. defer is appropriate for function-scoped ownership, while loops may need per-iteration helpers to avoid keeping many files open. File creation and truncation affect persistent state, so examples should use temporary paths rather than arbitrary user files.

Check write errors and relevant close errors when delivery matters. A successful write into buffering is not necessarily durable storage. Closing a file and crash-resistant persistence are separate concerns with platform-specific requirements. Keep filesystem replacement protocols outside a generic parsing contract.

A byte limit is one layer of validation

io.LimitReader stops after a bound but does not automatically tell you whether the original source had more data. Reading at most limit plus one can detect oversize content when the bound is chosen safely. After acquisition, validate syntax, numeric range, domain rules, and trailing records separately. A limited read can still contain invalid data.

Reusable parsers should return context-rich errors without logging the same failure at every layer. Keep synthetic malformed examples in tests: empty input, exact boundary, one byte beyond, truncated frame, and a reader returning both data and EOF. The correct result depends on record policy, not merely on whether some bytes arrived.

Common errors and fixes

The final bytes disappear at EOF.

Why it happens: The loop handles err before processing n.
How to fix it: Consume the first n bytes, then classify the error.
Prevent it next time: Test a Reader returning data and io.EOF together.

An oversized line looks like clean completion.

Why it happens: Scanner.Err was ignored.
How to fix it: Check the terminal error and configure a bounded token limit.
Prevent it next time: Test the limit and one byte beyond it.

A parser closes a stream its caller still needs.

Why it happens: Borrowing and ownership were unspecified.
How to fix it: Leave borrowed streams open unless transfer is explicit.
Prevent it next time: Document who acquires and closes each resource.

Check your understanding

1. A four-byte frame read returns n=2 and io.ErrUnexpectedEOF. What must the parser do with its application state?

Show worked solution

Treat the frame as incomplete and do not interpret unfilled bytes. Report truncation with the frame context and preserve the previous committed application state unless the API explicitly documents partial results. The two bytes may be retained only in a designed incremental parser. Cleanup remains required even though data validation failed.

2. Explain why io.ReadAll(io.LimitReader(r, 100)) does not prove the source has at most one hundred bytes.