Python / Reliable programs / Lesson 12

Reliable records, CSV, and time

Build a small record-processing pipeline with explicit schema, correct format handling, and meaningful time representations.

About 4 min readContent reviewed 2026-10-23

On this page

Lesson goals

Separate format and domain
Control size and reporting policy

Separate format and domain

A data pipeline usually acquires bytes, decodes text, parses a format, validates records, transforms values, and emits a report. Each stage has a different failure vocabulary. A syntactically valid CSV row may still contain an impossible quantity. Keep the raw row available for a safe location reference, such as row number, while constructing a validated domain value separately. Decide whether one invalid row rejects the whole batch or is collected into a report; neither policy should emerge accidentally from a broad exception handler. Explicit stage boundaries make the same calculation reusable for files, command-line inputs, or a future service.

CSV needs a parser

Comma-separated values can contain quoted commas, quotes, and embedded newlines. Splitting a line on a comma cannot handle those rules correctly. csv.reader and csv.DictReader implement the format’s quoting behavior, with a dialect describing separators and related conventions. When opening a real CSV text file, use newline="" as documented and specify the agreed encoding. Require the headers your application needs, and distinguish a missing column from an empty cell. Decide how to treat duplicate headers and unexpected fields rather than letting dictionary construction silently define that policy. Format compatibility is a documented contract between producer and consumer.

Parse a quoted record correctly

The first example uses StringIO as an in-memory text file so it is complete and does not depend on a local dataset. DictReader maps the header to string values; quantity remains text until int converts it. The product name contains a comma inside quotation marks and still becomes a single field. This is exactly the situation that a naive split would misread. A production parser would add row-number context and validation before aggregation. Even in this small example, keep the conversion visible so the boundary between format parsing and numeric interpretation is clear to a reader.

python Parse a quoted record correctly — worked example

import csv
import io

data = 'name,quantity\n"Tea, green",3\nBread,2\n'
rows = csv.DictReader(io.StringIO(data, newline=""))
for row in rows:
    print(row["name"], int(row["quantity"]) * 2)

Expected output

Tea, green 6
Bread 4

The comma inside the quoted product name belongs to that field. Numeric conversion occurs after CSV parsing and doubles each validated demonstration quantity.

Instants, local times, and durations

A datetime can be naive, without timezone interpretation, or aware, carrying timezone information. A local clock reading alone may be ambiguous or nonexistent around daylight-saving transitions. For instants exchanged between systems, use an explicit timezone and an agreed representation; timezone.utc is available throughout this course. A timedelta expresses a duration, while subtracting dates or aware instants requires a clear semantic goal. The example constructs two UTC instants and reports elapsed minutes, avoiding machine-local timezone assumptions. For named regional rules, zoneinfo uses timezone data, whose availability depends on the operating system or an installed data package.

python Instants, local times, and durations — worked example

from datetime import datetime, timezone

start = datetime(2026, 1, 1, 9, 0, tzinfo=timezone.utc)
end = datetime(2026, 1, 1, 10, 30, tzinfo=timezone.utc)
print(int((end - start).total_seconds() // 60))
print(start.isoformat())

Expected output

90
2026-01-01T09:00:00+00:00

Both aware instants use the same explicit UTC timezone, so their difference is ninety elapsed minutes and the serialized start includes its offset.

Control size and reporting policy

Record processing should have explicit limits for file size, field length, row count, and retained error samples when inputs may be large. Streaming rows avoids loading all raw text, but accumulating every record or every error can still exhaust memory. Use counters and bounded samples when only a summary is needed. For reproducible reports, define ordering and rounding before emission. CSV intended for a spreadsheet also has a separate interpretation boundary: some spreadsheet applications treat certain leading characters as formulas, so escaping policy must be designed for the target consumer. A format parser alone does not make every downstream use safe.

Common errors and fixes

A quoted comma creates an extra field

Why it happens: The implementation uses string splitting instead of CSV rules.
How to fix it: Use csv.reader or DictReader with the expected dialect.
Prevent it next time: Include quoted delimiters and embedded newlines in sample data.

A time difference changes with the machine timezone

Why it happens: Naive values were implicitly interpreted as local instants.
How to fix it: Use explicit aware instants and document timezone conversion.
Prevent it next time: Test boundaries and avoid ambient local-time assumptions.

Streaming still consumes unbounded memory

Why it happens: The report retains every row or every error after parsing.
How to fix it: Aggregate incrementally and bound retained diagnostic samples.
Prevent it next time: Review retained state separately from input reading strategy.

Check your understanding

1. How would you report three invalid quantities in a large CSV without storing every row?

Show worked solution

Iterate with row numbers, count invalid rows, and retain only the first three safe error summaries. Continue according to the documented batch policy. Report the total invalid count and the bounded sample, making clear that the sample is incomplete if more failures occurred.

2. Why should a timestamp exchanged across systems include an offset or explicit timezone agreement?