Data Engineering / Basic: contracts and repeatable batches / Lesson 2

Formats, encoding and file boundaries

Choose interchange and analytical formats from type, access and compatibility needs. Parse quoted records correctly and distinguish a file from an atomically published dataset.

About 4 min readContent reviewed 2026-10-26

On this page

Lesson goals

Explain CSV, JSON and columnar format tradeoffs.
Preserve encoding, record boundaries and source identity.

A file extension does not establish a usable schema

CSV is convenient for tabular interchange but requires agreed delimiter, quoting, header, encoding and value interpretation. JSON carries nested structure but does not represent every database type or exact numeric policy automatically. Columnar formats such as Parquet organize data for selective analytical reads with metadata and encoded column chunks. They are not interchangeable text serializations. Choose from consumer needs, row/column access, evolution and ecosystem support rather than assuming the smallest file or a familiar extension is universally best.

Parse records rather than splitting on commas

The complete offline fixture includes a comma inside a quoted customer label. Python’s CSV parser returns two records with the intended two fields; a naive split would invent an extra field. Newlines can also occur inside quoted CSV fields, so physical lines and logical records are not always identical. Keep the dialect and header policy explicit and preserve the original input when rejecting a malformed row. This example operates only on an in-memory string and does not claim to read every real supplier’s dialect.

python In-memory quoted CSV fixture

import csv
import io

text = 'id,label\n1,' + chr(34) + 'Quay, East' + chr(34) + '\n2,North\n'
rows = list(csv.DictReader(io.StringIO(text)))
assert rows == [{'id': '1', 'label': 'Quay, East'}, {'id': '2', 'label': 'North'}]
assert len(rows) == 2

The quoted comma remains part of the label, and all parsed values remain strings until an explicit type conversion.

Decode bytes before interpreting values

An agreed Unicode encoding prevents the same bytes being interpreted differently across systems. Decode errors should produce explicit evidence rather than arbitrary replacement that corrupts identifiers or joins. A byte-order mark, inconsistent line endings or a header repeated mid-file can also change ingestion. JSON number handling depends on the parser: a binary float may not preserve an exact decimal amount. Treat encoding, parsing and type conversion as separate boundaries so the error identifies the failed layer rather than silently returning a plausible empty dataset.

Choose the numeric parser deliberately

This JSON fixture uses Decimal for decimal tokens and keeps the currency unit beside the amount. It demonstrates parser configuration, not a general accounting or schema-validation library. The exact value is 12.30 and the currency remains the declared synthetic code. Missing fields, booleans, integer identifiers and very large values still need their own rules. Do not infer approved units from a numeric shape or coerce malformed financial text into zero merely to complete a batch. Document the destination representation as well as the input parser.

python In-memory JSON decimal parsing

import json
from decimal import Decimal

row = json.loads('{"amount":12.30,"currency":"LAB"}', parse_float=Decimal)
assert row['amount'] == Decimal('12.30')
assert str(row['amount']) == '12.30'
assert row['currency'] == 'LAB'

The decimal token preserves its exact declared value and scale in this parser. No external file, currency rate or service is involved.

Keep format metadata and dataset publication separate

Parquet readers use file metadata and row-group/column organization to avoid unnecessary reads when supported predicates and statistics permit it. A file being Parquet does not prove its partitions are complete or its schema compatible with every reader. Dataset publication should identify a complete file set or snapshot instead of exposing a half-written collection. This course does not install a Parquet engine; its format discussion is conceptual and grounded in official documentation. Validate chosen writer/reader versions and logical types in the actual approved environment before claiming interoperability or a performance gain.

Common errors and fixes

A quoted comma creates an extra column

Why it happens: The input was split as text instead of parsed with its CSV dialect.
How to fix it: Use a proper parser with an explicit dialect and header contract.
Prevent it next time: Test commas, quotes and embedded newlines.

Identifiers change during decoding

Why it happens: Encoding errors were silently replaced or misinterpreted.
How to fix it: Preserve source bytes and correct the agreed encoding boundary.
Prevent it next time: Include multilingual and invalid-byte fixtures.

A columnar dataset is assumed complete because files open

Why it happens: File readability was confused with dataset completeness and publication.
How to fix it: Validate the expected file set, schema and accepted version.
Prevent it next time: Publish through a declared completeness boundary.

Check your understanding

1. Why is counting physical CSV lines not always the number of data records?

Show worked solution

Quoted fields can contain newlines, and headers or blank-line policies affect the count. Parse logical records using the declared dialect, then report accepted and rejected records separately. Do not use a raw line count as exact completeness evidence without defining those conditions.

2. Does converting CSV to Parquet prove downstream meaning is preserved?