pandas / Basic: labels, types and accepted input / Lesson 4

CSV import contracts and validation

Preserve identifiers and missing-value meaning at ingestion, then validate structure and coverage before analysis.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Choose explicit dtype, delimiter, encoding and missing-token policies.
Separate successful parsing from valid analytical input.

A CSV file carries syntax but little schema

CSV represents rows and fields, but does not reliably declare identifier types, units, business grain or missing-value meaning. A field 0017 can be an item code rather than the number seventeen. The text NA can be a valid region code instead of a missing token. Define delimiter, quoting, encoding, header names, allowed forms and expected row coverage before selecting parser options. A syntactically accepted file can still contain duplicate identities, impossible quantities or the wrong reporting interval. Keep the original synthetic source and validation evidence separate from a cleaned result so conversion decisions remain reviewable.

Choose missing tokens per column

The illustrative read_csv program uses an in-memory source, explicit nullable dtypes and keep_default_na=False. It preserves 0017 and the literal identifier NA as strings, while treating an empty units field as missing through column-specific na_values. These choices are intentional and not a universal import recipe. A different source can use different missing markers or require rejecting blank quantities entirely. The declared output is derived from the fixture; pandas was not executed. A parser’s accepted dtype should still be followed by domain checks such as nonnegative units, allowed identifiers and row-grain uniqueness.

python Illustrative explicit CSV import policy

from io import StringIO
import pandas as pd

source = StringIO('item,units\n0017,2\nNA,\n')
frame = pd.read_csv(source, dtype={'item': 'string', 'units': 'Int64'}, keep_default_na=False, na_values={'units': ['']})
assert frame['item'].tolist() == ['0017', 'NA']
assert frame['units'].isna().tolist() == [False, True]
print(frame['item'].tolist())

Expected output

['0017', 'NA']

Expected identifiers are preserved as text and only the blank units field is missing; pandas was not run.

Parsing and conversion failures need accounting

A malformed row, an invalid numeric token and a valid missing observation are different cases. Skipping bad lines or coercing invalid values can reduce the population silently unless the pipeline records what changed. Decimal and thousands separators can alter numeric interpretation, while encoding errors can alter identifiers or prevent reading. Choose the parser engine only after checking support for the required options and file format. Do not infer correctness because read_csv returned a DataFrame. Reconcile source row counts, rejected rows, newly missing values and accepted rows under the declared contract rather than optimizing away evidence.

Inspect exact source tokens with the standard library

The csv.DictReader model reads the same two-row fixture as plain strings. It checks that leading zeros and the literal NA remain intact and that the second units token is empty. This is a source-token check, not verification of pandas extension dtypes or missing-value inference. Quoted commas and embedded newlines require a real CSV parser rather than splitting lines on commas. Validate expected headers before using dictionary keys, and distinguish an empty file from a valid header-only report. A complete input contract should state which condition is accepted and which requires a terminal quality error.

python Offline source-token parsing check

import csv
from io import StringIO

rows = list(csv.DictReader(StringIO('item,units\n0017,2\nNA,\n')))
assert [row['item'] for row in rows] == ['0017', 'NA']
assert rows[1]['units'] == ''
print([row['item'] for row in rows])

Expected output

['0017', 'NA']

The standard parser preserves exact text tokens; it does not establish pandas dtype or NA behavior.

Keep ingestion reproducible and bounded

Record source identity, content version, schema, parser options and selected pandas/Python versions. Do not read an arbitrary latest file when reproducing a historical result. Large inputs may need chunking or another storage format, but that choice must preserve global uniqueness and aggregation rules rather than checking each chunk in isolation. This lesson reads only an in-memory synthetic string. It installs no parser backend, accesses no remote file and opens no database. The pandas example remains non-runnable; only the standard-library parsing assertions are executed, with their narrower claim retained in the validation report.

Common errors and fixes

Item 0017 becomes 17 and no longer matches reference data.

Why it happens: Identifier text was inferred or converted as a number.
How to fix it: Read the identifier with an explicit text contract and reconcile altered keys.
Prevent it next time: Include leading-zero fixtures in ingestion tests.

A legitimate NA identifier disappears into missingness.

Why it happens: Default missing-token recognition conflicted with source meaning.
How to fix it: Choose deliberate missing-token settings, preferably column-specific where needed.
Prevent it next time: Test literal business codes and blank fields separately.

A cleaned result omits malformed rows without any count of rejection.

Why it happens: Parser error handling silently changed analytical coverage.
How to fix it: Preserve rejection evidence and reconcile accepted versus source population.
Prevent it next time: Make every skip or coercion policy observable.

Check your understanding

1. Does a successful read_csv call prove a file contains one valid row per item?

Show worked solution

No. Parsing establishes only part of ingestion. Validate identity, grain, allowed values, interval and source coverage separately. A frame can load successfully while containing duplicate items or invalid quantities, so retain a quality report rather than treating object construction as analytical acceptance.

2. Why does the fixture disable default NA tokens but still mark blank units as missing?