Data Analysis / Basic: Questions and Trustworthy Tables / Lesson 2

Data Types, Measurement, and CSV

Read tabular text under an explicit schema and keep identifiers, quantities, units, and encodings distinct.

About 4 min readContent reviewed 2026-10-25

On this page

Lesson goals

Classify identifiers, categories, counts, and continuous measurements.
Parse quoted CSV without guessing types from appearance.
Normalize units while preserving raw values.

A numeric-looking value may be a label

An identifier such as003 is a label, not a quantity to average. A nominal category has no implied order; an ordinal scale has order without guaranteed equal spacing. Counts and measured quantities permit different operations and may have different meaningful zeros. A codebook should state the interpretation, allowed values, units, and absence markers for each field.

Storage type and analytical type are separate. A CSV cell begins as text, while a Python float can hold a category code without making arithmetic meaningful. Choose a representation after identifying the field’s role. Preserve leading zeros in identifiers and do not infer calendar dates from ambiguous number patterns.

CSV is a dialect, not a split operation

CSV can contain quoted delimiters, embedded line endings, escaped quotes, and different separators. Use a parser with the documented dialect rather than splitting every line on commas. For real text files, specify encoding and open with newline="" for Python’s csv module. An accepted row still needs header, field-count, and domain validation.

The first complete program preserves a string identifier and parses a decimal amount explicitly. The quoted region contains a comma but remains one field. Decimal constructed from text avoids first converting through binary float; it does not automatically decide rounding or the business meaning of an amount.

python Parse a quoted field and preserve an identifier

import csv
import io
from decimal import Decimal

source = io.StringIO('id,region,amount\n003,"North, west",12.50\n', newline="")
row = next(csv.DictReader(source))
amount = Decimal(row["amount"])
print(f'id={row["id"]} region={row["region"]} amount={amount}')

Expected output

id=003 region=North, west amount=12.50

The comma stays inside the region field, and003 remains a three-character identifier.

A schema makes assumptions testable

Define required columns, their meanings, allowed null markers, uniqueness expectations, and conversion rules. Duplicate headers can silently overwrite meaning in dictionary-shaped records, so reject them before trusting row mappings. Extra or missing fields should follow an explicit policy rather than being ignored because some desired columns exist.

Inference is useful for exploration but risky as an unrecorded production contract. A column with early integer values and a later unknown token can change inferred type. In pandas, dtype and missing-value options deserve explicit review; this course does not execute pandas illustrations because that package is absent locally. The same schema reasoning applies to every tool.

Normalize units with a named rule

The second program converts milliseconds to seconds using an explicit factor. Keep the source unit and raw field available in an audit layer; do not overwrite a column named duration with a different unit without updating its contract. A thousand-fold discrepancy can be a unit mistake rather than an extraordinary observation.

Affine conversions need more than multiplication: temperatures with different zero points illustrate that distinction. Time zones and elapsed durations also differ. A timestamp identifies an instant or local civil time; a duration measures an interval. State which interpretation applies before subtracting or grouping values.

python An explicit unit conversion

milliseconds = [1000, 2500]
seconds = [value / 1000 for value in milliseconds]
print("seconds", ",".join(f"{value:.1f}" for value in seconds))

Expected output

seconds 1.0,2.5

Dividing by1000 maps1000ms to1.0s and2500ms to2.5s.

Record precision and decoding decisions

Binary floating-point represents many decimal fractions approximately. Formatting two decimals is a display choice, not evidence of exact arithmetic. Decide whether rounding belongs at collection, storage, calculation, or presentation, and avoid repeatedly rounding intermediates without justification. Exact decimal arithmetic can still embody an incorrect measurement or rounding rule.

Encoding errors and Unicode normalization can change categories or joins. Use known encoding, inspect failed decoding, and retain original tokens before canonicalizing. Case-folding or trimming is a domain transformation that can merge distinct labels. A correct parser only supplies candidate values; analytical validity still depends on their documented meaning.

Common errors and fixes

Identifier003 becomes3.

Why it happens: Type inference treated a label as an integer.
How to fix it: Preserve the identifier as text from ingestion.
Prevent it next time: Define identifier types before reading.

A quoted comma creates an extra column.

Why it happens: The input was split manually.
How to fix it: Use csv with the correct dialect.
Prevent it next time: Test quoted delimiters and embedded newlines.

Measurements differ by a factor of1000.

Why it happens: Milliseconds and seconds were mixed.
How to fix it: Normalize under a documented unit rule.
Prevent it next time: Store units in the data dictionary and tests.

Check your understanding

1. A column contains postal labels0012 and1200. Why should their average not be reported even if conversion to int succeeds?

Show worked solution

The labels identify areas rather than measured quantities. Their arithmetic mean has no defined postal interpretation, and conversion destroys leading-zero identity. Preserve strings, count categories, and join to a validated area dictionary if a geographic question requires it. Successful parsing does not establish a meaningful measurement scale.

2. Design checks before summing an amount column imported from CSV.