pandas / Basic: labels, types and accepted input / Lesson 3

Dtypes, nullable values and missingness

Choose a representation that preserves meaning and keep missing observations distinct from zero, empty text and invalid input.

About 5 min readContent reviewed 2026-10-28

On this page

Lesson goals

Explain nullable extension dtypes and dtype-dependent missing sentinels.
Design explicit missing-data and aggregation policies.

A dtype carries more than storage size

A dtype determines representation and influences arithmetic, comparison, missingness and conversion. Inventory units can use an explicit nullable integer dtype when a count is unknown; converting missing values to zero instead asserts a business fact. NumPy integer types, pandas Int64 and text representations have different behavior, so capitalization matters. A column that looks numeric can still contain strings or mixed objects after import. Inspect and validate the intended type rather than judging display alone. Keep identifiers such as0017 as text when leading zeros are meaningful, because converting them to an integer changes identity rather than improving data quality.

Detect missingness through the public API

pandas uses dtype-dependent sentinels such as NaN, NaT and pd.NA. isna and notna express missingness checks without assuming one sentinel compares equal to itself. Empty text, the literal word unknown and zero are not automatically the same missing fact. The explicit string extension dtype uses pd.NA by default, while the inferred str dtype introduced in 3.0 uses NaN semantics; backend choices can also affect representation. Nullable Boolean logic can preserve unknown results, and converting pd.NA directly to a scalar truth value is ambiguous. Decide how unknown conditions enter a filter instead of silently treating every unknown as a confirmed false business observation.

Read a deliberate nullable-count program

The illustrative pandas program creates counts one, missing and three with dtype Int64. isna identifies only the middle entry, and fillna(0) produces a declared comparison version with zero in that position. The code does not claim zero is the correct business imputation; it demonstrates the mechanical consequence of that choice. Aggregations commonly skip missing values under documented defaults, so a sum can exist even when some inputs are unknown. An all-missing result and an observed zero total need separate handling, including appropriate min_count behavior where supported. Expected assertions are derived and were not run against pandas.

python Illustrative nullable integer missingness

import pandas as pd

counts = pd.Series([1, None, 3], dtype='Int64')
assert counts.isna().tolist() == [False, True, False]
assert counts.fillna(0).tolist() == [1, 0, 3]
print(counts.isna().tolist())

Expected output

[False, True, False]

Expected missingness marks only the middle value; fillna demonstrates a policy choice rather than endorsing zero imputation.

Keep the denominator visible in an offline calculation

The standard-library model averages only the two observed values and separately reports one missing observation. The result two is the observed-value mean, not a complete-population estimate and not the mean after zero imputation. Stating the denominator makes that distinction reviewable. A missing value can mean unavailable, not applicable, redacted or parsing failure; those causes can need different treatment even when the analytical representation uses one sentinel. Preserve source-quality flags when important. Never coerce invalid text to missing and then silently drop it from a mean without accounting for how the population changed.

python Offline observed-value denominator model

values = [1, None, 3]
observed = [value for value in values if value is not None]
missing_count = len(values) - len(observed)
assert sum(observed) / len(observed) == 2
assert missing_count == 1
print(sum(observed) / len(observed), missing_count)

Expected output

2.0 1

The observed mean is 2.0 with one missing input; this does not implement pandas nullable arithmetic.

Treat conversion and imputation as separate decisions

A robust pipeline validates allowed source forms, converts to the selected dtype and then applies a documented missing-data policy. errors=coerce in a conversion API is a choice to represent failures as missing, not proof that the source is clean. Track newly introduced missing values and compare them with known missing input. Test a valid zero, an absent count, invalid text and an all-missing group. This lesson executes only the plain-Python denominator model. pandas conversion, nullable arithmetic, backend representation and missing-mask behavior remain unexecuted illustrations whose precise semantics are linked to the selected official documentation.

Common errors and fixes

Unknown units are silently filled with zero.

Why it happens: Missingness was confused with an observed absence of stock.
How to fix it: Define the business missing-value policy and retain uncertainty where required.
Prevent it next time: Test valid zero and unknown input as different fixtures.

boolean value of NA is ambiguous

Why it happens: A nullable unknown was forced into one scalar truth value.
How to fix it: Use explicit missingness checks and a reviewed Boolean policy.
Prevent it next time: Include unknown conditions in filtering acceptance tests.

A conversion creates missing values that disappear from summary statistics.

Why it happens: Coercion failures and aggregation denominators were not tracked.
How to fix it: Count newly invalid/missing values and reconcile the analyzed population.
Prevent it next time: Validate source forms before applying imputation or dropping rows.

Check your understanding

1. Is the mean of observed 1 and 3 with one missing value necessarily the mean of the whole population?

Show worked solution

No. Two is the mean of the observed subset. The missing observation can change the complete-population mean, and replacing it with zero answers another question. Report observed count, missing count and the policy used so readers can judge the result’s coverage rather than treating one number as complete evidence.

2. Why should inferred str and explicitly requested string not be assumed to have identical missing sentinels?