pandas / Advanced: ownership, windows and analytical acceptance / Lesson 16

Memory, IO and bounded chunk processing

Reduce input and intermediate cost without changing global analytical invariants or combining partial statistics incorrectly.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Distinguish source size, column storage estimates and peak process memory.
Combine additive chunk statistics and preserve cross-chunk identity checks.

Input file size does not predict peak memory

A compressed source file, decoded arrays, indexes, Python objects and intermediate results consume different amounts of memory. A merge or sort can allocate additional structures even when the input fits comfortably. memory_usage(deep=True) provides a useful column-oriented estimate under its documented behavior, not a guarantee of total resident process memory or peak allocation. Select needed columns and appropriate validated dtypes before calculating large intermediates. A smaller dtype must still represent the allowed range and missing policy. Do not downcast identifiers or money merely to make a memory number smaller while changing meaning or overflowing valid values.

A chunk algorithm needs a valid combine rule

Sums and observed counts can be combined additively, but an unweighted average of chunk means is generally wrong when chunk sizes differ. Retain sum/count pairs to derive an overall observed mean, and document all-missing behavior. Distinct counts, medians, global sorting and duplicate-key checks need additional state or another algorithm; processing each chunk independently is not enough. A duplicate event split between two chunks can pass every local uniqueness test. State which global invariants the bounded state preserves and how its memory grows. Chunking is a design choice with algebraic requirements, not a switch that makes every operation safely out-of-core.

Combine two declared partial sums and counts

The illustrative pandas program reads four synthetic counts in chunks of two and accumulates each chunk’s sum and observed count. Their total sum is ten and count four, yielding mean 2.5. It does not demonstrate an all-missing chunk or global distinct count, and the program was not executed in pandas. Explicit dtype and usecols reduce ambiguity in the example without claiming a universal capacity improvement. A real reader needs lifecycle/error handling and reconciliation of rejected inputs. Repeatedly concatenating an ever-growing result can defeat bounded-memory intent, so decide what final state is actually required instead of retaining every chunk automatically.

python Illustrative chunk sum/count combination

from io import StringIO
import pandas as pd

source = StringIO('units\n1\n2\n3\n4\n')
partials = []
for chunk in pd.read_csv(source, usecols=['units'], dtype={'units': 'Int64'}, chunksize=2):
    partials.append((int(chunk['units'].sum()), int(chunk['units'].count())))
total = sum(partial[0] for partial in partials)
count = sum(partial[1] for partial in partials)
assert (total, count) == (10, 4)
print(total / count)

Expected output

2.5

Expected mean is 2.5 for four observed values; pandas reader and memory behavior were not tested.

Expose the mean-of-means error offline

The standard-library fixture has one partial sum ten/count two and another sum ninety/count one. The combined observed mean is 100/3, while averaging the two means equally gives 47.5. The mismatch is caused by weighting, not pandas precision. This arithmetic does not model parser memory, DataFrame storage or parallel execution. Use it to state the combine invariant before optimizing IO. For floating calculations, summation order and numeric stability can introduce additional differences, so define tolerances from the measurement contract rather than accepting arbitrary drift as a side effect of chunking.

python Offline weighted aggregation counterexample

partials = [(10, 2), (90, 1)]
combined = sum(total for total, _ in partials) / sum(count for _, count in partials)
mean_of_means = sum(total / count for total, count in partials) / len(partials)
assert round(combined, 2) == 33.33
assert mean_of_means == 47.5
print(f'{combined:.2f} {mean_of_means:.2f}')

Expected output

33.33 47.50

33.33 is the rounded combined mean;47.50 weights unequal chunks equally and answers another question.

Choose storage from the required access pattern

Columnar formats can support selecting fewer columns and preserving richer schema, but optional engines, filtering and dtype round-trips must be reviewed for the exact backend. They are not installed or tested here. Keep source versions, chunk size, parser settings and combine-state limits with the result. Benchmarking needs a representative authorized environment and a defined peak-memory metric; no such experiment occurs in this course. Only the standard-library weighted arithmetic executes. pandas chunk reading and memory claims remain illustrative, and no external file, database, project dependency or optional IO package is accessed or changed.

Common errors and fixes

Chunk means are averaged equally despite different observed counts.

Why it happens: The combine rule discarded denominators.
How to fix it: Retain additive sums and observed counts or the intended explicit weights.
Prevent it next time: Test unequal chunk sizes and missing counts.

Every chunk is unique but the final input contains repeated event IDs.

Why it happens: A global uniqueness invariant was checked only locally.
How to fix it: Design bounded cross-chunk identity state or a suitable global method.
Prevent it next time: Place a deliberate duplicate across a chunk boundary.

A small compressed file causes unexpectedly high peak memory.

Why it happens: Decoded storage and intermediate allocations were omitted.
How to fix it: Measure the actual runtime envelope and reduce unneeded columns or intermediates.
Prevent it next time: Separate source bytes, column estimates and process peak metrics.

Check your understanding

1. Why is averaging chunk means not a safe general combine rule?

Show worked solution

It gives each chunk equal weight regardless of observed count. Unless that is the intended question or counts are equal, the result differs from the overall observed mean. Preserve sums and counts, handle zero-observation chunks explicitly and derive the final denominator before discussing performance.

2. Does chunksize alone prove the whole pipeline has bounded memory?