Data Engineering / Intermediate: change, time and coordination / Lesson 10

Partitions, file layout and compaction

Align dataset layout with legitimate access and correction patterns. Distinguish logical partitions, file statistics and compaction from guaranteed performance or atomic publication.

About 4 min readContent reviewed 2026-10-26

On this page

Lesson goals

Choose partition keys from query, retention and late-data needs.
Preserve logical content while changing physical files.

Partitioning is a tradeoff, not a universal accelerator

A partition groups data by a chosen key such as event day or tenant. Useful predicates can reduce inspected files, while excessive small partitions increase metadata and scheduling overhead. A high-cardinality key can create many tiny files, and a skewed key can concentrate most work in one partition. Choose layout from access patterns, write rates, retention and correction behavior rather than a rule that every dataset must partition by date. A partition path is not itself proof that its rows have matching values or that the expected partition is complete.

Select declared files without pretending to scan storage

The offline manifest lists two invented partitions with row counts. Selecting the desired event day identifies one file and twenty rows under this supplied metadata. It does not prove those rows exist, satisfy their declared day or have trusted statistics. Actual engines may prune partitions or row groups under supported expressions and metadata rules; incompatible casts or missing predicates can defeat useful pruning. Keep partition identity and file/snapshot version in the accepted dataset contract, and verify rows against metadata during a controlled publication check.

python Offline supplied-manifest selection

manifest = [
    {'file': 'part-a', 'event_day': '2026-10-01', 'rows': 20},
    {'file': 'part-b', 'event_day': '2026-10-02', 'rows': 30},
]
selected = [f for f in manifest if f['event_day'] == '2026-10-01']
assert [f['file'] for f in selected] == ['part-a']
assert sum(f['rows'] for f in selected) == 20

The result follows invented metadata only. No file system, object store or query engine is scanned.

A historical correction can touch an old partition

If partitioning uses event day, late updates may rewrite or append to historical partitions. A partition overwrite must include the complete intended replacement scope; overwriting with only the latest changed rows can delete valid untouched rows. Define merge, replace and append semantics separately. Readers should see an accepted file set or snapshot rather than a mix of old and new pieces. Concurrency between a backfill and live writer needs a supported conflict/publication protocol, not just different task names. Retention must account for readers and recovery references before old files are removed.

Prove content preservation when combining pieces

Compaction combines physical pieces to reduce overhead while preserving the declared logical records. The local model compares multisets before and after grouping into one list, so duplicate occurrence counts remain part of the invariant. It does not create Parquet files or implement a table-format transaction. Real compaction must preserve schema, delete semantics, partition meaning and snapshot references, and requires sufficient temporary capacity. A lower file count is not proof of correctness if valid records were dropped or duplicated. Record both physical improvements and logical reconciliation evidence.

python Offline multiset-preserving compaction model

from collections import Counter

parts = [[('e1', 10)], [('e2', 20), ('e2', 20)]]
before = Counter(row for part in parts for row in part)
combined = [row for part in parts for row in part]
assert Counter(combined) == before
assert len(parts) == 2 and len(combined) == 3

The repeated e2 remains repeated; this model changes physical grouping, not logical deduplication policy.

Evaluate bytes, files and mutation cost together

Column projection, compression, row-group statistics and partition pruning address different access costs. Parquet is a file format; Iceberg and similar table formats add their own metadata and commit protocols above files. Do not claim an atomic multi-file update merely because each file is valid. Measure representative read patterns, file count, scanned bytes, write amplification and maintenance capacity in the actual approved engine. Keep tool versions and layout decisions with the run lineage so a later performance regression or incorrect partition can be traced to a specific change rather than guessed from a dashboard.

Common errors and fixes

A changed-row overwrite removes unchanged records

Why it happens: Partial changes were treated as a complete partition replacement.
How to fix it: Rebuild the full replacement scope or use supported merge semantics.
Prevent it next time: Test corrections alongside untouched rows.

Many tiny partitions increase runtime

Why it happens: Metadata/task overhead was excluded from the layout decision.
How to fix it: Review cardinality, file sizes and supported compaction.
Prevent it next time: Measure representative file and byte costs.

Compaction silently deduplicates records

Why it happens: Physical reorganization was confused with logical cleanup.
How to fix it: Preserve the intended multiset unless deduplication is a separate accepted transform.
Prevent it next time: Reconcile occurrence counts and values.

Check your understanding

1. A backfill replaces yesterday’s partition with only corrected rows. What can be lost?

Show worked solution

Valid unchanged rows can disappear because replacement usually defines the whole selected scope. Reconstruct the complete partition or use an explicitly supported merge/update protocol, then reconcile all expected keys. A task named correction does not change storage overwrite semantics.

2. Does a successful Parquet compaction prove atomic dataset publication?