Apache Spark / Intermediate: distribution and execution evidence / Lesson 9

Parquet layout and safe publication

Design file-backed datasets with explicit schema, pruning, publication and recovery boundaries.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Separate column pruning, partition discovery and output layout.
Explain why plain file writes are not a universal transactional table.

A columnar format helps selected-column workloads

Parquet stores data in a columnar format with schema and metadata that readers can use. A report needing site_id and kwh should project those fields rather than transfer every diagnostic payload. Column pruning and supported predicate pushdown can reduce work, but their actual effect depends on the source, expression and plan. Compression also changes the relationship between file bytes and in-memory data size. A small compressed directory is not proof that collect fits in driver memory. Keep format, schema and query requirements together, and verify the actual read path instead of assuming a file extension guarantees optimal execution.

Read a deliberate snapshot with bounded projection

The PySpark fragment assumes a synthetic directory whose schema includes reading_day as a date. It builds a plan selecting one day and three columns. A reader can discover partition columns from supported directory structure, but reading only a subdirectory or choosing an incorrect base path can change discovery. Inspect the resulting schema and plan before trusting a missing partition column. No directory is opened here. File inventories should be stable for a published batch; a directory modified by another writer during a run is not automatically a coherent source snapshot.

python Illustrative technical example; logical results are reasoned, not engine-tested.

daily = (spark.read.parquet('lab-input/curated-readings')
    .where("reading_day = DATE '2026-10-01'")
    .select('site_id', 'reading_id', 'kwh'))
daily.explain(mode='formatted')

Illustrative plan over a fictional lab directory. The engine must establish actual pruning; none is measured here.

Choose directories from downstream access patterns

partitionBy('reading_day') creates directory layout by day when writing, not a promise of one physical file per day. Many tiny output files can increase listing and open overhead; one enormous file can reduce useful parallelism. Partitioning by a high-cardinality meter identifier can create a directory explosion. Select a bounded layout from common filters and retention needs, then tune execution partitions separately. Schema merge is not free semantic reconciliation, and changing units under the same column name remains a contract violation even when every Parquet footer can be read.

Publish a new version rather than casually overwriting

The illustrative write uses errorifexists at a new synthetic batch path, so the example does not recommend overwriting a configured dataset. In an actual approved system, a successful write still needs source-count reconciliation and an agreed publication step before readers use it. Spark output can be a directory containing part files and metadata rather than one named CSV or Parquet file. Object-store commit behavior and connectors have deployment-specific requirements. Plain Parquet does not itself supply multi-writer ACID transactions, rollback history or catalog governance; external table formats can add such capabilities under their own documented guarantees.

python Illustrative technical example; logical results are reasoned, not engine-tested.

(curated.write.mode('errorifexists')
    .partitionBy('reading_day')
    .parquet('lab-output/readings-batch-20261001-v1'))

Illustrative write fragment only; no path is created or changed. File count and commit behavior are not asserted.

A readable dataset needs a reproducible manifest

Record source batch identity, schema version, row counts, rejected counts, output location and the publication version. Verify representative keys and energy totals, not just a successful task status. A failure after some files exist requires the chosen commit and cleanup protocol, not arbitrary deletion of shared paths. Keep readers on the last validated version until the new version is accepted. This course performs no file read, write, deletion or storage benchmark. Standard-library validation checks only the illustrative plan text and safe manifest models, with no claim of Parquet or object-store engine testing.

Common errors and fixes

A date directory is expected to contain one file

Why it happens: Logical directory partitioning was confused with task output layout.
How to fix it: Inspect the produced file inventory and consumer contract.
Prevent it next time: Review directory cardinality and small-file overhead separately.

A missing partition column is filled with guessed values

Why it happens: The read path or partition-discovery base was not verified.
How to fix it: Inspect directory scope, basePath needs and the actual schema.
Prevent it next time: Test representative root and subdirectory reads in an authorized lab.

A successful overwrite is treated as a rollback plan

Why it happens: File replacement was confused with versioned transactional history.
How to fix it: Define versioned publication and supported recovery explicitly.
Prevent it next time: Verify connector and table-format guarantees before relying on them.

Check your understanding

1. Does partitionBy("reading_day") guarantee one Parquet file for each day?

Show worked solution

No. It defines directory layout; execution tasks, writer behavior and other factors determine files. Review both downstream pruning and file inventory. A high-cardinality directory key can create operational cost even if the write succeeds.

2. What should readers do while a new batch version is only partially written?