Apache Spark / Intermediate: distribution and execution evidence / Lesson 8

Partitions, shuffles and balanced work

Separate distribution, file layout and business keys while reasoning about skew and stage completion.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Explain narrow dependencies and shuffle boundaries.
Choose repartition or coalesce from an observed requirement.

Local transformations differ from key redistribution

A filter or projection can often process each input partition independently. Grouping readings by site usually requires related keys to meet, which can introduce an Exchange and shuffle. The exact physical plan can exploit existing distribution or other optimizations, so do not label every API call as one mandatory shuffle. Network transfer, serialization, shuffle files and downstream sorting all consume resources. Count the bytes and key distribution crossing a boundary rather than judging cost from the number of Python statements. A short groupBy can be more expensive than a long chain of selective projections.

Repartition expresses a requested distribution

The illustrative fragment requests four partitions by site_id. Four is a scenario value, not a recommendation. Hash distribution groups the same key consistently for the relevant operation, but it does not promise equal row counts or a particular public numeric partition identifier. A site containing most observations remains concentrated. repartition can introduce shuffle to change distribution, whereas coalesce commonly reduces partitions through a narrower dependency and does not provide general rebalancing. Inspect the resulting plan and workload rather than assuming fewer partitions always means less execution time. No repartition operation is run here.

python Illustrative technical example; logical results are reasoned, not engine-tested.

by_site = readings.repartition(4, 'site_id')
smaller = by_site.coalesce(2)

Assumes an existing synthetic readings DataFrame. The requests do not guarantee balanced bytes, two output files or a faster job.

The slowest necessary partition controls a stage tail

Imagine partition work units10,10,10,70. Total work is100, but the largest partition is70 rather than the balanced ideal25. This arithmetic is a local workload model, not measured task duration: real execution also depends on cores, scheduling, I/O and contention. More executors cannot split one indivisible skewed task automatically in every operation. Investigate hot keys, large records and join multiplication before increasing resource counts. A long tail can coexist with low average executor utilization, because averages hide the few remaining expensive tasks.

json Illustrative technical example; logical results are reasoned, not engine-tested.

{"work_units":[10,10,10,70],"total":100,"largest":70,"balanced_ideal":25,"scope":"local arithmetic model"}

Only the sum, maximum and ideal average are validated locally. These units are not milliseconds or Spark metrics.

Three meanings of partition must stay separate

Input scan partitions determine how files are read, shuffle partitions organize redistributed execution, and partitionBy in a file writer creates directory-based layout by column values. They influence one another but are not identical counts. A date directory can contain many files, and one file can contribute several scan splits. An output partition per meter identifier can create excessive directories and small files. Choose a bounded layout from downstream filtering and lifecycle needs, while choosing execution distribution from work balance. Do not make a business directory naming scheme carry every parallelism decision.

Change distribution after identifying the bottleneck

Compare authorized task duration, input and shuffle bytes, spill, record counts and key frequencies. Many tiny tasks suggest a different problem from one oversized task. Adaptive execution can coalesce or split eligible post-shuffle work in supported cases, but it does not redefine the business aggregation or cure every skew pattern. A proposed salting strategy must include a correct second aggregation and careful join semantics; adding random keys without reconstruction changes results. This course runs no cluster or load test. It validates only the stated arithmetic and leaves all Spark distribution fragments non-runnable.

Common errors and fixes

coalesce is expected to redistribute a hot key evenly

Why it happens: Partition reduction was confused with key rebalancing.
How to fix it: Inspect skew and choose a semantically valid redistribution strategy.
Prevent it next time: Compare per-task distributions rather than only partition count.

One output directory is assumed to mean one task

Why it happens: File layout partitioning was confused with execution partitioning.
How to fix it: Record scan, shuffle and writer layout decisions separately.
Prevent it next time: Test downstream file inventory alongside runtime metrics.

Random salting changes the energy total

Why it happens: The reconstruction step did not preserve the original grouping contract.
How to fix it: Prove the partial and final aggregation with a small exact fixture.
Prevent it next time: Review correctness before applying skew optimization.

Check your understanding

1. For work units10,10,10,70, what are total and balanced average?

Show worked solution

The total is100 and the ideal average is25 across four partitions. The maximum70 exposes imbalance, but these numbers cannot predict elapsed time without a runtime model. Inspect actual task metrics before attributing a slow stage solely to executor count.

2. Does repartition(4,"site_id") guarantee four equally sized site groups?