Apache Spark / Intermediate: distribution and execution evidence / Lesson 10
Execution plans, statistics and adaptive execution
Use logical and physical evidence to evaluate optimizer choices without claiming that a plan is a benchmark.
On this page
Lesson goals
- Distinguish plan stages and estimate quality.
- Explain adaptive execution and its semantic limits.
The optimizer rewrites a declared relation
DataFrame and SQL expressions describe logical results, leaving Spark opportunities to simplify projections, push eligible filters and choose physical operators. The analyzed plan resolves names and types; the optimized plan applies transformations; the physical plan describes execution strategies. These layers are related but answer different questions. An unresolved column is not a cluster-sizing failure, and an expensive Exchange is not necessarily an incorrect query. Begin with the business result contract and input grain, then inspect the layer relevant to the symptom. A physical plan cannot prove that the selected source rows represent complete daily reporting.
Label exactly which plan was inspected
The fragment requests formatted and cost-oriented explanations for an existing synthetic summary. explain describes planning evidence and does not establish actual elapsed time, spill or network traffic. Cost output can expose statistics used by planning, but missing or stale estimates still require interpretation. Keep query parameters, source version and relevant configuration with captured plans. A plan from a tiny local fixture may differ from a distributed run over realistic data. No plan is collected here, and no expected operator text is presented as if copied from an executed job.
summary.explain(mode='formatted')
summary.explain(mode='cost')Illustrative plan-inspection calls only. They do not supply measured runtime metrics or prove a performance improvement.
Read an operator in its data-flow context
An Exchange indicates a redistribution boundary in a physical plan. A HashAggregate may appear before and after that boundary as partial and final work; a Sort can support ordering or a join strategy. The schematic below is deliberately not exact Spark output. It makes a likely aggregation flow visible without promising the optimizer will choose it. Compare estimated row counts and bytes with authorized observed metrics when available. If a join accidentally multiplies readings, improving its physical operator only computes the wrong answer faster. Correctness remains the first acceptance check.
Parquet scan: required columns and eligible filters
Partial aggregation by site_id
Exchange: redistribute site groups
Final aggregation
Optional final orderingA schematic reasoning aid, not a captured physical plan or an assertion about operator names in every run.
AQE reacts to runtime evidence within supported rules
Adaptive Query Execution can revise eligible physical choices using runtime statistics, including post-shuffle partition coalescing, some join-strategy changes and skew handling. Its actual behavior depends on version, configuration and plan eligibility. It does not validate duplicate keys, repair an incorrect time zone or make an unsafe external side effect exactly-once. An initial adaptive plan can differ from its final executed form, so label which one the review uses. Avoid treating โAQE enabledโ as proof that manual workload design is unnecessary. Source layout, row width and severe key concentration can remain important constraints.
Evaluate one evidence-backed change at a time
Compare representative parameters and data distributions at the same measurement boundary. Record source counts and energy invariants before and after a candidate projection, join or partition change. Statistics collection and hints can influence other workloads, so review their scope rather than applying them as universal rituals. A broadcast hint on a growing site dimension can create executor pressure even if it helped yesterday. This course changes no optimizer setting and runs no benchmark. It teaches plan interpretation and safe local model checks only, with all SQL and PySpark fragments marked non-runnable.
Common errors and fixes
An explain plan is reported as a runtime speed test
- Why it happens
- Planning output was confused with execution measurements.
- How to fix it
- Label the plan source and obtain approved runtime evidence for timing claims.
- Prevent it next time
- Keep estimates, observed metrics and client timings separate.
AQE is assumed to repair duplicate-site totals
- Why it happens
- Physical optimization was mistaken for semantic validation.
- How to fix it
- Validate join keys and grain before performance review.
- Prevent it next time
- Retain exact small fixtures alongside plan evidence.
A permanent broadcast hint fails as reference data grows
- Why it happens
- A previous size assumption was treated as a lasting capacity guarantee.
- How to fix it
- Review current bytes, statistics and memory before retaining the hint.
- Prevent it next time
- Re-evaluate size-dependent decisions after source growth.
Check your understanding
1. Can a formatted plan prove a query finished in ten seconds?
Show worked solution
No. It describes execution structure, not measured duration. Capture authorized job and task evidence at a stated observation boundary, including source conditions. Preserve the result invariants so a faster but incomplete query is not accepted as an improvement.
2. What should be reviewed when an adaptive final plan differs from its initial plan?
Show worked solution
Identify the runtime statistics and eligible transformation that explain the change, then verify correctness and observed cost. Label both plans accurately. Do not infer that the SQL meaning changed merely because Spark chose a different join or partition strategy.
Key takeaways
- Plan evidence answers structural questions within a stated scope.
- Adaptive optimization preserves the need for semantic and workload review.