Apache Spark / Editorial solution guide
Diagnose slow stages, shuffle and memory pressure
Use task distributions and data-grain evidence to distinguish a long tail from a general capacity shortage.
Read-only knowledge base · No public comments
Recognize the symptoms
Most tasks finish quickly while a few remain slow, shuffle spill rises, an executor disappears or downstream tasks report FetchFailedException. A driver can also fail after collect even though executors had spare memory. Preserve the first relevant failure and its process identity, not only the final job message.
Understand the likely causes
- Hot keys, large records, duplicate reference joins and unnecessary wide shuffles can concentrate work. JVM heap, Python workers, caches and container overhead have different budgets. Adding retries or memory without identifying the data and process boundary can repeat or worsen the original problem.
Connect the symptom to the correct execution layer
Use approved application and stage evidence to identify where time or failure begins. Compare task input records, shuffle bytes, spill and duration rather than relying on an average across the cluster. Inspect whether a lost executor preceded a fetch failure. A driver collect failure points to result concentration, while a Python-worker error can involve a different memory process. Redact data paths and identifiers before sharing logs. This guide does not connect to a cluster, collect configured logs or start a diagnostic service.
Rule out multiplied facts before tuning distribution
A join to several site-history rows can inflate energy and output bytes simultaneously. First validate the dimension’s authoritative key and effective-time rule. A semi join can answer presence without multiplying readings, but it does not certify reference uniqueness. Inspect an authorized plan for Exchange, joins and projections after establishing the correct population. The fragment requests plan evidence only; it supplies no benchmark or measured operator output. Optimizing a duplicated relation is not an acceptable performance fix even when the job becomes faster.
enriched.explain(mode='formatted')
summary.explain(mode='cost')Existing synthetic DataFrames are assumed. Planning evidence must be paired with correctness and authorized runtime measurements.
Interpret a skew model as a question, not a diagnosis
The local fixture has work units5,5,5,45. Its total is60 and its largest partition is three times the balanced ideal15. Those units are not measured milliseconds. Investigate whether a hot site, unusually wide rows or join expansion explains the actual tail. coalesce can reduce partitions without redistributing a hot key evenly; repartition by that same hot key also need not balance it. A valid salting strategy requires correct recombination, while adaptive skew handling applies only to supported plans. More executors cannot universally split one indivisible task.
{"work_units":[5,5,5,45],"total":60,"balanced_ideal":15,"largest":45,"scope":"local model, not task telemetry"}Only exact local arithmetic is checked. Actual skew diagnosis needs records, bytes and task evidence.
Choose one narrow correction and preserve recovery
Remove unused wide columns, correct join grain and consider valid pre-aggregation before increasing resources. Review broadcast size in bytes and memory, not only dimension row count. Distinguish cache population from useful reuse, and include Python or native memory in the deployment’s allocation model. Reconcile external sink writes after failed attempts before replaying. Compare the complete representative workload after one justified change, preserving energy and identity invariants. No memory limit, retry count, shuffle setting or allocation is changed by this guide, and no stress or throughput experiment is performed.
Verify the fix
Reconcile keys, counts and energy before and after the candidate change, then compare authorized per-task and end-to-end evidence at the same boundary. Include cache population and recovery cost where relevant. Local models here do not establish any Spark speedup or failure-recovery result.
Keep it from recurring
- Keep distribution-aware fixtures, reference uniqueness checks, bounded driver results and versioned workload evidence. Revisit size-dependent hints after data growth, and protect publication protocols against duplicate task execution.