Apache Spark / Basic: typed relational foundations / Lesson 1
Driver, executors and lazy plans
Locate a DataFrame operation in Spark execution and distinguish constructing a plan from computing its result.
On this page
Lesson goals
- Explain application, driver, executor, partition and task.
- Identify transformations, actions and driver-bound results.
An application coordinates distributed work
A solar cooperative receives meter readings from many sites. Spark can process that collection as one application while distributing work over partitions. The driver coordinates the application and constructs execution plans; executors run tasks and retain relevant working data. A cluster manager allocates resources but does not decide the business meaning of a kilowatt-hour. The application can be local for independent learning or deployed through supported cluster environments. Neither configuration turns Spark into the authoritative meter database automatically. Define source completeness, reading identity and output publication separately from the execution architecture.
A DataFrame describes a computation
A DataFrame represents rows with named typed columns and a logical plan. Selecting columns or filtering normally builds another plan without immediately scanning all input. This laziness lets Spark optimize a chain before executing it. The fragment assumes an already approved SparkSession named spark, and creates a synthetic relation from local values. It is not a standalone program and is marked non-runnable. No SparkSession, Java process or cluster is started by the course. Naming a variable filtered does not prove that its rows have been evaluated or that source errors would already have surfaced.
readings = spark.createDataFrame(
[('north', 2), ('south', 4), ('north', 3)],
'site STRING, kwh INT'
)
filtered = readings.where('kwh >= 3').select('site', 'kwh')The logical filtered population contains south=4 and north=3. Transformations construct a plan; no engine execution is claimed.
An action creates an execution boundary
An action such as count, collect or a write asks Spark to produce a result. Spark can divide work into jobs, stages and tasks according to the plan and its shuffle boundaries; one source-code line does not correspond reliably to exactly one stage. Repeating actions can recompute work unless appropriate persistence or other reuse applies. count returns a small scalar to the driver, while collect returns every selected row and can exhaust driver memory. A dataset fitting across executors need not fit in the driver. Use bounded diagnostic results and distributed output for large data.
filtered.explain(mode='formatted')
# In an independent authorized lab only:
result_count = filtered.count()2explain describes a plan; count would return 2 for this fixed fixture. Neither call is executed during authoring.
A partition is a unit of work rather than a business key
A partition holds a portion of the distributed data and is commonly processed by one task at a stage. The same site can appear in several input partitions, and a physical file is not universally identical to one Spark partition. Grouping or joining by site can require redistributing rows so related keys meet. This movement is a shuffle and has network, serialization and storage consequences. An application with many tiny partitions can spend time scheduling overhead, while one oversized partition can dominate completion. Choose partitioning from observed workload rather than a universal number of rows per task.
Name the runtime boundary precisely
The course uses Apache Spark 4.0.1 documentation and PySpark or Spark SQL fragments as a named teaching baseline. Official release documentation lists Java 17 or 21, Scala 2.13 and Python 3.9 or later for this release; these are not installation instructions or claims about every managed platform. Spark Connect separates client and server differently from classic PySpark, so low-level RDD or SparkContext assumptions do not transfer automatically. No dependencies are installed here. Expected outputs are logical fixture results, and safe Python-only models used in validation are explicitly separate from actual Spark execution.
Common errors and fixes
A transformation is assumed to have validated all source data
- Why it happens
- Lazy plan construction was confused with evaluating an action.
- How to fix it
- Identify the execution boundary and validate the intended input during an authorized run.
- Prevent it next time
- Document which checks require source evaluation.
collect exhausts driver memory
- Why it happens
- The complete distributed result was moved into one process.
- How to fix it
- Use bounded inspection or distributed publication.
- Prevent it next time
- Budget driver results independently from executor storage.
A site is assumed to occupy one input partition
- Why it happens
- Business identity was confused with physical input layout.
- How to fix it
- Inspect the required key distribution and plan.
- Prevent it next time
- Keep logical grain separate from physical partitioning.
Check your understanding
1. Why does assigning filtered not prove that malformed source rows have been read?
Show worked solution
The assignment normally constructs a logical plan. Evaluation happens when an action needs the data, and some optimizations can avoid reading unused fields or rows. Put validation at a deliberate evaluated boundary and record its scope. Do not claim source quality from the existence of a DataFrame variable.
2. What does count return for the three-reading fixture after kwh >= 3?
Show worked solution
Two rows qualify: south with 4 and north with 3. This local reasoning establishes the expected count, not the number of executors, tasks or stages Spark will choose. Those physical details require the actual plan and authorized runtime evidence.
Key takeaways
- A logical plan and an executed result are different artifacts.
- Driver memory and distributed capacity are separate budgets.