Apache Spark / Intermediate: distribution and execution evidence / Lesson 11

Persistence, lineage and retry boundaries

Reuse expensive deterministic work without confusing cache, checkpoint and durable publication.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Choose persistence from actual reuse and memory cost.
Explain recomputation, checkpointing and repeated task effects.

Cache only an intentionally reused result

Suppose the cooperative derives the same accepted-reading population for energy totals and a completeness report. Recomputing its expensive deterministic preparation for each action may be wasteful, so persistence can be useful. A one-use DataFrame may gain nothing and incur storage overhead instead. Project only needed fields and choose the reuse boundary after semantic filters when that matches all consumers. Persistence is an execution optimization, not a requirement that every named variable be cached. Estimate the retained data and competing workloads before deciding that executor memory should hold another intermediate result.

A persistence request and populated storage are different

Calling persist marks a DataFrame for reuse, while an action evaluates data and can populate the selected storage level. The fragment explicitly names MEMORY_AND_DISK as a teaching choice, not a universal best level. count yields a small driver result but still evaluates the persisted population according to the plan. Record which columns and source snapshot are represented. A later consumer can still require additional work, and eviction or executor loss can trigger recomputation. The course does not allocate a cache, run count or measure its benefit.

python Illustrative technical example; logical results are reasoned, not engine-tested.

from pyspark import StorageLevel
prepared = accepted.select('reading_id', 'site_id', 'kwh')
prepared.persist(StorageLevel.MEMORY_AND_DISK)
prepared_count = prepared.count()

Assumes an existing accepted DataFrame. This illustrates a materialization boundary; no Spark storage is populated here.

Release storage when the reuse phase ends

A cache competes with execution memory and other retained data. Holding every intermediate can increase spill, eviction or garbage collection rather than improving time. Reuse the prepared population for the intended consumers, then unpersist at a controlled boundary. The illustrative try/finally expresses cleanup ownership even when a consumer fails. It does not roll back already published output or terminate unrelated application work. Verify total workload cost including cache population and cleanup; timing only the second warm action can overstate the benefit of a persistence design.

python Illustrative technical example; logical results are reasoned, not engine-tested.

try:
    energy = prepared.groupBy('site_id').sum('kwh')
    known = prepared.where('kwh IS NOT NULL').count()
finally:
    prepared.unpersist()

Only known.count is an action here; energy remains a plan until consumed. Cleanup does not constitute durable output rollback.

Recomputation needs a stable source and deterministic logic

Spark can reconstruct lost work from lineage when the necessary inputs and transformations remain available. That does not mean a mutable external file will magically return its old contents, or a random transformation will reproduce the same business value. Reliable checkpointing can truncate a long lineage through supported durable storage, whereas localCheckpoint relies on executor-local storage and has different failure guarantees. Both require an explicit lifecycle and environment assumptions. Neither is identical to publishing a governed dataset with a manifest. Protect source versions and checkpoint locations according to the actual recovery scenario.

A retried task can repeat external effects

Task retries and speculative execution mean user code can run more than once for a logical portion of data. Sending an email, charging a meter account or incrementing an external counter inside a row function is not made exactly-once by caching. Accumulators are not a general transactional accounting system either. Keep transformations pure where possible, and use an explicit idempotent publication or sink protocol for effects. A driver failure also needs application-level recovery ownership. No executor loss, checkpoint write, task retry or external call is exercised in this course; validation is limited to safe local models and syntax.

Common errors and fixes

Caching is reported as a durable source snapshot

Why it happens: Evictable execution storage was confused with governed persistent data.
How to fix it: Version the source and use an explicit durable publication or checkpoint design.
Prevent it next time: State cache and recovery guarantees separately.

Only the warm second action is timed

Why it happens: Population and cleanup costs were omitted from the comparison.
How to fix it: Measure the complete repeated workload under equivalent conditions.
Prevent it next time: Keep input, materialization and reuse boundaries in evidence.

A retried row function sends duplicate notifications

Why it happens: External effects were placed inside retryable distributed computation.
How to fix it: Move effects behind an idempotent sink or coordinated publication protocol.
Prevent it next time: Test duplicate execution and uncertain publication outcomes.

Check your understanding

1. In the cleanup example, has energy already been computed merely because groupBy.sum was assigned?

Show worked solution

No. It remains a plan until an action consumes it. The known count evaluates its own request, and unpersist removes the reuse request at the cleanup boundary. If energy must reuse populated storage, consume it within the intended reuse lifecycle and measure the complete workload.

2. Does a cache prevent a retried task from charging an external account twice?