Apache Spark / Advanced: streaming and recoverable operation / Lesson 18

Versions, catalogs and table-format boundaries

Review a release across SQL semantics, connectors, catalogs and stored data before approving compatibility or rollback.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Name Spark 4.0.1 assumptions and distinguish provider guarantees.
Build a compatible migration and recovery review.

A named baseline is more useful than a latest claim

This course uses Apache Spark 4.0.1 as an explicit reference, not a claim that it is the newest supported release in every environment. Java, Scala and Python compatibility must match that release and the deployment’s packaging. Spark 4.0 enables ANSI SQL behavior by default and changes some type conversions and table-provider defaults from earlier releases. Managed platforms can override settings or add features, so record observed configuration alongside documented defaults. An upgrade review must test the semantics the application relies on, including malformed casts, timestamps, decimals and historical data formats.

State a table provider rather than inheriting one accidentally

In Spark 4.0, CREATE TABLE without USING or STORED AS follows the configured default source rather than the former Hive-by-default behavior described in the migration guide. The illustrative definition states USING PARQUET explicitly. It is non-runnable and creates no catalog object. A table name also has catalog and namespace resolution rules; a temporary view is not the same persisted object. Record the intended provider and catalog in migrations so a different session default cannot silently route data elsewhere. A syntactically accepted definition does not itself establish uniqueness, transaction isolation or recovery history.

sql Illustrative technical example; logical results are reasoned, not engine-tested.

CREATE TABLE lab_readings (
 reading_id STRING, site_id STRING, kwh DECIMAL(10,3), reading_day DATE
) USING PARQUET;

Illustrative provider-explicit DDL for an independent lab only. It is not executed and does not imply ACID or primary-key enforcement.

File format and table format solve different problems

Parquet describes columnar files. External table formats and catalog implementations can add metadata, snapshots, concurrent-write protocols and schema or partition evolution, with different supported versions and guarantees. Do not attribute every such feature to plain Spark or assume one provider’s MERGE, time travel or rollback syntax works with another. Connectors are software dependencies whose Scala, Java and Spark compatibility matters. Cloud object-store behavior and commit protocols add another boundary. Select a documented combination and rehearse it with synthetic data; this course neither installs a provider nor operates a metastore, database or cloud service.

A release record must connect code to data meaning

The local manifest records the named baseline and semantic checks rather than a real deployment status. An actual release record should include exact artifacts, source schema versions, runtime settings, connector and catalog compatibility, output publication identity and the checkpoint history for streaming jobs. Preserve accepted data after the cutover through a durable source or explicitly approved recovery loss budget. Rolling back application code alone can misread data written with new semantics. Compare representative historical and boundary fixtures before approving the new version, and keep unresolved compatibility limits visible to operators.

json Illustrative technical example; logical results are reasoned, not engine-tested.

{"scope":"synthetic release review","spark_baseline":"4.0.1","semantic_checks":["ANSI casts","timestamp zones","decimal bounds","provider resolution","stream recovery compatibility"],"publication":"verify source and output versions before cutover"}

A local compatibility checklist, not Spark configuration or evidence that a migration ran.

Compatibility needs proof at each relevant boundary

A complete rehearsal includes the packaged job, worker dependencies, representative queries, rejected-row accounting, published data and a supported restart or restore path. Spark Connect and classic execution also need separate API-coverage assumptions. Check the official migration guides for the exact source and destination releases instead of disabling every new safety behavior. Apache Spark is Apache 2.0 software, while dependencies, connectors and managed-service capabilities can have separate terms and operational responsibilities. No runtime upgrade, catalog migration, checkpoint reset or restore is performed here. The course validates only local fixtures and structural contracts and reports those limitations explicitly.

Common errors and fixes

An upgrade changes table storage unexpectedly

Why it happens: A CREATE TABLE statement inherited a changed default provider.
How to fix it: State the intended provider and inspect catalog resolution.
Prevent it next time: Include provider-default changes in release tests.

A Parquet path is assumed to support provider-specific rollback

Why it happens: File-format storage was confused with an external table protocol.
How to fix it: Verify the actual provider, catalog and supported recovery operations.
Prevent it next time: Document capabilities per data asset, not by file extension alone.

A new binary is approved without data-semantic checks

Why it happens: Runtime startup was treated as end-to-end compatibility.
How to fix it: Rehearse casts, time, schemas, publication and streaming recovery.
Prevent it next time: Keep exact release assumptions and compatibility evidence together.

Check your understanding

1. Why does the teaching DDL include USING PARQUET explicitly?

Show worked solution

It names the intended provider instead of inheriting a configurable or changed default. That improves reproducibility but does not add transaction or key guarantees. Verify the catalog namespace, data location and provider-specific semantics in the separately authorized environment.

2. Can an application-code rollback alone restore prior streaming state?