Apache Spark / Basic: typed relational foundations / Lesson 2

Schemas, types and source boundaries

Read meter data through an explicit typed contract and preserve rejected values for diagnosis.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Choose schemas from measurement meaning.
Distinguish parsing, nullability metadata and business validation.

A schema states interpretation rather than column spelling

A meter identifier with leading zeros is text, energy may require bounded decimal arithmetic, and an observation time needs a declared temporal meaning. Inferring all of these from a sample can produce a different schema when another file arrives. Explicit schemas make the producer contract reviewable and can avoid repeated inference work. A schema still does not prove that energy is nonnegative, that identifiers are unique or that every expected site reported. Keep syntactic parsing, type conversion and domain validation as separate evidence. A file that parses successfully can remain incomplete or semantically wrong.

Declare the teaching relation deliberately

The SQL fixture uses DECIMAL(10,3) for exact bounded energy values and STRING for identity. The decimal literal retains scale intent without relying on binary floating point. A null measurement is deliberately present so later rules cannot pretend every reading is complete. This self-contained SQL relation is illustrative and not executed by Spark here. A schema declaration is metadata for interpretation; nullable flags on a DataFrame are not a substitute for persisted business constraints. Validate required fields explicitly before publishing a curated result.

sql Illustrative technical example; logical results are reasoned, not engine-tested.

SELECT * FROM VALUES
 ('0007', CAST(1.250 AS DECIMAL(10,3))),
 ('0008', CAST(NULL AS DECIMAL(10,3)))
AS readings(meter_id, kwh);

Expected output

meter_id | kwh
0007     | 1.250
0008     | NULL

Two logical rows with string identity and decimal energy; the second measurement remains missing.

File parsing options belong to the contract

A CSV reader needs decisions about header, delimiter, quote handling, encoding, timestamp format and malformed records. The fragment assumes a separately approved session and a fictional lab path. It does not access a file during authoring. FAILFAST can surface malformed input according to the reader’s rules, but business checks still need explicit evaluation. A permissive mode can retain or null problematic content depending on options and schema; if used, preserve a corrupt-record strategy rather than silently accepting losses. A JSON parser’s success is different from Spark accepting the declared column types.

python Illustrative technical example; logical results are reasoned, not engine-tested.

raw = (spark.read
    .option('header', True)
    .option('mode', 'FAILFAST')
    .schema('meter_id STRING, kwh DECIMAL(10,3)')
    .csv('lab-input/readings.csv'))

Illustrative read plan only. The path is a synthetic lab location and no file or Spark reader is opened.

Schema evolution needs an explicit compatibility rule

A new optional source column, a renamed field and a changed numeric scale have different consequences. Selecting by position from a changed CSV can misinterpret values even when column counts match. Merge-schema options for supported formats are not a general semantic migration engine and can add discovery cost. Keep the source schema version and map it to a curated schema deliberately. Quarantine incompatible versions with provenance. When using external catalogs or table formats, distinguish their evolution and transaction guarantees from plain file reads; Spark alone does not impose one universal schema registry on every source.

Preserve a rejection ledger as well as clean rows

A useful ingestion report reconciles source rows, accepted rows, rejected rows and intentionally excluded rows. Attach a source file or batch identity and a stable reading identifier where available. A row rejected for invalid energy should not be indistinguishable from a legitimate missing observation. Avoid collecting the entire rejection set to the driver; publish bounded diagnostics and an approved protected repair dataset. Treat source paths and meter identifiers according to the data policy. This course checks only synthetic type examples and source syntax, and claims no file-reader, decimal-engine or distributed parsing test.

Common errors and fixes

A meter code loses leading zeros

Why it happens: Schema inference or casting treated an identifier as a number.
How to fix it: Retain identity as STRING and repair from authoritative source.
Prevent it next time: Publish explicit schemas and identifier examples.

Nullable metadata is treated as a completeness guarantee

Why it happens: Interpretation metadata was mistaken for a business validation rule.
How to fix it: Evaluate explicit required-field checks before publication.
Prevent it next time: Reconcile accepted and rejected rows with source counts.

A schema merge silently changes meaning

Why it happens: Structural compatibility was assumed to establish equivalent units or scale.
How to fix it: Map source versions and units explicitly.
Prevent it next time: Review semantic changes independently from reader options.

Check your understanding

1. Does parsing kwh as DECIMAL prove the value is nonnegative?

Show worked solution

No. The type establishes representation and bounded precision. Add a domain rule for valid energy and decide whether zero, missing values and reverse-flow readings are meaningful in this dataset. Preserve rejected source values and reasons instead of converting every failure to zero.

2. Why should a file-schema change carry a version?