Data Engineering / Basic: contracts and repeatable batches / Lesson 3

Schemas, types, nulls and time

Preserve business meaning through explicit field types, units and temporal rules. Distinguish missing, null, zero and invalid values instead of coercing them into apparent success.

About 4 min readContent reviewed 2026-10-26

On this page

Lesson goals

Define field domains and accepted conversion failures.
Represent exact amounts and unambiguous instants.

A schema includes meaning beyond field names

A production field contract describes type, unit, domain, nullability and interpretation. weight could mean kilograms or grams, while updated_at could mean source modification, ingestion or publication time. A string that resembles an integer is not already a validated identifier. Avoid implicit conversions that blur categories such as false and zero or missing and empty text. Version semantic changes even when the physical type remains unchanged. Consumers depend on those meanings to join, aggregate and decide whether a record is fresh enough.

Preserve missing and explicit null as separate inputs

The offline classifier distinguishes an absent field, an explicit null and an actual zero. It does not claim all flows must use three states; the dataset owner decides whether some should be equivalent. A conversion routine must retain enough evidence to implement that decision. Python’s truthiness would collapse several values under if not value, which is often too broad. Test empty text, false and malformed values separately before using a convenience expression in a shared ingestion library.

python Offline field-presence cases

def classify(row):
    if 'qty' not in row:
        return 'missing'
    if row['qty'] is None:
        return 'null'
    return 'present'

assert classify({}) == 'missing'
assert classify({'qty': None}) == 'null'
assert classify({'qty': 0}) == 'present'

The zero is a present value. The function deliberately does not validate whether every present value is an allowed quantity.

Choose exactness and reject policies explicitly

Binary floating point is not a universal money or exact-decimal representation. Use integers in declared smallest units or an appropriate decimal type with an explicit scale and rounding policy. Overflow, excess scale and malformed numeric text need defined outcomes; silently clipping or replacing them can conceal financial errors. A parser accepting a value does not prove it lies in the business domain. Keep source representation alongside a scoped rejection record when needed, while avoiding unnecessary sensitive payload retention in ordinary operational logs.

Normalize an instant without losing source context

The standard-library example shows two offset timestamps referring to the same UTC instant. It uses explicit offsets rather than guessing a timezone for naive local input. Normalization does not retain the original named timezone or civil-time intent automatically, so store those separately when the contract needs them. Day boundaries can differ around daylight-saving changes; an ingestion date and an event-local date are different partition candidates. Do not label all timestamps UTC merely by appending a character to a string that originally represented an unknown clock.

python Offline explicit-offset normalization

from datetime import datetime, timezone

a = datetime.fromisoformat('2026-10-01T10:00:00+00:00')
b = datetime.fromisoformat('2026-10-01T12:00:00+02:00')
assert a == b
assert b.astimezone(timezone.utc).isoformat() == '2026-10-01T10:00:00+00:00'

Both inputs identify the same instant. No external timezone service or real event clock is consulted.

Test schema changes against actual readers

Adding a field, widening a type or changing a default can affect readers differently depending on the serialization and storage system. Avro’s writer/reader schema-resolution rules are one concrete example, not a promise that every format handles evolution identically. Identify old and new readers, required fields and conversion assumptions before rollout. A version marker helps only when producers and consumers honor its meaning. Test representative old, new, incomplete and invalid records, and publish the accepted schema with the data version so consumers do not discover a change through wrong totals.

Common errors and fixes

A missing value is converted to zero

Why it happens: Presence and quantity semantics were collapsed.
How to fix it: Represent the source states and apply the declared policy explicitly.
Prevent it next time: Test missing, null, zero, false and empty text.

A timestamp shifts when a client timezone changes

Why it happens: Naive local time was interpreted without an agreed zone.
How to fix it: Establish source clock/zone meaning and normalize valid instants.
Prevent it next time: Include offset and daylight-saving boundary fixtures.

A physical type stays the same but totals change

Why it happens: Units or defaults changed without a semantic contract revision.
How to fix it: Version and reconcile the meaning change with consumers.
Prevent it next time: Review units and accepted domains alongside schemas.

Check your understanding

1. A source changes weight from kilograms to grams but still sends integers. Is this a harmless compatible type change?

Show worked solution

No. The physical type can remain valid while the meaning changes by a factor of one thousand. Revise the contract, identify affected consumers and convert or version the data deliberately. Checks must include representative values and units rather than only successful parsing and non-null counts.

2. What remains uncertain after converting an offset timestamp to UTC?