Data Engineering / Advanced: distributed correctness and reliability / Lesson 13

Warehouses, lakes and snapshot tables

Separate file formats, table metadata, query engines and governance. Explain snapshot publication and concurrent changes without assuming every storage layer supplies the same transaction guarantees.

About 4 min readContent reviewed 2026-10-26

On this page

Lesson goals

Identify the responsibilities of storage, catalog and compute.
Reason about accepted snapshots, readers and maintenance.

A file format is not a complete table system

A warehouse, lake and lakehouse label can describe different combinations of storage, compute, catalog and access governance. Specify the actual components and guarantees instead of treating a product category as a consistency contract. Parquet describes files; a table format can define metadata, snapshots, schema evolution and commit rules above files. A query engine interprets that format with its supported version and features. Two engines reading the same path are not automatically compatible if one ignores delete metadata, schema IDs or a newer table feature.

A reader binds to one accepted metadata version

The offline model gives each synthetic snapshot an immutable tuple of file identities. A reader that selected snapshot one keeps its declared file set even after the current pointer advances to snapshot two. This illustrates a consistency goal, not an Iceberg implementation or object-store transaction. Real table formats require supported atomic catalog/metadata commit mechanisms and writer conflict validation. A path listing performed at different moments can instead mix versions. Bind lineage and quality results to the exact snapshot read, not only a table name whose current state changes.

python Offline accepted-snapshot reader model

snapshots = {1: ('file-a',), 2: ('file-a', 'file-b')}
current = 1
reader_snapshot = current
reader_files = snapshots[reader_snapshot]
current = 2
assert reader_files == ('file-a',)
assert snapshots[current] == ('file-a', 'file-b')

The existing reader remains bound to snapshot one. No catalog, object store or table-format engine is created.

Retry a commit only after validating its assumptions

Optimistic table commits can detect another writer’s update and retry under documented validation rules. That does not mean any stale overwrite may safely be replayed. An append, row-level delete, partition replacement and schema change have different conflict assumptions. Follow the chosen format and engine documentation for isolation and operation support. The metadata commit can be atomic while an external notification remains outside its boundary. Describe which state readers may observe and how a failed writer identifies whether its commit was accepted before resubmitting non-idempotent work.

Keep file set and schema identities together

This application-defined record associates an output snapshot with schema, partition interpretation and its source lineage. It is deliberately not valid Iceberg metadata or a cloud configuration. Real format specifications have exact fields, types and version requirements, so do not invent deployable payloads from this teaching shape. The point is that reading files without the corresponding schema and delete/partition metadata can produce wrong logical rows. Retain enough metadata history for reproducibility and rollback, and verify the selected readers understand the features written by the current producer.

json Nonexecuting table-publication relationship record

{
  "snapshot": "lab-output-v2",
  "schema": "shipment-schema-v1",
  "partitionMeaning": "UTC-event-day",
  "files": ["lab-file-a", "lab-file-b"],
  "lineage": ["lab-source-v1"],
  "format": "teaching-record-not-a-table-spec"
}

The record expresses related identities only; it implements no table-format compatibility or atomic commit.

Historical snapshots and cleanup have consumers

Snapshot expiration, orphan-file cleanup and compaction can remove data needed by older readers, investigations or recovery if retention assumptions are wrong. Use supported maintenance procedures with an explicit horizon and validated references. A table’s transaction history is not an independent backup against every failure or unauthorized change. Review permissions on catalog, metadata and data separately, along with engine support and cost. Choose the architecture from accepted consistency, evolution, recovery and workload requirements rather than assuming a fashionable storage label automatically solves all of them.

Common errors and fixes

Readable files yield wrong logical rows

Why it happens: The reader ignored required schema/delete/table metadata.
How to fix it: Use a compatible reader and accepted snapshot contract.
Prevent it next time: Test cross-engine feature compatibility explicitly.

A stale overwrite is blindly retried

Why it happens: Optimistic retry was assumed safe without conflict validation.
How to fix it: Revalidate the operation against current metadata and intended scope.
Prevent it next time: Separate append, replacement and delete conflict policies.

Cleanup breaks a retained reader or rollback

Why it happens: File/snapshot retention ignored active references and recovery needs.
How to fix it: Follow the supported maintenance and reference-validation process.
Prevent it next time: Document the minimum retained history and owner.

Check your understanding

1. Does an atomic table commit make an external status email part of the same transaction?

Show worked solution

No. The table commit coordinates its supported data/metadata boundary, while the external effect needs its own idempotent or outbox/reconciliation design. Publish a stable output identity and use it to avoid duplicate notifications. State each system’s guarantee rather than extending one atomic operation to unrelated services.

2. Why is listing every current file weaker than reading an accepted table snapshot?