Data Engineering / Basic: contracts and repeatable batches / Lesson 1

Data flows, ownership and contracts

Define what a production flow promises before choosing tools. Separate trustworthy delivery from analysis and connect each dataset to ownership, meaning and failure decisions.

About 4 min readContent reviewed 2026-10-26

On this page

Lesson goals

State source, destination, grain and acceptance rules.
Distinguish successful execution from usable published data.

Engineer a dependable flow rather than only a calculation

Data engineering makes data available with an explicit meaning, delivery boundary and operating responsibility. An analyst may calculate a shipping total; an engineering flow must also explain which events are included, how duplicates and corrections behave, when the result becomes visible and who handles failure. Name the source of truth, consumer, dataset grain, retention and freshness requirement. A completed process can still publish incomplete or incorrectly scoped data, so process success and data acceptance need separate evidence.

Write a small explicit dataset contract

The JSON is an application-defined planning record, not a standard schema language or a configured pipeline. One row means one accepted shipment event, and event_id is the stated identity. The owner must define whether corrections replace facts or create new events. A field list alone is insufficient: units, nullability, clock meaning, access scope and publication rules also belong to the contract. Review a change with affected consumers before treating an additional field or altered meaning as harmless.

json A synthetic dataset responsibility record

{
  "dataset": "synthetic-shipment-events",
  "grain": "one accepted event",
  "identity": ["event_id"],
  "owner": "training-flow-owner",
  "publication": "validated-complete-batch",
  "source": "invented-fixture-only"
}

The record creates no service or storage. It states the identity and publication decision that later examples must implement and verify.

Distinguish extraction, validation and publication

Extraction obtains an identified input; validation checks declared rules; transformation derives a new representation; publication makes an accepted version available to consumers. These may run in one program but remain different logical boundaries. Keep original input identity and rejection reasons so a failed transform can be investigated without guessing which source changed. Staging data is not automatically approved data. A consumer should read a declared completed version rather than partially written intermediate rows merely because a task has started.

Make one acceptance check observable

The offline function accepts only a nonempty batch with unique synthetic event identities. It does not verify every business field, authenticate a producer or implement a production contract registry. An empty batch can be legitimate in another flow, so the nonempty rule is explicitly local. The example demonstrates that expected data properties should be executable assertions where practical. Record counts with their population and input version, and preserve rejected cases instead of silently dropping them to make a job appear green.

python Offline synthetic batch-identity check

def accepted_batch(rows):
    ids = [row['event_id'] for row in rows]
    return bool(rows) and len(ids) == len(set(ids))

assert accepted_batch([{'event_id': 'e1'}, {'event_id': 'e2'}])
assert not accepted_batch([{'event_id': 'e1'}, {'event_id': 'e1'}])
assert not accepted_batch([])

The valid pair passes; duplicate and empty fixtures fail this declared policy. No external data or configured system is accessed.

Specify tool assumptions and owners through change

This course uses Python standard-library programs and explicitly isolated SQLite memory databases to demonstrate correctness properties. Models of distributed orchestration, streaming and storage are labeled as models, not executed platform integrations. Official framework documentation supplies concrete terminology, but adopting a tool does not transfer ownership of data meaning. Keep schema/version, source interval, run identity, acceptance evidence and rollback authority together. Changes in source semantics, consumer requirements or privacy scope should trigger review even when a scheduler still reports success.

Common errors and fixes

A green task publishes unusable data

Why it happens: Execution completion was substituted for dataset acceptance.
How to fix it: Define and enforce the consumer-visible acceptance boundary.
Prevent it next time: Track data checks separately from process status.

The same field changes meaning unnoticed

Why it happens: The contract recorded names but not units or semantics.
How to fix it: Version the meaning and review affected consumers.
Prevent it next time: Include examples and owner approval in contract changes.

Rejected rows disappear without evidence

Why it happens: Filtering concealed input failures.
How to fix it: Preserve scoped rejection reasons and reconcile input/output counts.
Prevent it next time: Make reject handling an explicit delivery outcome.

Check your understanding

1. A flow produces a correct total from only half the required files. Has it met its contract?

Show worked solution

No, if the promised population includes all required files. Arithmetic correctness is only one dimension; establish input completeness, interval and version before publication. The run should retain evidence of which files were expected, received and rejected rather than presenting a plausible total as a complete dataset.

2. Who should decide whether an empty daily batch is acceptable?