Data Engineering / Advanced: distributed correctness and reliability / Lesson 16

Security, privacy and access boundaries

Protect every data copy and processing identity without treating encryption or a private network as sufficient authorization. Keep diagnostic evidence useful and minimally exposed.

About 4 min readContent reviewed 2026-10-26

On this page

Lesson goals

Separate authentication, authorization, encryption and minimization.
Apply scoped access and safe operational evidence.

Every staging and recovery copy remains an asset

A pipeline can copy sensitive data into raw zones, staging tables, rejects, logs, exports, checkpoints and backups. Protecting only the final warehouse leaves those copies exposed. Inventory purpose, owner, allowed consumers and retention for each. Processing identities need only their required source, sink and metadata capabilities; a broad administrator credential is not justified by convenience. Authentication proves a subject under its mechanism, while authorization decides permitted actions. A private network and encrypted connection do not replace either data scope or correct business permission checks.

Project only the declared consumer fields

The offline function constructs a public training output from an explicit field allowlist, excluding an internal contact value. This is data minimization, not a complete anonymization or access-control system. A real flow must also classify whether the remaining fields or combinations can identify people, enforce the allowed consumer population and protect intermediate copies. Do not rely on deleting one obvious field after it has already been logged or loaded into a broad staging area. Review both the published representation and the earlier processing path.

python Offline explicit-field minimization

source = {'shipment_id': 's1', 'status': 'ready', 'internal_contact': 'synthetic-only'}
allowed_fields = ('shipment_id', 'status')
output = {field: source[field] for field in allowed_fields}
assert output == {'shipment_id': 's1', 'status': 'ready'}
assert 'internal_contact' not in output

One field is excluded from this output. The example provides no anonymization guarantee and uses no real identity or secret.

Protect transport and storage without inventing cryptography

Use supported encryption and key-management mechanisms appropriate to the platform, with recoverable approved key access where required. Encryption protects only under its threat and access assumptions; it does not make unnecessary collection harmless or authorize every key holder. Keep credentials out of code, manifests, task arguments and broad logs. Rotation, identity changes and restore workflows must preserve required access without retaining uncontrolled old credentials. A hash of personal data can remain linkable and sensitive, so do not call pseudonymization anonymous merely because clear text was replaced.

Test a narrow scope decision with synthetic subjects

The local policy permits one declared organization to read its own training records and rejects a different organization or a write request. It does not authenticate subjects or install database permissions. A real service should obtain organization identity from trusted server-side context, not an arbitrary request label. Pipeline service accounts also need separate privilege review for raw data, quality outputs and catalogs. Negative checks should be performed only on approved fixtures or known authorized test scopes; this course takes no account, cloud or production security action.

python Offline scope/operation policy

def scope_model(subject_org, dataset_org, action):
    return subject_org == dataset_org and action == 'read'

assert scope_model('lab-a', 'lab-a', 'read')
assert not scope_model('lab-a', 'lab-b', 'read')
assert not scope_model('lab-a', 'lab-a', 'write')

Only three supplied comparisons are verified. Real IAM, credentials and platform authorization remain outside this model.

Keep enough evidence without creating another leak

Operational records need run identity, affected dataset, error class and scoped reason more often than an entire private payload. Define who can access reject samples and lineage, their retention and how deletion obligations propagate through copies. A pipeline that removes a subject from the serving table may still retain it in raw files or backups under documented rules. Security and privacy requirements can constrain backfill and recovery, so include them in the data contract and change review. Evidence quality and minimum necessary exposure should be designed together rather than trading one away silently.

Common errors and fixes

A raw/reject copy exposes private data

Why it happens: Access and retention review covered only the final output.
How to fix it: Inventory and scope every meaningful copy.
Prevent it next time: Review raw, staging, logs and backups together.

A hashed identifier is called anonymous

Why it happens: Linkability and reidentification context were ignored.
How to fix it: Classify the transformed data under the actual privacy model.
Prevent it next time: Review combinations and access, not only visible text.

A service account receives administrator access to repair a flow

Why it happens: Convenience replaced required-capability analysis.
How to fix it: Identify the exact source/sink/catalog actions and narrow scope.
Prevent it next time: Test allowed and denied operations on approved fixtures.

Check your understanding

1. Is removing a contact field only at final publication sufficient protection?

Show worked solution

Not if it was unnecessarily exposed in raw access, staging, logs or reject records first. Minimize and restrict the full lifecycle, with appropriate retention and ownership. Final projection is one control, not a remedy for earlier copies already accessible outside the declared purpose.

2. Why should recovery plans include key and identity access?