Generative AI / Advanced: evaluation, reliability, and accountability / Lesson 13
Dataset lineage and governance
Record source, scope, rights, versions, and split boundaries so development and evaluation remain interpretable and responsibly maintained.
On this page
Lesson goals
- Explain the assumptions and trust boundaries of dataset lineage and governance.
- Interpret both examples and distinguish offline fixture results from model or production evidence.
- Diagnose the listed failures and justify the worked decisions while preserving task, data, permission, and uncertainty contracts.
Treat every corpus as an owned asset
A corpus used for prompting, retrieval, training, or evaluation needs a defined purpose, owner, source history, access scope, and retention policy. Public availability does not by itself establish suitability, permission for every use, or freedom from sensitive information. Record the review appropriate to the actual data and jurisdiction rather than making blanket legal or privacy claims. This course uses only authored public synthetic fixtures and makes no real dataset-rights assessment. The second example is a fictional documentation record, not a license, compliance certificate, or completed governance approval. Its fields make unresolved assumptions visible rather than hiding them behind a dataset name.
dataset_revision: synthetic-policy-fixtures-v3
intended_use: offline-grounding-and-access-tests
source: course-authored-public-synthetic-records
coverage_limits: explicitly-reviewed-per-task
lineage: source-transform-label-and-index-revisions
correction_owner: designated-dataset-owner
status: fictional-record-no-rights-certification
The record describes what should be documented and claims no license, real corpus audit, model training, or certification. It is not a platform configuration.
Preserve lineage through transformations
Cleaning, redaction, chunking, summarization, labeling, embedding, and export create derived artifacts. Keep links to the source version, transformation code or policy, schema, and relevant access restrictions. Deleting or correcting a source may require updates to indexes, caches, evaluations, and derived records. A content hash can detect changed bytes but does not prove source authority or usage permission. Record enough safe metadata to reproduce the pipeline without copying sensitive material into every diagnostic artifact. A transformation that improves formatting can still remove an exception, distort a label, or expose information that the original access policy protected.
Split by leakage risk, not convenience alone
Random rows can put near duplicates, the same source document, or related user records in both development and evaluation. Group by the unit relevant to leakage and define time boundaries where future information would be unavailable in use. The first offline example keeps two fragments from source group a together and holds group b apart. It demonstrates one grouping rule and does not prove there is no semantic overlap or training contamination. Keep held-out cases stable and protected from repeated prompt tuning. If a fixture is exposed during debugging, record that change rather than continuing to call the comparison untouched.
rows = [{"id": "a1", "source_group": "a"}, {"id": "a2", "source_group": "a"}, {"id": "b1", "source_group": "b"}]
development = [row for row in rows if row["source_group"] == "a"]
held_out = [row for row in rows if row["source_group"] == "b"]
print([row["id"] for row in development])
print([row["id"] for row in held_out])
print({row["source_group"] for row in development}.isdisjoint({row["source_group"] for row in held_out}))
['a1', 'a2']
['b1']
TrueThe two a fragments stay together and b is held out. This fixture checks the stated grouping rule only, not all duplicate, semantic, or pretraining leakage paths.
Review labels, coverage, and intended use
Labels reflect a rubric, evidence, and reviewer decisions and can contain uncertainty or disagreement. Document who or what produced them, the adjudication process, known omissions, language coverage, and limitations. A dataset suitable for one narrow extraction task may be unsuitable for open-ended assistance or another population. Keep negative, unanswerable, and policy-boundary cases alongside ordinary successes. Do not use aggregate dataset size as a substitute for representative coverage. Model and dataset cards can structure disclosure, but a completed card is evidence about documentation, not automatic proof of safety, fairness, factuality, or legal compliance in every deployment.
Maintain correction and retirement paths
An owned process should accept corrections, reassess access, retire invalid fixtures, and propagate changes to derived artifacts and release evidence. Preserve necessary audit history under the appropriate retention and privacy policy without exposing removed data. Evaluate whether a changed corpus requires a new baseline or model and index migration. No real personal record, external corpus, model training set, database, or licensing service is accessed here. The examples are tiny local records. Governance is useful when it makes a result traceable and a correction actionable, not when it produces a static form that nobody updates after the data changes.
Common errors and fixes
Related fragments leak across evaluation splits
- Why it happens
- Rows were split without source or user grouping.
- How to fix it
- Split by the relevant leakage unit and review time scope.
- Prevent it next time
- Audit duplicates and exposure history.
A corrected source remains in derived indexes
- Why it happens
- Lineage and update propagation were not owned.
- How to fix it
- Trace and rebuild or remove affected derivatives safely.
- Prevent it next time
- Maintain correction and deletion workflows.
A dataset card is treated as universal approval
- Why it happens
- Documentation was confused with verified suitability.
- How to fix it
- Review the actual task, users, rights, and controls.
- Prevent it next time
- Version limitations and review evidence.
Check your understanding
1. Two chunks from one policy are randomly assigned to development and evaluation. What risk exists?
Show worked solution
The system can benefit from near-duplicate content seen during prompt design, making held-out performance look stronger than genuine generalization. Group by source and evaluate overlap more broadly where needed; grouping alone is not a proof against every contamination path.
2. A source must be withdrawn. Which artifacts may need review?
Show worked solution
Review chunks, embeddings, retrieval indexes, caches, summaries, labels, evaluation sets, logs, and release evidence according to lineage and policy. Removing the original file alone may leave derived sensitive or invalid content available.
Key takeaways
- A corpus used for prompting, retrieval, training, or evaluation needs a defined purpose, owner, source history, access scope, and retention policy.
- Random rows can put near duplicates, the same source document, or related user records in both development and evaluation.
- An owned process should accept corrections, reassess access, retire invalid fixtures, and propagate changes to derived artifacts and release evidence.