dbt / Advanced: evidence, isolation and recoverable releases / Lesson 17
Orchestration, hooks and partial failures
Coordinate build outcomes and publication without assuming a project is one transaction or every retry is harmless.
On this page
Lesson goals
- Distinguish graph scheduling, test blocking and committed target state.
- Review hooks and replay through explicit side-effect contracts.
A build follows dependencies but is not one global commit
dbt build can process selected supported models, tests, seeds, snapshots and other resources in DAG order. A failing error-severity test can block relevant downstream work under documented selection and dependency rules. That does not mean a previously built relation was rolled back, or every independent branch stopped before creating output. A library release needs a publication policy separate from individual node success. Decide whether consumers stay on a validated prior version until all required checks pass, and preserve which resources were built, failed or skipped during an interrupted invocation.
Retain a partial outcome rather than flattening it to failure
The local outcome model has a successful fact build, a failed key assertion and a skipped downstream report. It is not a native run_results schema, but it illustrates why “the command failed” is insufficient recovery evidence. The fact relation may now contain new data while the report still reflects an older publication. Identify the intended operation and target state before rerunning. A retry that rebuilds a deterministic table differs from one that repeats an additive hook or appends unkeyed rows. Keep the previous validated publication and source checkpoint visible in the recovery record.
{"scope":"local partial-build model","outcomes":{"fct_loans":"built","loan_id_unique":"failed","daily_branch_activity":"skipped"},"publication":"retain previous validated report until reconciliation"}A synthetic recovery model, not captured dbt output or proof that a real relation was committed.
Hooks add SQL and transaction assumptions outside the model query
A pre-hook or post-hook can execute SQL around a resource build. The illustrative audit insert is a real kind of side effect if run, so it is non-runnable and targets a fictional lab relation only. Repeated attempts could create duplicate audit rows unless an operation identity or supported idempotent protocol exists. Hook accumulation and ordering also matter across package, project and resource contexts. Transaction behavior depends on the adapter: helpers intended for transactional targets must not be assumed portable to databases where dbt uses different commit semantics. Review generated SQL and required permissions for every hook.
models:
- name: fct_loans
config:
post_hook:
- "insert into lab_audit.model_runs(model_name) values ('fct_loans')"Illustrative side-effect configuration only. No audit relation is created or written, and this unkeyed insert is not presented as replay-safe.
A lost acknowledgment creates uncertainty at the sink
A database can commit before a client loses its connection or receives a timeout. Do not infer rollback solely from that transport symptom. Reconcile known target identities and operation records through the approved recovery path before repeating an incremental update, seed load or external notification. External orchestration also needs retry budgets and ownership for overlapping scheduled runs. Two jobs writing the same target can conflict even when each DAG is internally correct. A scheduler’s success flag and a data product’s acceptance record are related but different evidence.
Recovery should be rehearsed with exact resource semantics
A separately authorized rehearsal should cover failure after a model build, before a required test, inside a hook and after a database commit acknowledgment is lost. Verify target state, source retention and the publication decision for each case. Snapshot history, incremental tables and ordinary full rebuilds have different replay rules. Do not use a blanket full refresh as the answer to every failure. This course launches no job, writes no audit row and invokes no retry. Local validation checks the outcome JSON and pure replay models only, with no transaction, orchestration or warehouse recovery test claimed.
Common errors and fixes
A failed build is assumed to undo every earlier model
- Why it happens
- DAG scheduling was confused with a project-wide transaction.
- How to fix it
- Inspect actual per-resource outcomes and target publication state.
- Prevent it next time
- Keep recovery and consumer-switch policy separate from command status.
A retry repeats an unkeyed audit insert
- Why it happens
- Hook side effects were not designed for repeated attempts.
- How to fix it
- Use a supported operation-key or idempotent audit protocol.
- Prevent it next time
- Review every hook with failure and retry scenarios.
A connection timeout is treated as proof of no write
- Why it happens
- Transport uncertainty was confused with a known database outcome.
- How to fix it
- Reconcile committed state before replaying effects.
- Prevent it next time
- Persist source and operation identity for recovery.
Check your understanding
1. What can coexist after the local failed-build model?
Show worked solution
The fact may be newly built, its required assertion failed and the report skipped or left at an older state. Determine the actual target and publication versions before retrying. The JSON illustrates possible partial state, not an atomic rollback guarantee.
2. Why is the shown audit hook not replay-safe by itself?
Show worked solution
It inserts a new row without an operation identity or deduplication rule. A repeated attempt can add another row even if the model data is unchanged. Design and verify the audit protocol under the actual adapter transaction behavior rather than assuming hook timing provides exactly-once effects.
Key takeaways
- A graph build can leave meaningful partial data state.
- Hooks and retries require explicit side-effect and recovery contracts.