DevOps / Advanced: reliability, trust, and governance / Lesson 13
Reliable rollouts and data evolution
Coordinate application, schema, message, and configuration changes so recovery remains possible during mixed-version operation.
On this page
Lesson goals
- Model mixed-version compatibility across release phases.
- Plan bounded migrations and delayed contraction.
- Choose rollback or forward repair according to actual state.
Model the release as a protocol
A reliable release is a sequence of compatible state transitions, not one atomic command across every component. Application instances, database schema, messages, caches, and configuration often change at different times. Identify which combinations can exist and which component owns each transition. The first synthetic compatibility matrix makes overlap explicit. A clean deployment job cannot prove those combinations are safe. Review writes as carefully as reads because newly stored data can prevent old code from recovering. State the point after which rollback is no longer the preferred action and forward repair becomes necessary.
combination allowed
old app + expanded schema yes, after compatibility tests
new app + expanded schema yes, after compatibility tests
old app + contracted schema no
new app + unverified backfill no promotion
The matrix states a particular teaching contract, not universal database behavior. The forbidden old-app combination explains why contraction must wait.
Expand before contracting
An expand-and-contract change introduces a compatible structure, moves readers and writers through a reviewed transition, verifies adoption, and only later removes the old structure. Adding a nullable field may be compatible while renaming a required field immediately is not. A backfill needs bounded batches, resumability, progress evidence, and a policy for concurrent writes. Do not assume a dual-write implementation is consistent merely because both calls usually succeed; partial failure requires reconciliation or transactional coordination. Delay irreversible cleanup until old instances, consumers, and recovery requirements no longer depend on the previous representation.
Coordinate ordering and ownership
The second record describes a staged migration rather than executable database commands. It puts compatibility evidence before wider exposure and cleanup. Name one migration owner, serialize conflicting operations, and verify the selected environment and schema revision. Some database DDL can lock or rewrite large structures depending on engine and version, so confirm exact semantics and test representative data in an authorized isolated environment. A timeout does not establish whether a migration committed. Preserve execution evidence and inspect actual state before retrying a potentially nonidempotent operation.
phases:
- expand-compatible-schema
- deploy-compatible-readers-and-writers
- backfill-bounded-resumable-batches
- verify-invariants-and-consumer-adoption
- retire-old-contract-after-recovery-review
abort_on: integrity-or-compatibility-failure
Each phase requires its own evidence and owner. This record issues no SQL, migration, deployment, or data mutation.
Handle requests across the transition
A terminating instance should stop admission and drain accepted work within the platformโs grace period. Messages may be redelivered, clients may retry, and connections may persist across a route switch. Design idempotency and schema-version handling explicitly rather than expecting the orchestrator to infer them. Preserve correlation between the release and changed failure rates, and pause expansion when data integrity or important cohorts degrade. If a rollback is attempted, validate the previous artifact against current configuration and data. Moving traffic back does not undo already published messages or remote transactions.
Prove completion and retain recovery evidence
Completion requires evidence that every relevant reader and writer has moved, the backfill met its invariants, and no stale consumer still needs the old contract. Observe a suitable period before destructive cleanup and keep a verified recovery path. Update runbooks and compatibility records with the final state. Distinguish a successful experiment with tiny synthetic data from proof about production scale or lock duration. Release reliability comes from knowing which transitions are safe, measuring their effect, and stopping when assumptions fail. It cannot be reduced to a generic rollback instruction pasted into every pipeline.
Common errors and fixes
A rollback cannot read new data
- Why it happens
- Writes changed incompatibly during exposure.
- How to fix it
- Use the reviewed reconciliation or forward-repair plan.
- Prevent it next time
- Test previous readers against transition data.
Backfill competes with live writes
- Why it happens
- Batching, ordering, and concurrency semantics were undefined.
- How to fix it
- Pause according to policy and reconcile using explicit invariants.
- Prevent it next time
- Design resumability and concurrent-write handling before execution.
A timed-out migration is run twice
- Why it happens
- Timeout was assumed to mean no commit.
- How to fix it
- Inspect actual schema and execution evidence before retry.
- Prevent it next time
- Make migration ownership and completion records explicit.
Check your understanding
1. A new field replaces an old required field. Outline a compatible order.
Show worked solution
Add compatible storage, release code that tolerates both forms, migrate data with bounded resumable work, verify adoption and invariants, then remove old usage and finally old storage after recovery review. Exact dual-read or write rules depend on the application and database guarantees.
2. Can blue-green application capacity make an incompatible shared schema safe?
Show worked solution
No. Both environments can still read and write the same schema, and old code may return during recovery. Separate capacity does not provide data compatibility. Review shared state and irreversible effects independently before deciding whether a traffic switch is reversible.
Key takeaways
- A reliable release is a sequence of compatible state transitions, not one atomic command across every component.
- The second record describes a staged migration rather than executable database commands.
- Completion requires evidence that every relevant reader and writer has moved, the backfill met its invariants, and no stale consumer still needs the old contract.