SQLite / Advanced: concurrency, recovery, and application boundaries / Lesson 14
Backups, restoration, and integrity evidence
Design SQLite recovery as a tested capability using consistent copies, isolated restoration, integrity checks, and explicit application dependencies.
On this page
Lesson goals
- Distinguish a live file from a consistent backup.
- Exercise the online backup API safely.
- Verify a restored copy and its business meaning.
Start with acceptable loss and recovery time
Recovery point objective describes acceptable data-loss exposure, while recovery time objective describes how long service restoration may take. A daily copy can be recent-looking yet expose many hours of work. A quickly restored database may still fail because an application version, encryption extension, external file, or configuration dependency is missing. Inventory the complete supported recovery unit.
Decide who owns backup scheduling, retention, access, verification, and restoration. A copy on the same failing storage does not protect against that failure. SQLite's embedded nature moves these responsibilities into the application or deployment design; it does not remove them. A backup timestamp and a successful copy command are useful observations, not proof that a consistent business service can be recovered.
Use an engine-coordinated copy boundary
The SQLite backup API copies database content through connections, supporting memory and file databases without assuming an ordinary live-file copy is consistent. Python exposes it as Connection.backup. Its source, destination, completion, and error behavior need deliberate ownership. Do not start a backup of an unintended configured path because the API itself is convenient.
The first complete program copies a synthetic memory source, then changes the source. The destination still contains the earlier copied value. This demonstrates an independent completed copy, not crash recovery or a concurrent performance guarantee. Autocommit is explicit so no unfinished source write transaction accidentally makes the demonstration wait indefinitely.
import sqlite3
from contextlib import closing
with closing(sqlite3.connect(":memory:", isolation_level=None)) as source, \
closing(sqlite3.connect(":memory:", isolation_level=None)) as copy:
source.executescript("CREATE TABLE item(value INTEGER); INSERT INTO item VALUES(2);")
source.backup(copy)
source.execute("UPDATE item SET value=99")
print("source", source.execute("SELECT value FROM item").fetchone()[0])
print("copy", copy.execute("SELECT value FROM item").fetchone()[0])
source 99
copy 2
The completed backup retains2 even after the source changes to99. Both databases are private and closed.
Restore into a separate target and check it
The second program uses a new private temporary directory for both source and backup files, then restores into another memory connection. It checks integrity and a known business value. All handles close before temporary storage is removed. No real application file is copied, replaced, renamed, or opened. The actual check output is part of the tested example.
A production drill should also verify schema version, foreign-key consistency, representative queries, application compatibility, and the recovered point in time. Measure retrieval, restoration, validation, and service readiness together. An isolated successful toy restore proves the example's mechanics, not the organization's recovery time or the health of a different backup stored elsewhere.
import sqlite3
from pathlib import Path
from tempfile import TemporaryDirectory
from contextlib import closing
with TemporaryDirectory(prefix="sqlite-backup-training-") as temporary:
folder = Path(temporary)
with closing(sqlite3.connect(folder / "source.db", isolation_level=None)) as source, \
closing(sqlite3.connect(folder / "backup.db", isolation_level=None)) as backup, \
closing(sqlite3.connect(":memory:", isolation_level=None)) as restored:
source.executescript("CREATE TABLE account(id INTEGER PRIMARY KEY,amount INTEGER); INSERT INTO account VALUES(1,42);")
source.backup(backup)
print("backup_check", backup.execute("PRAGMA integrity_check").fetchone()[0])
backup.backup(restored)
print("restored", restored.execute("SELECT amount FROM account WHERE id=1").fetchone()[0])
print("foreign_key_issues", len(restored.execute("PRAGMA foreign_key_check").fetchall()))
backup_check ok
restored 42
foreign_key_issues 0
The backup passes its synthetic integrity check; the restored database contains42 and reports no foreign-key violations.
Do not discard live state while copying files
In WAL mode, committed data can reside in the WAL rather than only in the main database file. Copying the main file alone during active use can therefore create an incomplete or inconsistent recovery asset. Deleting the WAL to βclean upβ is not a backup procedure. Choose the documented backup API, a properly coordinated snapshot, or another approved consistent-copy method.
VACUUM INTO can create a compact copy on supported releases, with its own destination and operational restrictions. It is not identical to an ordinary backup schedule or an arbitrary overwrite command. Storage capacity, filesystem guarantees, destination ownership, and application quiescence requirements must be reviewed. Never add destructive cleanup just to make an example path appear reusable.
Treat corruption evidence as a recovery incident
integrity_check and quick_check inspect structural consistency with different cost and coverage, while foreign_key_check addresses relational violations separately. A result of ok is not proof of every business invariant or of correct application data. If a check reports corruption, preserve evidence, restrict further writes through the incident process, and evaluate known-good recovery before attempting best-effort extraction.
Investigate storage, runtime defects, application misuse, and the exact file/sidecar history. SQLite recovery tools can be useful but do not guarantee complete original semantics or privacy of recovered deleted content. Do not label a salvaged file healthy merely because it opens. A recovery record should include source identity, method, losses or uncertainties, integrity evidence, and application-level reconciliation.
Common errors and fixes
A copied main file misses recent committed records.
- Why it happens
- Live WAL content was omitted from the copy boundary.
- How to fix it
- Use an engine-coordinated consistent backup and test restoration.
- Prevent it next time
- Document journal mode and backup ownership.
A backup API call waits unexpectedly.
- Why it happens
- Source/destination transaction state or contention was not considered.
- How to fix it
- Review explicit transaction completion and documented backup behavior.
- Prevent it next time
- Test the exact host API with isolated data.
An opened recovered file was called fully correct.
- Why it happens
- File readability was mistaken for structural and business validity.
- How to fix it
- Run appropriate checks and reconcile application facts.
- Prevent it next time
- Keep measured isolated restoration drills.
Check your understanding
1. A backup restores quickly but contains yesterday's data. Which objective can still fail?
Show worked solution
The recovery point/data-loss objective can fail despite a good restore duration. Determine how much committed work is protected and test the full capture-to-restoration process. Speed alone does not measure acceptable loss.
2. Does integrity_check returning ok prove there are no orphaned references or incorrect business balances?
Show worked solution
No. Foreign-key checks have their own scope, and business invariants require application-level reconciliation. Combine structural, relational, schema-version, and representative business checks rather than assigning one command every guarantee.
Key takeaways
- A backup is a consistent recovery asset, not just a recent filename.
- Restore drills verify both mechanism and application readiness.
- Structural, relational, and business integrity are complementary checks.