diff --git a/docs/BACKUP_AND_MIGRATION.md b/docs/BACKUP_AND_MIGRATION.md new file mode 100644 index 0000000..4bc18a9 --- /dev/null +++ b/docs/BACKUP_AND_MIGRATION.md @@ -0,0 +1,223 @@ +# Backup, Restore & Docker Migration + +An in-place backup system that captures everything needed to rebuild the +library somewhere else: database rows, every uploaded PDF and image, covers, +generated thumbnails and all extra files. + +Built as the first half of the move to a Dockerised LIM — take a backup on the +Windows box, restore it into the container, done. + +--- + +## What a backup contains + +``` +lim-backup-YYYYMMDD-HHMMSS[-label].zip +├── manifest.json metadata, row counts, integrity report +├── checksums.sha256 SHA-256 of every payload entry +├── database/ +│ ├── data.json portable row dump of all four tables +│ └── lego_instructions.db raw SQLite snapshot (SQLite installs only) +└── uploads/ + ├── pdfs//… + ├── images//… + ├── covers//… + ├── thumbnails//… + └── extra_files//… +``` + +Two copies of the database on purpose: + +- **`data.json`** is engine-neutral. It restores into SQLite *or* Postgres and + it is what makes the Docker move possible. All path columns are rewritten to + forward slashes on the way out, so a Windows-made backup lands correctly on + Linux. +- **`lego_instructions.db`** is a consistent snapshot taken through SQLite's + own online-backup API (safe to run while the app is serving requests). It's + the belt-and-braces copy if you'd rather just drop the file in place. + +Uploads are walked from disk rather than from the database, so files that lost +their record still come across. PDFs and images are stored uncompressed — they +don't deflate, and skipping the attempt makes large collections much faster. + +### Integrity report + +Every backup records two lists in the manifest: + +| Field | Meaning | +|---|---| +| `missing_files` | Referenced by a database record but absent from disk. These records will survive the migration with a broken link. | +| `orphaned_files` | Present on disk with no record pointing at them. Backed up anyway. | + +Worth clearing `missing_files` before you cut over to Docker — it's the one +thing a restore cannot fix for you. + +--- + +## Using it from the admin UI + +**Admin → Backup & Restore** (`/admin/backup/`). + +- Shows current upload size, file count, and both integrity lists up front. +- **Start backup** runs in a background thread with a live progress bar, so a + multi-gigabyte collection won't time out the request. +- Each stored archive can be downloaded, checksum-verified, restored or deleted. +- `BACKUP_KEEP` (default 5) prunes older archives after each UI-initiated + backup. Set it to `0` to keep everything. + +Restores from the UI require typing `RESTORE` to confirm, and a destructive +wipe needs a second checkbox on top of that. + +--- + +## Using it from the CLI + +The CLI is the better path on a fresh instance, because it works before any +admin user exists. + +```bash +flask backup create [--label pre-docker] [--keep 5] [--no-sqlite-file] +flask backup list +flask backup inspect lim-backup-....zip # print the manifest +flask backup verify lim-backup-....zip # recompute every checksum +flask backup restore lim-backup-....zip --mode fresh [--force] + [--overwrite-files] [--skip-files] [--yes] +``` + +Archive arguments accept either a full path or a bare filename that exists in +the backup folder. + +### Restore modes + +| Mode | Behaviour | +|---|---| +| `fresh` | For an empty target. Primary keys are preserved exactly, so every existing URL and bookmark still resolves. Refuses to run against a non-empty database unless you pass `--force` (which wipes the four tables first). Postgres sequences are realigned afterwards. | +| `merge` | For folding a backup into a live instance. Users are matched by username (then email); sets by set number. A set that already exists is skipped whole — its instructions and extra files are *not* merged in, to avoid duplicating files. Clashing `brickset_id` values are cleared rather than failing the insert. | + +Upload files are never overwritten unless you pass `--overwrite-files`. +Password hashes come across intact, so everyone's existing logins keep working. + +--- + +## Migrating to Docker + +### 1. On the current Windows install + +```powershell +cd E:\LIM +flask backup create --label pre-docker +flask backup verify lim-backup-20260924-....zip +``` + +Check the summary. If it reports missing files, fix or delete those records +first, then take the backup again. + +### 2. Get the archive to the new host + +Copy the `.zip` into whatever directory you'll mount as the container's backup +folder, e.g. `/srv/lim/backups/`. + +### 3. Container volume layout + +Three things must live on volumes, not in the image: + +| Path in container | Holds | Env var | +|---|---|---| +| `/data/uploads` | all uploaded files | `UPLOAD_FOLDER` | +| `/data/backups` | backup archives | `BACKUP_FOLDER` | +| `/data/db` | SQLite file (if not using Postgres) | via `DATABASE_URL` | + +> **Note:** `UPLOAD_FOLDER` is currently hard-coded in `app/config.py` to +> `app/static/uploads`. When the Dockerfile lands, either make it read the +> `UPLOAD_FOLDER` environment variable or bind-mount the volume over +> `/app/app/static/uploads`. The backup format doesn't care which you pick — +> paths in the database are relative to whatever `UPLOAD_FOLDER` resolves to. + +Example `.env` for the container: + +```env +FLASK_APP=run.py +FLASK_ENV=production +SECRET_KEY= +DATABASE_URL=sqlite:////data/db/lego_instructions.db +BACKUP_FOLDER=/data/backups +BACKUP_KEEP=7 +``` + +### 4. Restore into the container + +```bash +docker compose exec lim flask backup verify /data/backups/lim-backup-....zip +docker compose exec lim flask backup restore /data/backups/lim-backup-....zip \ + --mode fresh --yes +``` + +Expected output: row counts matching the source, and a file count matching the +manifest's `uploads.file_count`. + +### 5. Verify + +- Log in with an existing account (hashes migrated, so the old password works). +- Open a set with a PDF, an image sequence, a cover image and an extra file. +- Spot-check that thumbnails render — they're backed up rather than + regenerated, so a missing one means the source was missing it too. + +### 6. Keep the old instance until you're happy + +Don't delete `E:\LIM` until the container has been in daily use for a while. +The backup is reversible in both directions: take one from the container and +restore it back onto Windows the same way. + +--- + +## Going to Postgres at the same time + +`data.json` restores into any SQLAlchemy-supported engine. Point +`DATABASE_URL` at Postgres, run `flask init-db` (or the migrations) to create +the schema, then `flask backup restore ... --mode fresh`. Sequences are reset +automatically after the insert, so new rows won't collide with the preserved +IDs. + +Keep the migration one change at a time if you can: container first on SQLite, +Postgres later. + +--- + +## Scheduling + +**Windows Task Scheduler** — daily at 2am: + +```powershell +cd /d E:\LIM && venv\Scripts\python.exe -m flask backup create --label nightly --keep 7 +``` + +**Docker host** — cron: + +```cron +0 2 * * * docker compose -f /srv/lim/docker-compose.yml exec -T lim flask backup create --label nightly --keep 7 +``` + +Then sync `/data/backups` off the host with your usual job. A backup sitting on +the same disk as the thing it's backing up is only half a backup. + +--- + +## Troubleshooting + +**"Target database is not empty"** — you're doing a `fresh` restore into a +populated database. Use `--mode merge`, or `--force` to wipe first. + +**Files restored but images don't load** — `UPLOAD_FOLDER` on the target +doesn't point where you think. Compare it against +`manifest.source.upload_folder` and the paths inside `uploads/` in the archive. + +**"Archive format vN is newer than this app supports"** — the backup came from +a newer LIM. Upgrade the target before restoring. + +**Backup is enormous** — expected; PDFs dominate and are stored uncompressed +for speed. Compare `manifest.uploads.total_bytes` against the size of your +uploads directory; they should be close. + +--- + +Created by: Jess Rogerson (yelling commands at Claude.AI)