# Backup, Restore & Docker Migration An in-place backup system that captures everything needed to rebuild the library somewhere else: database rows, every uploaded PDF and image, covers, generated thumbnails and all extra files. Built as the first half of the move to a Dockerised LIM — take a backup on the Windows box, restore it into the container, done. --- ## What a backup contains ``` lim-backup-YYYYMMDD-HHMMSS[-label].zip ├── manifest.json metadata, row counts, integrity report ├── checksums.sha256 SHA-256 of every payload entry ├── database/ │ ├── data.json portable row dump of all four tables │ └── lego_instructions.db raw SQLite snapshot (SQLite installs only) └── uploads/ ├── pdfs//… ├── images//… ├── covers//… ├── thumbnails//… └── extra_files//… ``` Two copies of the database on purpose: - **`data.json`** is engine-neutral. It restores into SQLite *or* Postgres and it is what makes the Docker move possible. All path columns are rewritten to forward slashes on the way out, so a Windows-made backup lands correctly on Linux. - **`lego_instructions.db`** is a consistent snapshot taken through SQLite's own online-backup API (safe to run while the app is serving requests). It's the belt-and-braces copy if you'd rather just drop the file in place. Uploads are walked from disk rather than from the database, so files that lost their record still come across. PDFs and images are stored uncompressed — they don't deflate, and skipping the attempt makes large collections much faster. ### Integrity report Every backup records two lists in the manifest: | Field | Meaning | |---|---| | `missing_files` | Referenced by a database record but absent from disk. These records will survive the migration with a broken link. | | `orphaned_files` | Present on disk with no record pointing at them. Backed up anyway. | Worth clearing `missing_files` before you cut over to Docker — it's the one thing a restore cannot fix for you. --- ## Using it from the admin UI **Admin → Backup & Restore** (`/admin/backup/`). - Shows current upload size, file count, and both integrity lists up front. - **Start backup** runs in a background thread with a live progress bar, so a multi-gigabyte collection won't time out the request. - Each stored archive can be downloaded, checksum-verified, restored or deleted. - `BACKUP_KEEP` (default 5) prunes older archives after each UI-initiated backup. Set it to `0` to keep everything. Restores from the UI require typing `RESTORE` to confirm, and a destructive wipe needs a second checkbox on top of that. --- ## Using it from the CLI The CLI is the better path on a fresh instance, because it works before any admin user exists. ```bash flask backup create [--label pre-docker] [--keep 5] [--no-sqlite-file] flask backup list flask backup inspect lim-backup-....zip # print the manifest flask backup verify lim-backup-....zip # recompute every checksum flask backup restore lim-backup-....zip --mode fresh [--force] [--overwrite-files] [--skip-files] [--yes] ``` Archive arguments accept either a full path or a bare filename that exists in the backup folder. ### Restore modes | Mode | Behaviour | |---|---| | `fresh` | For an empty target. Primary keys are preserved exactly, so every existing URL and bookmark still resolves. Refuses to run against a non-empty database unless you pass `--force` (which wipes the four tables first). Postgres sequences are realigned afterwards. | | `merge` | For folding a backup into a live instance. Users are matched by username (then email); sets by set number. A set that already exists is skipped whole — its instructions and extra files are *not* merged in, to avoid duplicating files. Clashing `brickset_id` values are cleared rather than failing the insert. | Upload files are never overwritten unless you pass `--overwrite-files`. Password hashes come across intact, so everyone's existing logins keep working. --- ## Migrating to Docker ### 1. On the current Windows install ```powershell cd E:\LIM flask backup create --label pre-docker flask backup verify lim-backup-20260924-....zip ``` Check the summary. If it reports missing files, fix or delete those records first, then take the backup again. ### 2. Get the archive to the new host Copy the `.zip` into whatever directory you'll mount as the container's backup folder, e.g. `/srv/lim/backups/`. ### 3. Container volume layout Three things must live on volumes, not in the image: | Path in container | Holds | Env var | |---|---|---| | `/data/uploads` | all uploaded files | `UPLOAD_FOLDER` | | `/data/backups` | backup archives | `BACKUP_FOLDER` | | `/data/db` | SQLite file (if not using Postgres) | via `DATABASE_URL` | > **Note:** `UPLOAD_FOLDER` is currently hard-coded in `app/config.py` to > `app/static/uploads`. When the Dockerfile lands, either make it read the > `UPLOAD_FOLDER` environment variable or bind-mount the volume over > `/app/app/static/uploads`. The backup format doesn't care which you pick — > paths in the database are relative to whatever `UPLOAD_FOLDER` resolves to. Example `.env` for the container: ```env FLASK_APP=run.py FLASK_ENV=production SECRET_KEY= DATABASE_URL=sqlite:////data/db/lego_instructions.db BACKUP_FOLDER=/data/backups BACKUP_KEEP=7 ``` ### 4. Restore into the container ```bash docker compose exec lim flask backup verify /data/backups/lim-backup-....zip docker compose exec lim flask backup restore /data/backups/lim-backup-....zip \ --mode fresh --yes ``` Expected output: row counts matching the source, and a file count matching the manifest's `uploads.file_count`. ### 5. Verify - Log in with an existing account (hashes migrated, so the old password works). - Open a set with a PDF, an image sequence, a cover image and an extra file. - Spot-check that thumbnails render — they're backed up rather than regenerated, so a missing one means the source was missing it too. ### 6. Keep the old instance until you're happy Don't delete `E:\LIM` until the container has been in daily use for a while. The backup is reversible in both directions: take one from the container and restore it back onto Windows the same way. --- ## Going to Postgres at the same time `data.json` restores into any SQLAlchemy-supported engine. Point `DATABASE_URL` at Postgres, run `flask init-db` (or the migrations) to create the schema, then `flask backup restore ... --mode fresh`. Sequences are reset automatically after the insert, so new rows won't collide with the preserved IDs. Keep the migration one change at a time if you can: container first on SQLite, Postgres later. --- ## Scheduling **Windows Task Scheduler** — daily at 2am: ```powershell cd /d E:\LIM && venv\Scripts\python.exe -m flask backup create --label nightly --keep 7 ``` **Docker host** — cron: ```cron 0 2 * * * docker compose -f /srv/lim/docker-compose.yml exec -T lim flask backup create --label nightly --keep 7 ``` Then sync `/data/backups` off the host with your usual job. A backup sitting on the same disk as the thing it's backing up is only half a backup. --- ## Troubleshooting **"Target database is not empty"** — you're doing a `fresh` restore into a populated database. Use `--mode merge`, or `--force` to wipe first. **Files restored but images don't load** — `UPLOAD_FOLDER` on the target doesn't point where you think. Compare it against `manifest.source.upload_folder` and the paths inside `uploads/` in the archive. **"Archive format vN is newer than this app supports"** — the backup came from a newer LIM. Upgrade the target before restoring. **Backup is enormous** — expected; PDFs dominate and are stored uncompressed for speed. Compare `manifest.uploads.total_bytes` against the size of your uploads directory; they should be close. --- Created by: Jess Rogerson (yelling commands at Claude.AI)