0 Fork 0

Files

T

jessikitty f01d4b3f58 Add backup and Docker migration runbook

2026-09-24 14:33:34 +10:00

7.9 KiB

Raw Blame History

Backup, Restore & Docker Migration

An in-place backup system that captures everything needed to rebuild the library somewhere else: database rows, every uploaded PDF and image, covers, generated thumbnails and all extra files.

Built as the first half of the move to a Dockerised LIM — take a backup on the Windows box, restore it into the container, done.

What a backup contains

lim-backup-YYYYMMDD-HHMMSS[-label].zip
├── manifest.json            metadata, row counts, integrity report
├── checksums.sha256         SHA-256 of every payload entry
├── database/
│   ├── data.json            portable row dump of all four tables
│   └── lego_instructions.db raw SQLite snapshot (SQLite installs only)
└── uploads/
    ├── pdfs/<set>/…
    ├── images/<set>/…
    ├── covers/<set>/…
    ├── thumbnails/<set>/…
    └── extra_files/<set>/…

Two copies of the database on purpose:

data.json is engine-neutral. It restores into SQLite or Postgres and it is what makes the Docker move possible. All path columns are rewritten to forward slashes on the way out, so a Windows-made backup lands correctly on Linux.
lego_instructions.db is a consistent snapshot taken through SQLite's own online-backup API (safe to run while the app is serving requests). It's the belt-and-braces copy if you'd rather just drop the file in place.

Uploads are walked from disk rather than from the database, so files that lost their record still come across. PDFs and images are stored uncompressed — they don't deflate, and skipping the attempt makes large collections much faster.

Integrity report

Every backup records two lists in the manifest:

Field	Meaning
`missing_files`	Referenced by a database record but absent from disk. These records will survive the migration with a broken link.
`orphaned_files`	Present on disk with no record pointing at them. Backed up anyway.

Worth clearing missing_files before you cut over to Docker — it's the one thing a restore cannot fix for you.

Using it from the admin UI

Admin → Backup & Restore (/admin/backup/).

Shows current upload size, file count, and both integrity lists up front.
Start backup runs in a background thread with a live progress bar, so a multi-gigabyte collection won't time out the request.
Each stored archive can be downloaded, checksum-verified, restored or deleted.
BACKUP_KEEP (default 5) prunes older archives after each UI-initiated backup. Set it to 0 to keep everything.

Restores from the UI require typing RESTORE to confirm, and a destructive wipe needs a second checkbox on top of that.

Using it from the CLI

The CLI is the better path on a fresh instance, because it works before any admin user exists.

flask backup create [--label pre-docker] [--keep 5] [--no-sqlite-file]
flask backup list
flask backup inspect  lim-backup-....zip     # print the manifest
flask backup verify   lim-backup-....zip     # recompute every checksum
flask backup restore  lim-backup-....zip --mode fresh [--force]
                                            [--overwrite-files] [--skip-files] [--yes]

Archive arguments accept either a full path or a bare filename that exists in the backup folder.

Restore modes

Mode	Behaviour
`fresh`	For an empty target. Primary keys are preserved exactly, so every existing URL and bookmark still resolves. Refuses to run against a non-empty database unless you pass `--force` (which wipes the four tables first). Postgres sequences are realigned afterwards.
`merge`	For folding a backup into a live instance. Users are matched by username (then email); sets by set number. A set that already exists is skipped whole — its instructions and extra files are not merged in, to avoid duplicating files. Clashing `brickset_id` values are cleared rather than failing the insert.

Upload files are never overwritten unless you pass --overwrite-files. Password hashes come across intact, so everyone's existing logins keep working.

Migrating to Docker

1. On the current Windows install

cd E:\LIM
flask backup create --label pre-docker
flask backup verify lim-backup-20260924-....zip

Check the summary. If it reports missing files, fix or delete those records first, then take the backup again.

2. Get the archive to the new host

Copy the .zip into whatever directory you'll mount as the container's backup folder, e.g. /srv/lim/backups/.

3. Container volume layout

Three things must live on volumes, not in the image:

Path in container	Holds	Env var
`/data/uploads`	all uploaded files	`UPLOAD_FOLDER`
`/data/backups`	backup archives	`BACKUP_FOLDER`
`/data/db`	SQLite file (if not using Postgres)	via `DATABASE_URL`

Note: UPLOAD_FOLDER is currently hard-coded in app/config.py to app/static/uploads. When the Dockerfile lands, either make it read the UPLOAD_FOLDER environment variable or bind-mount the volume over /app/app/static/uploads. The backup format doesn't care which you pick — paths in the database are relative to whatever UPLOAD_FOLDER resolves to.

Example .env for the container:

FLASK_APP=run.py
FLASK_ENV=production
SECRET_KEY=<your real secret>
DATABASE_URL=sqlite:////data/db/lego_instructions.db
BACKUP_FOLDER=/data/backups
BACKUP_KEEP=7

4. Restore into the container

docker compose exec lim flask backup verify  /data/backups/lim-backup-....zip
docker compose exec lim flask backup restore /data/backups/lim-backup-....zip \
    --mode fresh --yes

Expected output: row counts matching the source, and a file count matching the manifest's uploads.file_count.

5. Verify

Log in with an existing account (hashes migrated, so the old password works).
Open a set with a PDF, an image sequence, a cover image and an extra file.
Spot-check that thumbnails render — they're backed up rather than regenerated, so a missing one means the source was missing it too.

6. Keep the old instance until you're happy

Don't delete E:\LIM until the container has been in daily use for a while. The backup is reversible in both directions: take one from the container and restore it back onto Windows the same way.

Going to Postgres at the same time

data.json restores into any SQLAlchemy-supported engine. Point DATABASE_URL at Postgres, run flask init-db (or the migrations) to create the schema, then flask backup restore ... --mode fresh. Sequences are reset automatically after the insert, so new rows won't collide with the preserved IDs.

Keep the migration one change at a time if you can: container first on SQLite, Postgres later.

Scheduling

Windows Task Scheduler — daily at 2am:

cd /d E:\LIM && venv\Scripts\python.exe -m flask backup create --label nightly --keep 7

Docker host — cron:

0 2 * * * docker compose -f /srv/lim/docker-compose.yml exec -T lim flask backup create --label nightly --keep 7

Then sync /data/backups off the host with your usual job. A backup sitting on the same disk as the thing it's backing up is only half a backup.

Troubleshooting

"Target database is not empty" — you're doing a fresh restore into a populated database. Use --mode merge, or --force to wipe first.

Files restored but images don't load — UPLOAD_FOLDER on the target doesn't point where you think. Compare it against manifest.source.upload_folder and the paths inside uploads/ in the archive.

"Archive format vN is newer than this app supports" — the backup came from a newer LIM. Upgrade the target before restoring.

Backup is enormous — expected; PDFs dominate and are stored uncompressed for speed. Compare manifest.uploads.total_bytes against the size of your uploads directory; they should be close.

Created by: Jess Rogerson (yelling commands at Claude.AI)

7.9 KiB Raw Blame History