Add backup and Docker migration runbook
This commit is contained in:
jessikitty committed
1 parent
b16106db7d
commit
f01d4b3f58
1 file changed
+223
@@ -0,0 +1,223 @@
|
||||
# Backup, Restore & Docker Migration
|
||||
|
||||
An in-place backup system that captures everything needed to rebuild the
|
||||
library somewhere else: database rows, every uploaded PDF and image, covers,
|
||||
generated thumbnails and all extra files.
|
||||
|
||||
Built as the first half of the move to a Dockerised LIM — take a backup on the
|
||||
Windows box, restore it into the container, done.
|
||||
|
||||
---
|
||||
|
||||
## What a backup contains
|
||||
|
||||
```
|
||||
lim-backup-YYYYMMDD-HHMMSS[-label].zip
|
||||
├── manifest.json metadata, row counts, integrity report
|
||||
├── checksums.sha256 SHA-256 of every payload entry
|
||||
├── database/
|
||||
│ ├── data.json portable row dump of all four tables
|
||||
│ └── lego_instructions.db raw SQLite snapshot (SQLite installs only)
|
||||
└── uploads/
|
||||
├── pdfs/<set>/…
|
||||
├── images/<set>/…
|
||||
├── covers/<set>/…
|
||||
├── thumbnails/<set>/…
|
||||
└── extra_files/<set>/…
|
||||
```
|
||||
|
||||
Two copies of the database on purpose:
|
||||
|
||||
- **`data.json`** is engine-neutral. It restores into SQLite *or* Postgres and
|
||||
it is what makes the Docker move possible. All path columns are rewritten to
|
||||
forward slashes on the way out, so a Windows-made backup lands correctly on
|
||||
Linux.
|
||||
- **`lego_instructions.db`** is a consistent snapshot taken through SQLite's
|
||||
own online-backup API (safe to run while the app is serving requests). It's
|
||||
the belt-and-braces copy if you'd rather just drop the file in place.
|
||||
|
||||
Uploads are walked from disk rather than from the database, so files that lost
|
||||
their record still come across. PDFs and images are stored uncompressed — they
|
||||
don't deflate, and skipping the attempt makes large collections much faster.
|
||||
|
||||
### Integrity report
|
||||
|
||||
Every backup records two lists in the manifest:
|
||||
|
||||
| Field | Meaning |
|
||||
|---|---|
|
||||
| `missing_files` | Referenced by a database record but absent from disk. These records will survive the migration with a broken link. |
|
||||
| `orphaned_files` | Present on disk with no record pointing at them. Backed up anyway. |
|
||||
|
||||
Worth clearing `missing_files` before you cut over to Docker — it's the one
|
||||
thing a restore cannot fix for you.
|
||||
|
||||
---
|
||||
|
||||
## Using it from the admin UI
|
||||
|
||||
**Admin → Backup & Restore** (`/admin/backup/`).
|
||||
|
||||
- Shows current upload size, file count, and both integrity lists up front.
|
||||
- **Start backup** runs in a background thread with a live progress bar, so a
|
||||
multi-gigabyte collection won't time out the request.
|
||||
- Each stored archive can be downloaded, checksum-verified, restored or deleted.
|
||||
- `BACKUP_KEEP` (default 5) prunes older archives after each UI-initiated
|
||||
backup. Set it to `0` to keep everything.
|
||||
|
||||
Restores from the UI require typing `RESTORE` to confirm, and a destructive
|
||||
wipe needs a second checkbox on top of that.
|
||||
|
||||
---
|
||||
|
||||
## Using it from the CLI
|
||||
|
||||
The CLI is the better path on a fresh instance, because it works before any
|
||||
admin user exists.
|
||||
|
||||
```bash
|
||||
flask backup create [--label pre-docker] [--keep 5] [--no-sqlite-file]
|
||||
flask backup list
|
||||
flask backup inspect lim-backup-....zip # print the manifest
|
||||
flask backup verify lim-backup-....zip # recompute every checksum
|
||||
flask backup restore lim-backup-....zip --mode fresh [--force]
|
||||
[--overwrite-files] [--skip-files] [--yes]
|
||||
```
|
||||
|
||||
Archive arguments accept either a full path or a bare filename that exists in
|
||||
the backup folder.
|
||||
|
||||
### Restore modes
|
||||
|
||||
| Mode | Behaviour |
|
||||
|---|---|
|
||||
| `fresh` | For an empty target. Primary keys are preserved exactly, so every existing URL and bookmark still resolves. Refuses to run against a non-empty database unless you pass `--force` (which wipes the four tables first). Postgres sequences are realigned afterwards. |
|
||||
| `merge` | For folding a backup into a live instance. Users are matched by username (then email); sets by set number. A set that already exists is skipped whole — its instructions and extra files are *not* merged in, to avoid duplicating files. Clashing `brickset_id` values are cleared rather than failing the insert. |
|
||||
|
||||
Upload files are never overwritten unless you pass `--overwrite-files`.
|
||||
Password hashes come across intact, so everyone's existing logins keep working.
|
||||
|
||||
---
|
||||
|
||||
## Migrating to Docker
|
||||
|
||||
### 1. On the current Windows install
|
||||
|
||||
```powershell
|
||||
cd E:\LIM
|
||||
flask backup create --label pre-docker
|
||||
flask backup verify lim-backup-20260924-....zip
|
||||
```
|
||||
|
||||
Check the summary. If it reports missing files, fix or delete those records
|
||||
first, then take the backup again.
|
||||
|
||||
### 2. Get the archive to the new host
|
||||
|
||||
Copy the `.zip` into whatever directory you'll mount as the container's backup
|
||||
folder, e.g. `/srv/lim/backups/`.
|
||||
|
||||
### 3. Container volume layout
|
||||
|
||||
Three things must live on volumes, not in the image:
|
||||
|
||||
| Path in container | Holds | Env var |
|
||||
|---|---|---|
|
||||
| `/data/uploads` | all uploaded files | `UPLOAD_FOLDER` |
|
||||
| `/data/backups` | backup archives | `BACKUP_FOLDER` |
|
||||
| `/data/db` | SQLite file (if not using Postgres) | via `DATABASE_URL` |
|
||||
|
||||
> **Note:** `UPLOAD_FOLDER` is currently hard-coded in `app/config.py` to
|
||||
> `app/static/uploads`. When the Dockerfile lands, either make it read the
|
||||
> `UPLOAD_FOLDER` environment variable or bind-mount the volume over
|
||||
> `/app/app/static/uploads`. The backup format doesn't care which you pick —
|
||||
> paths in the database are relative to whatever `UPLOAD_FOLDER` resolves to.
|
||||
|
||||
Example `.env` for the container:
|
||||
|
||||
```env
|
||||
FLASK_APP=run.py
|
||||
FLASK_ENV=production
|
||||
SECRET_KEY=<your real secret>
|
||||
DATABASE_URL=sqlite:////data/db/lego_instructions.db
|
||||
BACKUP_FOLDER=/data/backups
|
||||
BACKUP_KEEP=7
|
||||
```
|
||||
|
||||
### 4. Restore into the container
|
||||
|
||||
```bash
|
||||
docker compose exec lim flask backup verify /data/backups/lim-backup-....zip
|
||||
docker compose exec lim flask backup restore /data/backups/lim-backup-....zip \
|
||||
--mode fresh --yes
|
||||
```
|
||||
|
||||
Expected output: row counts matching the source, and a file count matching the
|
||||
manifest's `uploads.file_count`.
|
||||
|
||||
### 5. Verify
|
||||
|
||||
- Log in with an existing account (hashes migrated, so the old password works).
|
||||
- Open a set with a PDF, an image sequence, a cover image and an extra file.
|
||||
- Spot-check that thumbnails render — they're backed up rather than
|
||||
regenerated, so a missing one means the source was missing it too.
|
||||
|
||||
### 6. Keep the old instance until you're happy
|
||||
|
||||
Don't delete `E:\LIM` until the container has been in daily use for a while.
|
||||
The backup is reversible in both directions: take one from the container and
|
||||
restore it back onto Windows the same way.
|
||||
|
||||
---
|
||||
|
||||
## Going to Postgres at the same time
|
||||
|
||||
`data.json` restores into any SQLAlchemy-supported engine. Point
|
||||
`DATABASE_URL` at Postgres, run `flask init-db` (or the migrations) to create
|
||||
the schema, then `flask backup restore ... --mode fresh`. Sequences are reset
|
||||
automatically after the insert, so new rows won't collide with the preserved
|
||||
IDs.
|
||||
|
||||
Keep the migration one change at a time if you can: container first on SQLite,
|
||||
Postgres later.
|
||||
|
||||
---
|
||||
|
||||
## Scheduling
|
||||
|
||||
**Windows Task Scheduler** — daily at 2am:
|
||||
|
||||
```powershell
|
||||
cd /d E:\LIM && venv\Scripts\python.exe -m flask backup create --label nightly --keep 7
|
||||
```
|
||||
|
||||
**Docker host** — cron:
|
||||
|
||||
```cron
|
||||
0 2 * * * docker compose -f /srv/lim/docker-compose.yml exec -T lim flask backup create --label nightly --keep 7
|
||||
```
|
||||
|
||||
Then sync `/data/backups` off the host with your usual job. A backup sitting on
|
||||
the same disk as the thing it's backing up is only half a backup.
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**"Target database is not empty"** — you're doing a `fresh` restore into a
|
||||
populated database. Use `--mode merge`, or `--force` to wipe first.
|
||||
|
||||
**Files restored but images don't load** — `UPLOAD_FOLDER` on the target
|
||||
doesn't point where you think. Compare it against
|
||||
`manifest.source.upload_folder` and the paths inside `uploads/` in the archive.
|
||||
|
||||
**"Archive format vN is newer than this app supports"** — the backup came from
|
||||
a newer LIM. Upgrade the target before restoring.
|
||||
|
||||
**Backup is enormous** — expected; PDFs dominate and are stored uncompressed
|
||||
for speed. Compare `manifest.uploads.total_bytes` against the size of your
|
||||
uploads directory; they should be close.
|
||||
|
||||
---
|
||||
|
||||
Created by: Jess Rogerson (yelling commands at Claude.AI)
|
||||
Reference in new issue
Block a user