224 lines
7.9 KiB
Markdown
224 lines
7.9 KiB
Markdown
# Backup, Restore & Docker Migration
|
|
|
|
An in-place backup system that captures everything needed to rebuild the
|
|
library somewhere else: database rows, every uploaded PDF and image, covers,
|
|
generated thumbnails and all extra files.
|
|
|
|
Built as the first half of the move to a Dockerised LIM — take a backup on the
|
|
Windows box, restore it into the container, done.
|
|
|
|
---
|
|
|
|
## What a backup contains
|
|
|
|
```
|
|
lim-backup-YYYYMMDD-HHMMSS[-label].zip
|
|
├── manifest.json metadata, row counts, integrity report
|
|
├── checksums.sha256 SHA-256 of every payload entry
|
|
├── database/
|
|
│ ├── data.json portable row dump of all four tables
|
|
│ └── lego_instructions.db raw SQLite snapshot (SQLite installs only)
|
|
└── uploads/
|
|
├── pdfs/<set>/…
|
|
├── images/<set>/…
|
|
├── covers/<set>/…
|
|
├── thumbnails/<set>/…
|
|
└── extra_files/<set>/…
|
|
```
|
|
|
|
Two copies of the database on purpose:
|
|
|
|
- **`data.json`** is engine-neutral. It restores into SQLite *or* Postgres and
|
|
it is what makes the Docker move possible. All path columns are rewritten to
|
|
forward slashes on the way out, so a Windows-made backup lands correctly on
|
|
Linux.
|
|
- **`lego_instructions.db`** is a consistent snapshot taken through SQLite's
|
|
own online-backup API (safe to run while the app is serving requests). It's
|
|
the belt-and-braces copy if you'd rather just drop the file in place.
|
|
|
|
Uploads are walked from disk rather than from the database, so files that lost
|
|
their record still come across. PDFs and images are stored uncompressed — they
|
|
don't deflate, and skipping the attempt makes large collections much faster.
|
|
|
|
### Integrity report
|
|
|
|
Every backup records two lists in the manifest:
|
|
|
|
| Field | Meaning |
|
|
|---|---|
|
|
| `missing_files` | Referenced by a database record but absent from disk. These records will survive the migration with a broken link. |
|
|
| `orphaned_files` | Present on disk with no record pointing at them. Backed up anyway. |
|
|
|
|
Worth clearing `missing_files` before you cut over to Docker — it's the one
|
|
thing a restore cannot fix for you.
|
|
|
|
---
|
|
|
|
## Using it from the admin UI
|
|
|
|
**Admin → Backup & Restore** (`/admin/backup/`).
|
|
|
|
- Shows current upload size, file count, and both integrity lists up front.
|
|
- **Start backup** runs in a background thread with a live progress bar, so a
|
|
multi-gigabyte collection won't time out the request.
|
|
- Each stored archive can be downloaded, checksum-verified, restored or deleted.
|
|
- `BACKUP_KEEP` (default 5) prunes older archives after each UI-initiated
|
|
backup. Set it to `0` to keep everything.
|
|
|
|
Restores from the UI require typing `RESTORE` to confirm, and a destructive
|
|
wipe needs a second checkbox on top of that.
|
|
|
|
---
|
|
|
|
## Using it from the CLI
|
|
|
|
The CLI is the better path on a fresh instance, because it works before any
|
|
admin user exists.
|
|
|
|
```bash
|
|
flask backup create [--label pre-docker] [--keep 5] [--no-sqlite-file]
|
|
flask backup list
|
|
flask backup inspect lim-backup-....zip # print the manifest
|
|
flask backup verify lim-backup-....zip # recompute every checksum
|
|
flask backup restore lim-backup-....zip --mode fresh [--force]
|
|
[--overwrite-files] [--skip-files] [--yes]
|
|
```
|
|
|
|
Archive arguments accept either a full path or a bare filename that exists in
|
|
the backup folder.
|
|
|
|
### Restore modes
|
|
|
|
| Mode | Behaviour |
|
|
|---|---|
|
|
| `fresh` | For an empty target. Primary keys are preserved exactly, so every existing URL and bookmark still resolves. Refuses to run against a non-empty database unless you pass `--force` (which wipes the four tables first). Postgres sequences are realigned afterwards. |
|
|
| `merge` | For folding a backup into a live instance. Users are matched by username (then email); sets by set number. A set that already exists is skipped whole — its instructions and extra files are *not* merged in, to avoid duplicating files. Clashing `brickset_id` values are cleared rather than failing the insert. |
|
|
|
|
Upload files are never overwritten unless you pass `--overwrite-files`.
|
|
Password hashes come across intact, so everyone's existing logins keep working.
|
|
|
|
---
|
|
|
|
## Migrating to Docker
|
|
|
|
### 1. On the current Windows install
|
|
|
|
```powershell
|
|
cd E:\LIM
|
|
flask backup create --label pre-docker
|
|
flask backup verify lim-backup-20260924-....zip
|
|
```
|
|
|
|
Check the summary. If it reports missing files, fix or delete those records
|
|
first, then take the backup again.
|
|
|
|
### 2. Get the archive to the new host
|
|
|
|
Copy the `.zip` into whatever directory you'll mount as the container's backup
|
|
folder, e.g. `/srv/lim/backups/`.
|
|
|
|
### 3. Container volume layout
|
|
|
|
Three things must live on volumes, not in the image:
|
|
|
|
| Path in container | Holds | Env var |
|
|
|---|---|---|
|
|
| `/data/uploads` | all uploaded files | `UPLOAD_FOLDER` |
|
|
| `/data/backups` | backup archives | `BACKUP_FOLDER` |
|
|
| `/data/db` | SQLite file (if not using Postgres) | via `DATABASE_URL` |
|
|
|
|
> **Note:** `UPLOAD_FOLDER` is currently hard-coded in `app/config.py` to
|
|
> `app/static/uploads`. When the Dockerfile lands, either make it read the
|
|
> `UPLOAD_FOLDER` environment variable or bind-mount the volume over
|
|
> `/app/app/static/uploads`. The backup format doesn't care which you pick —
|
|
> paths in the database are relative to whatever `UPLOAD_FOLDER` resolves to.
|
|
|
|
Example `.env` for the container:
|
|
|
|
```env
|
|
FLASK_APP=run.py
|
|
FLASK_ENV=production
|
|
SECRET_KEY=<your real secret>
|
|
DATABASE_URL=sqlite:////data/db/lego_instructions.db
|
|
BACKUP_FOLDER=/data/backups
|
|
BACKUP_KEEP=7
|
|
```
|
|
|
|
### 4. Restore into the container
|
|
|
|
```bash
|
|
docker compose exec lim flask backup verify /data/backups/lim-backup-....zip
|
|
docker compose exec lim flask backup restore /data/backups/lim-backup-....zip \
|
|
--mode fresh --yes
|
|
```
|
|
|
|
Expected output: row counts matching the source, and a file count matching the
|
|
manifest's `uploads.file_count`.
|
|
|
|
### 5. Verify
|
|
|
|
- Log in with an existing account (hashes migrated, so the old password works).
|
|
- Open a set with a PDF, an image sequence, a cover image and an extra file.
|
|
- Spot-check that thumbnails render — they're backed up rather than
|
|
regenerated, so a missing one means the source was missing it too.
|
|
|
|
### 6. Keep the old instance until you're happy
|
|
|
|
Don't delete `E:\LIM` until the container has been in daily use for a while.
|
|
The backup is reversible in both directions: take one from the container and
|
|
restore it back onto Windows the same way.
|
|
|
|
---
|
|
|
|
## Going to Postgres at the same time
|
|
|
|
`data.json` restores into any SQLAlchemy-supported engine. Point
|
|
`DATABASE_URL` at Postgres, run `flask init-db` (or the migrations) to create
|
|
the schema, then `flask backup restore ... --mode fresh`. Sequences are reset
|
|
automatically after the insert, so new rows won't collide with the preserved
|
|
IDs.
|
|
|
|
Keep the migration one change at a time if you can: container first on SQLite,
|
|
Postgres later.
|
|
|
|
---
|
|
|
|
## Scheduling
|
|
|
|
**Windows Task Scheduler** — daily at 2am:
|
|
|
|
```powershell
|
|
cd /d E:\LIM && venv\Scripts\python.exe -m flask backup create --label nightly --keep 7
|
|
```
|
|
|
|
**Docker host** — cron:
|
|
|
|
```cron
|
|
0 2 * * * docker compose -f /srv/lim/docker-compose.yml exec -T lim flask backup create --label nightly --keep 7
|
|
```
|
|
|
|
Then sync `/data/backups` off the host with your usual job. A backup sitting on
|
|
the same disk as the thing it's backing up is only half a backup.
|
|
|
|
---
|
|
|
|
## Troubleshooting
|
|
|
|
**"Target database is not empty"** — you're doing a `fresh` restore into a
|
|
populated database. Use `--mode merge`, or `--force` to wipe first.
|
|
|
|
**Files restored but images don't load** — `UPLOAD_FOLDER` on the target
|
|
doesn't point where you think. Compare it against
|
|
`manifest.source.upload_folder` and the paths inside `uploads/` in the archive.
|
|
|
|
**"Archive format vN is newer than this app supports"** — the backup came from
|
|
a newer LIM. Upgrade the target before restoring.
|
|
|
|
**Backup is enormous** — expected; PDFs dominate and are stored uncompressed
|
|
for speed. Compare `manifest.uploads.total_bytes` against the size of your
|
|
uploads directory; they should be close.
|
|
|
|
---
|
|
|
|
Created by: Jess Rogerson (yelling commands at Claude.AI)
|