Add automatic instruction fetcher for official sets

Downloads official instruction PDFs from LEGO, located via the Brickset API, one set per interval (10 minutes by default) so neither API gets hammered. Official sets only; MOCs are excluded throughout.

- New admin page at /admin/instruction-fetch: queue builder, pacing controls, live countdown and per-set job history

- Database-backed queue (instruction_fetch_jobs) and last-run timestamp, so pacing survives restarts

- Background worker thread, plus 'flask instructions queue/run/status' for Task Scheduler or cron

- SHA-256 dedupe (instructions.sha256), PDF header check, size cap and streaming downloads via temp files

- brickset_api: getInstructions2 by set number and getInstructions by setID (the old code sent a set number to the setID method); getSets params now proper JSON; getKeyUsageStats added

- Schema top-ups applied automatically at boot: new table plus instructions.sha256/source

- Docker: fetcher env vars passed through compose, no worker thread in the entrypoint bootstrap
This commit is contained in:
jessikitty committed 2026-09-26 13:43:17 +10:00
1 parent aac2b60517
commit 89f65e2d74
23 files changed
+2073 -76

No files matched your search

+6
View File
@@ -18,6 +18,7 @@ A Python web application for organizing and managing your LEGO instruction manua
- 🖼️ **Image Gallery**: View instruction images in an organized gallery
- 📄 **PDF Support**: View and download PDF instructions directly
- 💾 **Backup & Restore**: One-click archive of the database plus every uploaded file, ready to restore anywhere
- ⬇️ **Automatic Instruction Fetcher**: Downloads official instruction PDFs from LEGO (found via Brickset) for official sets, one every 10 minutes so nobody's API gets hammered — see [docs/INSTRUCTION_FETCHER.md](docs/INSTRUCTION_FETCHER.md)
## 🚀 Quick Start
@@ -185,6 +186,11 @@ flask --app run.py shell
### Instruction
- id, set_id, file_type, file_path, file_name, file_size, thumbnail_path
- page_number, user_id, uploaded_at
- sha256, source (set by the automatic fetcher; null for manual uploads)
### InstructionFetchJob
- id, set_id, status, attempts, files_added, source, message
- queued_at, not_before, started_at, finished_at, queued_by_id
### ExtraFile
- id, set_id, file_name, original_filename, file_path, file_type, file_size