Add automatic instruction fetcher for official sets
Downloads official instruction PDFs from LEGO, located via the Brickset API, one set per interval (10 minutes by default) so neither API gets hammered. Official sets only; MOCs are excluded throughout. - New admin page at /admin/instruction-fetch: queue builder, pacing controls, live countdown and per-set job history - Database-backed queue (instruction_fetch_jobs) and last-run timestamp, so pacing survives restarts - Background worker thread, plus 'flask instructions queue/run/status' for Task Scheduler or cron - SHA-256 dedupe (instructions.sha256), PDF header check, size cap and streaming downloads via temp files - brickset_api: getInstructions2 by set number and getInstructions by setID (the old code sent a set number to the setID method); getSets params now proper JSON; getKeyUsageStats added - Schema top-ups applied automatically at boot: new table plus instructions.sha256/source - Docker: fetcher env vars passed through compose, no worker thread in the entrypoint bootstrap
This commit is contained in:
jessikitty committed
1 parent
aac2b60517
commit
89f65e2d74
23 files changed
+2073
-76
No files matched your search
@@ -0,0 +1,181 @@
|
||||
# Automatic Instruction Fetcher
|
||||
|
||||
Downloads official building instruction PDFs for **official sets only**, one
|
||||
set at a time, on a timer you control. MOCs are never touched — they have no
|
||||
Brickset entry to look up.
|
||||
|
||||
Admin → **Instruction Fetcher** (`/admin/instruction-fetch`).
|
||||
|
||||
---
|
||||
|
||||
## Where the files actually come from
|
||||
|
||||
Brickset's API doesn't host the PDFs; it tells you where LEGO keeps them.
|
||||
|
||||
```
|
||||
getInstructions2(setNumber) -> [{ URL, description }, ...] (brickset.com)
|
||||
|
|
||||
v
|
||||
GET https://www.lego.com/.../6876-1.pdf (lego.com)
|
||||
```
|
||||
|
||||
So "from Brickset or LEGO" is one flow: Brickset for the lookup, LEGO for the
|
||||
file. Saved PDFs land in the normal uploads tree
|
||||
(`uploads/pdfs/<set number>/`) with a thumbnail, exactly like a manual upload,
|
||||
and appear on the set page as ordinary instructions.
|
||||
|
||||
### API quota
|
||||
|
||||
Per Brickset's documentation, **only `getSets` counts against the daily key
|
||||
limit**. `getInstructions` and `getInstructions2` do not. The fetcher is built
|
||||
around that:
|
||||
|
||||
| Call | When | Counts against quota |
|
||||
|------|------|----------------------|
|
||||
| `getInstructions` (by setID) | set already has `brickset_id` | no |
|
||||
| `getInstructions2` (by number) | normal path | no |
|
||||
| `getSets` | only if the number isn't found and set-ID lookup is enabled | **yes** |
|
||||
|
||||
When a `getSets` lookup does happen, the resulting Brickset ID is written back
|
||||
to the set, so it never happens twice for the same set.
|
||||
|
||||
---
|
||||
|
||||
## Pacing
|
||||
|
||||
One set per interval, 10 minutes by default (≈144 sets/day). Within a set,
|
||||
multi-book instructions are spaced by a few seconds each.
|
||||
|
||||
The "last run" timestamp is stored in `site_settings`, not in memory, so:
|
||||
|
||||
* restarting LIM doesn't reset the clock and let it stampede,
|
||||
* changing the interval takes effect on the next tick, no restart needed,
|
||||
* pausing keeps the queue intact.
|
||||
|
||||
All of it is configurable on the page:
|
||||
|
||||
| Setting | Default | Notes |
|
||||
|---------|---------|-------|
|
||||
| Work through the queue automatically | off | master switch; off = nothing happens |
|
||||
| Minutes between sets | 10 | 1–1440 |
|
||||
| Seconds between PDFs of the same set | 5 | 0–300 |
|
||||
| Attempts before giving up | 3 | failures back off two intervals, then retry |
|
||||
| Skip sets that already have a PDF | on | |
|
||||
| Allow set-ID lookups | on | the only quota-consuming call |
|
||||
|
||||
---
|
||||
|
||||
## Using it
|
||||
|
||||
1. **Queue sets missing instructions** — newest first. An optional limit lets
|
||||
you try 10 before committing to 900.
|
||||
2. Switch **Work through the queue automatically** on.
|
||||
3. Walk away. The page shows a live countdown and refreshes the job table as
|
||||
each set finishes.
|
||||
|
||||
**Run one now** processes a single set immediately in the background (tick
|
||||
*ignore the timer* to skip the cooldown). Useful for checking credentials
|
||||
before queueing hundreds.
|
||||
|
||||
### Job statuses
|
||||
|
||||
| Status | Meaning |
|
||||
|--------|---------|
|
||||
| `queued` | waiting its turn |
|
||||
| `running` | being fetched right now |
|
||||
| `done` | files saved, or already had them |
|
||||
| `empty` | Brickset lists no instructions for this set |
|
||||
| `failed` | gave up after the configured attempts |
|
||||
| `cancelled` | stopped by an admin, or the set was deleted |
|
||||
|
||||
---
|
||||
|
||||
## Safety rails
|
||||
|
||||
* **PDF only.** The first bytes must be `%PDF`; anything else is rejected with
|
||||
the reason recorded on the job.
|
||||
* **Size cap.** `FETCH_MAX_FILE_MB` (default 150) is checked both from the
|
||||
`Content-Length` header and while streaming.
|
||||
* **Deduplication.** Every downloaded file is hashed (SHA-256, stored on
|
||||
`instructions.sha256`). A file already on the set is skipped, so re-running
|
||||
a set never creates duplicates.
|
||||
* **Files are only written after a successful download** — partial downloads
|
||||
go to a temp file and are deleted.
|
||||
* **Official sets only**, enforced in the query, the queue, and the fetch
|
||||
itself.
|
||||
|
||||
---
|
||||
|
||||
## Running it from Task Scheduler instead
|
||||
|
||||
The background thread lives in the web process. If you'd rather drive it
|
||||
externally, set `INSTRUCTION_FETCH_WORKER=0` in `.env` and schedule:
|
||||
|
||||
```bat
|
||||
flask instructions run --count 1 --force
|
||||
```
|
||||
|
||||
Other commands:
|
||||
|
||||
```bat
|
||||
flask instructions status :: queue counts + countdown
|
||||
flask instructions queue --missing --limit 50 :: queue sets with no PDF
|
||||
flask instructions queue --all :: queue every official set
|
||||
flask instructions run --count 3 :: respect the interval
|
||||
```
|
||||
|
||||
`--force` ignores both the timer and the enabled switch, which is what you
|
||||
want when the schedule itself provides the pacing.
|
||||
|
||||
> Run **one** worker. If the web process has the thread running *and* Task
|
||||
> Scheduler calls the CLI, two of them can claim jobs at once. The CLI won't
|
||||
> start a worker thread of its own, so `INSTRUCTION_FETCH_WORKER=0` is the
|
||||
> only switch you need.
|
||||
|
||||
---
|
||||
|
||||
## Environment variables
|
||||
|
||||
```ini
|
||||
INSTRUCTION_FETCH_WORKER=1 # 0 to disable the in-process worker
|
||||
INSTRUCTION_FETCH_TICK_SECONDS=20 # how often it checks whether it's due
|
||||
FETCH_MAX_FILE_MB=150 # reject anything larger
|
||||
FETCH_TIMEOUT_SECONDS=60
|
||||
```
|
||||
|
||||
Everything else is in the admin UI, not the environment.
|
||||
|
||||
---
|
||||
|
||||
## Database changes
|
||||
|
||||
Applied automatically at startup by `services/schema_upgrade.py`; no manual
|
||||
migration:
|
||||
|
||||
* new table `instruction_fetch_jobs`
|
||||
* new columns `instructions.sha256`, `instructions.source`
|
||||
|
||||
Existing instruction rows keep `sha256 = NULL`, which simply means dedupe
|
||||
can't compare against them.
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**"Brickset API not configured"** — `BRICKSET_API_KEY`, `BRICKSET_USERNAME`
|
||||
and `BRICKSET_PASSWORD` in `.env`, then restart.
|
||||
|
||||
**Everything comes back `empty`** — check the set numbers. Brickset stores
|
||||
them as `{number}-{variant}`; the fetcher appends `-1` automatically, but an
|
||||
unusual variant (`-2`, `-3`) has to be in the set number itself.
|
||||
|
||||
**"The background worker thread isn't running"** — either
|
||||
`INSTRUCTION_FETCH_WORKER=0`, or LIM is being served in a way that loads the
|
||||
app without starting it. *Run one now* still works.
|
||||
|
||||
**Downloads fail with a connection error** — the server needs outbound HTTPS
|
||||
to `brickset.com` and `lego.com`.
|
||||
|
||||
---
|
||||
|
||||
Created by: Jess Rogerson (yelling commands at Claude.AI)
|
||||
Reference in new issue
Block a user