jessikitty
89f65e2d74
Downloads official instruction PDFs from LEGO, located via the Brickset API, one set per interval (10 minutes by default) so neither API gets hammered. Official sets only; MOCs are excluded throughout. - New admin page at /admin/instruction-fetch: queue builder, pacing controls, live countdown and per-set job history - Database-backed queue (instruction_fetch_jobs) and last-run timestamp, so pacing survives restarts - Background worker thread, plus 'flask instructions queue/run/status' for Task Scheduler or cron - SHA-256 dedupe (instructions.sha256), PDF header check, size cap and streaming downloads via temp files - brickset_api: getInstructions2 by set number and getInstructions by setID (the old code sent a set number to the setID method); getSets params now proper JSON; getKeyUsageStats added - Schema top-ups applied automatically at boot: new table plus instructions.sha256/source - Docker: fetcher env vars passed through compose, no worker thread in the entrypoint bootstrap
182 lines
6.0 KiB
Markdown
182 lines
6.0 KiB
Markdown
# Automatic Instruction Fetcher
|
||
|
||
Downloads official building instruction PDFs for **official sets only**, one
|
||
set at a time, on a timer you control. MOCs are never touched — they have no
|
||
Brickset entry to look up.
|
||
|
||
Admin → **Instruction Fetcher** (`/admin/instruction-fetch`).
|
||
|
||
---
|
||
|
||
## Where the files actually come from
|
||
|
||
Brickset's API doesn't host the PDFs; it tells you where LEGO keeps them.
|
||
|
||
```
|
||
getInstructions2(setNumber) -> [{ URL, description }, ...] (brickset.com)
|
||
|
|
||
v
|
||
GET https://www.lego.com/.../6876-1.pdf (lego.com)
|
||
```
|
||
|
||
So "from Brickset or LEGO" is one flow: Brickset for the lookup, LEGO for the
|
||
file. Saved PDFs land in the normal uploads tree
|
||
(`uploads/pdfs/<set number>/`) with a thumbnail, exactly like a manual upload,
|
||
and appear on the set page as ordinary instructions.
|
||
|
||
### API quota
|
||
|
||
Per Brickset's documentation, **only `getSets` counts against the daily key
|
||
limit**. `getInstructions` and `getInstructions2` do not. The fetcher is built
|
||
around that:
|
||
|
||
| Call | When | Counts against quota |
|
||
|------|------|----------------------|
|
||
| `getInstructions` (by setID) | set already has `brickset_id` | no |
|
||
| `getInstructions2` (by number) | normal path | no |
|
||
| `getSets` | only if the number isn't found and set-ID lookup is enabled | **yes** |
|
||
|
||
When a `getSets` lookup does happen, the resulting Brickset ID is written back
|
||
to the set, so it never happens twice for the same set.
|
||
|
||
---
|
||
|
||
## Pacing
|
||
|
||
One set per interval, 10 minutes by default (≈144 sets/day). Within a set,
|
||
multi-book instructions are spaced by a few seconds each.
|
||
|
||
The "last run" timestamp is stored in `site_settings`, not in memory, so:
|
||
|
||
* restarting LIM doesn't reset the clock and let it stampede,
|
||
* changing the interval takes effect on the next tick, no restart needed,
|
||
* pausing keeps the queue intact.
|
||
|
||
All of it is configurable on the page:
|
||
|
||
| Setting | Default | Notes |
|
||
|---------|---------|-------|
|
||
| Work through the queue automatically | off | master switch; off = nothing happens |
|
||
| Minutes between sets | 10 | 1–1440 |
|
||
| Seconds between PDFs of the same set | 5 | 0–300 |
|
||
| Attempts before giving up | 3 | failures back off two intervals, then retry |
|
||
| Skip sets that already have a PDF | on | |
|
||
| Allow set-ID lookups | on | the only quota-consuming call |
|
||
|
||
---
|
||
|
||
## Using it
|
||
|
||
1. **Queue sets missing instructions** — newest first. An optional limit lets
|
||
you try 10 before committing to 900.
|
||
2. Switch **Work through the queue automatically** on.
|
||
3. Walk away. The page shows a live countdown and refreshes the job table as
|
||
each set finishes.
|
||
|
||
**Run one now** processes a single set immediately in the background (tick
|
||
*ignore the timer* to skip the cooldown). Useful for checking credentials
|
||
before queueing hundreds.
|
||
|
||
### Job statuses
|
||
|
||
| Status | Meaning |
|
||
|--------|---------|
|
||
| `queued` | waiting its turn |
|
||
| `running` | being fetched right now |
|
||
| `done` | files saved, or already had them |
|
||
| `empty` | Brickset lists no instructions for this set |
|
||
| `failed` | gave up after the configured attempts |
|
||
| `cancelled` | stopped by an admin, or the set was deleted |
|
||
|
||
---
|
||
|
||
## Safety rails
|
||
|
||
* **PDF only.** The first bytes must be `%PDF`; anything else is rejected with
|
||
the reason recorded on the job.
|
||
* **Size cap.** `FETCH_MAX_FILE_MB` (default 150) is checked both from the
|
||
`Content-Length` header and while streaming.
|
||
* **Deduplication.** Every downloaded file is hashed (SHA-256, stored on
|
||
`instructions.sha256`). A file already on the set is skipped, so re-running
|
||
a set never creates duplicates.
|
||
* **Files are only written after a successful download** — partial downloads
|
||
go to a temp file and are deleted.
|
||
* **Official sets only**, enforced in the query, the queue, and the fetch
|
||
itself.
|
||
|
||
---
|
||
|
||
## Running it from Task Scheduler instead
|
||
|
||
The background thread lives in the web process. If you'd rather drive it
|
||
externally, set `INSTRUCTION_FETCH_WORKER=0` in `.env` and schedule:
|
||
|
||
```bat
|
||
flask instructions run --count 1 --force
|
||
```
|
||
|
||
Other commands:
|
||
|
||
```bat
|
||
flask instructions status :: queue counts + countdown
|
||
flask instructions queue --missing --limit 50 :: queue sets with no PDF
|
||
flask instructions queue --all :: queue every official set
|
||
flask instructions run --count 3 :: respect the interval
|
||
```
|
||
|
||
`--force` ignores both the timer and the enabled switch, which is what you
|
||
want when the schedule itself provides the pacing.
|
||
|
||
> Run **one** worker. If the web process has the thread running *and* Task
|
||
> Scheduler calls the CLI, two of them can claim jobs at once. The CLI won't
|
||
> start a worker thread of its own, so `INSTRUCTION_FETCH_WORKER=0` is the
|
||
> only switch you need.
|
||
|
||
---
|
||
|
||
## Environment variables
|
||
|
||
```ini
|
||
INSTRUCTION_FETCH_WORKER=1 # 0 to disable the in-process worker
|
||
INSTRUCTION_FETCH_TICK_SECONDS=20 # how often it checks whether it's due
|
||
FETCH_MAX_FILE_MB=150 # reject anything larger
|
||
FETCH_TIMEOUT_SECONDS=60
|
||
```
|
||
|
||
Everything else is in the admin UI, not the environment.
|
||
|
||
---
|
||
|
||
## Database changes
|
||
|
||
Applied automatically at startup by `services/schema_upgrade.py`; no manual
|
||
migration:
|
||
|
||
* new table `instruction_fetch_jobs`
|
||
* new columns `instructions.sha256`, `instructions.source`
|
||
|
||
Existing instruction rows keep `sha256 = NULL`, which simply means dedupe
|
||
can't compare against them.
|
||
|
||
---
|
||
|
||
## Troubleshooting
|
||
|
||
**"Brickset API not configured"** — `BRICKSET_API_KEY`, `BRICKSET_USERNAME`
|
||
and `BRICKSET_PASSWORD` in `.env`, then restart.
|
||
|
||
**Everything comes back `empty`** — check the set numbers. Brickset stores
|
||
them as `{number}-{variant}`; the fetcher appends `-1` automatically, but an
|
||
unusual variant (`-2`, `-3`) has to be in the set number itself.
|
||
|
||
**"The background worker thread isn't running"** — either
|
||
`INSTRUCTION_FETCH_WORKER=0`, or LIM is being served in a way that loads the
|
||
app without starting it. *Run one now* still works.
|
||
|
||
**Downloads fail with a connection error** — the server needs outbound HTTPS
|
||
to `brickset.com` and `lego.com`.
|
||
|
||
---
|
||
|
||
Created by: Jess Rogerson (yelling commands at Claude.AI)
|