lego-instructions-manager/docs/INSTRUCTION_FETCHER.md

Automatic Instruction Fetcher

Downloads official building instruction PDFs for official sets only, one set at a time, on a timer you control. MOCs are never touched — they have no Brickset entry to look up.

Admin → Instruction Fetcher (/admin/instruction-fetch).

---

Where the files actually come from

Brickset's API doesn't host the PDFs; it tells you where LEGO keeps them.

getInstructions2(setNumber)  ->  [{ URL, description }, ...]   (brickset.com)
         |
         v
GET https://www.lego.com/.../6876-1.pdf                        (lego.com)

So "from Brickset or LEGO" is one flow: Brickset for the lookup, LEGO for the file. Saved PDFs land in the normal uploads tree (uploads/pdfs/<set number>/) with a thumbnail, exactly like a manual upload, and appear on the set page as ordinary instructions.

API quota

Per Brickset's documentation, only getSets counts against the daily key limit. getInstructions and getInstructions2 do not. The fetcher is built around that:

Call	When	Counts against quota
getInstructions (by setID)	set already has brickset_id	no
getInstructions2 (by number)	normal path	no
getSets	only if the number isn't found and set-ID lookup is enabled	yes

When a getSets lookup does happen, the resulting Brickset ID is written back to the set, so it never happens twice for the same set.

---

Pacing

One set per interval, 10 minutes by default (≈144 sets/day). Within a set, multi-book instructions are spaced by a few seconds each.

The "last run" timestamp is stored in site_settings, not in memory, so:

• restarting LIM doesn't reset the clock and let it stampede,
• changing the interval takes effect on the next tick, no restart needed,
• pausing keeps the queue intact.

All of it is configurable on the page:

Setting	Default	Notes
Work through the queue automatically	off	master switch; off = nothing happens
Minutes between sets	10	1–1440
Seconds between PDFs of the same set	5	0–300
Attempts before giving up	3	failures back off two intervals, then retry
Skip sets that already have a PDF	on
Allow set-ID lookups	on	the only quota-consuming call

---

Using it

1. Queue sets missing instructions — newest first. An optional limit lets you try 10 before committing to 900.
2. Switch Work through the queue automatically on.
3. Walk away. The page shows a live countdown and refreshes the job table as each set finishes.

Run one now processes a single set immediately in the background (tick ignore the timer to skip the cooldown). Useful for checking credentials before queueing hundreds.

Job statuses

Status	Meaning
queued	waiting its turn
running	being fetched right now
done	files saved, or already had them
empty	Brickset lists no instructions for this set
failed	gave up after the configured attempts
cancelled	stopped by an admin, or the set was deleted

---

Safety rails

• PDF only. The first bytes must be %PDF; anything else is rejected with the reason recorded on the job.
• Size cap. FETCH_MAX_FILE_MB (default 150) is checked both from the Content-Length header and while streaming.
• Deduplication. Every downloaded file is hashed (SHA-256, stored on instructions.sha256). A file already on the set is skipped, so re-running a set never creates duplicates.
• Files are only written after a successful download — partial downloads go to a temp file and are deleted.
• Official sets only, enforced in the query, the queue, and the fetch itself.

---

Running it from Task Scheduler instead

The background thread lives in the web process. If you'd rather drive it externally, set INSTRUCTION_FETCH_WORKER=0 in .env and schedule:

flask instructions run --count 1 --force

Other commands:

flask instructions status                      :: queue counts + countdown
flask instructions queue --missing --limit 50  :: queue sets with no PDF
flask instructions queue --all                 :: queue every official set
flask instructions run --count 3               :: respect the interval

--force ignores both the timer and the enabled switch, which is what you want when the schedule itself provides the pacing.

Run one worker. If the web process has the thread running and Task Scheduler calls the CLI, two of them can claim jobs at once. The CLI won't start a worker thread of its own, so INSTRUCTION_FETCH_WORKER=0 is the only switch you need.

---

Environment variables

INSTRUCTION_FETCH_WORKER=1        # 0 to disable the in-process worker
INSTRUCTION_FETCH_TICK_SECONDS=20 # how often it checks whether it's due
FETCH_MAX_FILE_MB=150             # reject anything larger
FETCH_TIMEOUT_SECONDS=60

Everything else is in the admin UI, not the environment.

---

Database changes

Applied automatically at startup by services/schema_upgrade.py; no manual migration:

• new table instruction_fetch_jobs
• new columns instructions.sha256, instructions.source

Existing instruction rows keep sha256 = NULL, which simply means dedupe can't compare against them.

---

Troubleshooting

"Brickset API not configured" — BRICKSET_API_KEY, BRICKSET_USERNAME and BRICKSET_PASSWORD in .env, then restart.

Everything comes back empty — check the set numbers. Brickset stores them as {number}-{variant}; the fetcher appends -1 automatically, but an unusual variant (-2, -3) has to be in the set number itself.

"The background worker thread isn't running" — either INSTRUCTION_FETCH_WORKER=0, or LIM is being served in a way that loads the app without starting it. Run one now still works.

Downloads fail with a connection error — the server needs outbound HTTPS to brickset.com and lego.com.

---

Created by: Jess Rogerson (yelling commands at Claude.AI)