# Automatic Instruction Fetcher

Downloads official building instruction PDFs for **official sets only**, one
set at a time, on a timer you control. MOCs are never touched — they have no
Brickset entry to look up.

Admin → **Instruction Fetcher** (`/admin/instruction-fetch`).

---

## Where the files actually come from

Brickset's API doesn't host the PDFs; it tells you where LEGO keeps them.

```
getInstructions2(setNumber)  ->  [{ URL, description }, ...]   (brickset.com)
         |
         v
GET https://www.lego.com/.../6876-1.pdf                        (lego.com)
```

So "from Brickset or LEGO" is one flow: Brickset for the lookup, LEGO for the
file. Saved PDFs land in the normal uploads tree
(`uploads/pdfs/<set number>/`) with a thumbnail, exactly like a manual upload,
and appear on the set page as ordinary instructions.

### API quota

Per Brickset's documentation, **only `getSets` counts against the daily key
limit**. `getInstructions` and `getInstructions2` do not. The fetcher is built
around that:

| Call | When | Counts against quota |
|------|------|----------------------|
| `getInstructions` (by setID) | set already has `brickset_id` | no |
| `getInstructions2` (by number) | normal path | no |
| `getSets` | only if the number isn't found and set-ID lookup is enabled | **yes** |

When a `getSets` lookup does happen, the resulting Brickset ID is written back
to the set, so it never happens twice for the same set.

---

## Pacing

One set per interval, 10 minutes by default (≈144 sets/day). Within a set,
multi-book instructions are spaced by a few seconds each.

The "last run" timestamp is stored in `site_settings`, not in memory, so:

* restarting LIM doesn't reset the clock and let it stampede,
* changing the interval takes effect on the next tick, no restart needed,
* pausing keeps the queue intact.

All of it is configurable on the page:

| Setting | Default | Notes |
|---------|---------|-------|
| Work through the queue automatically | off | master switch; off = nothing happens |
| Minutes between sets | 10 | 1–1440 |
| Seconds between PDFs of the same set | 5 | 0–300 |
| Attempts before giving up | 3 | failures back off two intervals, then retry |
| Skip sets that already have a PDF | on | |
| Allow set-ID lookups | on | the only quota-consuming call |

---

## Using it

1. **Queue sets missing instructions** — newest first. An optional limit lets
   you try 10 before committing to 900.
2. Switch **Work through the queue automatically** on.
3. Walk away. The page shows a live countdown and refreshes the job table as
   each set finishes.

**Run one now** processes a single set immediately in the background (tick
*ignore the timer* to skip the cooldown). Useful for checking credentials
before queueing hundreds.

### Job statuses

| Status | Meaning |
|--------|---------|
| `queued` | waiting its turn |
| `running` | being fetched right now |
| `done` | files saved, or already had them |
| `empty` | Brickset lists no instructions for this set |
| `failed` | gave up after the configured attempts |
| `cancelled` | stopped by an admin, or the set was deleted |

---

## Safety rails

* **PDF only.** The first bytes must be `%PDF`; anything else is rejected with
  the reason recorded on the job.
* **Size cap.** `FETCH_MAX_FILE_MB` (default 150) is checked both from the
  `Content-Length` header and while streaming.
* **Deduplication.** Every downloaded file is hashed (SHA-256, stored on
  `instructions.sha256`). A file already on the set is skipped, so re-running
  a set never creates duplicates.
* **Files are only written after a successful download** — partial downloads
  go to a temp file and are deleted.
* **Official sets only**, enforced in the query, the queue, and the fetch
  itself.

---

## Running it from Task Scheduler instead

The background thread lives in the web process. If you'd rather drive it
externally, set `INSTRUCTION_FETCH_WORKER=0` in `.env` and schedule:

```bat
flask instructions run --count 1 --force
```

Other commands:

```bat
flask instructions status                      :: queue counts + countdown
flask instructions queue --missing --limit 50  :: queue sets with no PDF
flask instructions queue --all                 :: queue every official set
flask instructions run --count 3               :: respect the interval
```

`--force` ignores both the timer and the enabled switch, which is what you
want when the schedule itself provides the pacing.

> Run **one** worker. If the web process has the thread running *and* Task
> Scheduler calls the CLI, two of them can claim jobs at once. The CLI won't
> start a worker thread of its own, so `INSTRUCTION_FETCH_WORKER=0` is the
> only switch you need.

---

## Environment variables

```ini
INSTRUCTION_FETCH_WORKER=1        # 0 to disable the in-process worker
INSTRUCTION_FETCH_TICK_SECONDS=20 # how often it checks whether it's due
FETCH_MAX_FILE_MB=150             # reject anything larger
FETCH_TIMEOUT_SECONDS=60
```

Everything else is in the admin UI, not the environment.

---

## Database changes

Applied automatically at startup by `services/schema_upgrade.py`; no manual
migration:

* new table `instruction_fetch_jobs`
* new columns `instructions.sha256`, `instructions.source`

Existing instruction rows keep `sha256 = NULL`, which simply means dedupe
can't compare against them.

---

## Troubleshooting

**"Brickset API not configured"** — `BRICKSET_API_KEY`, `BRICKSET_USERNAME`
and `BRICKSET_PASSWORD` in `.env`, then restart.

**Everything comes back `empty`** — check the set numbers. Brickset stores
them as `{number}-{variant}`; the fetcher appends `-1` automatically, but an
unusual variant (`-2`, `-3`) has to be in the set number itself.

**"The background worker thread isn't running"** — either
`INSTRUCTION_FETCH_WORKER=0`, or LIM is being served in a way that loads the
app without starting it. *Run one now* still works.

**Downloads fail with a connection error** — the server needs outbound HTTPS
to `brickset.com` and `lego.com`.

---

Created by: Jess Rogerson (yelling commands at Claude.AI)