jessikitty
f7c20de5ac
LIM configured no logging at all, so the instruction fetcher had no trail to follow. Adds a rotating file handler on the root logger plus two owner-only admin pages. - Admin > Logs: filter by level, source and text; tracebacks kept with their error; auto-refresh, raw download, rotated-file browsing and rotate-now - Admin > Logs > Brickset API calls: every outbound call recorded with method, params, caller, duration, result count and error, plus daily quota accounting (only getSets counts) and a comparison against getKeyUsageStats - New owner concept (services/ownership.py): a single account, distinct from is_admin, since these pages expose other admins activity. Defaults to the first admin; flask owner show/set to change - Services now log through module loggers; retention setting and pruning for call records - New CLI: flask owner show/set, flask logs path/prune - New table brickset_api_calls, created automatically at boot; LOG_* env vars, /data/logs in Docker
146 lines
4.6 KiB
Markdown
146 lines
4.6 KiB
Markdown
# Logging and API call tracking
|
|
|
|
Two owner-only pages under **Admin → Logs**:
|
|
|
|
* **Application log** — the rotating file log, filterable and searchable.
|
|
* **Brickset API calls** — every outbound request, with quota accounting.
|
|
|
|
---
|
|
|
|
## Owner, not admin
|
|
|
|
These pages show everything happening inside the instance, including other
|
|
admins' activity, so `is_admin` isn't enough. They're restricted to a single
|
|
**owner** account.
|
|
|
|
The owner is whoever `owner_user_id` in `site_settings` points at. If that's
|
|
unset (or points at a deleted account) it falls back to the lowest-numbered
|
|
admin — in practice the account `flask init_db` created — and remembers that
|
|
answer.
|
|
|
|
```bat
|
|
flask owner show
|
|
flask owner set jessikitty :: promotes them to admin if needed
|
|
```
|
|
|
|
Non-owner admins don't get a permission error on the dashboard; the Logs tile
|
|
simply isn't rendered for them.
|
|
|
|
---
|
|
|
|
## Application log
|
|
|
|
LIM previously configured no logging at all, which left Flask's logger at
|
|
WARNING with nothing written to disk. Now a rotating file handler is attached
|
|
to the **root** logger, so every module logger lands in one file:
|
|
|
|
```
|
|
2026-09-26 13:38:00 INFO app.services.instruction_fetcher | Set 10497: saved 10497-1-building-instructions.pdf (48261234 bytes)
|
|
```
|
|
|
|
Rotation is by size, so disk use is bounded no matter how chatty things get:
|
|
`LOG_MAX_BYTES` per file (2 MB), `LOG_BACKUP_COUNT` old copies (5). Worst case
|
|
is 12 MB.
|
|
|
|
### The viewer
|
|
|
|
Filter by minimum level, by source, and by free text; the source dropdown has
|
|
shortcuts for the fetcher, the worker and the Brickset client, so you can
|
|
isolate one feature's activity. Multi-line entries (tracebacks) stay attached
|
|
to the error that produced them rather than scattering.
|
|
|
|
Also on the page: auto-refresh every 10s, download the raw file, switch to a
|
|
rotated backup, and *Rotate now* to start a clean file before reproducing a
|
|
problem.
|
|
|
|
Reading is capped at the last 1 MB of the file, so a huge log won't stall the
|
|
page.
|
|
|
|
### Environment
|
|
|
|
```ini
|
|
LOG_FOLDER=logs # /data/logs in Docker
|
|
LOG_LEVEL=INFO # DEBUG when chasing something
|
|
LOG_MAX_BYTES=2097152
|
|
LOG_BACKUP_COUNT=5
|
|
LOG_ACCESS=0 # 1 to include per-request logging (noisy)
|
|
```
|
|
|
|
A relative `LOG_FOLDER` resolves against the project root, not the working
|
|
directory, same as `UPLOAD_FOLDER`.
|
|
|
|
If the folder can't be created or opened, LIM logs a warning and carries on
|
|
rather than refusing to boot. The viewer says so at the top of the page.
|
|
|
|
```bat
|
|
flask logs path --tail 50
|
|
```
|
|
|
|
---
|
|
|
|
## Brickset API calls
|
|
|
|
Every call is recorded: which method, what was asked, which part of LIM asked,
|
|
how long it took, how many results came back, and the error if it failed.
|
|
Credentials are never stored — the `login` row records the username only.
|
|
|
|
### Quota accounting
|
|
|
|
Per Brickset's documentation, **only `getSets` counts against the daily key
|
|
limit**. The page leads with that number for today, separately from the total,
|
|
because the total is mostly quota-free traffic:
|
|
|
|
| Method | Counts |
|
|
|--------|--------|
|
|
| `getSets` | **yes** |
|
|
| `getInstructions`, `getInstructions2` | no |
|
|
| `getThemes`, `login`, `getKeyUsageStats` | no |
|
|
|
|
Rows that count are flagged `q` in the table and broken out per day.
|
|
|
|
*Compare with Brickset's own figures* calls `getKeyUsageStats`, which reports
|
|
what Brickset thinks you've used over the last 30 days. It's free, and useful
|
|
for confirming LIM's count matches theirs — a gap means something else is
|
|
using the same key.
|
|
|
|
The instruction fetcher is deliberately quota-light: it uses
|
|
`getInstructions2` by default and only falls back to `getSets` when a set
|
|
number can't be found, caching the returned Brickset ID so it never pays
|
|
twice for the same set. On the calls page that shows up as a long run of
|
|
`fetcher` rows with no `q` flags.
|
|
|
|
### Retention
|
|
|
|
90 days by default, changeable on the page. Pruning is manual (*Prune now*) or
|
|
scheduled:
|
|
|
|
```bat
|
|
flask logs prune --days 30
|
|
```
|
|
|
|
---
|
|
|
|
## Where to look when something's wrong
|
|
|
|
**Fetcher downloaded nothing.** Logs → source *Instruction fetcher*. Each job
|
|
logs how many files Brickset listed, and each download failure with its
|
|
reason. The Brickset page shows whether the lookup even went out.
|
|
|
|
**Suspected quota problems.** Brickset page, "Quota calls today". If that's
|
|
low but Brickset is refusing you, something else is using the key — compare
|
|
with their figures.
|
|
|
|
**Something broke, no idea what.** Logs, level ERROR, no filter. Tracebacks
|
|
are kept intact.
|
|
|
|
---
|
|
|
|
## Database
|
|
|
|
`brickset_api_calls`, created automatically at startup by
|
|
`services/schema_upgrade.py`. No migration step.
|
|
|
|
---
|
|
|
|
Created by: Jess Rogerson (yelling commands at Claude.AI)
|