Add owner-only logging and Brickset API call tracking
LIM configured no logging at all, so the instruction fetcher had no trail to follow. Adds a rotating file handler on the root logger plus two owner-only admin pages. - Admin > Logs: filter by level, source and text; tracebacks kept with their error; auto-refresh, raw download, rotated-file browsing and rotate-now - Admin > Logs > Brickset API calls: every outbound call recorded with method, params, caller, duration, result count and error, plus daily quota accounting (only getSets counts) and a comparison against getKeyUsageStats - New owner concept (services/ownership.py): a single account, distinct from is_admin, since these pages expose other admins activity. Defaults to the first admin; flask owner show/set to change - Services now log through module loggers; retention setting and pruning for call records - New CLI: flask owner show/set, flask logs path/prune - New table brickset_api_calls, created automatically at boot; LOG_* env vars, /data/logs in Docker
This commit is contained in:
1 parent
89f65e2d74
commit
f7c20de5ac
27 files changed
+1914
-35
No files matched your search
+145
@@ -0,0 +1,145 @@
|
||||
# Logging and API call tracking
|
||||
|
||||
Two owner-only pages under **Admin → Logs**:
|
||||
|
||||
* **Application log** — the rotating file log, filterable and searchable.
|
||||
* **Brickset API calls** — every outbound request, with quota accounting.
|
||||
|
||||
---
|
||||
|
||||
## Owner, not admin
|
||||
|
||||
These pages show everything happening inside the instance, including other
|
||||
admins' activity, so `is_admin` isn't enough. They're restricted to a single
|
||||
**owner** account.
|
||||
|
||||
The owner is whoever `owner_user_id` in `site_settings` points at. If that's
|
||||
unset (or points at a deleted account) it falls back to the lowest-numbered
|
||||
admin — in practice the account `flask init_db` created — and remembers that
|
||||
answer.
|
||||
|
||||
```bat
|
||||
flask owner show
|
||||
flask owner set jessikitty :: promotes them to admin if needed
|
||||
```
|
||||
|
||||
Non-owner admins don't get a permission error on the dashboard; the Logs tile
|
||||
simply isn't rendered for them.
|
||||
|
||||
---
|
||||
|
||||
## Application log
|
||||
|
||||
LIM previously configured no logging at all, which left Flask's logger at
|
||||
WARNING with nothing written to disk. Now a rotating file handler is attached
|
||||
to the **root** logger, so every module logger lands in one file:
|
||||
|
||||
```
|
||||
2026-09-26 13:38:00 INFO app.services.instruction_fetcher | Set 10497: saved 10497-1-building-instructions.pdf (48261234 bytes)
|
||||
```
|
||||
|
||||
Rotation is by size, so disk use is bounded no matter how chatty things get:
|
||||
`LOG_MAX_BYTES` per file (2 MB), `LOG_BACKUP_COUNT` old copies (5). Worst case
|
||||
is 12 MB.
|
||||
|
||||
### The viewer
|
||||
|
||||
Filter by minimum level, by source, and by free text; the source dropdown has
|
||||
shortcuts for the fetcher, the worker and the Brickset client, so you can
|
||||
isolate one feature's activity. Multi-line entries (tracebacks) stay attached
|
||||
to the error that produced them rather than scattering.
|
||||
|
||||
Also on the page: auto-refresh every 10s, download the raw file, switch to a
|
||||
rotated backup, and *Rotate now* to start a clean file before reproducing a
|
||||
problem.
|
||||
|
||||
Reading is capped at the last 1 MB of the file, so a huge log won't stall the
|
||||
page.
|
||||
|
||||
### Environment
|
||||
|
||||
```ini
|
||||
LOG_FOLDER=logs # /data/logs in Docker
|
||||
LOG_LEVEL=INFO # DEBUG when chasing something
|
||||
LOG_MAX_BYTES=2097152
|
||||
LOG_BACKUP_COUNT=5
|
||||
LOG_ACCESS=0 # 1 to include per-request logging (noisy)
|
||||
```
|
||||
|
||||
A relative `LOG_FOLDER` resolves against the project root, not the working
|
||||
directory, same as `UPLOAD_FOLDER`.
|
||||
|
||||
If the folder can't be created or opened, LIM logs a warning and carries on
|
||||
rather than refusing to boot. The viewer says so at the top of the page.
|
||||
|
||||
```bat
|
||||
flask logs path --tail 50
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Brickset API calls
|
||||
|
||||
Every call is recorded: which method, what was asked, which part of LIM asked,
|
||||
how long it took, how many results came back, and the error if it failed.
|
||||
Credentials are never stored — the `login` row records the username only.
|
||||
|
||||
### Quota accounting
|
||||
|
||||
Per Brickset's documentation, **only `getSets` counts against the daily key
|
||||
limit**. The page leads with that number for today, separately from the total,
|
||||
because the total is mostly quota-free traffic:
|
||||
|
||||
| Method | Counts |
|
||||
|--------|--------|
|
||||
| `getSets` | **yes** |
|
||||
| `getInstructions`, `getInstructions2` | no |
|
||||
| `getThemes`, `login`, `getKeyUsageStats` | no |
|
||||
|
||||
Rows that count are flagged `q` in the table and broken out per day.
|
||||
|
||||
*Compare with Brickset's own figures* calls `getKeyUsageStats`, which reports
|
||||
what Brickset thinks you've used over the last 30 days. It's free, and useful
|
||||
for confirming LIM's count matches theirs — a gap means something else is
|
||||
using the same key.
|
||||
|
||||
The instruction fetcher is deliberately quota-light: it uses
|
||||
`getInstructions2` by default and only falls back to `getSets` when a set
|
||||
number can't be found, caching the returned Brickset ID so it never pays
|
||||
twice for the same set. On the calls page that shows up as a long run of
|
||||
`fetcher` rows with no `q` flags.
|
||||
|
||||
### Retention
|
||||
|
||||
90 days by default, changeable on the page. Pruning is manual (*Prune now*) or
|
||||
scheduled:
|
||||
|
||||
```bat
|
||||
flask logs prune --days 30
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Where to look when something's wrong
|
||||
|
||||
**Fetcher downloaded nothing.** Logs → source *Instruction fetcher*. Each job
|
||||
logs how many files Brickset listed, and each download failure with its
|
||||
reason. The Brickset page shows whether the lookup even went out.
|
||||
|
||||
**Suspected quota problems.** Brickset page, "Quota calls today". If that's
|
||||
low but Brickset is refusing you, something else is using the key — compare
|
||||
with their figures.
|
||||
|
||||
**Something broke, no idea what.** Logs, level ERROR, no filter. Tracebacks
|
||||
are kept intact.
|
||||
|
||||
---
|
||||
|
||||
## Database
|
||||
|
||||
`brickset_api_calls`, created automatically at startup by
|
||||
`services/schema_upgrade.py`. No migration step.
|
||||
|
||||
---
|
||||
|
||||
Created by: Jess Rogerson (yelling commands at Claude.AI)
|
||||
Reference in new issue
Block a user