211 lines
10 KiB
Markdown
211 lines
10 KiB
Markdown
# Architecture
|
||
|
||
Current as of 2026-09-23. Supersedes the v2 tracking design throughout.
|
||
|
||
## Pipeline
|
||
|
||
```
|
||
camera video
|
||
-> ROI crops around last-seen crests, at NATIVE resolution (steady state)
|
||
or full detection canvas, long side capped (acquire / every 600ms / on loss)
|
||
-> ArUco decode, maxHamming 0
|
||
-> sub-pixel corner refinement (gradient centroid per border edge)
|
||
-> filtered to scene anchor ids, deduped by id
|
||
-> TRACKING ENGINE -> camera pose in world
|
||
-> three.js render: server-driven ghosts + depth-only occluder meshes
|
||
```
|
||
|
||
Rendering runs every animation frame; detection and solving do not.
|
||
|
||
## Tracking engine
|
||
|
||
**Primary: `opencv`** — a joint multi-marker *board* solve. Every detected crest
|
||
corner becomes a 3D world point (from scene anchors via `anchorWorldMatrix`), and
|
||
**one `solvePnP`** recovers the camera pose. This replaced v2's per-marker POS-IT
|
||
plus confidence-weighted fusion: stability now scales with marker spread instead
|
||
of degrading into fusion tuning, and there is no per-marker slerp base or area
|
||
bucketing.
|
||
|
||
**Fallback: `posit`** — the complete v2 pipeline (`detect` → `pose` → `fuse`),
|
||
retained for on-table A/B and as an automatic fallback if the OpenCV worker fails
|
||
to load. To be deleted once opencv is confirmed on the real town.
|
||
|
||
Engine is runtime-switchable from the dev HUD and persists in
|
||
`localStorage.nbx.engine`.
|
||
|
||
### Why OpenCV runs in a Web Worker
|
||
|
||
The `@techstark/opencv-js` build embeds ~8 MB of WASM as a base64 data URI inside
|
||
a 10.8 MB script. Parsing, base64-decoding and compiling that on the main thread
|
||
froze the page for seconds and triggered "page unresponsive" on mobile. In
|
||
`cv-worker.js` the whole cost — and every per-frame `solvePnP` — is off-thread.
|
||
|
||
Solves are **fire-and-forget, one in flight**: if a solve is still running the
|
||
frame is skipped rather than queued, so the main thread never blocks and latency
|
||
can't accumulate. The 1€ filter (main thread) absorbs the ~1 frame delay.
|
||
|
||
Pinned to `4.12.0-release.1`. Note for future work: this build lacks
|
||
`solvePnPGeneric` and `cornerSubPix`, which is why pose ambiguity is handled by
|
||
gating rather than by comparing both IPPE solutions.
|
||
|
||
## Detection: ROI first, full frame as fallback
|
||
|
||
The crests do not move; the phone does. Once a crest has been seen, the next
|
||
tick crops around where it was and decodes that, rather than rescanning 518k
|
||
pixels to re-find something that shifted by a few. Crops are taken from the
|
||
**native** video, so each is both smaller than the corresponding patch of a full
|
||
scan and higher resolution than it (`@1.3x` on a 1280 stream against a 960
|
||
canvas). Five typical crests come to ~40k pixels against 518k.
|
||
|
||
Mapping back into detection space is two scale changes and an origin shift per
|
||
crop, which is exactly the kind of arithmetic that fails silently and puts
|
||
ghosts a centimetre off. It is tested directly in `tools/test-roi.mjs`.
|
||
|
||
A full scan runs on first acquisition, when targets go stale, at least every
|
||
600ms, when there are more than 6 targets, and when the crops would cover more
|
||
than 55% of the frame. **If an ROI pass returns fewer markers than it had
|
||
targets, a full scan runs in the same tick** — dropping from 4 crests to 2 is
|
||
precisely what makes the pose ambiguous, so it is never allowed to happen
|
||
silently. Worst case is one wasted crop pass. Off: `localStorage.nbx.roi = 0`.
|
||
|
||
## Corner precision: sub-pixel refinement
|
||
|
||
js-aruco2 returns corners on the detection canvas lattice. At 960px a 60mm crest
|
||
spans ~140px, so one pixel of corner error is ~0.7% of the marker — millimetres
|
||
of ghost placement, re-rolled every frame.
|
||
|
||
`subpix.js` refines each quad by fitting its four border edges rather than by
|
||
looking at corners. Along each edge it scans perpendicular at ~12 points, takes
|
||
the **first moment of the gradient** at each (not the gradient peak), fits a line
|
||
through the crossings by total least squares with one outlier pass, and
|
||
intersects consecutive lines.
|
||
|
||
The centroid matters. Parabolic interpolation of the gradient peak — the obvious
|
||
method, and what a `cornerSubPix`-style operator reduces to — is biased by
|
||
sampling phase, because the gradient of an anti-aliased step is roughly
|
||
triangular. Measured against synthetic edges it carried up to 0.29px of bias
|
||
sharp and 0.75px defocused, i.e. **worse than the whole-pixel corners it was
|
||
meant to improve**. The gradient centroid is unbiased for any symmetric edge
|
||
profile: 0.018px on the same test. End to end on synthetic markers, RMS corner
|
||
error drops ~12x (`tools/test-subpix.mjs`).
|
||
|
||
`cv.cornerSubPix` was not an option: it is absent from the pinned build. Probed
|
||
directly rather than assumed — `solvePnPRefineLM` IS present, though it refines
|
||
a pose from corners and so cannot help here.
|
||
|
||
Refinement is all-or-nothing per marker and falls back to the raw corners if any
|
||
edge is too short, too flat, too diffuse, or if a corner would move more than
|
||
3px. A bad refinement is worse than none, because the pose gates are tuned
|
||
against honest corner noise. Off: `localStorage.nbx.subpix = 0`.
|
||
|
||
## Anchor conditioning
|
||
|
||
`geom.js` measures, once per scene load, the RMS distance of every anchor corner
|
||
from the best-fit plane through all of them, and the HUD shows it. Below 1.5cm
|
||
the layout is effectively coplanar and the mirrored pose is live — the physics
|
||
gates below become load-bearing rather than a safety net.
|
||
|
||
This exists because "the anchors are coplanar" is otherwise invisible: it looks
|
||
normal in the JSON and shows up days later as ghosts intermittently flipping
|
||
across the table. A layout re-saved by an editor that does not understand wall
|
||
anchors can silently become flat. Reference values: A3 rig with the tower ~3.5cm,
|
||
a town layout with one wall crest at 8cm ~2.7cm, any all-tabletop layout 0.0cm.
|
||
|
||
## Admin chrome
|
||
|
||
`public/admin/admin.js` holds a `TOOLS` registry — id, display name, group and
|
||
a one-line blurb for every admin page. The nav bar (`nav()`) and the hub
|
||
(`/admin/index.html`) are both generated from it, so a new page is registered
|
||
once and cannot appear in one and not the other. `tools/check-admin-nav.mjs`
|
||
checks the other direction: every `.html` in `public/admin` except the hub and
|
||
the login screen must be registered.
|
||
|
||
This replaced a hand-kept array of filenames, under which `testsheet.html`
|
||
existed but was listed nowhere, and `calibrate.html` was listed but never
|
||
rendered the bar — and never called `requireAuth()`, so in final mode it was
|
||
the one admin page reachable without the password.
|
||
|
||
Two pages cannot host the bar: `calibrate.html` is a full-bleed camera view and
|
||
`testsheet.html` must print edge to edge. They carry `chrome: false` and get a
|
||
link home instead — a floating pill (`backLink()`, styled by the separately
|
||
exported `BACK_CSS`) for calibrate, an inline link in the existing toolbar for
|
||
testsheet, where a floating pill would cover the Print button.
|
||
|
||
`nav()`'s root element stays `<header>`: `print.html` hides `header` on print
|
||
and `plan.html` hides its `#hdr` wrapper, and both rules depend on it.
|
||
|
||
## Camera model
|
||
|
||
Everything derives from one assumption — `HFOV_DEG` (default 60, override via
|
||
`localStorage.nbx.hfovDeg`) — **applied to the image's LONG axis**, because iOS
|
||
may hand back a portrait stream (720×1280). Keying it to `videoWidth` made the
|
||
focal length 1.78× too short.
|
||
|
||
```js
|
||
focalPxFor(w, h) = (max(w, h) / 2) / tan(HFOV_DEG / 2)
|
||
```
|
||
|
||
The same function feeds both the detector and the render camera, so they cannot
|
||
disagree. The render camera additionally accounts for `object-fit: cover`: only
|
||
part of the frame is on screen, so `camera3.fov` comes from the *visible* height,
|
||
and `_crop` records the visible fraction so on-screen diagnostics compare like
|
||
with like.
|
||
|
||
**Frame conversion** (validated against synthetic projections, error ~0):
|
||
solvePnP returns world→cvCamera `(R, t)`; cv camera is +X right, +Y down,
|
||
+Z forward. Camera position `C = -Rᵀt`; three.js world-from-camera rotation is
|
||
`Rᵀ · diag(1, -1, -1)`.
|
||
|
||
## Pose validity: the coplanar ambiguity
|
||
|
||
**A coplanar set of markers admits two poses that reproject almost equally well**
|
||
— the true one and a mirrored twin. With few crests visible the solver
|
||
intermittently returns the twin, which looks like ghosts snapping to the wrong
|
||
side of the table. Reprojection error cannot separate them: a flipped pose was
|
||
observed accepted at 1.3 px.
|
||
|
||
Two independent measures, in order of preference:
|
||
|
||
1. **Geometry (the cure).** Non-coplanar anchors make the twin mathematically
|
||
impossible. The A3 test rig provides this with a folded tower carrying crests
|
||
100 mm above the sheet; the real town should keep at least one raised or
|
||
wall-mounted crest for the same reason.
|
||
2. **Physics gates (the safety net),** in `cv-worker.js`: the camera must be
|
||
above the table, and cannot teleport 20 cm between frames. A rejected pose
|
||
leaves the previous one in place. An escape hatch after 12 consecutive
|
||
rejections prevents a bad reference locking tracking out. The HUD reports the
|
||
rejection reason.
|
||
|
||
Detection also **deduplicates ids within a frame**, keeping the largest quad:
|
||
two quads decoding to one id would hand solvePnP contradictory image points for
|
||
identical world points.
|
||
|
||
## Scene and state flow
|
||
|
||
- **Scene is fetched over HTTP at startup**, not awaited from the WebSocket.
|
||
Detection filters against scene anchor ids, so a slow or flapping socket used
|
||
to make every detection get silently discarded — indistinguishable on screen
|
||
from "the camera sees nothing". The WebSocket `scene` message now only
|
||
refreshes an already-working client.
|
||
- The WebSocket remains authoritative for **ghosts** (`spawn`, `despawn`,
|
||
`active`, `characters`) and clock sync (`ping`/`pong`).
|
||
- **Motion amounts are server-authoritative.** `playlist.js` sends explicit
|
||
`bobAmp`, `radius` and `speed` with every spawn record, so client-side defaults
|
||
never apply to a real ghost. Tune motion in the playlist config.
|
||
|
||
## Occlusion
|
||
|
||
Invisible depth-only meshes (`colorWrite: false`) let physical models hide
|
||
ghosts. Two shapes: boxes for buildings, and cylinders (`shape:'cylinder'` with
|
||
`radiusCm`/`heightCm`) for round props, where a box silhouette would wrongly hide
|
||
ghosts at the corners. Positions are floor-level; meshes are lifted by half their
|
||
height.
|
||
|
||
## Caching
|
||
|
||
App JS and HTML are served `no-cache` (ETag makes revalidation cheap); the 13 MB
|
||
`opencv.js` is `immutable, max-age=1yr`. A cache-busting query on the entry module
|
||
would not work — ES module imports fetch under their own URLs, so children would
|
||
stay stale. Note that an upstream openresty proxy has been seen overriding this
|
||
with a daily `expires`; that needs a vhost fix.
|