# Architecture

Current as of 2026-09-23. Supersedes the v2 tracking design throughout.

## Pipeline

```
camera video
  -> ROI crops around last-seen crests, at NATIVE resolution   (steady state)
     or full detection canvas, long side capped                (acquire / every 600ms / on loss)
  -> ArUco decode, maxHamming 0
  -> sub-pixel corner refinement (gradient centroid per border edge)
  -> filtered to scene anchor ids, deduped by id
  -> TRACKING ENGINE  ->  camera pose in world
  -> three.js render: server-driven ghosts + depth-only occluder meshes
```

Rendering runs every animation frame; detection and solving do not.

## Tracking engine

**Primary: `opencv`** — a joint multi-marker *board* solve. Every detected crest
corner becomes a 3D world point (from scene anchors via `anchorWorldMatrix`), and
**one `solvePnP`** recovers the camera pose. This replaced v2's per-marker POS-IT
plus confidence-weighted fusion: stability now scales with marker spread instead
of degrading into fusion tuning, and there is no per-marker slerp base or area
bucketing.

**Fallback: `posit`** — the complete v2 pipeline (`detect` → `pose` → `fuse`),
retained for on-table A/B and as an automatic fallback if the OpenCV worker fails
to load. To be deleted once opencv is confirmed on the real town.

Engine is runtime-switchable from the dev HUD and persists in
`localStorage.nbx.engine`.

### Why OpenCV runs in a Web Worker

The `@techstark/opencv-js` build embeds ~8 MB of WASM as a base64 data URI inside
a 10.8 MB script. Parsing, base64-decoding and compiling that on the main thread
froze the page for seconds and triggered "page unresponsive" on mobile. In
`cv-worker.js` the whole cost — and every per-frame `solvePnP` — is off-thread.

Solves are **fire-and-forget, one in flight**: if a solve is still running the
frame is skipped rather than queued, so the main thread never blocks and latency
can't accumulate. The 1€ filter (main thread) absorbs the ~1 frame delay.

Pinned to `4.12.0-release.1`. Note for future work: this build lacks
`solvePnPGeneric` and `cornerSubPix`, which is why pose ambiguity is handled by
gating rather than by comparing both IPPE solutions.

## Detection: ROI first, full frame as fallback

The crests do not move; the phone does. Once a crest has been seen, the next
tick crops around where it was and decodes that, rather than rescanning 518k
pixels to re-find something that shifted by a few. Crops are taken from the
**native** video, so each is both smaller than the corresponding patch of a full
scan and higher resolution than it (`@1.3x` on a 1280 stream against a 960
canvas). Five typical crests come to ~40k pixels against 518k.

Mapping back into detection space is two scale changes and an origin shift per
crop, which is exactly the kind of arithmetic that fails silently and puts
ghosts a centimetre off. It is tested directly in `tools/test-roi.mjs`.

A full scan runs on first acquisition, when targets go stale, at least every
600ms, when there are more than 6 targets, and when the crops would cover more
than 55% of the frame. **If an ROI pass returns fewer markers than it had
targets, a full scan runs in the same tick** — dropping from 4 crests to 2 is
precisely what makes the pose ambiguous, so it is never allowed to happen
silently. Worst case is one wasted crop pass. Off: `localStorage.nbx.roi = 0`.

## Corner precision: sub-pixel refinement

js-aruco2 returns corners on the detection canvas lattice. At 960px a 60mm crest
spans ~140px, so one pixel of corner error is ~0.7% of the marker — millimetres
of ghost placement, re-rolled every frame.

`subpix.js` refines each quad by fitting its four border edges rather than by
looking at corners. Along each edge it scans perpendicular at ~12 points, takes
the **first moment of the gradient** at each (not the gradient peak), fits a line
through the crossings by total least squares with one outlier pass, and
intersects consecutive lines.

The centroid matters. Parabolic interpolation of the gradient peak — the obvious
method, and what a `cornerSubPix`-style operator reduces to — is biased by
sampling phase, because the gradient of an anti-aliased step is roughly
triangular. Measured against synthetic edges it carried up to 0.29px of bias
sharp and 0.75px defocused, i.e. **worse than the whole-pixel corners it was
meant to improve**. The gradient centroid is unbiased for any symmetric edge
profile: 0.018px on the same test. End to end on synthetic markers, RMS corner
error drops ~12x (`tools/test-subpix.mjs`).

`cv.cornerSubPix` was not an option: it is absent from the pinned build. Probed
directly rather than assumed — `solvePnPRefineLM` IS present, though it refines
a pose from corners and so cannot help here.

Refinement is all-or-nothing per marker and falls back to the raw corners if any
edge is too short, too flat, too diffuse, or if a corner would move more than
3px. A bad refinement is worse than none, because the pose gates are tuned
against honest corner noise. Off: `localStorage.nbx.subpix = 0`.

## Anchor conditioning

`geom.js` measures, once per scene load, the RMS distance of every anchor corner
from the best-fit plane through all of them, and the HUD shows it. Below 1.5cm
the layout is effectively coplanar and the mirrored pose is live — the physics
gates below become load-bearing rather than a safety net.

This exists because "the anchors are coplanar" is otherwise invisible: it looks
normal in the JSON and shows up days later as ghosts intermittently flipping
across the table. A layout re-saved by an editor that does not understand wall
anchors can silently become flat. Reference values: A3 rig with the tower ~3.5cm,
a town layout with one wall crest at 8cm ~2.7cm, any all-tabletop layout 0.0cm.

## Admin chrome

`public/admin/admin.js` holds a `TOOLS` registry — id, display name, group and
a one-line blurb for every admin page. The nav bar (`nav()`) and the hub
(`/admin/index.html`) are both generated from it, so a new page is registered
once and cannot appear in one and not the other. `tools/check-admin-nav.mjs`
checks the other direction: every `.html` in `public/admin` except the hub and
the login screen must be registered.

This replaced a hand-kept array of filenames, under which `testsheet.html`
existed but was listed nowhere, and `calibrate.html` was listed but never
rendered the bar — and never called `requireAuth()`, so in final mode it was
the one admin page reachable without the password.

Two pages cannot host the bar: `calibrate.html` is a full-bleed camera view and
`testsheet.html` must print edge to edge. They carry `chrome: false` and get a
link home instead — a floating pill (`backLink()`, styled by the separately
exported `BACK_CSS`) for calibrate, an inline link in the existing toolbar for
testsheet, where a floating pill would cover the Print button.

`nav()`'s root element stays `<header>`: `print.html` hides `header` on print
and `plan.html` hides its `#hdr` wrapper, and both rules depend on it.

## Camera model

Everything derives from one assumption — `HFOV_DEG` (default 60, override via
`localStorage.nbx.hfovDeg`) — **applied to the image's LONG axis**, because iOS
may hand back a portrait stream (720×1280). Keying it to `videoWidth` made the
focal length 1.78× too short.

```js
focalPxFor(w, h) = (max(w, h) / 2) / tan(HFOV_DEG / 2)
```

The same function feeds both the detector and the render camera, so they cannot
disagree. The render camera additionally accounts for `object-fit: cover`: only
part of the frame is on screen, so `camera3.fov` comes from the *visible* height,
and `_crop` records the visible fraction so on-screen diagnostics compare like
with like.

**Frame conversion** (validated against synthetic projections, error ~0):
solvePnP returns world→cvCamera `(R, t)`; cv camera is +X right, +Y down,
+Z forward. Camera position `C = -Rᵀt`; three.js world-from-camera rotation is
`Rᵀ · diag(1, -1, -1)`.

## Pose validity: the coplanar ambiguity

**A coplanar set of markers admits two poses that reproject almost equally well**
— the true one and a mirrored twin. With few crests visible the solver
intermittently returns the twin, which looks like ghosts snapping to the wrong
side of the table. Reprojection error cannot separate them: a flipped pose was
observed accepted at 1.3 px.

Two independent measures, in order of preference:

1. **Geometry (the cure).** Non-coplanar anchors make the twin mathematically
   impossible. The A3 test rig provides this with a folded tower carrying crests
   100 mm above the sheet; the real town should keep at least one raised or
   wall-mounted crest for the same reason.
2. **Physics gates (the safety net),** in `cv-worker.js`: the camera must be
   above the table, and cannot teleport 20 cm between frames. A rejected pose
   leaves the previous one in place. An escape hatch after 12 consecutive
   rejections prevents a bad reference locking tracking out. The HUD reports the
   rejection reason.

Detection also **deduplicates ids within a frame**, keeping the largest quad:
two quads decoding to one id would hand solvePnP contradictory image points for
identical world points.

## Scene and state flow

- **Scene is fetched over HTTP at startup**, not awaited from the WebSocket.
  Detection filters against scene anchor ids, so a slow or flapping socket used
  to make every detection get silently discarded — indistinguishable on screen
  from "the camera sees nothing". The WebSocket `scene` message now only
  refreshes an already-working client.
- The WebSocket remains authoritative for **ghosts** (`spawn`, `despawn`,
  `active`, `characters`) and clock sync (`ping`/`pong`).
- **Motion amounts are server-authoritative.** `playlist.js` sends explicit
  `bobAmp`, `radius` and `speed` with every spawn record, so client-side defaults
  never apply to a real ghost. Tune motion in the playlist config.

## Occlusion

Invisible depth-only meshes (`colorWrite: false`) let physical models hide
ghosts. Two shapes: boxes for buildings, and cylinders (`shape:'cylinder'` with
`radiusCm`/`heightCm`) for round props, where a box silhouette would wrongly hide
ghosts at the corners. Positions are floor-level; meshes are lifted by half their
height.

## Caching

App JS and HTML are served `no-cache` (ETag makes revalidation cheap); the 13 MB
`opencv.js` is `immutable, max-age=1yr`. A cache-busting query on the entry module
would not work — ES module imports fetch under their own URLs, so children would
stay stale. Note that an upstream openresty proxy has been seen overriding this
with a daily `expires`; that needs a vhost fix.