# Architecture

Current as of 2026-09-23. Supersedes the v2 tracking design throughout.

## Pipeline

```
camera video
  -> detection canvas (long side capped, ~20Hz)
  -> ArUco decode, maxHamming 0, filtered to scene anchor ids, deduped by id
  -> TRACKING ENGINE  ->  camera pose in world
  -> three.js render: server-driven ghosts + depth-only occluder meshes
```

Rendering runs every animation frame; detection and solving do not.

## Tracking engine

**Primary: `opencv`** — a joint multi-marker *board* solve. Every detected crest
corner becomes a 3D world point (from scene anchors via `anchorWorldMatrix`), and
**one `solvePnP`** recovers the camera pose. This replaced v2's per-marker POS-IT
plus confidence-weighted fusion: stability now scales with marker spread instead
of degrading into fusion tuning, and there is no per-marker slerp base or area
bucketing.

**Fallback: `posit`** — the complete v2 pipeline (`detect` → `pose` → `fuse`),
retained for on-table A/B and as an automatic fallback if the OpenCV worker fails
to load. To be deleted once opencv is confirmed on the real town.

Engine is runtime-switchable from the dev HUD and persists in
`localStorage.nbx.engine`.

### Why OpenCV runs in a Web Worker

The `@techstark/opencv-js` build embeds ~8 MB of WASM as a base64 data URI inside
a 10.8 MB script. Parsing, base64-decoding and compiling that on the main thread
froze the page for seconds and triggered "page unresponsive" on mobile. In
`cv-worker.js` the whole cost — and every per-frame `solvePnP` — is off-thread.

Solves are **fire-and-forget, one in flight**: if a solve is still running the
frame is skipped rather than queued, so the main thread never blocks and latency
can't accumulate. The 1€ filter (main thread) absorbs the ~1 frame delay.

Pinned to `4.12.0-release.1`. Note for future work: this build lacks
`solvePnPGeneric` and `cornerSubPix`, which is why pose ambiguity is handled by
gating rather than by comparing both IPPE solutions.

## Camera model

Everything derives from one assumption — `HFOV_DEG` (default 60, override via
`localStorage.nbx.hfovDeg`) — **applied to the image's LONG axis**, because iOS
may hand back a portrait stream (720×1280). Keying it to `videoWidth` made the
focal length 1.78× too short.

```js
focalPxFor(w, h) = (max(w, h) / 2) / tan(HFOV_DEG / 2)
```

The same function feeds both the detector and the render camera, so they cannot
disagree. The render camera additionally accounts for `object-fit: cover`: only
part of the frame is on screen, so `camera3.fov` comes from the *visible* height,
and `_crop` records the visible fraction so on-screen diagnostics compare like
with like.

**Frame conversion** (validated against synthetic projections, error ~0):
solvePnP returns world→cvCamera `(R, t)`; cv camera is +X right, +Y down,
+Z forward. Camera position `C = -Rᵀt`; three.js world-from-camera rotation is
`Rᵀ · diag(1, -1, -1)`.

## Pose validity: the coplanar ambiguity

**A coplanar set of markers admits two poses that reproject almost equally well**
— the true one and a mirrored twin. With few crests visible the solver
intermittently returns the twin, which looks like ghosts snapping to the wrong
side of the table. Reprojection error cannot separate them: a flipped pose was
observed accepted at 1.3 px.

Two independent measures, in order of preference:

1. **Geometry (the cure).** Non-coplanar anchors make the twin mathematically
   impossible. The A3 test rig provides this with a folded tower carrying crests
   100 mm above the sheet; the real town should keep at least one raised or
   wall-mounted crest for the same reason.
2. **Physics gates (the safety net),** in `cv-worker.js`: the camera must be
   above the table, and cannot teleport 20 cm between frames. A rejected pose
   leaves the previous one in place. An escape hatch after 12 consecutive
   rejections prevents a bad reference locking tracking out. The HUD reports the
   rejection reason.

Detection also **deduplicates ids within a frame**, keeping the largest quad:
two quads decoding to one id would hand solvePnP contradictory image points for
identical world points.

## Scene and state flow

- **Scene is fetched over HTTP at startup**, not awaited from the WebSocket.
  Detection filters against scene anchor ids, so a slow or flapping socket used
  to make every detection get silently discarded — indistinguishable on screen
  from "the camera sees nothing". The WebSocket `scene` message now only
  refreshes an already-working client.
- The WebSocket remains authoritative for **ghosts** (`spawn`, `despawn`,
  `active`, `characters`) and clock sync (`ping`/`pong`).
- **Motion amounts are server-authoritative.** `playlist.js` sends explicit
  `bobAmp`, `radius` and `speed` with every spawn record, so client-side defaults
  never apply to a real ghost. Tune motion in the playlist config.

## Occlusion

Invisible depth-only meshes (`colorWrite: false`) let physical models hide
ghosts. Two shapes: boxes for buildings, and cylinders (`shape:'cylinder'` with
`radiusCm`/`heightCm`) for round props, where a box silhouette would wrongly hide
ghosts at the corners. Positions are floor-level; meshes are lifted by half their
height.

## Caching

App JS and HTML are served `no-cache` (ETag makes revalidation cheap); the 13 MB
`opencv.js` is `immutable, max-age=1yr`. A cache-busting query on the entry module
would not work — ES module imports fetch under their own URLs, so children would
stay stale. Note that an upstream openresty proxy has been seen overriding this
with a daily `expires`; that needs a vhost fix.