newbury-exhibit-v3/.claude-context/sessions/2026-09-23-ar-tracking-overhaul.md

# AR Tracking Overhaul: Worker Offload, Camera Model, and Coplanar Pose Ambiguity

**Date:** 2026-09-23
**Project:** newbury-exhibit-v3
**Status:** ongoing
**Tags:** ar, opencv, solvepnp, aruco, web-worker, camera-calibration, performance, caching, test-rig

## Context

v3 replaced v2's per-marker POS-IT tracking with a joint multi-marker OpenCV
board solve. On-device testing surfaced a cascade of distinct faults that all
presented as "the ghost is in the wrong place or missing": a frozen page, ghosts
rendering off-screen, ghosts vanishing entirely, and stale builds masking fixes.
This session diagnosed each from HUD telemetry rather than guesswork, fixed them,
and built a printable A3 test rig — including a foldable marker tower — to make
tracking evaluation repeatable.

## Key Decisions

- **Run OpenCV.js in a Web Worker** (`cv-worker.js`), not on the main thread.
  The `@techstark/opencv-js` build embeds ~8 MB of WASM as a base64 data URI
  inside a 10.8 MB script, so parsing + base64 decode + WASM compile all happen
  synchronously — seconds of frozen UI and a "page unresponsive" prompt.
  *Rejected:* loading via Blob URL with a progress bar (made memory worse: three
  copies of 10.8 MB); extracting the WASM to a sibling file at Docker build time
  to enable streaming compile (mechanical string surgery on emscripten glue, too
  fragile).

- **Solves are fire-and-forget, one in flight at a time.** If a solve is still
  running the frame is skipped rather than queued, so the main thread never waits
  and latency can't accumulate. The 1€ filter absorbs the ~1 frame delay.

- **FOV applies to the image's LONG axis, not `videoWidth`.** iOS hands back
  portrait streams (720×1280); assuming the angle spanned `videoWidth` made the
  focal length 1280/720 = 1.78× too short. This was measured on device twice —
  the first "fix" (deriving vertical FOV from the video + `object-fit: cover`
  crop) was correct in principle but still keyed off `videoWidth`, so the same
  1.78× error survived it.

- **Throttle detection to 20 Hz while rendering at full rate.** Detection
  (`getImageData` + ArUco decode) dominates frame cost, and the crests don't
  move — only the phone does.

- **Cap the detection canvas LONG side, not its width.** Fixing width at 960
  tripled pixel count on portrait streams (960×1707 vs 960×540).

- **Kept detection resolution at 960 after checking the arithmetic.** Mid-change
  I was raising it to 1280 on the theory that missed crests were a decode
  problem — then computed that 960 already yields ~23 px per marker cell, far
  above ArUco's needs. Raising it would have cost +78% CPU for probably nothing.
  Exposed as a knob instead so the theory can be tested rather than assumed.

- **Deduplicate marker ids within a frame, keeping the largest quad.** Two quads
  decoding to the same id hand solvePnP two contradictory image points for one
  set of world points, collapsing the pose (observed: `seen [0,2,0,4]`,
  reproj 5.4 px).

- **Gate poses on physics, not reprojection error.** A coplanar point set has two
  poses that reproject almost equally well, so error can't separate them (a
  flipped pose was accepted at 1.3 px). Gates: camera must be above the sheet;
  camera can't teleport 20 cm between frames. Rejections keep the *previous*
  pose rather than applying a bad one. Escape hatch after 12 consecutive
  rejections so a bad reference can't lock tracking out.

- **The real cure for ambiguity is geometry: a marker tower.** Gates suppress a
  symptom; non-coplanar anchors make the mirrored pose mathematically impossible.
  *Rejected:* wrapping a crest sheet around the 85 mm cylinder — ArUco needs
  planar facets. Chose a four-sided prism + top face so at least one crest is
  readable from any angle (worst case 45° incidence).

- **Tower face orientations derived, not guessed.** Unrolling a prism advances
  arc position and the marker's printed-right axis together, so left-to-right on
  the flat net is yaw *increasing*: front(180), left(270), back(0), right(90).
  Verified against the real `anchorWorldMatrix` before shipping, because a wrong
  yaw yields plausible-but-wrong ghost placement — the hardest kind to debug.

- **Motion amounts are server-authoritative.** `playlist.js` sends explicit
  `bobAmp`/`radius`/`speed` with every spawn, so client defaults never applied.
  A client-side "3 mm bob" fix was dead code until the server default changed.

- **Test rig is a single source of truth.** The page that draws the printed
  sheets also builds the scene + playlist JSON from the same constants, so print
  and config cannot drift apart.

## Technical Details

**World frame:** origin at front-left of the sheet; +X right, +Z away from
viewer, +Y up; centimetres. Marker print sizes stay in mm (`sizeMM`), converted
once at the solver boundary.

**Camera model** (`exhibit.js`):
```js
const focalPxFor = (w, h) => (Math.max(w, h) / 2) / Math.tan(degToRad(HFOV_DEG) / 2);
// updateCameraIntrinsics(): scale = max(innerWidth/vw, innerHeight/vh)  // object-fit: cover
// visH = min(vh, innerHeight/scale);  camera3.fov = 2*atan((visH/2)/focal)
// _crop.{x,y} = visible fraction, used to put the aim-check px into viewport NDC
```
Verified rendered frustum matches true visible frame to <0.05° in portrait,
landscape, and landscape-screen cases.

**Worker protocol** (`public/js/ar/cv-worker.js`):
- `{type:'init'}` → `{type:'ready'}` | `{type:'failed', msg}`
- `{type:'solve', id, obj, img, n, W, H, f}` → `{type:'pose', id, ok, C, m3, reproj, n, why}`
- `{type:'reset'}` — clears warm start + gate state
- `obj`/`img` are transferred `Float64Array`s (no copy). Worker returns camera
  position `C` and the 3×3 world-from-camera rotation `m3`; the client builds the
  quaternion (worker never touches three.js).

**Frame conversion** (validated against synthetic projections, error ~0):
solvePnP gives world→cvCamera `(R,t)`; cv cam is +X right, +Y down, +Z forward.
`C = -Rᵀt`; three.js world-from-camera = `Rᵀ · diag(1,-1,-1)`.

**Worker constants:** `MAX_REPROJ_PX = 4` (was 8), `MIN_CAM_Y_CM = 1`,
`MAX_JUMP_CM = 20` (allowance scales with elapsed time), `JUMP_GRACE_MS = 500`,
`MAX_REJECT_STREAK = 12`, `PREV_TTL = 1500`.

**localStorage knobs:** `nbx.engine` (opencv|posit), `nbx.hfovDeg` (default 60),
`nbx.detectHz` (20), `nbx.detectLong` (960).

**Server motion defaults** (`server/playlist.js`, these override the client):
`bobAmp 0.3`, `bobHz 0.4`, `wanderRadius 2.5`, `wanderVertical 0.4`,
`pathSpeed 1.5` — previously 5–10, 60, 10, 8, sized for a large floor.

**A3 test rig** (`/admin/testsheet.html`), A3 landscape = 42.0 × 29.7 cm:
- Page 1: crests 0–5 at (5,4) (21,4) (37,4) (5,25.7) (21,25.7) (37,25.7) cm,
  60 mm, `mount:'flat'`, `yawDeg:180` (prints upright). 100 mm scale bar.
- Page 2: tower net — 85 mm square × 160 mm tall, four 85 mm panels + 15 mm glue
  tab + 85 mm top flap; net 355 × 245 mm. Crests 6–9 on the sides at 100 mm
  height (`mount:'wall'`, yaw 180/270/0/90, position pushed ±4.25 cm along the
  face normal), crest 10 on top (`mount:'flat'`, `yawDeg:180`).
- Anchor heights now 0 / 10 / 16 cm — genuinely non-coplanar.
- Walker loop: 12–30 cm × 8.85–20.85 cm, 60 cm perimeter, ~40 s/lap at 1.5 cm/s.
- Static ghost at (21, 22) cm, behind the tower from the front (occlusion test).

**Cache policy** (`server/server.js`): `no-cache` for `.js`/`.html`,
`immutable, max-age=31536000` for `/vendor/opencv.js`. A cache-busting query on
the entry module would NOT have worked — ES module imports fetch under their own
URLs, so `detect.js` etc. would stay stale.

**Deploy:** `cd ~/newbury-exhibit-v3 && bash deploy.sh`. Rollback to v2:
`cd ~/newbury-exhibit && docker compose up -d --build`.

## Code Changes

- `public/js/ar/cv-worker.js` — **new.** OpenCV board solve off-thread; reusable
  Mats; warm-start + SQPNP/IPPE cold start; reprojection, camera-height and
  teleport gates with reported rejection reason.
- `public/js/ar/solve-cv.js` — rewritten as a worker client; keeps anchor
  geometry and the 1€ filter on the main thread; retains the previous pose on
  rejection; exposes `lastReject`.
- `public/js/ar/detect.js` — report raw vs scene-filtered detections; dedupe
  repeated ids keeping the largest quad; expose `dupes`.
- `public/js/exhibit.js` — worker-backed solver; long-axis focal model; camera
  intrinsics from video + cover-crop; 20 Hz detection throttle; long-side canvas
  cap; scene fetched over HTTP so tracking survives a dead WebSocket; cylinder
  occluders; dev Motion button; aim-check and rejection telemetry in the HUD.
- `public/index.html` — moved `#dbg` below the HUD row (it was covering the
  tracking chip and swallowing taps); added the `#motionBtn` control.
- `public/js/ghosts/behavior.js` — idle float scaled to the display: 3 mm bob
  (was 6 cm), matching shimmer for wander/path; table-scale fallbacks.
- `server/playlist.js` — server-side motion defaults rescaled and made tunable
  via `behaviors.bobAmp`/`bobHz`.
- `server/server.js` — explicit cache policy for app code vs the WASM blob.
- `public/admin/testsheet.html` — **new.** Two-page printable A3 test rig plus
  one-click scene + playlist load, all from shared constants.
- `docs/PAGES.md` — **new.** Page index, API endpoints, HUD reading guide, and
  the gotchas worth not rediscovering.

## Open Questions / Next Steps

- **Why do crests fail to decode?** Observed 5 clearly visible, only 2 decoded.
  Resolution is NOT the explanation (~23 px/cell at 960). Candidates: strict
  `maxHammingDistance: 0` rejecting marginal reads under blur/glare; adaptive
  thresholding on uneven lighting. Relaxing to hamming 1 is plausible now that a
  known-id whitelist, dedupe and pose gates exist, but it risks a misread landing
  on a whitelisted id. Test `nbx.detectLong = 1280` first to settle it empirically.
- **Validate the tower on-device.** Success criteria: `markers` count rises,
  `REJ:` stays quiet, and `proj` tracks `px` while circling the tower.
- **Tune `nbx.hfovDeg`.** 60° is a guess; if a constant `proj`/`px` ratio remains,
  adjust and watch `reproj` minimise.
- **WebSocket flapping.** HUD showed `ws conn` persisting with ghosts clearing on
  reconnect. Tracking no longer depends on it, but ghosts do. Likely the
  openresty vhost missing `Upgrade`/`Connection` headers on `/ws`.
- **Upstream proxy overrides cache headers.** openresty replaces the app's
  `no-cache` with a daily `expires`, so deploys don't reach devices until it
  lapses. Needs a vhost change; private-tab testing is the workaround.
- **Layout editor doesn't understand cylinder occluders or wall anchors** — it
  may mangle the tower layout on save.
- **Delete the posit engine** once the opencv path is confirmed on the real town.

## Artifacts

- Repo: `https://gitea.hideawaygaming.com.au/jessikitty/newbury-exhibit-v3`
- v2 (rollback path): `https://gitea.hideawaygaming.com.au/jessikitty/newbury-exhibit-v2`
- `docs/PAGES.md` — page index, HUD guide, gotchas
- `public/admin/testsheet.html` — the printable test rig
- OpenCV.js build pinned at `@techstark/opencv-js@4.12.0-release.1`
  (5.0.0 does not initialise; `solvePnPGeneric` and `cornerSubPix` are absent
  from this build, which is why ambiguity is handled by gating rather than by
  comparing both IPPE solutions)