diff --git a/.claude-context/sessions/2026-09-23-ar-tracking-overhaul.md b/.claude-context/sessions/2026-09-23-ar-tracking-overhaul.md new file mode 100644 index 0000000..7c68f1d --- /dev/null +++ b/.claude-context/sessions/2026-09-23-ar-tracking-overhaul.md @@ -0,0 +1,200 @@ +# AR Tracking Overhaul: Worker Offload, Camera Model, and Coplanar Pose Ambiguity + +**Date:** 2026-09-23 +**Project:** newbury-exhibit-v3 +**Status:** ongoing +**Tags:** ar, opencv, solvepnp, aruco, web-worker, camera-calibration, performance, caching, test-rig + +## Context + +v3 replaced v2's per-marker POS-IT tracking with a joint multi-marker OpenCV +board solve. On-device testing surfaced a cascade of distinct faults that all +presented as "the ghost is in the wrong place or missing": a frozen page, ghosts +rendering off-screen, ghosts vanishing entirely, and stale builds masking fixes. +This session diagnosed each from HUD telemetry rather than guesswork, fixed them, +and built a printable A3 test rig — including a foldable marker tower — to make +tracking evaluation repeatable. + +## Key Decisions + +- **Run OpenCV.js in a Web Worker** (`cv-worker.js`), not on the main thread. + The `@techstark/opencv-js` build embeds ~8 MB of WASM as a base64 data URI + inside a 10.8 MB script, so parsing + base64 decode + WASM compile all happen + synchronously — seconds of frozen UI and a "page unresponsive" prompt. + *Rejected:* loading via Blob URL with a progress bar (made memory worse: three + copies of 10.8 MB); extracting the WASM to a sibling file at Docker build time + to enable streaming compile (mechanical string surgery on emscripten glue, too + fragile). + +- **Solves are fire-and-forget, one in flight at a time.** If a solve is still + running the frame is skipped rather than queued, so the main thread never waits + and latency can't accumulate. The 1€ filter absorbs the ~1 frame delay. + +- **FOV applies to the image's LONG axis, not `videoWidth`.** iOS hands back + portrait streams (720×1280); assuming the angle spanned `videoWidth` made the + focal length 1280/720 = 1.78× too short. This was measured on device twice — + the first "fix" (deriving vertical FOV from the video + `object-fit: cover` + crop) was correct in principle but still keyed off `videoWidth`, so the same + 1.78× error survived it. + +- **Throttle detection to 20 Hz while rendering at full rate.** Detection + (`getImageData` + ArUco decode) dominates frame cost, and the crests don't + move — only the phone does. + +- **Cap the detection canvas LONG side, not its width.** Fixing width at 960 + tripled pixel count on portrait streams (960×1707 vs 960×540). + +- **Kept detection resolution at 960 after checking the arithmetic.** Mid-change + I was raising it to 1280 on the theory that missed crests were a decode + problem — then computed that 960 already yields ~23 px per marker cell, far + above ArUco's needs. Raising it would have cost +78% CPU for probably nothing. + Exposed as a knob instead so the theory can be tested rather than assumed. + +- **Deduplicate marker ids within a frame, keeping the largest quad.** Two quads + decoding to the same id hand solvePnP two contradictory image points for one + set of world points, collapsing the pose (observed: `seen [0,2,0,4]`, + reproj 5.4 px). + +- **Gate poses on physics, not reprojection error.** A coplanar point set has two + poses that reproject almost equally well, so error can't separate them (a + flipped pose was accepted at 1.3 px). Gates: camera must be above the sheet; + camera can't teleport 20 cm between frames. Rejections keep the *previous* + pose rather than applying a bad one. Escape hatch after 12 consecutive + rejections so a bad reference can't lock tracking out. + +- **The real cure for ambiguity is geometry: a marker tower.** Gates suppress a + symptom; non-coplanar anchors make the mirrored pose mathematically impossible. + *Rejected:* wrapping a crest sheet around the 85 mm cylinder — ArUco needs + planar facets. Chose a four-sided prism + top face so at least one crest is + readable from any angle (worst case 45° incidence). + +- **Tower face orientations derived, not guessed.** Unrolling a prism advances + arc position and the marker's printed-right axis together, so left-to-right on + the flat net is yaw *increasing*: front(180), left(270), back(0), right(90). + Verified against the real `anchorWorldMatrix` before shipping, because a wrong + yaw yields plausible-but-wrong ghost placement — the hardest kind to debug. + +- **Motion amounts are server-authoritative.** `playlist.js` sends explicit + `bobAmp`/`radius`/`speed` with every spawn, so client defaults never applied. + A client-side "3 mm bob" fix was dead code until the server default changed. + +- **Test rig is a single source of truth.** The page that draws the printed + sheets also builds the scene + playlist JSON from the same constants, so print + and config cannot drift apart. + +## Technical Details + +**World frame:** origin at front-left of the sheet; +X right, +Z away from +viewer, +Y up; centimetres. Marker print sizes stay in mm (`sizeMM`), converted +once at the solver boundary. + +**Camera model** (`exhibit.js`): +```js +const focalPxFor = (w, h) => (Math.max(w, h) / 2) / Math.tan(degToRad(HFOV_DEG) / 2); +// updateCameraIntrinsics(): scale = max(innerWidth/vw, innerHeight/vh) // object-fit: cover +// visH = min(vh, innerHeight/scale); camera3.fov = 2*atan((visH/2)/focal) +// _crop.{x,y} = visible fraction, used to put the aim-check px into viewport NDC +``` +Verified rendered frustum matches true visible frame to <0.05° in portrait, +landscape, and landscape-screen cases. + +**Worker protocol** (`public/js/ar/cv-worker.js`): +- `{type:'init'}` → `{type:'ready'}` | `{type:'failed', msg}` +- `{type:'solve', id, obj, img, n, W, H, f}` → `{type:'pose', id, ok, C, m3, reproj, n, why}` +- `{type:'reset'}` — clears warm start + gate state +- `obj`/`img` are transferred `Float64Array`s (no copy). Worker returns camera + position `C` and the 3×3 world-from-camera rotation `m3`; the client builds the + quaternion (worker never touches three.js). + +**Frame conversion** (validated against synthetic projections, error ~0): +solvePnP gives world→cvCamera `(R,t)`; cv cam is +X right, +Y down, +Z forward. +`C = -Rᵀt`; three.js world-from-camera = `Rᵀ · diag(1,-1,-1)`. + +**Worker constants:** `MAX_REPROJ_PX = 4` (was 8), `MIN_CAM_Y_CM = 1`, +`MAX_JUMP_CM = 20` (allowance scales with elapsed time), `JUMP_GRACE_MS = 500`, +`MAX_REJECT_STREAK = 12`, `PREV_TTL = 1500`. + +**localStorage knobs:** `nbx.engine` (opencv|posit), `nbx.hfovDeg` (default 60), +`nbx.detectHz` (20), `nbx.detectLong` (960). + +**Server motion defaults** (`server/playlist.js`, these override the client): +`bobAmp 0.3`, `bobHz 0.4`, `wanderRadius 2.5`, `wanderVertical 0.4`, +`pathSpeed 1.5` — previously 5–10, 60, 10, 8, sized for a large floor. + +**A3 test rig** (`/admin/testsheet.html`), A3 landscape = 42.0 × 29.7 cm: +- Page 1: crests 0–5 at (5,4) (21,4) (37,4) (5,25.7) (21,25.7) (37,25.7) cm, + 60 mm, `mount:'flat'`, `yawDeg:180` (prints upright). 100 mm scale bar. +- Page 2: tower net — 85 mm square × 160 mm tall, four 85 mm panels + 15 mm glue + tab + 85 mm top flap; net 355 × 245 mm. Crests 6–9 on the sides at 100 mm + height (`mount:'wall'`, yaw 180/270/0/90, position pushed ±4.25 cm along the + face normal), crest 10 on top (`mount:'flat'`, `yawDeg:180`). +- Anchor heights now 0 / 10 / 16 cm — genuinely non-coplanar. +- Walker loop: 12–30 cm × 8.85–20.85 cm, 60 cm perimeter, ~40 s/lap at 1.5 cm/s. +- Static ghost at (21, 22) cm, behind the tower from the front (occlusion test). + +**Cache policy** (`server/server.js`): `no-cache` for `.js`/`.html`, +`immutable, max-age=31536000` for `/vendor/opencv.js`. A cache-busting query on +the entry module would NOT have worked — ES module imports fetch under their own +URLs, so `detect.js` etc. would stay stale. + +**Deploy:** `cd ~/newbury-exhibit-v3 && bash deploy.sh`. Rollback to v2: +`cd ~/newbury-exhibit && docker compose up -d --build`. + +## Code Changes + +- `public/js/ar/cv-worker.js` — **new.** OpenCV board solve off-thread; reusable + Mats; warm-start + SQPNP/IPPE cold start; reprojection, camera-height and + teleport gates with reported rejection reason. +- `public/js/ar/solve-cv.js` — rewritten as a worker client; keeps anchor + geometry and the 1€ filter on the main thread; retains the previous pose on + rejection; exposes `lastReject`. +- `public/js/ar/detect.js` — report raw vs scene-filtered detections; dedupe + repeated ids keeping the largest quad; expose `dupes`. +- `public/js/exhibit.js` — worker-backed solver; long-axis focal model; camera + intrinsics from video + cover-crop; 20 Hz detection throttle; long-side canvas + cap; scene fetched over HTTP so tracking survives a dead WebSocket; cylinder + occluders; dev Motion button; aim-check and rejection telemetry in the HUD. +- `public/index.html` — moved `#dbg` below the HUD row (it was covering the + tracking chip and swallowing taps); added the `#motionBtn` control. +- `public/js/ghosts/behavior.js` — idle float scaled to the display: 3 mm bob + (was 6 cm), matching shimmer for wander/path; table-scale fallbacks. +- `server/playlist.js` — server-side motion defaults rescaled and made tunable + via `behaviors.bobAmp`/`bobHz`. +- `server/server.js` — explicit cache policy for app code vs the WASM blob. +- `public/admin/testsheet.html` — **new.** Two-page printable A3 test rig plus + one-click scene + playlist load, all from shared constants. +- `docs/PAGES.md` — **new.** Page index, API endpoints, HUD reading guide, and + the gotchas worth not rediscovering. + +## Open Questions / Next Steps + +- **Why do crests fail to decode?** Observed 5 clearly visible, only 2 decoded. + Resolution is NOT the explanation (~23 px/cell at 960). Candidates: strict + `maxHammingDistance: 0` rejecting marginal reads under blur/glare; adaptive + thresholding on uneven lighting. Relaxing to hamming 1 is plausible now that a + known-id whitelist, dedupe and pose gates exist, but it risks a misread landing + on a whitelisted id. Test `nbx.detectLong = 1280` first to settle it empirically. +- **Validate the tower on-device.** Success criteria: `markers` count rises, + `REJ:` stays quiet, and `proj` tracks `px` while circling the tower. +- **Tune `nbx.hfovDeg`.** 60° is a guess; if a constant `proj`/`px` ratio remains, + adjust and watch `reproj` minimise. +- **WebSocket flapping.** HUD showed `ws conn` persisting with ghosts clearing on + reconnect. Tracking no longer depends on it, but ghosts do. Likely the + openresty vhost missing `Upgrade`/`Connection` headers on `/ws`. +- **Upstream proxy overrides cache headers.** openresty replaces the app's + `no-cache` with a daily `expires`, so deploys don't reach devices until it + lapses. Needs a vhost change; private-tab testing is the workaround. +- **Layout editor doesn't understand cylinder occluders or wall anchors** — it + may mangle the tower layout on save. +- **Delete the posit engine** once the opencv path is confirmed on the real town. + +## Artifacts + +- Repo: `https://gitea.hideawaygaming.com.au/jessikitty/newbury-exhibit-v3` +- v2 (rollback path): `https://gitea.hideawaygaming.com.au/jessikitty/newbury-exhibit-v2` +- `docs/PAGES.md` — page index, HUD guide, gotchas +- `public/admin/testsheet.html` — the printable test rig +- OpenCV.js build pinned at `@techstark/opencv-js@4.12.0-release.1` + (5.0.0 does not initialise; `solvePnPGeneric` and `cornerSubPix` are absent + from this build, which is why ambiguity is handled by gating rather than by + comparing both IPPE solutions)