From 7947eb61a0fb200eee9ab8ececd126b39d7f5516 Mon Sep 17 00:00:00 2001 From: jessikitty Date: Wed, 23 Sep 2026 16:21:05 +1000 Subject: [PATCH] =?UTF-8?q?docs:=20architecture=20record=20=E2=80=94=20boa?= =?UTF-8?q?rd=20solve,=20worker=20offload,=20camera=20model,=20pose=20vali?= =?UTF-8?q?dity,=20scene=20flow?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .claude-context/01-architecture.md | 121 +++++++++++++++++++++++++++++ 1 file changed, 121 insertions(+) create mode 100644 .claude-context/01-architecture.md diff --git a/.claude-context/01-architecture.md b/.claude-context/01-architecture.md new file mode 100644 index 0000000..008e95e --- /dev/null +++ b/.claude-context/01-architecture.md @@ -0,0 +1,121 @@ +# Architecture + +Current as of 2026-09-23. Supersedes the v2 tracking design throughout. + +## Pipeline + +``` +camera video + -> detection canvas (long side capped, ~20Hz) + -> ArUco decode, maxHamming 0, filtered to scene anchor ids, deduped by id + -> TRACKING ENGINE -> camera pose in world + -> three.js render: server-driven ghosts + depth-only occluder meshes +``` + +Rendering runs every animation frame; detection and solving do not. + +## Tracking engine + +**Primary: `opencv`** — a joint multi-marker *board* solve. Every detected crest +corner becomes a 3D world point (from scene anchors via `anchorWorldMatrix`), and +**one `solvePnP`** recovers the camera pose. This replaced v2's per-marker POS-IT +plus confidence-weighted fusion: stability now scales with marker spread instead +of degrading into fusion tuning, and there is no per-marker slerp base or area +bucketing. + +**Fallback: `posit`** — the complete v2 pipeline (`detect` → `pose` → `fuse`), +retained for on-table A/B and as an automatic fallback if the OpenCV worker fails +to load. To be deleted once opencv is confirmed on the real town. + +Engine is runtime-switchable from the dev HUD and persists in +`localStorage.nbx.engine`. + +### Why OpenCV runs in a Web Worker + +The `@techstark/opencv-js` build embeds ~8 MB of WASM as a base64 data URI inside +a 10.8 MB script. Parsing, base64-decoding and compiling that on the main thread +froze the page for seconds and triggered "page unresponsive" on mobile. In +`cv-worker.js` the whole cost — and every per-frame `solvePnP` — is off-thread. + +Solves are **fire-and-forget, one in flight**: if a solve is still running the +frame is skipped rather than queued, so the main thread never blocks and latency +can't accumulate. The 1€ filter (main thread) absorbs the ~1 frame delay. + +Pinned to `4.12.0-release.1`. Note for future work: this build lacks +`solvePnPGeneric` and `cornerSubPix`, which is why pose ambiguity is handled by +gating rather than by comparing both IPPE solutions. + +## Camera model + +Everything derives from one assumption — `HFOV_DEG` (default 60, override via +`localStorage.nbx.hfovDeg`) — **applied to the image's LONG axis**, because iOS +may hand back a portrait stream (720×1280). Keying it to `videoWidth` made the +focal length 1.78× too short. + +```js +focalPxFor(w, h) = (max(w, h) / 2) / tan(HFOV_DEG / 2) +``` + +The same function feeds both the detector and the render camera, so they cannot +disagree. The render camera additionally accounts for `object-fit: cover`: only +part of the frame is on screen, so `camera3.fov` comes from the *visible* height, +and `_crop` records the visible fraction so on-screen diagnostics compare like +with like. + +**Frame conversion** (validated against synthetic projections, error ~0): +solvePnP returns world→cvCamera `(R, t)`; cv camera is +X right, +Y down, ++Z forward. Camera position `C = -Rᵀt`; three.js world-from-camera rotation is +`Rᵀ · diag(1, -1, -1)`. + +## Pose validity: the coplanar ambiguity + +**A coplanar set of markers admits two poses that reproject almost equally well** +— the true one and a mirrored twin. With few crests visible the solver +intermittently returns the twin, which looks like ghosts snapping to the wrong +side of the table. Reprojection error cannot separate them: a flipped pose was +observed accepted at 1.3 px. + +Two independent measures, in order of preference: + +1. **Geometry (the cure).** Non-coplanar anchors make the twin mathematically + impossible. The A3 test rig provides this with a folded tower carrying crests + 100 mm above the sheet; the real town should keep at least one raised or + wall-mounted crest for the same reason. +2. **Physics gates (the safety net),** in `cv-worker.js`: the camera must be + above the table, and cannot teleport 20 cm between frames. A rejected pose + leaves the previous one in place. An escape hatch after 12 consecutive + rejections prevents a bad reference locking tracking out. The HUD reports the + rejection reason. + +Detection also **deduplicates ids within a frame**, keeping the largest quad: +two quads decoding to one id would hand solvePnP contradictory image points for +identical world points. + +## Scene and state flow + +- **Scene is fetched over HTTP at startup**, not awaited from the WebSocket. + Detection filters against scene anchor ids, so a slow or flapping socket used + to make every detection get silently discarded — indistinguishable on screen + from "the camera sees nothing". The WebSocket `scene` message now only + refreshes an already-working client. +- The WebSocket remains authoritative for **ghosts** (`spawn`, `despawn`, + `active`, `characters`) and clock sync (`ping`/`pong`). +- **Motion amounts are server-authoritative.** `playlist.js` sends explicit + `bobAmp`, `radius` and `speed` with every spawn record, so client-side defaults + never apply to a real ghost. Tune motion in the playlist config. + +## Occlusion + +Invisible depth-only meshes (`colorWrite: false`) let physical models hide +ghosts. Two shapes: boxes for buildings, and cylinders (`shape:'cylinder'` with +`radiusCm`/`heightCm`) for round props, where a box silhouette would wrongly hide +ghosts at the corners. Positions are floor-level; meshes are lifted by half their +height. + +## Caching + +App JS and HTML are served `no-cache` (ETag makes revalidation cheap); the 13 MB +`opencv.js` is `immutable, max-age=1yr`. A cache-busting query on the entry module +would not work — ES module imports fetch under their own URLs, so children would +stay stale. Note that an upstream openresty proxy has been seen overriding this +with a daily `expires`; that needs a vhost fix.