122 lines
5.5 KiB
Markdown
122 lines
5.5 KiB
Markdown
# Architecture
|
||
|
||
Current as of 2026-09-23. Supersedes the v2 tracking design throughout.
|
||
|
||
## Pipeline
|
||
|
||
```
|
||
camera video
|
||
-> detection canvas (long side capped, ~20Hz)
|
||
-> ArUco decode, maxHamming 0, filtered to scene anchor ids, deduped by id
|
||
-> TRACKING ENGINE -> camera pose in world
|
||
-> three.js render: server-driven ghosts + depth-only occluder meshes
|
||
```
|
||
|
||
Rendering runs every animation frame; detection and solving do not.
|
||
|
||
## Tracking engine
|
||
|
||
**Primary: `opencv`** — a joint multi-marker *board* solve. Every detected crest
|
||
corner becomes a 3D world point (from scene anchors via `anchorWorldMatrix`), and
|
||
**one `solvePnP`** recovers the camera pose. This replaced v2's per-marker POS-IT
|
||
plus confidence-weighted fusion: stability now scales with marker spread instead
|
||
of degrading into fusion tuning, and there is no per-marker slerp base or area
|
||
bucketing.
|
||
|
||
**Fallback: `posit`** — the complete v2 pipeline (`detect` → `pose` → `fuse`),
|
||
retained for on-table A/B and as an automatic fallback if the OpenCV worker fails
|
||
to load. To be deleted once opencv is confirmed on the real town.
|
||
|
||
Engine is runtime-switchable from the dev HUD and persists in
|
||
`localStorage.nbx.engine`.
|
||
|
||
### Why OpenCV runs in a Web Worker
|
||
|
||
The `@techstark/opencv-js` build embeds ~8 MB of WASM as a base64 data URI inside
|
||
a 10.8 MB script. Parsing, base64-decoding and compiling that on the main thread
|
||
froze the page for seconds and triggered "page unresponsive" on mobile. In
|
||
`cv-worker.js` the whole cost — and every per-frame `solvePnP` — is off-thread.
|
||
|
||
Solves are **fire-and-forget, one in flight**: if a solve is still running the
|
||
frame is skipped rather than queued, so the main thread never blocks and latency
|
||
can't accumulate. The 1€ filter (main thread) absorbs the ~1 frame delay.
|
||
|
||
Pinned to `4.12.0-release.1`. Note for future work: this build lacks
|
||
`solvePnPGeneric` and `cornerSubPix`, which is why pose ambiguity is handled by
|
||
gating rather than by comparing both IPPE solutions.
|
||
|
||
## Camera model
|
||
|
||
Everything derives from one assumption — `HFOV_DEG` (default 60, override via
|
||
`localStorage.nbx.hfovDeg`) — **applied to the image's LONG axis**, because iOS
|
||
may hand back a portrait stream (720×1280). Keying it to `videoWidth` made the
|
||
focal length 1.78× too short.
|
||
|
||
```js
|
||
focalPxFor(w, h) = (max(w, h) / 2) / tan(HFOV_DEG / 2)
|
||
```
|
||
|
||
The same function feeds both the detector and the render camera, so they cannot
|
||
disagree. The render camera additionally accounts for `object-fit: cover`: only
|
||
part of the frame is on screen, so `camera3.fov` comes from the *visible* height,
|
||
and `_crop` records the visible fraction so on-screen diagnostics compare like
|
||
with like.
|
||
|
||
**Frame conversion** (validated against synthetic projections, error ~0):
|
||
solvePnP returns world→cvCamera `(R, t)`; cv camera is +X right, +Y down,
|
||
+Z forward. Camera position `C = -Rᵀt`; three.js world-from-camera rotation is
|
||
`Rᵀ · diag(1, -1, -1)`.
|
||
|
||
## Pose validity: the coplanar ambiguity
|
||
|
||
**A coplanar set of markers admits two poses that reproject almost equally well**
|
||
— the true one and a mirrored twin. With few crests visible the solver
|
||
intermittently returns the twin, which looks like ghosts snapping to the wrong
|
||
side of the table. Reprojection error cannot separate them: a flipped pose was
|
||
observed accepted at 1.3 px.
|
||
|
||
Two independent measures, in order of preference:
|
||
|
||
1. **Geometry (the cure).** Non-coplanar anchors make the twin mathematically
|
||
impossible. The A3 test rig provides this with a folded tower carrying crests
|
||
100 mm above the sheet; the real town should keep at least one raised or
|
||
wall-mounted crest for the same reason.
|
||
2. **Physics gates (the safety net),** in `cv-worker.js`: the camera must be
|
||
above the table, and cannot teleport 20 cm between frames. A rejected pose
|
||
leaves the previous one in place. An escape hatch after 12 consecutive
|
||
rejections prevents a bad reference locking tracking out. The HUD reports the
|
||
rejection reason.
|
||
|
||
Detection also **deduplicates ids within a frame**, keeping the largest quad:
|
||
two quads decoding to one id would hand solvePnP contradictory image points for
|
||
identical world points.
|
||
|
||
## Scene and state flow
|
||
|
||
- **Scene is fetched over HTTP at startup**, not awaited from the WebSocket.
|
||
Detection filters against scene anchor ids, so a slow or flapping socket used
|
||
to make every detection get silently discarded — indistinguishable on screen
|
||
from "the camera sees nothing". The WebSocket `scene` message now only
|
||
refreshes an already-working client.
|
||
- The WebSocket remains authoritative for **ghosts** (`spawn`, `despawn`,
|
||
`active`, `characters`) and clock sync (`ping`/`pong`).
|
||
- **Motion amounts are server-authoritative.** `playlist.js` sends explicit
|
||
`bobAmp`, `radius` and `speed` with every spawn record, so client-side defaults
|
||
never apply to a real ghost. Tune motion in the playlist config.
|
||
|
||
## Occlusion
|
||
|
||
Invisible depth-only meshes (`colorWrite: false`) let physical models hide
|
||
ghosts. Two shapes: boxes for buildings, and cylinders (`shape:'cylinder'` with
|
||
`radiusCm`/`heightCm`) for round props, where a box silhouette would wrongly hide
|
||
ghosts at the corners. Positions are floor-level; meshes are lifted by half their
|
||
height.
|
||
|
||
## Caching
|
||
|
||
App JS and HTML are served `no-cache` (ETag makes revalidation cheap); the 13 MB
|
||
`opencv.js` is `immutable, max-age=1yr`. A cache-busting query on the entry module
|
||
would not work — ES module imports fetch under their own URLs, so children would
|
||
stay stale. Note that an upstream openresty proxy has been seen overriding this
|
||
with a daily `expires`; that needs a vhost fix.
|