Architecture

Current as of 2026-09-23. Supersedes the v2 tracking design throughout.

Pipeline

camera video
  -> detection canvas (long side capped, ~20Hz)
  -> ArUco decode, maxHamming 0, filtered to scene anchor ids, deduped by id
  -> TRACKING ENGINE  ->  camera pose in world
  -> three.js render: server-driven ghosts + depth-only occluder meshes

Rendering runs every animation frame; detection and solving do not.

Tracking engine

Primary: opencv — a joint multi-marker board solve. Every detected crest corner becomes a 3D world point (from scene anchors via anchorWorldMatrix), and one solvePnP recovers the camera pose. This replaced v2's per-marker POS-IT plus confidence-weighted fusion: stability now scales with marker spread instead of degrading into fusion tuning, and there is no per-marker slerp base or area bucketing.

Fallback: posit — the complete v2 pipeline (detect → pose → fuse), retained for on-table A/B and as an automatic fallback if the OpenCV worker fails to load. To be deleted once opencv is confirmed on the real town.

Engine is runtime-switchable from the dev HUD and persists in localStorage.nbx.engine.

Why OpenCV runs in a Web Worker

The @techstark/opencv-js build embeds ~8 MB of WASM as a base64 data URI inside a 10.8 MB script. Parsing, base64-decoding and compiling that on the main thread froze the page for seconds and triggered "page unresponsive" on mobile. In cv-worker.js the whole cost — and every per-frame solvePnP — is off-thread.

Solves are fire-and-forget, one in flight: if a solve is still running the frame is skipped rather than queued, so the main thread never blocks and latency can't accumulate. The 1€ filter (main thread) absorbs the ~1 frame delay.

Pinned to 4.12.0-release.1. Note for future work: this build lacks solvePnPGeneric and cornerSubPix, which is why pose ambiguity is handled by gating rather than by comparing both IPPE solutions.

Camera model

Everything derives from one assumption — HFOV_DEG (default 60, override via localStorage.nbx.hfovDeg) — applied to the image's LONG axis, because iOS may hand back a portrait stream (720×1280). Keying it to videoWidth made the focal length 1.78× too short.

focalPxFor(w, h) = (max(w, h) / 2) / tan(HFOV_DEG / 2)

The same function feeds both the detector and the render camera, so they cannot disagree. The render camera additionally accounts for object-fit: cover: only part of the frame is on screen, so camera3.fov comes from the visible height, and _crop records the visible fraction so on-screen diagnostics compare like with like.

Frame conversion (validated against synthetic projections, error ~0): solvePnP returns world→cvCamera (R, t); cv camera is +X right, +Y down, +Z forward. Camera position C = -Rᵀt; three.js world-from-camera rotation is Rᵀ · diag(1, -1, -1).

Pose validity: the coplanar ambiguity

A coplanar set of markers admits two poses that reproject almost equally well — the true one and a mirrored twin. With few crests visible the solver intermittently returns the twin, which looks like ghosts snapping to the wrong side of the table. Reprojection error cannot separate them: a flipped pose was observed accepted at 1.3 px.

Two independent measures, in order of preference:

Geometry (the cure). Non-coplanar anchors make the twin mathematically impossible. The A3 test rig provides this with a folded tower carrying crests 100 mm above the sheet; the real town should keep at least one raised or wall-mounted crest for the same reason.
Physics gates (the safety net), in cv-worker.js: the camera must be above the table, and cannot teleport 20 cm between frames. A rejected pose leaves the previous one in place. An escape hatch after 12 consecutive rejections prevents a bad reference locking tracking out. The HUD reports the rejection reason.

Detection also deduplicates ids within a frame, keeping the largest quad: two quads decoding to one id would hand solvePnP contradictory image points for identical world points.

Scene and state flow

Scene is fetched over HTTP at startup, not awaited from the WebSocket. Detection filters against scene anchor ids, so a slow or flapping socket used to make every detection get silently discarded — indistinguishable on screen from "the camera sees nothing". The WebSocket scene message now only refreshes an already-working client.
The WebSocket remains authoritative for ghosts (spawn, despawn, active, characters) and clock sync (ping/pong).
Motion amounts are server-authoritative. playlist.js sends explicit bobAmp, radius and speed with every spawn record, so client-side defaults never apply to a real ghost. Tune motion in the playlist config.

Occlusion

Invisible depth-only meshes (colorWrite: false) let physical models hide ghosts. Two shapes: boxes for buildings, and cylinders (shape:'cylinder' with radiusCm/heightCm) for round props, where a box silhouette would wrongly hide ghosts at the corners. Positions are floor-level; meshes are lifted by half their height.

Caching

App JS and HTML are served no-cache (ETag makes revalidation cheap); the 13 MB opencv.js is immutable, max-age=1yr. A cache-busting query on the entry module would not work — ES module imports fetch under their own URLs, so children would stay stale. Note that an upstream openresty proxy has been seen overriding this with a daily expires; that needs a vhost fix.

5.5 KiB Raw Blame History Unescape Escape