AR Tracking Overhaul: Worker Offload, Camera Model, and Coplanar Pose Ambiguity

Date: 2026-09-23 Project: newbury-exhibit-v3 Status: ongoing Tags: ar, opencv, solvepnp, aruco, web-worker, camera-calibration, performance, caching, test-rig

Context

v3 replaced v2's per-marker POS-IT tracking with a joint multi-marker OpenCV board solve. On-device testing surfaced a cascade of distinct faults that all presented as "the ghost is in the wrong place or missing": a frozen page, ghosts rendering off-screen, ghosts vanishing entirely, and stale builds masking fixes. This session diagnosed each from HUD telemetry rather than guesswork, fixed them, and built a printable A3 test rig — including a foldable marker tower — to make tracking evaluation repeatable.

Key Decisions

Run OpenCV.js in a Web Worker (cv-worker.js), not on the main thread. The @techstark/opencv-js build embeds ~8 MB of WASM as a base64 data URI inside a 10.8 MB script, so parsing + base64 decode + WASM compile all happen synchronously — seconds of frozen UI and a "page unresponsive" prompt. Rejected: loading via Blob URL with a progress bar (made memory worse: three copies of 10.8 MB); extracting the WASM to a sibling file at Docker build time to enable streaming compile (mechanical string surgery on emscripten glue, too fragile).
Solves are fire-and-forget, one in flight at a time. If a solve is still running the frame is skipped rather than queued, so the main thread never waits and latency can't accumulate. The 1€ filter absorbs the ~1 frame delay.
FOV applies to the image's LONG axis, not videoWidth. iOS hands back portrait streams (720×1280); assuming the angle spanned videoWidth made the focal length 1280/720 = 1.78× too short. This was measured on device twice — the first "fix" (deriving vertical FOV from the video + object-fit: cover crop) was correct in principle but still keyed off videoWidth, so the same 1.78× error survived it.
Throttle detection to 20 Hz while rendering at full rate. Detection (getImageData + ArUco decode) dominates frame cost, and the crests don't move — only the phone does.
Cap the detection canvas LONG side, not its width. Fixing width at 960 tripled pixel count on portrait streams (960×1707 vs 960×540).
Kept detection resolution at 960 after checking the arithmetic. Mid-change I was raising it to 1280 on the theory that missed crests were a decode problem — then computed that 960 already yields ~23 px per marker cell, far above ArUco's needs. Raising it would have cost +78% CPU for probably nothing. Exposed as a knob instead so the theory can be tested rather than assumed.
Deduplicate marker ids within a frame, keeping the largest quad. Two quads decoding to the same id hand solvePnP two contradictory image points for one set of world points, collapsing the pose (observed: seen [0,2,0,4], reproj 5.4 px).
Gate poses on physics, not reprojection error. A coplanar point set has two poses that reproject almost equally well, so error can't separate them (a flipped pose was accepted at 1.3 px). Gates: camera must be above the sheet; camera can't teleport 20 cm between frames. Rejections keep the previous pose rather than applying a bad one. Escape hatch after 12 consecutive rejections so a bad reference can't lock tracking out.
The real cure for ambiguity is geometry: a marker tower. Gates suppress a symptom; non-coplanar anchors make the mirrored pose mathematically impossible. Rejected: wrapping a crest sheet around the 85 mm cylinder — ArUco needs planar facets. Chose a four-sided prism + top face so at least one crest is readable from any angle (worst case 45° incidence).
Tower face orientations derived, not guessed. Unrolling a prism advances arc position and the marker's printed-right axis together, so left-to-right on the flat net is yaw increasing: front(180), left(270), back(0), right(90). Verified against the real anchorWorldMatrix before shipping, because a wrong yaw yields plausible-but-wrong ghost placement — the hardest kind to debug.
Motion amounts are server-authoritative. playlist.js sends explicit bobAmp/radius/speed with every spawn, so client defaults never applied. A client-side "3 mm bob" fix was dead code until the server default changed.
Test rig is a single source of truth. The page that draws the printed sheets also builds the scene + playlist JSON from the same constants, so print and config cannot drift apart.

Technical Details

World frame: origin at front-left of the sheet; +X right, +Z away from viewer, +Y up; centimetres. Marker print sizes stay in mm (sizeMM), converted once at the solver boundary.

Camera model (exhibit.js):

const focalPxFor = (w, h) => (Math.max(w, h) / 2) / Math.tan(degToRad(HFOV_DEG) / 2);
// updateCameraIntrinsics(): scale = max(innerWidth/vw, innerHeight/vh)  // object-fit: cover
// visH = min(vh, innerHeight/scale);  camera3.fov = 2*atan((visH/2)/focal)
// _crop.{x,y} = visible fraction, used to put the aim-check px into viewport NDC

Verified rendered frustum matches true visible frame to <0.05° in portrait, landscape, and landscape-screen cases.

Worker protocol (public/js/ar/cv-worker.js):

{type:'init'} → {type:'ready'} | {type:'failed', msg}
{type:'solve', id, obj, img, n, W, H, f} → {type:'pose', id, ok, C, m3, reproj, n, why}
{type:'reset'} — clears warm start + gate state
obj/img are transferred Float64Arrays (no copy). Worker returns camera position C and the 3×3 world-from-camera rotation m3; the client builds the quaternion (worker never touches three.js).

Frame conversion (validated against synthetic projections, error ~0): solvePnP gives world→cvCamera (R,t); cv cam is +X right, +Y down, +Z forward. C = -Rᵀt; three.js world-from-camera = Rᵀ · diag(1,-1,-1).

Worker constants: MAX_REPROJ_PX = 4 (was 8), MIN_CAM_Y_CM = 1, MAX_JUMP_CM = 20 (allowance scales with elapsed time), JUMP_GRACE_MS = 500, MAX_REJECT_STREAK = 12, PREV_TTL = 1500.

localStorage knobs: nbx.engine (opencv|posit), nbx.hfovDeg (default 60), nbx.detectHz (20), nbx.detectLong (960).

Server motion defaults (server/playlist.js, these override the client): bobAmp 0.3, bobHz 0.4, wanderRadius 2.5, wanderVertical 0.4, pathSpeed 1.5 — previously 5–10, 60, 10, 8, sized for a large floor.

A3 test rig (/admin/testsheet.html), A3 landscape = 42.0 × 29.7 cm:

Page 1: crests 0–5 at (5,4) (21,4) (37,4) (5,25.7) (21,25.7) (37,25.7) cm, 60 mm, mount:'flat', yawDeg:180 (prints upright). 100 mm scale bar.
Page 2: tower net — 85 mm square × 160 mm tall, four 85 mm panels + 15 mm glue tab + 85 mm top flap; net 355 × 245 mm. Crests 6–9 on the sides at 100 mm height (mount:'wall', yaw 180/270/0/90, position pushed ±4.25 cm along the face normal), crest 10 on top (mount:'flat', yawDeg:180).
Anchor heights now 0 / 10 / 16 cm — genuinely non-coplanar.
Walker loop: 12–30 cm × 8.85–20.85 cm, 60 cm perimeter, ~40 s/lap at 1.5 cm/s.
Static ghost at (21, 22) cm, behind the tower from the front (occlusion test).

Cache policy (server/server.js): no-cache for .js/.html, immutable, max-age=31536000 for /vendor/opencv.js. A cache-busting query on the entry module would NOT have worked — ES module imports fetch under their own URLs, so detect.js etc. would stay stale.

Deploy: cd ~/newbury-exhibit-v3 && bash deploy.sh. Rollback to v2: cd ~/newbury-exhibit && docker compose up -d --build.

Code Changes

public/js/ar/cv-worker.js — new. OpenCV board solve off-thread; reusable Mats; warm-start + SQPNP/IPPE cold start; reprojection, camera-height and teleport gates with reported rejection reason.
public/js/ar/solve-cv.js — rewritten as a worker client; keeps anchor geometry and the 1€ filter on the main thread; retains the previous pose on rejection; exposes lastReject.
public/js/ar/detect.js — report raw vs scene-filtered detections; dedupe repeated ids keeping the largest quad; expose dupes.
public/js/exhibit.js — worker-backed solver; long-axis focal model; camera intrinsics from video + cover-crop; 20 Hz detection throttle; long-side canvas cap; scene fetched over HTTP so tracking survives a dead WebSocket; cylinder occluders; dev Motion button; aim-check and rejection telemetry in the HUD.
public/index.html — moved #dbg below the HUD row (it was covering the tracking chip and swallowing taps); added the #motionBtn control.
public/js/ghosts/behavior.js — idle float scaled to the display: 3 mm bob (was 6 cm), matching shimmer for wander/path; table-scale fallbacks.
server/playlist.js — server-side motion defaults rescaled and made tunable via behaviors.bobAmp/bobHz.
server/server.js — explicit cache policy for app code vs the WASM blob.
public/admin/testsheet.html — new. Two-page printable A3 test rig plus one-click scene + playlist load, all from shared constants.
docs/PAGES.md — new. Page index, API endpoints, HUD reading guide, and the gotchas worth not rediscovering.

Open Questions / Next Steps

Why do crests fail to decode? Observed 5 clearly visible, only 2 decoded. Resolution is NOT the explanation (~23 px/cell at 960). Candidates: strict maxHammingDistance: 0 rejecting marginal reads under blur/glare; adaptive thresholding on uneven lighting. Relaxing to hamming 1 is plausible now that a known-id whitelist, dedupe and pose gates exist, but it risks a misread landing on a whitelisted id. Test nbx.detectLong = 1280 first to settle it empirically.
Validate the tower on-device. Success criteria: markers count rises, REJ: stays quiet, and proj tracks px while circling the tower.
Tune nbx.hfovDeg. 60° is a guess; if a constant proj/px ratio remains, adjust and watch reproj minimise.
WebSocket flapping. HUD showed ws conn persisting with ghosts clearing on reconnect. Tracking no longer depends on it, but ghosts do. Likely the openresty vhost missing Upgrade/Connection headers on /ws.
Upstream proxy overrides cache headers. openresty replaces the app's no-cache with a daily expires, so deploys don't reach devices until it lapses. Needs a vhost change; private-tab testing is the workaround.
Layout editor doesn't understand cylinder occluders or wall anchors — it may mangle the tower layout on save.
Delete the posit engine once the opencv path is confirmed on the real town.

Artifacts

Repo: https://gitea.hideawaygaming.com.au/jessikitty/newbury-exhibit-v3
v2 (rollback path): https://gitea.hideawaygaming.com.au/jessikitty/newbury-exhibit-v2
docs/PAGES.md — page index, HUD guide, gotchas
public/admin/testsheet.html — the printable test rig
OpenCV.js build pinned at @techstark/opencv-js@4.12.0-release.1 (5.0.0 does not initialise; solvePnPGeneric and cornerSubPix are absent from this build, which is why ambiguity is handled by gating rather than by comparing both IPPE solutions)

11 KiB Raw Blame History Unescape Escape