newbury-exhibit-v3/.claude-context/01-architecture.md

Architecture

Current as of 2026-09-23. Supersedes the v2 tracking design throughout.

Pipeline

camera video
  -> ROI crops around last-seen crests, at NATIVE resolution   (steady state)
     or full detection canvas, long side capped                (acquire / every 600ms / on loss)
  -> ArUco decode, maxHamming 0
  -> sub-pixel corner refinement (gradient centroid per border edge)
  -> filtered to scene anchor ids, deduped by id
  -> TRACKING ENGINE  ->  camera pose in world
  -> three.js render: server-driven ghosts + depth-only occluder meshes

Rendering runs every animation frame; detection and solving do not.

Tracking engine

Primary: opencv — a joint multi-marker board solve. Every detected crest corner becomes a 3D world point (from scene anchors via anchorWorldMatrix), and one solvePnP recovers the camera pose. This replaced v2's per-marker POS-IT plus confidence-weighted fusion: stability now scales with marker spread instead of degrading into fusion tuning, and there is no per-marker slerp base or area bucketing.

Fallback: posit — the complete v2 pipeline (detect → pose → fuse), retained for on-table A/B and as an automatic fallback if the OpenCV worker fails to load. To be deleted once opencv is confirmed on the real town.

Engine is runtime-switchable from the dev HUD and persists in localStorage.nbx.engine.

Why OpenCV runs in a Web Worker

The @techstark/opencv-js build embeds ~8 MB of WASM as a base64 data URI inside a 10.8 MB script. Parsing, base64-decoding and compiling that on the main thread froze the page for seconds and triggered "page unresponsive" on mobile. In cv-worker.js the whole cost — and every per-frame solvePnP — is off-thread.

Solves are fire-and-forget, one in flight: if a solve is still running the frame is skipped rather than queued, so the main thread never blocks and latency can't accumulate. The 1€ filter (main thread) absorbs the ~1 frame delay.

Pinned to 4.12.0-release.1. Note for future work: this build lacks solvePnPGeneric and cornerSubPix, which is why pose ambiguity is handled by gating rather than by comparing both IPPE solutions.

Detection: ROI first, full frame as fallback

The crests do not move; the phone does. Once a crest has been seen, the next tick crops around where it was and decodes that, rather than rescanning 518k pixels to re-find something that shifted by a few. Crops are taken from the native video, so each is both smaller than the corresponding patch of a full scan and higher resolution than it (@1.3x on a 1280 stream against a 960 canvas). Five typical crests come to ~40k pixels against 518k.

Mapping back into detection space is two scale changes and an origin shift per crop, which is exactly the kind of arithmetic that fails silently and puts ghosts a centimetre off. It is tested directly in tools/test-roi.mjs.

A full scan runs on first acquisition, when targets go stale, at least every 600ms, when there are more than 6 targets, and when the crops would cover more than 55% of the frame. If an ROI pass returns fewer markers than it had targets, a full scan runs in the same tick — dropping from 4 crests to 2 is precisely what makes the pose ambiguous, so it is never allowed to happen silently. Worst case is one wasted crop pass. Off: localStorage.nbx.roi = 0.

Corner precision: sub-pixel refinement

js-aruco2 returns corners on the detection canvas lattice. At 960px a 60mm crest spans ~140px, so one pixel of corner error is ~0.7% of the marker — millimetres of ghost placement, re-rolled every frame.

subpix.js refines each quad by fitting its four border edges rather than by looking at corners. Along each edge it scans perpendicular at ~12 points, takes the first moment of the gradient at each (not the gradient peak), fits a line through the crossings by total least squares with one outlier pass, and intersects consecutive lines.

The centroid matters. Parabolic interpolation of the gradient peak — the obvious method, and what a cornerSubPix-style operator reduces to — is biased by sampling phase, because the gradient of an anti-aliased step is roughly triangular. Measured against synthetic edges it carried up to 0.29px of bias sharp and 0.75px defocused, i.e. worse than the whole-pixel corners it was meant to improve. The gradient centroid is unbiased for any symmetric edge profile: 0.018px on the same test. End to end on synthetic markers, RMS corner error drops ~12x (tools/test-subpix.mjs).

cv.cornerSubPix was not an option: it is absent from the pinned build. Probed directly rather than assumed — solvePnPRefineLM IS present, though it refines a pose from corners and so cannot help here.

Refinement is all-or-nothing per marker and falls back to the raw corners if any edge is too short, too flat, too diffuse, or if a corner would move more than 3px. A bad refinement is worse than none, because the pose gates are tuned against honest corner noise. Off: localStorage.nbx.subpix = 0.

Anchor conditioning

geom.js measures, once per scene load, the RMS distance of every anchor corner from the best-fit plane through all of them, and the HUD shows it. Below 1.5cm the layout is effectively coplanar and the mirrored pose is live — the physics gates below become load-bearing rather than a safety net.

This exists because "the anchors are coplanar" is otherwise invisible: it looks normal in the JSON and shows up days later as ghosts intermittently flipping across the table. A layout re-saved by an editor that does not understand wall anchors can silently become flat. Reference values: A3 rig with the tower ~3.5cm, a town layout with one wall crest at 8cm ~2.7cm, any all-tabletop layout 0.0cm.

Admin chrome

public/admin/admin.js holds a TOOLS registry — id, display name, group and a one-line blurb for every admin page. The nav bar (nav()) and the hub (/admin/index.html) are both generated from it, so a new page is registered once and cannot appear in one and not the other. tools/check-admin-nav.mjs checks the other direction: every .html in public/admin except the hub and the login screen must be registered.

This replaced a hand-kept array of filenames, under which testsheet.html existed but was listed nowhere, and calibrate.html was listed but never rendered the bar — and never called requireAuth(), so in final mode it was the one admin page reachable without the password.

Two pages cannot host the bar: calibrate.html is a full-bleed camera view and testsheet.html must print edge to edge. They carry chrome: false and get a link home instead — a floating pill (backLink(), styled by the separately exported BACK_CSS) for calibrate, an inline link in the existing toolbar for testsheet, where a floating pill would cover the Print button.

nav()'s root element stays <header>: print.html hides header on print and plan.html hides its #hdr wrapper, and both rules depend on it.

Camera model

Everything derives from one assumption — HFOV_DEG (default 60, override via localStorage.nbx.hfovDeg) — applied to the image's LONG axis, because iOS may hand back a portrait stream (720×1280). Keying it to videoWidth made the focal length 1.78× too short.

focalPxFor(w, h) = (max(w, h) / 2) / tan(HFOV_DEG / 2)

The same function feeds both the detector and the render camera, so they cannot disagree. The render camera additionally accounts for object-fit: cover: only part of the frame is on screen, so camera3.fov comes from the visible height, and _crop records the visible fraction so on-screen diagnostics compare like with like.

Frame conversion (validated against synthetic projections, error ~0): solvePnP returns world→cvCamera (R, t); cv camera is +X right, +Y down, +Z forward. Camera position C = -Rᵀt; three.js world-from-camera rotation is Rᵀ · diag(1, -1, -1).

Pose validity: the coplanar ambiguity

A coplanar set of markers admits two poses that reproject almost equally well — the true one and a mirrored twin. With few crests visible the solver intermittently returns the twin, which looks like ghosts snapping to the wrong side of the table. Reprojection error cannot separate them: a flipped pose was observed accepted at 1.3 px.

Two independent measures, in order of preference:

1. Geometry (the cure). Non-coplanar anchors make the twin mathematically impossible. The A3 test rig provides this with a folded tower carrying crests 100 mm above the sheet; the real town should keep at least one raised or wall-mounted crest for the same reason.
2. Physics gates (the safety net), in cv-worker.js: the camera must be above the table, and cannot teleport 20 cm between frames. A rejected pose leaves the previous one in place. An escape hatch after 12 consecutive rejections prevents a bad reference locking tracking out. The HUD reports the rejection reason.

Detection also deduplicates ids within a frame, keeping the largest quad: two quads decoding to one id would hand solvePnP contradictory image points for identical world points.

Scene and state flow

• Scene is fetched over HTTP at startup, not awaited from the WebSocket. Detection filters against scene anchor ids, so a slow or flapping socket used to make every detection get silently discarded — indistinguishable on screen from "the camera sees nothing". The WebSocket scene message now only refreshes an already-working client.
• The WebSocket remains authoritative for ghosts (spawn, despawn, active, characters) and clock sync (ping/pong).
• Motion amounts are server-authoritative. playlist.js sends explicit bobAmp, radius and speed with every spawn record, so client-side defaults never apply to a real ghost. Tune motion in the playlist config.

Occlusion

Invisible depth-only meshes (colorWrite: false) let physical models hide ghosts. Two shapes: boxes for buildings, and cylinders (shape:'cylinder' with radiusCm/heightCm) for round props, where a box silhouette would wrongly hide ghosts at the corners. Positions are floor-level; meshes are lifted by half their height.

Caching

App JS and HTML are served no-cache (ETag makes revalidation cheap); the 13 MB opencv.js is immutable, max-age=1yr. A cache-busting query on the entry module would not work — ES module imports fetch under their own URLs, so children would stay stale. Note that an upstream openresty proxy has been seen overriding this with a daily expires; that needs a vhost fix.