docs: session record — AR tracking overhaul, camera model, coplanar ambiguity, A3 test rig
This commit is contained in:
jessikitty committed
1 parent
12d30b3b8f
commit
9aac46eba4
1 file changed
+200
@@ -0,0 +1,200 @@
|
||||
# AR Tracking Overhaul: Worker Offload, Camera Model, and Coplanar Pose Ambiguity
|
||||
|
||||
**Date:** 2026-09-23
|
||||
**Project:** newbury-exhibit-v3
|
||||
**Status:** ongoing
|
||||
**Tags:** ar, opencv, solvepnp, aruco, web-worker, camera-calibration, performance, caching, test-rig
|
||||
|
||||
## Context
|
||||
|
||||
v3 replaced v2's per-marker POS-IT tracking with a joint multi-marker OpenCV
|
||||
board solve. On-device testing surfaced a cascade of distinct faults that all
|
||||
presented as "the ghost is in the wrong place or missing": a frozen page, ghosts
|
||||
rendering off-screen, ghosts vanishing entirely, and stale builds masking fixes.
|
||||
This session diagnosed each from HUD telemetry rather than guesswork, fixed them,
|
||||
and built a printable A3 test rig — including a foldable marker tower — to make
|
||||
tracking evaluation repeatable.
|
||||
|
||||
## Key Decisions
|
||||
|
||||
- **Run OpenCV.js in a Web Worker** (`cv-worker.js`), not on the main thread.
|
||||
The `@techstark/opencv-js` build embeds ~8 MB of WASM as a base64 data URI
|
||||
inside a 10.8 MB script, so parsing + base64 decode + WASM compile all happen
|
||||
synchronously — seconds of frozen UI and a "page unresponsive" prompt.
|
||||
*Rejected:* loading via Blob URL with a progress bar (made memory worse: three
|
||||
copies of 10.8 MB); extracting the WASM to a sibling file at Docker build time
|
||||
to enable streaming compile (mechanical string surgery on emscripten glue, too
|
||||
fragile).
|
||||
|
||||
- **Solves are fire-and-forget, one in flight at a time.** If a solve is still
|
||||
running the frame is skipped rather than queued, so the main thread never waits
|
||||
and latency can't accumulate. The 1€ filter absorbs the ~1 frame delay.
|
||||
|
||||
- **FOV applies to the image's LONG axis, not `videoWidth`.** iOS hands back
|
||||
portrait streams (720×1280); assuming the angle spanned `videoWidth` made the
|
||||
focal length 1280/720 = 1.78× too short. This was measured on device twice —
|
||||
the first "fix" (deriving vertical FOV from the video + `object-fit: cover`
|
||||
crop) was correct in principle but still keyed off `videoWidth`, so the same
|
||||
1.78× error survived it.
|
||||
|
||||
- **Throttle detection to 20 Hz while rendering at full rate.** Detection
|
||||
(`getImageData` + ArUco decode) dominates frame cost, and the crests don't
|
||||
move — only the phone does.
|
||||
|
||||
- **Cap the detection canvas LONG side, not its width.** Fixing width at 960
|
||||
tripled pixel count on portrait streams (960×1707 vs 960×540).
|
||||
|
||||
- **Kept detection resolution at 960 after checking the arithmetic.** Mid-change
|
||||
I was raising it to 1280 on the theory that missed crests were a decode
|
||||
problem — then computed that 960 already yields ~23 px per marker cell, far
|
||||
above ArUco's needs. Raising it would have cost +78% CPU for probably nothing.
|
||||
Exposed as a knob instead so the theory can be tested rather than assumed.
|
||||
|
||||
- **Deduplicate marker ids within a frame, keeping the largest quad.** Two quads
|
||||
decoding to the same id hand solvePnP two contradictory image points for one
|
||||
set of world points, collapsing the pose (observed: `seen [0,2,0,4]`,
|
||||
reproj 5.4 px).
|
||||
|
||||
- **Gate poses on physics, not reprojection error.** A coplanar point set has two
|
||||
poses that reproject almost equally well, so error can't separate them (a
|
||||
flipped pose was accepted at 1.3 px). Gates: camera must be above the sheet;
|
||||
camera can't teleport 20 cm between frames. Rejections keep the *previous*
|
||||
pose rather than applying a bad one. Escape hatch after 12 consecutive
|
||||
rejections so a bad reference can't lock tracking out.
|
||||
|
||||
- **The real cure for ambiguity is geometry: a marker tower.** Gates suppress a
|
||||
symptom; non-coplanar anchors make the mirrored pose mathematically impossible.
|
||||
*Rejected:* wrapping a crest sheet around the 85 mm cylinder — ArUco needs
|
||||
planar facets. Chose a four-sided prism + top face so at least one crest is
|
||||
readable from any angle (worst case 45° incidence).
|
||||
|
||||
- **Tower face orientations derived, not guessed.** Unrolling a prism advances
|
||||
arc position and the marker's printed-right axis together, so left-to-right on
|
||||
the flat net is yaw *increasing*: front(180), left(270), back(0), right(90).
|
||||
Verified against the real `anchorWorldMatrix` before shipping, because a wrong
|
||||
yaw yields plausible-but-wrong ghost placement — the hardest kind to debug.
|
||||
|
||||
- **Motion amounts are server-authoritative.** `playlist.js` sends explicit
|
||||
`bobAmp`/`radius`/`speed` with every spawn, so client defaults never applied.
|
||||
A client-side "3 mm bob" fix was dead code until the server default changed.
|
||||
|
||||
- **Test rig is a single source of truth.** The page that draws the printed
|
||||
sheets also builds the scene + playlist JSON from the same constants, so print
|
||||
and config cannot drift apart.
|
||||
|
||||
## Technical Details
|
||||
|
||||
**World frame:** origin at front-left of the sheet; +X right, +Z away from
|
||||
viewer, +Y up; centimetres. Marker print sizes stay in mm (`sizeMM`), converted
|
||||
once at the solver boundary.
|
||||
|
||||
**Camera model** (`exhibit.js`):
|
||||
```js
|
||||
const focalPxFor = (w, h) => (Math.max(w, h) / 2) / Math.tan(degToRad(HFOV_DEG) / 2);
|
||||
// updateCameraIntrinsics(): scale = max(innerWidth/vw, innerHeight/vh) // object-fit: cover
|
||||
// visH = min(vh, innerHeight/scale); camera3.fov = 2*atan((visH/2)/focal)
|
||||
// _crop.{x,y} = visible fraction, used to put the aim-check px into viewport NDC
|
||||
```
|
||||
Verified rendered frustum matches true visible frame to <0.05° in portrait,
|
||||
landscape, and landscape-screen cases.
|
||||
|
||||
**Worker protocol** (`public/js/ar/cv-worker.js`):
|
||||
- `{type:'init'}` → `{type:'ready'}` | `{type:'failed', msg}`
|
||||
- `{type:'solve', id, obj, img, n, W, H, f}` → `{type:'pose', id, ok, C, m3, reproj, n, why}`
|
||||
- `{type:'reset'}` — clears warm start + gate state
|
||||
- `obj`/`img` are transferred `Float64Array`s (no copy). Worker returns camera
|
||||
position `C` and the 3×3 world-from-camera rotation `m3`; the client builds the
|
||||
quaternion (worker never touches three.js).
|
||||
|
||||
**Frame conversion** (validated against synthetic projections, error ~0):
|
||||
solvePnP gives world→cvCamera `(R,t)`; cv cam is +X right, +Y down, +Z forward.
|
||||
`C = -Rᵀt`; three.js world-from-camera = `Rᵀ · diag(1,-1,-1)`.
|
||||
|
||||
**Worker constants:** `MAX_REPROJ_PX = 4` (was 8), `MIN_CAM_Y_CM = 1`,
|
||||
`MAX_JUMP_CM = 20` (allowance scales with elapsed time), `JUMP_GRACE_MS = 500`,
|
||||
`MAX_REJECT_STREAK = 12`, `PREV_TTL = 1500`.
|
||||
|
||||
**localStorage knobs:** `nbx.engine` (opencv|posit), `nbx.hfovDeg` (default 60),
|
||||
`nbx.detectHz` (20), `nbx.detectLong` (960).
|
||||
|
||||
**Server motion defaults** (`server/playlist.js`, these override the client):
|
||||
`bobAmp 0.3`, `bobHz 0.4`, `wanderRadius 2.5`, `wanderVertical 0.4`,
|
||||
`pathSpeed 1.5` — previously 5–10, 60, 10, 8, sized for a large floor.
|
||||
|
||||
**A3 test rig** (`/admin/testsheet.html`), A3 landscape = 42.0 × 29.7 cm:
|
||||
- Page 1: crests 0–5 at (5,4) (21,4) (37,4) (5,25.7) (21,25.7) (37,25.7) cm,
|
||||
60 mm, `mount:'flat'`, `yawDeg:180` (prints upright). 100 mm scale bar.
|
||||
- Page 2: tower net — 85 mm square × 160 mm tall, four 85 mm panels + 15 mm glue
|
||||
tab + 85 mm top flap; net 355 × 245 mm. Crests 6–9 on the sides at 100 mm
|
||||
height (`mount:'wall'`, yaw 180/270/0/90, position pushed ±4.25 cm along the
|
||||
face normal), crest 10 on top (`mount:'flat'`, `yawDeg:180`).
|
||||
- Anchor heights now 0 / 10 / 16 cm — genuinely non-coplanar.
|
||||
- Walker loop: 12–30 cm × 8.85–20.85 cm, 60 cm perimeter, ~40 s/lap at 1.5 cm/s.
|
||||
- Static ghost at (21, 22) cm, behind the tower from the front (occlusion test).
|
||||
|
||||
**Cache policy** (`server/server.js`): `no-cache` for `.js`/`.html`,
|
||||
`immutable, max-age=31536000` for `/vendor/opencv.js`. A cache-busting query on
|
||||
the entry module would NOT have worked — ES module imports fetch under their own
|
||||
URLs, so `detect.js` etc. would stay stale.
|
||||
|
||||
**Deploy:** `cd ~/newbury-exhibit-v3 && bash deploy.sh`. Rollback to v2:
|
||||
`cd ~/newbury-exhibit && docker compose up -d --build`.
|
||||
|
||||
## Code Changes
|
||||
|
||||
- `public/js/ar/cv-worker.js` — **new.** OpenCV board solve off-thread; reusable
|
||||
Mats; warm-start + SQPNP/IPPE cold start; reprojection, camera-height and
|
||||
teleport gates with reported rejection reason.
|
||||
- `public/js/ar/solve-cv.js` — rewritten as a worker client; keeps anchor
|
||||
geometry and the 1€ filter on the main thread; retains the previous pose on
|
||||
rejection; exposes `lastReject`.
|
||||
- `public/js/ar/detect.js` — report raw vs scene-filtered detections; dedupe
|
||||
repeated ids keeping the largest quad; expose `dupes`.
|
||||
- `public/js/exhibit.js` — worker-backed solver; long-axis focal model; camera
|
||||
intrinsics from video + cover-crop; 20 Hz detection throttle; long-side canvas
|
||||
cap; scene fetched over HTTP so tracking survives a dead WebSocket; cylinder
|
||||
occluders; dev Motion button; aim-check and rejection telemetry in the HUD.
|
||||
- `public/index.html` — moved `#dbg` below the HUD row (it was covering the
|
||||
tracking chip and swallowing taps); added the `#motionBtn` control.
|
||||
- `public/js/ghosts/behavior.js` — idle float scaled to the display: 3 mm bob
|
||||
(was 6 cm), matching shimmer for wander/path; table-scale fallbacks.
|
||||
- `server/playlist.js` — server-side motion defaults rescaled and made tunable
|
||||
via `behaviors.bobAmp`/`bobHz`.
|
||||
- `server/server.js` — explicit cache policy for app code vs the WASM blob.
|
||||
- `public/admin/testsheet.html` — **new.** Two-page printable A3 test rig plus
|
||||
one-click scene + playlist load, all from shared constants.
|
||||
- `docs/PAGES.md` — **new.** Page index, API endpoints, HUD reading guide, and
|
||||
the gotchas worth not rediscovering.
|
||||
|
||||
## Open Questions / Next Steps
|
||||
|
||||
- **Why do crests fail to decode?** Observed 5 clearly visible, only 2 decoded.
|
||||
Resolution is NOT the explanation (~23 px/cell at 960). Candidates: strict
|
||||
`maxHammingDistance: 0` rejecting marginal reads under blur/glare; adaptive
|
||||
thresholding on uneven lighting. Relaxing to hamming 1 is plausible now that a
|
||||
known-id whitelist, dedupe and pose gates exist, but it risks a misread landing
|
||||
on a whitelisted id. Test `nbx.detectLong = 1280` first to settle it empirically.
|
||||
- **Validate the tower on-device.** Success criteria: `markers` count rises,
|
||||
`REJ:` stays quiet, and `proj` tracks `px` while circling the tower.
|
||||
- **Tune `nbx.hfovDeg`.** 60° is a guess; if a constant `proj`/`px` ratio remains,
|
||||
adjust and watch `reproj` minimise.
|
||||
- **WebSocket flapping.** HUD showed `ws conn` persisting with ghosts clearing on
|
||||
reconnect. Tracking no longer depends on it, but ghosts do. Likely the
|
||||
openresty vhost missing `Upgrade`/`Connection` headers on `/ws`.
|
||||
- **Upstream proxy overrides cache headers.** openresty replaces the app's
|
||||
`no-cache` with a daily `expires`, so deploys don't reach devices until it
|
||||
lapses. Needs a vhost change; private-tab testing is the workaround.
|
||||
- **Layout editor doesn't understand cylinder occluders or wall anchors** — it
|
||||
may mangle the tower layout on save.
|
||||
- **Delete the posit engine** once the opencv path is confirmed on the real town.
|
||||
|
||||
## Artifacts
|
||||
|
||||
- Repo: `https://gitea.hideawaygaming.com.au/jessikitty/newbury-exhibit-v3`
|
||||
- v2 (rollback path): `https://gitea.hideawaygaming.com.au/jessikitty/newbury-exhibit-v2`
|
||||
- `docs/PAGES.md` — page index, HUD guide, gotchas
|
||||
- `public/admin/testsheet.html` — the printable test rig
|
||||
- OpenCV.js build pinned at `@techstark/opencv-js@4.12.0-release.1`
|
||||
(5.0.0 does not initialise; `solvePnPGeneric` and `cornerSubPix` are absent
|
||||
from this build, which is why ambiguity is handled by gating rather than by
|
||||
comparing both IPPE solutions)
|
||||
Reference in new issue
Block a user