docs: session record — AR tracking overhaul, camera model, coplanar ambiguity, A3 test rig
This commit is contained in:
jessikitty committed
1 parent
12d30b3b8f
commit
9aac46eba4
1 file changed
+200
@@ -0,0 +1,200 @@
|
|||||||
|
# AR Tracking Overhaul: Worker Offload, Camera Model, and Coplanar Pose Ambiguity
|
||||||
|
|
||||||
|
**Date:** 2026-09-23
|
||||||
|
**Project:** newbury-exhibit-v3
|
||||||
|
**Status:** ongoing
|
||||||
|
**Tags:** ar, opencv, solvepnp, aruco, web-worker, camera-calibration, performance, caching, test-rig
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
v3 replaced v2's per-marker POS-IT tracking with a joint multi-marker OpenCV
|
||||||
|
board solve. On-device testing surfaced a cascade of distinct faults that all
|
||||||
|
presented as "the ghost is in the wrong place or missing": a frozen page, ghosts
|
||||||
|
rendering off-screen, ghosts vanishing entirely, and stale builds masking fixes.
|
||||||
|
This session diagnosed each from HUD telemetry rather than guesswork, fixed them,
|
||||||
|
and built a printable A3 test rig — including a foldable marker tower — to make
|
||||||
|
tracking evaluation repeatable.
|
||||||
|
|
||||||
|
## Key Decisions
|
||||||
|
|
||||||
|
- **Run OpenCV.js in a Web Worker** (`cv-worker.js`), not on the main thread.
|
||||||
|
The `@techstark/opencv-js` build embeds ~8 MB of WASM as a base64 data URI
|
||||||
|
inside a 10.8 MB script, so parsing + base64 decode + WASM compile all happen
|
||||||
|
synchronously — seconds of frozen UI and a "page unresponsive" prompt.
|
||||||
|
*Rejected:* loading via Blob URL with a progress bar (made memory worse: three
|
||||||
|
copies of 10.8 MB); extracting the WASM to a sibling file at Docker build time
|
||||||
|
to enable streaming compile (mechanical string surgery on emscripten glue, too
|
||||||
|
fragile).
|
||||||
|
|
||||||
|
- **Solves are fire-and-forget, one in flight at a time.** If a solve is still
|
||||||
|
running the frame is skipped rather than queued, so the main thread never waits
|
||||||
|
and latency can't accumulate. The 1€ filter absorbs the ~1 frame delay.
|
||||||
|
|
||||||
|
- **FOV applies to the image's LONG axis, not `videoWidth`.** iOS hands back
|
||||||
|
portrait streams (720×1280); assuming the angle spanned `videoWidth` made the
|
||||||
|
focal length 1280/720 = 1.78× too short. This was measured on device twice —
|
||||||
|
the first "fix" (deriving vertical FOV from the video + `object-fit: cover`
|
||||||
|
crop) was correct in principle but still keyed off `videoWidth`, so the same
|
||||||
|
1.78× error survived it.
|
||||||
|
|
||||||
|
- **Throttle detection to 20 Hz while rendering at full rate.** Detection
|
||||||
|
(`getImageData` + ArUco decode) dominates frame cost, and the crests don't
|
||||||
|
move — only the phone does.
|
||||||
|
|
||||||
|
- **Cap the detection canvas LONG side, not its width.** Fixing width at 960
|
||||||
|
tripled pixel count on portrait streams (960×1707 vs 960×540).
|
||||||
|
|
||||||
|
- **Kept detection resolution at 960 after checking the arithmetic.** Mid-change
|
||||||
|
I was raising it to 1280 on the theory that missed crests were a decode
|
||||||
|
problem — then computed that 960 already yields ~23 px per marker cell, far
|
||||||
|
above ArUco's needs. Raising it would have cost +78% CPU for probably nothing.
|
||||||
|
Exposed as a knob instead so the theory can be tested rather than assumed.
|
||||||
|
|
||||||
|
- **Deduplicate marker ids within a frame, keeping the largest quad.** Two quads
|
||||||
|
decoding to the same id hand solvePnP two contradictory image points for one
|
||||||
|
set of world points, collapsing the pose (observed: `seen [0,2,0,4]`,
|
||||||
|
reproj 5.4 px).
|
||||||
|
|
||||||
|
- **Gate poses on physics, not reprojection error.** A coplanar point set has two
|
||||||
|
poses that reproject almost equally well, so error can't separate them (a
|
||||||
|
flipped pose was accepted at 1.3 px). Gates: camera must be above the sheet;
|
||||||
|
camera can't teleport 20 cm between frames. Rejections keep the *previous*
|
||||||
|
pose rather than applying a bad one. Escape hatch after 12 consecutive
|
||||||
|
rejections so a bad reference can't lock tracking out.
|
||||||
|
|
||||||
|
- **The real cure for ambiguity is geometry: a marker tower.** Gates suppress a
|
||||||
|
symptom; non-coplanar anchors make the mirrored pose mathematically impossible.
|
||||||
|
*Rejected:* wrapping a crest sheet around the 85 mm cylinder — ArUco needs
|
||||||
|
planar facets. Chose a four-sided prism + top face so at least one crest is
|
||||||
|
readable from any angle (worst case 45° incidence).
|
||||||
|
|
||||||
|
- **Tower face orientations derived, not guessed.** Unrolling a prism advances
|
||||||
|
arc position and the marker's printed-right axis together, so left-to-right on
|
||||||
|
the flat net is yaw *increasing*: front(180), left(270), back(0), right(90).
|
||||||
|
Verified against the real `anchorWorldMatrix` before shipping, because a wrong
|
||||||
|
yaw yields plausible-but-wrong ghost placement — the hardest kind to debug.
|
||||||
|
|
||||||
|
- **Motion amounts are server-authoritative.** `playlist.js` sends explicit
|
||||||
|
`bobAmp`/`radius`/`speed` with every spawn, so client defaults never applied.
|
||||||
|
A client-side "3 mm bob" fix was dead code until the server default changed.
|
||||||
|
|
||||||
|
- **Test rig is a single source of truth.** The page that draws the printed
|
||||||
|
sheets also builds the scene + playlist JSON from the same constants, so print
|
||||||
|
and config cannot drift apart.
|
||||||
|
|
||||||
|
## Technical Details
|
||||||
|
|
||||||
|
**World frame:** origin at front-left of the sheet; +X right, +Z away from
|
||||||
|
viewer, +Y up; centimetres. Marker print sizes stay in mm (`sizeMM`), converted
|
||||||
|
once at the solver boundary.
|
||||||
|
|
||||||
|
**Camera model** (`exhibit.js`):
|
||||||
|
```js
|
||||||
|
const focalPxFor = (w, h) => (Math.max(w, h) / 2) / Math.tan(degToRad(HFOV_DEG) / 2);
|
||||||
|
// updateCameraIntrinsics(): scale = max(innerWidth/vw, innerHeight/vh) // object-fit: cover
|
||||||
|
// visH = min(vh, innerHeight/scale); camera3.fov = 2*atan((visH/2)/focal)
|
||||||
|
// _crop.{x,y} = visible fraction, used to put the aim-check px into viewport NDC
|
||||||
|
```
|
||||||
|
Verified rendered frustum matches true visible frame to <0.05° in portrait,
|
||||||
|
landscape, and landscape-screen cases.
|
||||||
|
|
||||||
|
**Worker protocol** (`public/js/ar/cv-worker.js`):
|
||||||
|
- `{type:'init'}` → `{type:'ready'}` | `{type:'failed', msg}`
|
||||||
|
- `{type:'solve', id, obj, img, n, W, H, f}` → `{type:'pose', id, ok, C, m3, reproj, n, why}`
|
||||||
|
- `{type:'reset'}` — clears warm start + gate state
|
||||||
|
- `obj`/`img` are transferred `Float64Array`s (no copy). Worker returns camera
|
||||||
|
position `C` and the 3×3 world-from-camera rotation `m3`; the client builds the
|
||||||
|
quaternion (worker never touches three.js).
|
||||||
|
|
||||||
|
**Frame conversion** (validated against synthetic projections, error ~0):
|
||||||
|
solvePnP gives world→cvCamera `(R,t)`; cv cam is +X right, +Y down, +Z forward.
|
||||||
|
`C = -Rᵀt`; three.js world-from-camera = `Rᵀ · diag(1,-1,-1)`.
|
||||||
|
|
||||||
|
**Worker constants:** `MAX_REPROJ_PX = 4` (was 8), `MIN_CAM_Y_CM = 1`,
|
||||||
|
`MAX_JUMP_CM = 20` (allowance scales with elapsed time), `JUMP_GRACE_MS = 500`,
|
||||||
|
`MAX_REJECT_STREAK = 12`, `PREV_TTL = 1500`.
|
||||||
|
|
||||||
|
**localStorage knobs:** `nbx.engine` (opencv|posit), `nbx.hfovDeg` (default 60),
|
||||||
|
`nbx.detectHz` (20), `nbx.detectLong` (960).
|
||||||
|
|
||||||
|
**Server motion defaults** (`server/playlist.js`, these override the client):
|
||||||
|
`bobAmp 0.3`, `bobHz 0.4`, `wanderRadius 2.5`, `wanderVertical 0.4`,
|
||||||
|
`pathSpeed 1.5` — previously 5–10, 60, 10, 8, sized for a large floor.
|
||||||
|
|
||||||
|
**A3 test rig** (`/admin/testsheet.html`), A3 landscape = 42.0 × 29.7 cm:
|
||||||
|
- Page 1: crests 0–5 at (5,4) (21,4) (37,4) (5,25.7) (21,25.7) (37,25.7) cm,
|
||||||
|
60 mm, `mount:'flat'`, `yawDeg:180` (prints upright). 100 mm scale bar.
|
||||||
|
- Page 2: tower net — 85 mm square × 160 mm tall, four 85 mm panels + 15 mm glue
|
||||||
|
tab + 85 mm top flap; net 355 × 245 mm. Crests 6–9 on the sides at 100 mm
|
||||||
|
height (`mount:'wall'`, yaw 180/270/0/90, position pushed ±4.25 cm along the
|
||||||
|
face normal), crest 10 on top (`mount:'flat'`, `yawDeg:180`).
|
||||||
|
- Anchor heights now 0 / 10 / 16 cm — genuinely non-coplanar.
|
||||||
|
- Walker loop: 12–30 cm × 8.85–20.85 cm, 60 cm perimeter, ~40 s/lap at 1.5 cm/s.
|
||||||
|
- Static ghost at (21, 22) cm, behind the tower from the front (occlusion test).
|
||||||
|
|
||||||
|
**Cache policy** (`server/server.js`): `no-cache` for `.js`/`.html`,
|
||||||
|
`immutable, max-age=31536000` for `/vendor/opencv.js`. A cache-busting query on
|
||||||
|
the entry module would NOT have worked — ES module imports fetch under their own
|
||||||
|
URLs, so `detect.js` etc. would stay stale.
|
||||||
|
|
||||||
|
**Deploy:** `cd ~/newbury-exhibit-v3 && bash deploy.sh`. Rollback to v2:
|
||||||
|
`cd ~/newbury-exhibit && docker compose up -d --build`.
|
||||||
|
|
||||||
|
## Code Changes
|
||||||
|
|
||||||
|
- `public/js/ar/cv-worker.js` — **new.** OpenCV board solve off-thread; reusable
|
||||||
|
Mats; warm-start + SQPNP/IPPE cold start; reprojection, camera-height and
|
||||||
|
teleport gates with reported rejection reason.
|
||||||
|
- `public/js/ar/solve-cv.js` — rewritten as a worker client; keeps anchor
|
||||||
|
geometry and the 1€ filter on the main thread; retains the previous pose on
|
||||||
|
rejection; exposes `lastReject`.
|
||||||
|
- `public/js/ar/detect.js` — report raw vs scene-filtered detections; dedupe
|
||||||
|
repeated ids keeping the largest quad; expose `dupes`.
|
||||||
|
- `public/js/exhibit.js` — worker-backed solver; long-axis focal model; camera
|
||||||
|
intrinsics from video + cover-crop; 20 Hz detection throttle; long-side canvas
|
||||||
|
cap; scene fetched over HTTP so tracking survives a dead WebSocket; cylinder
|
||||||
|
occluders; dev Motion button; aim-check and rejection telemetry in the HUD.
|
||||||
|
- `public/index.html` — moved `#dbg` below the HUD row (it was covering the
|
||||||
|
tracking chip and swallowing taps); added the `#motionBtn` control.
|
||||||
|
- `public/js/ghosts/behavior.js` — idle float scaled to the display: 3 mm bob
|
||||||
|
(was 6 cm), matching shimmer for wander/path; table-scale fallbacks.
|
||||||
|
- `server/playlist.js` — server-side motion defaults rescaled and made tunable
|
||||||
|
via `behaviors.bobAmp`/`bobHz`.
|
||||||
|
- `server/server.js` — explicit cache policy for app code vs the WASM blob.
|
||||||
|
- `public/admin/testsheet.html` — **new.** Two-page printable A3 test rig plus
|
||||||
|
one-click scene + playlist load, all from shared constants.
|
||||||
|
- `docs/PAGES.md` — **new.** Page index, API endpoints, HUD reading guide, and
|
||||||
|
the gotchas worth not rediscovering.
|
||||||
|
|
||||||
|
## Open Questions / Next Steps
|
||||||
|
|
||||||
|
- **Why do crests fail to decode?** Observed 5 clearly visible, only 2 decoded.
|
||||||
|
Resolution is NOT the explanation (~23 px/cell at 960). Candidates: strict
|
||||||
|
`maxHammingDistance: 0` rejecting marginal reads under blur/glare; adaptive
|
||||||
|
thresholding on uneven lighting. Relaxing to hamming 1 is plausible now that a
|
||||||
|
known-id whitelist, dedupe and pose gates exist, but it risks a misread landing
|
||||||
|
on a whitelisted id. Test `nbx.detectLong = 1280` first to settle it empirically.
|
||||||
|
- **Validate the tower on-device.** Success criteria: `markers` count rises,
|
||||||
|
`REJ:` stays quiet, and `proj` tracks `px` while circling the tower.
|
||||||
|
- **Tune `nbx.hfovDeg`.** 60° is a guess; if a constant `proj`/`px` ratio remains,
|
||||||
|
adjust and watch `reproj` minimise.
|
||||||
|
- **WebSocket flapping.** HUD showed `ws conn` persisting with ghosts clearing on
|
||||||
|
reconnect. Tracking no longer depends on it, but ghosts do. Likely the
|
||||||
|
openresty vhost missing `Upgrade`/`Connection` headers on `/ws`.
|
||||||
|
- **Upstream proxy overrides cache headers.** openresty replaces the app's
|
||||||
|
`no-cache` with a daily `expires`, so deploys don't reach devices until it
|
||||||
|
lapses. Needs a vhost change; private-tab testing is the workaround.
|
||||||
|
- **Layout editor doesn't understand cylinder occluders or wall anchors** — it
|
||||||
|
may mangle the tower layout on save.
|
||||||
|
- **Delete the posit engine** once the opencv path is confirmed on the real town.
|
||||||
|
|
||||||
|
## Artifacts
|
||||||
|
|
||||||
|
- Repo: `https://gitea.hideawaygaming.com.au/jessikitty/newbury-exhibit-v3`
|
||||||
|
- v2 (rollback path): `https://gitea.hideawaygaming.com.au/jessikitty/newbury-exhibit-v2`
|
||||||
|
- `docs/PAGES.md` — page index, HUD guide, gotchas
|
||||||
|
- `public/admin/testsheet.html` — the printable test rig
|
||||||
|
- OpenCV.js build pinned at `@techstark/opencv-js@4.12.0-release.1`
|
||||||
|
(5.0.0 does not initialise; `solvePnPGeneric` and `cornerSubPix` are absent
|
||||||
|
from this build, which is why ambiguity is handled by gating rather than by
|
||||||
|
comparing both IPPE solutions)
|
||||||
Reference in new issue
Block a user