103 lines
5.2 KiB
Markdown
103 lines
5.2 KiB
Markdown
# Newbury Exhibit v3
|
|
|
|
Fan-made shared multi-viewer AR experience for a physical LEGO® Hidden Side™ town
|
|
display (tribute project — not affiliated with the LEGO Group). Multiple phones
|
|
detect printed "Newbury Crests" (ArUco markers) and see the same server-driven
|
|
ghosts anchored to real-world positions, each from their own viewpoint.
|
|
|
|
**v3 = v2 with the tracking core replaced.** Everything else — playlist engine,
|
|
residents/tracks, character editor, admin suite, dev error overlay, password-
|
|
protected final mode, deployment — carries over unchanged.
|
|
|
|
**[docs/PAGES.md](docs/PAGES.md) — every page the exhibit serves, and the API
|
|
endpoints worth opening while debugging.** Keep it updated as pages change.
|
|
|
|
## What changed in v3
|
|
|
|
The per-marker POS-IT → confidence-weighted fusion pipeline is replaced by a
|
|
**joint multi-marker board solve** (`public/js/ar/solve-cv.js`, OpenCV.js):
|
|
|
|
- Every detected crest corner becomes a 3D world point (from `data/scene.json`
|
|
anchors, same `anchorWorldMatrix` convention as v2) and **one `solvePnP` call**
|
|
recovers the camera pose. No per-marker fusion, no slerp base, no area buckets.
|
|
- Stability scales with marker spread. Synthetic validation at 0.5px corner noise:
|
|
~0.06 cm position / 0.03° rotation error on the full 5-crest board
|
|
(single-marker POS-IT on the same test: ~2.6 cm / 0.9°).
|
|
- The **wall crest makes the point set non-coplanar**, which removes the POS-IT
|
|
planar ambiguity (the flip problem) outright whenever it's visible.
|
|
- Single visible crest still works: SQPNP/IPPE cold start, then iterative refine
|
|
warm-started from the previous frame (temporal continuity resolves flips).
|
|
- A reprojection-error gate (mean px, shown in the dev HUD) rejects poisoned
|
|
frames before they reach the 1€ filter.
|
|
|
|
OpenCV.js (~13 MB WASM) is fetched at Docker build time (pinned
|
|
`@techstark/opencv-js@4.12.0-release.1`) and lazy-loaded on the client after
|
|
Start, with download progress shown in the dev HUD (`cv:loading 42%`). If it
|
|
fails or stalls past 45 s, the app falls back to the v2 engine automatically.
|
|
|
|
## Dual engines (on-table A/B)
|
|
|
|
The complete v2 pipeline (`detect.js` → `pose.js` → `fuse.js`) ships alongside
|
|
the new one until the board solve is confirmed on the real table.
|
|
|
|
Dev HUD overlay tap zones (thirds):
|
|
|
|
| zone | opencv engine | posit engine |
|
|
|--------|----------------------------|-----------------------------|
|
|
| top | switch engine (retries load if it failed) | switch engine |
|
|
| middle | axis mode (`std`/`ymirror`)| rotation mode (`fr`/…) |
|
|
| bottom | — | camera correction (`none`/…)|
|
|
|
|
Engine choice persists in `localStorage` (`nbx.engine`). Optional per-device
|
|
horizontal FOV override: `localStorage nbx.hfovDeg` (default 60, both engines).
|
|
|
|
`axis ymirror` exists as on-device insurance: if the world appears mirrored /
|
|
camera under the table in opencv mode, tap the middle third once.
|
|
|
|
## Reading the dev HUD
|
|
|
|
```
|
|
ws open rx scene,active,pong <- WebSocket state + recent message types
|
|
ghost recs 1 <- ghost records the client is holding
|
|
eng opencv FROZEN <- engine; FROZEN = ghost motion paused (tap tracking chip)
|
|
axis std cv:ready <- engine mode + OpenCV load state
|
|
markers 2 reproj 0.4px <- crests detected + mean reprojection error
|
|
raw / view / up / pos <- solved camera pose
|
|
m0 px(0.02,-0.05) proj(-0.71,0.63) <- AIM CHECK, see below
|
|
ghost ndc(-0.90,0.80) off-screen <- first ghost's screen position
|
|
```
|
|
|
|
**Aim check.** `px` is where the crest actually sits in the camera frame
|
|
(`0,0` = centre); `proj` is where the solved pose *projects* it. Point straight
|
|
at a crest so `px` ≈ `(0,0)` — if `proj` disagrees, the camera-frame convention
|
|
is wrong, not the tracking. A sign flip on x means mirrored, on y means
|
|
Y-inverted, on both means a 180° roll. Ghosts also carry a magenta always-on-top
|
|
beacon in dev mode, so "no ghost" and "ghost off-screen" can't be confused.
|
|
|
|
## First-time load (in place of v2)
|
|
|
|
```
|
|
git clone https://gitea.hideawaygaming.com.au/jessikitty/newbury-exhibit-v3.git ~/newbury-exhibit-v3
|
|
cd ~/newbury-exhibit-v3 && bash load-v3.sh
|
|
```
|
|
|
|
`load-v3.sh` imports the unchanged v2 base files into this repo (one commit,
|
|
pushed — set `GITEA_TOKEN=…` if the machine has no git credentials), copies
|
|
`.env` and any locally-dropped `public/models` from the v2 checkout, stops the
|
|
v2 container, and brings v3 up **on the same host port, container name, and
|
|
`exhibit-live` volume** — live scene/playlist/character state carries over.
|
|
|
|
- Future deploys: `cd ~/newbury-exhibit-v3 && bash deploy.sh` (same as v2).
|
|
- Rollback any time: `cd ~/newbury-exhibit && docker compose up -d --build`.
|
|
|
|
## Conventions (unchanged from v2)
|
|
|
|
- World units cm; origin front-left table corner; +X right, +Z back, +Y up.
|
|
- Marker print sizes stay in mm (`sizeMM`); single mm→cm conversion at the
|
|
solver boundary.
|
|
- `data/*.json` are committed seeds; `data/live/` (volume) is runtime state.
|
|
- `maxHammingDistance: 0` — hamming-1 detections are 100% phantoms here.
|
|
|
|
LEGO® and Hidden Side™ are trademarks of the LEGO Group, which does not sponsor,
|
|
authorise or endorse this project.
|