207 lines
11 KiB
Markdown
207 lines
11 KiB
Markdown
# Newbury Exhibit
|
||
|
||
Shared, view-only AR experience for a **physical Newbury LEGO display** (max 700 × 1500 mm).
|
||
Multiple phones/tablets see the *same* ghosts, in the *same* real-world spots on the build,
|
||
each from their own viewpoint — the "Hidden Side" made visible to a room full of people at once.
|
||
|
||
This is a sibling project to **newbury-nights** (the single-player hunt). The architecture is
|
||
deliberately different: instead of ghosts spawning around one player, ghosts live at fixed
|
||
**world anchors** on the build, and every device localises itself against printed/LEGO markers
|
||
so they all agree on where those anchors are.
|
||
|
||
---
|
||
|
||
## How it fits together
|
||
|
||
```
|
||
physical marker ──► world origin ──► anchors ──► ghosts ──► broadcast ──► every phone
|
||
(Newbury Crest) (one corner) (scene.json) (motion) (WebSocket) (renders identically)
|
||
```
|
||
|
||
1. **Localisation.** Each device's camera detects an ArUco 4×4 marker of known size and known
|
||
world position. From the marker's four corners it solves where the device is in the build's
|
||
coordinate frame. (Proof of concept: `public/ar-test.html`.)
|
||
2. **Authoritative state.** `server/sync-server.js` runs one motion loop and broadcasts ghost
|
||
poses at 20 Hz over WebSocket (`/sync`). The server is the single source of truth, so all
|
||
viewers see synchronised motion.
|
||
3. **Rendering.** Each device renders its own scene locally. `public/js/viewer.js` is the desktop
|
||
"god-view" for authoring/testing; the phone AR client (next milestone) will render ghosts at
|
||
anchor world-positions seen through the camera.
|
||
|
||
All coordinates are in **millimetres**, world origin at the **front-left corner** of the baseplate.
|
||
This maps 1:1 to the physical build, so tape-measure values drop straight into `data/scene.json`.
|
||
|
||
---
|
||
|
||
## Markers — "Newbury Crests"
|
||
|
||
The markers are **ArUco 4×4** (dictionary `ARUCO_4X4_1000`), chosen because a 4×4 data grid maps
|
||
cleanly onto LEGO tiles. Physically each marker is a **6×6 grid**: a 1-cell black quiet border
|
||
around the 4×4 code.
|
||
|
||
**Design decision — colour wheel as frame, not code.** ArUco detection is brightness-based
|
||
(each cell reads black or white), which is exactly what makes it robust under bad lighting and
|
||
shallow angles. Encoding the Red/Yellow/Blue Hidden Side colours *into* the code cells would make
|
||
it pretty but fragile. So:
|
||
|
||
- **Detector core (6×6):** black & white flat tiles — this is what the camera reads.
|
||
- **Colour-wheel ring:** a decorative Red/Yellow/Blue border *around* the core, in the zone the
|
||
detector ignores. On-theme for humans, invisible to the maths.
|
||
|
||
See `markers/` for printable SVGs, per-anchor LEGO build maps, and `crest-01_buildplan.svg`
|
||
(a visual of core + colour ring). Print a marker now to test robustness before committing to a build.
|
||
|
||
### Marker → anchor mapping
|
||
`markers/manifest.json` maps each physical ArUco id to a scene anchor. The `fiducialId` in
|
||
`data/scene.json` must match the `arucoId` of the marker you place there.
|
||
|
||
| Crest | ArUco id | Ward (placeholder) |
|
||
|-------|----------|--------------------|
|
||
| crest-01 | 0 | North-West |
|
||
| crest-02 | 1 | North-East |
|
||
| crest-03 | 2 | Town Square |
|
||
| crest-04 | 3 | South-West |
|
||
| crest-05 | 4 | South-East |
|
||
|
||
---
|
||
|
||
## Proof of concept: `ar-test.html`
|
||
|
||
Camera → marker detection → a ghost sprite locked to the marker, bobbing above it. No server
|
||
needed for this page. Use it to test detection robustness on your iPhone 15 Pro.
|
||
|
||
**Must be served over HTTPS** for the camera to work on iOS. Behind your existing nginx that's
|
||
already handled; for the exhibit it'll sit at the same kind of URL as newbury-nights.
|
||
|
||
Drop an animated WebP at `public/assets/ghost_blue.webp` to use a real sprite; otherwise it draws
|
||
a procedural blue "Sad" wisp.
|
||
|
||
---
|
||
|
||
## Scene workshop (admin GUI): `admin.html`
|
||
|
||
A 3D authoring tool over `data/scene.json` — no more hand-editing JSON. Orbit the build, click to
|
||
select, drag on the table to position. Four tabs:
|
||
|
||
- **World** — table dimensions (mm). X = length, Z = depth, Y = max ghost height.
|
||
- **Anchors** — the AR crests: marker type/id, size, position, yaw. (Note: existing scene used
|
||
`apriltag`; the detector is ArUco — the editor lets you set `aruco` and the matching id.)
|
||
- **Ghosts** — spawns with `hover` (position + radius + bob + yaw) or `patrol` (editable path of
|
||
points, loop period, face-travel) motion, plus lure colour. Drag the ghost to move it; drag
|
||
individual patrol points to reshape the path.
|
||
- **Buildings** — simple stackable box blockers for occlusion. Drag to place, set W×H×D, and
|
||
"Duplicate (stack +Y)" to pile boxes into a structure. (Occlusion rendering comes next; this
|
||
authors the geometry.)
|
||
|
||
Tools: select (V), add ghost (G), add building (B), add anchor (A). Delete/Esc work on selection.
|
||
|
||
**Save** POSTs to the server, which **backs up the old `scene.json`** to `data/backups/` first,
|
||
validates, writes, and **hot-reloads** — so connected viewers/phones get the new anchors and
|
||
ghosts immediately. **Export** downloads the JSON; **Reload** discards changes and re-fetches.
|
||
|
||
Server endpoints: `GET /api/scene` (full scene), `POST /api/scene` (validated save + backup).
|
||
|
||
> The scene schema gained a `blockers` array: `{ id, label, position:{x,y,z}, size:{x,y,z} }`,
|
||
> where `position` is the footprint corner (origin-relative) and `size` is the box extent in mm.
|
||
|
||
## Forgiving scanning (`js/detect-tuned.js`)
|
||
|
||
Raw js-aruco can feel finicky — slow to acquire, quick to drop on blur, jittery once locked.
|
||
`detect-tuned.js` wraps it with three tolerances, used by `ar-head.html`:
|
||
|
||
1. **Looser detector params** — smaller min marker size, looser polygon fit, adjustable light
|
||
threshold. Three presets: `strict` / `forgiving` (default) / `greedy`. Switch live from the
|
||
"scan:" dropdown in `ar-head.html`.
|
||
2. **Multi-scale pass** — also detects on a half-res copy, catching distant / motion-blurred
|
||
markers that fail at full res (corners scaled back up).
|
||
3. **Coast + smooth tracker** — `MarkerTracker` keeps the last good pose for ~280 ms through a
|
||
dropout (no more snapping off when a hand passes or the image blurs), and smooths corners with
|
||
an adaptive 1€ filter: heavy smoothing when you hold still (kills jitter), light when you move
|
||
fast (stays responsive).
|
||
|
||
Tuning knobs: `DETECT_PRESETS` (params per preset), `MarkerTracker({ coastMs, smooth })`.
|
||
Verified that the looser presets don't introduce false negatives — all presets still cleanly
|
||
detect a known marker, including small + blurred.
|
||
|
||
> The jitter meter (`jitter.html`) deliberately does **not** smooth — it reports raw pose truth so
|
||
> the numbers are honest. `ar-head.html` smooths because there the goal is a stable-feeling ghost.
|
||
|
||
## 3D LEGO-head ghost: `ar-head.html`
|
||
|
||
The real-3D successor to the flat-sprite PoC. Camera → ArUco → **full 6-DoF pose** → a Three.js
|
||
ghost shaped like a classic minifig head (cylinder body, domed top, the stud) in a translucent
|
||
Hidden Side gradient with a fresnel rim-glow. Because it's driven by the marker's solved
|
||
rotation+translation, the head holds its orientation in world space — **walk around the crest and
|
||
it turns**, showing its sides. Eyes on the front (+Z) face; simple original ghost eyes, no
|
||
reproduced LEGO face print. Tap the swatches to switch lure colour (Blue/Red/Yellow).
|
||
|
||
Geometry lives in `js/lego-head-ghost.js` (`buildLegoHeadGhost(THREE, colorKey)`), built from
|
||
primitives so there's no model file to load. Set `MARKER_SIZE_MM` in `js/ar-head.js` to match your
|
||
marker.
|
||
|
||
> Pose convention: POS-IT gives the marker pose in a camera frame (X right, Y up, Z toward viewer);
|
||
> we flip Y/Z into Three.js's (Y up, looking down −Z). Verified: anchor lands in front of camera at
|
||
> unit scale.
|
||
|
||
## Pose jitter meter: `jitter.html`
|
||
|
||
Measures how *stable* the solved camera pose is per marker while you hold still — the number
|
||
that predicts whether occlusion will look clean. "Jitter" = standard deviation of the marker's
|
||
solved position (mm) and orientation (deg) over a ~1s rolling window.
|
||
|
||
**Workflow for the en-masse test:**
|
||
1. Open `jitter.html` on the phone (HTTPS).
|
||
2. Label the run in the text box (e.g. `1 marker, 30cm, room light`).
|
||
3. Hold steady on a crest, **Record** ~10s, **Stop**.
|
||
4. Repeat for: closer / further, sharper angle, 1 marker vs several crests in frame, bright vs dim.
|
||
5. **Export → Download JSON** and send the file back.
|
||
|
||
The HUD shows live pos-jitter (green <0.5 mm, amber, red >1.5 mm), angle jitter, distance, and
|
||
view angle. `distMeanMm`, `markersPerFrame`, and per-marker `posJitMedMm` in the export are the
|
||
key numbers — they'll tell us how much multi-marker fusion is worth before we build occlusion.
|
||
|
||
> Set `MARKER_SIZE_MM` at the top of `public/js/jitter.js` to the outer black-square size of the
|
||
> marker you actually test with (default 96 mm = 6 cells × 16 mm). Distance readings depend on it.
|
||
|
||
## Run
|
||
|
||
```bash
|
||
npm install
|
||
npm start # PORT defaults to 34034
|
||
# open http://localhost:34034/admin.html scene workshop (authoring GUI)
|
||
# open http://localhost:34034/ar-head.html 3D ghost AR (camera needs HTTPS on phones)
|
||
# open http://localhost:34034/ god-view viewer (viewer.js)
|
||
```
|
||
|
||
Env: `PORT` (default 34034), `TICK_HZ` (default 20).
|
||
|
||
### Deploy box (blackpearl)
|
||
`./run.sh {start|stop|restart|status|logs|update}` — background runner for testing.
|
||
`./run.sh update` syncs to origin/dev and restarts. `./deploy.sh` is the pull-only one-shot.
|
||
|
||
---
|
||
|
||
## Status / roadmap
|
||
|
||
- [x] Authoritative sync server (hover + patrol motion solvers, 20 Hz broadcast)
|
||
- [x] God-view desktop viewer
|
||
- [x] Scene + anchor + spawn data model (mm, corner origin)
|
||
- [x] ArUco 4×4 marker system + LEGO build maps ("Newbury Crests")
|
||
- [x] AR PoC: camera + detection + ghost locked to marker (flat sprite)
|
||
- [x] 3D LEGO-head ghost holding world orientation (`ar-head.html`)
|
||
- [x] Pose jitter meter (`jitter.html`) with session logging + JSON export
|
||
- [x] Forgiving detection: tuned params, multi-scale, coast + smooth (`detect-tuned.js`)
|
||
- [x] Scene workshop admin GUI (`admin.html`) — author anchors, ghosts, motion, buildings; save+backup+hot-reload
|
||
- [ ] Multi-marker pose fusion (one combined camera pose from all visible crests)
|
||
- [ ] Occluder geometry: ghosts vanish behind the building blockers (depth-only boxes)
|
||
- [ ] Phone AR client: render anchored ghosts through the camera using marker pose
|
||
- [ ] Calibration page: scan each crest, record its measured world position
|
||
- [ ] Per-ghost colour/sprite wiring from the enriched roster
|
||
- [ ] Multi-device soak test on the real build under exhibition lighting
|
||
|
||
---
|
||
|
||
## Licence
|
||
MIT (project). Vendored `js-aruco2` is MIT (Damiano Falcioni / Juan Mellado); ArUco dictionary
|
||
data is 3-clause BSD (OpenCV). See headers in `public/vendor/`.
|