# Newbury Exhibit

Shared, view-only AR experience for a **physical Newbury LEGO display** (max 700 × 1500 mm).
Multiple phones/tablets see the *same* ghosts, in the *same* real-world spots on the build,
each from their own viewpoint — the "Hidden Side" made visible to a room full of people at once.

This is a sibling project to **newbury-nights** (the single-player hunt). The architecture is
deliberately different: instead of ghosts spawning around one player, ghosts live at fixed
**world anchors** on the build, and every device localises itself against printed/LEGO markers
so they all agree on where those anchors are.

---

## How it fits together

```
 physical marker  ──►  world origin  ──►  anchors  ──►  ghosts  ──►  broadcast  ──►  every phone
 (Newbury Crest)       (one corner)       (scene.json)   (motion)     (WebSocket)     (renders identically)
```

1. **Localisation.** Each device's camera detects an ArUco 4×4 marker of known size and known
   world position. From the marker's four corners it solves where the device is in the build's
   coordinate frame. (Proof of concept: `public/ar-test.html`.)
2. **Authoritative state.** `server/sync-server.js` runs one motion loop and broadcasts ghost
   poses at 20 Hz over WebSocket (`/sync`). The server is the single source of truth, so all
   viewers see synchronised motion.
3. **Rendering.** Each device renders its own scene locally. `public/js/viewer.js` is the desktop
   "god-view" for authoring/testing; the phone AR client (next milestone) will render ghosts at
   anchor world-positions seen through the camera.

All coordinates are in **millimetres**, world origin at the **front-left corner** of the baseplate.
This maps 1:1 to the physical build, so tape-measure values drop straight into `data/scene.json`.

---

## Markers — "Newbury Crests"

The markers are **ArUco 4×4** (dictionary `ARUCO_4X4_1000`), chosen because a 4×4 data grid maps
cleanly onto LEGO tiles. Physically each marker is a **6×6 grid**: a 1-cell black quiet border
around the 4×4 code.

**Design decision — colour wheel as frame, not code.** ArUco detection is brightness-based
(each cell reads black or white), which is exactly what makes it robust under bad lighting and
shallow angles. Encoding the Red/Yellow/Blue Hidden Side colours *into* the code cells would make
it pretty but fragile. So:

- **Detector core (6×6):** black & white flat tiles — this is what the camera reads.
- **Colour-wheel ring:** a decorative Red/Yellow/Blue border *around* the core, in the zone the
  detector ignores. On-theme for humans, invisible to the maths.

See `markers/` for printable SVGs, per-anchor LEGO build maps, and `crest-01_buildplan.svg`
(a visual of core + colour ring). Print a marker now to test robustness before committing to a build.

### Marker → anchor mapping
`markers/manifest.json` maps each physical ArUco id to a scene anchor. The `fiducialId` in
`data/scene.json` must match the `arucoId` of the marker you place there.

| Crest | ArUco id | Ward (placeholder) |
|-------|----------|--------------------|
| crest-01 | 0 | North-West |
| crest-02 | 1 | North-East |
| crest-03 | 2 | Town Square |
| crest-04 | 3 | South-West |
| crest-05 | 4 | South-East |

---

## Proof of concept: `ar-test.html`

Camera → marker detection → a ghost sprite locked to the marker, bobbing above it. No server
needed for this page. Use it to test detection robustness on your iPhone 15 Pro.

**Must be served over HTTPS** for the camera to work on iOS. Behind your existing nginx that's
already handled; for the exhibit it'll sit at the same kind of URL as newbury-nights.

Drop an animated WebP at `public/assets/ghost_blue.webp` to use a real sprite; otherwise it draws
a procedural blue "Sad" wisp.

---

## Forgiving scanning (`js/detect-tuned.js`)

Raw js-aruco can feel finicky — slow to acquire, quick to drop on blur, jittery once locked.
`detect-tuned.js` wraps it with three tolerances, used by `ar-head.html`:

1. **Looser detector params** — smaller min marker size, looser polygon fit, adjustable light
   threshold. Three presets: `strict` / `forgiving` (default) / `greedy`. Switch live from the
   "scan:" dropdown in `ar-head.html`.
2. **Multi-scale pass** — also detects on a half-res copy, catching distant / motion-blurred
   markers that fail at full res (corners scaled back up).
3. **Coast + smooth tracker** — `MarkerTracker` keeps the last good pose for ~280 ms through a
   dropout (no more snapping off when a hand passes or the image blurs), and smooths corners with
   an adaptive 1€ filter: heavy smoothing when you hold still (kills jitter), light when you move
   fast (stays responsive).

Tuning knobs: `DETECT_PRESETS` (params per preset), `MarkerTracker({ coastMs, smooth })`.
Verified that the looser presets don't introduce false negatives — all presets still cleanly
detect a known marker, including small + blurred.

> The jitter meter (`jitter.html`) deliberately does **not** smooth — it reports raw pose truth so
> the numbers are honest. `ar-head.html` smooths because there the goal is a stable-feeling ghost.

## 3D LEGO-head ghost: `ar-head.html`

The real-3D successor to the flat-sprite PoC. Camera → ArUco → **full 6-DoF pose** → a Three.js
ghost shaped like a classic minifig head (cylinder body, domed top, the stud) in a translucent
Hidden Side gradient with a fresnel rim-glow. Because it's driven by the marker's solved
rotation+translation, the head holds its orientation in world space — **walk around the crest and
it turns**, showing its sides. Eyes on the front (+Z) face; simple original ghost eyes, no
reproduced LEGO face print. Tap the swatches to switch lure colour (Blue/Red/Yellow).

Geometry lives in `js/lego-head-ghost.js` (`buildLegoHeadGhost(THREE, colorKey)`), built from
primitives so there's no model file to load. Set `MARKER_SIZE_MM` in `js/ar-head.js` to match your
marker.

> Pose convention: POS-IT gives the marker pose in a camera frame (X right, Y up, Z toward viewer);
> we flip Y/Z into Three.js's (Y up, looking down −Z). Verified: anchor lands in front of camera at
> unit scale.

## Pose jitter meter: `jitter.html`

Measures how *stable* the solved camera pose is per marker while you hold still — the number
that predicts whether occlusion will look clean. "Jitter" = standard deviation of the marker's
solved position (mm) and orientation (deg) over a ~1s rolling window.

**Workflow for the en-masse test:**
1. Open `jitter.html` on the phone (HTTPS).
2. Label the run in the text box (e.g. `1 marker, 30cm, room light`).
3. Hold steady on a crest, **Record** ~10s, **Stop**.
4. Repeat for: closer / further, sharper angle, 1 marker vs several crests in frame, bright vs dim.
5. **Export → Download JSON** and send the file back.

The HUD shows live pos-jitter (green <0.5 mm, amber, red >1.5 mm), angle jitter, distance, and
view angle. `distMeanMm`, `markersPerFrame`, and per-marker `posJitMedMm` in the export are the
key numbers — they'll tell us how much multi-marker fusion is worth before we build occlusion.

> Set `MARKER_SIZE_MM` at the top of `public/js/jitter.js` to the outer black-square size of the
> marker you actually test with (default 96 mm = 6 cells × 16 mm). Distance readings depend on it.

## Run

```bash
npm install
npm start            # PORT defaults to 34034
# open http://localhost:34034/ar-test.html  (camera needs HTTPS on phones)
# open http://localhost:34034/               god-view viewer (viewer.js)
```

Env: `PORT` (default 34034), `TICK_HZ` (default 20).

---

## Status / roadmap

- [x] Authoritative sync server (hover + patrol motion solvers, 20 Hz broadcast)
- [x] God-view desktop viewer
- [x] Scene + anchor + spawn data model (mm, corner origin)
- [x] ArUco 4×4 marker system + LEGO build maps ("Newbury Crests")
- [x] AR PoC: camera + detection + ghost locked to marker (flat sprite)
- [x] 3D LEGO-head ghost holding world orientation (`ar-head.html`)
- [x] Pose jitter meter (`jitter.html`) with session logging + JSON export
- [ ] Multi-marker pose fusion (one combined camera pose from all visible crests)
- [ ] Occluder geometry: ghosts vanish behind premapped buildings (depth-only boxes)
- [ ] Phone AR client: render anchored ghosts through the camera using marker pose
- [ ] Calibration page: scan each crest, record its measured world position
- [ ] Per-ghost colour/sprite wiring from the enriched roster
- [ ] Multi-device soak test on the real build under exhibition lighting

---

## Licence
MIT (project). Vendored `js-aruco2` is MIT (Damiano Falcioni / Juan Mellado); ArUco dictionary
data is 3-clause BSD (OpenCV). See headers in `public/vendor/`.