newbury-exhibit/README.md

Newbury Exhibit

Shared, view-only AR experience for a physical Newbury LEGO display (max 700 × 1500 mm). Multiple phones/tablets see the same ghosts, in the same real-world spots on the build, each from their own viewpoint — the "Hidden Side" made visible to a room full of people at once.

This is a sibling project to newbury-nights (the single-player hunt). The architecture is deliberately different: instead of ghosts spawning around one player, ghosts live at fixed world anchors on the build, and every device localises itself against printed/LEGO markers so they all agree on where those anchors are.

---

How it fits together

 physical marker  ──►  world origin  ──►  anchors  ──►  ghosts  ──►  broadcast  ──►  every phone
 (Newbury Crest)       (one corner)       (scene.json)   (motion)     (WebSocket)     (renders identically)

1. Localisation. Each device's camera detects an ArUco 4×4 marker of known size and known world position. From the marker's four corners it solves where the device is in the build's coordinate frame. (Proof of concept: public/ar-test.html.)
2. Authoritative state. server/sync-server.js runs one motion loop and broadcasts ghost poses at 20 Hz over WebSocket (/sync). The server is the single source of truth, so all viewers see synchronised motion.
3. Rendering. Each device renders its own scene locally. public/js/viewer.js is the desktop "god-view" for authoring/testing; the phone AR client (next milestone) will render ghosts at anchor world-positions seen through the camera.

All coordinates are in millimetres, world origin at the front-left corner of the baseplate. This maps 1:1 to the physical build, so tape-measure values drop straight into data/scene.json.

---

Markers — "Newbury Crests"

The markers are ArUco 4×4 (dictionary ARUCO_4X4_1000), chosen because a 4×4 data grid maps cleanly onto LEGO tiles. Physically each marker is a 6×6 grid: a 1-cell black quiet border around the 4×4 code.

Design decision — colour wheel as frame, not code. ArUco detection is brightness-based (each cell reads black or white), which is exactly what makes it robust under bad lighting and shallow angles. Encoding the Red/Yellow/Blue Hidden Side colours into the code cells would make it pretty but fragile. So:

• Detector core (6×6): black & white flat tiles — this is what the camera reads.
• Colour-wheel ring: a decorative Red/Yellow/Blue border around the core, in the zone the detector ignores. On-theme for humans, invisible to the maths.

See markers/ for printable SVGs, per-anchor LEGO build maps, and crest-01_buildplan.svg (a visual of core + colour ring). Print a marker now to test robustness before committing to a build.

Marker → anchor mapping

markers/manifest.json maps each physical ArUco id to a scene anchor. The fiducialId in data/scene.json must match the arucoId of the marker you place there.

Crest	ArUco id	Ward (placeholder)
crest-01	0	North-West
crest-02	1	North-East
crest-03	2	Town Square
crest-04	3	South-West
crest-05	4	South-East

---

Proof of concept: ar-test.html

Camera → marker detection → a ghost sprite locked to the marker, bobbing above it. No server needed for this page. Use it to test detection robustness on your iPhone 15 Pro.

Must be served over HTTPS for the camera to work on iOS. Behind your existing nginx that's already handled; for the exhibit it'll sit at the same kind of URL as newbury-nights.

Drop an animated WebP at public/assets/ghost_blue.webp to use a real sprite; otherwise it draws a procedural blue "Sad" wisp.

---

Forgiving scanning (js/detect-tuned.js)

Raw js-aruco can feel finicky — slow to acquire, quick to drop on blur, jittery once locked. detect-tuned.js wraps it with three tolerances, used by ar-head.html:

1. Looser detector params — smaller min marker size, looser polygon fit, adjustable light threshold. Three presets: strict / forgiving (default) / greedy. Switch live from the "scan:" dropdown in ar-head.html.
2. Multi-scale pass — also detects on a half-res copy, catching distant / motion-blurred markers that fail at full res (corners scaled back up).
3. Coast + smooth tracker — MarkerTracker keeps the last good pose for ~280 ms through a dropout (no more snapping off when a hand passes or the image blurs), and smooths corners with an adaptive 1€ filter: heavy smoothing when you hold still (kills jitter), light when you move fast (stays responsive).

Tuning knobs: DETECT_PRESETS (params per preset), MarkerTracker({ coastMs, smooth }). Verified that the looser presets don't introduce false negatives — all presets still cleanly detect a known marker, including small + blurred.

The jitter meter (jitter.html) deliberately does not smooth — it reports raw pose truth so the numbers are honest. ar-head.html smooths because there the goal is a stable-feeling ghost.

3D LEGO-head ghost: ar-head.html

The real-3D successor to the flat-sprite PoC. Camera → ArUco → full 6-DoF pose → a Three.js ghost shaped like a classic minifig head (cylinder body, domed top, the stud) in a translucent Hidden Side gradient with a fresnel rim-glow. Because it's driven by the marker's solved rotation+translation, the head holds its orientation in world space — walk around the crest and it turns, showing its sides. Eyes on the front (+Z) face; simple original ghost eyes, no reproduced LEGO face print. Tap the swatches to switch lure colour (Blue/Red/Yellow).

Geometry lives in js/lego-head-ghost.js (buildLegoHeadGhost(THREE, colorKey)), built from primitives so there's no model file to load. Set MARKER_SIZE_MM in js/ar-head.js to match your marker.

Pose convention: POS-IT gives the marker pose in a camera frame (X right, Y up, Z toward viewer); we flip Y/Z into Three.js's (Y up, looking down −Z). Verified: anchor lands in front of camera at unit scale.

Pose jitter meter: jitter.html

Measures how stable the solved camera pose is per marker while you hold still — the number that predicts whether occlusion will look clean. "Jitter" = standard deviation of the marker's solved position (mm) and orientation (deg) over a ~1s rolling window.

Workflow for the en-masse test:

1. Open jitter.html on the phone (HTTPS).
2. Label the run in the text box (e.g. 1 marker, 30cm, room light).
3. Hold steady on a crest, Record ~10s, Stop.
4. Repeat for: closer / further, sharper angle, 1 marker vs several crests in frame, bright vs dim.
5. Export → Download JSON and send the file back.

The HUD shows live pos-jitter (green <0.5 mm, amber, red >1.5 mm), angle jitter, distance, and view angle. distMeanMm, markersPerFrame, and per-marker posJitMedMm in the export are the key numbers — they'll tell us how much multi-marker fusion is worth before we build occlusion.

Set MARKER_SIZE_MM at the top of public/js/jitter.js to the outer black-square size of the marker you actually test with (default 96 mm = 6 cells × 16 mm). Distance readings depend on it.

Run

npm install
npm start            # PORT defaults to 34034
# open http://localhost:34034/ar-test.html  (camera needs HTTPS on phones)
# open http://localhost:34034/               god-view viewer (viewer.js)

Env: PORT (default 34034), TICK_HZ (default 20).

---

Status / roadmap

• Authoritative sync server (hover + patrol motion solvers, 20 Hz broadcast)
• God-view desktop viewer
• Scene + anchor + spawn data model (mm, corner origin)
• ArUco 4×4 marker system + LEGO build maps ("Newbury Crests")
• AR PoC: camera + detection + ghost locked to marker (flat sprite)
• 3D LEGO-head ghost holding world orientation (ar-head.html)
• Pose jitter meter (jitter.html) with session logging + JSON export
• Multi-marker pose fusion (one combined camera pose from all visible crests)
• Occluder geometry: ghosts vanish behind premapped buildings (depth-only boxes)
• Phone AR client: render anchored ghosts through the camera using marker pose
• Calibration page: scan each crest, record its measured world position
• Per-ghost colour/sprite wiring from the enriched roster
• Multi-device soak test on the real build under exhibition lighting

---

Licence

MIT (project). Vendored js-aruco2 is MIT (Damiano Falcioni / Juan Mellado); ArUco dictionary data is 3-clause BSD (OpenCV). See headers in public/vendor/.