diff --git a/README.md b/README.md index c04fb3e..7fd70ea 100644 --- a/README.md +++ b/README.md @@ -78,6 +78,65 @@ a procedural blue "Sad" wisp. --- +## Forgiving scanning (`js/detect-tuned.js`) + +Raw js-aruco can feel finicky — slow to acquire, quick to drop on blur, jittery once locked. +`detect-tuned.js` wraps it with three tolerances, used by `ar-head.html`: + +1. **Looser detector params** — smaller min marker size, looser polygon fit, adjustable light + threshold. Three presets: `strict` / `forgiving` (default) / `greedy`. Switch live from the + "scan:" dropdown in `ar-head.html`. +2. **Multi-scale pass** — also detects on a half-res copy, catching distant / motion-blurred + markers that fail at full res (corners scaled back up). +3. **Coast + smooth tracker** — `MarkerTracker` keeps the last good pose for ~280 ms through a + dropout (no more snapping off when a hand passes or the image blurs), and smooths corners with + an adaptive 1€ filter: heavy smoothing when you hold still (kills jitter), light when you move + fast (stays responsive). + +Tuning knobs: `DETECT_PRESETS` (params per preset), `MarkerTracker({ coastMs, smooth })`. +Verified that the looser presets don't introduce false negatives — all presets still cleanly +detect a known marker, including small + blurred. + +> The jitter meter (`jitter.html`) deliberately does **not** smooth — it reports raw pose truth so +> the numbers are honest. `ar-head.html` smooths because there the goal is a stable-feeling ghost. + +## 3D LEGO-head ghost: `ar-head.html` + +The real-3D successor to the flat-sprite PoC. Camera → ArUco → **full 6-DoF pose** → a Three.js +ghost shaped like a classic minifig head (cylinder body, domed top, the stud) in a translucent +Hidden Side gradient with a fresnel rim-glow. Because it's driven by the marker's solved +rotation+translation, the head holds its orientation in world space — **walk around the crest and +it turns**, showing its sides. Eyes on the front (+Z) face; simple original ghost eyes, no +reproduced LEGO face print. Tap the swatches to switch lure colour (Blue/Red/Yellow). + +Geometry lives in `js/lego-head-ghost.js` (`buildLegoHeadGhost(THREE, colorKey)`), built from +primitives so there's no model file to load. Set `MARKER_SIZE_MM` in `js/ar-head.js` to match your +marker. + +> Pose convention: POS-IT gives the marker pose in a camera frame (X right, Y up, Z toward viewer); +> we flip Y/Z into Three.js's (Y up, looking down −Z). Verified: anchor lands in front of camera at +> unit scale. + +## Pose jitter meter: `jitter.html` + +Measures how *stable* the solved camera pose is per marker while you hold still — the number +that predicts whether occlusion will look clean. "Jitter" = standard deviation of the marker's +solved position (mm) and orientation (deg) over a ~1s rolling window. + +**Workflow for the en-masse test:** +1. Open `jitter.html` on the phone (HTTPS). +2. Label the run in the text box (e.g. `1 marker, 30cm, room light`). +3. Hold steady on a crest, **Record** ~10s, **Stop**. +4. Repeat for: closer / further, sharper angle, 1 marker vs several crests in frame, bright vs dim. +5. **Export → Download JSON** and send the file back. + +The HUD shows live pos-jitter (green <0.5 mm, amber, red >1.5 mm), angle jitter, distance, and +view angle. `distMeanMm`, `markersPerFrame`, and per-marker `posJitMedMm` in the export are the +key numbers — they'll tell us how much multi-marker fusion is worth before we build occlusion. + +> Set `MARKER_SIZE_MM` at the top of `public/js/jitter.js` to the outer black-square size of the +> marker you actually test with (default 96 mm = 6 cells × 16 mm). Distance readings depend on it. + ## Run ```bash @@ -97,7 +156,11 @@ Env: `PORT` (default 34034), `TICK_HZ` (default 20). - [x] God-view desktop viewer - [x] Scene + anchor + spawn data model (mm, corner origin) - [x] ArUco 4×4 marker system + LEGO build maps ("Newbury Crests") -- [x] AR PoC: camera + detection + ghost locked to marker +- [x] AR PoC: camera + detection + ghost locked to marker (flat sprite) +- [x] 3D LEGO-head ghost holding world orientation (`ar-head.html`) +- [x] Pose jitter meter (`jitter.html`) with session logging + JSON export +- [ ] Multi-marker pose fusion (one combined camera pose from all visible crests) +- [ ] Occluder geometry: ghosts vanish behind premapped buildings (depth-only boxes) - [ ] Phone AR client: render anchored ghosts through the camera using marker pose - [ ] Calibration page: scan each crest, record its measured world position - [ ] Per-ghost colour/sprite wiring from the enriched roster diff --git a/public/ar-head.html b/public/ar-head.html new file mode 100644 index 0000000..2b4aced --- /dev/null +++ b/public/ar-head.html @@ -0,0 +1,83 @@ + + + + + +Newbury Exhibit — 3D LEGO-head ghost + + + +
+ + +
+ + + + + +
+

Newbury LEGO-head ghost

+

Point at a crest. A 3D ghost head locks onto it — walk around and it turns to face you from every side.

+ +
+
+ + + + + + + + + diff --git a/public/jitter.html b/public/jitter.html new file mode 100644 index 0000000..279c5b0 --- /dev/null +++ b/public/jitter.html @@ -0,0 +1,149 @@ + + + + + +Newbury Exhibit — pose jitter meter + + + +
+ + +
+ +
REC 0.0s
+ + + + + + + +
+

Newbury jitter meter

+

Hold the phone steady on a crest, hit Record for ~10s, then move closer / further / to a sharper angle and record again. Export and send me the file.

+ +
+
+ + + + + + + + + diff --git a/public/js/ar-head.js b/public/js/ar-head.js new file mode 100644 index 0000000..84d2049 --- /dev/null +++ b/public/js/ar-head.js @@ -0,0 +1,191 @@ +/* Newbury Exhibit — 3D LEGO-head ghost AR driver + * + * Camera frame -> ArUco detection -> POS-IT 6-DoF pose -> Three.js transform. + * The ghost is placed in a Three.js scene whose camera sits at the origin; we + * apply the marker's solved rotation+translation to the ghost so it stays fixed + * in world space. Walk around the marker and you see the head's sides — it turns. + * + * This is the real-3D successor to ar-test.js (which was a flat sprite). + */ +import * as THREE from 'https://cdn.jsdelivr.net/npm/three@0.160.0/build/three.module.js'; +import { buildLegoHeadGhost } from './lego-head-ghost.js?v=1'; +import { TunedDetector, MarkerTracker } from './detect-tuned.js?v=1'; + +const MARKER_SIZE_MM = 96; // outer black square; match what you build/print. + +const $ = (id) => document.getElementById(id); +const video = $('video'); +const canvas = $('three'); +const startScreen = $('start'), goBtn = $('go'), errEl = $('err'); +const hud = $('hud'), picker = $('picker'); +const lockTag = $('lock'), midEl = $('mid'), distEl = $('dist'), fpsEl = $('fps'); + +const grab = document.createElement('canvas'); +const gctx = grab.getContext('2d', { willReadFrequently: true }); + +let detector = null, tracker = null, posit = null, focalPx = 0, running = false; + +// ---- Three.js setup ------------------------------------------------------- +const renderer = new THREE.WebGLRenderer({ canvas, alpha: true, antialias: true }); +renderer.setClearColor(0x000000, 0); // transparent over video + +const scene = new THREE.Scene(); +scene.add(new THREE.AmbientLight(0xb8c4e0, 1.1)); +const key = new THREE.DirectionalLight(0xffffff, 0.8); +key.position.set(0.3, 1, 0.6); +scene.add(key); + +// Camera at origin looking down -Z (we move the ghost, not the camera). +let camera = new THREE.PerspectiveCamera(50, 1, 1, 5000); // units: mm +camera.position.set(0, 0, 0); + +// The ghost head, parented to an "anchor" object we drive from pose. +const anchor = new THREE.Object3D(); +scene.add(anchor); +let ghost = buildLegoHeadGhost(THREE, 'Blue'); +ghost.userData.setBaseY(MARKER_SIZE_MM * 0.9); // float above the plaque +anchor.add(ghost); +anchor.visible = false; + +// tether line marker->ghost so it reads as bound to the crest +const tetherMat = new THREE.LineBasicMaterial({ color: 0x51eaf1, transparent: true, opacity: 0.35 }); +const tetherGeo = new THREE.BufferGeometry().setFromPoints([ + new THREE.Vector3(0, 0, 0), new THREE.Vector3(0, MARKER_SIZE_MM * 0.65, 0), +]); +const tether = new THREE.Line(tetherGeo, tetherMat); +anchor.add(tether); + +// ---- Pose -> matrix ------------------------------------------------------- +// js-aruco POS-IT gives rotation (3x3) + translation (mm) of the marker in a +// camera frame with X right, Y up, Z toward viewer. Three.js camera looks down +// -Z with Y up, so we flip Y and Z to convert. Marker plane is the anchor's XZ. +const poseMatrix = new THREE.Matrix4(); +const flip = new THREE.Matrix4().makeScale(1, -1, -1); + +function centredCorners(m, w, h) { + return m.corners.map((c) => ({ x: c.x - w / 2, y: h / 2 - c.y })); +} + +function applyPose(m) { + const pose = posit.pose(centredCorners(m, grab.width, grab.height)); + if (!pose) return false; + const R = pose.bestRotation, t = pose.bestTranslation; + // Build matrix: rows of R, translation in last column. + poseMatrix.set( + R[0][0], R[0][1], R[0][2], t[0], + R[1][0], R[1][1], R[1][2], t[1], + R[2][0], R[2][1], R[2][2], t[2], + 0, 0, 0, 1 + ); + // Convert axis convention. + poseMatrix.premultiply(flip); + anchor.matrixAutoUpdate = false; + anchor.matrix.copy(poseMatrix); + // The marker's own plane is XY in POS-IT; we want the ghost to stand UP off + // the table, i.e. along the marker normal. Rotate the head so its +Y aligns + // with the marker normal (marker's local +Z). + ghost.rotation.x = -Math.PI / 2; // stand head up out of the marker plane + tether.rotation.x = -Math.PI / 2; + return { dist: Math.hypot(t[0], t[1], t[2]) }; +} + +// ---- sizing --------------------------------------------------------------- +function sizeAll() { + const vw = video.videoWidth, vh = video.videoHeight; + if (!vw || !vh) return; + grab.width = vw; grab.height = vh; + const scale = Math.max(window.innerWidth / vw, window.innerHeight / vh); + const dw = vw * scale, dh = vh * scale; + for (const el of [video, canvas]) { el.style.width = dw + 'px'; el.style.height = dh + 'px'; } + renderer.setPixelRatio(Math.min(window.devicePixelRatio, 2)); + renderer.setSize(dw, dh, false); + camera.aspect = dw / dh; + // Approximate focal length; vertical FOV from focal & sensor height in px. + focalPx = vw; + const fovY = 2 * Math.atan((vh / 2) / focalPx) * (180 / Math.PI); + camera.fov = fovY * (dh / vh) / (dw / vw < 1 ? 1 : 1); // keep simple + camera.fov = fovY; + camera.updateProjectionMatrix(); + posit = new POS.Posit(MARKER_SIZE_MM, focalPx); +} + +// ---- start ---------------------------------------------------------------- +async function start() { + errEl.textContent = ''; + try { + const stream = await navigator.mediaDevices.getUserMedia({ + video: { facingMode: { ideal: 'environment' }, width: { ideal: 1280 }, height: { ideal: 720 } }, audio: false, + }); + video.srcObject = stream; await video.play(); + } catch (e) { + errEl.textContent = 'Camera unavailable: ' + (e.message || e.name) + '. iOS needs HTTPS + camera permission.'; + return; + } + detector = new TunedDetector({ dictionaryName: 'ARUCO_4X4_1000' }).setPreset('forgiving'); + tracker = new MarkerTracker({ coastMs: 280, smooth: true }); + await new Promise((res) => { if (video.readyState >= 2) res(); else video.onloadeddata = () => res(); }); + sizeAll(); + window.addEventListener('resize', sizeAll); + startScreen.classList.add('hidden'); + hud.classList.remove('hidden'); picker.classList.remove('hidden'); + running = true; requestAnimationFrame(loop); +} + +// ---- loop ----------------------------------------------------------------- +let frames = 0, fpsStamp = performance.now(); + +function loop(now) { + if (!running) return; + requestAnimationFrame(loop); + if (video.readyState < 2 || !grab.width) return; + + gctx.drawImage(video, 0, 0, grab.width, grab.height); + const img = gctx.getImageData(0, 0, grab.width, grab.height); + let raw = []; + try { raw = detector.detect(img, { width: grab.width, height: grab.height }); } catch (_) { raw = []; } + + // coast through dropouts + smooth corners + const markers = tracker.update(raw, now); + + if (markers.length) { + // nearest-ish: smallest marker id-agnostic; just take first track + const m = markers[0]; + const res = applyPose(m); + if (res) { + anchor.visible = true; + if (m.coasting) { + lockTag.textContent = 'holding…'; lockTag.className = 'tag on'; + } else { + lockTag.textContent = 'locked'; lockTag.className = 'tag on'; + } + midEl.textContent = m.id; + distEl.textContent = Math.round(res.dist); + } + } else { + anchor.visible = false; + lockTag.textContent = 'searching…'; lockTag.className = 'tag off'; + midEl.textContent = '—'; distEl.textContent = '—'; + } + + ghost.userData.update(now / 1000); + renderer.render(scene, camera); + + frames++; + if (now - fpsStamp > 500) { fpsEl.textContent = Math.round((frames * 1000) / (now - fpsStamp)); frames = 0; fpsStamp = now; } +} + +// ---- colour picker -------------------------------------------------------- +picker.addEventListener('click', (e) => { + const sw = e.target.closest('.swatch'); if (!sw) return; + picker.querySelectorAll('.swatch').forEach((s) => s.classList.remove('sel')); + sw.classList.add('sel'); + ghost.userData.setColor(sw.dataset.c); +}); + +goBtn.addEventListener('click', start); + +// live sensitivity switch +const presetSel = document.getElementById('preset'); +if (presetSel) presetSel.addEventListener('change', () => { + if (detector) detector.setPreset(presetSel.value); +}); diff --git a/public/js/detect-tuned.js b/public/js/detect-tuned.js new file mode 100644 index 0000000..faf5afc --- /dev/null +++ b/public/js/detect-tuned.js @@ -0,0 +1,181 @@ +/* Newbury Exhibit — forgiving detection + tracking + * + * Wraps js-aruco2 to make marker scanning more tolerant: + * 1. Looser detector params (find smaller / dimmer / sharper-angle markers). + * 2. Optional multi-scale pass (re-detect on a downscaled frame to catch + * distant or motion-blurred markers that fail at full res). + * 3. A stateful Tracker that COASTS through brief dropouts (keeps the last + * good pose for a short window instead of snapping off) and SMOOTHS the + * corners with an adaptive 1€-style filter (steady when still, responsive + * when moving). + * + * Usage: + * const td = new TunedDetector({ dictionaryName:'ARUCO_4X4_1000' }); + * const markers = td.detect(imageData, { width, height }); // raw, tuned + * const tracker = new MarkerTracker(); + * const tracked = tracker.update(markers, nowMs); // coasted+smoothed + * + * Each tracked marker: { id, corners:[{x,y}*4], coasting:bool, age:ms }. + * Corners are in the SAME pixel space js-aruco returns (full-res frame). + */ + +/* global AR, CV */ + +export const DETECT_PRESETS = { + // crisp, fast — good light, marker fills a decent part of frame + strict: { minSizeRatio: 0.04, epsilon: 0.05, threshKernel: 2, threshBias: 7, warp: 49, multiScale: false }, + // the sensible default for an exhibit — noticeably more forgiving + forgiving:{ minSizeRatio: 0.015, epsilon: 0.06, threshKernel: 2, threshBias: 7, warp: 49, multiScale: true }, + // last resort — distant/dim/awkward; costs more CPU + greedy: { minSizeRatio: 0.008, epsilon: 0.08, threshKernel: 3, threshBias: 9, warp: 49, multiScale: true }, +}; + +export class TunedDetector { + constructor(config) { + this.det = new AR.Detector(config); + this.preset = DETECT_PRESETS.forgiving; + // scratch image objects for the downscaled pass + this._small = null; + this._smallCanvas = null; + } + + setPreset(name) { this.preset = DETECT_PRESETS[name] || this.preset; return this; } + + // One detection pass at the detector's native resolution, using tuned params. + _pass(image) { + const d = this.det; + CV.grayscale(image, d.grey); + // threshKernel/threshBias widen tolerance to uneven lighting. + CV.adaptiveThreshold(d.grey, d.thres, this.preset.threshKernel, this.preset.threshBias); + d.contours = CV.findContours(d.thres, d.binary); + d.candidates = d.findCandidates( + d.contours, + image.width * this.preset.minSizeRatio, // smaller => accepts smaller markers + this.preset.epsilon, // looser polygon approximation + 10 + ); + d.candidates = d.clockwiseCorners(d.candidates); + d.candidates = d.notTooNear(d.candidates, 10); + return d.findMarkers(d.grey, d.candidates, this.preset.warp); + } + + // Optional second pass on a half-res copy; corners scaled back to full res. + _passDownscaled(imageData, width, height) { + const sw = Math.round(width / 2), sh = Math.round(height / 2); + if (!this._smallCanvas) { + this._smallCanvas = document.createElement('canvas'); + this._smallCtx = this._smallCanvas.getContext('2d', { willReadFrequently: true }); + } + this._smallCanvas.width = sw; this._smallCanvas.height = sh; + // draw the full-res ImageData down by putting it on a temp canvas then scaling + if (!this._fullCanvas) { + this._fullCanvas = document.createElement('canvas'); + this._fullCtx = this._fullCanvas.getContext('2d', { willReadFrequently: true }); + } + this._fullCanvas.width = width; this._fullCanvas.height = height; + this._fullCtx.putImageData(imageData, 0, 0); + this._smallCtx.drawImage(this._fullCanvas, 0, 0, sw, sh); + const smallImg = this._smallCtx.getImageData(0, 0, sw, sh); + const found = this._pass(smallImg); + for (const m of found) m.corners = m.corners.map((c) => ({ x: c.x * 2, y: c.y * 2 })); + return found; + } + + detect(imageData, dims) { + let markers = []; + try { markers = this._pass(imageData); } catch (_) { markers = []; } + + if (this.preset.multiScale && dims) { + const seen = new Set(markers.map((m) => m.id)); + let extra = []; + try { extra = this._passDownscaled(imageData, dims.width, dims.height); } catch (_) { extra = []; } + for (const m of extra) if (!seen.has(m.id)) markers.push(m); + } + return markers; + } +} + +/* ---- 1€ filter (adaptive smoothing) ---- + * Smooths a value with a cutoff that rises with speed: still hand => heavy + * smoothing (kills jitter); moving fast => light smoothing (stays responsive). + */ +class OneEuro { + constructor(minCutoff = 1.2, beta = 0.012, dCutoff = 1.0) { + this.minCutoff = minCutoff; this.beta = beta; this.dCutoff = dCutoff; + this.xPrev = null; this.dxPrev = 0; this.tPrev = null; + } + _alpha(cutoff, dt) { + const tau = 1 / (2 * Math.PI * cutoff); + return 1 / (1 + tau / dt); + } + filter(x, tMs) { + if (this.xPrev == null) { this.xPrev = x; this.tPrev = tMs; return x; } + let dt = (tMs - this.tPrev) / 1000; if (dt <= 0) dt = 1 / 60; + this.tPrev = tMs; + const dx = (x - this.xPrev) / dt; + const aD = this._alpha(this.dCutoff, dt); + const dxHat = aD * dx + (1 - aD) * this.dxPrev; + this.dxPrev = dxHat; + const cutoff = this.minCutoff + this.beta * Math.abs(dxHat); + const a = this._alpha(cutoff, dt); + const xHat = a * x + (1 - a) * this.xPrev; + this.xPrev = xHat; + return xHat; + } +} + +export class MarkerTracker { + /* coastMs: how long to keep showing a marker after it stops being detected. + * Higher = more stable through blur/occlusion, but a truly-gone marker lingers. */ + constructor({ coastMs = 250, smooth = true } = {}) { + this.coastMs = coastMs; + this.smooth = smooth; + this.tracks = new Map(); // id -> { corners, filters:[8 OneEuro], lastSeen, firstSeen } + } + + update(markers, nowMs) { + // refresh / create tracks for detected markers + const detectedIds = new Set(); + for (const m of markers) { + detectedIds.add(m.id); + let tr = this.tracks.get(m.id); + if (!tr) { + tr = { + id: m.id, + filters: Array.from({ length: 8 }, () => new OneEuro()), + firstSeen: nowMs, + corners: m.corners.map((c) => ({ x: c.x, y: c.y })), + }; + this.tracks.set(m.id, tr); + } + // smooth each corner coordinate + const sm = []; + for (let i = 0; i < 4; i++) { + const cx = m.corners[i].x, cy = m.corners[i].y; + sm.push(this.smooth + ? { x: tr.filters[i * 2].filter(cx, nowMs), y: tr.filters[i * 2 + 1].filter(cy, nowMs) } + : { x: cx, y: cy }); + } + tr.corners = sm; + tr.lastSeen = nowMs; + tr.coasting = false; + } + + // emit detected + still-coasting tracks; drop expired ones + const out = []; + for (const [id, tr] of this.tracks) { + const age = nowMs - tr.lastSeen; + if (detectedIds.has(id)) { + out.push({ id, corners: tr.corners, coasting: false, age: nowMs - tr.firstSeen }); + } else if (age <= this.coastMs) { + // coast: hold last good corners so the ghost doesn't snap off + out.push({ id, corners: tr.corners, coasting: true, age: nowMs - tr.firstSeen }); + } else { + this.tracks.delete(id); + } + } + return out; + } + + reset() { this.tracks.clear(); } +} diff --git a/public/js/jitter.js b/public/js/jitter.js new file mode 100644 index 0000000..7674507 --- /dev/null +++ b/public/js/jitter.js @@ -0,0 +1,371 @@ +/* Newbury Exhibit — pose jitter meter + * + * Measures how stable the solved camera pose is, per marker, while you hold still. + * "Jitter" = standard deviation of the marker's solved position (mm) and orientation + * (deg) over a short rolling window. Low jitter = occlusion will look clean. + * + * Records timestamped samples during a run and exports JSON + a human summary you + * can hand back for analysis. + * + * Pose: js-aruco2 POS.Posit. Detector dictionary ARUCO_4X4_1000. + */ +(function () { + 'use strict'; + + // ---- Physical constants ------------------------------------------------- + // The full printed/built marker the POS-IT model spans corner-to-corner. + // For an ArUco 4x4 the *outer black square* is 6 cells. With 16mm cells that + // is 96mm. If you test with a different printed size, change this and re-run. + const MARKER_SIZE_MM = 96; + + // ---- DOM ---------------------------------------------------------------- + const $ = (id) => document.getElementById(id); + const video = $('video'), overlay = $('overlay'), ctx = overlay.getContext('2d'); + const startScreen = $('start'), goBtn = $('go'), errEl = $('err'); + const hud = $('hud'), controls = $('controls'); + const lockTag = $('lock'), fpsEl = $('fps'); + const posJitEl = $('posJit'), angJitEl = $('angJit'), distEl = $('dist'), angEl = $('ang'); + const markersListEl = $('markersList'); + const recBtn = $('rec'), sampleCountEl = $('sampleCount'), exportBtn = $('export'), markPointBtn = $('markPoint'); + const noteEl = $('note'); + const recBadge = $('recBadge'), recTimeEl = $('recTime'); + const modal = $('modal'), summaryEl = $('summary'); + + const grab = document.createElement('canvas'); + const gctx = grab.getContext('2d', { willReadFrequently: true }); + + let detector = null, posit = null, focalPx = 0; + let running = false; + + // ---- Rolling jitter window (per marker id) ------------------------------ + // Keep recent pose samples; jitter = stddev over the window. + const WINDOW = 30; // ~1s at 30fps + const histories = new Map(); // id -> { pos:[{x,y,z}], rot:[ [3x3] ], t:[] } + + function pushHistory(id, pos, rotEuler) { + let h = histories.get(id); + if (!h) { h = { pos: [], rot: [] }; histories.set(id, h); } + h.pos.push(pos); h.rot.push(rotEuler); + if (h.pos.length > WINDOW) { h.pos.shift(); h.rot.shift(); } + return h; + } + + function stddev(arr) { + const n = arr.length; if (n < 2) return 0; + const m = arr.reduce((a, b) => a + b, 0) / n; + const v = arr.reduce((a, b) => a + (b - m) * (b - m), 0) / (n - 1); + return Math.sqrt(v); + } + + // Position jitter: RMS of per-axis stddev (mm). Angle jitter: RMS of euler stddev (deg). + function jitterOf(h) { + if (h.pos.length < 4) return null; + const sx = stddev(h.pos.map((p) => p.x)); + const sy = stddev(h.pos.map((p) => p.y)); + const sz = stddev(h.pos.map((p) => p.z)); + const posJit = Math.sqrt(sx * sx + sy * sy + sz * sz); + const ra = stddev(h.rot.map((r) => r.x)); + const rb = stddev(h.rot.map((r) => r.y)); + const rc = stddev(h.rot.map((r) => r.z)); + const angJit = Math.sqrt(ra * ra + rb * rb + rc * rc); + return { posJit, angJit, samples: h.pos.length }; + } + + // ---- Pose helpers ------------------------------------------------------- + // js-aruco image points must be centred (origin = image centre, y up). + function centredCorners(m, w, h) { + return m.corners.map((c) => ({ x: c.x - w / 2, y: h / 2 - c.y })); + } + + function rotToEuler(R) { + // R is 3x3 array. Return degrees (x=pitch,y=yaw,z=roll). Good enough for jitter. + const sy = Math.sqrt(R[0][0] * R[0][0] + R[1][0] * R[1][0]); + let x, y, z; + if (sy > 1e-6) { + x = Math.atan2(R[2][1], R[2][2]); + y = Math.atan2(-R[2][0], sy); + z = Math.atan2(R[1][0], R[0][0]); + } else { + x = Math.atan2(-R[1][2], R[1][1]); y = Math.atan2(-R[2][0], sy); z = 0; + } + const d = 180 / Math.PI; + return { x: x * d, y: y * d, z: z * d }; + } + + function poseForMarker(m, w, h) { + const pts = centredCorners(m, w, h); + const pose = posit.pose(pts); + if (!pose) return null; + const t = pose.bestTranslation; // [x,y,z] in mm, camera space + const R = pose.bestRotation; // 3x3 + const pos = { x: t[0], y: t[1], z: t[2] }; + const dist = Math.sqrt(pos.x * pos.x + pos.y * pos.y + pos.z * pos.z); + // View angle: angle between camera->marker axis and the marker's normal (z col of R). + const normal = { x: R[0][2], y: R[1][2], z: R[2][2] }; + const toCam = { x: -pos.x / dist, y: -pos.y / dist, z: -pos.z / dist }; + let dot = normal.x * toCam.x + normal.y * toCam.y + normal.z * toCam.z; + dot = Math.max(-1, Math.min(1, Math.abs(dot))); + const viewAngle = Math.acos(dot) * 180 / Math.PI; // 0 = head-on + return { pos, euler: rotToEuler(R), dist, viewAngle, reprojErr: pose.bestError }; + } + + // ---- Session recording -------------------------------------------------- + const session = { + startedAt: null, + markerSizeMm: MARKER_SIZE_MM, + device: navigator.userAgent, + runs: [], // each: { label, startedAt, durationMs, samples:[...], flags:[...] } + }; + let currentRun = null; + let recording = false, recStart = 0; + + function startRun() { + currentRun = { + label: noteEl.value.trim() || `run ${session.runs.length + 1}`, + startedAt: new Date().toISOString(), + videoWidth: grab.width, videoHeight: grab.height, + focalPx: Math.round(focalPx), + samples: [], flags: [], + }; + recording = true; recStart = performance.now(); + recBtn.classList.add('recording'); recBtn.textContent = '■ Stop'; + recBadge.classList.add('show'); + } + + function stopRun() { + recording = false; + currentRun.durationMs = Math.round(performance.now() - recStart); + // attach computed summary for this run + currentRun.summary = summariseRun(currentRun); + session.runs.push(currentRun); + recBtn.classList.remove('recording'); recBtn.textContent = '● Record'; + recBadge.classList.remove('show'); + currentRun = null; + persist(); + } + + function summariseRun(run) { + // Aggregate per-marker jitter + distance across the recorded samples. + const byId = new Map(); + for (const s of run.samples) { + for (const mk of s.markers) { + let g = byId.get(mk.id); + if (!g) { g = { id: mk.id, posJit: [], angJit: [], dist: [], viewAngle: [], reproj: [] }; byId.set(mk.id, g); } + if (mk.posJit != null) g.posJit.push(mk.posJit); + if (mk.angJit != null) g.angJit.push(mk.angJit); + g.dist.push(mk.dist); g.viewAngle.push(mk.viewAngle); g.reproj.push(mk.reproj); + } + } + const mean = (a) => a.length ? a.reduce((x, y) => x + y, 0) / a.length : null; + const med = (a) => { if (!a.length) return null; const b = [...a].sort((x, y) => x - y); return b[Math.floor(b.length / 2)]; }; + const out = []; + for (const g of byId.values()) { + out.push({ + id: g.id, frames: g.dist.length, + posJitMeanMm: round(mean(g.posJit)), posJitMedMm: round(med(g.posJit)), + angJitMeanDeg: round(mean(g.angJit)), angJitMedDeg: round(med(g.angJit)), + distMeanMm: round(mean(g.dist)), viewAngleMeanDeg: round(mean(g.viewAngle)), + reprojMeanPx: round(mean(g.reproj)), + }); + } + // multi-marker stat: how many frames saw N markers + const multi = {}; + for (const s of run.samples) { const n = s.markers.length; multi[n] = (multi[n] || 0) + 1; } + return { perMarker: out, markersPerFrame: multi }; + } + + function round(n) { return n == null ? null : Math.round(n * 100) / 100; } + + function persist() { + try { localStorage.setItem('newbury-jitter-session', JSON.stringify(session)); } catch (_) {} + } + function restore() { + try { + const raw = localStorage.getItem('newbury-jitter-session'); + if (raw) { const s = JSON.parse(raw); if (s.runs) { session.runs = s.runs; } } + } catch (_) {} + } + + // ---- Camera + loop ------------------------------------------------------ + function sizeToVideo() { + const vw = video.videoWidth, vh = video.videoHeight; + if (!vw || !vh) return; + grab.width = vw; grab.height = vh; + const scale = Math.max(window.innerWidth / vw, window.innerHeight / vh); + const dw = vw * scale, dh = vh * scale; + for (const el of [video, overlay]) { el.style.width = dw + 'px'; el.style.height = dh + 'px'; } + overlay.width = dw; overlay.height = dh; overlay._scale = scale; + // Focal length estimate in px. Without calibration, ~1.0*width is a decent + // phone-camera approximation; jitter is relative so exact focal isn't critical. + focalPx = vw; + posit = new POS.Posit(MARKER_SIZE_MM, focalPx); + } + + async function start() { + errEl.textContent = ''; + try { + const stream = await navigator.mediaDevices.getUserMedia({ + video: { facingMode: { ideal: 'environment' }, width: { ideal: 1280 }, height: { ideal: 720 } }, audio: false, + }); + video.srcObject = stream; await video.play(); + } catch (e) { + errEl.textContent = 'Camera unavailable: ' + (e.message || e.name) + '. iOS needs HTTPS + camera permission.'; + return; + } + detector = new AR.Detector({ dictionaryName: 'ARUCO_4X4_1000' }); + await new Promise((res) => { if (video.readyState >= 2) res(); else video.onloadeddata = () => res(); }); + sizeToVideo(); + window.addEventListener('resize', sizeToVideo); + startScreen.classList.add('hidden'); + hud.classList.remove('hidden'); controls.classList.remove('hidden'); + running = true; requestAnimationFrame(loop); + } + + let frames = 0, fpsStamp = performance.now(); + function loop(now) { + if (!running) return; + requestAnimationFrame(loop); + if (video.readyState < 2 || !grab.width) return; + + gctx.drawImage(video, 0, 0, grab.width, grab.height); + const img = gctx.getImageData(0, 0, grab.width, grab.height); + let markers = []; try { markers = detector.detect(img); } catch (_) { markers = []; } + + const s = overlay._scale || 1; + ctx.clearRect(0, 0, overlay.width, overlay.height); + + const frameSample = { t: Math.round(now), markers: [] }; + let primary = null; + + for (const m of markers) { + drawOutline(m, s, markers.length > 1); + const p = poseForMarker(m, grab.width, grab.height); + if (!p) continue; + const h = pushHistory(m.id, p.pos, p.euler); + const j = jitterOf(h); + const rec = { + id: m.id, dist: round(p.dist), viewAngle: round(p.viewAngle), + reproj: round(p.reprojErr), + posJit: j ? round(j.posJit) : null, angJit: j ? round(j.angJit) : null, + hamming: m.hammingDistance, + }; + frameSample.markers.push(rec); + if (!primary || p.dist < primary.dist) primary = { ...rec, dist: p.dist }; + } + + // HUD + if (primary) { + lockTag.textContent = markers.length > 1 ? `${markers.length} markers` : 'locked'; + lockTag.className = 'tag on'; + setJit(posJitEl, primary.posJit, 0.5, 1.5); // <0.5mm good, >1.5 bad + setJit(angJitEl, primary.angJit, 0.3, 1.0); // degrees + distEl.textContent = Math.round(primary.dist); + angEl.textContent = primary.viewAngle != null ? Math.round(primary.viewAngle) : '—'; + markersListEl.innerHTML = markers.map((m) => { + const r = frameSample.markers.find((x) => x.id === m.id); + return `id ${m.id} · ${r ? Math.round(r.dist) : '?'}mm · jit ${r && r.posJit != null ? r.posJit.toFixed(2) : '—'}mm`; + }).join('   '); + } else { + lockTag.textContent = 'no marker'; lockTag.className = 'tag off'; + posJitEl.textContent = angJitEl.textContent = distEl.textContent = angEl.textContent = '—'; + posJitEl.className = angJitEl.className = ''; + markersListEl.textContent = 'point at a crest…'; + } + + if (recording) { + currentRun.samples.push(frameSample); + sampleCountEl.textContent = currentRun.samples.length; + recTimeEl.textContent = ((now - recStart) / 1000).toFixed(1) + 's'; + } + + frames++; + if (now - fpsStamp > 500) { fpsEl.textContent = Math.round((frames * 1000) / (now - fpsStamp)); frames = 0; fpsStamp = now; } + } + + function setJit(el, val, goodT, badT) { + if (val == null) { el.textContent = '—'; el.className = ''; return; } + el.textContent = val.toFixed(2); + el.className = val <= goodT ? 'jitter-good' : (val >= badT ? 'jitter-bad' : 'jitter-warn'); + } + + function drawOutline(m, s, multi) { + const c = m.corners.map((p) => ({ x: p.x * s, y: p.y * s })); + ctx.lineWidth = 3; + ctx.strokeStyle = multi ? 'rgba(81,234,241,0.95)' : 'rgba(82,158,255,0.9)'; + ctx.beginPath(); ctx.moveTo(c[0].x, c[0].y); + for (let i = 1; i < c.length; i++) ctx.lineTo(c[i].x, c[i].y); + ctx.closePath(); ctx.stroke(); + ctx.fillStyle = '#fff35d'; + ctx.beginPath(); ctx.arc(c[0].x, c[0].y, 5, 0, Math.PI * 2); ctx.fill(); + // id label + const cx = (c[0].x + c[2].x) / 2, cy = (c[0].y + c[2].y) / 2; + ctx.fillStyle = 'rgba(255,255,255,0.95)'; ctx.font = 'bold 16px system-ui'; + ctx.textAlign = 'center'; ctx.fillText(String(m.id), cx, cy); + } + + // ---- Export ------------------------------------------------------------- + function buildSummaryText() { + const lines = []; + lines.push('NEWBURY EXHIBIT — pose jitter report'); + lines.push('generated: ' + new Date().toISOString()); + lines.push('marker size: ' + MARKER_SIZE_MM + 'mm · device: ' + shortUA()); + lines.push('runs: ' + session.runs.length); + lines.push(''); + session.runs.forEach((run, i) => { + lines.push(`── Run ${i + 1}: "${run.label}"`); + lines.push(` ${run.videoWidth}x${run.videoHeight}px · focal≈${run.focalPx}px · ${(run.durationMs / 1000).toFixed(1)}s · ${run.samples.length} frames`); + const mpf = run.summary.markersPerFrame || {}; + lines.push(' markers/frame: ' + Object.entries(mpf).map(([n, c]) => `${n}→${c}`).join(' ')); + for (const pm of run.summary.perMarker) { + lines.push(` • id ${pm.id}: posJit ${pm.posJitMedMm}mm (med) / ${pm.posJitMeanMm} (mean), angJit ${pm.angJitMedDeg}° , dist ${pm.distMeanMm}mm, view ${pm.viewAngleMeanDeg}°, reproj ${pm.reprojMeanPx}px, ${pm.frames}f`); + } + if (run.flags.length) lines.push(' flags: ' + run.flags.map((f) => `${(f.atMs / 1000).toFixed(1)}s`).join(', ')); + lines.push(''); + }); + lines.push('Lower posJit = steadier pose = cleaner occlusion. Compare 1-marker vs multi-marker runs.'); + return lines.join('\n'); + } + + function shortUA() { + const ua = navigator.userAgent; + const m = ua.match(/(iPhone|iPad|Android)[^)]*/); + return m ? m[0].slice(0, 40) : ua.slice(0, 40); + } + + function openExport() { + if (recording) stopRun(); + summaryEl.textContent = session.runs.length ? buildSummaryText() : 'No runs recorded yet. Hit Record, hold on a crest ~10s, Stop, then Export.'; + modal.classList.add('show'); + } + + function downloadJson() { + const blob = new Blob([JSON.stringify(session, null, 2)], { type: 'application/json' }); + const url = URL.createObjectURL(blob); + const a = document.createElement('a'); + const stamp = new Date().toISOString().replace(/[:.]/g, '-').slice(0, 19); + a.href = url; a.download = `newbury-jitter-${stamp}.json`; + document.body.appendChild(a); a.click(); a.remove(); + setTimeout(() => URL.revokeObjectURL(url), 1000); + } + + async function copyText() { + try { await navigator.clipboard.writeText(buildSummaryText()); $('copyText').textContent = 'Copied ✓'; setTimeout(() => $('copyText').textContent = 'Copy summary', 1500); } + catch (_) { /* clipboard may be blocked; JSON download is the reliable path */ } + } + + // ---- Wire up ------------------------------------------------------------ + goBtn.addEventListener('click', start); + recBtn.addEventListener('click', () => { recording ? stopRun() : startRun(); }); + markPointBtn.addEventListener('click', () => { + if (recording && currentRun) currentRun.flags.push({ atMs: Math.round(performance.now() - recStart) }); + }); + exportBtn.addEventListener('click', openExport); + $('closeModal').addEventListener('click', () => modal.classList.remove('show')); + $('downloadJson').addEventListener('click', downloadJson); + $('copyText').addEventListener('click', copyText); + $('clearAll').addEventListener('click', () => { + if (confirm('Clear all recorded runs?')) { session.runs = []; persist(); summaryEl.textContent = 'Cleared.'; sampleCountEl.textContent = '0'; } + }); + + restore(); +})(); diff --git a/public/js/lego-head-ghost.js b/public/js/lego-head-ghost.js new file mode 100644 index 0000000..9e16b98 --- /dev/null +++ b/public/js/lego-head-ghost.js @@ -0,0 +1,192 @@ +/* Newbury Exhibit — LEGO-head ghost (3D) + * + * A classic minifig-head SILHOUETTE (cylinder body, domed top, stud) given a + * translucent ghost treatment in a Hidden Side gradient. Built from primitives + * so there's no external model file to load. Original eyes + wispy tail — we + * deliberately do NOT reproduce any printed LEGO face or licensed character. + * + * Exports buildLegoHeadGhost(THREE, colorKey) -> THREE.Group (units: mm). + * The group's local origin is at the centre of the head so it sits nicely on a + * marker anchor. Call group.userData.update(tSeconds) each frame for the idle + * bob/sway/blink animation. + */ + +// Hidden Side ghost-lure gradients (recovered GhostType data). +export const GHOST_GRADIENTS = { + Red: { top: 0xf65151, bottom: 0xff2678 }, + Yellow: { top: 0xb57f0b, bottom: 0xfff35d }, + Blue: { top: 0x529eff, bottom: 0x51eaf1 }, +}; + +// Vertical gradient + soft rim glow, applied to the head body. +const GHOST_VERT = ` + varying vec3 vNormalW; + varying vec3 vViewDir; + varying float vY; + uniform float uMinY; + uniform float uMaxY; + void main() { + vec4 wp = modelMatrix * vec4(position, 1.0); + vNormalW = normalize(mat3(modelMatrix) * normal); + vViewDir = normalize(cameraPosition - wp.xyz); + vY = clamp((position.y - uMinY) / (uMaxY - uMinY), 0.0, 1.0); + gl_Position = projectionMatrix * viewMatrix * wp; + } +`; + +const GHOST_FRAG = ` + varying vec3 vNormalW; + varying vec3 vViewDir; + varying float vY; + uniform vec3 uTop; + uniform vec3 uBottom; + uniform float uOpacity; + uniform float uTime; + void main() { + // body gradient (top lighter -> bottom saturated) + vec3 base = mix(uBottom, uTop, vY); + // fresnel rim so the silhouette edge glows like a spectre + float fres = pow(1.0 - max(dot(normalize(vNormalW), normalize(vViewDir)), 0.0), 2.2); + vec3 col = base + fres * vec3(0.55, 0.85, 0.95); + // gentle internal shimmer + float shimmer = 0.06 * sin(vY * 18.0 - uTime * 2.2); + col += shimmer; + float alpha = uOpacity * (0.55 + 0.45 * fres); + gl_FragColor = vec4(col, alpha); + } +`; + +function ghostBodyMaterial(THREE, colorKey, minY, maxY) { + const g = GHOST_GRADIENTS[colorKey] || GHOST_GRADIENTS.Blue; + return new THREE.ShaderMaterial({ + uniforms: { + uTop: { value: new THREE.Color(g.bottom) }, // brighter accent at top + uBottom: { value: new THREE.Color(g.top) }, // deeper hue at base + uOpacity: { value: 0.82 }, + uMinY: { value: minY }, + uMaxY: { value: maxY }, + uTime: { value: 0 }, + }, + vertexShader: GHOST_VERT, + fragmentShader: GHOST_FRAG, + transparent: true, + depthWrite: false, // translucent: don't occlude its own backfaces oddly + side: THREE.DoubleSide, + blending: THREE.NormalBlending, + }); +} + +export function buildLegoHeadGhost(THREE, colorKey = 'Blue') { + const group = new THREE.Group(); + + // --- dimensions in mm (classic minifig head proportions, ghost-scaled) --- + const R = 22; // head radius + const BODY_H = 46; // cylinder height + const DOME_H = 16; // top dome rise + const STUD_R = 9; + const STUD_H = 9; + + const minY = -BODY_H / 2 - 26; // include tail reach for gradient base + const maxY = BODY_H / 2 + DOME_H + STUD_H; + const bodyMat = ghostBodyMaterial(THREE, colorKey, minY, maxY); + + const mats = [bodyMat]; // track shader mats for time uniform + + // cylinder body + const body = new THREE.Mesh( + new THREE.CylinderGeometry(R, R, BODY_H, 40, 1, true), + bodyMat + ); + group.add(body); + + // domed top — half-sphere squashed to DOME_H + const dome = new THREE.Mesh( + new THREE.SphereGeometry(R, 40, 20, 0, Math.PI * 2, 0, Math.PI / 2), + bodyMat + ); + dome.scale.set(1, DOME_H / R, 1); + dome.position.y = BODY_H / 2; + group.add(dome); + + // bottom cap (rounded) so the base reads solid + const cap = new THREE.Mesh( + new THREE.SphereGeometry(R, 40, 16, 0, Math.PI * 2, Math.PI / 2, Math.PI / 2), + bodyMat + ); + cap.scale.set(1, 0.35, 1); + cap.position.y = -BODY_H / 2; + group.add(cap); + + // the stud on top — the unmistakable LEGO tell + const stud = new THREE.Mesh( + new THREE.CylinderGeometry(STUD_R, STUD_R, STUD_H, 28), + bodyMat + ); + stud.position.y = BODY_H / 2 + DOME_H + STUD_H / 2 - 2; + group.add(stud); + + // --- eyes (original, simple) — solid dark ovals on the front (+Z) face --- + const eyeMat = new THREE.MeshBasicMaterial({ color: 0x081026 }); + const glintMat = new THREE.MeshBasicMaterial({ color: 0xcfeeff }); + function eye(sign) { + const e = new THREE.Group(); + const ball = new THREE.Mesh(new THREE.SphereGeometry(5.2, 20, 16), eyeMat); + ball.scale.set(0.8, 1.15, 0.5); + e.add(ball); + const glint = new THREE.Mesh(new THREE.SphereGeometry(1.5, 10, 8), glintMat); + glint.position.set(sign * -0.8, 2.0, 4.6); + e.add(glint); + e.position.set(sign * 9, 2, R - 2); + return e; + } + const eyes = new THREE.Group(); + eyes.add(eye(-1), eye(1)); + group.add(eyes); + + // --- wispy tail: a few translucent "drips" below the body --- + const tail = new THREE.Group(); + const tailMat = bodyMat; + const drips = 4; + for (let i = 0; i < drips; i++) { + const dr = R * (0.5 - i * 0.06); + const drip = new THREE.Mesh( + new THREE.ConeGeometry(dr * 0.6, 22 + i * 2, 18, 1, true), + tailMat + ); + const ang = (i / drips) * Math.PI * 2; + drip.position.set(Math.cos(ang) * R * 0.5, -BODY_H / 2 - 8, Math.sin(ang) * R * 0.5); + drip.userData.phase = ang; + tail.add(drip); + } + group.add(tail); + + // --- idle animation --- + let baseY = 0; + group.userData.colorKey = colorKey; + group.userData.materials = mats; + group.userData.update = function (t) { + for (const m of mats) m.uniforms.uTime.value = t; + // bob + sway + group.position.y = baseY + Math.sin(t * 1.8) * 6; + eyes.rotation.y = Math.sin(t * 0.6) * 0.15; + // tail drips waggle + tail.children.forEach((d, i) => { + d.position.y = -BODY_H / 2 - 8 + Math.sin(t * 3 + d.userData.phase) * 3; + d.rotation.z = Math.sin(t * 2 + i) * 0.12; + }); + // occasional blink (scale eyes flat briefly) + const blink = (Math.sin(t * 0.9) > 0.985) ? 0.1 : 1.0; + eyes.scale.y = eyes.scale.y + (blink - eyes.scale.y) * 0.4; + }; + group.userData.setBaseY = (y) => { baseY = y; }; + + // recolour at runtime (e.g. when server assigns a ghost's lure colour) + group.userData.setColor = function (key) { + const g = GHOST_GRADIENTS[key] || GHOST_GRADIENTS.Blue; + bodyMat.uniforms.uTop.value.set(g.bottom); + bodyMat.uniforms.uBottom.value.set(g.top); + group.userData.colorKey = key; + }; + + return group; +}