AR: sub-pixel corner refinement, ROI-first detection, anchor coplanarity check

.claude-context/00-overview.md

@@ -42,4 +42,5 @@ volume, so live state carries over and rollback is one command.
 
 ## Session Log
 
+- 2026-10-07 — Corner Precision and ROI Detection — sub-pixel corner refinement (own implementation; `cornerSubPix` is absent from the pinned build), ROI-first detection cropping from the native video, and an anchor-coplanarity check surfaced in the HUD.
 - 2026-09-23 — AR Tracking Overhaul: Worker Offload, Camera Model, and Coplanar Pose Ambiguity — moved OpenCV off the main thread, fixed the focal/FOV model for portrait streams, deduped marker ids, gated the mirrored coplanar pose, and built a two-page A3 test rig with a foldable marker tower.

.claude-context/01-architecture.md

@@ -6,8 +6,11 @@ Current as of 2026-09-23. Supersedes the v2 tracking design throughout.
 
 ```
 camera video
-  -> detection canvas (long side capped, ~20Hz)
-  -> ArUco decode, maxHamming 0, filtered to scene anchor ids, deduped by id
+  -> ROI crops around last-seen crests, at NATIVE resolution   (steady state)
+     or full detection canvas, long side capped                (acquire / every 600ms / on loss)
+  -> ArUco decode, maxHamming 0
+  -> sub-pixel corner refinement (gradient centroid per border edge)
+  -> filtered to scene anchor ids, deduped by id
   -> TRACKING ENGINE  ->  camera pose in world
   -> three.js render: server-driven ghosts + depth-only occluder meshes
 ```
@@ -45,6 +48,69 @@ Pinned to `4.12.0-release.1`. Note for future work: this build lacks
 `solvePnPGeneric` and `cornerSubPix`, which is why pose ambiguity is handled by
 gating rather than by comparing both IPPE solutions.
 
+## Detection: ROI first, full frame as fallback
+
+The crests do not move; the phone does. Once a crest has been seen, the next
+tick crops around where it was and decodes that, rather than rescanning 518k
+pixels to re-find something that shifted by a few. Crops are taken from the
+**native** video, so each is both smaller than the corresponding patch of a full
+scan and higher resolution than it (`@1.3x` on a 1280 stream against a 960
+canvas). Five typical crests come to ~40k pixels against 518k.
+
+Mapping back into detection space is two scale changes and an origin shift per
+crop, which is exactly the kind of arithmetic that fails silently and puts
+ghosts a centimetre off. It is tested directly in `tools/test-roi.mjs`.
+
+A full scan runs on first acquisition, when targets go stale, at least every
+600ms, when there are more than 6 targets, and when the crops would cover more
+than 55% of the frame. **If an ROI pass returns fewer markers than it had
+targets, a full scan runs in the same tick** — dropping from 4 crests to 2 is
+precisely what makes the pose ambiguous, so it is never allowed to happen
+silently. Worst case is one wasted crop pass. Off: `localStorage.nbx.roi = 0`.
+
+## Corner precision: sub-pixel refinement
+
+js-aruco2 returns corners on the detection canvas lattice. At 960px a 60mm crest
+spans ~140px, so one pixel of corner error is ~0.7% of the marker — millimetres
+of ghost placement, re-rolled every frame.
+
+`subpix.js` refines each quad by fitting its four border edges rather than by
+looking at corners. Along each edge it scans perpendicular at ~12 points, takes
+the **first moment of the gradient** at each (not the gradient peak), fits a line
+through the crossings by total least squares with one outlier pass, and
+intersects consecutive lines.
+
+The centroid matters. Parabolic interpolation of the gradient peak — the obvious
+method, and what a `cornerSubPix`-style operator reduces to — is biased by
+sampling phase, because the gradient of an anti-aliased step is roughly
+triangular. Measured against synthetic edges it carried up to 0.29px of bias
+sharp and 0.75px defocused, i.e. **worse than the whole-pixel corners it was
+meant to improve**. The gradient centroid is unbiased for any symmetric edge
+profile: 0.018px on the same test. End to end on synthetic markers, RMS corner
+error drops ~12x (`tools/test-subpix.mjs`).
+
+`cv.cornerSubPix` was not an option: it is absent from the pinned build. Probed
+directly rather than assumed — `solvePnPRefineLM` IS present, though it refines
+a pose from corners and so cannot help here.
+
+Refinement is all-or-nothing per marker and falls back to the raw corners if any
+edge is too short, too flat, too diffuse, or if a corner would move more than
+3px. A bad refinement is worse than none, because the pose gates are tuned
+against honest corner noise. Off: `localStorage.nbx.subpix = 0`.
+
+## Anchor conditioning
+
+`geom.js` measures, once per scene load, the RMS distance of every anchor corner
+from the best-fit plane through all of them, and the HUD shows it. Below 1.5cm
+the layout is effectively coplanar and the mirrored pose is live — the physics
+gates below become load-bearing rather than a safety net.
+
+This exists because "the anchors are coplanar" is otherwise invisible: it looks
+normal in the JSON and shows up days later as ghosts intermittently flipping
+across the table. A layout re-saved by an editor that does not understand wall
+anchors can silently become flat. Reference values: A3 rig with the tower ~3.5cm,
+a town layout with one wall crest at 8cm ~2.7cm, any all-tabletop layout 0.0cm.
+
 ## Camera model
 
 Everything derives from one assumption — `HFOV_DEG` (default 60, override via

.claude-context/sessions/2026-10-07-corner-precision-roi.md

@@ -0,0 +1,154 @@
+# Corner Precision and ROI Detection
+
+**Date:** 2026-10-07
+**Project:** newbury-exhibit-v3
+**Status:** written and unit-tested offline; NOT yet validated on device
+**Tags:** ar, aruco, subpixel, roi, performance, coplanarity, opencv
+
+## Context
+
+Three improvements picked off a longer list: cheaper detection, more accurate
+corners, and closing out the coplanar-ambiguity work started on 2026-09-23. Two
+of the three changed shape once the actual code was read.
+
+## Key Decisions
+
+- **`cv.cornerSubPix` is genuinely absent from the pinned build** — the
+  2026-09-23 note was right. Verified properly this time rather than inferred:
+  the binding names live in the embedded WASM, not in the JS glue, so grepping
+  `opencv.js` reports nothing either way. Loading the module under node and
+  reading the exports is the only reliable check.
+
+  Also found: **`solvePnPRefineLM` IS exported.** Not useful here (it refines a
+  pose from corners, not the corners), but worth knowing it exists.
+  `solvePnPGeneric` remains absent, so the two-IPPE-solutions route to ambiguity
+  resolution is still closed.
+
+- **Wrote sub-pixel refinement rather than going without.** Edge fitting, not
+  corner finding: scan perpendicular across each border edge at ~12 points, find
+  the crossing, fit a line, intersect consecutive lines. Averaging twelve
+  crossings along a 140px edge beats any single window near a corner, which is
+  the least reliable part of a printed marker anyway.
+
+- **Gradient CENTROID, not gradient peak.** The first implementation used
+  parabolic interpolation of the gradient peak and made things *worse* — RMS
+  corner error went from 0.57px to 0.73px. The gradient of an anti-aliased step
+  is roughly triangular and a parabola fits it badly, so the estimate is pulled
+  toward the sample grid: up to 0.29px of phase-dependent bias sharp, 0.75px
+  defocused. The first moment of |gradient| has no such bias — for any symmetric
+  edge profile the centroid lands on the edge. Measured on 1-D synthetic edges:
+  parabolic 0.20px RMS, centroid 0.018px.
+
+  *Rejected:* half-maximum crossing (0.066px, needs clean plateaus); centroid of
+  the principal gradient lobe (tried as a defence against a second transition in
+  the window — it made the clean case 4x worse and did not fix the case it was
+  for). The second transition is handled by capping the scan at 0.42 of a marker
+  cell, so the nearest other edge is 2.4 scan-radii away, plus a gradient-spread
+  gate that drops the sample when the window is not looking at one clean edge.
+
+- **Two re-centring iterations per scan.** Costs a second pass; worth it on
+  small markers (0.063px -> 0.018px) because it makes the window symmetric about
+  the edge rather than about the initial guess.
+
+- **Refinement is all-or-nothing per marker.** Any failed edge, near-parallel
+  intersection, or corner moving more than 3px returns the originals untouched.
+  The pose gates in `cv-worker.js` are tuned against honest corner noise; a
+  confidently wrong corner is worse than a quantised one.
+
+- **ROI detection instead of lowering detection resolution.** The original idea
+  was to detect at 640 instead of 960 for the CPU saving. Reading the 2026-09-23
+  note killed it: decode failures are already an open question and the
+  arithmetic showing 960 is adequate was already done. Lowering resolution would
+  have pushed on exactly the wrong variable. Cropping gets the same CPU saving
+  and moves resolution the *other* way.
+
+- **Crops come from the native video, not the detection canvas.** A crop is then
+  both smaller and more detailed than the same patch of a full scan (1.33x on a
+  1280 stream at 960). A tight crop also gives the adaptive threshold a much
+  narrower brightness range, which is the other live candidate for the decode
+  failures.
+
+- **The crop scale is floored at detection resolution.** First version capped
+  the crop's long side at 224px unconditionally, which made a large near crest
+  come back at 0.41x — worse through an ROI than through a full scan. Caught by
+  the round-trip test, which prints the upscale factor for exactly this reason.
+
+- **An incomplete ROI pass triggers a full scan in the same tick.** Fewer markers
+  than targets means a crest left its crop. Solving from the shrunken set is how
+  a 4-crest frame becomes an ambiguous 2-crest frame, so it is never allowed.
+
+- **Coplanarity is measured and shown, not assumed.** `geom.js` reports the RMS
+  distance of every anchor corner from the best-fit plane at scene load. The
+  2026-09-23 session identified non-coplanar geometry as the cure for the
+  mirrored pose and built the tower, but nothing tells you whether the *loaded
+  layout* actually has it — and the layout editor is known not to understand
+  wall anchors, so a save can silently flatten it.
+
+## Technical Details
+
+**`public/js/ar/subpix.js`** — `refineQuad(imageData, corners, opts)` and
+`refineMarkers(imageData, markers, opts)`. Samples bilinear luma straight from
+the RGBA buffer, so there is no grayscale pre-pass. ~800 taps per marker, under
+1% of detection cost. Key options: `scanCellFrac 0.42` (scan radius as a
+fraction of one marker cell, assuming 6 cells across for ARUCO_4X4),
+`recentreIters 2`, `sigmaMaxPx 2.4`, `maxShiftPx 3.0`.
+
+**`public/js/ar/roi.js`** — `RoiScanner`. Full scan when: no targets, targets
+older than 400ms, 600ms since the last full scan, more than 6 targets, or crops
+would cover more than 55% of the frame. Crop margin 40% of the bbox (min 14px),
+long side soft-capped at 224px with a hard floor at detection resolution.
+
+**`public/js/ar/geom.js`** — `planarity(flatPoints)` via Jacobi eigen-decomposition
+of the 3x3 covariance; smallest eigenvector is the plane normal. `COPLANAR_CM = 1.5`.
+Reference: all-flat layout 0.0cm, A3 rig with tower 3.5cm, town with one wall
+crest at 8cm 2.7cm.
+
+**`public/js/ar/detect.js`** — split into `detectRaw` (decode any image) and
+`finaliseMarkers` (dedupe, scene filter, diagnostics) so both paths finalise
+identically. `detectMarkers` kept as a wrapper; `calibrate.html` still uses it.
+
+**`public/js/ar/solve-cv.js`** — `setScene` now also computes `this.planarity`,
+which warms the `cornersFor` cache as a side effect.
+
+**New localStorage knobs:** `nbx.subpix` (default 1), `nbx.roi` (default 1).
+Setting both to 0 restores the previous behaviour exactly — that is the A/B.
+
+**New HUD lines:**
+```
+scan roi 4x @1.3x 38kpx   subpix 4/4 0.21px
+geom spread 10.4cm (max 16.0)
+```
+
+**Measured, offline, synthetic:**
+```
+tools/test-subpix.mjs   RMS corner error 0.569px -> 0.046px   (12.4x, 192/192 accepted)
+tools/test-roi.mjs      crop -> detection space mapping exact to 1e-9 over 9 cases
+```
+
+A harness bug is worth recording: the first run showed refinement making things
+worse because the test renderer integrated pixel `(i,j)` over `[i, i+1]` while
+`subpix.js` samples with pixel centres on integers, the convention js-aruco2
+corners and OpenCV intrinsics both use. Half a pixel, and it pointed the finger
+at the wrong file.
+
+## Open Questions / Next Steps
+
+- **Validate on device.** Success: `reproj` falls (sub-pixel corners should show
+  up directly in it); `scan` reads `roi Nx` most ticks with `kpx` well under
+  518; `subpix` shows `N/N` with a mean shift of roughly 0.2-0.5px; `geom`
+  reports the tower's ~3.5cm.
+- **Does ROI fix the decode-rate problem?** Compare `markers` seen with
+  `nbx.roi=1` against `nbx.roi=0` on the same framing. If crops decode crests
+  that full scans miss, the cause was local contrast, not resolution — and
+  `nbx.detectLong=1280` need never be tried.
+- **Re-tune the 1€ filter.** Cutoffs in `solve-cv.js` were set for POS-IT-era
+  corner noise and then raised for the board solve. With corner error down ~12x
+  there is probably room to raise them again, trading smoothing for latency.
+- **Re-tune `MAX_REPROJ_PX`.** Currently 4, set against quantised corners. If
+  reproj now sits well below that, tightening it makes the gate more selective
+  and the mirrored pose easier to reject.
+- **Carry a raised crest into the real town layout**, not just the A3 rig, and
+  watch the geom line to confirm it survived the editor.
+- Everything in the 2026-09-23 open-questions list still stands: openresty
+  cache/WS headers, layout editor vs cylinder occluders and wall anchors,
+  deleting the posit engine.