AR: sub-pixel corner refinement, ROI-first detection, anchor coplanarity check

.claude-context/sessions/2026-10-07-corner-precision-roi.md

@@ -0,0 +1,154 @@
+# Corner Precision and ROI Detection
+
+**Date:** 2026-10-07
+**Project:** newbury-exhibit-v3
+**Status:** written and unit-tested offline; NOT yet validated on device
+**Tags:** ar, aruco, subpixel, roi, performance, coplanarity, opencv
+
+## Context
+
+Three improvements picked off a longer list: cheaper detection, more accurate
+corners, and closing out the coplanar-ambiguity work started on 2026-09-23. Two
+of the three changed shape once the actual code was read.
+
+## Key Decisions
+
+- **`cv.cornerSubPix` is genuinely absent from the pinned build** — the
+  2026-09-23 note was right. Verified properly this time rather than inferred:
+  the binding names live in the embedded WASM, not in the JS glue, so grepping
+  `opencv.js` reports nothing either way. Loading the module under node and
+  reading the exports is the only reliable check.
+
+  Also found: **`solvePnPRefineLM` IS exported.** Not useful here (it refines a
+  pose from corners, not the corners), but worth knowing it exists.
+  `solvePnPGeneric` remains absent, so the two-IPPE-solutions route to ambiguity
+  resolution is still closed.
+
+- **Wrote sub-pixel refinement rather than going without.** Edge fitting, not
+  corner finding: scan perpendicular across each border edge at ~12 points, find
+  the crossing, fit a line, intersect consecutive lines. Averaging twelve
+  crossings along a 140px edge beats any single window near a corner, which is
+  the least reliable part of a printed marker anyway.
+
+- **Gradient CENTROID, not gradient peak.** The first implementation used
+  parabolic interpolation of the gradient peak and made things *worse* — RMS
+  corner error went from 0.57px to 0.73px. The gradient of an anti-aliased step
+  is roughly triangular and a parabola fits it badly, so the estimate is pulled
+  toward the sample grid: up to 0.29px of phase-dependent bias sharp, 0.75px
+  defocused. The first moment of |gradient| has no such bias — for any symmetric
+  edge profile the centroid lands on the edge. Measured on 1-D synthetic edges:
+  parabolic 0.20px RMS, centroid 0.018px.
+
+  *Rejected:* half-maximum crossing (0.066px, needs clean plateaus); centroid of
+  the principal gradient lobe (tried as a defence against a second transition in
+  the window — it made the clean case 4x worse and did not fix the case it was
+  for). The second transition is handled by capping the scan at 0.42 of a marker
+  cell, so the nearest other edge is 2.4 scan-radii away, plus a gradient-spread
+  gate that drops the sample when the window is not looking at one clean edge.
+
+- **Two re-centring iterations per scan.** Costs a second pass; worth it on
+  small markers (0.063px -> 0.018px) because it makes the window symmetric about
+  the edge rather than about the initial guess.
+
+- **Refinement is all-or-nothing per marker.** Any failed edge, near-parallel
+  intersection, or corner moving more than 3px returns the originals untouched.
+  The pose gates in `cv-worker.js` are tuned against honest corner noise; a
+  confidently wrong corner is worse than a quantised one.
+
+- **ROI detection instead of lowering detection resolution.** The original idea
+  was to detect at 640 instead of 960 for the CPU saving. Reading the 2026-09-23
+  note killed it: decode failures are already an open question and the
+  arithmetic showing 960 is adequate was already done. Lowering resolution would
+  have pushed on exactly the wrong variable. Cropping gets the same CPU saving
+  and moves resolution the *other* way.
+
+- **Crops come from the native video, not the detection canvas.** A crop is then
+  both smaller and more detailed than the same patch of a full scan (1.33x on a
+  1280 stream at 960). A tight crop also gives the adaptive threshold a much
+  narrower brightness range, which is the other live candidate for the decode
+  failures.
+
+- **The crop scale is floored at detection resolution.** First version capped
+  the crop's long side at 224px unconditionally, which made a large near crest
+  come back at 0.41x — worse through an ROI than through a full scan. Caught by
+  the round-trip test, which prints the upscale factor for exactly this reason.
+
+- **An incomplete ROI pass triggers a full scan in the same tick.** Fewer markers
+  than targets means a crest left its crop. Solving from the shrunken set is how
+  a 4-crest frame becomes an ambiguous 2-crest frame, so it is never allowed.
+
+- **Coplanarity is measured and shown, not assumed.** `geom.js` reports the RMS
+  distance of every anchor corner from the best-fit plane at scene load. The
+  2026-09-23 session identified non-coplanar geometry as the cure for the
+  mirrored pose and built the tower, but nothing tells you whether the *loaded
+  layout* actually has it — and the layout editor is known not to understand
+  wall anchors, so a save can silently flatten it.
+
+## Technical Details
+
+**`public/js/ar/subpix.js`** — `refineQuad(imageData, corners, opts)` and
+`refineMarkers(imageData, markers, opts)`. Samples bilinear luma straight from
+the RGBA buffer, so there is no grayscale pre-pass. ~800 taps per marker, under
+1% of detection cost. Key options: `scanCellFrac 0.42` (scan radius as a
+fraction of one marker cell, assuming 6 cells across for ARUCO_4X4),
+`recentreIters 2`, `sigmaMaxPx 2.4`, `maxShiftPx 3.0`.
+
+**`public/js/ar/roi.js`** — `RoiScanner`. Full scan when: no targets, targets
+older than 400ms, 600ms since the last full scan, more than 6 targets, or crops
+would cover more than 55% of the frame. Crop margin 40% of the bbox (min 14px),
+long side soft-capped at 224px with a hard floor at detection resolution.
+
+**`public/js/ar/geom.js`** — `planarity(flatPoints)` via Jacobi eigen-decomposition
+of the 3x3 covariance; smallest eigenvector is the plane normal. `COPLANAR_CM = 1.5`.
+Reference: all-flat layout 0.0cm, A3 rig with tower 3.5cm, town with one wall
+crest at 8cm 2.7cm.
+
+**`public/js/ar/detect.js`** — split into `detectRaw` (decode any image) and
+`finaliseMarkers` (dedupe, scene filter, diagnostics) so both paths finalise
+identically. `detectMarkers` kept as a wrapper; `calibrate.html` still uses it.
+
+**`public/js/ar/solve-cv.js`** — `setScene` now also computes `this.planarity`,
+which warms the `cornersFor` cache as a side effect.
+
+**New localStorage knobs:** `nbx.subpix` (default 1), `nbx.roi` (default 1).
+Setting both to 0 restores the previous behaviour exactly — that is the A/B.
+
+**New HUD lines:**
+```
+scan roi 4x @1.3x 38kpx   subpix 4/4 0.21px
+geom spread 10.4cm (max 16.0)
+```
+
+**Measured, offline, synthetic:**
+```
+tools/test-subpix.mjs   RMS corner error 0.569px -> 0.046px   (12.4x, 192/192 accepted)
+tools/test-roi.mjs      crop -> detection space mapping exact to 1e-9 over 9 cases
+```
+
+A harness bug is worth recording: the first run showed refinement making things
+worse because the test renderer integrated pixel `(i,j)` over `[i, i+1]` while
+`subpix.js` samples with pixel centres on integers, the convention js-aruco2
+corners and OpenCV intrinsics both use. Half a pixel, and it pointed the finger
+at the wrong file.
+
+## Open Questions / Next Steps
+
+- **Validate on device.** Success: `reproj` falls (sub-pixel corners should show
+  up directly in it); `scan` reads `roi Nx` most ticks with `kpx` well under
+  518; `subpix` shows `N/N` with a mean shift of roughly 0.2-0.5px; `geom`
+  reports the tower's ~3.5cm.
+- **Does ROI fix the decode-rate problem?** Compare `markers` seen with
+  `nbx.roi=1` against `nbx.roi=0` on the same framing. If crops decode crests
+  that full scans miss, the cause was local contrast, not resolution — and
+  `nbx.detectLong=1280` need never be tried.
+- **Re-tune the 1€ filter.** Cutoffs in `solve-cv.js` were set for POS-IT-era
+  corner noise and then raised for the board solve. With corner error down ~12x
+  there is probably room to raise them again, trading smoothing for latency.
+- **Re-tune `MAX_REPROJ_PX`.** Currently 4, set against quantised corners. If
+  reproj now sits well below that, tightening it makes the gate more selective
+  and the mirrored pose easier to reject.
+- **Carry a raised crest into the real town layout**, not just the A3 rig, and
+  watch the geom line to confirm it survived the editor.
+- Everything in the 2026-09-23 open-questions list still stands: openresty
+  cache/WS headers, layout editor vs cylinder occluders and wall anchors,
+  deleting the posit engine.