newbury-exhibit-v3/.claude-context/sessions/2026-10-07-corner-precision-roi.md

Corner Precision and ROI Detection

Date: 2026-10-07 Project: newbury-exhibit-v3 Status: written and unit-tested offline; NOT yet validated on device Tags: ar, aruco, subpixel, roi, performance, coplanarity, opencv

Context

Three improvements picked off a longer list: cheaper detection, more accurate corners, and closing out the coplanar-ambiguity work started on 2026-09-23. Two of the three changed shape once the actual code was read.

Key Decisions

• cv.cornerSubPix is genuinely absent from the pinned build — the 2026-09-23 note was right. Verified properly this time rather than inferred: the binding names live in the embedded WASM, not in the JS glue, so grepping opencv.js reports nothing either way. Loading the module under node and reading the exports is the only reliable check.

  Also found: solvePnPRefineLM IS exported. Not useful here (it refines a pose from corners, not the corners), but worth knowing it exists. solvePnPGeneric remains absent, so the two-IPPE-solutions route to ambiguity resolution is still closed.

• Wrote sub-pixel refinement rather than going without. Edge fitting, not corner finding: scan perpendicular across each border edge at ~12 points, find the crossing, fit a line, intersect consecutive lines. Averaging twelve crossings along a 140px edge beats any single window near a corner, which is the least reliable part of a printed marker anyway.

• Gradient CENTROID, not gradient peak. The first implementation used parabolic interpolation of the gradient peak and made things worse — RMS corner error went from 0.57px to 0.73px. The gradient of an anti-aliased step is roughly triangular and a parabola fits it badly, so the estimate is pulled toward the sample grid: up to 0.29px of phase-dependent bias sharp, 0.75px defocused. The first moment of |gradient| has no such bias — for any symmetric edge profile the centroid lands on the edge. Measured on 1-D synthetic edges: parabolic 0.20px RMS, centroid 0.018px.

  Rejected: half-maximum crossing (0.066px, needs clean plateaus); centroid of the principal gradient lobe (tried as a defence against a second transition in the window — it made the clean case 4x worse and did not fix the case it was for). The second transition is handled by capping the scan at 0.42 of a marker cell, so the nearest other edge is 2.4 scan-radii away, plus a gradient-spread gate that drops the sample when the window is not looking at one clean edge.

• Two re-centring iterations per scan. Costs a second pass; worth it on small markers (0.063px -> 0.018px) because it makes the window symmetric about the edge rather than about the initial guess.

• Refinement is all-or-nothing per marker. Any failed edge, near-parallel intersection, or corner moving more than 3px returns the originals untouched. The pose gates in cv-worker.js are tuned against honest corner noise; a confidently wrong corner is worse than a quantised one.

• ROI detection instead of lowering detection resolution. The original idea was to detect at 640 instead of 960 for the CPU saving. Reading the 2026-09-23 note killed it: decode failures are already an open question and the arithmetic showing 960 is adequate was already done. Lowering resolution would have pushed on exactly the wrong variable. Cropping gets the same CPU saving and moves resolution the other way.

• Crops come from the native video, not the detection canvas. A crop is then both smaller and more detailed than the same patch of a full scan (1.33x on a 1280 stream at 960). A tight crop also gives the adaptive threshold a much narrower brightness range, which is the other live candidate for the decode failures.

• The crop scale is floored at detection resolution. First version capped the crop's long side at 224px unconditionally, which made a large near crest come back at 0.41x — worse through an ROI than through a full scan. Caught by the round-trip test, which prints the upscale factor for exactly this reason.

• An incomplete ROI pass triggers a full scan in the same tick. Fewer markers than targets means a crest left its crop. Solving from the shrunken set is how a 4-crest frame becomes an ambiguous 2-crest frame, so it is never allowed.

• Coplanarity is measured and shown, not assumed. geom.js reports the RMS distance of every anchor corner from the best-fit plane at scene load. The 2026-09-23 session identified non-coplanar geometry as the cure for the mirrored pose and built the tower, but nothing tells you whether the loaded layout actually has it — and the layout editor is known not to understand wall anchors, so a save can silently flatten it.

Technical Details

public/js/ar/subpix.js — refineQuad(imageData, corners, opts) and refineMarkers(imageData, markers, opts). Samples bilinear luma straight from the RGBA buffer, so there is no grayscale pre-pass. ~800 taps per marker, under 1% of detection cost. Key options: scanCellFrac 0.42 (scan radius as a fraction of one marker cell, assuming 6 cells across for ARUCO_4X4), recentreIters 2, sigmaMaxPx 2.4, maxShiftPx 3.0.

public/js/ar/roi.js — RoiScanner. Full scan when: no targets, targets older than 400ms, 600ms since the last full scan, more than 6 targets, or crops would cover more than 55% of the frame. Crop margin 40% of the bbox (min 14px), long side soft-capped at 224px with a hard floor at detection resolution.

public/js/ar/geom.js — planarity(flatPoints) via Jacobi eigen-decomposition of the 3x3 covariance; smallest eigenvector is the plane normal. COPLANAR_CM = 1.5. Reference: all-flat layout 0.0cm, A3 rig with tower 3.5cm, town with one wall crest at 8cm 2.7cm.

public/js/ar/detect.js — split into detectRaw (decode any image) and finaliseMarkers (dedupe, scene filter, diagnostics) so both paths finalise identically. detectMarkers kept as a wrapper; calibrate.html still uses it.

public/js/ar/solve-cv.js — setScene now also computes this.planarity, which warms the cornersFor cache as a side effect.

New localStorage knobs: nbx.subpix (default 1), nbx.roi (default 1). Setting both to 0 restores the previous behaviour exactly — that is the A/B.

New HUD lines:

scan roi 4x @1.3x 38kpx   subpix 4/4 0.21px
geom spread 10.4cm (max 16.0)

Measured, offline, synthetic:

tools/test-subpix.mjs   RMS corner error 0.569px -> 0.046px   (12.4x, 192/192 accepted)
tools/test-roi.mjs      crop -> detection space mapping exact to 1e-9 over 9 cases

A harness bug is worth recording: the first run showed refinement making things worse because the test renderer integrated pixel (i,j) over [i, i+1] while subpix.js samples with pixel centres on integers, the convention js-aruco2 corners and OpenCV intrinsics both use. Half a pixel, and it pointed the finger at the wrong file.

Open Questions / Next Steps

• Validate on device. Success: reproj falls (sub-pixel corners should show up directly in it); scan reads roi Nx most ticks with kpx well under 518; subpix shows N/N with a mean shift of roughly 0.2-0.5px; geom reports the tower's ~3.5cm.
• Does ROI fix the decode-rate problem? Compare markers seen with nbx.roi=1 against nbx.roi=0 on the same framing. If crops decode crests that full scans miss, the cause was local contrast, not resolution — and nbx.detectLong=1280 need never be tried.
• Re-tune the 1€ filter. Cutoffs in solve-cv.js were set for POS-IT-era corner noise and then raised for the board solve. With corner error down ~12x there is probably room to raise them again, trading smoothing for latency.
• Re-tune MAX_REPROJ_PX. Currently 4, set against quantised corners. If reproj now sits well below that, tightening it makes the gate more selective and the mirrored pose easier to reject.
• Carry a raised crest into the real town layout, not just the A3 rig, and watch the geom line to confirm it survived the editor.
• Everything in the 2026-09-23 open-questions list still stands: openresty cache/WS headers, layout editor vs cylinder occluders and wall anchors, deleting the posit engine.