155 lines
8.2 KiB
Markdown
155 lines
8.2 KiB
Markdown
# Corner Precision and ROI Detection
|
|
|
|
**Date:** 2026-10-07
|
|
**Project:** newbury-exhibit-v3
|
|
**Status:** written and unit-tested offline; NOT yet validated on device
|
|
**Tags:** ar, aruco, subpixel, roi, performance, coplanarity, opencv
|
|
|
|
## Context
|
|
|
|
Three improvements picked off a longer list: cheaper detection, more accurate
|
|
corners, and closing out the coplanar-ambiguity work started on 2026-09-23. Two
|
|
of the three changed shape once the actual code was read.
|
|
|
|
## Key Decisions
|
|
|
|
- **`cv.cornerSubPix` is genuinely absent from the pinned build** — the
|
|
2026-09-23 note was right. Verified properly this time rather than inferred:
|
|
the binding names live in the embedded WASM, not in the JS glue, so grepping
|
|
`opencv.js` reports nothing either way. Loading the module under node and
|
|
reading the exports is the only reliable check.
|
|
|
|
Also found: **`solvePnPRefineLM` IS exported.** Not useful here (it refines a
|
|
pose from corners, not the corners), but worth knowing it exists.
|
|
`solvePnPGeneric` remains absent, so the two-IPPE-solutions route to ambiguity
|
|
resolution is still closed.
|
|
|
|
- **Wrote sub-pixel refinement rather than going without.** Edge fitting, not
|
|
corner finding: scan perpendicular across each border edge at ~12 points, find
|
|
the crossing, fit a line, intersect consecutive lines. Averaging twelve
|
|
crossings along a 140px edge beats any single window near a corner, which is
|
|
the least reliable part of a printed marker anyway.
|
|
|
|
- **Gradient CENTROID, not gradient peak.** The first implementation used
|
|
parabolic interpolation of the gradient peak and made things *worse* — RMS
|
|
corner error went from 0.57px to 0.73px. The gradient of an anti-aliased step
|
|
is roughly triangular and a parabola fits it badly, so the estimate is pulled
|
|
toward the sample grid: up to 0.29px of phase-dependent bias sharp, 0.75px
|
|
defocused. The first moment of |gradient| has no such bias — for any symmetric
|
|
edge profile the centroid lands on the edge. Measured on 1-D synthetic edges:
|
|
parabolic 0.20px RMS, centroid 0.018px.
|
|
|
|
*Rejected:* half-maximum crossing (0.066px, needs clean plateaus); centroid of
|
|
the principal gradient lobe (tried as a defence against a second transition in
|
|
the window — it made the clean case 4x worse and did not fix the case it was
|
|
for). The second transition is handled by capping the scan at 0.42 of a marker
|
|
cell, so the nearest other edge is 2.4 scan-radii away, plus a gradient-spread
|
|
gate that drops the sample when the window is not looking at one clean edge.
|
|
|
|
- **Two re-centring iterations per scan.** Costs a second pass; worth it on
|
|
small markers (0.063px -> 0.018px) because it makes the window symmetric about
|
|
the edge rather than about the initial guess.
|
|
|
|
- **Refinement is all-or-nothing per marker.** Any failed edge, near-parallel
|
|
intersection, or corner moving more than 3px returns the originals untouched.
|
|
The pose gates in `cv-worker.js` are tuned against honest corner noise; a
|
|
confidently wrong corner is worse than a quantised one.
|
|
|
|
- **ROI detection instead of lowering detection resolution.** The original idea
|
|
was to detect at 640 instead of 960 for the CPU saving. Reading the 2026-09-23
|
|
note killed it: decode failures are already an open question and the
|
|
arithmetic showing 960 is adequate was already done. Lowering resolution would
|
|
have pushed on exactly the wrong variable. Cropping gets the same CPU saving
|
|
and moves resolution the *other* way.
|
|
|
|
- **Crops come from the native video, not the detection canvas.** A crop is then
|
|
both smaller and more detailed than the same patch of a full scan (1.33x on a
|
|
1280 stream at 960). A tight crop also gives the adaptive threshold a much
|
|
narrower brightness range, which is the other live candidate for the decode
|
|
failures.
|
|
|
|
- **The crop scale is floored at detection resolution.** First version capped
|
|
the crop's long side at 224px unconditionally, which made a large near crest
|
|
come back at 0.41x — worse through an ROI than through a full scan. Caught by
|
|
the round-trip test, which prints the upscale factor for exactly this reason.
|
|
|
|
- **An incomplete ROI pass triggers a full scan in the same tick.** Fewer markers
|
|
than targets means a crest left its crop. Solving from the shrunken set is how
|
|
a 4-crest frame becomes an ambiguous 2-crest frame, so it is never allowed.
|
|
|
|
- **Coplanarity is measured and shown, not assumed.** `geom.js` reports the RMS
|
|
distance of every anchor corner from the best-fit plane at scene load. The
|
|
2026-09-23 session identified non-coplanar geometry as the cure for the
|
|
mirrored pose and built the tower, but nothing tells you whether the *loaded
|
|
layout* actually has it — and the layout editor is known not to understand
|
|
wall anchors, so a save can silently flatten it.
|
|
|
|
## Technical Details
|
|
|
|
**`public/js/ar/subpix.js`** — `refineQuad(imageData, corners, opts)` and
|
|
`refineMarkers(imageData, markers, opts)`. Samples bilinear luma straight from
|
|
the RGBA buffer, so there is no grayscale pre-pass. ~800 taps per marker, under
|
|
1% of detection cost. Key options: `scanCellFrac 0.42` (scan radius as a
|
|
fraction of one marker cell, assuming 6 cells across for ARUCO_4X4),
|
|
`recentreIters 2`, `sigmaMaxPx 2.4`, `maxShiftPx 3.0`.
|
|
|
|
**`public/js/ar/roi.js`** — `RoiScanner`. Full scan when: no targets, targets
|
|
older than 400ms, 600ms since the last full scan, more than 6 targets, or crops
|
|
would cover more than 55% of the frame. Crop margin 40% of the bbox (min 14px),
|
|
long side soft-capped at 224px with a hard floor at detection resolution.
|
|
|
|
**`public/js/ar/geom.js`** — `planarity(flatPoints)` via Jacobi eigen-decomposition
|
|
of the 3x3 covariance; smallest eigenvector is the plane normal. `COPLANAR_CM = 1.5`.
|
|
Reference: all-flat layout 0.0cm, A3 rig with tower 3.5cm, town with one wall
|
|
crest at 8cm 2.7cm.
|
|
|
|
**`public/js/ar/detect.js`** — split into `detectRaw` (decode any image) and
|
|
`finaliseMarkers` (dedupe, scene filter, diagnostics) so both paths finalise
|
|
identically. `detectMarkers` kept as a wrapper; `calibrate.html` still uses it.
|
|
|
|
**`public/js/ar/solve-cv.js`** — `setScene` now also computes `this.planarity`,
|
|
which warms the `cornersFor` cache as a side effect.
|
|
|
|
**New localStorage knobs:** `nbx.subpix` (default 1), `nbx.roi` (default 1).
|
|
Setting both to 0 restores the previous behaviour exactly — that is the A/B.
|
|
|
|
**New HUD lines:**
|
|
```
|
|
scan roi 4x @1.3x 38kpx subpix 4/4 0.21px
|
|
geom spread 10.4cm (max 16.0)
|
|
```
|
|
|
|
**Measured, offline, synthetic:**
|
|
```
|
|
tools/test-subpix.mjs RMS corner error 0.569px -> 0.046px (12.4x, 192/192 accepted)
|
|
tools/test-roi.mjs crop -> detection space mapping exact to 1e-9 over 9 cases
|
|
```
|
|
|
|
A harness bug is worth recording: the first run showed refinement making things
|
|
worse because the test renderer integrated pixel `(i,j)` over `[i, i+1]` while
|
|
`subpix.js` samples with pixel centres on integers, the convention js-aruco2
|
|
corners and OpenCV intrinsics both use. Half a pixel, and it pointed the finger
|
|
at the wrong file.
|
|
|
|
## Open Questions / Next Steps
|
|
|
|
- **Validate on device.** Success: `reproj` falls (sub-pixel corners should show
|
|
up directly in it); `scan` reads `roi Nx` most ticks with `kpx` well under
|
|
518; `subpix` shows `N/N` with a mean shift of roughly 0.2-0.5px; `geom`
|
|
reports the tower's ~3.5cm.
|
|
- **Does ROI fix the decode-rate problem?** Compare `markers` seen with
|
|
`nbx.roi=1` against `nbx.roi=0` on the same framing. If crops decode crests
|
|
that full scans miss, the cause was local contrast, not resolution — and
|
|
`nbx.detectLong=1280` need never be tried.
|
|
- **Re-tune the 1€ filter.** Cutoffs in `solve-cv.js` were set for POS-IT-era
|
|
corner noise and then raised for the board solve. With corner error down ~12x
|
|
there is probably room to raise them again, trading smoothing for latency.
|
|
- **Re-tune `MAX_REPROJ_PX`.** Currently 4, set against quantised corners. If
|
|
reproj now sits well below that, tightening it makes the gate more selective
|
|
and the mirrored pose easier to reject.
|
|
- **Carry a raised crest into the real town layout**, not just the A3 rig, and
|
|
watch the geom line to confirm it survived the editor.
|
|
- Everything in the 2026-09-23 open-questions list still stands: openresty
|
|
cache/WS headers, layout editor vs cylinder occluders and wall anchors,
|
|
deleting the posit engine.
|