react-native-image-stitcher 0.5.0 → 0.6.0

package/CHANGELOG.md

@@ -16,6 +16,177 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.6.0] — 2026-05-25
+
+> [!WARNING]
+> **Breaking changes.**  v0.6.0 retires the deprecated JS-driver
+> non-AR path that was marked for removal in v0.5.0's *Deprecated*
+> section.  Hosts using the default `<Camera>` flow (`legacyDriver`
+> unset) are not affected — they were already on
+> `useFrameProcessorDriver`.  Hosts that opted into the legacy
+> driver (`legacyDriver={true}` on `<Camera>`, or a direct
+> `useIncrementalJSDriver()` consumer) MUST migrate to the Frame
+> Processor driver — see *Migration from 0.5.x* below.
+
+### Removed (breaking)
+
+- **`useIncrementalJSDriver` hook** + its `UseIncrementalJSDriverOptions`
+  / `IncrementalJSDriverHandle` types.  Deprecated in v0.5.0; the
+  v0.5 deprecation warning has now been replaced by deletion.
+- **`legacyDriver?: boolean` prop on `<Camera>`**.  The escape hatch
+  back to the JS driver is gone.  Hosts that set this prop will
+  get a TS-level error; at runtime the prop is silently ignored.
+- **`frameSourceMode: 'jsDriver'`** enum value in
+  `IncrementalStartOptions`.  The TS type is now narrowed to
+  `'arSession' | 'frameProcessor'`.  Passing `'jsDriver'` is a
+  compile error; at the native bridge layer the value falls through
+  to the default (now `'arSession'`).
+- **`IncrementalStitcher.processFrameAtPath` native method** on both
+  iOS and Android.  The only JS caller was `useIncrementalJSDriver`,
+  also deleted.  Hosts calling
+  `NativeModules.IncrementalStitcher.processFrameAtPath(...)` via
+  raw `NativeModules` access will get a runtime "method does not
+  exist" error.  Use the Frame Processor driver instead.
+
+### Changed (breaking)
+
+- **Android `frameSourceMode` default switched from `"jsDriver"` to
+  `"arSession"`** for parity with iOS.  Raw `NativeModules` callers
+  that omitted `frameSourceMode` were previously getting an inert
+  capture (the "jsDriver" branch dropped all engine input on
+  Android since v0.5.0); they now get AR-mode behaviour, matching
+  iOS.  The production `<Camera>` is unaffected — it always passes
+  `frameSourceMode: 'arSession'` explicitly for AR captures.
+
+### Changed (non-breaking)
+
+- **`RNSARCameraView` (AR mode) no longer eager-encodes a JPEG per
+  ARCore frame.**  Migrated to the pixel-data path introduced for
+  the Frame Processor in v0.5.1's F8.6 work.  AR-mode captures now
+  pass `nv21PixelData` / `nv21PixelWidth` / `nv21PixelHeight`
+  through `ingestFromARCameraView`; `legacyJpegPath` is always null
+  on this path.  Expected gain on Galaxy A35: ~30-50 ms per
+  accepted frame, with the dominant savings on rejected frames
+  (no JPEG encode → no imread round-trip).  Closes the v0.5.0
+  follow-up.
+
+### Removed (internal cleanup; no external API impact)
+
+- **F8.6 perf-diagnostic logs** (`F8.6-route`, `F8.6-perf`)
+  introduced in v0.5.1 stripped from `IncrementalStitcher` +
+  `IncrementalFirstwinsEngine` — F8.6 is now baked in for
+  production and the diagnostic spam is no longer informative.
+- **Orphaned native helpers** dropped after `processFrameAtPath`
+  removal:
+  - iOS: `addBatchKeyframePath(path:pose:)`, `isBatchKeyframeMode`
+    getter, `decodeJpegToGrayscalePixelBuffer` (only callers were
+    `processFrameAtPath`).
+  - Android: `decodeJpegToGrayscale` + `GrayscaleFrame` data class,
+    `isBatchKeyframeMode` getter (only callers were
+    `processFrameAtPath` and the AR-mode eager-encode branch).
+- **Stale comments** referencing removed code paths swept across
+  Kotlin/Swift/Obj-C/TS.  Historical "removed in v0.6" markers
+  retained; comments that described live code in terms of the
+  removed names rewritten to describe current behaviour.
+
+### Migration from 0.5.x
+
+**Default `<Camera>` hosts (no `legacyDriver` prop set):** no
+action required.  `<Camera>` already used `useFrameProcessorDriver`
+in non-AR mode and `RNSARSession` in AR mode since v0.5.0.
+
+**Hosts with `legacyDriver={true}` on `<Camera>`:** remove the
+prop.  `<Camera>` will use the Frame Processor driver, which has
+been the default since v0.5.0 and the only path since this release.
+
+```tsx
+// Before (v0.5.x)
+<Camera legacyDriver={true} ... />
+
+// After (v0.6.0)
+<Camera ... />
+```
+
+**Hosts directly using `useIncrementalJSDriver`:** migrate to
+`useFrameProcessorDriver`.  The handle shape (`{ start, stop,
+frameProcessor, isRunning }`) is preserved, but the new hook is a
+Frame Processor + gyro driver instead of a `takeSnapshot` + JS
+interval driver.  See
+[`src/stitching/useFrameProcessorDriver.ts`](src/stitching/useFrameProcessorDriver.ts)
+for the migration mapping; the gyro pose synthesis convention
+(`q = q_yaw * q_pitch * q_roll`) is identical, so existing pose
+math at call sites continues to work.
+
+**Hosts passing `frameSourceMode: 'jsDriver'` to
+`incremental.start(...)`:** change to `'frameProcessor'`.  The
+TypeScript type now rejects `'jsDriver'` at compile time.
+
+## [0.5.1] — 2026-05-25
+
+### Added — F8.6 Android pixel-buffer engine parity
+
+Closes the v0.5.0 follow-up tracked in the [0.5.0] section.
+
+**Live engine ingest no longer requires a JPEG round-trip.**
+The `IncrementalFirstwinsEngine` (slit-scan / first-wins) and the
+hybrid `IncrementalEngine` both gained a new
+`addFramePixelData(nv21, w, h, ...)` method.  It builds the BGR
+`cv::Mat` in-process via
+`Imgproc.cvtColor(yuv, COLOR_YUV2BGR_NV21)`, then delegates to a
+newly-extracted shared `addFrameMat` helper that runs the original
+engine pipeline verbatim.  The legacy `addFrameAtPath(path, ...)`
+is now a thin wrapper: `imread → downsample → addFrameMat`.
+
+**Routing.**  `IncrementalStitcher.ingestFromARCameraView` got
+three optional parameters — `nv21PixelData: ByteArray?`,
+`nv21PixelWidth: Int`, `nv21PixelHeight: Int`.  When supplied (and
+`batchKeyframeMode == false`), the live engine ingests via
+`addFramePixelData`; otherwise falls back to `addFrameAtPath` with
+`legacyJpegPath`.  Backwards-compatible — all-null defaults
+preserve every existing caller.
+
+**Frame Processor wiring.**  `consumeFrameFromPlugin` now packs the
+incoming `Image` NV21 once at the top (was twice — gate consumed
+Y only, then the `onAccept` lambda re-packed for JPEG encode) and
+threads the bytes through to both the gate (which reads only the
+Y subset) AND the new `nv21PixelData` parameter.  Net: single
+`packNV21` per producer-thread frame.
+
+**Measured on Galaxy A35, `engine: 'firstwins-rectilinear'`,
+non-AR Frame Processor capture:**
+
+| Outcome | F8.6 pixel-data | Legacy JPEG path (estimated) |
+|---|---|---|
+| `AcceptedHigh` (first-frame init) | 7–11 ms | 50–70 ms |
+| `SkippedTooClose` (gate bail) | 0.5–2 ms | 50–60 ms (imread is unconditional) |
+
+`SkippedTooClose` dominates the producer-thread frame budget
+(~95% of frames at 30 fps with a slow pan).  Eliminating the
+imread on those frames is the bulk of the F8.6 win.
+
+### Added
+
+* New `<Camera engine={...}>` prop exposes the live engine
+  selection (`'batch-keyframe'` (default) / `'firstwins-rectilinear'`
+  / `'hybrid'` / `'slitscan-*'`).  Lets hosts opt into in-flight
+  stitching for low-latency previews; previously the choice was
+  hardcoded.
+
+### Changed
+
+* `New: F8.6 perf-diagnostic logs` (`F8.6-route`, `F8.6-perf`) fire
+  in live-engine mode only — inert under the default
+  `batch-keyframe`.  Will be removed in v0.6 once F8.6 is baked in
+  production.
+
+### Fixed
+
+* In `IncrementalStitcher.consumeFrameFromPlugin`, the `onAccept`
+  lambda was re-packing the live `Image` instead of reusing the
+  already-packed NV21 from the outer scope.  Now it reuses the
+  outer `packed` — saves a redundant `packNV21` call on every
+  accepted frame.
+
 ## [0.5.0] — 2026-05-25
 
 ### Added — F8 Frame Processor port
@@ -91,13 +262,29 @@ Frame Processor** on the camera producer thread instead of the
 
 ### Tracking — known follow-ups (don't gate this release)
 
-- **F8.6** — Android engine refactor for pixel-buffer-direct
-  ingest (true zero-copy parity with iOS).
-- **F8.3.H2-target** — `swift test` currently can't run the
-  iOS test target due to mixed Swift/.mm sources.  The
-  `FrameProcessorPluginSelectorTests` selector guard is in place
-  as a documentation artifact; CI test-runner fix is a separate
-  task.
+- **F8.6 (v0.5.1)** — Android engine refactor for pixel-buffer-
+  direct ingest (true zero-copy parity with iOS).  Would extract
+  an `addFrameMat` helper from `IncrementalFirstwinsEngine` and
+  `IncrementalEngine`'s `addFrameAtPath`, add a parallel
+  `addFramePixelData` that constructs the BGR `cv::Mat` from NV21
+  bytes via `cvtColor`, and rewire `RNSARCameraView` to skip the
+  per-frame JPEG encode.  Expected gain: ~30–50 ms per accepted
+  frame.  Deferred because the engine bodies are 400+ lines of
+  complex AR-mode code; needs A35 device verification before
+  merge, which the v0.5.0 prep session didn't have.
+
+- **F8.3-followup-roll** — resolved in v0.5.0.
+
+- **F8.3.H2-target** — RESOLVED in v0.5.0 via a different
+  mechanism than originally planned.  The selector pin is now a
+  compile-time `#selector(...)` reference inside
+  `IncrementalStitcher.swift` plus a dev-build runtime assert in
+  `IncrementalStitcher.init()` — both fire if the Swift method
+  signature drifts from what `KeyframeGateFrameProcessor.mm`
+  expects.  The obsolete test file was deleted.  `swift test` now
+  runs the (8-test) `QualityCheckerTests` suite cleanly because
+  `Package.swift` switched from an exclude list (broke every time
+  a new `.mm` landed) to an explicit `sources` allowlist.
 
 ## [0.4.1] — 2026-05-23
 
@@ -974,7 +1161,12 @@ Native module names also changed:
 - iOS pod: `RetaiLensCaptureSDK` → `RNImageStitcher`
 - iOS xcframework: shipped as `opencv2.xcframework` (linked from `RNImageStitcher.podspec`)
 
-[Unreleased]: https://github.com/bhargavkanda/react-native-image-stitcher/compare/v0.3.0...HEAD
+[Unreleased]: https://github.com/bhargavkanda/react-native-image-stitcher/compare/v0.6.0...HEAD
+[0.6.0]: https://github.com/bhargavkanda/react-native-image-stitcher/compare/v0.5.1...v0.6.0
+[0.5.1]: https://github.com/bhargavkanda/react-native-image-stitcher/compare/v0.5.0...v0.5.1
+[0.5.0]: https://github.com/bhargavkanda/react-native-image-stitcher/compare/v0.4.1...v0.5.0
+[0.4.1]: https://github.com/bhargavkanda/react-native-image-stitcher/compare/v0.4.0...v0.4.1
+[0.4.0]: https://github.com/bhargavkanda/react-native-image-stitcher/compare/v0.3.0...v0.4.0
 [0.3.0]: https://github.com/bhargavkanda/react-native-image-stitcher/compare/v0.2.1...v0.3.0
 [0.2.1]: https://github.com/bhargavkanda/react-native-image-stitcher/compare/v0.2.0...v0.2.1
 [0.2.0]: https://github.com/bhargavkanda/react-native-image-stitcher/compare/v0.1.3...v0.2.0

package/android/src/main/java/io/imagestitcher/rn/CvFlowGateFrameProcessor.kt

@@ -29,8 +29,8 @@ import com.mrousavy.camera.frameprocessors.VisionCameraProxy
  *   3. Call `IncrementalStitcher.consumeFrameFromPlugin(image, …)`
  *      which:
  *        - Drops the call if `frameSourceMode != "frameProcessor"`
- *          (prevents double-feeding the engine alongside the legacy
- *          `processFrameAtPath` path).
+ *          (prevents double-feeding the engine alongside the
+ *          AR-mode `ingestFromARCameraView` path).
  *        - Otherwise: extracts the Y plane, evaluates the keyframe
  *          gate via `KeyframeGate.evaluateWithFrame`, encodes the
  *          accepted frame to JPEG synchronously, and hands the path

package/android/src/main/java/io/imagestitcher/rn/IncrementalFirstwinsEngine.kt

@@ -197,7 +197,127 @@ internal class IncrementalFirstwinsEngine(
         }
         val frameBGR = downsampleToCompose(srcRaw)
         if (frameBGR !== srcRaw) srcRaw.release()
+        return addFrameMat(
+            frameBGR,
+            qx, qy, qz, qw,
+            fx, fy, cx, cy,
+            imageWidth, imageHeight,
+            yaw, pitch,
+            fovHorizDegrees, fovVertDegrees,
+            t0,
+        )
+    }
+
+    /**
+     * F8.6 — pixel-data twin of [addFrameAtPath].  Accepts the
+     * camera frame as an NV21 byte buffer instead of a JPEG file
+     * path; skips the JPEG decode round-trip (~30–50 ms per
+     * accepted frame on a mid-tier device).
+     *
+     * Use this from the vision-camera Frame Processor path (where
+     * we already have direct producer-thread access to YUV bytes)
+     * and from the ARCore path (where the previous
+     * JPEG-encode-on-every-frame in `RNSARCameraView` was a
+     * measurable hot-spot).
+     *
+     * The body matches [addFrameAtPath] one-for-one except for the
+     * Mat construction: an `Mat(h*3/2, w, CV_8UC1)` wraps the
+     * NV21 bytes, then `Imgproc.cvtColor` produces the BGR Mat the
+     * engine pipeline already expects.  Everything downstream is
+     * the shared [addFrameMat] helper.
+     *
+     * `nv21Width`/`nv21Height` describe the buffer's actual
+     * dimensions.  `imageWidth`/`imageHeight` describe the
+     * camera's reported sensor dims used for intrinsics scaling —
+     * these can differ when the camera is downsampling for Frame
+     * Processor output.
+     */
+    fun addFramePixelData(
+        nv21: ByteArray,
+        nv21Width: Int,
+        nv21Height: Int,
+        qx: Double, qy: Double, qz: Double, qw: Double,
+        fx: Double, fy: Double, cx: Double, cy: Double,
+        imageWidth: Int, imageHeight: Int,
+        yaw: Double, pitch: Double,
+        fovHorizDegrees: Double, fovVertDegrees: Double,
+        trackingPoor: Boolean,
+    ): FrameTelemetry {
+        val t0 = System.nanoTime()
+        if (trackingPoor) {
+            return FrameTelemetry(
+                FrameOutcome.SkippedTrackingPoor, -1.0, 0, yaw, pitch,
+                msSince(t0),
+                isLandscape = isLandscape,
+            )
+        }
+        // NV21 layout: Y plane (w*h bytes) + interleaved VU
+        // (w*h/2 bytes).  A single CV_8UC1 Mat of height h*3/2
+        // packs the whole thing; cvtColor with COLOR_YUV2BGR_NV21
+        // does the planar-aware decode in one call.
+        //
+        // F8.6 IS-1 — length guard.  If the caller supplied a
+        // short buffer, `yuv.put(0,0,nv21)` would copy only
+        // `nv21.size` bytes and leave the rest zero-init; cvtColor
+        // would then read stale/zero UV and produce silently
+        // corrupt colour.  Fail fast instead.
+        val expectedBytes = nv21Width * nv21Height * 3 / 2
+        require(nv21.size >= expectedBytes) {
+            "addFramePixelData: nv21 buffer too small " +
+                "(${nv21.size} bytes < $expectedBytes for " +
+                "${nv21Width}x${nv21Height})"
+        }
+        val yuv = Mat(nv21Height + nv21Height / 2, nv21Width, CvType.CV_8UC1)
+        yuv.put(0, 0, nv21)
+        val srcRaw = Mat()
+        Imgproc.cvtColor(yuv, srcRaw, Imgproc.COLOR_YUV2BGR_NV21)
+        yuv.release()
+        if (srcRaw.empty()) {
+            return FrameTelemetry(
+                FrameOutcome.RejectedAlignmentLost, -1.0, 0, yaw, pitch,
+                msSince(t0),
+                isLandscape = isLandscape,
+            )
+        }
+        val frameBGR = downsampleToCompose(srcRaw)
+        if (frameBGR !== srcRaw) srcRaw.release()
+        return addFrameMat(
+            frameBGR,
+            qx, qy, qz, qw,
+            fx, fy, cx, cy,
+            imageWidth, imageHeight,
+            yaw, pitch,
+            fovHorizDegrees, fovVertDegrees,
+            t0,
+        )
+    }
 
+    /**
+     * F8.6 — the body extracted from [addFrameAtPath].  Takes a
+     * BGR `Mat` (downsampled to compose dims) and runs the full
+     * engine pipeline: first-frame init or subsequent-frame paste
+     * via either rectilinear or cylindrical warp.
+     *
+     * Behaviour is identical to the pre-F8.6 `addFrameAtPath`
+     * (the body is a verbatim move).  Both `addFrameAtPath` and
+     * `addFramePixelData` delegate here after their respective
+     * Mat constructions.
+     */
+    private fun addFrameMat(
+        frameBGR: Mat,
+        qx: Double, qy: Double, qz: Double, qw: Double,
+        fx: Double, fy: Double, cx: Double, cy: Double,
+        imageWidth: Int, imageHeight: Int,
+        yaw: Double, pitch: Double,
+        // FOV params kept for symmetry with `IncrementalEngine.addFrameMat`
+        // (the hybrid engine, which uses them in `computeOverlapPct`).
+        // The firstwins/slit-scan engine here doesn't consume them —
+        // its paste decision is driven by pose-projected pixel
+        // displacement, not FoV-overlap percent.
+        @Suppress("UNUSED_PARAMETER") fovHorizDegrees: Double,
+        @Suppress("UNUSED_PARAMETER") fovVertDegrees: Double,
+        t0: Long,
+    ): FrameTelemetry {
         val rNew = quaternionToRotationMat(qx, qy, qz, qw)
 
         if (!hasFirstFrame) {