react-native-image-stitcher 0.5.1 → 0.7.0

package/CHANGELOG.md

@@ -16,6 +16,198 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.7.0] — 2026-05-26
+
+### Added — Tier 1: `useKeyframeStream`
+
+JS-thread subscription hook for **accepted-keyframe events** — the
+small subset of camera frames the stitching engine actually chose to
+include in the panorama.  Foundation for plugin-pattern host features:
+OCR on each saved keyframe, packet detection, server-side analysis,
+analytics, etc.
+
+Fires 4-6 times per panorama (once per accepted keyframe), NOT per
+camera frame — the lowest-frequency, highest-value frame stream.
+
+```tsx
+import { useKeyframeStream, type AcceptedKeyframe } from 'react-native-image-stitcher';
+
+function OcrPlugin() {
+  useKeyframeStream(useCallback(async (kf: AcceptedKeyframe) => {
+    const text = await runOCR(kf.jpegPath);
+    console.log(`Keyframe ${kf.index} pose=${kf.pose.rotation}:`, text);
+  }, []));
+  return null;
+}
+```
+
+- **`useKeyframeStream(handler)`** exported from
+  `react-native-image-stitcher`.  Subscribes to the existing
+  `IncrementalStateUpdate` event channel; surfaces accepted-keyframe
+  events through a typed callback.  Re-subscribes on handler-identity
+  changes; async handler rejections are surfaced via `console.error`
+  rather than swallowed.
+- **`AcceptedKeyframe` type** exported.  Fields: `jpegPath` (absolute
+  path, no `file://` prefix); `pose` (rotation quaternion + optional
+  translation vector); `timestamp` (ms since epoch); `index`
+  (zero-based position in current panorama).
+- **`IncrementalState.batchKeyframePose?`** + **`batchKeyframeAcceptedAtMs?`**
+  new optional fields.  Populated by the native emit alongside the
+  existing `batchKeyframeThumbnailPath` + `batchKeyframeIndex` on
+  accept events.  Direct readers of `IncrementalState` can consume
+  these without going through the new hook.
+
+### Changed (internal — externally invisible)
+
+- **Native `emitBatchKeyframeAcceptedState` populates pose + timestamp.**
+  Both `IncrementalStitcher.swift::emitBatchKeyframeAcceptedState` and
+  `IncrementalStitcher.kt::emitBatchKeyframeAcceptedState` grew
+  parameters for the pose snapshot (quaternion + translation) and
+  accept-time wall-clock millis.  The existing call sites in the
+  batch-keyframe accept path thread the pose they already have in
+  scope.
+
+### Engine-mode caveat
+
+`useKeyframeStream` only fires under the `batch-keyframe` engine (the
+`<Camera>` component's default).  Live engines (`firstwins-rectilinear`,
+`hybrid`, `slitscan-*`) paint into a live canvas instead of saving
+per-accept JPEGs and do not surface accept events through this channel
+— the hook silently does not fire in those modes.  Live-engine accept
+emit may land as a v0.7.1 follow-up if a real consumer needs it.
+
+### Translation semantics
+
+`AcceptedKeyframe.pose.translation` is always populated by the native
+emit.  In AR mode it carries the real ARKit / ARCore camera transform
+in metres (world coords).  In non-AR (Frame Processor) mode the
+translation reads as `[0, 0, 0]` because gyroscope provides only
+rotation (no spatial anchor).  Hosts that need to distinguish can
+either check the active `frameSourceMode` or threshold the translation
+magnitude.
+
+### Compatibility
+
+Strict additive over v0.6.0.  No host changes required.  Existing
+`subscribeIncrementalState` consumers see new optional fields but
+their existing reads are unaffected.
+
+### Verification
+
+- iPhone 17 Pro (real device, iOS 26.5): hold-and-release AR-mode
+  panorama produced four accepted-keyframe events with real pose
+  data (unit quaternion + non-zero translation in metres matching
+  the physical pan).
+- Android (Galaxy A35): `compileDebugKotlin` BUILD SUCCESSFUL;
+  on-device runtime verification deferred for this release (the
+  Kotlin emit mirrors the iOS emit at the byte-for-byte payload
+  level — same field names, same types, same call-site pattern).
+
+## [0.6.0] — 2026-05-25
+
+> [!WARNING]
+> **Breaking changes.**  v0.6.0 retires the deprecated JS-driver
+> non-AR path that was marked for removal in v0.5.0's *Deprecated*
+> section.  Hosts using the default `<Camera>` flow (`legacyDriver`
+> unset) are not affected — they were already on
+> `useFrameProcessorDriver`.  Hosts that opted into the legacy
+> driver (`legacyDriver={true}` on `<Camera>`, or a direct
+> `useIncrementalJSDriver()` consumer) MUST migrate to the Frame
+> Processor driver — see *Migration from 0.5.x* below.
+
+### Removed (breaking)
+
+- **`useIncrementalJSDriver` hook** + its `UseIncrementalJSDriverOptions`
+  / `IncrementalJSDriverHandle` types.  Deprecated in v0.5.0; the
+  v0.5 deprecation warning has now been replaced by deletion.
+- **`legacyDriver?: boolean` prop on `<Camera>`**.  The escape hatch
+  back to the JS driver is gone.  Hosts that set this prop will
+  get a TS-level error; at runtime the prop is silently ignored.
+- **`frameSourceMode: 'jsDriver'`** enum value in
+  `IncrementalStartOptions`.  The TS type is now narrowed to
+  `'arSession' | 'frameProcessor'`.  Passing `'jsDriver'` is a
+  compile error; at the native bridge layer the value falls through
+  to the default (now `'arSession'`).
+- **`IncrementalStitcher.processFrameAtPath` native method** on both
+  iOS and Android.  The only JS caller was `useIncrementalJSDriver`,
+  also deleted.  Hosts calling
+  `NativeModules.IncrementalStitcher.processFrameAtPath(...)` via
+  raw `NativeModules` access will get a runtime "method does not
+  exist" error.  Use the Frame Processor driver instead.
+
+### Changed (breaking)
+
+- **Android `frameSourceMode` default switched from `"jsDriver"` to
+  `"arSession"`** for parity with iOS.  Raw `NativeModules` callers
+  that omitted `frameSourceMode` were previously getting an inert
+  capture (the "jsDriver" branch dropped all engine input on
+  Android since v0.5.0); they now get AR-mode behaviour, matching
+  iOS.  The production `<Camera>` is unaffected — it always passes
+  `frameSourceMode: 'arSession'` explicitly for AR captures.
+
+### Changed (non-breaking)
+
+- **`RNSARCameraView` (AR mode) no longer eager-encodes a JPEG per
+  ARCore frame.**  Migrated to the pixel-data path introduced for
+  the Frame Processor in v0.5.1's F8.6 work.  AR-mode captures now
+  pass `nv21PixelData` / `nv21PixelWidth` / `nv21PixelHeight`
+  through `ingestFromARCameraView`; `legacyJpegPath` is always null
+  on this path.  Expected gain on Galaxy A35: ~30-50 ms per
+  accepted frame, with the dominant savings on rejected frames
+  (no JPEG encode → no imread round-trip).  Closes the v0.5.0
+  follow-up.
+
+### Removed (internal cleanup; no external API impact)
+
+- **F8.6 perf-diagnostic logs** (`F8.6-route`, `F8.6-perf`)
+  introduced in v0.5.1 stripped from `IncrementalStitcher` +
+  `IncrementalFirstwinsEngine` — F8.6 is now baked in for
+  production and the diagnostic spam is no longer informative.
+- **Orphaned native helpers** dropped after `processFrameAtPath`
+  removal:
+  - iOS: `addBatchKeyframePath(path:pose:)`, `isBatchKeyframeMode`
+    getter, `decodeJpegToGrayscalePixelBuffer` (only callers were
+    `processFrameAtPath`).
+  - Android: `decodeJpegToGrayscale` + `GrayscaleFrame` data class,
+    `isBatchKeyframeMode` getter (only callers were
+    `processFrameAtPath` and the AR-mode eager-encode branch).
+- **Stale comments** referencing removed code paths swept across
+  Kotlin/Swift/Obj-C/TS.  Historical "removed in v0.6" markers
+  retained; comments that described live code in terms of the
+  removed names rewritten to describe current behaviour.
+
+### Migration from 0.5.x
+
+**Default `<Camera>` hosts (no `legacyDriver` prop set):** no
+action required.  `<Camera>` already used `useFrameProcessorDriver`
+in non-AR mode and `RNSARSession` in AR mode since v0.5.0.
+
+**Hosts with `legacyDriver={true}` on `<Camera>`:** remove the
+prop.  `<Camera>` will use the Frame Processor driver, which has
+been the default since v0.5.0 and the only path since this release.
+
+```tsx
+// Before (v0.5.x)
+<Camera legacyDriver={true} ... />
+
+// After (v0.6.0)
+<Camera ... />
+```
+
+**Hosts directly using `useIncrementalJSDriver`:** migrate to
+`useFrameProcessorDriver`.  The handle shape (`{ start, stop,
+frameProcessor, isRunning }`) is preserved, but the new hook is a
+Frame Processor + gyro driver instead of a `takeSnapshot` + JS
+interval driver.  See
+[`src/stitching/useFrameProcessorDriver.ts`](src/stitching/useFrameProcessorDriver.ts)
+for the migration mapping; the gyro pose synthesis convention
+(`q = q_yaw * q_pitch * q_roll`) is identical, so existing pose
+math at call sites continues to work.
+
+**Hosts passing `frameSourceMode: 'jsDriver'` to
+`incremental.start(...)`:** change to `'frameProcessor'`.  The
+TypeScript type now rejects `'jsDriver'` at compile time.
+
 ## [0.5.1] — 2026-05-25
 
 ### Added — F8.6 Android pixel-buffer engine parity
@@ -1056,7 +1248,13 @@ Native module names also changed:
 - iOS pod: `RetaiLensCaptureSDK` → `RNImageStitcher`
 - iOS xcframework: shipped as `opencv2.xcframework` (linked from `RNImageStitcher.podspec`)
 
-[Unreleased]: https://github.com/bhargavkanda/react-native-image-stitcher/compare/v0.3.0...HEAD
+[Unreleased]: https://github.com/bhargavkanda/react-native-image-stitcher/compare/v0.7.0...HEAD
+[0.7.0]: https://github.com/bhargavkanda/react-native-image-stitcher/compare/v0.6.0...v0.7.0
+[0.6.0]: https://github.com/bhargavkanda/react-native-image-stitcher/compare/v0.5.1...v0.6.0
+[0.5.1]: https://github.com/bhargavkanda/react-native-image-stitcher/compare/v0.5.0...v0.5.1
+[0.5.0]: https://github.com/bhargavkanda/react-native-image-stitcher/compare/v0.4.1...v0.5.0
+[0.4.1]: https://github.com/bhargavkanda/react-native-image-stitcher/compare/v0.4.0...v0.4.1
+[0.4.0]: https://github.com/bhargavkanda/react-native-image-stitcher/compare/v0.3.0...v0.4.0
 [0.3.0]: https://github.com/bhargavkanda/react-native-image-stitcher/compare/v0.2.1...v0.3.0
 [0.2.1]: https://github.com/bhargavkanda/react-native-image-stitcher/compare/v0.2.0...v0.2.1
 [0.2.0]: https://github.com/bhargavkanda/react-native-image-stitcher/compare/v0.1.3...v0.2.0

package/android/src/main/java/io/imagestitcher/rn/CvFlowGateFrameProcessor.kt

@@ -29,8 +29,8 @@ import com.mrousavy.camera.frameprocessors.VisionCameraProxy
  *   3. Call `IncrementalStitcher.consumeFrameFromPlugin(image, …)`
  *      which:
  *        - Drops the call if `frameSourceMode != "frameProcessor"`
- *          (prevents double-feeding the engine alongside the legacy
- *          `processFrameAtPath` path).
+ *          (prevents double-feeding the engine alongside the
+ *          AR-mode `ingestFromARCameraView` path).
  *        - Otherwise: extracts the Y plane, evaluates the keyframe
  *          gate via `KeyframeGate.evaluateWithFrame`, encodes the
  *          accepted frame to JPEG synchronously, and hands the path

package/android/src/main/java/io/imagestitcher/rn/IncrementalFirstwinsEngine.kt

@@ -197,7 +197,7 @@ internal class IncrementalFirstwinsEngine(
         }
         val frameBGR = downsampleToCompose(srcRaw)
         if (frameBGR !== srcRaw) srcRaw.release()
-        val tele = addFrameMat(
+        return addFrameMat(
             frameBGR,
             qx, qy, qz, qw,
             fx, fy, cx, cy,
@@ -206,8 +206,6 @@ internal class IncrementalFirstwinsEngine(
             fovHorizDegrees, fovVertDegrees,
             t0,
         )
-        f8_6_logPerf("firstwins/jpeg", t0, tele.outcome)
-        return tele
     }
 
     /**
@@ -283,7 +281,7 @@ internal class IncrementalFirstwinsEngine(
         }
         val frameBGR = downsampleToCompose(srcRaw)
         if (frameBGR !== srcRaw) srcRaw.release()
-        val tele = addFrameMat(
+        return addFrameMat(
             frameBGR,
             qx, qy, qz, qw,
             fx, fy, cx, cy,
@@ -292,32 +290,6 @@ internal class IncrementalFirstwinsEngine(
             fovHorizDegrees, fovVertDegrees,
             t0,
         )
-        f8_6_logPerf("firstwins/pixel", t0, tele.outcome)
-        return tele
-    }
-
-    /**
-     * F8.6 perf-diagnostic counter.  Logs ingest timing every Nth
-     * call so a single capture session yields enough samples to
-     * eyeball the JPEG-path vs pixel-data-path delta.  Remove this
-     * once F8.6 is verified in production.
-     */
-    @Volatile private var f8_6_perfCallCounter: Long = 0L
-    private fun f8_6_logPerf(
-        path: String,
-        t0Nanos: Long,
-        outcome: FrameOutcome,
-    ) {
-        val n = ++f8_6_perfCallCounter
-        // Every 5th call ≈ ~1 line/sec at 5 Hz live-engine rate;
-        // first call always logs so we see something on capture
-        // start without waiting.
-        if (n == 1L || n % 5L == 0L) {
-            Log.i(
-                "F8.6-perf",
-                "$path took ${msSince(t0Nanos)}ms outcome=$outcome (call #$n)",
-            )
-        }
     }
 
     /**