react-native-image-stitcher 0.4.1 → 0.5.0

package/CHANGELOG.md

@@ -16,6 +16,89 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.5.0] — 2026-05-25
+
+### Added — F8 Frame Processor port
+
+`<Camera>` now drives **non-AR captures through a vision-camera
+Frame Processor** on the camera producer thread instead of the
+4 Hz `takeSnapshot` → JPEG → cache-file path the v0.4 series used.
+
+- **`useFrameProcessorDriver`** (`src/stitching/useFrameProcessorDriver.ts`)
+  — new hook with the same `{ start, stop, frameProcessor,
+  isRunning }` shape as the legacy `useIncrementalJSDriver`.  Gyro
+  yaw / pitch / **roll** are integrated on the JS thread and
+  published via `useSharedValue` so the worklet reads pose
+  zero-hop.  Plugin acquisition uses a mount-once + 16 ms
+  setTimeout retry pattern to side-step the vision-camera
+  registry init race.
+- **`cv_flow_gate_process_frame` JSI plugin** — registered on both
+  platforms:
+  - iOS: `ios/Sources/RNImageStitcher/KeyframeGateFrameProcessor.mm`
+    + `@objc IncrementalStitcher.consumeFrameFromPlugin(...)`
+    wrapper.  `CVPixelBuffer` flows end-to-end into
+    `IncrementalStitcher.consumeFrame` — the SAME entry point AR
+    mode already uses.  Zero JPEG round-trip on accept.
+  - Android: `android/src/main/java/io/imagestitcher/rn/CvFlowGateFrameProcessor.kt`
+    + Kotlin `consumeFrameFromPlugin(...)` wrapper.  Extracts the
+    Y plane on the producer thread, encodes inline JPEG on accept
+    via the existing `YuvImageConverter`, hands the path to
+    `ingestFromARCameraView`.  Pixel-buffer parity tracked as F8.6.
+- **`frameSourceMode: 'frameProcessor'`** in
+  `IncrementalStitcher.start()` options — flips
+  `frameProcessorIngestEnabled` ON so the plugin's producer-thread
+  feed reaches the engine.  Default for non-AR captures from v0.5.
+- **`legacyDriver?: boolean`** prop on `<Camera>` — opt-in escape
+  hatch back to `useIncrementalJSDriver` for hosts that hit a
+  vision-camera incompatibility.  Will be removed in v0.6.
+- **`VISION_CAMERA_RUNTIME` error code** for vision-camera
+  runtime errors that aren't transient lifecycle events.
+- **Roll axis** (gyro-Z) in the synthesised pose quaternion —
+  `q = q_yaw * q_pitch * q_roll`.  Field captures with wrist-twist
+  no longer lie to the cv::Stitcher's intrinsic estimator.
+
+### Changed
+
+- Default non-AR driver is now `useFrameProcessorDriver`.  Hosts
+  using `<Camera>` opt in transparently — no code change needed
+  unless you want the legacy path (`legacyDriver={true}`).
+- `host-supplied frameProcessor` prop on `<Camera>` is now treated
+  as a legacy escape hatch: silently overridden by the SDK driver
+  in default mode with a one-shot `console.warn`.
+
+### Deprecated
+
+- **`useIncrementalJSDriver`** — works through v0.5, removed in
+  v0.6.  Hosts that drove non-AR captures with this hook should
+  migrate to letting `<Camera>` do it by default
+  (`legacyDriver` unset).  The hook now emits a one-shot
+  `console.warn` from its `start()` call.
+
+### Fixed
+
+- **Vision-camera transient lifecycle errors** (screen-lock,
+  app-switch, DoNotDisturb, MDM camera restriction) are now
+  filtered inside `<CameraView>` instead of propagating to the
+  host's `onError`.  Auto-recovery happens on resume; hosts no
+  longer get spurious crash reports on every phone-lock.
+
+### Added — peer dependency
+
+- **`react-native-worklets-core`** is now a declared peer
+  dependency (`>=1.3.0`).  It was already required transitively
+  via `react-native-vision-camera@^4`; the explicit declaration
+  documents the contract.
+
+### Tracking — known follow-ups (don't gate this release)
+
+- **F8.6** — Android engine refactor for pixel-buffer-direct
+  ingest (true zero-copy parity with iOS).
+- **F8.3.H2-target** — `swift test` currently can't run the
+  iOS test target due to mixed Swift/.mm sources.  The
+  `FrameProcessorPluginSelectorTests` selector guard is in place
+  as a documentation artifact; CI test-runner fix is a separate
+  task.
+
 ## [0.4.1] — 2026-05-23
 
 ### Fixed

package/README.md

@@ -160,6 +160,7 @@ The component owns the runtime state; the parent persists across launches via th
 | **Android namespace** | `io.imagestitcher.rn`. |
 | **Stitching pipeline** | Shared C++ under `cpp/stitcher.cpp` invoked from both iOS Obj-C++ and Android JNI.  PANORAMA + SCANS modes; C+D progressive-confidence retry over keyframes. |
 | **Two capture-source paths** | AR uses ARKit (iOS) / ARCore (Android) pose stream.  Non-AR uses vision-camera + IMU integration via `useIMUTranslationGate`. |
+| **Frame Processor driver (v0.5+)** | Non-AR captures evaluate the keyframe gate on the camera producer thread at native frame rate via a vision-camera Frame Processor (`cv_flow_gate_process_frame`).  iOS passes `CVPixelBuffer` end-to-end; Android writes a Y-plane-derived JPEG on accept.  Opt-out via `<Camera legacyDriver />` for one minor cycle.  See `docs/f8-frame-processor-plan.md` for the design. |
 | **Two supported pan modes** | Landscape phone + vertical pan; portrait phone + horizontal pan.  Any other combination is a user deviation, not a supported mode. |
 
 ## License

package/android/build.gradle

@@ -215,6 +215,39 @@ dependencies {
     // on demand (~30 MB) the first time the user opens an AR
     // capture screen on a supported device.
     implementation "com.google.ar:core:1.45.0"
+
+    // F8.4 — vision-camera as a compile-time peer dep.  Same
+    // `compileOnly` pattern as React Native above: host apps that
+    // wire `<Camera>` already include the autolinked
+    // `:react-native-vision-camera` Gradle project; we just need
+    // `com.mrousavy.camera.frameprocessors.*` types on the compile
+    // classpath to build `CvFlowGateFrameProcessor.kt` + the
+    // registration in `RNImageStitcherPackage`.  Runtime
+    // resolution is the host's responsibility —
+    // `RNImageStitcherPackage`'s static initialiser catches
+    // `NoClassDefFoundError` so the SDK still loads if a non-
+    // camera consumer omits the dep.
+    //
+    // `findProject(...)` guard: lets the SDK still compile in
+    // hosts that haven't installed react-native-vision-camera.
+    // The plugin file's import resolution will fail in that case,
+    // which is fine — we conditionally exclude the plugin sources
+    // below.
+    if (findProject(':react-native-vision-camera') != null) {
+        compileOnly project(':react-native-vision-camera')
+        // CameraX `ImageProxy` / `ImageInfo` types — vision-camera
+        // exposes them through `Frame.getImageProxy()` and we need
+        // the compile-time class for `imageInfo.rotationDegrees`.
+        // `compileOnly` because the host app already ships these via
+        // vision-camera's transitive runtime dep.
+        compileOnly "androidx.camera:camera-core:1.5.0-alpha03"
+    } else {
+        // Without vision-camera on the classpath the Frame
+        // Processor plugin source can't compile (imports unresolved).
+        // Exclude it from the source set so the rest of the SDK
+        // still builds for non-camera consumers.
+        android.sourceSets.main.java.exclude '**/CvFlowGateFrameProcessor.kt'
+    }
 }
 
 // Helper from the React Native gradle convention to read host-app

package/android/src/main/java/io/imagestitcher/rn/CvFlowGateFrameProcessor.kt

@@ -0,0 +1,163 @@
+// SPDX-License-Identifier: Apache-2.0
+package io.imagestitcher.rn
+
+import android.media.Image
+import androidx.annotation.Keep
+import com.facebook.proguard.annotations.DoNotStrip
+import com.mrousavy.camera.frameprocessors.Frame
+import com.mrousavy.camera.frameprocessors.FrameProcessorPlugin
+import com.mrousavy.camera.frameprocessors.VisionCameraProxy
+
+/**
+ * F8.4 — Android vision-camera Frame Processor plugin that mirrors
+ * iOS' `KeyframeGateFrameProcessor.mm`.
+ *
+ * Plugin name (must match the iOS plugin):
+ *   `cv_flow_gate_process_frame`
+ *
+ * JS-side usage is identical to iOS — the same `useFrameProcessorDriver`
+ * hook + the same `plugin.call(frame, args)` contract.  The JS layer
+ * is 100% platform-agnostic.
+ *
+ * ## What this plugin does
+ *
+ * Per producer-thread frame:
+ *   1. Pull the `android.media.Image` out of vision-camera's `Frame`.
+ *   2. Extract pose primitives from the worklet's `params` dict
+ *      (defaults safe for non-AR: tx/ty/tz=0, qw=1 identity, fx/fy=0
+ *      → engine uses 65°×50° FoV fallback).
+ *   3. Call `IncrementalStitcher.consumeFrameFromPlugin(image, …)`
+ *      which:
+ *        - Drops the call if `frameSourceMode != "frameProcessor"`
+ *          (prevents double-feeding the engine alongside the legacy
+ *          `processFrameAtPath` path).
+ *        - Otherwise: extracts the Y plane, evaluates the keyframe
+ *          gate via `KeyframeGate.evaluateWithFrame`, encodes the
+ *          accepted frame to JPEG synchronously, and hands the path
+ *          to the existing `ingestFromARCameraView` engine entry.
+ *
+ * ## Lifetime / threading
+ *
+ * The `Frame` (and the underlying `Image` / `ImageProxy`) is valid
+ * only for the duration of this callback — vision-camera closes it
+ * on return.  All Image access (including the JPEG encode on
+ * accept) MUST happen synchronously inside `callback()`.
+ *
+ * ## Divergence vs iOS
+ *
+ * iOS keeps the `CVPixelBuffer` reachable end-to-end into the
+ * stitcher engine (zero-copy).  Android's engine entry point
+ * (`ingestFromARCameraView`) takes a Y `ByteArray` + a JPEG file
+ * path, so we copy Y bytes here and encode JPEG inline on accept.
+ * Cross-platform parity at the engine level is tracked as F8.6.
+ *
+ * ## Registration
+ *
+ * Registered in `RNImageStitcherPackage.kt`'s companion-object
+ * static initialiser via `FrameProcessorPluginRegistry`.  Vision-
+ * camera docs say "should be called as soon as possible — ideally
+ * on app start or in a static initialiser"; the package class is
+ * loaded by RN autolinking at app startup, so the registration
+ * fires before any JS Frame Processor can `initFrameProcessorPlugin`
+ * the plugin.
+ */
+@DoNotStrip
+@Keep
+class CvFlowGateFrameProcessor(
+    proxy: VisionCameraProxy,
+    options: Map<String, Any>?,
+) : FrameProcessorPlugin() {
+
+    // The `proxy` and `options` are accepted by the
+    // `PluginInitializer` contract but the plugin is stateless —
+    // all gate tunables live on `IncrementalStitcher` and are
+    // configured at its `start()` time from the host-app settings.
+    // The plugin is a thin pose-injector.
+    //
+    // Lint suppressors: we intentionally don't read these.
+    @Suppress("unused", "UNUSED_PARAMETER")
+    private val unused = proxy to options
+
+    @Suppress("UNCHECKED_CAST")
+    override fun callback(frame: Frame, params: Map<String, Any>?): Any? {
+        // Frame may throw `FrameInvalidError` if vision-camera has
+        // already released it.  Defensive: swallow and return.
+        val image: Image = try {
+            frame.image
+        } catch (e: Throwable) {
+            return mapOf("submitted" to false, "error" to "frame invalid")
+        }
+
+        val stitcher = IncrementalStitcher.bridgeInstance
+        if (stitcher == null) {
+            // Module never registered (host hasn't initialised the
+            // React bridge yet, or autolinking skipped us).  Drop
+            // the call; JS sees `submitted: false` and can detect.
+            return mapOf("submitted" to false, "error" to "stitcher not registered")
+        }
+
+        // F8.4-Android-c rotation fix: read CameraX's authoritative
+        // "rotation needed to display upright" value via
+        // `imageProxy.imageInfo.rotationDegrees`.
+        //
+        // The earlier attempt used `Frame.orientation` (the enum),
+        // but vision-camera's `getOrientation()` returns the REVERSE
+        // of the rotation-needed value (see Frame.java:88, the
+        // "Reverse it" comment).  Trying to invert the enum
+        // ourselves was off by 90° on the A35.  The raw
+        // `imageInfo.rotationDegrees` is unambiguous.
+        //
+        // Used by the engine's JPEG encoder to write the correct
+        // EXIF Orientation tag so thumbnails (and any other
+        // EXIF-honoring viewer) display upright.  The raw cv::Mat
+        // the stitcher sees is unaffected — see consumeFrameFromPlugin
+        // docstring for the no-double-rotation rationale.
+        val sensorRotationDegrees = try {
+            frame.imageProxy.imageInfo.rotationDegrees
+        } catch (_: Throwable) {
+            // FrameInvalidError or null mid-callback — treat as
+            // portrait back-camera default (sensor mounted 90° CW).
+            90
+        }
+
+        stitcher.consumeFrameFromPlugin(
+            image = image,
+            tx = argDouble(params, "tx", 0.0),
+            ty = argDouble(params, "ty", 0.0),
+            tz = argDouble(params, "tz", 0.0),
+            qx = argDouble(params, "qx", 0.0),
+            qy = argDouble(params, "qy", 0.0),
+            qz = argDouble(params, "qz", 0.0),
+            qw = argDouble(params, "qw", 1.0),
+            fx = argDouble(params, "fx", 0.0),
+            fy = argDouble(params, "fy", 0.0),
+            cx = argDouble(params, "cx", image.width / 2.0),
+            cy = argDouble(params, "cy", image.height / 2.0),
+            timestampMs = argDouble(params, "timestampMs", 0.0),
+            // Default 2 == `.tracking` so the worklet doesn't need
+            // to send a tracking-state field on every frame.
+            trackingStateRaw = argInt(params, "trackingStateRaw", 2),
+            sensorRotationDegrees = sensorRotationDegrees,
+        )
+
+        return mapOf("submitted" to true)
+    }
+
+    private fun argDouble(args: Map<String, Any>?, key: String, default: Double): Double {
+        if (args == null) return default
+        val v = args[key] ?: return default
+        return when (v) {
+            is Number -> v.toDouble()
+            else -> default
+        }
+    }
+
+    private fun argInt(args: Map<String, Any>?, key: String, default: Int): Int {
+        if (args == null) return default
+        val v = args[key] ?: return default
+        return when (v) {
+            is Number -> v.toInt()
+            else -> default
+        }
+    }
+}

package/android/src/main/java/io/imagestitcher/rn/IncrementalStitcher.kt

@@ -36,6 +36,7 @@ import org.opencv.features2d.ORB
 import org.opencv.imgcodecs.Imgcodecs
 import org.opencv.imgproc.Imgproc
 import java.io.File
+import io.imagestitcher.rn.ar.YuvImageConverter
 import java.util.concurrent.atomic.AtomicBoolean
 
 /**
@@ -76,6 +77,13 @@ class IncrementalStitcher(
     private val reactContext: ReactApplicationContext,
 ) : ReactContextBaseJavaModule(reactContext) {
 
+    // F8.4 note: the static singleton accessor for cross-thread
+    // lookup (used by `CvFlowGateFrameProcessor` running on vision-
+    // camera's producer thread) is the existing `bridgeInstance`
+    // companion field below — same pattern that `RNSARCameraView`
+    // uses to call back into the bridge.  No new companion object
+    // needed.
+
     override fun getName(): String = "IncrementalStitcher"
 
     /// Required by RCTEventEmitter contract.  No-op on Android because
@@ -217,6 +225,21 @@ class IncrementalStitcher(
     private var consumeFrameCounter: Long = 0L
 
     private val isRunning = AtomicBoolean(false)
+
+    /// F8.4 — gate for `consumeFrameFromPlugin` (the vision-camera
+    /// Frame Processor producer-thread entry point on Android).
+    /// TRUE only when the current capture was started with
+    /// `frameSourceMode == "frameProcessor"`.  In other modes
+    /// (especially the legacy `"jsDriver"` path that feeds via
+    /// `processFrameAtPath`), the plugin would double-feed the
+    /// engine — bytes from the producer thread + JPEG paths from
+    /// the JS interval, racing on the same workScope serial
+    /// dispatcher — so we drop the producer-thread call.
+    ///
+    /// AtomicBoolean: producer thread reads lock-free, JS thread
+    /// (start/cancel/finalize) writes via `set()`/`compareAndSet()`.
+    /// Mirror of iOS' `frameProcessorIngestEnabled` ivar.
+    private val frameProcessorIngestEnabled = AtomicBoolean(false)
     /// Critic #5 fix: serial dispatcher so concurrent
     /// processFrameAtPath() calls can't race on the engine's canvas.
     /// `limitedParallelism(1)` guarantees one-at-a-time execution
@@ -301,6 +324,15 @@ class IncrementalStitcher(
         }
         try {
             ensureOpenCv()
+            // F8.4 — frameSourceMode honoured on Android.  Pre-F8.4,
+            // Android ignored this option (only iOS interpreted it).
+            // Now `"frameProcessor"` unlocks `consumeFrameFromPlugin`'s
+            // producer-thread ingest path; everything else (the
+            // implicit default + the legacy `"jsDriver"`) keeps the
+            // ingest path dormant so the existing `processFrameAtPath`
+            // / ARCore paths run unmodified.
+            val frameSourceMode = options.getString("frameSourceMode") ?: "jsDriver"
+            frameProcessorIngestEnabled.set(frameSourceMode == "frameProcessor")
             val rotation = options.getIntOrDefault("frameRotationDegrees", 90)
             val composeW = options.getIntOrDefault("composeWidth",  960)
             val composeH = options.getIntOrDefault("composeHeight", 720)
@@ -548,6 +580,7 @@ class IncrementalStitcher(
             promise.resolve(map)
         } catch (t: Throwable) {
             isRunning.set(false)
+            frameProcessorIngestEnabled.set(false)  // F8.4 — symmetric clear on error path
             promise.reject("incremental-start-failed", t.message, t)
         }
     }
@@ -935,6 +968,7 @@ class IncrementalStitcher(
         // bail at the re-check (see processFrameAtPath above).
         // Matches iOS V12.1 fix.
         isRunning.set(false)
+        frameProcessorIngestEnabled.set(false)  // F8.4 — cut producer-thread ingest at finalize
 
         // V16 batch-keyframe finalize: snapshot the keyframe state
         // synchronously under the same "stop ingestion before
@@ -1133,6 +1167,7 @@ class IncrementalStitcher(
         // iOS V12.1 cancel path.
         arCameraViewRef?.setIncrementalIngestionActive(false)
         isRunning.set(false)
+        frameProcessorIngestEnabled.set(false)  // F8.4 — cut producer-thread ingest at cancel
         val hybrid = engine
         val firstwins = firstwinsEngine
         engine = null
@@ -1444,6 +1479,179 @@ class IncrementalStitcher(
         }
     }
 
+    // ─── F8.4 — Frame Processor entry point ──────────────────────
+    //
+    // `consumeFrameFromPlugin` is the producer-thread ingress for
+    // the vision-camera Frame Processor plugin
+    // (`CvFlowGateFrameProcessor`).  It takes a live
+    // `android.media.Image` (held open by vision-camera for the
+    // duration of the plugin callback) plus pose primitives, and
+    // delegates to the existing `ingestFromARCameraView` after
+    // extracting the Y plane bytes and wiring an inline JPEG
+    // encoder for the on-accept lambda.
+    //
+    // ## Why this lives here (not on the plugin class)
+    //
+    // The plugin needs zero knowledge of the engine's internals
+    // (batchKeyframeMode, eval-throttling, plane-latching, etc.)
+    // — that's all in `ingestFromARCameraView`.  Mirroring iOS'
+    // `consumeFrameFromPlugin`, the wrapper just maps the public
+    // primitive contract to the existing engine entry point.
+    //
+    // ## Why pass `Image` (not just the Y bytes)
+    //
+    // The engine's `ingestFromARCameraView` uses Y-only for the
+    // keyframe gate.  But when the gate ACCEPTS, the host (us) is
+    // responsible for encoding the accepted frame as JPEG before
+    // `ingestFromARCameraView` returns.  YuvImage / NV21 needs the
+    // full Y + interleaved VU planes, so we keep the Image
+    // reachable through the lambda.  Image's lifetime is bounded
+    // by the plugin callback's return — vision-camera closes the
+    // ImageProxy automatically — so the encode MUST be synchronous.
+    //
+    // ## Threading
+    //
+    // Called on vision-camera's frame-processor thread (a single-
+    // thread executor).  `frameProcessorIngestEnabled` is read
+    // lock-free via AtomicBoolean.  `ingestFromARCameraView`
+    // dispatches the heavy engine work to `workScope` (serial),
+    // so producer-thread blocking is bounded to the synchronous
+    // gate evaluation + (on accept) JPEG encode — typically
+    // 5–10 ms reject, 30–50 ms accept on a mid-tier device.
+    fun consumeFrameFromPlugin(
+        image: android.media.Image,
+        tx: Double, ty: Double, tz: Double,
+        qx: Double, qy: Double, qz: Double, qw: Double,
+        fx: Double, fy: Double, cx: Double, cy: Double,
+        timestampMs: Double,
+        trackingStateRaw: Int,
+        // F8.4-Android-c rotation fix: how many degrees the sensor
+        // data needs to be rotated CW to display upright.  Comes
+        // from vision-camera's `Frame.imageProxy.imageInfo.rotationDegrees`.
+        // Typically 90 for a portrait-held back camera on Samsung
+        // devices (sensor mounted 90° rotated from screen-up).
+        sensorRotationDegrees: Int,
+    ) {
+        // F8.4 — drop the call unless this capture was started in
+        // frameProcessor mode.  Otherwise the plugin would double-
+        // feed the engine alongside the legacy jsDriver /
+        // processFrameAtPath path.  See the flag's declaration
+        // for the full reasoning.  Mirrors iOS H1.
+        if (!frameProcessorIngestEnabled.get()) return
+
+        val width = image.width
+        val height = image.height
+        val yPlane = image.planes[0]
+        val yRowStride = yPlane.rowStride
+
+        // Read Y plane bytes.  ByteBuffer.get() advances position;
+        // copy into our own ByteArray so the engine's downstream
+        // workScope can safely outlive this method (the Image
+        // closes after callback returns, but we've already copied).
+        val yBuffer = yPlane.buffer
+        val yBytes = ByteArray(yBuffer.remaining())
+        yBuffer.get(yBytes)
+
+        // Compute derived params expected by the existing ingest
+        // API.  Quaternion-to-yaw/pitch follows the same convention
+        // useFrameProcessorDriver synthesises on JS (q_yaw * q_pitch).
+        //
+        //   yaw   = atan2(2(qw*qy + qx*qz), 1 - 2(qy² + qz²))
+        //   pitch = asin(clamp(2(qw*qx - qz*qy), -1, 1))
+        val yaw = kotlin.math.atan2(
+            2.0 * (qw * qy + qx * qz),
+            1.0 - 2.0 * (qy * qy + qz * qz),
+        )
+        val pitch = kotlin.math.asin(
+            (2.0 * (qw * qx - qz * qy)).coerceIn(-1.0, 1.0),
+        )
+
+        // FoV from intrinsics + dims.  fx == 0 is the "JS didn't
+        // supply" signal (the iOS wrapper has the same default);
+        // fall back to a 65°×50° estimate so the engine doesn't
+        // see NaN.
+        val fovHorizDegrees = if (fx > 0.0)
+            2.0 * kotlin.math.atan(width.toDouble() / (2.0 * fx)) * 180.0 / Math.PI
+        else 65.0
+        val fovVertDegrees = if (fy > 0.0)
+            2.0 * kotlin.math.atan(height.toDouble() / (2.0 * fy)) * 180.0 / Math.PI
+        else 50.0
+
+        // `2` == `.tracking` per the iOS RNSARTrackingState enum.
+        // Anything else maps to trackingPoor=true, routing the
+        // frame through the engine's degraded-tracking branches
+        // (failing closed; symmetric with iOS C2).
+        val trackingPoor = trackingStateRaw != 2
+
+        ingestFromARCameraView(
+            tx = tx, ty = ty, tz = tz,
+            qx = qx, qy = qy, qz = qz, qw = qw,
+            fx = fx, fy = fy, cx = cx, cy = cy,
+            imageWidth = width, imageHeight = height,
+            yaw = yaw, pitch = pitch,
+            fovHorizDegrees = fovHorizDegrees,
+            fovVertDegrees = fovVertDegrees,
+            trackingPoor = trackingPoor,
+            grayData = yBytes,
+            grayWidth = width,
+            grayHeight = height,
+            grayStride = yRowStride,
+            onAccept = { targetPath ->
+                // Synchronous JPEG encode via the existing
+                // YuvImageConverter (also used by RNSARCameraView's
+                // ARCore path).  Handles both the NV21 conversion
+                // (stride / pixelStride aware) and the EXIF
+                // Orientation tag write so the JPEG displays upright
+                // in the UI thumbnail strip + any RN Image consumer.
+                //
+                // EXIF rotation is BAKED-AS-METADATA, not pixel-
+                // rotated.  cv::imread in the stitcher ignores EXIF
+                // by default (see BatchStitcher.applyExifOrientation),
+                // so the engine's stored `frameRotationDegrees` still
+                // governs how the cv::Mat is interpreted downstream.
+                // No double-rotation.
+                //
+                // Returning `true` tells the engine the keyframe was
+                // persisted; `false` tells it to drop the accept.
+                try {
+                    val packed = YuvImageConverter.packNV21(image)
+                        ?: run {
+                            android.util.Log.w(
+                                "IncrementalStitcher",
+                                "consumeFrameFromPlugin: packNV21 returned null for $targetPath",
+                            )
+                            return@run null
+                        }
+                    if (packed == null) {
+                        false
+                    } else {
+                        val displayRotation = when (sensorRotationDegrees) {
+                            0   -> android.view.Surface.ROTATION_90
+                            90  -> android.view.Surface.ROTATION_0
+                            180 -> android.view.Surface.ROTATION_270
+                            270 -> android.view.Surface.ROTATION_180
+                            else -> android.view.Surface.ROTATION_0
+                        }
+                        val outPath = YuvImageConverter.encodeJpegFromNV21(
+                            packed,
+                            targetPath,
+                            jpegQuality = 80,
+                            displayRotation = displayRotation,
+                        )
+                        outPath != null
+                    }
+                } catch (e: Throwable) {
+                    android.util.Log.w(
+                        "IncrementalStitcher",
+                        "consumeFrameFromPlugin: JPEG encode failed for $targetPath: ${e.javaClass.simpleName}: ${e.message}",
+                        e,
+                    )
+                    false
+                }
+            },
+        )
+    }
+
     @ReactMethod
     fun getState(promise: Promise) {
         val state = firstwinsEngine?.lastState ?: engine?.lastState
@@ -1944,6 +2152,12 @@ class IncrementalStitcher(
             // Ignore — not critical at teardown.
         }
         captureSessionDir = null
+        // F8.4 — release the static back-pointer so the Frame
+        // Processor plugin sees a clean nil after bridge teardown.
+        // A new bridge will set it again via the init block.
+        if (bridgeInstance === this) {
+            bridgeInstance = null
+        }
         super.onCatalystInstanceDestroy()
     }
 

package/android/src/main/java/io/imagestitcher/rn/RNImageStitcherPackage.kt

@@ -5,6 +5,7 @@ import com.facebook.react.ReactPackage
 import com.facebook.react.bridge.NativeModule
 import com.facebook.react.bridge.ReactApplicationContext
 import com.facebook.react.uimanager.ViewManager
+import com.mrousavy.camera.frameprocessors.FrameProcessorPluginRegistry
 
 /**
  * ReactPackage that registers the SDK's two native modules with
@@ -22,15 +23,72 @@ import com.facebook.react.uimanager.ViewManager
  * JS layer.
  */
 class RNImageStitcherPackage : ReactPackage {
+
+    companion object {
+        @Volatile
+        private var fpPluginRegistered = false
+
+        /**
+         * F8.4 — register the vision-camera Frame Processor plugin.
+         * Called lazily from `createNativeModules` (which fires
+         * AFTER the React bridge has booted, side-stepping the
+         * bridgeless TurboModule init race we'd hit if we did this
+         * in a class-level static initialiser).
+         *
+         * No-op when vision-camera isn't on the runtime classpath
+         * (the SDK doesn't hard-depend on it — consumers that don't
+         * use `<Camera>` don't pay the dep).  Catches
+         * `NoClassDefFoundError` defensively because the runtime
+         * classpath is what matters, not the compile-time one.
+         *
+         * Idempotent: guarded by `fpPluginRegistered` so a host
+         * with multiple React instances doesn't double-register
+         * (would throw "name already exists" from the registry).
+         */
+        @JvmStatic
+        @Synchronized
+        fun ensureFrameProcessorPluginRegistered() {
+            if (fpPluginRegistered) return
+            try {
+                FrameProcessorPluginRegistry.addFrameProcessorPlugin(
+                    "cv_flow_gate_process_frame",
+                ) { proxy, options ->
+                    CvFlowGateFrameProcessor(proxy, options)
+                }
+                fpPluginRegistered = true
+            } catch (e: NoClassDefFoundError) {
+                android.util.Log.i(
+                    "RNImageStitcherPackage",
+                    "vision-camera FrameProcessorPluginRegistry not on classpath — "
+                    + "skipping cv_flow_gate_process_frame plugin registration "
+                    + "(host app doesn't appear to use Frame Processors).",
+                )
+                fpPluginRegistered = true  // don't retry every package init
+            } catch (e: Throwable) {
+                android.util.Log.w(
+                    "RNImageStitcherPackage",
+                    "Failed to register cv_flow_gate_process_frame plugin: ${e.message}",
+                )
+                fpPluginRegistered = true
+            }
+        }
+    }
+
     override fun createNativeModules(
         reactContext: ReactApplicationContext,
-    ): List<NativeModule> = listOf(
-        QualityChecker(reactContext),
-        BatchStitcher(reactContext),
-        RNSARSession(reactContext),
-        IncrementalStitcher(reactContext),
-        FileBridge(reactContext),
-    )
+    ): List<NativeModule> {
+        // F8.4 — register the Frame Processor plugin here, after the
+        // bridge is fully booted.  See `ensureFrameProcessorPluginRegistered`
+        // for the rationale (vs. a class-load-time static init).
+        ensureFrameProcessorPluginRegistered()
+        return listOf(
+            QualityChecker(reactContext),
+            BatchStitcher(reactContext),
+            RNSARSession(reactContext),
+            IncrementalStitcher(reactContext),
+            FileBridge(reactContext),
+        )
+    }
 
     override fun createViewManagers(
         reactContext: ReactApplicationContext,