react-native-image-stitcher 0.10.0 → 0.11.0

package/CHANGELOG.md

@@ -16,6 +16,102 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.11.0] — 2026-05-28
+
+### Added — `useStitcherWorklet` for non-AR composition
+
+Closes the v0.8.0 Phase 5 either-or constraint: hosts that want to
+write their OWN `useFrameProcessor` worklet body can now COMPOSE
+first-party stitching back in with a single `stitcher.call(frame)`
+call, instead of having to choose between their worklet and the
+lib's stitching.
+
+```tsx
+import {
+  Camera, useFrameProcessor, useStitcherWorklet,
+  type StitcherFrame,
+} from 'react-native-image-stitcher';
+
+function MyScreen() {
+  const stitcher = useStitcherWorklet();
+  const fp = useFrameProcessor((frame: StitcherFrame) => {
+    'worklet';
+    hostPreLogic(frame);
+    stitcher.call(frame);   // ← first-party stitching
+    hostPostLogic(frame);
+  }, [stitcher.call]);
+  return <Camera frameProcessor={fp} ... />;
+}
+```
+
+Migrating from v0.10.x is a one-line diff:
+
+```diff
++ const stitcher = useStitcherWorklet();
+  const fp = useFrameProcessor((frame: StitcherFrame) => {
+    'worklet';
++   stitcher.call(frame);   // ← first-party stitching back in
+    hostLogic(frame);
+- }, [hostLogic]);
++ }, [stitcher.call, hostLogic]);
+```
+
+### Changed
+
+- `useFrameProcessorDriver` is now a thin wrapper around
+  `useStitcherWorklet`.  Public API (`start` / `stop` /
+  `isRunning` / `frameProcessor`) is unchanged.  Pose-reset
+  semantics preserved via the new `stitcher.reset()` method which
+  the driver calls internally from `start()` and `stop()`.
+- The gyro subscription that powers pose tracking now lives in
+  `useStitcherWorklet` and runs for the lifetime of the hook
+  (mount → unmount) rather than being tied to the driver's
+  `start()` / `stop()`.  In practice this matches all observed
+  host integrations (capture screens mount `<Camera>` for the
+  duration of capture; idle screens don't).  Battery delta is
+  small (≪1% CPU at 33 ms gyro sampling).
+- `<Camera frameProcessor>` JSDoc rewritten: the "Non-AR mode
+  tradeoff (HONEST)" section is replaced by a "Non-AR mode
+  composition" section that shows the v0.11.0 composition
+  pattern.  The runtime `console.info` text is softened from
+  "your worklet REPLACES first-party stitching, panorama capture
+  will not produce stitched output" to "if you want first-party
+  stitching alongside, call `useStitcherWorklet()` from your
+  worklet body".
+- Example app (`example/App.tsx`) now demonstrates the
+  composition pattern end-to-end: one
+  `useFrameProcessor` body that calls both `stitcher.call(frame)`
+  and the existing 1 Hz host tick log.  `<Camera>` mounts with
+  `frameProcessor={exampleFrameProcessor}` (previously left
+  unwired with an "intentionally unused" comment block).
+- `docs/frame-access-tiers.md` adds a `useStitcherWorklet`
+  reference section + 1-line migration diff.  Softens the
+  "either-or" language in the Tier 3 + AR-vs-non-AR sections.
+
+### Files changed
+- NEW: `src/stitching/useStitcherWorklet.ts`
+- `src/stitching/useFrameProcessorDriver.ts` (refactored thin wrapper)
+- `src/index.ts` (export new hook + types)
+- `src/camera/Camera.tsx` (docstring + console.info softened)
+- `example/App.tsx` (composition demo)
+- `docs/frame-access-tiers.md` (new section + softened wording)
+- `docs/v0.11.0-manual-verification-checklist.md` (Phase 4 human-loop checklist)
+
+### Not touched
+- All native code (`ios/Sources/`, `android/src/main/cpp/`,
+  `android/src/main/java/io/imagestitcher/rn/`) — pure TS refactor.
+- AR-mode dispatch path — already composes natively.
+- `useFrameProcessor` (v0.8.0 public hook) — unchanged.
+
+### Verified
+- JS Jest: **69 / 69 pass**
+- C++ Gtest: **17 / 17 pass**
+- Android JUnit: **6 / 6 pass**
+- iOS build (Debug, generic iOS device): clean
+- Android `:app:assembleDebug`: clean
+- Real-device panorama capture verification deferred to the
+  human-in-the-loop checklist (`docs/v0.11.0-manual-verification-checklist.md`).
+
 ## [0.10.0] — 2026-05-28
 
 ### Added — v0.10.0 PR A: host-side test infrastructure (`#9A` + `#11A`)

package/dist/camera/Camera.d.ts

@@ -221,26 +221,42 @@ export interface CameraProps {
      * }
      * ```
      *
-     * ## Non-AR mode tradeoff (HONEST)
+     * ## Non-AR mode composition (v0.11.0+)
      *
      * vision-camera's `<Camera>` accepts ONLY ONE frame processor.
      * The lib's internal `useFrameProcessorDriver` produces the
      * processor that drives first-party panorama stitching in non-AR
-     * mode.  If you supply your own via this prop, **the lib's
-     * first-party stitching is replaced** — panorama capture in
-     * non-AR mode will not produce stitched output until you remove
-     * the prop or fork the SDK to compose both worklets manually.
+     * mode.  If you supply your own via this prop, the lib's
+     * default processor is REPLACED — but as of v0.11.0 you can
+     * COMPOSE first-party stitching back into your worklet body
+     * using `useStitcherWorklet`:
      *
-     * For the common case (host wants worklet + lib wants stitching
-     * concurrently), prefer AR mode: the AR-mode path natively fans
-     * out to both the lib's first-party stitching AND every
-     * registered host worklet on every frame, with per-worklet
-     * failure isolation.
+     * ```tsx
+     * import {
+     *   Camera, useFrameProcessor, useStitcherWorklet,
+     *   type StitcherFrame,
+     * } from 'react-native-image-stitcher';
+     *
+     * function MyScreen() {
+     *   const stitcher = useStitcherWorklet();
+     *   const fp = useFrameProcessor((frame: StitcherFrame) => {
+     *     'worklet';
+     *     hostPreLogic(frame);
+     *     stitcher.call(frame);   // ← first-party stitching
+     *     hostPostLogic(frame);
+     *   }, [stitcher.call]);
+     *   return <Camera frameProcessor={fp} ... />;
+     * }
+     * ```
      *
-     * Composition for non-AR mode (lib stitching + host worklet on
-     * the same vc processor) is tracked as a v0.9+ follow-up;
-     * needs the lib's first-party logic exposed as a vc Frame
-     * Processor plugin the host's worklet can call.
+     * Hosts that DON'T call `useStitcherWorklet` from their worklet
+     * body replace first-party stitching for non-AR captures (a
+     * one-shot console.info documents this when the prop is first
+     * supplied).  AR mode is unaffected either way — the AR-mode
+     * dispatch path (v0.8.0 Phase 4b.i / 4b.iii) natively fans out
+     * to both the lib's first-party stitching AND every registered
+     * host worklet on every frame, with per-worklet failure
+     * isolation.
      *
      * ## AR mode behaviour
      *

package/dist/camera/Camera.js

@@ -435,36 +435,36 @@ function Camera(props) {
     // Safety: stop the driver if the component unmounts mid-recording.
     // eslint-disable-next-line react-hooks/exhaustive-deps
     (0, react_1.useEffect)(() => () => { fpDriver.stop(); }, []);
-    // v0.8.0 Phase 5 — frameProcessor prop semantics:
+    // v0.8.0 Phase 5 / v0.11.0 — frameProcessor prop semantics:
     //
-    //   - Host supplied? → use host's processor; lib's first-party
-    //     stitching is DISABLED in non-AR mode (vc accepts only one
-    //     processor).  One-shot console.info documents the tradeoff
-    //     so the host isn't surprised by "panorama capture stopped
-    //     producing output" in non-AR mode.  AR-mode capture is
-    //     unaffected — the AR-session dispatch path fans out to BOTH
-    //     first-party and host worklets independently.
+    //   - Host supplied? → use host's processor.  The host's worklet
+    //     body controls whether first-party stitching also fires:
+    //     call `stitcher.call(frame)` (from `useStitcherWorklet`)
+    //     inside the body to compose; omit to replace.  One-shot
+    //     console.info documents the choice so the host can spot a
+    //     missing `useStitcherWorklet` call before they go hunting
+    //     for "why is non-AR panorama capture not producing output".
+    //     AR-mode capture is unaffected either way — the AR-session
+    //     dispatch path fans out to BOTH first-party stitching AND
+    //     every host worklet independently.
     //
     //   - No host processor? → use `fpDriver.frameProcessor` which is
     //     the lib's internal worklet driving first-party stitching
     //     via `useFrameProcessorDriver`.  Default behaviour for the
     //     common "I just want panorama capture" case.
-    //
-    // The pre-v0.8.0 behaviour (host's prop silently ignored with
-    // a warning) is gone — Phase 5 plumbs the prop through.  The
-    // tradeoff is honestly documented in the CameraProps docstring.
     const hostFrameProcessorAcceptedWarnedRef = (0, react_1.useRef)(false);
     if (hostFrameProcessor != null
         && !hostFrameProcessorAcceptedWarnedRef.current) {
         hostFrameProcessorAcceptedWarnedRef.current = true;
         // eslint-disable-next-line no-console
         console.info('[react-native-image-stitcher] Host frameProcessor supplied — '
-            + 'non-AR mode will run YOUR worklet instead of the lib\'s '
-            + 'first-party stitching plugin (vc accepts only one frame '
-            + 'processor).  Non-AR panorama capture will not produce '
-            + 'stitched output until this prop is removed.  AR-mode '
-            + 'capture is unaffected (AR-session dispatch fans out to '
-            + 'both first-party and host worklets independently).');
+            + 'non-AR mode will run YOUR composed worklet.  If you want '
+            + 'first-party panorama stitching alongside your own logic, '
+            + 'call `useStitcherWorklet()` and invoke `stitcher.call(frame)` '
+            + 'from your worklet body (see `<Camera>` `frameProcessor` '
+            + 'JSDoc for the composition pattern).  AR-mode capture is '
+            + 'unaffected (AR-session dispatch fans out to both '
+            + 'first-party and host worklets independently).');
     }
     // The Frame Processor worklet bound to vision-camera's Camera.
     // Host's wins if supplied; lib's internal driver otherwise.

package/dist/index.d.ts

@@ -73,5 +73,7 @@ export { useFrameStream } from './stitching/useFrameStream';
 export type { FrameStreamOptions, SampledFrame } from './types';
 export { useFrameProcessorDriver } from './stitching/useFrameProcessorDriver';
 export type { UseFrameProcessorDriverOptions, FrameProcessorDriverHandle, } from './stitching/useFrameProcessorDriver';
+export { useStitcherWorklet } from './stitching/useStitcherWorklet';
+export type { UseStitcherWorkletOptions, StitcherWorkletHandle, StitcherWorkletInput, } from './stitching/useStitcherWorklet';
 export { stitchVideo } from './stitching/stitchVideo';
 //# sourceMappingURL=index.d.ts.map

package/dist/index.js

@@ -22,7 +22,7 @@
  * adds RetaiLens-specific features on top.
  */
 Object.defineProperty(exports, "__esModule", { value: true });
-exports.stitchVideo = exports.useFrameProcessorDriver = exports.useFrameStream = exports.useThrottledFrameProcessor = exports.useFrameProcessor = exports.useKeyframeStream = exports.useIncrementalStitcher = exports.cleanupOldKeyframes = exports.getIncrementalNativeModule = exports.subscribeIncrementalState = exports.incrementalStitcherIsAvailable = exports.IncrementalOutcome = exports.useDeviceOrientation = exports.useVideoCapture = exports.useCapture = exports.ViewportCropOverlay = exports.hybridSettingsToNativeConfig = exports.slitscanSettingsToNativeConfig = exports.panoramaSettingsToNativeConfig = exports.DEFAULT_HYBRID_SETTINGS = exports.DEFAULT_SLITSCAN_SETTINGS = exports.DEFAULT_FLOW_GATE_SETTINGS = exports.DEFAULT_PANORAMA_SETTINGS = exports.PanoramaSettingsModal = exports.PanoramaGuidance = exports.PanoramaBandOverlay = exports.IncrementalPanGuide = exports.CaptureThumbnailStrip = exports.useStitchStatsToast = exports.CaptureStitchStatsToast = exports.CaptureOrientationPill = exports.CaptureKeyframePill = exports.CaptureMemoryPill = exports.CaptureDebugOverlay = exports.CaptureStatusOverlay = exports.CapturePreview = exports.CaptureControlsBar = exports.CaptureHeader = exports.CameraView = exports.ARCameraView = exports.useIMUTranslationGate = exports.ARTrackingState = exports.useARSession = exports.CameraError = exports.Camera = void 0;
+exports.stitchVideo = exports.useStitcherWorklet = exports.useFrameProcessorDriver = exports.useFrameStream = exports.useThrottledFrameProcessor = exports.useFrameProcessor = exports.useKeyframeStream = exports.useIncrementalStitcher = exports.cleanupOldKeyframes = exports.getIncrementalNativeModule = exports.subscribeIncrementalState = exports.incrementalStitcherIsAvailable = exports.IncrementalOutcome = exports.useDeviceOrientation = exports.useVideoCapture = exports.useCapture = exports.ViewportCropOverlay = exports.hybridSettingsToNativeConfig = exports.slitscanSettingsToNativeConfig = exports.panoramaSettingsToNativeConfig = exports.DEFAULT_HYBRID_SETTINGS = exports.DEFAULT_SLITSCAN_SETTINGS = exports.DEFAULT_FLOW_GATE_SETTINGS = exports.DEFAULT_PANORAMA_SETTINGS = exports.PanoramaSettingsModal = exports.PanoramaGuidance = exports.PanoramaBandOverlay = exports.IncrementalPanGuide = exports.CaptureThumbnailStrip = exports.useStitchStatsToast = exports.CaptureStitchStatsToast = exports.CaptureOrientationPill = exports.CaptureKeyframePill = exports.CaptureMemoryPill = exports.CaptureDebugOverlay = exports.CaptureStatusOverlay = exports.CapturePreview = exports.CaptureControlsBar = exports.CaptureHeader = exports.CameraView = exports.ARCameraView = exports.useIMUTranslationGate = exports.ARTrackingState = exports.useARSession = exports.CameraError = exports.Camera = void 0;
 // ─────────────────────────────────────────────────────────────────────
 // Layer 1 — the high-level <Camera> component
 // ─────────────────────────────────────────────────────────────────────
@@ -179,6 +179,14 @@ Object.defineProperty(exports, "useFrameStream", { enumerable: true, get: functi
 // `useIncrementalJSDriver` was removed; was deprecated in v0.5).
 var useFrameProcessorDriver_1 = require("./stitching/useFrameProcessorDriver");
 Object.defineProperty(exports, "useFrameProcessorDriver", { enumerable: true, get: function () { return useFrameProcessorDriver_1.useFrameProcessorDriver; } });
+// v0.11.0 — composable first-party stitching as a worklet function.
+// Hosts that want to COMPOSE their own per-frame logic with the
+// lib's stitching (instead of REPLACING it via the <Camera>
+// `frameProcessor` prop) call this hook + invoke `stitcher.call`
+// inside their own `useFrameProcessor` body.  See
+// `docs/host-app-integration.md` § Tier 3 for the full pattern.
+var useStitcherWorklet_1 = require("./stitching/useStitcherWorklet");
+Object.defineProperty(exports, "useStitcherWorklet", { enumerable: true, get: function () { return useStitcherWorklet_1.useStitcherWorklet; } });
 // ── Batch stitching ───────────────────────────────────────────────────
 // Feed a video file straight to OpenCV's cv::Stitcher, bypassing the
 // incremental pipeline.  Useful when you have content captured

package/dist/stitching/useFrameProcessorDriver.d.ts

@@ -4,99 +4,54 @@
  * from v0.6 onward (replaced the deprecated `useIncrementalJSDriver`
  * hook, which was removed in v0.6).
  *
- * Why this exists (vs the pre-v0.6 JS-driver predecessor)
- *
- *   The old JS driver took a JPEG snapshot every ~250 ms and fed the
- *   path to `IncrementalStitcher.processFrameAtPath` (both removed in
- *   v0.6).  That path had three costs:
- *
- *     1. JPEG encode (`takeSnapshot` ≈ 30–80 ms on iPhone 16 Pro)
- *     2. Disk write of the JPEG
- *     3. JPEG decode + cv::Mat alloc inside the engine
- *
- *   Per-frame round-trip ~80 ms means ~4 Hz max throughput, and
- *   ~80 ms latency between "this is the moment to accept" and "this
- *   frame is in the engine".  Both numbers caused operator-felt lag
- *   on long shelf pans.
- *
- *   This hook uses vision-camera's Frame Processor instead.  The
- *   worklet runs on the camera producer thread at the native frame
- *   rate (30 fps on iOS).  Each frame goes through a JSI plugin
- *   (`cv_flow_gate_process_frame`) directly into
- *   `IncrementalStitcher.consumeFrame` — the SAME entry point AR
- *   mode uses, with the engine's existing KeyframeGate making the
- *   accept/reject decision.  Rejected frames cost ~3–8 ms; accepted
- *   frames take the same deep-copy + workQueue path AR mode takes.
- *
- *   Net wins: no JPEG round-trip on rejected frames, no disk thrash
- *   during recording, lower latency to accept, full 30 fps gate
- *   evaluation budget.
- *
- * Pose synthesis
- *
- *   Non-AR mode has no ARKit pose.  We integrate the gyroscope on
- *   the JS thread (`react-native-sensors`), accumulate yaw + pitch,
- *   and publish them via Reanimated `useSharedValue` so the worklet
- *   can read them WITHOUT a thread hop.  Translation is reported as
- *   zero (no IMU translation; this is a known limitation we share
- *   with the legacy driver — drift ~1–2°/min over a 30 s capture is
- *   below the gate's overlap threshold and rarely matters).
- *
- *   Quaternion synthesis (q = q_yaw * q_pitch * q_roll, Tait-Bryan
- *   YPR order to match the legacy driver's body-frame intent):
- *     q_yaw   = (0, sin(yaw/2), 0, cos(yaw/2))
- *     q_pitch = (sin(pitch/2), 0, 0, cos(pitch/2))
- *     q_roll  = (0, 0, sin(roll/2), cos(roll/2))
- *
- *   Expanded (cy, sy = cos/sin(yaw/2); analogous for cp/sp, cr/sr):
- *     qx = cy*sp*cr + sy*cp*sr
- *     qy = sy*cp*cr - cy*sp*sr
- *     qz = cy*cp*sr - sy*sp*cr
- *     qw = cy*cp*cr + sy*sp*sr
- *
- *   When roll=0 this collapses to the 2-axis form
- *   `(cy*sp, sy*cp, -sy*sp, cy*cp)` the legacy driver used, so
- *   captures held perfectly level produce identical poses to the
- *   pre-roll behaviour.
- *
- *   Intrinsics are synthesised from the actual frame dimensions
- *   (`frame.width`, `frame.height`) plus the host-provided
- *   horizontal/vertical FoV defaults.  The stitcher derives its FoV-
- *   overlap window from these, so the assumed FoV matters for the
- *   gate's overlap math but not for the panorama itself (the
- *   stitcher feature-matches + RANSACs the final alignment).
- *
- * Throttling
- *
- *   `evalEveryNFrames` controls how often the worklet calls the
- *   plugin.  Default 1 (every frame).  Set higher to amortise the
- *   plugin call + consumeFrame's gate evaluation across multiple
- *   producer-thread frames on lower-end devices.  Independent of —
- *   and stacks on top of — the stitcher's own internal
- *   `flowEvalEveryNFrames` (see `KeyframeGate.swift`); both
- *   throttles can be active simultaneously and the effective cadence
- *   is `evalEveryNFrames * flowEvalEveryNFrames`.
- *
- * Lifecycle
- *
- *   `start()` subscribes to the gyro and resets pose accumulators.
- *   `stop()` unsubscribes and resets.  The returned `frameProcessor`
- *   is meant to be passed to `<Camera frameProcessor={...} />` —
- *   it's stable as long as the plugin reference and the FoV props
- *   haven't changed.  Returns `null` when the plugin isn't loaded
- *   yet; pass `null`-or-fallback to the Camera in that case.
- *
- * Pairing with `IncrementalStitcher.start({frameSourceMode})`
- *
- *   The plugin's per-frame call into `consumeFrameFromPlugin` is
- *   gated by `IncrementalStitcher.frameProcessorIngestEnabled`,
- *   which is TRUE only when the stitcher was started with
- *   `frameSourceMode === 'frameProcessor'`.  Hosts MUST call
- *   `incrementalStitcher.start({ frameSourceMode: 'frameProcessor',
- *   ... })` to actually get frames into the engine — otherwise the
- *   worklet runs to completion but the wrapper drops the call.
- *   `Camera.tsx` does this wiring automatically when the host opts
- *   into this driver.
+ * v0.11.0 — refactored to a thin wrapper around `useStitcherWorklet`.
+ * The plugin acquisition + shared-value declarations + gyro
+ * subscription + worklet body all live in `useStitcherWorklet` now;
+ * this hook just binds the returned worklet via vision-camera's
+ * `useFrameProcessor` and exposes the legacy `start` / `stop` /
+ * `isRunning` API for backwards compatibility with v0.10.x.
+ *
+ * ## Why the v0.11.0 split
+ *
+ * vision-camera v4 allows ONE frame processor per `<Camera>` mount.
+ * Pre-v0.11.0, hosts that wanted to compose their own worklet with
+ * the lib's first-party stitching couldn't — passing a host
+ * `frameProcessor` REPLACED the lib's processor.  v0.11.0 closes
+ * this gap by exposing the worklet body via `useStitcherWorklet`
+ * so hosts can write:
+ *
+ *   const stitcher = useStitcherWorklet();
+ *   const fp = useFrameProcessor((frame) => {
+ *     'worklet';
+ *     hostPreLogic(frame);
+ *     stitcher.call(frame);   // ← first-party stitching
+ *     hostPostLogic(frame);
+ *   }, [stitcher.call]);
+ *
+ * `useFrameProcessorDriver` keeps the legacy default-integration
+ * shape (start / stop / isRunning) for the `<Camera>` component's
+ * built-in non-AR path and for any host still using the v0.10.x API
+ * directly.  No behavioural change for those callers.
+ *
+ * ## start / stop behaviour
+ *
+ *   - `start()` calls `stitcher.reset()` to zero the accumulated
+ *     pose (preserves the pre-v0.11.0 "each capture starts with
+ *     pose = (0, 0, 0)" contract).
+ *   - `stop()` also resets the pose (idempotent; matches the
+ *     pre-v0.11.0 stop() side effect of zeroing yaw / pitch / roll).
+ *   - The gyro subscription itself is owned by `useStitcherWorklet`
+ *     and runs for the lifetime of the hook.  In the default
+ *     `<Camera>` integration this means gyro is on while the camera
+ *     screen is mounted — same practical scope as pre-v0.11.0 in
+ *     all observed host integrations (capture screens mount
+ *     `<Camera>` for the duration of capture; idle screens don't).
+ *
+ * ## Pose synthesis / intrinsics / throttling
+ *
+ * Owned by `useStitcherWorklet`.  See that file's header for the
+ * quaternion math, FoV-to-intrinsics derivation, throttle gate, and
+ * pairing-with-IncrementalStitcher.start docs.
  */
 import type { ReadonlyFrameProcessor } from 'react-native-vision-camera';
 export interface UseFrameProcessorDriverOptions {
@@ -131,9 +86,9 @@ export interface UseFrameProcessorDriverOptions {
     evalEveryNFrames?: number;
 }
 export interface FrameProcessorDriverHandle {
-    /** Subscribe to the gyro + reset pose accumulators.  Idempotent. */
+    /** Reset pose accumulators + mark the driver as running.  Idempotent. */
     start: () => void;
-    /** Unsubscribe + reset pose. */
+    /** Reset pose + mark the driver as stopped.  Idempotent. */
     stop: () => void;
     /**
      * Pass this to `<Camera frameProcessor={...} />`.  `null` until