react-native-image-stitcher 0.14.2 → 0.15.1

package/CHANGELOG.md

@@ -16,6 +16,170 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.15.1] — 2026-06-08
+
+### Fixed
+
+- **Camera preview now matches capture FOV on all paths (letterbox WYSIWYG).**
+  The preview and captured photo now share the same field of view regardless of
+  the container size the host app uses.  Black letterbox bars fill any extra
+  space rather than cropping or stretching the camera feed.
+  - *VisionCamera path:* `CameraView` measures its rendered bounds via
+    `onLayout`, pins the format to 4:3 with `useCameraFormat`, then sizes the
+    `<Camera>` component to the largest axis-aligned box that fits the container
+    while preserving the format aspect ratio.
+  - *ARCore path (Android):* `RNSARCameraView` now selects a camera config
+    whose image aspect and texture aspect match within 2% (`selectMatchingCameraConfig`).
+    On devices (e.g. Galaxy A35) where no 4:3 matched config exists, the best
+    available 16:9 config is chosen — both preview and capture are 16:9.
+    The GL renderer letterboxes the camera texture inside the GL surface using
+    `setDisplayGeometry` + `glViewport`, centred on a black-cleared surface.
+  - *ARKit path (iOS):* `RNSARCameraView.layoutSubviews()` reads
+    `imageResolution` from the ARKit session and centres the scene view inside
+    the container bounds using the same aspect-correct letterbox calculation.
+
+- **ARCore CPU image resolution upgraded automatically.**  `selectMatchingCameraConfig`
+  prefers the highest-resolution matched config, so CPU image captures used for
+  stitching are now at full sensor resolution (1920×1080 on the Galaxy A35,
+  up from 640×480) with no API change required.
+
+### Changed
+
+- **`defaultCaptureSource` changed from `'ar'` to `'non-ar'`.**  AR mode is now
+  opt-in.  Host apps that want AR must pass `defaultCaptureSource="ar"` or
+  implement a toggle; the plain camera path is the default.
+
+## [0.15.0] — 2026-06-07
+
+### Breaking — only `batch-keyframe` remains; host-worklet / frame-stream hooks removed
+
+The live/incremental stitching engines (hybrid, slit-scan, firstwins) and the
+third-party host-worklet / frame-stream observer API were archived (kept under
+`archive/`, excluded from every build surface) so the SDK now ships only the
+`batch-keyframe` capture path.  Removed from the public API:
+
+- **Hooks** `useFrameProcessor`, `useThrottledFrameProcessor`, `useFrameStream`
+  and their option types (`ThrottledFrameProcessorOptions`, `FrameStreamOptions`,
+  `SampledFrame`).  To compose first-party stitching, use vision-camera's own
+  `useFrameProcessor` with `useStitcherWorklet().call(frame)` (see the example app).
+- The **slit-scan / hybrid** panorama-engine settings types and their
+  native-config adapters (`slitscanSettingsToNativeConfig`,
+  `hybridSettingsToNativeConfig`).
+
+A type-only break for the default batch-keyframe path; per the 0.x stability
+policy this bumps a new MINOR.
+
+### Changed — iOS + Android unified on the manual `cv::detail` stitch pipeline
+
+Both platforms now run the **same** manual `cv::detail` stitch pipeline
+(`useManualPipeline=true` on both), so a given capture produces consistent,
+more robust output regardless of platform.  Previously iOS used the manual
+pipeline while Android used the high-level `cv::Stitcher` — the two diverged on
+resolution, exposure handling, and wide-capture robustness.  The unified manual
+path carries:
+
+- **Exposure compensation** (`cv::detail::GainCompensator`, GAIN_BLOCKS) — evens
+  brightness/colour across frames before blending, removing the visible seam
+  steps the manual path previously had.
+- **Matched registration / compositing resolution** (registration 0.6 MP,
+  composite 1.0 MP) on both platforms.
+- The **cylindrical warp fallback** (below), so wide / 0.5× captures survive on
+  both platforms.
+
+The decision was made after an on-device A/B (manual vs high-level at matched
+resolution): with parity the manual path matched the high-level on quality and
+was strictly more robust on wide captures.  Background + the verification trail
+are recorded in [`docs/stitch-pipeline-architecture.md`](docs/stitch-pipeline-architecture.md).
+
+### Added — cylindrical warp fallback for wide / 0.5× captures
+
+When the configured (plane) warper would diverge on a wide or 0.5× ultra-wide
+capture — a single frame's warp canvas exceeding the 100 MP guard — the stitcher
+now auto-retries with the bounded cylindrical projection instead of failing with
+`STITCH_CAMERA_PARAMS_FAIL`.  Wide and ultra-wide (0.5×) panoramas that
+previously errored out now complete.  Because the pipeline is now unified
+(above), this fallback applies on both iOS and Android.
+
+### Added — `userFacingStitchError()` for friendly recoverable-stitch copy
+
+New public SDK export that maps a recoverable stitch `CameraErrorCode`
+(`STITCH_NEED_MORE_IMGS`, `STITCH_CAMERA_PARAMS_FAIL`, `STITCH_HOMOGRAPHY_FAIL`,
+`STITCH_OOM`) to friendly, action-guiding `{ title, message }` copy for a host
+`Alert` / toast — so the user sees "pan more slowly" / "pivot in place" instead
+of the raw `cv::Stitcher` diagnostic.  Returns `null` for every non-recoverable
+code (permission denied, device unavailable, generic finalize failure, unknown,
+…), so the host falls back to its generic error UI.  Call it from `onError`:
+
+```tsx
+import { userFacingStitchError } from 'react-native-image-stitcher';
+
+onError={(err) => {
+  const friendly = userFacingStitchError(err.code);
+  if (friendly) Alert.alert(friendly.title, friendly.message);
+  else reportGenericError(err);
+}}
+```
+
+Also exports the `UserFacingStitchError` type (`{ title, message }`).  Lives in
+the SDK (not per-host) so every consumer shows the same vetted guidance for the
+same failure, and so the mapping is unit-testable in isolation.
+
+### Fixed — friendlier stitch-failure classification + example UX
+
+- `STITCH_NEED_MORE_IMGS` now also classifies the manual pipeline's "0 valid
+  pairwise matches / frames may not overlap enough" failure, which previously
+  surfaced as a generic `PANORAMA_FINALIZE_FAILED`.  Both insufficient-overlap
+  signals now map to the same recoverable "pan more slowly" outcome (and so pick
+  up the `userFacingStitchError` copy above).
+- The example app now shows friendly, action-guiding guidance — via
+  `userFacingStitchError` (an Alert) on a stitch failure (`onError`), and a
+  transient **toast** when frames are dropped for insufficient overlap
+  (`onFramesDropped`) — shown only when **>30%** of the requested frames are
+  missing from the final stitch (e.g. ≥2 of 6), so minor drops stay silent.  The toast (`CaptureStitchStatsToast`) also
+  gained optional `title` (bold, above the message) and `placement`
+  (`'top'` | `'center'`) props; the example shows a centered title+body toast.
+  Failure alerts now lead with the corrective ask as the title (e.g. "Please
+  pan more slowly" / "Try a shorter sweep") and explain the cause in the body.
+
+### Fixed — reach the ultra-wide by device-swap when a logical multi-cam can't (Samsung / Camera2)
+
+`selectCaptureDevice` now device-swaps to a standalone ultra-wide camera when a
+logical multi-cam device merely *lists* the ultra-wide but can't reach it by
+`zoom` (its zoom range starts at 1.0 — common on Android / Camera2 / Samsung,
+where the ultra-wide is a separate physical camera rather than a zoom target).
+Previously such devices stayed on the multi-cam device and 0.5× showed the
+wide-angle FOV.  A logical device whose zoom range genuinely extends to the
+ultra-wide (e.g. iOS virtual devices, `minZoom ≈ 0.5`) is still preferred and
+lens-switches via zoom as before.
+
+### Added — time-budget keyframe force-accept (`maxKeyframeIntervalMs`)
+
+The keyframe gate now force-accepts a keyframe when a configurable wall-clock
+interval has elapsed since the last accepted keyframe — even if the novelty /
+overlap threshold wasn't met — so a slow or static pan never leaves a temporal
+gap.  Default **2000 ms (2 s)**; `0` disables it.  Configurable via the
+`<Camera defaultMaxKeyframeIntervalMs>` prop, the `FrameSelectionSettings.maxKeyframeIntervalMs`
+field, or the in-app settings panel.  Applies to BOTH AR (plane-overlap) and
+non-AR (flow) capture paths; force-accepted keyframes count toward
+`maxKeyframes` (the cap still finalises the capture).
+
+### Added — inscribed-rect panorama crop (opt-in)
+
+`<Camera maxInscribedRectCrop={true}>` (and the `enableMaxInscribedRectCrop`
+panorama setting) crops the finished panorama to the largest axis-aligned
+rectangle inscribed in the coverage mask — clean edges with no black corners
+from unfilled projection regions.  **It is opt-in; the default is off.**  The
+default crop stays the bounding box of non-black pixels, which preserves all
+stitched content but can leave black corners.  Inscribed-rect can shrink the
+output substantially on lopsided or ultra-wide masks, so it isn't the default.
+
+### Fixed — Android keyframe-gate flow reason labels
+
+`KeyframeGate.reasonFromCode` (Android) didn't map the v0.3.0 flow-strategy
+reason codes 12–15, so accepted keyframes logged as `unknown(12)`.  They now
+read `ok-flow` / `first-flow` / `overlap-too-high (flow)` / `ok-flow-translation`,
+matching the iOS labels.  Logging only — keyframe selection is unchanged.
+
 ## [0.14.2] — 2026-06-03
 
 ### Fixed — AR preview blank on first entry (intermittent camera-handoff race)

package/README.md

@@ -212,7 +212,9 @@ These mirror the in-app settings panel; most apps never set them.
 | `defaultFlowMaxTranslationCm` | `number` | `50` | Max IMU translation between keyframes; 0 = disabled. |
 | `defaultKeyframeMaxCount` | `number` | `6` | Keyframe cap per capture (3–10). |
 | `defaultKeyframeOverlapThreshold` | `number` | `0.20` | Min overlap to accept a keyframe (0.20–0.60). |
+| `defaultMaxKeyframeIntervalMs` | `number` | `2000` | Time-budget force-accept: take a keyframe at least every N ms during a pan even if the overlap/novelty threshold isn't met, so a slow or static pan never leaves a temporal gap. Force-accepted keyframes still count toward the keyframe cap. `0` = disabled. AR + non-AR. Also exposed as the `FrameSelectionSettings.maxKeyframeIntervalMs` settings field and in the in-app settings panel. |
 | `defaultCompositingResolMP` / `defaultRegistrationResolMP` / `defaultSeamEstimationResolMP` | `number` | — | Forward-looking cv::Stitcher resolution knobs (currently no-ops). |
+| `maxInscribedRectCrop` | `boolean` | `false` | Opt in with `true` to crop the panorama to the largest inscribed rectangle (clean edges, no black corners) instead of the bounding box. Default keeps the bounding-box crop (all stitched content; may show black corners). Inscribed-rect can shrink the output on lopsided / ultra-wide masks. |
 
 ### UI toggles
 
@@ -287,6 +289,39 @@ type CameraCaptureResult =
 `STITCH_HOMOGRAPHY_FAIL`, `STITCH_CAMERA_PARAMS_FAIL`, `STITCH_OOM`,
 `OUTPUT_WRITE_FAILED`, plus `VISION_CAMERA_RUNTIME`.
 
+#### Friendly copy for recoverable stitch failures — `userFacingStitchError`
+
+The four `STITCH_*` codes are *recoverable*: the user can usually fix them by
+re-capturing (pan more slowly, pivot in place, shorten the sweep). For those,
+the SDK exports `userFacingStitchError(code)` — it returns
+`{ title, message }` of vetted, action-guiding copy you can drop straight into a
+host `Alert`/toast (instead of surfacing the raw `cv::Stitcher` diagnostic), and
+returns `null` for every non-recoverable code so you fall back to your generic
+error UI:
+
+```tsx
+import {
+  Camera,
+  userFacingStitchError,
+  type UserFacingStitchError,
+} from 'react-native-image-stitcher';
+import { Alert } from 'react-native';
+
+<Camera
+  onError={(err) => {
+    const friendly: UserFacingStitchError | null = userFacingStitchError(err.code);
+    if (friendly) {
+      Alert.alert(friendly.title, friendly.message); // "pan more slowly", "pivot in place", …
+    } else {
+      reportGenericError(err); // permission denied, device unavailable, etc.
+    }
+  }}
+/>;
+```
+
+It lives in the SDK (not per-host) so every consumer shows the same guidance for
+the same failure. The `example/` app uses it end-to-end.
+
 ### Migration from 0.13.x
 
 - **Removed:** the `panGuide` and `panoramaGuidance` props (the

package/RNImageStitcher.podspec

@@ -35,14 +35,15 @@ Pod::Spec.new do |s|
 
   # Sources: iOS-specific Swift/Obj-C/Obj-C++ AND the shared C++ port
   # (cpp/) that both iOS and Android compile from a single source.
+  # cpp/ glob is NON-RECURSIVE on purpose: it picks up the shared C++
+  # port (all top-level cpp/*.cpp) but skips the maintainer-only
+  # GoogleTest harnesses under cpp/tests/ (which would otherwise fail
+  # the pod with `'gtest/gtest.h' file not found`). NOTE: using
+  # `cpp/**` + `s.exclude_files = ['cpp/tests/**/*']` instead broke the
+  # vendored opencv2.xcframework header integration for the remaining
+  # cpp/ files — keep this as a single non-recursive glob.
   s.source_files = ['ios/Sources/**/*.{swift,h,m,mm}',
-                    'cpp/**/*.{h,hpp,cpp}']
-  # Exclude the lib's own C++ unit tests — they #include <gtest/gtest.h>,
-  # which consumer apps don't vendor.  The `cpp/**/*.cpp` glob above
-  # otherwise slurps cpp/tests/*.cpp into every host pod build, failing
-  # with `'gtest/gtest.h' file not found`.  Tests build only in the lib's
-  # CI / example app, never in a consumer.
-  s.exclude_files = ['cpp/tests/**/*']
+                    'cpp/*.{h,hpp,cpp}']
   # Restrict the umbrella header to ONLY the iOS-side Obj-C `.h`
   # files.  Without this, CocoaPods defaults every header in
   # `source_files` (including the C++ `.hpp` files under cpp/) to

package/android/build.gradle

@@ -267,22 +267,6 @@ dependencies {
         android.sourceSets.main.java.exclude '**/CvFlowGateFrameProcessor.kt'
     }
 
-    // v0.8.0 Phase 4b.ii — react-native-worklets-core's Android
-    // prefab (`rnworklets`) is consumed by the native shim
-    // (`stitcher_worklet_registry.cpp` constructs
-    // `RNWorklet::WorkletInvoker`s).  `implementation` not
-    // `compileOnly` because we need the prefab's `.so` available at
-    // both link time AND runtime — without the runtime presence,
-    // `dlopen` would fail when our `libimage_stitcher.so` is loaded.
-    //
-    // Host apps that use this lib already declare worklets-core as
-    // a peer dep (see package.json's peerDependencies); RN
-    // autolinking + Gradle deduplicates, so the host doesn't get
-    // a second copy.
-    if (findProject(':react-native-worklets-core') != null) {
-        implementation project(':react-native-worklets-core')
-    }
-
     // v0.10.0 audit #11A — Android JUnit test scaffold.  JVM unit
     // tests for pure-Kotlin data wrappers + algorithm helpers that
     // don't need an Android device.  Run via

package/android/src/main/cpp/CMakeLists.txt

@@ -80,34 +80,6 @@ if(NOT EXISTS "${SHARED_CPP_DIR}/keyframe_gate.hpp")
         "Expected react-native-image-stitcher/cpp/ — was the package layout broken?")
 endif()
 
-# ── React Native prefab packages for JSI ──────────────────────────
-#
-# v0.8.0 Phase 3 — activating the previously-deferred JSI integration.
-# The shared C++ host object (cpp/stitcher_frame_jsi.cpp) depends on
-# `facebook::jsi`.  ReactAndroid ships JSI as a prefab starting
-# RN 0.71+; the lib targets RN 0.84 so this is always available.
-#
-# `buildFeatures { prefab true }` in android/build.gradle enables
-# consumption + `ANDROID_STL=c++_shared` aligns the STL with what
-# the prefabs require.  The Phase-2 STL probe (`llvm-nm
-# libopencv_stitching.a | grep '__ndk1'`) confirmed OpenCV's
-# stitching archive was already built with c++_shared (768
-# __ndk1 symbols + 0 __cxx11 / NSt3) — switching the lib's flag
-# from c++_static to c++_shared just aligns + matches.  The
-# previous c++_static was working only because the JNI shim's
-# `.so` boundary used POD/C types; the new c++_shared is properly
-# matched throughout.
-find_package(ReactAndroid REQUIRED CONFIG)
-find_package(fbjni REQUIRED CONFIG)
-
-# v0.8.0 Phase 4b.ii — react-native-worklets-core prefab.  The
-# Gradle module name is `react-native-worklets-core`; inside it
-# publishes a library named `rnworklets` (matches vc's consumption
-# pattern in node_modules/react-native-vision-camera/android/CMakeLists.txt).
-# We consume both the headers (for `WKTJsiWorklet.h` etc.) AND
-# the .so (for `RNWorklet::WorkletInvoker` + `JsiWrapper::unwrap`
-# symbols, which are defined in worklets-core's WKTJsiWrapper.cpp).
-find_package(react-native-worklets-core REQUIRED CONFIG)
 
 # ── Our shim ───────────────────────────────────────────────────────
 add_library(image_stitcher SHARED
@@ -119,30 +91,7 @@ add_library(image_stitcher SHARED
     # retry + dimension/memory instrumentation.  Used to live in this
     # file (image_stitcher_jni.cpp).  See cpp/stitcher.hpp for design
     # rationale.
-    "${SHARED_CPP_DIR}/stitcher.cpp"
-    # v0.8.0 Phase 3 — shared JSI host object for `StitcherFrame`.
-    # Compiles to identical dispatch on both platforms; iOS consumes
-    # it via the .mm shim at
-    # `ios/Sources/RNImageStitcher/StitcherFrameHostObject.mm`.
-    # See cpp/stitcher_frame_jsi.hpp for the class API.
-    "${SHARED_CPP_DIR}/stitcher_frame_jsi.cpp"
-    # v0.8.0 Phase 4b.ii — shared C++ registry of host-supplied
-    # worklets + the `globalThis.__stitcherProxy` host object that
-    # JS calls into.  iOS picked these up via the podspec glob in
-    # Phase 4b.i; Android adds them here.
-    "${SHARED_CPP_DIR}/stitcher_worklet_registry.cpp"
-    "${SHARED_CPP_DIR}/stitcher_proxy_jsi.cpp"
-    # v0.8.0 Phase 4b.iii — shared per-frame fan-out helper.  Posts
-    # a `StitcherFrameData` onto worklets-core's default context's
-    # worklet thread; iterates the host worklet registry; invalidates
-    # the JSI host object after dispatch completes.
-    "${SHARED_CPP_DIR}/stitcher_worklet_dispatch.cpp"
-    # v0.8.0 Phase 4b.ii — Android JNI bindings for the JSI install
-    # (`StitcherJsiInstallerModule.nativeInstall`).  Reaches into the
-    # main JS runtime via the `long` JSI handle Kotlin pulls from
-    # `ReactApplicationContext.getJavaScriptContextHolder()`.  See
-    # worklets-core's `WorkletsModule.java` for the canonical pattern.
-    stitcher_jsi_install_jni.cpp)
+    "${SHARED_CPP_DIR}/stitcher.cpp")
 
 target_include_directories(image_stitcher PRIVATE
     "${OPENCV_INCLUDE_DIR}"
@@ -169,17 +118,7 @@ target_link_libraries(image_stitcher
     opencv_stitching
     -Wl,--no-whole-archive
     opencv_java
-    log
-    # v0.8.0 Phase 3 — JSI for the shared C++ host object
-    # (cpp/stitcher_frame_jsi.cpp's `facebook::jsi::HostObject`
-    # subclass).  fbjni for the Phase 3c JNI bridge between Kotlin
-    # worklet runtime + C++ host object construction.
-    ReactAndroid::jsi
-    fbjni::fbjni
-    # v0.8.0 Phase 4b.ii — worklets-core's `RNWorklet::WorkletInvoker`
-    # is constructed in the C++ registry's `install` method and
-    # invoked from the Android per-frame dispatch path.
-    react-native-worklets-core::rnworklets)
+    log)
 
 target_compile_options(image_stitcher PRIVATE
     -fvisibility=hidden

package/android/src/main/cpp/image_stitcher_jni.cpp

@@ -116,6 +116,20 @@ Java_io_imagestitcher_rn_BatchStitcher_nativeStitchFramePaths(
         ? retailens::StitchMode::Panorama
         : retailens::StitchMode::Scans;
 
+    // 2026-06-07 — unify on the manual cv::detail pipeline.  It won the
+    // on-device A/B: equals the high-level cv::Stitcher on quality after
+    // parity AND is strictly more robust — the cylindrical fallback, warp
+    // guard, and exposure comp all live only in the manual path, so the
+    // high-level path garbages wide/0.5x captures.  Mirrors iOS'
+    // OpenCVStitcher.mm.  See docs/stitch-pipeline-architecture.md §7.
+    cfg.useManualPipeline = true;
+    // Match iOS' parity resolution: the manual entry's default registration
+    // is 0.3 MP (vs the high-level's 0.6); bump to 0.6 unless the caller set
+    // an explicit value.  (compositingResolMP already arrives as 1.0.)
+    if (cfg.registrationResolMP <= 0.0) {
+        cfg.registrationResolMP = 0.6;
+    }
+
     const std::string outPath = jstring_to_string(env, outputPath);
 
     retailens::StitchResult result = retailens::stitchFramePaths(

package/android/src/main/cpp/keyframe_gate_jni.cpp

@@ -110,6 +110,19 @@ Java_io_imagestitcher_rn_KeyframeGate_nativeSetFlowMaxTranslationM(
     gate(handle)->setFlowMaxTranslationM(static_cast<double>(metres));
 }
 
+// Wall-clock keyframe-interval budget (milliseconds).  Force-accepts a
+// frame once the elapsed time since the last accepted keyframe exceeds
+// this value (applies to BOTH Pose and Flow strategies); 0 disables.
+// Passed straight through — no unit conversion.  See
+// setMaxKeyframeIntervalMs doc in keyframe_gate.hpp.  Android JNI
+// counterpart of the iOS bridge method in KeyframeGateBridge.
+JNIEXPORT void JNICALL
+Java_io_imagestitcher_rn_KeyframeGate_nativeSetMaxKeyframeIntervalMs(
+    JNIEnv*, jclass, jlong handle, jdouble ms)
+{
+    gate(handle)->setMaxKeyframeIntervalMs(static_cast<double>(ms));
+}
+
 // 2026-05-14 — Android JNI for the percentile setter so JS Settings
 // can tune novelty aggregation on Android (was iOS-only until now).
 // See setFlowNoveltyPercentile doc in keyframe_gate.hpp.