react-native-image-stitcher 0.16.2 → 0.18.0

package/CHANGELOG.md

@@ -14,6 +14,160 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 > during 0.x are bumped to a new MINOR (e.g., 0.1 → 0.2), and the
 > upgrade path is documented in this CHANGELOG.
 
+## [0.18.0] — 2026-06-18
+
+### ⚠️ Breaking — `StitcherFrame` → `CameraFrame`
+
+The frame type a worklet receives is renamed **`StitcherFrame` →
+`CameraFrame`** (and `StitcherFrameProcessor` → `CameraFrameProcessor`).
+The shape is unchanged; only the names changed, to match the
+`arFrameProcessor` prop's role (it's the camera frame, not a "stitcher"
+frame). Update your imports:
+
+```diff
+- import { type StitcherFrame } from 'react-native-image-stitcher';
++ import { type CameraFrame } from 'react-native-image-stitcher';
+```
+
+### Added — AR depth, anchors, scene mesh, and intrinsics on `CameraFrame`
+
+The AR frame a worklet receives can now carry rich per-frame metadata,
+each behind an **opt-in `<Camera>` prop** (all off by default — you pay
+only for what you request):
+
+- **`enableDepth`** → `frame.arDepth` — a depth map normalised to **one
+  cross-platform shape**: `Float32` **metres** in `depthMap`, optional
+  `Uint8` `confidenceMap` (`0`/`1`/`2`). Sourced from ARKit
+  `sceneDepth`/`smoothedSceneDepth` (LiDAR) and the ARCore Depth API.
+- **`enableAnchors`** → `frame.arAnchors` — detected planes / images,
+  now with plane **`alignment`** (`'horizontal'` | `'vertical'`),
+  **`extent`** (`[x, z]` metres), and (iOS only) semantic
+  **`classification`** (`'wall'`/`'floor'`/…).
+- **`enableMesh`** → `type: 'mesh'` entries in `arAnchors` carrying
+  `meshGeometry` (`vertices`/`faces`/optional `classifications`
+  ArrayBuffers). iOS uses ARKit `ARMeshAnchor` scene reconstruction
+  (LiDAR); **Android reconstructs a rough mesh from the depth map**
+  (camera-local vertices, identity transform, no per-face
+  classifications) — so Android mesh requires a Depth-API device and is
+  geometry-only.
+- **`planeDetection`** (`'vertical'` (default) | `'horizontal'` |
+  `'both'`) — which plane orientations reach `arAnchors`. iOS changes
+  ARKit `planeDetection`; Android keeps detecting both (ARCore needs
+  horizontal planes to bootstrap tracking) and filters the emitted set,
+  so the JS-observable result is identical on both platforms. The
+  `'vertical'` default preserves the plane-projected stitch path's
+  long-standing behaviour.
+- **`frame.intrinsics`** — per-frame `fx`/`fy`/`cx`/`cy` (px) plus the
+  capture resolution, for lifting 2D image coordinates to 3D. Always
+  present on AR frames; `undefined` on non-AR (vision-camera) frames,
+  which have no intrinsics surface.
+
+Depth/anchor/mesh bytes are **eager-copied** out of the native frame at
+extraction time, so they're safe to read anywhere in the worklet (no
+buffer-lifetime footgun). See the new **[Testing the AR frame
+processor](https://bhargavkanda.github.io/react-native-image-stitcher/docs/dev-testing)**
+guide for a copy-paste verification recipe and the expected on-device
+output per platform.
+
+### Added — `onArFrame`: worklet-free AR metadata on the main thread
+
+`<Camera onArFrame={(meta) => …}>` is a new callback (also on
+`<ARCameraView>`) that delivers **light per-frame AR metadata on the JS
+main thread** — no worklet involved:
+
+```ts
+onArFrame={(m: ARFrameMeta) => {
+  // m.trackingState, m.pose, m.intrinsics,
+  // m.depth?.{width,height,hasConfidence},
+  // m.anchors[] (id/type/alignment/extent/classification/transform),
+  // m.mesh?.{anchorCount,vertexCount,faceCount}
+}}
+```
+
+Native builds the metadata each frame (reusing the same extraction as
+above) and emits it as a throttled event (default ≈10 Hz; tune with
+`arFrameMetaInterval` ms). Costly fields are gated by the same opt-ins
+(`depth` needs `enableDepth`, `mesh` needs `enableMesh`, `anchors` needs
+`enableAnchors`); `pose`/`trackingState`/`intrinsics` are always present.
+
+**This is the recommended way to read AR data in JS** for observe/measure
+use cases — it carries *light* data (dims, counts, intrinsics, plane
+geometry), never heavy buffers. For zero-copy access to raw per-frame
+buffers (depth pixels, mesh vertices) you'd use the `arFrameProcessor`
+worklet — see the limitation below.
+
+### Known limitation — `arFrameProcessor` worklets must capture nothing
+
+In this release an `arFrameProcessor` worklet must **not capture host
+objects** (a `runOnJS` callback or a shared value) in its closure:
+`react-native-worklets-core` deep-copies the worklet's closure when it's
+installed into the AR worklet runtime, and a captured host object makes
+that copy recurse until the stack overflows (a hard crash, on both Debug
+and Release). A worklet that captures **nothing** installs and runs fine.
+Until this is resolved upstream, **use `onArFrame`** (above) to get AR
+data into JS; reserve the worklet for capture-free per-frame work.
+
+### Known limitation — `enableMesh` is memory-heavy on sustained sessions
+
+`enableMesh` turns on ARKit **continuous scene reconstruction**, the most
+memory-intensive AR mode — the mesh model grows as you scan, and a long
+session with depth + mesh both on can be **jetsam-killed by iOS** after a
+few seconds on memory-constrained devices. `onArFrame` reports mesh as
+light *counts* (`anchorCount`/`vertexCount`/`faceCount`) without copying
+geometry, so reading mesh stats is cheap; it's the **underlying ARKit
+meshing** that's heavy. For now, enable `mesh` only for short captures (the
+example demos depth + planes + intrinsics with mesh off). Proper memory
+management for sustained meshing — bounded reconstruction, single depth
+semantic, on-demand geometry — lands with the 0.20 reconstruction work.
+
+### Internal — `StitcherFrameData` → `CameraFrameData`
+
+The shared C++ frame struct and its JSI/Obj-C++ host objects were renamed
+(`StitcherFrameData` → `CameraFrameData`, `StitcherFrameJsiHostObject` →
+`CameraFrameJsiHostObject`, `StitcherFrameHostObject` →
+`CameraFrameHostObject`) to match the public `CameraFrame` type. No public
+API change.
+
+### Notes
+
+- Compile-verified on both platforms (iOS `xcodebuild` + Android
+  `assembleDebug`); all unit tests + typecheck pass. On-device
+  observation of depth/planes/mesh/intrinsics against real surfaces is
+  the recommended pre-adoption check (see the dev-testing guide).
+
+## [0.17.0] — 2026-06-19
+
+### Added — `arFrameProcessor`: observe AR frames with a host worklet
+
+`<Camera>` gains an **`arFrameProcessor`** prop — a `'worklet'` invoked once per
+**ARKit / ARCore frame** while in AR capture, dispatched natively and running
+*alongside* first-party stitching (composition, not replacement). The worklet
+receives a `StitcherFrame` tagged `source: 'ar'` with the world-space `pose` and
+`arTrackingState`. It fires during preview too (continuous observation), at zero
+per-frame cost when no worklet is registered.
+
+This restores the previously-archived AR host-worklet capability and re-exposes
+it as an explicit prop (rather than the old auto-registering hook). Under the
+hood it installs `globalThis.__stitcherProxy` (JSI) on first use and fans frames
+out through a shared C++ proxy / registry / dispatch layer on both platforms
+(verified against `react-native-worklets-core` 1.6.3).
+
+The non-AR equivalent remains `frameProcessor` (vision-camera); the two modes use
+different runtimes and frame shapes, hence the separate prop. The
+`StitcherFrame` / `StitcherFrameProcessor` type names are unchanged.
+
+Verified on device: the worklet fires per frame on **iOS (ARKit, iPhone 16 Pro)**
+and **Android (ARCore, Galaxy A35)**.
+
+### Fixed
+
+- **Example app crashed at launch on Android** (`PlatformConstants could not be
+  found`). The v0.16.2 OpenCV-reuse demo added an app-level `externalNativeBuild`
+  to `example/android/app/build.gradle` that displaced React Native's own
+  New-Architecture app native build (so core TurboModules weren't compiled in).
+  Removed it; React Native owns the app native build again. **Example-app only —
+  the published SDK was never affected.**
+
 ## [0.16.2] — 2026-06-17
 
 ### Added — reuse the bundled OpenCV from your host app's native code (Android)

package/RNImageStitcher.podspec

@@ -60,6 +60,18 @@ Pod::Spec.new do |s|
 
   s.dependency 'React-Core'
 
+  # react-native-worklets-core — provides the `RNWorklet::WorkletInvoker`
+  # + `JsiWorkletContext` primitives the AR-mode JSI fan-out is built on
+  # (StitcherJsiInstaller.mm / RNSARWorkletRuntime.mm + the shared
+  # cpp/stitcher_worklet_{registry,dispatch}.cpp).  In practice this pod
+  # is already in every host's graph (vision-camera depends on it), but
+  # declaring it here makes the dependency explicit and guarantees its
+  # headers are present even for a host that uses AR mode without
+  # vision-camera.  The bare `WKTJsiWorklet.h` includes in the .mm files
+  # resolve via the HEADER_SEARCH_PATHS entry below (the package's own
+  # node_modules copy of the worklets-core cpp/ dir).
+  s.dependency 'react-native-worklets-core'
+
   # ─────────────────────────────────────────────────────────────────────
   # OpenCV — pre-built custom xcframework fetched by postinstall
   # ─────────────────────────────────────────────────────────────────────
@@ -84,6 +96,19 @@ Pod::Spec.new do |s|
     'CLANG_CXX_LANGUAGE_STANDARD' => 'c++17',
     'CLANG_CXX_LIBRARY' => 'libc++',
     'OTHER_CPLUSPLUSFLAGS' => '$(inherited) -std=c++17',
-    'HEADER_SEARCH_PATHS' => '$(inherited) "${PODS_TARGET_SRCROOT}/cpp"',
+    # HEADER_SEARCH_PATHS:
+    #   - "${PODS_TARGET_SRCROOT}/cpp" — the shared C++ port's own
+    #     headers (keyframe_gate.hpp, camera_frame_jsi.hpp, …).
+    #   - the worklets-core cpp/ dir — so the bare `#include
+    #     "WKTJsiWorklet.h"` / "WKTJsiWorkletContext.h" lines in
+    #     StitcherJsiInstaller.mm + RNSARWorkletRuntime.mm resolve.
+    #     PODS_ROOT is `<host>/ios/Pods`; the package's worklets-core
+    #     copy lives at `<host>/node_modules/react-native-worklets-core/
+    #     cpp`, i.e. `${PODS_ROOT}/../node_modules/...`.  (The shared
+    #     cpp/*.cpp files instead use the namespace-prefixed
+    #     `<react-native-worklets-core/WKTJsiWorklet.h>` form, which
+    #     resolves against `${PODS_ROOT}/Headers/Public` — already on
+    #     the inherited path — and works on Android's prefab too.)
+    'HEADER_SEARCH_PATHS' => '$(inherited) "${PODS_TARGET_SRCROOT}/cpp" "${PODS_ROOT}/../node_modules/react-native-worklets-core/cpp"',
   }
 end

package/android/build.gradle

@@ -301,6 +301,26 @@ dependencies {
         android.sourceSets.main.java.exclude '**/CvFlowGateFrameProcessor.kt'
     }
 
+    // v0.8.0 Phase 4b.ii/iii — react-native-worklets-core, consumed
+    // for its `rnworklets` PREFAB module (RNWorklet::JsiWorkletContext /
+    // WorkletInvoker) by the AR frame-processor JSI fan-out
+    // (src/main/cpp/stitcher_jsi_install_jni.cpp + the shared cpp/
+    // stitcher_*_jsi.cpp).  `find_package(react-native-worklets-core ...)`
+    // in CMakeLists.txt needs this gradle project on the path so AGP
+    // wires the prefab into the native build.  Same consumption pattern
+    // react-native-vision-camera uses for Frame Processors.
+    //
+    // `findProject(...)` guard mirrors the vision-camera block above:
+    // the SDK still compiles in hosts that don't install worklets-core
+    // (those hosts simply don't use the AR frame processor — the JSI
+    // sources still compile, but the prefab link only happens when the
+    // host provides worklets-core).  Autolinking adds the project for
+    // any host that depends on vision-camera (which transitively pulls
+    // in worklets-core).
+    if (findProject(':react-native-worklets-core') != null) {
+        implementation project(':react-native-worklets-core')
+    }
+
     // v0.10.0 audit #11A — Android JUnit test scaffold.  JVM unit
     // tests for pure-Kotlin data wrappers + algorithm helpers that
     // don't need an Android device.  Run via

package/android/src/main/cpp/CMakeLists.txt

@@ -66,6 +66,20 @@ add_library(opencv_stitching STATIC IMPORTED)
 set_target_properties(opencv_stitching PROPERTIES
     IMPORTED_LOCATION "${OPENCV_STITCHING_A}")
 
+# ── React Native + worklets-core prefabs (v0.8.0 Phase 4b.ii/iii) ──
+#
+# The AR frame-processor's JSI fan-out (stitcher_jsi_install_jni.cpp +
+# the shared cpp/stitcher_*_jsi.cpp) needs RN's JSI runtime, fbjni, and
+# react-native-worklets-core's `RNWorklet::JsiWorkletContext` /
+# `WorkletInvoker`.  RN 0.71+ ships jsi + the native-modules umbrella as
+# prefab packages; worklets-core ships a `rnworklets` prefab module.
+# `buildFeatures.prefab true` (android/build.gradle) makes these
+# discoverable.  We mirror react-native-vision-camera's consumption of
+# the SAME prefabs in this example app (verified working on RN 0.84.1).
+find_package(ReactAndroid REQUIRED CONFIG)
+find_package(fbjni REQUIRED CONFIG)
+find_package(react-native-worklets-core REQUIRED CONFIG)
+
 # ── Shared C++ port (KeyframeGate) ────────────────────────────────
 #
 # `cpp/` at the SDK root holds C++ that's compiled into BOTH the iOS
@@ -91,12 +105,26 @@ add_library(image_stitcher SHARED
     # retry + dimension/memory instrumentation.  Used to live in this
     # file (image_stitcher_jni.cpp).  See cpp/stitcher.hpp for design
     # rationale.
-    "${SHARED_CPP_DIR}/stitcher.cpp")
+    "${SHARED_CPP_DIR}/stitcher.cpp"
+    # v0.8.0 Phase 4b.ii/iii — AR frame-processor JSI fan-out.  The JNI
+    # binding (stitcher_jsi_install_jni.cpp) installs
+    # `globalThis.__stitcherProxy` and fans each AR frame out to the
+    # registered host worklets.  The 4 shared cpp/ JSI files below are
+    # the SAME source the iOS pod compiles — one cross-platform JSI
+    # surface (proxy host object + native worklet registry + per-frame
+    # dispatch helper + CameraFrame JSI host object).
+    stitcher_jsi_install_jni.cpp
+    "${SHARED_CPP_DIR}/stitcher_proxy_jsi.cpp"
+    "${SHARED_CPP_DIR}/stitcher_worklet_registry.cpp"
+    "${SHARED_CPP_DIR}/stitcher_worklet_dispatch.cpp"
+    "${SHARED_CPP_DIR}/camera_frame_jsi.cpp")
 
 target_include_directories(image_stitcher PRIVATE
     "${OPENCV_INCLUDE_DIR}"
     # cpp/ holds keyframe_gate.hpp + ar_frame_pose.h that the JNI
-    # bindings include without relative-path spelunking.
+    # bindings include without relative-path spelunking.  Also the
+    # shared JSI sources (stitcher_proxy_jsi.hpp, camera_frame_data.hpp,
+    # stitcher_worklet_*.hpp, camera_frame_jsi.hpp).
     "${SHARED_CPP_DIR}")
 
 # Link order matters:
@@ -118,7 +146,22 @@ target_link_libraries(image_stitcher
     opencv_stitching
     -Wl,--no-whole-archive
     opencv_java
-    log)
+    log
+    # v0.8.0 Phase 4b.ii/iii — JSI fan-out deps.  Mirrors
+    # react-native-vision-camera's link set on RN 0.84.1:
+    #   ReactAndroid::jsi         — facebook::jsi::Runtime / Value / Object
+    #   ReactAndroid::reactnative — RN's native-modules umbrella prefab
+    #                               (RN >= 0.76; CallInvoker + friends that
+    #                               worklets-core's context depends on)
+    #   fbjni::fbjni              — JNI helpers worklets-core links against
+    #   react-native-worklets-core::rnworklets — JsiWorkletContext +
+    #                               WorkletInvoker.  Carries its own
+    #                               include dir so <react-native-worklets-core/
+    #                               WKTJsiWorkletContext.h> resolves.
+    ReactAndroid::jsi
+    ReactAndroid::reactnative
+    fbjni::fbjni
+    react-native-worklets-core::rnworklets)
 
 target_compile_options(image_stitcher PRIVATE
     -fvisibility=hidden