react-native-image-stitcher 0.17.0 → 0.18.0

package/CHANGELOG.md

@@ -14,6 +14,127 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 > during 0.x are bumped to a new MINOR (e.g., 0.1 → 0.2), and the
 > upgrade path is documented in this CHANGELOG.
 
+## [0.18.0] — 2026-06-18
+
+### ⚠️ Breaking — `StitcherFrame` → `CameraFrame`
+
+The frame type a worklet receives is renamed **`StitcherFrame` →
+`CameraFrame`** (and `StitcherFrameProcessor` → `CameraFrameProcessor`).
+The shape is unchanged; only the names changed, to match the
+`arFrameProcessor` prop's role (it's the camera frame, not a "stitcher"
+frame). Update your imports:
+
+```diff
+- import { type StitcherFrame } from 'react-native-image-stitcher';
++ import { type CameraFrame } from 'react-native-image-stitcher';
+```
+
+### Added — AR depth, anchors, scene mesh, and intrinsics on `CameraFrame`
+
+The AR frame a worklet receives can now carry rich per-frame metadata,
+each behind an **opt-in `<Camera>` prop** (all off by default — you pay
+only for what you request):
+
+- **`enableDepth`** → `frame.arDepth` — a depth map normalised to **one
+  cross-platform shape**: `Float32` **metres** in `depthMap`, optional
+  `Uint8` `confidenceMap` (`0`/`1`/`2`). Sourced from ARKit
+  `sceneDepth`/`smoothedSceneDepth` (LiDAR) and the ARCore Depth API.
+- **`enableAnchors`** → `frame.arAnchors` — detected planes / images,
+  now with plane **`alignment`** (`'horizontal'` | `'vertical'`),
+  **`extent`** (`[x, z]` metres), and (iOS only) semantic
+  **`classification`** (`'wall'`/`'floor'`/…).
+- **`enableMesh`** → `type: 'mesh'` entries in `arAnchors` carrying
+  `meshGeometry` (`vertices`/`faces`/optional `classifications`
+  ArrayBuffers). iOS uses ARKit `ARMeshAnchor` scene reconstruction
+  (LiDAR); **Android reconstructs a rough mesh from the depth map**
+  (camera-local vertices, identity transform, no per-face
+  classifications) — so Android mesh requires a Depth-API device and is
+  geometry-only.
+- **`planeDetection`** (`'vertical'` (default) | `'horizontal'` |
+  `'both'`) — which plane orientations reach `arAnchors`. iOS changes
+  ARKit `planeDetection`; Android keeps detecting both (ARCore needs
+  horizontal planes to bootstrap tracking) and filters the emitted set,
+  so the JS-observable result is identical on both platforms. The
+  `'vertical'` default preserves the plane-projected stitch path's
+  long-standing behaviour.
+- **`frame.intrinsics`** — per-frame `fx`/`fy`/`cx`/`cy` (px) plus the
+  capture resolution, for lifting 2D image coordinates to 3D. Always
+  present on AR frames; `undefined` on non-AR (vision-camera) frames,
+  which have no intrinsics surface.
+
+Depth/anchor/mesh bytes are **eager-copied** out of the native frame at
+extraction time, so they're safe to read anywhere in the worklet (no
+buffer-lifetime footgun). See the new **[Testing the AR frame
+processor](https://bhargavkanda.github.io/react-native-image-stitcher/docs/dev-testing)**
+guide for a copy-paste verification recipe and the expected on-device
+output per platform.
+
+### Added — `onArFrame`: worklet-free AR metadata on the main thread
+
+`<Camera onArFrame={(meta) => …}>` is a new callback (also on
+`<ARCameraView>`) that delivers **light per-frame AR metadata on the JS
+main thread** — no worklet involved:
+
+```ts
+onArFrame={(m: ARFrameMeta) => {
+  // m.trackingState, m.pose, m.intrinsics,
+  // m.depth?.{width,height,hasConfidence},
+  // m.anchors[] (id/type/alignment/extent/classification/transform),
+  // m.mesh?.{anchorCount,vertexCount,faceCount}
+}}
+```
+
+Native builds the metadata each frame (reusing the same extraction as
+above) and emits it as a throttled event (default ≈10 Hz; tune with
+`arFrameMetaInterval` ms). Costly fields are gated by the same opt-ins
+(`depth` needs `enableDepth`, `mesh` needs `enableMesh`, `anchors` needs
+`enableAnchors`); `pose`/`trackingState`/`intrinsics` are always present.
+
+**This is the recommended way to read AR data in JS** for observe/measure
+use cases — it carries *light* data (dims, counts, intrinsics, plane
+geometry), never heavy buffers. For zero-copy access to raw per-frame
+buffers (depth pixels, mesh vertices) you'd use the `arFrameProcessor`
+worklet — see the limitation below.
+
+### Known limitation — `arFrameProcessor` worklets must capture nothing
+
+In this release an `arFrameProcessor` worklet must **not capture host
+objects** (a `runOnJS` callback or a shared value) in its closure:
+`react-native-worklets-core` deep-copies the worklet's closure when it's
+installed into the AR worklet runtime, and a captured host object makes
+that copy recurse until the stack overflows (a hard crash, on both Debug
+and Release). A worklet that captures **nothing** installs and runs fine.
+Until this is resolved upstream, **use `onArFrame`** (above) to get AR
+data into JS; reserve the worklet for capture-free per-frame work.
+
+### Known limitation — `enableMesh` is memory-heavy on sustained sessions
+
+`enableMesh` turns on ARKit **continuous scene reconstruction**, the most
+memory-intensive AR mode — the mesh model grows as you scan, and a long
+session with depth + mesh both on can be **jetsam-killed by iOS** after a
+few seconds on memory-constrained devices. `onArFrame` reports mesh as
+light *counts* (`anchorCount`/`vertexCount`/`faceCount`) without copying
+geometry, so reading mesh stats is cheap; it's the **underlying ARKit
+meshing** that's heavy. For now, enable `mesh` only for short captures (the
+example demos depth + planes + intrinsics with mesh off). Proper memory
+management for sustained meshing — bounded reconstruction, single depth
+semantic, on-demand geometry — lands with the 0.20 reconstruction work.
+
+### Internal — `StitcherFrameData` → `CameraFrameData`
+
+The shared C++ frame struct and its JSI/Obj-C++ host objects were renamed
+(`StitcherFrameData` → `CameraFrameData`, `StitcherFrameJsiHostObject` →
+`CameraFrameJsiHostObject`, `StitcherFrameHostObject` →
+`CameraFrameHostObject`) to match the public `CameraFrame` type. No public
+API change.
+
+### Notes
+
+- Compile-verified on both platforms (iOS `xcodebuild` + Android
+  `assembleDebug`); all unit tests + typecheck pass. On-device
+  observation of depth/planes/mesh/intrinsics against real surfaces is
+  the recommended pre-adoption check (see the dev-testing guide).
+
 ## [0.17.0] — 2026-06-19
 
 ### Added — `arFrameProcessor`: observe AR frames with a host worklet

package/RNImageStitcher.podspec

@@ -98,7 +98,7 @@ Pod::Spec.new do |s|
     'OTHER_CPLUSPLUSFLAGS' => '$(inherited) -std=c++17',
     # HEADER_SEARCH_PATHS:
     #   - "${PODS_TARGET_SRCROOT}/cpp" — the shared C++ port's own
-    #     headers (keyframe_gate.hpp, stitcher_frame_jsi.hpp, …).
+    #     headers (keyframe_gate.hpp, camera_frame_jsi.hpp, …).
     #   - the worklets-core cpp/ dir — so the bare `#include
     #     "WKTJsiWorklet.h"` / "WKTJsiWorkletContext.h" lines in
     #     StitcherJsiInstaller.mm + RNSARWorkletRuntime.mm resolve.

package/android/src/main/cpp/CMakeLists.txt

@@ -112,19 +112,19 @@ add_library(image_stitcher SHARED
     # registered host worklets.  The 4 shared cpp/ JSI files below are
     # the SAME source the iOS pod compiles — one cross-platform JSI
     # surface (proxy host object + native worklet registry + per-frame
-    # dispatch helper + StitcherFrame JSI host object).
+    # dispatch helper + CameraFrame JSI host object).
     stitcher_jsi_install_jni.cpp
     "${SHARED_CPP_DIR}/stitcher_proxy_jsi.cpp"
     "${SHARED_CPP_DIR}/stitcher_worklet_registry.cpp"
     "${SHARED_CPP_DIR}/stitcher_worklet_dispatch.cpp"
-    "${SHARED_CPP_DIR}/stitcher_frame_jsi.cpp")
+    "${SHARED_CPP_DIR}/camera_frame_jsi.cpp")
 
 target_include_directories(image_stitcher PRIVATE
     "${OPENCV_INCLUDE_DIR}"
     # cpp/ holds keyframe_gate.hpp + ar_frame_pose.h that the JNI
     # bindings include without relative-path spelunking.  Also the
-    # shared JSI sources (stitcher_proxy_jsi.hpp, stitcher_frame_data.hpp,
-    # stitcher_worklet_*.hpp, stitcher_frame_jsi.hpp).
+    # shared JSI sources (stitcher_proxy_jsi.hpp, camera_frame_data.hpp,
+    # stitcher_worklet_*.hpp, camera_frame_jsi.hpp).
     "${SHARED_CPP_DIR}")
 
 # Link order matters:

package/android/src/main/cpp/stitcher_jsi_install_jni.cpp

@@ -33,8 +33,8 @@
 // v0.8.0 Phase 4b.iii — per-frame fan-out support.  The shared
 // `dispatchToHostWorklets` posts to worklets-core's default context;
 // this JNI file's `nativeDispatchToHostWorklets` constructs the
-// `StitcherFrameData` from raw bytes + pose + dims and forwards it.
-#include "stitcher_frame_data.hpp"
+// `CameraFrameData` from raw bytes + pose + dims and forwards it.
+#include "camera_frame_data.hpp"
 #include "stitcher_worklet_dispatch.hpp"
 #include "stitcher_worklet_registry.hpp"
 
@@ -76,7 +76,7 @@ Java_io_imagestitcher_rn_StitcherJsiInstallerModule_nativeInstall(
 // Owns a heap-allocated `std::vector<uint8_t>` of pre-copied NV21
 // bytes.  Constructed by `nativeDispatchToHostWorklets` after one
 // JNI byte-array copy from Kotlin; outlives the AR render thread
-// scope via `StitcherFrameData::pixelReader`'s `shared_ptr` —
+// scope via `CameraFrameData::pixelReader`'s `shared_ptr` —
 // dropped when the host object is invalidated.
 
 namespace {
@@ -117,6 +117,29 @@ Java_io_imagestitcher_rn_StitcherWorkletRuntime_nativeRegistryCount(
       retailens::StitcherWorkletRegistry::shared().count());
 }
 
+// ─── per-frame extraction-config gate ──────────────────────────────
+//
+// Returns the current `retailens::getExtractionConfig()` packed into a
+// `jint` bitmask so Kotlin can cheaply read the JS-driven
+// enableDepth/enableAnchors/enableMesh toggles once per frame and skip
+// the costly ARCore depth-acquire / anchor-collect / mesh-build work
+// when a host hasn't opted in.  Same atomic-snapshot read the JSI
+// `setExtractionConfig` host function writes.
+//
+//   bit0 (0x1) = depth
+//   bit1 (0x2) = anchors
+//   bit2 (0x4) = mesh
+extern "C" JNIEXPORT jint JNICALL
+Java_io_imagestitcher_rn_StitcherWorkletRuntime_nativeExtractionFlags(
+    JNIEnv* /*env*/, jobject /*thiz*/) {
+  const retailens::ExtractionConfig cfg = retailens::getExtractionConfig();
+  jint flags = 0;
+  if (cfg.depth) flags |= 0x1;
+  if (cfg.anchors) flags |= 0x2;
+  if (cfg.mesh) flags |= 0x4;
+  return flags;
+}
+
 // ─── v0.8.0 Phase 4b.iii — per-frame dispatch JNI binding ──────────
 //
 // Called from Kotlin's `StitcherWorkletRuntime.dispatchToHostWorklets`
@@ -143,7 +166,18 @@ Java_io_imagestitcher_rn_StitcherWorkletRuntime_nativeDispatchToHostWorklets(
     jdouble qx, jdouble qy, jdouble qz, jdouble qw,
     jdouble tx, jdouble ty, jdouble tz,
     jdouble timestampNs,
-    jstring trackingState) {
+    jstring trackingState,
+    jbyteArray depthBytes,
+    jint depthWidth, jint depthHeight,
+    jobjectArray anchorIds,
+    jobjectArray anchorTypes,
+    jobjectArray anchorTransforms,
+    jobjectArray anchorMeshVertices,
+    jobjectArray anchorMeshFaces,
+    jdouble fx, jdouble fy, jdouble cx, jdouble cy,
+    jint intrinsicsImageWidth, jint intrinsicsImageHeight,
+    jobjectArray anchorAlignments,
+    jobjectArray anchorExtents) {
   // Fast-path early-exit BEFORE the JNI byte-array copy.  Saves the
   // ~3MB memcpy + JSI host object alloc on every frame in the
   // common first-party-only case.
@@ -183,10 +217,10 @@ Java_io_imagestitcher_rn_StitcherWorkletRuntime_nativeDispatchToHostWorklets(
     }
   }
 
-  // Build StitcherFrameData.  Field semantics match the iOS
-  // `StitcherFrameHostObject::fromARFrame:pose:` factory; this is
+  // Build CameraFrameData.  Field semantics match the iOS
+  // `CameraFrameHostObject::fromARFrame:pose:` factory; this is
   // the Android equivalent path.
-  retailens::StitcherFrameData data;
+  retailens::CameraFrameData data;
   data.source = "ar";
   data.width = static_cast<int32_t>(width);
   data.height = static_cast<int32_t>(height);
@@ -214,6 +248,181 @@ Java_io_imagestitcher_rn_StitcherWorkletRuntime_nativeDispatchToHostWorklets(
   data.pixelReader =
       std::make_shared<AndroidNV21BufferReader>(std::move(bytes));
 
+  // ── AR depth (ARCore DEPTH16, "u16packed") ────────────────────────
+  //
+  // Kotlin hands us a dense, row-packed uint16-per-pixel byte array
+  // (depthWidth*depthHeight*2 bytes; low 13 bits = mm, high 3 bits =
+  // confidence 0..7) or null when depth is unavailable this frame.  We
+  // copy the bytes verbatim into `data.arDepth.depthBytes` with
+  // `format = "u16packed"` and leave `confidenceBytes` EMPTY — the
+  // shared JSI layer (`cpp/camera_frame_jsi.cpp`) unpacks mm->metres
+  // and confidence 0..7 -> 0..2 from the high bits.
+  if (depthBytes != nullptr && depthWidth > 0 && depthHeight > 0) {
+    const jsize depthLen = env->GetArrayLength(depthBytes);
+    if (depthLen > 0) {
+      retailens::ArDepth depth;
+      depth.width = static_cast<int32_t>(depthWidth);
+      depth.height = static_cast<int32_t>(depthHeight);
+      depth.format = "u16packed";
+      depth.depthBytes.resize(static_cast<std::size_t>(depthLen));
+      env->GetByteArrayRegion(
+          depthBytes, 0, depthLen,
+          reinterpret_cast<jbyte*>(depth.depthBytes.data()));
+      // confidenceBytes intentionally left empty (packed into the high
+      // 3 bits of each uint16 — unpacked JSI-side).
+      data.arDepth = std::move(depth);
+    }
+  }
+
+  // ── Per-frame camera intrinsics ───────────────────────────────────
+  //
+  // Kotlin passes camera.imageIntrinsics (fx,fy,cx,cy in pixels) + the
+  // capture resolution they're expressed at.  fx <= 0.0 is the "no
+  // intrinsics" sentinel (defensive — AR frames always carry valid
+  // intrinsics, but a degenerate session could yield 0).  The shared
+  // JSI layer exposes `intrinsics === undefined` when !hasIntrinsics.
+  if (fx > 0.0) {
+    data.hasIntrinsics = true;
+    data.fx = fx;
+    data.fy = fy;
+    data.cx = cx;
+    data.cy = cy;
+    data.intrinsicsImageWidth = static_cast<int32_t>(intrinsicsImageWidth);
+    data.intrinsicsImageHeight = static_cast<int32_t>(intrinsicsImageHeight);
+  }
+
+  // ── AR anchors ────────────────────────────────────────────────────
+  //
+  // Five parallel arrays from Kotlin: ids (String[]), types (String[]),
+  // transforms (double[16][]), and the per-anchor mesh byte arrays
+  // meshVertices (byte[][], Float32 xyz triplets) + meshFaces (byte[][],
+  // Uint32 triangle indices) — both NULL for non-mesh anchors.  Build one
+  // `retailens::ArAnchor` per entry; the transform is already ROW-MAJOR
+  // (anchor->world) — Kotlin transposed ARCore's column-major OpenGL
+  // matrix before marshaling (mesh anchors emit identity: the vertices
+  // are camera-local).  Empty arrays (the common case — no host opted
+  // into anchors/mesh) leave `data.arAnchors` empty, which the JSI layer
+  // surfaces as `[]` for source=="ar".
+  if (anchorIds != nullptr && anchorTypes != nullptr &&
+      anchorTransforms != nullptr) {
+    const jsize anchorCount = env->GetArrayLength(anchorIds);
+    data.arAnchors.reserve(static_cast<std::size_t>(anchorCount));
+    for (jsize i = 0; i < anchorCount; ++i) {
+      retailens::ArAnchor anchor;
+
+      auto idObj = reinterpret_cast<jstring>(
+          env->GetObjectArrayElement(anchorIds, i));
+      if (idObj != nullptr) {
+        const char* cs = env->GetStringUTFChars(idObj, nullptr);
+        if (cs != nullptr) {
+          anchor.id = cs;
+          env->ReleaseStringUTFChars(idObj, cs);
+        }
+        env->DeleteLocalRef(idObj);
+      }
+
+      auto typeObj = reinterpret_cast<jstring>(
+          env->GetObjectArrayElement(anchorTypes, i));
+      if (typeObj != nullptr) {
+        const char* cs = env->GetStringUTFChars(typeObj, nullptr);
+        if (cs != nullptr) {
+          anchor.type = cs;
+          env->ReleaseStringUTFChars(typeObj, cs);
+        }
+        env->DeleteLocalRef(typeObj);
+      }
+
+      auto transformObj = reinterpret_cast<jdoubleArray>(
+          env->GetObjectArrayElement(anchorTransforms, i));
+      if (transformObj != nullptr) {
+        const jsize n = env->GetArrayLength(transformObj);
+        jdouble* elems = env->GetDoubleArrayElements(transformObj, nullptr);
+        if (elems != nullptr) {
+          const jsize copyN = (n < 16) ? n : 16;
+          for (jsize j = 0; j < copyN; ++j) {
+            anchor.transform[static_cast<std::size_t>(j)] =
+                static_cast<double>(elems[j]);
+          }
+          env->ReleaseDoubleArrayElements(transformObj, elems, JNI_ABORT);
+        }
+        env->DeleteLocalRef(transformObj);
+      }
+
+      // ── per-anchor plane alignment + extent ─────────────────────────
+      //
+      // anchorAlignments[i] is "" for image/mesh anchors (→ JS
+      // `alignment === undefined`) or "horizontal"/"vertical" for plane
+      // anchors.  anchorExtents[i] is null for non-plane anchors or a
+      // double[2] = {extentX, extentZ} (metres) for planes.  Both arrays
+      // are parallel to anchorIds; guard for null (a caller passing the
+      // older arg shape) the same way the mesh arrays are guarded.  We do
+      // NOT set classification — Android has no plane semantics (iOS-only).
+      if (anchorAlignments != nullptr) {
+        auto alignObj = reinterpret_cast<jstring>(
+            env->GetObjectArrayElement(anchorAlignments, i));
+        if (alignObj != nullptr) {
+          const char* cs = env->GetStringUTFChars(alignObj, nullptr);
+          if (cs != nullptr) {
+            if (cs[0] != '\0') {
+              anchor.alignment = cs;
+            }
+            env->ReleaseStringUTFChars(alignObj, cs);
+          }
+          env->DeleteLocalRef(alignObj);
+        }
+      }
+      if (anchorExtents != nullptr) {
+        auto extObj = reinterpret_cast<jdoubleArray>(
+            env->GetObjectArrayElement(anchorExtents, i));
+        if (extObj != nullptr) {
+          if (env->GetArrayLength(extObj) >= 2) {
+            jdouble vals[2] = {0.0, 0.0};
+            env->GetDoubleArrayRegion(extObj, 0, 2, vals);
+            anchor.hasExtent = true;
+            anchor.extentX = static_cast<double>(vals[0]);
+            anchor.extentZ = static_cast<double>(vals[1]);
+          }
+          env->DeleteLocalRef(extObj);
+        }
+      }
+
+      // ── per-anchor mesh geometry (depth-derived; type=="mesh") ──────
+      //
+      // anchorMeshVertices[i] / anchorMeshFaces[i] are null for non-mesh
+      // anchors and a byte[] for a mesh anchor.  When BOTH are present we
+      // copy them verbatim into the ArAnchor's vectors and flag hasMesh —
+      // the JSI layer (`cpp/camera_frame_jsi.cpp`) emits them as
+      // ArrayBuffers (Float32 vertices / Uint32 faces) unchanged.
+      // meshClassifications stays empty (Android depth meshes carry no
+      // per-face semantics).
+      if (anchorMeshVertices != nullptr && anchorMeshFaces != nullptr) {
+        auto vertObj = reinterpret_cast<jbyteArray>(
+            env->GetObjectArrayElement(anchorMeshVertices, i));
+        auto faceObj = reinterpret_cast<jbyteArray>(
+            env->GetObjectArrayElement(anchorMeshFaces, i));
+        if (vertObj != nullptr && faceObj != nullptr) {
+          const jsize vLen = env->GetArrayLength(vertObj);
+          const jsize fLen = env->GetArrayLength(faceObj);
+          if (vLen > 0 && fLen > 0) {
+            anchor.meshVertices.resize(static_cast<std::size_t>(vLen));
+            env->GetByteArrayRegion(
+                vertObj, 0, vLen,
+                reinterpret_cast<jbyte*>(anchor.meshVertices.data()));
+            anchor.meshFaces.resize(static_cast<std::size_t>(fLen));
+            env->GetByteArrayRegion(
+                faceObj, 0, fLen,
+                reinterpret_cast<jbyte*>(anchor.meshFaces.data()));
+            anchor.hasMesh = true;
+          }
+        }
+        if (vertObj != nullptr) env->DeleteLocalRef(vertObj);
+        if (faceObj != nullptr) env->DeleteLocalRef(faceObj);
+      }
+
+      data.arAnchors.push_back(std::move(anchor));
+    }
+  }
+
   // Dispatch on worklets-core's default context.  That context is
   // initialised by JS' `Worklets.install()` (which runs at lib
   // bootstrap when worklets-core's module is imported); by the