react-native-image-stitcher 0.17.0 → 0.19.0

package/CHANGELOG.md

@@ -14,6 +14,157 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 > during 0.x are bumped to a new MINOR (e.g., 0.1 → 0.2), and the
 > upgrade path is documented in this CHANGELOG.
 
+## [0.19.0] — 2026-06-19
+
+### Added — Native AR frame-processor plugins
+
+AR-mode `<Camera>` can now run **native per-frame plugins** with zero-copy
+access to the AR frame — the foundation for on-device CV (OCR, object
+detection, reconstruction feeds) **without** baking that domain code into the
+SDK. The SDK ships only the generic framework; plugins live in your app.
+
+- **Plugin interface:** implement `RNISARFramePlugin` (iOS) / `ARFramePlugin`
+  (Android) — `name()` + `process(context)`.
+- **`ARFrameContext`** hands the plugin the frame **zero-copy**: the camera
+  buffer, `pose`, `intrinsics`, tracking state, timestamp, and — when the
+  matching `enable*` prop is on — `depth` + `anchors`. The buffer is valid
+  **only during `process()`**; copy it before offloading to another thread.
+- **Register at startup:** `RNISARPluginRegistry.shared.register(…)` (iOS) /
+  `RNSARPluginRegistry.register(…)` (Android). The SDK invokes registered
+  plugins per AR frame, gated on a **non-empty registry** — zero-plugin apps
+  pay nothing.
+- **Two result channels:** light **synchronous** `process()` returns fold into
+  `onArFrame`'s `ARFrameMeta.plugins` (keyed by plugin name); heavy / **async**
+  results are pushed via `registry.emit(name, result)` → the new
+  **`onArPluginResult`** callback prop (delivered off the AR thread — for slow
+  work like OCR that must not block frame capture).
+- The example ships a sample `FrameBrightnessPlugin` (both platforms),
+  surfaced live in the AR overlay.
+
+Device-verified on iPhone 16 Pro. The SDK stays dependency-light — no OCR / ML
+runtimes are added to core.
+
+## [0.18.0] — 2026-06-18
+
+### ⚠️ Breaking — `StitcherFrame` → `CameraFrame`
+
+The frame type a worklet receives is renamed **`StitcherFrame` →
+`CameraFrame`** (and `StitcherFrameProcessor` → `CameraFrameProcessor`).
+The shape is unchanged; only the names changed, to match the
+`arFrameProcessor` prop's role (it's the camera frame, not a "stitcher"
+frame). Update your imports:
+
+```diff
+- import { type StitcherFrame } from 'react-native-image-stitcher';
++ import { type CameraFrame } from 'react-native-image-stitcher';
+```
+
+### Added — AR depth, anchors, scene mesh, and intrinsics on `CameraFrame`
+
+The AR frame a worklet receives can now carry rich per-frame metadata,
+each behind an **opt-in `<Camera>` prop** (all off by default — you pay
+only for what you request):
+
+- **`enableDepth`** → `frame.arDepth` — a depth map normalised to **one
+  cross-platform shape**: `Float32` **metres** in `depthMap`, optional
+  `Uint8` `confidenceMap` (`0`/`1`/`2`). Sourced from ARKit
+  `sceneDepth`/`smoothedSceneDepth` (LiDAR) and the ARCore Depth API.
+- **`enableAnchors`** → `frame.arAnchors` — detected planes / images,
+  now with plane **`alignment`** (`'horizontal'` | `'vertical'`),
+  **`extent`** (`[x, z]` metres), and (iOS only) semantic
+  **`classification`** (`'wall'`/`'floor'`/…).
+- **`enableMesh`** → `type: 'mesh'` entries in `arAnchors` carrying
+  `meshGeometry` (`vertices`/`faces`/optional `classifications`
+  ArrayBuffers). iOS uses ARKit `ARMeshAnchor` scene reconstruction
+  (LiDAR); **Android reconstructs a rough mesh from the depth map**
+  (camera-local vertices, identity transform, no per-face
+  classifications) — so Android mesh requires a Depth-API device and is
+  geometry-only.
+- **`planeDetection`** (`'vertical'` (default) | `'horizontal'` |
+  `'both'`) — which plane orientations reach `arAnchors`. iOS changes
+  ARKit `planeDetection`; Android keeps detecting both (ARCore needs
+  horizontal planes to bootstrap tracking) and filters the emitted set,
+  so the JS-observable result is identical on both platforms. The
+  `'vertical'` default preserves the plane-projected stitch path's
+  long-standing behaviour.
+- **`frame.intrinsics`** — per-frame `fx`/`fy`/`cx`/`cy` (px) plus the
+  capture resolution, for lifting 2D image coordinates to 3D. Always
+  present on AR frames; `undefined` on non-AR (vision-camera) frames,
+  which have no intrinsics surface.
+
+Depth/anchor/mesh bytes are **eager-copied** out of the native frame at
+extraction time, so they're safe to read anywhere in the worklet (no
+buffer-lifetime footgun). See the new **[Testing the AR frame
+processor](https://bhargavkanda.github.io/react-native-image-stitcher/docs/dev-testing)**
+guide for a copy-paste verification recipe and the expected on-device
+output per platform.
+
+### Added — `onArFrame`: worklet-free AR metadata on the main thread
+
+`<Camera onArFrame={(meta) => …}>` is a new callback (also on
+`<ARCameraView>`) that delivers **light per-frame AR metadata on the JS
+main thread** — no worklet involved:
+
+```ts
+onArFrame={(m: ARFrameMeta) => {
+  // m.trackingState, m.pose, m.intrinsics,
+  // m.depth?.{width,height,hasConfidence},
+  // m.anchors[] (id/type/alignment/extent/classification/transform),
+  // m.mesh?.{anchorCount,vertexCount,faceCount}
+}}
+```
+
+Native builds the metadata each frame (reusing the same extraction as
+above) and emits it as a throttled event (default ≈10 Hz; tune with
+`arFrameMetaInterval` ms). Costly fields are gated by the same opt-ins
+(`depth` needs `enableDepth`, `mesh` needs `enableMesh`, `anchors` needs
+`enableAnchors`); `pose`/`trackingState`/`intrinsics` are always present.
+
+**This is the recommended way to read AR data in JS** for observe/measure
+use cases — it carries *light* data (dims, counts, intrinsics, plane
+geometry), never heavy buffers. For zero-copy access to raw per-frame
+buffers (depth pixels, mesh vertices) you'd use the `arFrameProcessor`
+worklet — see the limitation below.
+
+### Known limitation — `arFrameProcessor` worklets must capture nothing
+
+In this release an `arFrameProcessor` worklet must **not capture host
+objects** (a `runOnJS` callback or a shared value) in its closure:
+`react-native-worklets-core` deep-copies the worklet's closure when it's
+installed into the AR worklet runtime, and a captured host object makes
+that copy recurse until the stack overflows (a hard crash, on both Debug
+and Release). A worklet that captures **nothing** installs and runs fine.
+Until this is resolved upstream, **use `onArFrame`** (above) to get AR
+data into JS; reserve the worklet for capture-free per-frame work.
+
+### Known limitation — `enableMesh` is memory-heavy on sustained sessions
+
+`enableMesh` turns on ARKit **continuous scene reconstruction**, the most
+memory-intensive AR mode — the mesh model grows as you scan, and a long
+session with depth + mesh both on can be **jetsam-killed by iOS** after a
+few seconds on memory-constrained devices. `onArFrame` reports mesh as
+light *counts* (`anchorCount`/`vertexCount`/`faceCount`) without copying
+geometry, so reading mesh stats is cheap; it's the **underlying ARKit
+meshing** that's heavy. For now, enable `mesh` only for short captures (the
+example demos depth + planes + intrinsics with mesh off). Proper memory
+management for sustained meshing — bounded reconstruction, single depth
+semantic, on-demand geometry — lands with the 0.20 reconstruction work.
+
+### Internal — `StitcherFrameData` → `CameraFrameData`
+
+The shared C++ frame struct and its JSI/Obj-C++ host objects were renamed
+(`StitcherFrameData` → `CameraFrameData`, `StitcherFrameJsiHostObject` →
+`CameraFrameJsiHostObject`, `StitcherFrameHostObject` →
+`CameraFrameHostObject`) to match the public `CameraFrame` type. No public
+API change.
+
+### Notes
+
+- Compile-verified on both platforms (iOS `xcodebuild` + Android
+  `assembleDebug`); all unit tests + typecheck pass. On-device
+  observation of depth/planes/mesh/intrinsics against real surfaces is
+  the recommended pre-adoption check (see the dev-testing guide).
+
 ## [0.17.0] — 2026-06-19
 
 ### Added — `arFrameProcessor`: observe AR frames with a host worklet

package/RNImageStitcher.podspec

@@ -98,7 +98,7 @@ Pod::Spec.new do |s|
     'OTHER_CPLUSPLUSFLAGS' => '$(inherited) -std=c++17',
     # HEADER_SEARCH_PATHS:
     #   - "${PODS_TARGET_SRCROOT}/cpp" — the shared C++ port's own
-    #     headers (keyframe_gate.hpp, stitcher_frame_jsi.hpp, …).
+    #     headers (keyframe_gate.hpp, camera_frame_jsi.hpp, …).
     #   - the worklets-core cpp/ dir — so the bare `#include
     #     "WKTJsiWorklet.h"` / "WKTJsiWorkletContext.h" lines in
     #     StitcherJsiInstaller.mm + RNSARWorkletRuntime.mm resolve.

package/android/src/main/cpp/CMakeLists.txt

@@ -112,19 +112,19 @@ add_library(image_stitcher SHARED
     # registered host worklets.  The 4 shared cpp/ JSI files below are
     # the SAME source the iOS pod compiles — one cross-platform JSI
     # surface (proxy host object + native worklet registry + per-frame
-    # dispatch helper + StitcherFrame JSI host object).
+    # dispatch helper + CameraFrame JSI host object).
     stitcher_jsi_install_jni.cpp
     "${SHARED_CPP_DIR}/stitcher_proxy_jsi.cpp"
     "${SHARED_CPP_DIR}/stitcher_worklet_registry.cpp"
     "${SHARED_CPP_DIR}/stitcher_worklet_dispatch.cpp"
-    "${SHARED_CPP_DIR}/stitcher_frame_jsi.cpp")
+    "${SHARED_CPP_DIR}/camera_frame_jsi.cpp")
 
 target_include_directories(image_stitcher PRIVATE
     "${OPENCV_INCLUDE_DIR}"
     # cpp/ holds keyframe_gate.hpp + ar_frame_pose.h that the JNI
     # bindings include without relative-path spelunking.  Also the
-    # shared JSI sources (stitcher_proxy_jsi.hpp, stitcher_frame_data.hpp,
-    # stitcher_worklet_*.hpp, stitcher_frame_jsi.hpp).
+    # shared JSI sources (stitcher_proxy_jsi.hpp, camera_frame_data.hpp,
+    # stitcher_worklet_*.hpp, camera_frame_jsi.hpp).
     "${SHARED_CPP_DIR}")
 
 # Link order matters:

package/android/src/main/cpp/stitcher_jsi_install_jni.cpp

@@ -33,8 +33,8 @@
 // v0.8.0 Phase 4b.iii — per-frame fan-out support.  The shared
 // `dispatchToHostWorklets` posts to worklets-core's default context;
 // this JNI file's `nativeDispatchToHostWorklets` constructs the
-// `StitcherFrameData` from raw bytes + pose + dims and forwards it.
-#include "stitcher_frame_data.hpp"
+// `CameraFrameData` from raw bytes + pose + dims and forwards it.
+#include "camera_frame_data.hpp"
 #include "stitcher_worklet_dispatch.hpp"
 #include "stitcher_worklet_registry.hpp"
 
@@ -76,7 +76,7 @@ Java_io_imagestitcher_rn_StitcherJsiInstallerModule_nativeInstall(
 // Owns a heap-allocated `std::vector<uint8_t>` of pre-copied NV21
 // bytes.  Constructed by `nativeDispatchToHostWorklets` after one
 // JNI byte-array copy from Kotlin; outlives the AR render thread
-// scope via `StitcherFrameData::pixelReader`'s `shared_ptr` —
+// scope via `CameraFrameData::pixelReader`'s `shared_ptr` —
 // dropped when the host object is invalidated.
 
 namespace {
@@ -117,6 +117,29 @@ Java_io_imagestitcher_rn_StitcherWorkletRuntime_nativeRegistryCount(
       retailens::StitcherWorkletRegistry::shared().count());
 }
 
+// ─── per-frame extraction-config gate ──────────────────────────────
+//
+// Returns the current `retailens::getExtractionConfig()` packed into a
+// `jint` bitmask so Kotlin can cheaply read the JS-driven
+// enableDepth/enableAnchors/enableMesh toggles once per frame and skip
+// the costly ARCore depth-acquire / anchor-collect / mesh-build work
+// when a host hasn't opted in.  Same atomic-snapshot read the JSI
+// `setExtractionConfig` host function writes.
+//
+//   bit0 (0x1) = depth
+//   bit1 (0x2) = anchors
+//   bit2 (0x4) = mesh
+extern "C" JNIEXPORT jint JNICALL
+Java_io_imagestitcher_rn_StitcherWorkletRuntime_nativeExtractionFlags(
+    JNIEnv* /*env*/, jobject /*thiz*/) {
+  const retailens::ExtractionConfig cfg = retailens::getExtractionConfig();
+  jint flags = 0;
+  if (cfg.depth) flags |= 0x1;
+  if (cfg.anchors) flags |= 0x2;
+  if (cfg.mesh) flags |= 0x4;
+  return flags;
+}
+
 // ─── v0.8.0 Phase 4b.iii — per-frame dispatch JNI binding ──────────
 //
 // Called from Kotlin's `StitcherWorkletRuntime.dispatchToHostWorklets`
@@ -143,7 +166,18 @@ Java_io_imagestitcher_rn_StitcherWorkletRuntime_nativeDispatchToHostWorklets(
     jdouble qx, jdouble qy, jdouble qz, jdouble qw,
     jdouble tx, jdouble ty, jdouble tz,
     jdouble timestampNs,
-    jstring trackingState) {
+    jstring trackingState,
+    jbyteArray depthBytes,
+    jint depthWidth, jint depthHeight,
+    jobjectArray anchorIds,
+    jobjectArray anchorTypes,
+    jobjectArray anchorTransforms,
+    jobjectArray anchorMeshVertices,
+    jobjectArray anchorMeshFaces,
+    jdouble fx, jdouble fy, jdouble cx, jdouble cy,
+    jint intrinsicsImageWidth, jint intrinsicsImageHeight,
+    jobjectArray anchorAlignments,
+    jobjectArray anchorExtents) {
   // Fast-path early-exit BEFORE the JNI byte-array copy.  Saves the
   // ~3MB memcpy + JSI host object alloc on every frame in the
   // common first-party-only case.
@@ -183,10 +217,10 @@ Java_io_imagestitcher_rn_StitcherWorkletRuntime_nativeDispatchToHostWorklets(
     }
   }
 
-  // Build StitcherFrameData.  Field semantics match the iOS
-  // `StitcherFrameHostObject::fromARFrame:pose:` factory; this is
+  // Build CameraFrameData.  Field semantics match the iOS
+  // `CameraFrameHostObject::fromARFrame:pose:` factory; this is
   // the Android equivalent path.
-  retailens::StitcherFrameData data;
+  retailens::CameraFrameData data;
   data.source = "ar";
   data.width = static_cast<int32_t>(width);
   data.height = static_cast<int32_t>(height);
@@ -214,6 +248,181 @@ Java_io_imagestitcher_rn_StitcherWorkletRuntime_nativeDispatchToHostWorklets(
   data.pixelReader =
       std::make_shared<AndroidNV21BufferReader>(std::move(bytes));
 
+  // ── AR depth (ARCore DEPTH16, "u16packed") ────────────────────────
+  //
+  // Kotlin hands us a dense, row-packed uint16-per-pixel byte array
+  // (depthWidth*depthHeight*2 bytes; low 13 bits = mm, high 3 bits =
+  // confidence 0..7) or null when depth is unavailable this frame.  We
+  // copy the bytes verbatim into `data.arDepth.depthBytes` with
+  // `format = "u16packed"` and leave `confidenceBytes` EMPTY — the
+  // shared JSI layer (`cpp/camera_frame_jsi.cpp`) unpacks mm->metres
+  // and confidence 0..7 -> 0..2 from the high bits.
+  if (depthBytes != nullptr && depthWidth > 0 && depthHeight > 0) {
+    const jsize depthLen = env->GetArrayLength(depthBytes);
+    if (depthLen > 0) {
+      retailens::ArDepth depth;
+      depth.width = static_cast<int32_t>(depthWidth);
+      depth.height = static_cast<int32_t>(depthHeight);
+      depth.format = "u16packed";
+      depth.depthBytes.resize(static_cast<std::size_t>(depthLen));
+      env->GetByteArrayRegion(
+          depthBytes, 0, depthLen,
+          reinterpret_cast<jbyte*>(depth.depthBytes.data()));
+      // confidenceBytes intentionally left empty (packed into the high
+      // 3 bits of each uint16 — unpacked JSI-side).
+      data.arDepth = std::move(depth);
+    }
+  }
+
+  // ── Per-frame camera intrinsics ───────────────────────────────────
+  //
+  // Kotlin passes camera.imageIntrinsics (fx,fy,cx,cy in pixels) + the
+  // capture resolution they're expressed at.  fx <= 0.0 is the "no
+  // intrinsics" sentinel (defensive — AR frames always carry valid
+  // intrinsics, but a degenerate session could yield 0).  The shared
+  // JSI layer exposes `intrinsics === undefined` when !hasIntrinsics.
+  if (fx > 0.0) {
+    data.hasIntrinsics = true;
+    data.fx = fx;
+    data.fy = fy;
+    data.cx = cx;
+    data.cy = cy;
+    data.intrinsicsImageWidth = static_cast<int32_t>(intrinsicsImageWidth);
+    data.intrinsicsImageHeight = static_cast<int32_t>(intrinsicsImageHeight);
+  }
+
+  // ── AR anchors ────────────────────────────────────────────────────
+  //
+  // Five parallel arrays from Kotlin: ids (String[]), types (String[]),
+  // transforms (double[16][]), and the per-anchor mesh byte arrays
+  // meshVertices (byte[][], Float32 xyz triplets) + meshFaces (byte[][],
+  // Uint32 triangle indices) — both NULL for non-mesh anchors.  Build one
+  // `retailens::ArAnchor` per entry; the transform is already ROW-MAJOR
+  // (anchor->world) — Kotlin transposed ARCore's column-major OpenGL
+  // matrix before marshaling (mesh anchors emit identity: the vertices
+  // are camera-local).  Empty arrays (the common case — no host opted
+  // into anchors/mesh) leave `data.arAnchors` empty, which the JSI layer
+  // surfaces as `[]` for source=="ar".
+  if (anchorIds != nullptr && anchorTypes != nullptr &&
+      anchorTransforms != nullptr) {
+    const jsize anchorCount = env->GetArrayLength(anchorIds);
+    data.arAnchors.reserve(static_cast<std::size_t>(anchorCount));
+    for (jsize i = 0; i < anchorCount; ++i) {
+      retailens::ArAnchor anchor;
+
+      auto idObj = reinterpret_cast<jstring>(
+          env->GetObjectArrayElement(anchorIds, i));
+      if (idObj != nullptr) {
+        const char* cs = env->GetStringUTFChars(idObj, nullptr);
+        if (cs != nullptr) {
+          anchor.id = cs;
+          env->ReleaseStringUTFChars(idObj, cs);
+        }
+        env->DeleteLocalRef(idObj);
+      }
+
+      auto typeObj = reinterpret_cast<jstring>(
+          env->GetObjectArrayElement(anchorTypes, i));
+      if (typeObj != nullptr) {
+        const char* cs = env->GetStringUTFChars(typeObj, nullptr);
+        if (cs != nullptr) {
+          anchor.type = cs;
+          env->ReleaseStringUTFChars(typeObj, cs);
+        }
+        env->DeleteLocalRef(typeObj);
+      }
+
+      auto transformObj = reinterpret_cast<jdoubleArray>(
+          env->GetObjectArrayElement(anchorTransforms, i));
+      if (transformObj != nullptr) {
+        const jsize n = env->GetArrayLength(transformObj);
+        jdouble* elems = env->GetDoubleArrayElements(transformObj, nullptr);
+        if (elems != nullptr) {
+          const jsize copyN = (n < 16) ? n : 16;
+          for (jsize j = 0; j < copyN; ++j) {
+            anchor.transform[static_cast<std::size_t>(j)] =
+                static_cast<double>(elems[j]);
+          }
+          env->ReleaseDoubleArrayElements(transformObj, elems, JNI_ABORT);
+        }
+        env->DeleteLocalRef(transformObj);
+      }
+
+      // ── per-anchor plane alignment + extent ─────────────────────────
+      //
+      // anchorAlignments[i] is "" for image/mesh anchors (→ JS
+      // `alignment === undefined`) or "horizontal"/"vertical" for plane
+      // anchors.  anchorExtents[i] is null for non-plane anchors or a
+      // double[2] = {extentX, extentZ} (metres) for planes.  Both arrays
+      // are parallel to anchorIds; guard for null (a caller passing the
+      // older arg shape) the same way the mesh arrays are guarded.  We do
+      // NOT set classification — Android has no plane semantics (iOS-only).
+      if (anchorAlignments != nullptr) {
+        auto alignObj = reinterpret_cast<jstring>(
+            env->GetObjectArrayElement(anchorAlignments, i));
+        if (alignObj != nullptr) {
+          const char* cs = env->GetStringUTFChars(alignObj, nullptr);
+          if (cs != nullptr) {
+            if (cs[0] != '\0') {
+              anchor.alignment = cs;
+            }
+            env->ReleaseStringUTFChars(alignObj, cs);
+          }
+          env->DeleteLocalRef(alignObj);
+        }
+      }
+      if (anchorExtents != nullptr) {
+        auto extObj = reinterpret_cast<jdoubleArray>(
+            env->GetObjectArrayElement(anchorExtents, i));
+        if (extObj != nullptr) {
+          if (env->GetArrayLength(extObj) >= 2) {
+            jdouble vals[2] = {0.0, 0.0};
+            env->GetDoubleArrayRegion(extObj, 0, 2, vals);
+            anchor.hasExtent = true;
+            anchor.extentX = static_cast<double>(vals[0]);
+            anchor.extentZ = static_cast<double>(vals[1]);
+          }
+          env->DeleteLocalRef(extObj);
+        }
+      }
+
+      // ── per-anchor mesh geometry (depth-derived; type=="mesh") ──────
+      //
+      // anchorMeshVertices[i] / anchorMeshFaces[i] are null for non-mesh
+      // anchors and a byte[] for a mesh anchor.  When BOTH are present we
+      // copy them verbatim into the ArAnchor's vectors and flag hasMesh —
+      // the JSI layer (`cpp/camera_frame_jsi.cpp`) emits them as
+      // ArrayBuffers (Float32 vertices / Uint32 faces) unchanged.
+      // meshClassifications stays empty (Android depth meshes carry no
+      // per-face semantics).
+      if (anchorMeshVertices != nullptr && anchorMeshFaces != nullptr) {
+        auto vertObj = reinterpret_cast<jbyteArray>(
+            env->GetObjectArrayElement(anchorMeshVertices, i));
+        auto faceObj = reinterpret_cast<jbyteArray>(
+            env->GetObjectArrayElement(anchorMeshFaces, i));
+        if (vertObj != nullptr && faceObj != nullptr) {
+          const jsize vLen = env->GetArrayLength(vertObj);
+          const jsize fLen = env->GetArrayLength(faceObj);
+          if (vLen > 0 && fLen > 0) {
+            anchor.meshVertices.resize(static_cast<std::size_t>(vLen));
+            env->GetByteArrayRegion(
+                vertObj, 0, vLen,
+                reinterpret_cast<jbyte*>(anchor.meshVertices.data()));
+            anchor.meshFaces.resize(static_cast<std::size_t>(fLen));
+            env->GetByteArrayRegion(
+                faceObj, 0, fLen,
+                reinterpret_cast<jbyte*>(anchor.meshFaces.data()));
+            anchor.hasMesh = true;
+          }
+        }
+        if (vertObj != nullptr) env->DeleteLocalRef(vertObj);
+        if (faceObj != nullptr) env->DeleteLocalRef(faceObj);
+      }
+
+      data.arAnchors.push_back(std::move(anchor));
+    }
+  }
+
   // Dispatch on worklets-core's default context.  That context is
   // initialised by JS' `Worklets.install()` (which runs at lib
   // bootstrap when worklets-core's module is imported); by the

package/android/src/main/java/io/imagestitcher/rn/ARFrameContext.kt

@@ -0,0 +1,89 @@
+// SPDX-License-Identifier: Apache-2.0
+package io.imagestitcher.rn
+
+/**
+ * 0.19.0 — zero-copy native view of one ARCore frame, handed to every
+ * registered [ARFramePlugin.process] (iOS twin: `RNISARFrameContext`).
+ *
+ * The SDK builds ONE of these per AR frame — only when [RNSARPluginRegistry]
+ * is non-empty — from the SAME `ARCore Frame` + pose the `onArFrame` meta
+ * path uses, then passes it to each plugin in turn.  Zero-plugin apps never
+ * pay for the build.
+ *
+ * ## Camera image
+ *
+ * ARCore hands the SDK a `YUV_420_888` camera image which is already packed
+ * into a contiguous JVM-side NV21 byte array ([nv21]) before the ARCore
+ * `Image` is closed (the SDK does this once per frame for its own stitch /
+ * worklet paths — we reuse it here, no extra acquire).  Layout:
+ *   - bytes `[0 .. width*height)`              = Y plane (luminance), dense,
+ *                                                row stride = [width]
+ *   - bytes `[width*height .. width*height*3/2)` = interleaved V-U chroma
+ * [yPlane] is a convenience read-only window onto just the Y plane.
+ *
+ * ## Lifetime — COPY BEFORE OFFLOADING
+ *
+ * [nv21] / [yPlane] / [depthBytes] are the SDK's own arrays, reused on the
+ * next frame.  They are valid ONLY for the duration of the synchronous
+ * [ARFramePlugin.process] call.  A plugin that hands bytes to another
+ * thread (async OCR, network upload, etc.) **MUST copy** them first
+ * (`bytes.copyOf()`), or it will read torn/overwritten data on the next AR
+ * frame.
+ *
+ * @property nv21            Full NV21 camera image (Y plane then interleaved VU).
+ * @property width           Camera image width (px).
+ * @property height          Camera image height (px).
+ * @property timestampNs      ARCore frame timestamp (nanoseconds).
+ * @property fx              Focal length x (px, at capture resolution).
+ * @property fy              Focal length y (px).
+ * @property cx              Principal point x (px).
+ * @property cy              Principal point y (px).
+ * @property imageWidth      Intrinsics reference image width (px).
+ * @property imageHeight     Intrinsics reference image height (px).
+ * @property poseRotation    Camera pose rotation quaternion `[x, y, z, w]`.
+ * @property poseTranslation Camera pose translation `[x, y, z]` (metres, world).
+ * @property trackingState   Contract enum string: "normal" | "limited" | "notAvailable".
+ * @property depthBytes      Row-packed DEPTH16 (uint16/px, w*h*2 bytes) or null
+ *                           (null unless `enableDepth` AND depth available this frame).
+ * @property depthWidth      Depth map width (px), 0 when [depthBytes] is null.
+ * @property depthHeight     Depth map height (px), 0 when [depthBytes] is null.
+ * @property anchors         Anchor descriptor maps already collected for the
+ *                           `onArFrame` event (empty unless `enableAnchors`).
+ *                           Each map: { id, type, transform[16 row-major],
+ *                           alignment?, extent? } — same shape the JS
+ *                           `ARAnchor` contract uses.
+ */
+class ARFrameContext(
+    @JvmField val nv21: ByteArray,
+    @JvmField val width: Int,
+    @JvmField val height: Int,
+    @JvmField val timestampNs: Double,
+    @JvmField val fx: Double,
+    @JvmField val fy: Double,
+    @JvmField val cx: Double,
+    @JvmField val cy: Double,
+    @JvmField val imageWidth: Int,
+    @JvmField val imageHeight: Int,
+    @JvmField val poseRotation: DoubleArray,
+    @JvmField val poseTranslation: DoubleArray,
+    @JvmField val trackingState: String,
+    @JvmField val depthBytes: ByteArray? = null,
+    @JvmField val depthWidth: Int = 0,
+    @JvmField val depthHeight: Int = 0,
+    @JvmField val anchors: List<Map<String, Any?>> = emptyList(),
+) {
+    /**
+     * Read-only window onto JUST the Y (luminance) plane of [nv21] — the
+     * first `width * height` bytes.  Cheap (no copy): a sliced, read-only
+     * [java.nio.ByteBuffer] over the same backing array.  Convenient for
+     * plugins that only need luma (brightness, simple CV gates).
+     *
+     * Like [nv21], valid only during [ARFramePlugin.process]; copy before
+     * offloading.
+     */
+    val yPlane: java.nio.ByteBuffer
+        get() = java.nio.ByteBuffer
+            .wrap(nv21, 0, width * height)
+            .slice()
+            .asReadOnlyBuffer()
+}

package/android/src/main/java/io/imagestitcher/rn/ARFramePlugin.kt

@@ -0,0 +1,57 @@
+// SPDX-License-Identifier: Apache-2.0
+package io.imagestitcher.rn
+
+import com.facebook.react.bridge.WritableMap
+
+/**
+ * 0.19.0 — Android AR frame-processor plugin SPI (Swift twin:
+ * `ios/Sources/RNImageStitcher/RNISARFramePlugin.swift`).
+ *
+ * Mirrors vision-camera's `FrameProcessorPlugin` registration ergonomics:
+ * the *host app* implements this interface, registers an instance with
+ * [RNSARPluginRegistry] at startup, and the SDK invokes [process] for every
+ * ARCore frame while the registry is non-empty.  The SDK ships ONLY this
+ * generic framework — no OCR or any other concrete plugin (the host writes
+ * those against this contract).
+ *
+ * ## Threading & lifetime
+ *
+ * [process] runs on the **AR (GL render) thread**, synchronously, once per
+ * ARCore frame.  The [ARFrameContext] handed in is a zero-copy view onto
+ * the live frame — its byte buffers (`yPlane` / `nv21` / depth `bytes`) are
+ * the SDK's own arrays and are reused on subsequent frames.  A plugin that
+ * offloads heavy work to another thread **MUST copy** any bytes it needs
+ * before returning from [process] (see [ARFrameContext]).
+ *
+ * ## Sync vs async results
+ *
+ *  - Return a non-null [WritableMap] for a *light, synchronous* result.  The
+ *    SDK folds it into the throttled `onArFrame` `ARFrameMeta` event under
+ *    `plugins[name]`, so it rides the existing channel for free.
+ *  - Return `null` and call [RNSARPluginRegistry.emit] later (from the
+ *    plugin's own queue) to deliver an *async* result over the dedicated
+ *    `RNImageStitcherARPluginResult` device event.
+ *
+ * Plugins are responsible for their own throttling / work-offloading — the
+ * SDK calls [process] on EVERY AR frame while the registry is non-empty.
+ */
+interface ARFramePlugin {
+    /**
+     * Stable, unique name for this plugin.  Used as the key under
+     * `ARFrameMeta.plugins` for sync results and as the `plugin` field of
+     * the `RNImageStitcherARPluginResult` event for async results.  The JS
+     * side keys off this string, so keep it stable across app launches.
+     */
+    fun name(): String
+
+    /**
+     * Process one ARCore frame.  Return a light synchronous result map
+     * (folded into the `onArFrame` event) or `null` (no sync result — emit
+     * later via [RNSARPluginRegistry.emit] if needed).
+     *
+     * Runs on the AR thread.  Do NOT block: self-throttle and offload heavy
+     * work.  Copy any [ARFrameContext] byte buffers you retain past the
+     * call.
+     */
+    fun process(context: ARFrameContext): WritableMap?
+}