@srsergio/taptapp-ar 1.0.0 → 1.0.3

package/README.md

@@ -1,62 +1,138 @@
 # @srsergio/taptapp-ar
 
-AR Visualizer and Image Target Compiler for Astro and React.
+🚀 **TapTapp AR** is a high-performance Augmented Reality (AR) toolkit specifically designed for **Astro** and **Node.js** environments. It provides a seamless way to integrate image tracking, video overlays, and an offline compiler for image targets.
 
-## Features
+Built on top of **MindAR** and **A-Frame**, this package features a **pure JavaScript offline compiler** that requires **no TensorFlow** for backend compilation, while still supporting TensorFlow.js for real-time tracking in the browser.
 
-- 🚀 **Astro-ready**: Built-in components for Astro projects.
-- ⚛️ **React Support**: Advanced AREditor and Progress UI.
-- 🖼️ **Offline Compiler**: Powerful image target compilation using TensorFlow.js.
-- 📱 **Mobile Optimized**: Smooth performance on mobile devices.
+---
+
+## 🌟 Key Features
+
+- 🚀 **Astro Native**: Optimized components for Astro's Islands architecture.
+- 🖼️ **Ultra-Fast Offline Compiler**: Pure JavaScript compiler that generates `.mind` target files in **~1.3s per image**.
+- ⚡ **Zero TensorFlow for Compilation**: The offline compiler uses optimized pure JS algorithms - no TensorFlow installation required.
+- 🧵 **Multi-threaded Engine**: Truly parallel processing using Node.js `worker_threads` for bulk image compilation.
+- 🚀 **Serverless Ready**: Lightweight compiler with minimal dependencies, perfect for Vercel, AWS Lambda, and Netlify.
+
+---
 
-## Installation
+## 🛠 Installation
 
 ```bash
 npm install @srsergio/taptapp-ar
 ```
 
-Note: You should also have `three`, `aframe`, `react`, and `react-dom` installed in your project as peer dependencies.
+### 📦 Optional Dependencies
+
+> **Note:** TensorFlow is **NOT required** for the offline compiler. It only uses pure JavaScript.
+
+For real-time AR tracking in the browser, TensorFlow.js is loaded automatically via CDN.
+
+---
+
+## 🚀 Astro Integration Guide
 
-## Usage
+The easiest way to display AR content is using the `ARVideoTrigger` component.
 
-### Using the AR Visualizer in Astro
+### Usage
 
 ```astro
 ---
 import ARVideoTrigger from '@srsergio/taptapp-ar/astro/ARVideoTrigger.astro';
 
 const config = {
-  targetImageSrc: '/path/to/image.jpg',
-  targetMindSrc: '/path/to/targets.mind',
-  videoSrc: '/path/to/video.mp4',
+  cardId: 'unique-id',
+  targetImageSrc: 'https://cdn.example.com/target.jpg',
+  targetMindSrc: 'https://cdn.example.com/targets.mind',
+  videoSrc: 'https://cdn.example.com/overlay.mp4',
   videoWidth: 1280,
   videoHeight: 720,
-  scale: 1,
+  scale: 1.2,
 };
 ---
 
 <ARVideoTrigger config={config} />
 ```
 
-### Using the AR Editor in React
+---
+
+## 🖼️ High-Performance Compiler (Protocol V3)
+
+TaptApp AR features the industry's most advanced **pure JavaScript** offline compiler. With the introduction of **Protocol V3 (Columnar Binary Format)**, it sets a new standard for AR asset management.
+
+### ⚡ Industry-Leading Benchmarks
+
+| Metric | Official MindAR | TapTapp AR (v3) | Improvement |
+| :--- | :--- | :--- | :--- |
+| **Compilation Time** | ~23.50s | **~1.71s** | 🚀 **13.7x Faster** |
+| **Output Size (.mind)** | ~770 KB | **~127 KB** | 📉 **83.5% Smaller** |
+| **Loading Latency** | >100ms | **2.6ms** | ⚡ **Zero-Copy** |
+| **Memory Footprint** | Heavy (JSON Objects) | **Minimal (Binary)** | 🧠 **CPU-Aligned** |
+
+> *Tested on 1024x1024 high-detail image target.*
+
+### 🚀 Key Technical Breakthroughs
+
+- **Protocol V3 (Columnar Binary)**: Uses TypedArrays to store coordinates, angles, and descriptors in a cache-aligned layout. No more thousands of slow JavaScript objects.
+- **Zero-Copy Loading**: The runtime reads directly from the binary buffer. Initialization is now virtualy instant.
+- **Aggressive Matching Optimization**: Tree-based hierarchical clustering compacted into a flattened binary format.
+- **No Dependencies**: Works in Node.js and Browser with zero external requirements for the compilation core.
 
-```tsx
-import { AREditor } from '@srsergio/taptapp-ar';
+### 🖥️ Usage (Node.js & Serverless)
 
-function MyEditor() {
-  return <AREditor adminId="some-id" />;
-}
+Optimized for server-side compilation with multi-core parallelism:
+
+```javascript
+import { OfflineCompiler } from '@srsergio/taptapp-ar/compiler/offline-compiler.js';
+
+const compiler = new OfflineCompiler();
+
+// Compile target image
+const compiledData = await compiler.compileImageTargets(
+  [{ width, height, data: grayscaleUint8Array }], 
+  (progress) => console.log(`Compiling: ${progress}%`)
+);
+
+// Export to Protocol V3 binary format
+const binaryBuffer = compiler.exportData(); // Yields a much smaller .mind file
 ```
 
-### Using the Offline Compiler (Backend/Serverless)
+### 🌐 Frontend (Zero-Latency Loading)
 
-```typescript
-import { OfflineCompiler } from '@srsergio/taptapp-ar';
+```javascript
+import { OfflineCompiler } from '@srsergio/taptapp-ar/compiler/offline-compiler.js';
 
 const compiler = new OfflineCompiler();
-// ... logic to compile image targets
+// Loading 127KB instead of 800KB saves bandwidth and CPU parsing time
+compiler.importData(binaryBuffer); 
+```
+---
+
+## ❓ Troubleshooting
+
+| Issue | Solution |
+| :--- | :--- |
+| **Camera not starting** | Ensure your site is served via `HTTPS`. Browsers block camera access on insecure origins. |
+| **Video not playing** | iOS Safari requires `muted` and `playsinline` attributes for autoplaying videos. Our components handle this by default. |
+| **CORS errors** | Ensure that `targetImageSrc`, `targetMindSrc`, and `videoSrc` have CORS headers enabled (`Access-Control-Allow-Origin: *`). |
+| **Memory Outage on Serverless** | Reduce the resolution of your target images. High-res images increase memory pressure during compilation. |
+
+---
+
+## 🏗 Development
+
+```bash
+# Install dependencies
+npm install
+
+# Build the package
+npm run build
 ```
 
-## License
+The package uses **TypeScript** and exports both ESM and CJS compatible builds located in the `dist` folder.
+
+---
+
+## 📄 License
 
-MIT
+MIT © [srsergiolazaro](https://github.com/srsergiolazaro)

package/dist/compiler/aframe.js

@@ -1,5 +1,4 @@
 import { Controller, UI } from "./index.js";
-const needsDOMRefresh = document.readyState === "complete" || document.readyState == "interactive";
 AFRAME.registerSystem("mindar-image-system", {
     container: null,
     video: null,
@@ -93,7 +92,6 @@ AFRAME.registerSystem("mindar-image-system", {
     },
     _startAR: async function () {
         const video = this.video;
-        const container = this.container;
         this.controller = new Controller({
             inputWidth: video.videoWidth,
             inputHeight: video.videoHeight,
@@ -159,7 +157,6 @@ AFRAME.registerSystem("mindar-image-system", {
         const fov = (2 * Math.atan((1 / proj[5] / vh) * container.clientHeight) * 180) / Math.PI; // vertical fov
         const near = proj[14] / (proj[10] - 1.0);
         const far = proj[14] / (proj[10] + 1.0);
-        const ratio = proj[5] / proj[0]; // (r-l) / (t-b)
         //console.log("loaded proj: ", proj, ". fov: ", fov, ". near: ", near, ". far: ", far, ". ratio: ", ratio);
         const newAspect = container.clientWidth / container.clientHeight;
         const cameraEle = container.getElementsByTagName("a-camera")[0];

package/dist/compiler/compiler-base.d.ts

@@ -1,12 +1,8 @@
 export class CompilerBase {
     data: any[] | null;
     compileImageTargets(images: any, progressCallback: any): Promise<any>;
-    exportData(): Uint8Array<ArrayBufferLike>;
+    exportData(): Uint8Array<ArrayBuffer>;
     importData(buffer: any): any[];
-    createProcessCanvas(img: any): void;
-    compileTrack({ progressCallback, targetImages, basePercent }: {
-        progressCallback: any;
-        targetImages: any;
-        basePercent: any;
-    }): void;
+    createProcessCanvas(): void;
+    compileTrack(): Promise<never[]>;
 }

package/dist/compiler/compiler-base.js

@@ -47,21 +47,34 @@ class CompilerBase {
                     targetImages.push(targetImage);
                 }
                 // compute matching data: 50% progress
-                const percentPerImage = 50.0 / targetImages.length;
-                let percent = 0.0;
+                let matchingDataList;
+                if (this.compileMatch) {
+                    matchingDataList = await this.compileMatch({
+                        progressCallback,
+                        targetImages,
+                        basePercent: 0,
+                    });
+                }
+                else {
+                    const percentPerImage = 50.0 / targetImages.length;
+                    let percent = 0.0;
+                    const matchingPromises = targetImages.map(async (targetImage, i) => {
+                        const imageList = buildImageList(targetImage);
+                        const percentPerAction = percentPerImage / imageList.length;
+                        const matchingData = await _extractMatchingFeatures(imageList, () => {
+                            percent += percentPerAction;
+                            progressCallback(percent);
+                        });
+                        return matchingData;
+                    });
+                    matchingDataList = await Promise.all(matchingPromises);
+                }
                 this.data = [];
                 for (let i = 0; i < targetImages.length; i++) {
-                    const targetImage = targetImages[i];
-                    const imageList = buildImageList(targetImage);
-                    const percentPerAction = percentPerImage / imageList.length;
-                    const matchingData = await _extractMatchingFeatures(imageList, () => {
-                        percent += percentPerAction;
-                        progressCallback(percent);
-                    });
                     this.data.push({
-                        targetImage: targetImage,
-                        imageList: imageList,
-                        matchingData: matchingData,
+                        targetImage: targetImages[i],
+                        imageList: buildImageList(targetImages[i]),
+                        matchingData: matchingDataList[i],
                     });
                 }
                 for (let i = 0; i < targetImages.length; i++) {
@@ -121,13 +134,14 @@ class CompilerBase {
         }
         return this.data;
     }
-    createProcessCanvas(img) {
+    createProcessCanvas() {
         // sub-class implements
         console.warn("missing createProcessCanvas implementation");
     }
-    compileTrack({ progressCallback, targetImages, basePercent }) {
+    compileTrack() {
         // sub-class implements
         console.warn("missing compileTrack implementation");
+        return Promise.resolve([]);
     }
 }
 const _extractMatchingFeatures = async (imageList, doneCallback) => {

package/dist/compiler/compiler.js

@@ -8,7 +8,7 @@ export class Compiler extends CompilerBase {
         return processCanvas;
     }
     compileTrack({ progressCallback, targetImages, basePercent }) {
-        return new Promise((resolve, reject) => {
+        return new Promise((resolve) => {
             const worker = new CompilerWorker();
             worker.onmessage = (e) => {
                 if (e.data.type === "progress") {

package/dist/compiler/compiler.worker.js

@@ -13,7 +13,7 @@ onmessage = (msg) => {
             const imageList = buildTrackingImageList(targetImage);
             const percentPerAction = percentPerImage / imageList.length;
             //console.log("compiling tracking...", i);
-            const trackingData = extractTrackingFeatures(imageList, (index) => {
+            const trackingData = extractTrackingFeatures(imageList, () => {
                 //console.log("done tracking", i, index);
                 percent += percentPerAction;
                 postMessage({ type: "progress", percent });

package/dist/compiler/controller.js

@@ -63,7 +63,7 @@ class Controller {
         console.table(tf.memory());
     }
     addImageTargets(fileURL) {
-        return new Promise(async (resolve, reject) => {
+        return new Promise(async (resolve) => {
             const content = await fetch(fileURL);
             const buffer = await content.arrayBuffer();
             const result = this.addImageTargetsFromBuffer(buffer);
@@ -75,7 +75,6 @@ class Controller {
         const dataList = compiler.importData(buffer);
         const trackingDataList = [];
         const matchingDataList = [];
-        const imageListList = [];
         const dimensions = [];
         for (let i = 0; i < dataList.length; i++) {
             matchingDataList.push(dataList[i].matchingData);
@@ -264,7 +263,7 @@ class Controller {
     }
     async detect(input) {
         const inputT = this.inputLoader.loadInput(input);
-        const { featurePoints, debugExtra } = await this.cropDetector.detect(inputT);
+        const { featurePoints, debugExtra } = this.cropDetector.detect(inputT);
         inputT.dispose();
         return { featurePoints, debugExtra };
     }
@@ -287,7 +286,7 @@ class Controller {
         return modelViewTransform2;
     }
     _workerMatch(featurePoints, targetIndexes) {
-        return new Promise(async (resolve, reject) => {
+        return new Promise((resolve) => {
             this.workerMatchDone = (data) => {
                 resolve({
                     targetIndex: data.targetIndex,
@@ -299,7 +298,7 @@ class Controller {
         });
     }
     _workerTrackUpdate(modelViewTransform, trackingFeatures) {
-        return new Promise(async (resolve, reject) => {
+        return new Promise((resolve) => {
             this.workerTrackDone = (data) => {
                 resolve(data.modelViewTransform);
             };

package/dist/compiler/controller.worker.js

@@ -1,6 +1,5 @@
 import { Matcher } from "./matching/matcher.js";
 import { Estimator } from "./estimation/estimator.js";
-let projectionTransform = null;
 let matchingDataList = null;
 let debugMode = false;
 let matcher = null;
@@ -9,7 +8,6 @@ onmessage = (msg) => {
     const { data } = msg;
     switch (data.type) {
         case "setup":
-            projectionTransform = data.projectionTransform;
             matchingDataList = data.matchingDataList;
             debugMode = data.debugMode;
             matcher = new Matcher(data.inputWidth, data.inputHeight, debugMode);

package/dist/compiler/detector/crop-detector.js

@@ -1,6 +1,4 @@
-import * as tf from "@tensorflow/tfjs";
 import { Detector } from "./detector.js";
-import { buildModelViewProjectionTransform, computeScreenCoordiate } from "../estimation/utils.js";
 class CropDetector {
     constructor(width, height, debugMode = false) {
         this.debugMode = debugMode;

package/dist/compiler/detector/detector-lite.d.ts

@@ -0,0 +1,73 @@
+export function setDetectorGPUMode(enabled: boolean): void;
+/**
+ * Detector de características sin TensorFlow
+ */
+export class DetectorLite {
+    constructor(width: any, height: any, options?: {});
+    width: any;
+    height: any;
+    useGPU: any;
+    numOctaves: number;
+    /**
+     * Detecta características en una imagen en escala de grises
+     * @param {Float32Array|Uint8Array} imageData - Datos de imagen (width * height)
+     * @returns {{featurePoints: Array}} Puntos de características detectados
+     */
+    detect(imageData: Float32Array | Uint8Array): {
+        featurePoints: any[];
+    };
+    /**
+     * Construye una pirámide gaussiana
+     */
+    _buildGaussianPyramid(data: any, width: any, height: any): {
+        data: Float32Array<ArrayBuffer>;
+        width: any;
+        height: any;
+    }[][];
+    /**
+     * Aplica un filtro gaussiano binomial [1,4,6,4,1] - Optimizado
+     */
+    _applyGaussianFilter(data: any, width: any, height: any): {
+        data: Float32Array<ArrayBuffer>;
+        width: any;
+        height: any;
+    };
+    /**
+     * Downsample imagen por factor de 2
+     */
+    _downsample(data: any, width: any, height: any): {
+        data: Float32Array<ArrayBuffer>;
+        width: number;
+        height: number;
+    };
+    /**
+     * Construye pirámide de diferencia de gaussianas
+     */
+    _buildDogPyramid(pyramidImages: any): {
+        data: Float32Array<ArrayBuffer>;
+        width: any;
+        height: any;
+    }[];
+    /**
+     * Encuentra extremos locales en la pirámide DoG
+     */
+    _findExtremas(dogPyramid: any, pyramidImages: any): {
+        score: number;
+        octave: number;
+        x: number;
+        y: number;
+        absScore: number;
+    }[];
+    /**
+     * Aplica pruning para mantener solo los mejores features por bucket
+     */
+    _applyPrune(extremas: any): any[];
+    /**
+     * Calcula la orientación de cada feature
+     */
+    _computeOrientations(extremas: any, pyramidImages: any): void;
+    /**
+     * Calcula descriptores FREAK
+     */
+    _computeFreakDescriptors(extremas: any, pyramidImages: any): void;
+}