@srsergio/taptapp-ar 1.0.2 → 1.0.4

package/README.md

@@ -2,16 +2,17 @@
 
 🚀 **TapTapp AR** is a high-performance Augmented Reality (AR) toolkit specifically designed for **Astro** and **Node.js** environments. It provides a seamless way to integrate image tracking, video overlays, and an offline compiler for image targets.
 
-Built on top of **MindAR**, **A-Frame**, and **TensorFlow.js**, this package is optimized for both frontend visualization in Astro and backend/serverless image target compilation.
+Built on top of **MindAR** and **A-Frame**, this package features a **pure JavaScript offline compiler** that requires **no TensorFlow** for backend compilation, while still supporting TensorFlow.js for real-time tracking in the browser.
 
 ---
 
 ## 🌟 Key Features
 
 - 🚀 **Astro Native**: Optimized components for Astro's Islands architecture.
-- 🖼️ **Offline Compiler**: A powerful server-side compiler that generates `.mind` target files without a browser.
-- ⚡ **Optimized Performance**: Pre-warmed TensorFlow backends and adaptive memory management for serverless environments (Vercel, AWS Lambda).
-- 📱 **Mobile First**: Designed for smooth performance on iOS (Safari) and Android.
+- 🖼️ **Ultra-Fast Offline Compiler**: Pure JavaScript compiler that generates `.mind` target files in **~1.3s per image**.
+- ⚡ **Zero TensorFlow for Compilation**: The offline compiler uses optimized pure JS algorithms - no TensorFlow installation required.
+- 🧵 **Multi-threaded Engine**: Truly parallel processing using Node.js `worker_threads` for bulk image compilation.
+- 🚀 **Serverless Ready**: Lightweight compiler with minimal dependencies, perfect for Vercel, AWS Lambda, and Netlify.
 
 ---
 
@@ -21,15 +22,11 @@ Built on top of **MindAR**, **A-Frame**, and **TensorFlow.js**, this package is
 npm install @srsergio/taptapp-ar
 ```
 
-### 📦 Peer Dependencies
+### 📦 Optional Dependencies
 
-Make sure you have the following packages installed in your host project:
+> **Note:** TensorFlow is **NOT required** for the offline compiler. It only uses pure JavaScript.
 
-```bash
-npm install three aframe astro
-```
-
-Note: If you are using the `OfflineCompiler` in a Node.js environment, ensure you have the necessary TensorFlow.js backends installed.
+For real-time AR tracking in the browser, TensorFlow.js is loaded automatically via CDN.
 
 ---
 
@@ -57,53 +54,58 @@ const config = {
 <ARVideoTrigger config={config} />
 ```
 
-### `ARVideoTrigger` Props (Config)
+---
 
-| Prop | Type | Description |
-| :--- | :--- | :--- |
-| `cardId` | `string` | Unique identifier for tracking/session. |
-| `targetImageSrc` | `string` | URL of the image being tracked. |
-| `targetMindSrc` | `string` | URL of the compiled `.mind` target file. |
-| `videoSrc` | `string` | URL of the video to overlay on the target. |
-| `videoWidth` | `number` | Original width of the video. |
-| `videoHeight` | `number` | Original height of the video. |
-| `scale` | `number` | Scaling factor for the video overlay (Default: `1`). |
+## 🖼️ High-Performance Compiler (Protocol V3)
 
----
+TaptApp AR features the industry's most advanced **pure JavaScript** offline compiler. With the introduction of **Protocol V3 (Columnar Binary Format)**, it sets a new standard for AR asset management.
+
+### ⚡ Industry-Leading Benchmarks
 
-## 🖼 Offline Compiler Guide
+| Metric | Official MindAR | TapTapp AR (v3) | Improvement |
+| :--- | :--- | :--- | :--- |
+| **Compilation Time** | ~23.50s | **~1.71s** | 🚀 **13.7x Faster** |
+| **Output Size (.mind)** | ~770 KB | **~127 KB** | 📉 **83.5% Smaller** |
+| **Loading Latency** | >100ms | **2.6ms** | ⚡ **Zero-Copy** |
+| **Memory Footprint** | Heavy (JSON Objects) | **Minimal (Binary)** | 🧠 **CPU-Aligned** |
 
-The `OfflineCompiler` allows you to compile image targets on the backend. This is the heart of the TapTapp asset pipeline.
+> *Tested on 1024x1024 high-detail image target.*
 
-### Why use the Offline Compiler?
-Standard MindAR tools require a browser canvas to compile images. This compiler uses **TensorFlow.js** backends (CPU/WebGL/Node) to perform the computation as a background task.
+### 🚀 Key Technical Breakthroughs
 
-### Basic Usage
+- **Protocol V3 (Columnar Binary)**: Uses TypedArrays to store coordinates, angles, and descriptors in a cache-aligned layout. No more thousands of slow JavaScript objects.
+- **Zero-Copy Loading**: The runtime reads directly from the binary buffer. Initialization is now virtualy instant.
+- **Aggressive Matching Optimization**: Tree-based hierarchical clustering compacted into a flattened binary format.
+- **No Dependencies**: Works in Node.js and Browser with zero external requirements for the compilation core.
 
-```typescript
-import { OfflineCompiler } from '@srsergio/taptapp-ar';
+### 🖥️ Usage (Node.js & Serverless)
+
+Optimized for server-side compilation with multi-core parallelism:
+
+```javascript
+import { OfflineCompiler } from '@srsergio/taptapp-ar/compiler/offline-compiler.js';
 
 const compiler = new OfflineCompiler();
 
-async function compile(imageBuffer: Buffer) {
-  // targetImages is an array of images to compile into the same .mind file
-  const result = await compiler.compileTrack({
-    targetImages: [imageBuffer],
-    progressCallback: (progress) => console.log(`Compiling: ${progress}%`),
-    basePercent: 0
-  });
-  
-  // result is the compiled target data
-  return result;
-}
+// Compile target image
+const compiledData = await compiler.compileImageTargets(
+  [{ width, height, data: grayscaleUint8Array }], 
+  (progress) => console.log(`Compiling: ${progress}%`)
+);
+
+// Export to Protocol V3 binary format
+const binaryBuffer = compiler.exportData(); // Yields a much smaller .mind file
 ```
 
-### ⚡ Serverless Optimization
-The compiler is optimized for environments like Vercel Functions:
-- **Early Initialization**: TensorFlow is pre-warmed on module import.
-- **Memory Management**: Aggressive garbage collection (`tf.dispose()`) and tensor cleanup.
-- **Batch Processing**: Automatically splits work to avoid memory spikes.
+### 🌐 Frontend (Zero-Latency Loading)
 
+```javascript
+import { OfflineCompiler } from '@srsergio/taptapp-ar/compiler/offline-compiler.js';
+
+const compiler = new OfflineCompiler();
+// Loading 127KB instead of 800KB saves bandwidth and CPU parsing time
+compiler.importData(binaryBuffer); 
+```
 ---
 
 ## ❓ Troubleshooting

package/dist/compiler/aframe.js

@@ -1,5 +1,4 @@
 import { Controller, UI } from "./index.js";
-const needsDOMRefresh = document.readyState === "complete" || document.readyState == "interactive";
 AFRAME.registerSystem("mindar-image-system", {
     container: null,
     video: null,
@@ -93,7 +92,6 @@ AFRAME.registerSystem("mindar-image-system", {
     },
     _startAR: async function () {
         const video = this.video;
-        const container = this.container;
         this.controller = new Controller({
             inputWidth: video.videoWidth,
             inputHeight: video.videoHeight,
@@ -159,7 +157,6 @@ AFRAME.registerSystem("mindar-image-system", {
         const fov = (2 * Math.atan((1 / proj[5] / vh) * container.clientHeight) * 180) / Math.PI; // vertical fov
         const near = proj[14] / (proj[10] - 1.0);
         const far = proj[14] / (proj[10] + 1.0);
-        const ratio = proj[5] / proj[0]; // (r-l) / (t-b)
         //console.log("loaded proj: ", proj, ". fov: ", fov, ". near: ", near, ". far: ", far, ". ratio: ", ratio);
         const newAspect = container.clientWidth / container.clientHeight;
         const cameraEle = container.getElementsByTagName("a-camera")[0];

package/dist/compiler/compiler-base.d.ts

@@ -1,12 +1,8 @@
 export class CompilerBase {
     data: any[] | null;
     compileImageTargets(images: any, progressCallback: any): Promise<any>;
-    exportData(): Uint8Array<ArrayBufferLike>;
+    exportData(): any;
     importData(buffer: any): any[];
-    createProcessCanvas(img: any): void;
-    compileTrack({ progressCallback, targetImages, basePercent }: {
-        progressCallback: any;
-        targetImages: any;
-        basePercent: any;
-    }): void;
+    createProcessCanvas(): void;
+    compileTrack(): Promise<never[]>;
 }

package/dist/compiler/compiler-base.js

@@ -47,21 +47,34 @@ class CompilerBase {
                     targetImages.push(targetImage);
                 }
                 // compute matching data: 50% progress
-                const percentPerImage = 50.0 / targetImages.length;
-                let percent = 0.0;
+                let matchingDataList;
+                if (this.compileMatch) {
+                    matchingDataList = await this.compileMatch({
+                        progressCallback,
+                        targetImages,
+                        basePercent: 0,
+                    });
+                }
+                else {
+                    const percentPerImage = 50.0 / targetImages.length;
+                    let percent = 0.0;
+                    const matchingPromises = targetImages.map(async (targetImage, i) => {
+                        const imageList = buildImageList(targetImage);
+                        const percentPerAction = percentPerImage / imageList.length;
+                        const matchingData = await _extractMatchingFeatures(imageList, () => {
+                            percent += percentPerAction;
+                            progressCallback(percent);
+                        });
+                        return matchingData;
+                    });
+                    matchingDataList = await Promise.all(matchingPromises);
+                }
                 this.data = [];
                 for (let i = 0; i < targetImages.length; i++) {
-                    const targetImage = targetImages[i];
-                    const imageList = buildImageList(targetImage);
-                    const percentPerAction = percentPerImage / imageList.length;
-                    const matchingData = await _extractMatchingFeatures(imageList, () => {
-                        percent += percentPerAction;
-                        progressCallback(percent);
-                    });
                     this.data.push({
-                        targetImage: targetImage,
-                        imageList: imageList,
-                        matchingData: matchingData,
+                        targetImage: targetImages[i],
+                        imageList: buildImageList(targetImages[i]),
+                        matchingData: matchingDataList[i],
                     });
                 }
                 for (let i = 0; i < targetImages.length; i++) {
@@ -121,13 +134,14 @@ class CompilerBase {
         }
         return this.data;
     }
-    createProcessCanvas(img) {
+    createProcessCanvas() {
         // sub-class implements
         console.warn("missing createProcessCanvas implementation");
     }
-    compileTrack({ progressCallback, targetImages, basePercent }) {
+    compileTrack() {
         // sub-class implements
         console.warn("missing compileTrack implementation");
+        return Promise.resolve([]);
     }
 }
 const _extractMatchingFeatures = async (imageList, doneCallback) => {

package/dist/compiler/compiler.js

@@ -8,7 +8,7 @@ export class Compiler extends CompilerBase {
         return processCanvas;
     }
     compileTrack({ progressCallback, targetImages, basePercent }) {
-        return new Promise((resolve, reject) => {
+        return new Promise((resolve) => {
             const worker = new CompilerWorker();
             worker.onmessage = (e) => {
                 if (e.data.type === "progress") {

package/dist/compiler/compiler.worker.js

@@ -13,7 +13,7 @@ onmessage = (msg) => {
             const imageList = buildTrackingImageList(targetImage);
             const percentPerAction = percentPerImage / imageList.length;
             //console.log("compiling tracking...", i);
-            const trackingData = extractTrackingFeatures(imageList, (index) => {
+            const trackingData = extractTrackingFeatures(imageList, () => {
                 //console.log("done tracking", i, index);
                 percent += percentPerAction;
                 postMessage({ type: "progress", percent });

package/dist/compiler/controller.d.ts

@@ -60,12 +60,12 @@ export class Controller {
             descriptors: any[];
         }[];
         debugExtra: {
-            pyramidImages: (number | number[] | number[][] | number[][][] | number[][][][] | number[][][][][] | number[][][][][][])[][];
-            dogPyramidImages: (number | number[] | number[][] | number[][][] | number[][][][] | number[][][][][] | number[][][][][][] | null)[];
-            extremasResults: (number | number[] | number[][] | number[][][] | number[][][][] | number[][][][][] | number[][][][][][])[];
+            pyramidImages: any[][];
+            dogPyramidImages: any[];
+            extremasResults: any[];
             extremaAngles: any;
             prunedExtremas: number[][];
-            localizedExtremas: number | number[] | number[][] | number[][][] | number[][][][] | number[][][][][] | number[][][][][][];
+            localizedExtremas: any;
         } | null;
     }>;
     match(featurePoints: any, targetIndex: any): Promise<{

package/dist/compiler/controller.js

@@ -63,7 +63,7 @@ class Controller {
         console.table(tf.memory());
     }
     addImageTargets(fileURL) {
-        return new Promise(async (resolve, reject) => {
+        return new Promise(async (resolve) => {
             const content = await fetch(fileURL);
             const buffer = await content.arrayBuffer();
             const result = this.addImageTargetsFromBuffer(buffer);
@@ -75,7 +75,6 @@ class Controller {
         const dataList = compiler.importData(buffer);
         const trackingDataList = [];
         const matchingDataList = [];
-        const imageListList = [];
         const dimensions = [];
         for (let i = 0; i < dataList.length; i++) {
             matchingDataList.push(dataList[i].matchingData);
@@ -264,7 +263,7 @@ class Controller {
     }
     async detect(input) {
         const inputT = this.inputLoader.loadInput(input);
-        const { featurePoints, debugExtra } = await this.cropDetector.detect(inputT);
+        const { featurePoints, debugExtra } = this.cropDetector.detect(inputT);
         inputT.dispose();
         return { featurePoints, debugExtra };
     }
@@ -287,7 +286,7 @@ class Controller {
         return modelViewTransform2;
     }
     _workerMatch(featurePoints, targetIndexes) {
-        return new Promise(async (resolve, reject) => {
+        return new Promise((resolve) => {
             this.workerMatchDone = (data) => {
                 resolve({
                     targetIndex: data.targetIndex,
@@ -299,7 +298,7 @@ class Controller {
         });
     }
     _workerTrackUpdate(modelViewTransform, trackingFeatures) {
-        return new Promise(async (resolve, reject) => {
+        return new Promise((resolve) => {
             this.workerTrackDone = (data) => {
                 resolve(data.modelViewTransform);
             };

package/dist/compiler/controller.worker.js

@@ -1,6 +1,5 @@
 import { Matcher } from "./matching/matcher.js";
 import { Estimator } from "./estimation/estimator.js";
-let projectionTransform = null;
 let matchingDataList = null;
 let debugMode = false;
 let matcher = null;
@@ -9,7 +8,6 @@ onmessage = (msg) => {
     const { data } = msg;
     switch (data.type) {
         case "setup":
-            projectionTransform = data.projectionTransform;
             matchingDataList = data.matchingDataList;
             debugMode = data.debugMode;
             matcher = new Matcher(data.inputWidth, data.inputHeight, debugMode);

package/dist/compiler/detector/crop-detector.d.ts

@@ -17,12 +17,12 @@ export class CropDetector {
             descriptors: any[];
         }[];
         debugExtra: {
-            pyramidImages: (number | number[] | number[][] | number[][][] | number[][][][] | number[][][][][] | number[][][][][][])[][];
-            dogPyramidImages: (number | number[] | number[][] | number[][][] | number[][][][] | number[][][][][] | number[][][][][][] | null)[];
-            extremasResults: (number | number[] | number[][] | number[][][] | number[][][][] | number[][][][][] | number[][][][][][])[];
+            pyramidImages: any[][];
+            dogPyramidImages: any[];
+            extremasResults: any[];
             extremaAngles: any;
             prunedExtremas: number[][];
-            localizedExtremas: number | number[] | number[][] | number[][][] | number[][][][] | number[][][][][] | number[][][][][][];
+            localizedExtremas: any;
         } | null;
     };
     detectMoving(inputImageT: any): {
@@ -35,12 +35,12 @@ export class CropDetector {
             descriptors: any[];
         }[];
         debugExtra: {
-            pyramidImages: (number | number[] | number[][] | number[][][] | number[][][][] | number[][][][][] | number[][][][][][])[][];
-            dogPyramidImages: (number | number[] | number[][] | number[][][] | number[][][][] | number[][][][][] | number[][][][][][] | null)[];
-            extremasResults: (number | number[] | number[][] | number[][][] | number[][][][] | number[][][][][] | number[][][][][][])[];
+            pyramidImages: any[][];
+            dogPyramidImages: any[];
+            extremasResults: any[];
             extremaAngles: any;
             prunedExtremas: number[][];
-            localizedExtremas: number | number[] | number[][] | number[][][] | number[][][][] | number[][][][][] | number[][][][][][];
+            localizedExtremas: any;
         } | null;
     };
     _detect(inputImageT: any, startX: any, startY: any): {
@@ -53,12 +53,12 @@ export class CropDetector {
             descriptors: any[];
         }[];
         debugExtra: {
-            pyramidImages: (number | number[] | number[][] | number[][][] | number[][][][] | number[][][][][] | number[][][][][][])[][];
-            dogPyramidImages: (number | number[] | number[][] | number[][][] | number[][][][] | number[][][][][] | number[][][][][][] | null)[];
-            extremasResults: (number | number[] | number[][] | number[][][] | number[][][][] | number[][][][][] | number[][][][][][])[];
+            pyramidImages: any[][];
+            dogPyramidImages: any[];
+            extremasResults: any[];
             extremaAngles: any;
             prunedExtremas: number[][];
-            localizedExtremas: number | number[] | number[][] | number[][][] | number[][][][] | number[][][][][] | number[][][][][][];
+            localizedExtremas: any;
         } | null;
     };
 }

package/dist/compiler/detector/crop-detector.js

@@ -1,6 +1,4 @@
-import * as tf from "@tensorflow/tfjs";
 import { Detector } from "./detector.js";
-import { buildModelViewProjectionTransform, computeScreenCoordiate } from "../estimation/utils.js";
 class CropDetector {
     constructor(width, height, debugMode = false) {
         this.debugMode = debugMode;

package/dist/compiler/detector/detector-lite.d.ts

@@ -0,0 +1,73 @@
+export function setDetectorGPUMode(enabled: boolean): void;
+/**
+ * Detector de características sin TensorFlow
+ */
+export class DetectorLite {
+    constructor(width: any, height: any, options?: {});
+    width: any;
+    height: any;
+    useGPU: any;
+    numOctaves: number;
+    /**
+     * Detecta características en una imagen en escala de grises
+     * @param {Float32Array|Uint8Array} imageData - Datos de imagen (width * height)
+     * @returns {{featurePoints: Array}} Puntos de características detectados
+     */
+    detect(imageData: Float32Array | Uint8Array): {
+        featurePoints: any[];
+    };
+    /**
+     * Construye una pirámide gaussiana
+     */
+    _buildGaussianPyramid(data: any, width: any, height: any): {
+        data: Float32Array<ArrayBuffer>;
+        width: any;
+        height: any;
+    }[][];
+    /**
+     * Aplica un filtro gaussiano binomial [1,4,6,4,1] - Optimizado
+     */
+    _applyGaussianFilter(data: any, width: any, height: any): {
+        data: Float32Array<ArrayBuffer>;
+        width: any;
+        height: any;
+    };
+    /**
+     * Downsample imagen por factor de 2
+     */
+    _downsample(data: any, width: any, height: any): {
+        data: Float32Array<ArrayBuffer>;
+        width: number;
+        height: number;
+    };
+    /**
+     * Construye pirámide de diferencia de gaussianas
+     */
+    _buildDogPyramid(pyramidImages: any): {
+        data: Float32Array<ArrayBuffer>;
+        width: any;
+        height: any;
+    }[];
+    /**
+     * Encuentra extremos locales en la pirámide DoG
+     */
+    _findExtremas(dogPyramid: any, pyramidImages: any): {
+        score: number;
+        octave: number;
+        x: number;
+        y: number;
+        absScore: number;
+    }[];
+    /**
+     * Aplica pruning para mantener solo los mejores features por bucket
+     */
+    _applyPrune(extremas: any): any[];
+    /**
+     * Calcula la orientación de cada feature
+     */
+    _computeOrientations(extremas: any, pyramidImages: any): void;
+    /**
+     * Calcula descriptores FREAK
+     */
+    _computeFreakDescriptors(extremas: any, pyramidImages: any): void;
+}