@srsergio/taptapp-ar 1.0.87 → 1.0.89

package/README.md

@@ -29,10 +29,12 @@
 
 ## 🌟 Key Features
 
-- 🖼️ **Hyper-Fast Compiler**: Pure JavaScript compiler that generates `.taar` files in **< 3s**.
+- 🎭 **Non-Rigid Surface Tracking**: Supports curved and deformable surfaces using **Delaunay Meshes** and **Mass-Spring Relaxation**.
+- 🚀 **Hyper-Fast Compiler**: Pure JavaScript compiler that generates `.taar` files in **< 3s**.
 - ⚡ **No TensorFlow Dependency**: No TFJS at all. Works natively in any JS environment (Node, Browser, Workers).
 - 🧬 **Fourier Positional Encoding**: Uses high-frequency sine/cosine mappings (GPT-style) for neural-like spatial consistency.
 - 🚀 **Protocol V7 (Moonshot)**: 
+  - **Delaunay Triangular Grid**: Adaptive mesh that tracks surface deformations.
   - **16-bit Fourier Signatures**: Spatial ADN embedded in every feature for harmonic matching.
   - **4-bit Packed Tracking Data**: Grayscale images are compressed to 4-bit depth, slashing file size.
   - **64-bit LSH Descriptors**: Optimized Locality Sensitive Hashing for descriptors.
@@ -54,9 +56,9 @@ npm install @srsergio/taptapp-ar
 
 | Metric | Official MindAR | TapTapp AR V7 | Improvement |
 | :--- | :--- | :--- | :--- |
-| **Compilation Time** | ~23.50s | **~2.61s** | 🚀 **~9x Faster** |
-| **Output Size (.taar)** | ~770 KB | **~50 KB** | 📉 **93% Smaller** |
-| **Descriptor Format** | 84-byte Float | **64-bit LSH** | 🧠 **Massive Data Saving** |
+| **Compilation Time** | ~23.50s | **~0.93s** | 🚀 **~25x Faster** |
+| **Output Size (.taar)** | ~770 KB | **~338 KB** | 📉 **56% Smaller** |
+| **Descriptor Format** | 84-byte Float | **128-bit LSH** | 🧠 **Massive Data Saving** |
 | **Tracking Data** | 8-bit Gray | **4-bit Packed** | 📦 **50% Data Saving** |
 | **Dependency Size** | ~20MB (TFJS) | **< 100KB** | 📦 **99% Smaller Bundle** |
 
@@ -68,11 +70,11 @@ The latest version has been rigorously tested with an adaptive stress test (`rob
 
 | Metric | Result | Description |
 | :--- | :--- | :--- |
-| **Pass Rate** | **96.3%** | High success rate across resolutions. |
-| **Drift Tolerance** | **< 15%** | Validated geometrically against ground truth metadata. |
+| **Pass Rate** | **93.5%** | High success rate across resolutions (202/216). |
+| **Drift Tolerance** | **< 10%** | Validated geometrically against ground truth metadata. |
 | **Tracking Precision** | **Float32** | Full 32-bit precision for optical flow tracking. |
-| **Detection Time** | **~21ms** | Ultra-fast initial detection on standard CPU. |
-| **Total Pipeline** | **~64ms** | Complete loop (Detect + Match + Track + Validate). |
+| **Detection Time** | **< 20ms** | Ultra-fast initial detection on standard CPU. |
+| **Total Pipeline** | **~45ms** | Complete loop (Detect + Match + Track + Validate). |
 
 ---
 
@@ -258,6 +260,8 @@ ar.stop();
 ## 🏗️ Protocol V7 (Moonshot Packed Format)
 TapTapp AR uses a proprietary **Moonshot Vision Codec** that is significantly more efficient than standard AR formats.
 
+- **Non-Rigid Surface Tracking**: Replaces the standard rigid homography with a dynamic **Delaunay Mesh**. This allows the tracker to follow the curvature of posters on cylinders, t-shirts, or slightly bent magazines.
+- **Mass-Spring Relaxation**: The tracking mesh is optimized using physical relaxation, minimizing L2 distance between predicted and tracked points while maintaining topological rigidity.
 - **Fourier Positional Encoding**: Maps 2D coordinates into a 16-dimensional frequency space. This creates a "Neural Consistency Check" that filters out noise and motion blur by checking for harmonic spatial agreement.
 - **4-bit Packed Tracking Data**: Image data used for optical flow is compressed to 4-bit depth.
 - **64-bit LSH Fingerprinting**: Feature descriptors are compressed to just 8 bytes using LSH.
@@ -271,3 +275,4 @@ TapTapp AR uses a proprietary **Moonshot Vision Codec** that is significantly mo
 MIT © [srsergiolazaro](https://github.com/srsergiolazaro)
 
 Based on the core research of MindAR, but completely re-written for high-performance binary processing and JS-only execution.
+

package/dist/compiler/offline-compiler.js

@@ -10,6 +10,7 @@ import { extractTrackingFeatures } from "../core/tracker/extract-utils.js";
 import { DetectorLite } from "../core/detector/detector-lite.js";
 import { build as hierarchicalClusteringBuild } from "../core/matching/hierarchical-clustering.js";
 import * as protocol from "../core/protocol.js";
+import { triangulate, getEdges } from "../core/utils/delaunay.js";
 // Detect environment
 const isNode = typeof process !== "undefined" &&
     process.versions != null &&
@@ -141,6 +142,14 @@ export class OfflineCompiler {
                         px[i] = td.points[i].x;
                         py[i] = td.points[i].y;
                     }
+                    const triangles = triangulate(td.points);
+                    const edges = getEdges(triangles);
+                    const restLengths = new Float32Array(edges.length);
+                    for (let j = 0; j < edges.length; j++) {
+                        const p1 = td.points[edges[j][0]];
+                        const p2 = td.points[edges[j][1]];
+                        restLengths[j] = Math.sqrt((p1.x - p2.x) ** 2 + (p1.y - p2.y) ** 2);
+                    }
                     return {
                         w: td.width,
                         h: td.height,
@@ -148,6 +157,11 @@ export class OfflineCompiler {
                         px,
                         py,
                         d: td.data,
+                        mesh: {
+                            t: new Uint16Array(triangles.flat()),
+                            e: new Uint16Array(edges.flat()),
+                            rl: restLengths
+                        }
                     };
                 }),
                 matchingData: item.matchingData.map((kf) => ({

package/dist/core/estimation/non-rigid-refine.d.ts

@@ -0,0 +1,16 @@
+/**
+ * 🚀 Moonshot: Non-Rigid Surface Refinement (Mass-Spring System)
+ *
+ * Instead of a single homography, we relax a triangle mesh to match tracked points
+ * while preserving edge lengths (Isometric constraint).
+ */
+export function refineNonRigid({ mesh, trackedPoints, currentVertices, iterations }: {
+    mesh: any;
+    trackedPoints: any;
+    currentVertices: any;
+    iterations?: number | undefined;
+}): Float32Array<any>;
+/**
+ * Maps a mesh from reference space to screen space using a homography
+ */
+export function projectMesh(mesh: any, homography: any, width: any, height: any): Float32Array<ArrayBuffer>;

package/dist/core/estimation/non-rigid-refine.js

@@ -0,0 +1,70 @@
+/**
+ * 🚀 Moonshot: Non-Rigid Surface Refinement (Mass-Spring System)
+ *
+ * Instead of a single homography, we relax a triangle mesh to match tracked points
+ * while preserving edge lengths (Isometric constraint).
+ */
+export function refineNonRigid({ mesh, trackedPoints, currentVertices, iterations = 5 }) {
+    const { e: edges, rl: restLengths } = mesh;
+    const numVertices = currentVertices.length / 2;
+    const vertices = new Float32Array(currentVertices); // copy
+    // lambda for stiffness
+    const stiffness = 0.8;
+    const dataFidelity = 0.5;
+    for (let iter = 0; iter < iterations; iter++) {
+        // 1. Edge Length Constraints (Isometric term)
+        for (let i = 0; i < restLengths.length; i++) {
+            const idx1 = edges[i * 2];
+            const idx2 = edges[i * 2 + 1];
+            const restL = restLengths[i];
+            const vx1 = vertices[idx1 * 2];
+            const vy1 = vertices[idx1 * 2 + 1];
+            const vx2 = vertices[idx2 * 2];
+            const vy2 = vertices[idx2 * 2 + 1];
+            const dx = vx2 - vx1;
+            const dy = vy2 - vy1;
+            const currentL = Math.sqrt(dx * dx + dy * dy);
+            if (currentL < 0.0001)
+                continue;
+            const diff = (currentL - restL) / currentL;
+            const moveX = dx * 0.5 * diff * stiffness;
+            const moveY = dy * 0.5 * diff * stiffness;
+            vertices[idx1 * 2] += moveX;
+            vertices[idx1 * 2 + 1] += moveY;
+            vertices[idx2 * 2] -= moveX;
+            vertices[idx2 * 2 + 1] -= moveY;
+        }
+        // 2. Data Fidelity Constraints (Alignment with NCC tracker)
+        for (const tp of trackedPoints) {
+            const idx = tp.meshIndex;
+            if (idx === undefined)
+                continue;
+            const targetX = tp.x;
+            const targetY = tp.y;
+            vertices[idx * 2] += (targetX - vertices[idx * 2]) * dataFidelity;
+            vertices[idx * 2 + 1] += (targetY - vertices[idx * 2 + 1]) * dataFidelity;
+        }
+    }
+    return vertices;
+}
+/**
+ * Maps a mesh from reference space to screen space using a homography
+ */
+export function projectMesh(mesh, homography, width, height) {
+    const { px, py } = mesh; // original octave points used as mesh vertices
+    const numVertices = px.length;
+    const projected = new Float32Array(numVertices * 2);
+    const h = homography;
+    const h00 = h[0][0], h01 = h[0][1], h02 = h[0][3];
+    const h10 = h[1][0], h11 = h[1][1], h12 = h[1][3];
+    const h20 = h[2][0], h21 = h[2][1], h22 = h[2][3];
+    for (let i = 0; i < numVertices; i++) {
+        const x = px[i];
+        const y = py[i];
+        const uz = (x * h20) + (y * h21) + h22;
+        const invZ = 1.0 / uz;
+        projected[i * 2] = ((x * h00) + (y * h01) + h02) * invZ;
+        projected[i * 2 + 1] = ((x * h10) + (y * h11) + h12) * invZ;
+    }
+    return projected;
+}

package/dist/core/protocol.d.ts

@@ -1,4 +1,4 @@
-export declare const CURRENT_VERSION = 7;
+export declare const CURRENT_VERSION = 8;
 /**
  * Morton Order calculation for spatial sorting
  */

package/dist/core/protocol.js

@@ -1,5 +1,5 @@
 import * as msgpack from "@msgpack/msgpack";
-export const CURRENT_VERSION = 7;
+export const CURRENT_VERSION = 8;
 /**
  * Morton Order calculation for spatial sorting
  */
@@ -137,6 +137,11 @@ export function decodeTaar(buffer) {
                 if (td.d)
                     td.d = unpacked;
             }
+            if (td.mesh) {
+                td.mesh.t = normalizeBuffer(td.mesh.t, Uint16Array);
+                td.mesh.e = normalizeBuffer(td.mesh.e, Uint16Array);
+                td.mesh.rl = normalizeBuffer(td.mesh.rl, Float32Array);
+            }
         }
         // 2. Process Matching Data
         for (const kf of item.matchingData) {

package/dist/core/tracker/tracker.d.ts

@@ -8,6 +8,7 @@ export class Tracker {
     debugMode: boolean;
     trackingKeyframeList: any[];
     prebuiltData: any[];
+    meshVerticesState: any[];
     templateBuffer: Float32Array<ArrayBuffer>;
     dummyRun(inputData: any): void;
     track(inputData: any, lastModelViewTransform: any, targetIndex: any): {
@@ -17,6 +18,7 @@ export class Tracker {
         debugExtra: {};
         indices?: undefined;
         octaveIndex?: undefined;
+        deformedMesh?: undefined;
     } | {
         worldCoords: {
             x: number;
@@ -30,6 +32,10 @@ export class Tracker {
         reliabilities: number[];
         indices: number[];
         octaveIndex: any;
+        deformedMesh: {
+            vertices: Float32Array<ArrayBuffer>;
+            triangles: any;
+        } | null;
         debugExtra: {};
     };
     lastOctaveIndex: any[] | undefined;

package/dist/core/tracker/tracker.js

@@ -1,4 +1,5 @@
 import { buildModelViewProjectionTransform, computeScreenCoordiate } from "../estimation/utils.js";
+import { refineNonRigid, projectMesh } from "../estimation/non-rigid-refine.js";
 const AR2_DEFAULT_TS = 6;
 const AR2_DEFAULT_TS_GAP = 1;
 const AR2_SEARCH_SIZE = 12; // Reduced from 25 to 12 for high-speed tracking (25 is overkill)
@@ -25,10 +26,13 @@ class Tracker {
                 width: keyframe.w,
                 height: keyframe.h,
                 scale: keyframe.s,
+                mesh: keyframe.mesh,
                 // Recyclable projected image buffer
                 projectedImage: new Float32Array(keyframe.w * keyframe.h)
             }));
         }
+        // Maintain mesh vertices state for temporal continuity
+        this.meshVerticesState = []; // [targetIndex][octaveIndex]
         // Pre-allocate template data buffer to avoid garbage collection
         const templateOneSize = AR2_DEFAULT_TS;
         const templateSize = templateOneSize * 2 + 1;
@@ -84,7 +88,8 @@ class Tracker {
         const { px, py, s: scale } = trackingFrame;
         const reliabilities = [];
         for (let i = 0; i < matchingPoints.length; i++) {
-            if (sim[i] > AR2_SIM_THRESH && i < px.length) {
+            const reliability = sim[i];
+            if (reliability > AR2_SIM_THRESH && i < px.length) {
                 goodTrack.push(i);
                 const point = computeScreenCoordiate(modelViewProjectionTransform, matchingPoints[i][0], matchingPoints[i][1]);
                 screenCoords.push(point);
@@ -93,9 +98,53 @@ class Tracker {
                     y: py[i] / scale,
                     z: 0,
                 });
-                reliabilities.push(sim[i]);
+                reliabilities.push(reliability);
             }
         }
+        // --- 🚀 MOONSHOT: Non-Rigid Mesh Refinement ---
+        let deformedMesh = null;
+        if (prebuilt.mesh && goodTrack.length >= 4) {
+            if (!this.meshVerticesState[targetIndex])
+                this.meshVerticesState[targetIndex] = [];
+            let currentOctaveVertices = this.meshVerticesState[targetIndex][octaveIndex];
+            // Initial setup: If no state, use the reference points (normalized) as first guess
+            if (!currentOctaveVertices) {
+                currentOctaveVertices = new Float32Array(px.length * 2);
+                for (let i = 0; i < px.length; i++) {
+                    currentOctaveVertices[i * 2] = px[i];
+                    currentOctaveVertices[i * 2 + 1] = py[i];
+                }
+            }
+            // Data fidelity: Prepare tracked targets for mass-spring
+            const trackedTargets = [];
+            for (let j = 0; j < goodTrack.length; j++) {
+                const idx = goodTrack[j];
+                trackedTargets.push({
+                    meshIndex: idx,
+                    x: matchingPoints[idx][0] * scale, // Convert back to octave space pixels
+                    y: matchingPoints[idx][1] * scale
+                });
+            }
+            // Relax mesh in octave space
+            const refinedOctaveVertices = refineNonRigid({
+                mesh: prebuilt.mesh,
+                trackedPoints: trackedTargets,
+                currentVertices: currentOctaveVertices,
+                iterations: 5
+            });
+            this.meshVerticesState[targetIndex][octaveIndex] = refinedOctaveVertices;
+            // Project deformed mesh to screen space
+            const screenMeshVertices = new Float32Array(refinedOctaveVertices.length);
+            for (let i = 0; i < refinedOctaveVertices.length; i += 2) {
+                const p = computeScreenCoordiate(modelViewProjectionTransform, refinedOctaveVertices[i] / scale, refinedOctaveVertices[i + 1] / scale);
+                screenMeshVertices[i] = p.x;
+                screenMeshVertices[i + 1] = p.y;
+            }
+            deformedMesh = {
+                vertices: screenMeshVertices,
+                triangles: prebuilt.mesh.t
+            };
+        }
         // 2.1 Spatial distribution check: Avoid getting stuck in corners/noise
         if (screenCoords.length >= 8) {
             let minX = Infinity, minY = Infinity, maxX = -Infinity, maxY = -Infinity;
@@ -125,7 +174,7 @@ class Tracker {
                 trackedPoints: screenCoords,
             };
         }
-        return { worldCoords, screenCoords, reliabilities, indices: goodTrack, octaveIndex, debugExtra };
+        return { worldCoords, screenCoords, reliabilities, indices: goodTrack, octaveIndex, deformedMesh, debugExtra };
     }
     /**
      * Pure JS implementation of NCC matching

package/dist/core/utils/delaunay.d.ts

@@ -0,0 +1,10 @@
+/**
+ * 🚀 Moonshot: Simple incremental Delaunay Triangulation (Bowyer-Watson)
+ *
+ * Used to create a topological mesh covering the image target features.
+ */
+export function triangulate(points: any): number[][];
+/**
+ * Extract edges from triangles for Mass-Spring logic
+ */
+export function getEdges(triangles: any): number[][];

package/dist/core/utils/delaunay.js

@@ -0,0 +1,125 @@
+/**
+ * 🚀 Moonshot: Simple incremental Delaunay Triangulation (Bowyer-Watson)
+ *
+ * Used to create a topological mesh covering the image target features.
+ */
+export function triangulate(points) {
+    if (points.length < 3)
+        return [];
+    // 1. Create a super-triangle that contains all points
+    // Find min/max bounds
+    let minX = Infinity, minY = Infinity, maxX = -Infinity, maxY = -Infinity;
+    for (const p of points) {
+        if (p.x < minX)
+            minX = p.x;
+        if (p.x > maxX)
+            maxX = p.x;
+        if (p.y < minY)
+            minY = p.y;
+        if (p.y > maxY)
+            maxY = p.y;
+    }
+    const dx = maxX - minX;
+    const dy = maxY - minY;
+    const deltaMax = Math.max(dx, dy);
+    const midX = (minX + maxX) / 2;
+    const midY = (minY + maxY) / 2;
+    // A triangle large enough to cover all points
+    const p1 = { x: midX - 20 * deltaMax, y: midY - deltaMax };
+    const p2 = { x: midX, y: midY + 20 * deltaMax };
+    const p3 = { x: midX + 20 * deltaMax, y: midY - deltaMax };
+    let triangles = [
+        { p1, p2, p3, indices: [-1, -2, -3] }
+    ];
+    // 2. Add points one by one
+    for (let i = 0; i < points.length; i++) {
+        const p = points[i];
+        const badTriangles = [];
+        for (const t of triangles) {
+            if (isInCircumcircle(p, t)) {
+                badTriangles.push(t);
+            }
+        }
+        const polygon = [];
+        for (const t of badTriangles) {
+            const edges = [
+                { a: t.p1, b: t.p2, i1: t.indices[0], i2: t.indices[1] },
+                { a: t.p2, b: t.p3, i1: t.indices[1], i2: t.indices[2] },
+                { a: t.p3, b: t.p1, i1: t.indices[2], i2: t.indices[0] }
+            ];
+            for (const edge of edges) {
+                let isShared = false;
+                for (const t2 of badTriangles) {
+                    if (t === t2)
+                        continue;
+                    if (isSameEdge(edge, t2)) {
+                        isShared = true;
+                        break;
+                    }
+                }
+                if (!isShared) {
+                    polygon.push(edge);
+                }
+            }
+        }
+        // Remove bad triangles
+        triangles = triangles.filter(t => !badTriangles.includes(t));
+        // Add new triangles from polygon edges to point p
+        for (const edge of polygon) {
+            triangles.push({
+                p1: edge.a,
+                p2: edge.b,
+                p3: p,
+                indices: [edge.i1, edge.i2, i]
+            });
+        }
+    }
+    // 3. Remove triangles that share vertices with the super-triangle
+    return triangles.filter(t => {
+        return t.indices[0] >= 0 && t.indices[1] >= 0 && t.indices[2] >= 0;
+    }).map(t => t.indices);
+}
+function isInCircumcircle(p, t) {
+    const x1 = t.p1.x, y1 = t.p1.y;
+    const x2 = t.p2.x, y2 = t.p2.y;
+    const x3 = t.p3.x, y3 = t.p3.y;
+    const D = 2 * (x1 * (y2 - y3) + x2 * (y3 - y1) + x3 * (y1 - y2));
+    const centerX = ((x1 * x1 + y1 * y1) * (y2 - y3) + (x2 * x2 + y2 * y2) * (y3 - y1) + (x3 * x3 + y3 * y3) * (y1 - y2)) / D;
+    const centerY = ((x1 * x1 + y1 * y1) * (x3 - x2) + (x2 * x2 + y2 * y2) * (x1 - x3) + (x3 * x3 + y3 * y3) * (x2 - x1)) / D;
+    const radiusSq = (x1 - centerX) * (x1 - centerX) + (y1 - centerY) * (y1 - centerY);
+    const distSq = (p.x - centerX) * (p.x - centerX) + (p.y - centerY) * (p.y - centerY);
+    return distSq <= radiusSq;
+}
+function isSameEdge(edge, triangle) {
+    const tEdges = [
+        [triangle.indices[0], triangle.indices[1]],
+        [triangle.indices[1], triangle.indices[2]],
+        [triangle.indices[2], triangle.indices[0]]
+    ];
+    for (const te of tEdges) {
+        if ((edge.i1 === te[0] && edge.i2 === te[1]) || (edge.i1 === te[1] && edge.i2 === te[0])) {
+            return true;
+        }
+    }
+    return false;
+}
+/**
+ * Extract edges from triangles for Mass-Spring logic
+ */
+export function getEdges(triangles) {
+    const edgeSet = new Set();
+    const edges = [];
+    for (const t of triangles) {
+        const pairs = [[t[0], t[1]], [t[1], t[2]], [t[2], t[0]]];
+        for (const pair of pairs) {
+            const low = Math.min(pair[0], pair[1]);
+            const high = Math.max(pair[0], pair[1]);
+            const key = `${low}-${high}`;
+            if (!edgeSet.has(key)) {
+                edgeSet.add(key);
+                edges.push([low, high]);
+            }
+        }
+    }
+    return edges;
+}

package/dist/runtime/controller.d.ts

@@ -68,10 +68,17 @@ declare class Controller {
         featurePoints: any;
     }>;
     _trackAndUpdate(inputData: any, lastModelViewTransform: number[][], targetIndex: number): Promise<{
+        modelViewTransform: null;
+        screenCoords: any[];
+        reliabilities: number[];
+        stabilities: number[];
+        deformedMesh?: undefined;
+    } | {
         modelViewTransform: any;
         screenCoords: any[];
         reliabilities: number[];
         stabilities: number[];
+        deformedMesh: any;
     }>;
     processVideo(input: any): void;
     stopProcessVideo(): void;
@@ -93,6 +100,7 @@ declare class Controller {
         debugExtra: {};
         indices?: undefined;
         octaveIndex?: undefined;
+        deformedMesh?: undefined;
     } | {
         worldCoords: {
             x: number;
@@ -106,6 +114,10 @@ declare class Controller {
         reliabilities: number[];
         indices: number[];
         octaveIndex: any;
+        deformedMesh: {
+            vertices: Float32Array<ArrayBuffer>;
+            triangles: any;
+        } | null;
         debugExtra: {};
     }>;
     trackUpdate(modelViewTransform: number[][], trackFeatures: any): Promise<any>;

package/dist/runtime/controller.js

@@ -193,9 +193,9 @@ class Controller {
         return { targetIndex, modelViewTransform, screenCoords, worldCoords, featurePoints };
     }
     async _trackAndUpdate(inputData, lastModelViewTransform, targetIndex) {
-        const { worldCoords, screenCoords, reliabilities, indices = [], octaveIndex = 0 } = await this._workerTrack(inputData, lastModelViewTransform, targetIndex);
+        const { worldCoords, screenCoords, reliabilities, indices = [], octaveIndex = 0, deformedMesh } = await this._workerTrack(inputData, lastModelViewTransform, targetIndex);
         if (!worldCoords || worldCoords.length === 0) {
-            return { modelViewTransform: null, screenCoords: [], reliabilities: [], stabilities: [] };
+            return { modelViewTransform: null, screenCoords: [], reliabilities: [], stabilities: [], deformedMesh: null };
         }
         const state = this.trackingStates[targetIndex];
         if (!state.pointStabilities)
@@ -264,13 +264,15 @@ class Controller {
             stabilities: finalWorldCoords.map((_, i) => {
                 const globalIdx = indices[i];
                 return stabilities[globalIdx];
-            })
+            }),
+            deformedMesh
         });
         return {
             modelViewTransform,
             screenCoords: finalScreenCoords,
             reliabilities: finalReliabilities,
-            stabilities: finalStabilities
+            stabilities: finalStabilities,
+            deformedMesh
         };
     }
     processVideo(input) {
@@ -329,6 +331,7 @@ class Controller {
                             trackingState.screenCoords = result.screenCoords;
                             trackingState.reliabilities = result.reliabilities;
                             trackingState.stabilities = result.stabilities;
+                            trackingState.deformedMesh = result.deformedMesh;
                         }
                     }
                     const wasShowing = trackingState.showing;
@@ -365,7 +368,8 @@ class Controller {
                             modelViewTransform: trackingState.currentModelViewTransform,
                             screenCoords: trackingState.screenCoords,
                             reliabilities: trackingState.reliabilities,
-                            stabilities: trackingState.stabilities
+                            stabilities: trackingState.stabilities,
+                            deformedMesh: trackingState.deformedMesh
                         });
                     }
                 }

package/package.json

@@ -1,6 +1,6 @@
 {
     "name": "@srsergio/taptapp-ar",
-    "version": "1.0.87",
+    "version": "1.0.89",
     "description": "Ultra-fast Augmented Reality (AR) SDK for Node.js and Browser. Image tracking with 100% pure JavaScript, zero-dependencies, and high-performance compilation.",
     "keywords": [
         "augmented reality",

package/src/compiler/offline-compiler.ts

@@ -11,6 +11,7 @@ import { extractTrackingFeatures } from "../core/tracker/extract-utils.js";
 import { DetectorLite } from "../core/detector/detector-lite.js";
 import { build as hierarchicalClusteringBuild } from "../core/matching/hierarchical-clustering.js";
 import * as protocol from "../core/protocol.js";
+import { triangulate, getEdges } from "../core/utils/delaunay.js";
 
 // Detect environment
 const isNode = typeof process !== "undefined" &&
@@ -176,6 +177,15 @@ export class OfflineCompiler {
                         px[i] = td.points[i].x;
                         py[i] = td.points[i].y;
                     }
+                    const triangles = triangulate(td.points);
+                    const edges = getEdges(triangles);
+                    const restLengths = new Float32Array(edges.length);
+                    for (let j = 0; j < edges.length; j++) {
+                        const p1 = td.points[edges[j][0]];
+                        const p2 = td.points[edges[j][1]];
+                        restLengths[j] = Math.sqrt((p1.x - p2.x) ** 2 + (p1.y - p2.y) ** 2);
+                    }
+
                     return {
                         w: td.width,
                         h: td.height,
@@ -183,6 +193,11 @@ export class OfflineCompiler {
                         px,
                         py,
                         d: td.data,
+                        mesh: {
+                            t: new Uint16Array(triangles.flat()),
+                            e: new Uint16Array(edges.flat()),
+                            rl: restLengths
+                        }
                     };
                 }),
                 matchingData: item.matchingData.map((kf: any) => ({

package/src/core/estimation/non-rigid-refine.js

@@ -0,0 +1,85 @@
+/**
+ * 🚀 Moonshot: Non-Rigid Surface Refinement (Mass-Spring System)
+ * 
+ * Instead of a single homography, we relax a triangle mesh to match tracked points
+ * while preserving edge lengths (Isometric constraint).
+ */
+
+export function refineNonRigid({ mesh, trackedPoints, currentVertices, iterations = 5 }) {
+    const { e: edges, rl: restLengths } = mesh;
+    const numVertices = currentVertices.length / 2;
+    const vertices = new Float32Array(currentVertices); // copy
+
+    // lambda for stiffness
+    const stiffness = 0.8;
+    const dataFidelity = 0.5;
+
+    for (let iter = 0; iter < iterations; iter++) {
+        // 1. Edge Length Constraints (Isometric term)
+        for (let i = 0; i < restLengths.length; i++) {
+            const idx1 = edges[i * 2];
+            const idx2 = edges[i * 2 + 1];
+            const restL = restLengths[i];
+
+            const vx1 = vertices[idx1 * 2];
+            const vy1 = vertices[idx1 * 2 + 1];
+            const vx2 = vertices[idx2 * 2];
+            const vy2 = vertices[idx2 * 2 + 1];
+
+            const dx = vx2 - vx1;
+            const dy = vy2 - vy1;
+            const currentL = Math.sqrt(dx * dx + dy * dy);
+
+            if (currentL < 0.0001) continue;
+
+            const diff = (currentL - restL) / currentL;
+            const moveX = dx * 0.5 * diff * stiffness;
+            const moveY = dy * 0.5 * diff * stiffness;
+
+            vertices[idx1 * 2] += moveX;
+            vertices[idx1 * 2 + 1] += moveY;
+            vertices[idx2 * 2] -= moveX;
+            vertices[idx2 * 2 + 1] -= moveY;
+        }
+
+        // 2. Data Fidelity Constraints (Alignment with NCC tracker)
+        for (const tp of trackedPoints) {
+            const idx = tp.meshIndex;
+            if (idx === undefined) continue;
+
+            const targetX = tp.x;
+            const targetY = tp.y;
+
+            vertices[idx * 2] += (targetX - vertices[idx * 2]) * dataFidelity;
+            vertices[idx * 2 + 1] += (targetY - vertices[idx * 2 + 1]) * dataFidelity;
+        }
+    }
+
+    return vertices;
+}
+
+/**
+ * Maps a mesh from reference space to screen space using a homography
+ */
+export function projectMesh(mesh, homography, width, height) {
+    const { px, py } = mesh; // original octave points used as mesh vertices
+    const numVertices = px.length;
+    const projected = new Float32Array(numVertices * 2);
+
+    const h = homography;
+    const h00 = h[0][0], h01 = h[0][1], h02 = h[0][3];
+    const h10 = h[1][0], h11 = h[1][1], h12 = h[1][3];
+    const h20 = h[2][0], h21 = h[2][1], h22 = h[2][3];
+
+    for (let i = 0; i < numVertices; i++) {
+        const x = px[i];
+        const y = py[i];
+
+        const uz = (x * h20) + (y * h21) + h22;
+        const invZ = 1.0 / uz;
+        projected[i * 2] = ((x * h00) + (y * h01) + h02) * invZ;
+        projected[i * 2 + 1] = ((x * h10) + (y * h11) + h12) * invZ;
+    }
+
+    return projected;
+}

package/src/core/protocol.ts

@@ -1,6 +1,6 @@
 import * as msgpack from "@msgpack/msgpack";
 
-export const CURRENT_VERSION = 7;
+export const CURRENT_VERSION = 8;
 
 /**
  * Morton Order calculation for spatial sorting
@@ -157,6 +157,12 @@ export function decodeTaar(buffer: ArrayBuffer | Uint8Array) {
                 if (td.data) td.data = unpacked;
                 if (td.d) td.d = unpacked;
             }
+
+            if (td.mesh) {
+                td.mesh.t = normalizeBuffer(td.mesh.t, Uint16Array);
+                td.mesh.e = normalizeBuffer(td.mesh.e, Uint16Array);
+                td.mesh.rl = normalizeBuffer(td.mesh.rl, Float32Array);
+            }
         }
 
         // 2. Process Matching Data

package/src/core/tracker/tracker.js

@@ -1,4 +1,5 @@
 import { buildModelViewProjectionTransform, computeScreenCoordiate } from "../estimation/utils.js";
+import { refineNonRigid, projectMesh } from "../estimation/non-rigid-refine.js";
 
 const AR2_DEFAULT_TS = 6;
 const AR2_DEFAULT_TS_GAP = 1;
@@ -37,11 +38,15 @@ class Tracker {
         width: keyframe.w,
         height: keyframe.h,
         scale: keyframe.s,
+        mesh: keyframe.mesh,
         // Recyclable projected image buffer
         projectedImage: new Float32Array(keyframe.w * keyframe.h)
       }));
     }
 
+    // Maintain mesh vertices state for temporal continuity
+    this.meshVerticesState = []; // [targetIndex][octaveIndex]
+
     // Pre-allocate template data buffer to avoid garbage collection
     const templateOneSize = AR2_DEFAULT_TS;
     const templateSize = templateOneSize * 2 + 1;
@@ -122,7 +127,8 @@ class Tracker {
     const reliabilities = [];
 
     for (let i = 0; i < matchingPoints.length; i++) {
-      if (sim[i] > AR2_SIM_THRESH && i < px.length) {
+      const reliability = sim[i];
+      if (reliability > AR2_SIM_THRESH && i < px.length) {
         goodTrack.push(i);
         const point = computeScreenCoordiate(
           modelViewProjectionTransform,
@@ -135,8 +141,63 @@ class Tracker {
           y: py[i] / scale,
           z: 0,
         });
-        reliabilities.push(sim[i]);
+        reliabilities.push(reliability);
+      }
+    }
+
+    // --- 🚀 MOONSHOT: Non-Rigid Mesh Refinement ---
+    let deformedMesh = null;
+    if (prebuilt.mesh && goodTrack.length >= 4) {
+      if (!this.meshVerticesState[targetIndex]) this.meshVerticesState[targetIndex] = [];
+
+      let currentOctaveVertices = this.meshVerticesState[targetIndex][octaveIndex];
+
+      // Initial setup: If no state, use the reference points (normalized) as first guess
+      if (!currentOctaveVertices) {
+        currentOctaveVertices = new Float32Array(px.length * 2);
+        for (let i = 0; i < px.length; i++) {
+          currentOctaveVertices[i * 2] = px[i];
+          currentOctaveVertices[i * 2 + 1] = py[i];
+        }
+      }
+
+      // Data fidelity: Prepare tracked targets for mass-spring
+      const trackedTargets = [];
+      for (let j = 0; j < goodTrack.length; j++) {
+        const idx = goodTrack[j];
+        trackedTargets.push({
+          meshIndex: idx,
+          x: matchingPoints[idx][0] * scale, // Convert back to octave space pixels
+          y: matchingPoints[idx][1] * scale
+        });
       }
+
+      // Relax mesh in octave space
+      const refinedOctaveVertices = refineNonRigid({
+        mesh: prebuilt.mesh,
+        trackedPoints: trackedTargets,
+        currentVertices: currentOctaveVertices,
+        iterations: 5
+      });
+
+      this.meshVerticesState[targetIndex][octaveIndex] = refinedOctaveVertices;
+
+      // Project deformed mesh to screen space
+      const screenMeshVertices = new Float32Array(refinedOctaveVertices.length);
+      for (let i = 0; i < refinedOctaveVertices.length; i += 2) {
+        const p = computeScreenCoordiate(
+          modelViewProjectionTransform,
+          refinedOctaveVertices[i] / scale,
+          refinedOctaveVertices[i + 1] / scale
+        );
+        screenMeshVertices[i] = p.x;
+        screenMeshVertices[i + 1] = p.y;
+      }
+
+      deformedMesh = {
+        vertices: screenMeshVertices,
+        triangles: prebuilt.mesh.t
+      };
     }
 
     // 2.1 Spatial distribution check: Avoid getting stuck in corners/noise
@@ -164,7 +225,7 @@ class Tracker {
       };
     }
 
-    return { worldCoords, screenCoords, reliabilities, indices: goodTrack, octaveIndex, debugExtra };
+    return { worldCoords, screenCoords, reliabilities, indices: goodTrack, octaveIndex, deformedMesh, debugExtra };
   }
 
   /**

package/src/core/utils/delaunay.js

@@ -0,0 +1,139 @@
+/**
+ * 🚀 Moonshot: Simple incremental Delaunay Triangulation (Bowyer-Watson)
+ * 
+ * Used to create a topological mesh covering the image target features.
+ */
+
+export function triangulate(points) {
+    if (points.length < 3) return [];
+
+    // 1. Create a super-triangle that contains all points
+    // Find min/max bounds
+    let minX = Infinity, minY = Infinity, maxX = -Infinity, maxY = -Infinity;
+    for (const p of points) {
+        if (p.x < minX) minX = p.x;
+        if (p.x > maxX) maxX = p.x;
+        if (p.y < minY) minY = p.y;
+        if (p.y > maxY) maxY = p.y;
+    }
+
+    const dx = maxX - minX;
+    const dy = maxY - minY;
+    const deltaMax = Math.max(dx, dy);
+    const midX = (minX + maxX) / 2;
+    const midY = (minY + maxY) / 2;
+
+    // A triangle large enough to cover all points
+    const p1 = { x: midX - 20 * deltaMax, y: midY - deltaMax };
+    const p2 = { x: midX, y: midY + 20 * deltaMax };
+    const p3 = { x: midX + 20 * deltaMax, y: midY - deltaMax };
+
+    let triangles = [
+        { p1, p2, p3, indices: [-1, -2, -3] }
+    ];
+
+    // 2. Add points one by one
+    for (let i = 0; i < points.length; i++) {
+        const p = points[i];
+        const badTriangles = [];
+
+        for (const t of triangles) {
+            if (isInCircumcircle(p, t)) {
+                badTriangles.push(t);
+            }
+        }
+
+        const polygon = [];
+        for (const t of badTriangles) {
+            const edges = [
+                { a: t.p1, b: t.p2, i1: t.indices[0], i2: t.indices[1] },
+                { a: t.p2, b: t.p3, i1: t.indices[1], i2: t.indices[2] },
+                { a: t.p3, b: t.p1, i1: t.indices[2], i2: t.indices[0] }
+            ];
+
+            for (const edge of edges) {
+                let isShared = false;
+                for (const t2 of badTriangles) {
+                    if (t === t2) continue;
+                    if (isSameEdge(edge, t2)) {
+                        isShared = true;
+                        break;
+                    }
+                }
+                if (!isShared) {
+                    polygon.push(edge);
+                }
+            }
+        }
+
+        // Remove bad triangles
+        triangles = triangles.filter(t => !badTriangles.includes(t));
+
+        // Add new triangles from polygon edges to point p
+        for (const edge of polygon) {
+            triangles.push({
+                p1: edge.a,
+                p2: edge.b,
+                p3: p,
+                indices: [edge.i1, edge.i2, i]
+            });
+        }
+    }
+
+    // 3. Remove triangles that share vertices with the super-triangle
+    return triangles.filter(t => {
+        return t.indices[0] >= 0 && t.indices[1] >= 0 && t.indices[2] >= 0;
+    }).map(t => t.indices);
+}
+
+function isInCircumcircle(p, t) {
+    const x1 = t.p1.x, y1 = t.p1.y;
+    const x2 = t.p2.x, y2 = t.p2.y;
+    const x3 = t.p3.x, y3 = t.p3.y;
+
+    const D = 2 * (x1 * (y2 - y3) + x2 * (y3 - y1) + x3 * (y1 - y2));
+    const centerX = ((x1 * x1 + y1 * y1) * (y2 - y3) + (x2 * x2 + y2 * y2) * (y3 - y1) + (x3 * x3 + y3 * y3) * (y1 - y2)) / D;
+    const centerY = ((x1 * x1 + y1 * y1) * (x3 - x2) + (x2 * x2 + y2 * y2) * (x1 - x3) + (x3 * x3 + y3 * y3) * (x2 - x1)) / D;
+
+    const radiusSq = (x1 - centerX) * (x1 - centerX) + (y1 - centerY) * (y1 - centerY);
+    const distSq = (p.x - centerX) * (p.x - centerX) + (p.y - centerY) * (p.y - centerY);
+
+    return distSq <= radiusSq;
+}
+
+function isSameEdge(edge, triangle) {
+    const tEdges = [
+        [triangle.indices[0], triangle.indices[1]],
+        [triangle.indices[1], triangle.indices[2]],
+        [triangle.indices[2], triangle.indices[0]]
+    ];
+
+    for (const te of tEdges) {
+        if ((edge.i1 === te[0] && edge.i2 === te[1]) || (edge.i1 === te[1] && edge.i2 === te[0])) {
+            return true;
+        }
+    }
+    return false;
+}
+
+/**
+ * Extract edges from triangles for Mass-Spring logic
+ */
+export function getEdges(triangles) {
+    const edgeSet = new Set();
+    const edges = [];
+
+    for (const t of triangles) {
+        const pairs = [[t[0], t[1]], [t[1], t[2]], [t[2], t[0]]];
+        for (const pair of pairs) {
+            const low = Math.min(pair[0], pair[1]);
+            const high = Math.max(pair[0], pair[1]);
+            const key = `${low}-${high}`;
+            if (!edgeSet.has(key)) {
+                edgeSet.add(key);
+                edges.push([low, high]);
+            }
+        }
+    }
+    return edges;
+}

package/src/runtime/controller.ts

@@ -262,14 +262,14 @@ class Controller {
     }
 
     async _trackAndUpdate(inputData: any, lastModelViewTransform: number[][], targetIndex: number) {
-        const { worldCoords, screenCoords, reliabilities, indices = [], octaveIndex = 0 } = await this._workerTrack(
+        const { worldCoords, screenCoords, reliabilities, indices = [], octaveIndex = 0, deformedMesh } = await this._workerTrack(
             inputData,
             lastModelViewTransform,
             targetIndex,
         );
 
         if (!worldCoords || worldCoords.length === 0) {
-            return { modelViewTransform: null, screenCoords: [], reliabilities: [], stabilities: [] };
+            return { modelViewTransform: null, screenCoords: [], reliabilities: [], stabilities: [], deformedMesh: null };
         }
 
         const state = this.trackingStates[targetIndex];
@@ -344,14 +344,16 @@ class Controller {
             stabilities: finalWorldCoords.map((_, i) => {
                 const globalIdx = indices[i];
                 return stabilities[globalIdx];
-            })
+            }),
+            deformedMesh
         });
 
         return {
             modelViewTransform,
             screenCoords: finalScreenCoords,
             reliabilities: finalReliabilities,
-            stabilities: finalStabilities
+            stabilities: finalStabilities,
+            deformedMesh
         };
     }
 
@@ -420,6 +422,7 @@ class Controller {
                             trackingState.screenCoords = result.screenCoords;
                             trackingState.reliabilities = result.reliabilities;
                             trackingState.stabilities = result.stabilities;
+                            (trackingState as any).deformedMesh = result.deformedMesh;
                         }
                     }
 
@@ -465,7 +468,8 @@ class Controller {
                             modelViewTransform: trackingState.currentModelViewTransform,
                             screenCoords: trackingState.screenCoords,
                             reliabilities: trackingState.reliabilities,
-                            stabilities: trackingState.stabilities
+                            stabilities: trackingState.stabilities,
+                            deformedMesh: (trackingState as any).deformedMesh
                         });
                     }
                 }