prism-mcp-server 9.2.6 → 9.3.0

package/README.md

@@ -503,6 +503,43 @@ A gorgeous glassmorphism UI at `localhost:3000` that lets you see exactly what y
 ### 🧬 10× Memory Compression
 Powered by a pure TypeScript port of Google's TurboQuant (inspired by Google's ICLR research), Prism compresses 768-dim embeddings from **3,072 bytes → ~400 bytes** — enabling decades of session history on a standard laptop. No native modules. No vector database required. To mitigate quantization degradation (where repeated compress/decompress cycles could smear subtle corrections after 10k+ memories), Prism leverages autonomous **ledger compaction** and **Deep Storage cleanup** to guarantee high-fidelity memory integrity over time.
 
+<details>
+<summary><strong>📊 1M-Vector Benchmark (d=768, 4-bit)</strong></summary>
+
+Validated on 1,000,000 synthetic unit vectors at production dimension (d=768), run on Apple M4 Max (36GB):
+
+| Metric | Value |
+|--------|-------|
+| **Compression ratio** | 7.7× (3,072 → 400 bytes) |
+| **Throughput** | 833 vectors/sec |
+| **Peak heap** | 329 MB |
+| **Total time** | 57.6 minutes |
+
+**Residual norm distribution** — the quantization error after Householder rotation + Lloyd-Max scalar quantization:
+
+| Statistic | Value |
+|-----------|-------|
+| Mean | 0.1855 |
+| CV (coefficient of variation) | **0.038** |
+| P99/P50 ratio | **1.11** |
+| P99.9/P50 ratio | 1.16 |
+| Max/Min ratio | 1.46 |
+| IQR | 0.009 |
+
+A CV of 0.038 means the residual norm barely varies across 1M vectors — **there is effectively no long tail**. The QJL correction term (which scales linearly with residualNorm) remains stable even for P99.9 outliers.
+
+**R@k retrieval accuracy** (global corpus, 30 trials):
+
+| Corpus Size | R@1 | R@5 |
+|-------------|-----|-----|
+| N=1,000 | 20.0% | 60.0% |
+| N=10,000 | 36.7% | 76.7% |
+| N=50,000 | 53.3% | **90.0%** |
+
+> **Note:** R@k on random high-dimensional vectors is inherently harder than on real embeddings (all vectors are near-equidistant in d=768). Real-world retrieval with clustered embeddings produces higher accuracy. See [tests/residual-distribution.test.ts](tests/residual-distribution.test.ts) and [tests/benchmarks/residual-1m.ts](tests/benchmarks/residual-1m.ts) for full methodology.
+
+</details>
+
 ### 🐝 Multi-Agent Hivemind & Enterprise Sync
 While local SQLite is amazing for solo developers, enterprise teams cannot share a local SQLite file. Prism breaks the "local-only" ceiling via **Supabase Sync** and the **Multi-Agent Hivemind**—scaling effortlessly to teams of 50+ developers using agents. Multiple agents (dev, QA, PM) can work on the same project with **role-isolated memory**, discover each other automatically, and share context in real-time via Telepathy sync to a shared Postgres backend. → [Multi-agent setup example](examples/multi-agent-hivemind/)
 
@@ -789,8 +826,12 @@ The Generator strips the `console.log`, resubmits, and the next `EVALUATE` retur
 
 ## 🆕 What's New
 
-> **Current release: v9.2.4 — Cross-Backend Reconciliation**
+> **Current release: v9.3.0 — TurboQuant ResidualNorm Tiebreaker**
 
+- 🎯 **v9.3.0 — TurboQuant ResidualNorm Tiebreaker:** Configurable ranking optimization for Tier-2 search. When compressed cosine scores are within ε of each other, prefers the candidate with lower `residualNorm` (more trustworthy compressed representation). `PRISM_TURBOQUANT_TIEBREAKER_EPSILON=0.005` gives +2pp R@1, +1pp R@5. Empirically validated at N=5K with A/B test. 1066 tests, 0 regressions. Inspired by [@m13v's suggestion](https://github.com/xiaowu0162/LongMemEval/issues/31).
+- 🔒 **v9.2.7 — Security Hardening:** Typed `PrototypePollutionError` class (replaces generic `Error` in `sanitizeForMerge()` — enables catch-site discrimination and forensic logging with `offendingKey`), explicit null-byte path injection guard in `SafetyController.validateActionsInScope()` (C-string truncation attack vector), and corrected CRDT merge semantics documentation (Remove-Wins-from-Either, not Add-Wins). 1055 tests, 0 regressions.
+- 🪟 **v9.2.6 — Windows CI Timeout Fix:** CLI integration tests timed out on Windows + Node 22.x GitHub Actions runners. Added `{ timeout: 30_000 }` to the describe block. 6 new residual distribution tests validating TurboQuant's QJL correction stability (zero R@5 delta between P50 and P95 residual vectors at d=128, 2K corpus).
+- 🔧 **v9.2.5 — Reconciliation Credential Probe Fix:** `supabaseReady` guard only resolved credentials when `requestedBackend === "supabase"`, causing reconciliation to silently skip. Added second credential probe for local + reconciliation path. Fixed Supabase schema mismatch on `key_context` column.
 - 🔄 **v9.2.4 — Cross-Backend Reconciliation:** Automatic two-layer sync from Supabase → SQLite on startup. When Claude Desktop writes handoffs and ledger entries to Supabase, Antigravity (local SQLite) now automatically detects stale data and pulls newer handoffs + the 20 most recent ledger entries. 5-second timeout prevents startup freeze. Targeted ID lookups (not full table scans) keep it safe for large databases. 13 tests including malformed JSON resilience, multi-role dedup, and timeout handling.
 - 🔧 **v9.2.3 — Code Review Hardening:** 10x faster split-brain detection (lightweight direct queries replace full `StorageBackend` construction), variable shadowing fix in CLI, resource leak fix in SQLite alternate client.
 - 🚨 **v9.2.2 — Critical: Split-Brain Detection & Prevention:** When multiple MCP clients use different storage backends (e.g., Claude Desktop → Supabase, Antigravity → SQLite), session state could silently diverge, causing agents to act on stale TODOs and outdated context. **New: `--storage` flag** on `prism load` CLI lets callers explicitly select which backend to read from. **New: Split-Brain Drift Detection** in `session_load_context` — compares active and alternate backend versions at load time and warns prominently when they diverge. Session loader script updated to respect `PRISM_STORAGE` environment variable.
@@ -817,7 +858,7 @@ Standard memory servers (like Mem0, Zep, or the baseline Anthropic MCP) act as p
 | **Storage Engine** | **BYO SQLite or Supabase** | Managed Cloud / VectorDBs | Managed Cloud / Postgres | Local SQLite only |
 | **Context Assembly** | **Progressive (Quick/Std/Deep)** | Top-K Semantic Search | Top-K + Temporal Summaries | Basic Entity Search |
 | **Memory Mechanics** | **ACT-R Activation, Spreading Activation, Hebbian Consolidation, Rejection Gate** | Basic Vector + Entity | Fading Temporal Graph | None (Infinite growth) |
-| **Multi-Agent Sync** | **CRDT (Add-Wins / LWW)** | Cloud locks | Postgres locks | ❌ None (Data races) |
+| **Multi-Agent Sync** | **CRDT (Remove-Wins / LWW)** | Cloud locks | Postgres locks | ❌ None (Data races) |
 | **Data Compression** | **TurboQuant (7x smaller vectors)** | ❌ Standard F32 Vectors | ❌ Standard Vectors | ❌ No Vectors |
 | **Observability** | **OTel Traces + Built-in PWA UI** | Cloud Dashboard | Cloud Dashboard | ❌ None |
 | **Maintenance** | **Autonomous Background Scheduler** | Manual/API driven | Automated (Cloud) | ❌ Manual |
@@ -1188,8 +1229,16 @@ Prism has evolved from smart session logging into a **cognitive memory architect
 | **v7.8** | Multi-Hop Causal Reasoning — spreading activation traverses `caused_by`/`led_to` edges with damped fan effect (`1/ln(fan+e)`) and lateral inhibition | ACT-R spreading activation (Anderson), Collins & Loftus (1975) | ✅ Shipped |
 | **v7.8** | Uncertainty-Aware Rejection Gate — dual-signal (similarity floor + gap distance) safety layer prevents hallucination from low-confidence retrievals | Metacognition research, uncertainty quantification | ✅ Shipped |
 | **v7.8** | Dynamic Fast Weight Decay — `is_rollup` semantic nodes decay 50% slower (`ageModifier = 0.5`) than episodic entries, creating Long-Term Context anchors | ACT-R base-level activation with differential decay rates | ✅ Shipped |
-| **v7.x** | Affect-Tagged Memory — sentiment shapes what gets recalled | Affect-modulated retrieval (neuroscience) | 🔭 Horizon |
-| **v8+** | Zero-Search Retrieval — no index, no ANN, just ask the vector | Holographic Reduced Representations | 🔭 Horizon |
+| **v9.0** | Affect-Tagged Memory — valence-scored retrieval where `\|valence\|` boosts ranking; UX warnings surface historically negative topics | Affect-modulated retrieval (neuroscience), somatic marker hypothesis | ✅ Shipped |
+| **v9.0** | Surprisal Gate — vector-based novelty pricing: high-surprisal saves cost 0.5× tokens, low-surprisal 2.0×; forces LLM data compression | Information-theoretic surprisal (Shannon), predictive coding | ✅ Shipped |
+| **v9.0** | Cognitive Budget — per-project token economy with passive UBI recovery (+100 tokens/hr); agents that over-save enter Cognitive Debt | Resource-bounded rationality (Simon, 1955) | ✅ Shipped |
+| **v9.1** | Task Router v2 — 6-signal weighted heuristic engine routing tasks between cloud host and local LLM based on file-type complexity, scope, and multi-step detection | Heuristic classification, cognitive load theory | ✅ Shipped |
+| **v9.2** | Cross-Backend Reconciliation — automatic Supabase → SQLite sync with idempotent dedup and 5s timeout | Eventual consistency, crdt-style reconciliation | ✅ Shipped |
+| **v9.2** | Split-Brain Drift Detection — dual-backend version comparison with prominent divergence warnings at load time | Byzantine fault detection, split-brain resolution | ✅ Shipped |
+| **v9.2** | TurboQuant QJL Validation — zero R@5 delta between P50 and P95 residual vectors (d=128, N=2K); CV=0.038 at d=768 proves no long tail | QJL estimator (ICLR 2026), Householder orthogonal rotation | ✅ Shipped |
+| **v9.2** | Typed Security Errors — `PrototypePollutionError` with `offendingKey` for forensic logging; null-byte path injection guard in SafetyController | Defense-in-depth (NIST), C-string truncation attack mitigation | ✅ Shipped |
+| **v9.3** | ResidualNorm Tiebreaker — within-ε candidates ranked by compression fidelity (`PRISM_TURBOQUANT_TIEBREAKER_EPSILON`); +2pp R@1, +1pp R@5 at ε=0.005 | Quantization confidence scoring, compression-aware retrieval | ✅ Shipped |
+| **v10+** | Zero-Search Retrieval — no index, no ANN, just ask the vector | Holographic Reduced Representations | 🔭 Horizon |
 
 > Informed by Anderson's ACT-R (Adaptive Control of Thought—Rational), Collins & Loftus spreading activation networks (1975), Kanerva's SDM (1988), Hebb's learning rule, and LeCun's "Why AI Systems Don't Learn" (Dupoux, LeCun, Malik).
 
@@ -1221,7 +1270,7 @@ Prism MCP is open-source and free for individual developers. For teams and enter
 
 ## 📦 Milestones & Roadmap
 
-> **Current: v9.2.4** — Cross-Backend Reconciliation ([CHANGELOG](CHANGELOG.md))
+> **Current: v9.3.0** — TurboQuant ResidualNorm Tiebreaker ([CHANGELOG](CHANGELOG.md))
 
 | Release | Headline |
 |---------|----------|

package/dist/config.js

@@ -268,3 +268,17 @@ export const PRISM_DARK_FACTORY_ENABLED = process.env.PRISM_DARK_FACTORY_ENABLED
 export const PRISM_DARK_FACTORY_POLL_MS = parseInt(process.env.PRISM_DARK_FACTORY_POLL_MS || "30000", 10);
 /** Default max wall-clock time per pipeline (ms). Default: 15 minutes. */
 export const PRISM_DARK_FACTORY_MAX_RUNTIME_MS = parseInt(process.env.PRISM_DARK_FACTORY_MAX_RUNTIME_MS || "900000", 10);
+// ─── v9.3: TurboQuant ResidualNorm Tiebreaker ─────────────────
+// When two compressed cosine scores are within ε of each other,
+// prefer the candidate with lower residualNorm (its compressed
+// representation captured more signal energy, making its score
+// more trustworthy). Empirically validated: ε=0.005 gives +2pp
+// R@1, +1pp R@5 on random d=128 vectors. Set to 0 to disable.
+//
+// Only affects Tier-2 TurboQuant JS-side search (both SQLite and
+// Supabase backends). Tier-1 native vector search is unaffected.
+/** Tiebreaker threshold for TurboQuant Tier-2 ranking. 0 = disabled (default). */
+const rawTiebreakerEpsilon = parseFloat(process.env.PRISM_TURBOQUANT_TIEBREAKER_EPSILON || "0");
+export const PRISM_TURBOQUANT_TIEBREAKER_EPSILON = Number.isFinite(rawTiebreakerEpsilon) && rawTiebreakerEpsilon >= 0
+    ? rawTiebreakerEpsilon
+    : 0;

package/dist/darkfactory/safetyController.js

@@ -84,6 +84,12 @@ export class SafetyController {
             if (!action.targetPath || typeof action.targetPath !== 'string' || action.targetPath.trim() === '') {
                 return `Action[${i}]: targetPath is empty or missing`;
             }
+            // Null-byte injection guard: C-string truncation attack vector.
+            // A path like "src/\0../../etc/passwd" would be truncated at the null byte
+            // by native fs syscalls, potentially resolving to an unintended location.
+            if (action.targetPath.includes('\0')) {
+                return `Action[${i}]: targetPath contains null byte (injection attempt)`;
+            }
             // Resolve targetPath relative to workingDirectory for scope check
             const resolvedTarget = spec.workingDirectory
                 ? path.resolve(spec.workingDirectory, action.targetPath)

package/dist/storage/sqlite.js

@@ -1594,6 +1594,9 @@ export class SqliteStorage {
         `;
                 const fallbackResult = await this.db.execute({ sql: fallbackSql, args: fallbackArgs });
                 // Score each entry using asymmetric cosine similarity
+                // Track residualNorm for optional tiebreaker (v9.3)
+                const { PRISM_TURBOQUANT_TIEBREAKER_EPSILON } = await import("../config.js");
+                const eps = PRISM_TURBOQUANT_TIEBREAKER_EPSILON;
                 const scored = [];
                 for (const row of fallbackResult.rows) {
                     try {
@@ -1613,6 +1616,7 @@ export class SqliteStorage {
                                 is_rollup: Boolean(row.is_rollup),
                                 importance: row.importance ?? 0,
                                 last_accessed_at: row.last_accessed_at || null,
+                                _residualNorm: eps > 0 ? compressed.residualNorm : undefined,
                             });
                         }
                     }
@@ -1620,9 +1624,20 @@ export class SqliteStorage {
                         // Skip entries with corrupt compressed data
                     }
                 }
-                // Sort by similarity descending and limit
-                scored.sort((a, b) => b.similarity - a.similarity);
+                // Sort by similarity descending, with optional residualNorm tiebreaker
+                // When ε > 0: candidates within ε of each other are ranked by lower
+                // residualNorm (its compressed representation is more trustworthy).
+                scored.sort((a, b) => {
+                    const diff = b.similarity - a.similarity;
+                    if (eps > 0 && Math.abs(diff) < eps && a._residualNorm != null && b._residualNorm != null) {
+                        return a._residualNorm - b._residualNorm;
+                    }
+                    return diff;
+                });
                 const baseResults = scored.slice(0, params.limit);
+                // Strip internal tiebreaker field before returning
+                for (const r of baseResults)
+                    delete r._residualNorm;
                 debugLog(`[SqliteStorage] Tier-2 TurboQuant fallback: scored ${fallbackResult.rows.length} entries, ` +
                     `${scored.length} above threshold`);
                 if (params.activation?.enabled) {

package/dist/storage/supabase.js

@@ -280,6 +280,9 @@ export class SupabaseStorage {
                     queryParams.role = `eq.${params.role}`;
                 const rows = await supabaseGet("session_ledger", queryParams);
                 const scored = [];
+                // v9.3: Import tiebreaker config for optional residualNorm ranking
+                const { PRISM_TURBOQUANT_TIEBREAKER_EPSILON } = await import("../config.js");
+                const eps = PRISM_TURBOQUANT_TIEBREAKER_EPSILON;
                 for (const row of (Array.isArray(rows) ? rows : [])) {
                     try {
                         const compressedBase64 = row.embedding_compressed;
@@ -295,6 +298,7 @@ export class SupabaseStorage {
                                 session_date: (row.session_date || row.created_at),
                                 decisions: Array.isArray(row.decisions) ? row.decisions : [],
                                 files_changed: Array.isArray(row.files_changed) ? row.files_changed : [],
+                                _residualNorm: eps > 0 ? compressed.residualNorm : undefined,
                             });
                         }
                     }
@@ -302,10 +306,21 @@ export class SupabaseStorage {
                         // Skip entries with corrupt compressed data
                     }
                 }
-                scored.sort((a, b) => b.similarity - a.similarity);
+                // Sort by similarity descending, with optional residualNorm tiebreaker
+                scored.sort((a, b) => {
+                    const diff = b.similarity - a.similarity;
+                    if (eps > 0 && Math.abs(diff) < eps && a._residualNorm != null && b._residualNorm != null) {
+                        return a._residualNorm - b._residualNorm;
+                    }
+                    return diff;
+                });
                 debugLog(`[SupabaseStorage] Tier-2 TurboQuant fallback: scored ${rows.length} entries, ` +
                     `${scored.length} above threshold`);
-                return scored.slice(0, params.limit);
+                const results = scored.slice(0, params.limit);
+                // Strip internal tiebreaker field before returning
+                for (const r of results)
+                    delete r._residualNorm;
+                return results;
             }
             catch (tier2Err) {
                 // Both tiers failed — return empty; caller falls through to FTS5

package/dist/utils/crdtMerge.js

@@ -40,13 +40,28 @@
 //
 // This is a zero-dependency, fast (~10ms for a typical handoff object)
 // solution appropriate for Prism's small merge surfaces.
+/**
+ * Typed error thrown when sanitizeForMerge() detects prototype pollution.
+ * Provides the offending key for forensic logging and distinct catch handling.
+ */
+export class PrototypePollutionError extends Error {
+    offendingKey;
+    constructor(key) {
+        super(`Security violation: prototype pollution attempt detected via key "${key}"`);
+        this.name = "PrototypePollutionError";
+        this.offendingKey = key;
+        if (Error.captureStackTrace) {
+            Error.captureStackTrace(this, PrototypePollutionError);
+        }
+    }
+}
 const FORBIDDEN_KEYS = new Set(["__proto__", "constructor", "prototype"]);
 function walkForForbiddenKeys(current) {
     if (!current || typeof current !== "object")
         return;
     for (const key of Object.keys(current)) {
         if (FORBIDDEN_KEYS.has(key)) {
-            throw new Error(`Security violation: prototype pollution attempt detected via key "${key}"`);
+            throw new PrototypePollutionError(key);
         }
         walkForForbiddenKeys(current[key]);
     }
@@ -63,18 +78,19 @@ export function sanitizeForMerge(obj) {
     walkForForbiddenKeys(obj);
     return JSON.parse(JSON.stringify(obj));
 }
-// ─── OR-Set Logic (Add-Wins) ────────────────────────────────────
+// ─── OR-Set Logic (Remove-Wins-from-Either) ────────────────────
 //
 // 3-way set merge:
 //   added_by_incoming = incoming - base
 //   removed_by_incoming = base - incoming
 //   added_by_current = current - base
 //   removed_by_current = base - current
-//   result = (base - removals) ∪ all_adds
+//   result = (base - all_removals) ∪ all_adds
 //
-// "Add-Wins" means: if Agent A removes "X" and Agent B adds "X",
-// the add wins. This is safe for TODOs (better to have a duplicate
-// than lose work) and keywords (idempotent).
+// SEMANTICS: Items removed by EITHER agent are dropped from the base.
+// Fresh additions from either agent are always preserved (union).
+// This means a removal by one agent wins over non-action by the other,
+// but cannot override a fresh add. Safe for TODOs and keywords.
 function mergeArray(b = [], i = [], c = []) {
     const bSet = new Set(b);
     const iSet = new Set(i);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "prism-mcp-server",
-  "version": "9.2.6",
+  "version": "9.3.0",
   "mcpName": "io.github.dcostenco/prism-mcp",
   "description": "The Mind Palace for AI Agents — a true Cognitive Architecture with Hebbian learning (episodic→semantic consolidation), ACT-R spreading activation (multi-hop causal reasoning), uncertainty-aware rejection gates (agents that know when they don't know), adversarial evaluation (anti-sycophancy), fail-closed Dark Factory pipelines, persistent memory (SQLite/Supabase), multi-agent Hivemind, time travel & visual dashboard. Zero-config local mode.",
   "module": "index.ts",