@jeremiaheth/neolata-mem 0.8.2 → 0.8.6

package/README.md

@@ -520,20 +520,57 @@ Decay Cycle:
 |---------|------------|------|-------|-----|
 | Zettelkasten linking | ✅ | ❌ | ❌ | ❌ |
 | Biological decay | ✅ | ❌ | ❌ | ❌ |
-| Graph traversal | ✅ | ❌ | ❌ | ✅ |
-| Multi-agent native | ✅ | ❌ | ❌ | ❌ |
-| Conflict resolution | ✅ | ✅ | ❌ | ❌ |
+| Graph traversal | ✅ | ❌ | ❌ | ✅ ¹ |
+| Multi-agent native | ✅ | ⚠️ ² | ⚠️ ³ | ❌ |
+| Conflict resolution | ✅ | ✅ | ❌ | ⚠️ ⁴ |
 | Quarantine lane | ✅ | ❌ | ❌ | ❌ |
-| Predicate schemas | ✅ | ❌ | ❌ | ❌ |
+| Predicate schemas | ✅ | ❌ | ❌ | ⚠️ ⁵ |
 | Runtime helpers (heartbeat/recall/dump) | ✅ | ❌ | ❌ | ❌ |
 | Explainability API | ✅ | ❌ | ❌ | ❌ |
-| Episodes & compression | ✅ | ❌ | ❌ | ❌ |
+| Episodes & compression | ✅ | ❌ | ❌ | ⚠️ ⁶ |
 | Labeled clusters | ✅ | ❌ | ❌ | ❌ |
 | Works offline | ✅ | ✅ | ✅ | ❌ |
-| No Python needed | ✅ | ❌ | ❌ | ❌ |
+| No Python needed | ✅ | ❌ ⁷ | ❌ | ❌ |
 | Zero-config start | ✅ | ❌ | ❌ | ❌ |
 | LLM optional | ✅ | ❌ | ❌ | ❌ |
 
+<details>
+<summary><b>Comparison notes</b> (click to expand)</summary>
+
+1. **Zep graph traversal** — Zep uses a temporal knowledge graph with entity/relation extraction. Different architecture from neolata-mem's Zettelkasten-style memory-to-memory links, but supports graph queries.
+2. **Mem0 multi-agent** — Mem0 supports User, Session, and Agent memory levels. Agent-scoped storage exists, but it's designed around user personalization rather than independent agent collaboration.
+3. **Letta multi-agent** — Letta supports shared memory blocks between agents and has a multi-agent orchestration layer. Not the same as neolata-mem's native agent-scoped store/search.
+4. **Zep conflict resolution** — Zep tracks temporal validity of facts (marking outdated facts as invalid when new info arrives). This is implicit conflict handling, not explicit quarantine/review like neolata-mem.
+5. **Zep predicate schemas** — Zep supports custom graph ontologies via Pydantic/Zod for domain-specific entity types. Different from neolata-mem's per-predicate conflict/dedup/normalization rules.
+6. **Zep episodes** — Zep structures interactions into episodic sequences with temporal awareness. neolata-mem episodes are explicitly named groups with time-window capture, compression, and search.
+7. **Mem0 npm package** — Mem0 offers an npm SDK (`mem0ai`), but the core engine and self-hosted deployment require Python.
+
+*Last updated: February 2026. Comparison based on publicly available documentation.*
+
+</details>
+
+## Roadmap
+
+### v0.9 — Production Hardening (Q1 2026)
+- [ ] **Supabase storage adapter** — first-class `storage: { type: 'supabase' }` config (currently requires custom wiring)
+- [ ] **Write-ahead log** — crash recovery for JSON storage beyond atomic writes
+- [ ] **Streaming search** — async iterator for large result sets
+- [ ] **Memory import/export** — portable JSON/JSONL format for backup and migration
+- [ ] **Benchmarks** — LOCOMO and LongMemEval comparison against Mem0, Zep, and raw vector stores
+
+### v1.0 — Stable API (Q2 2026)
+- [ ] **API stabilization** — semantic versioning contract, deprecation policy
+- [ ] **Plugin system** — bring-your-own storage/embeddings/LLM via a documented interface (beyond current provider pattern)
+- [ ] **Web dashboard** — visual graph explorer, memory inspector, health monitor
+- [ ] **Multi-backend sync** — replicate between JSON ↔ Supabase ↔ custom backends
+- [ ] **Access control** — per-agent read/write permissions for shared deployments
+
+### Future
+- [ ] **Real-time subscriptions** — watch for memory changes (useful for multi-agent coordination)
+- [ ] **Temporal queries** — "what did agent X know about Y on date Z?"
+- [ ] **Federation** — cross-instance memory sharing with trust boundaries
+- [ ] **Browser SDK** — lightweight client for web apps (IndexedDB storage)
+
 ## Security
 
 neolata-mem includes several hardening measures:

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@jeremiaheth/neolata-mem",
-  "version": "0.8.2",
+  "version": "0.8.6",
   "description": "Trustworthy graph-native memory engine for AI agents - belief updates, provenance tracking, trust-gated supersession, and poisoning resistance",
   "type": "module",
   "main": "src/index.mjs",
@@ -9,9 +9,14 @@
     "./graph": "./src/graph.mjs",
     "./embeddings": "./src/embeddings.mjs",
     "./storage": "./src/storage.mjs",
+    "./wal": "./src/wal.mjs",
+    "./wal-replay": "./src/wal-replay.mjs",
+    "./wal-snapshot": "./src/wal-snapshot.mjs",
+    "./wal-equivalence": "./src/wal-equivalence.mjs",
     "./supabase-storage": "./src/supabase-storage.mjs",
     "./extraction": "./src/extraction.mjs",
-    "./llm": "./src/llm.mjs"
+    "./llm": "./src/llm.mjs",
+    "./wal-recovery": "./src/wal-recovery.mjs"
   },
   "bin": {
     "neolata-mem": "./cli/index.mjs"
@@ -19,14 +24,18 @@
   "files": [
     "src/",
     "cli/",
-    "docs/*.md",
+    "docs/guide.md",
+    "docs/implementation-notes.md",
+    "docs/runtime-helpers.md",
     "LICENSE",
     "README.md"
   ],
   "scripts": {
     "test": "vitest run",
+    "validate:wal": "node --preserve-symlinks --preserve-symlinks-main scripts/validate-wal.mjs",
     "verify-skill": "node scripts/verify-skill.mjs",
-    "prepublishOnly": "vitest run && node scripts/verify-skill.mjs"
+    "prepublishOnly": "vitest run && node scripts/verify-skill.mjs",
+    "validate:recovery": "node --preserve-symlinks --preserve-symlinks-main scripts/validate-recovery.mjs"
   },
   "keywords": [
     "ai",
@@ -52,7 +61,7 @@
   "homepage": "https://github.com/Jeremiaheth/neolata-mem#readme",
   "repository": {
     "type": "git",
-    "url": "https://github.com/Jeremiaheth/neolata-mem"
+    "url": "git+https://github.com/Jeremiaheth/neolata-mem.git"
   },
   "engines": {
     "node": ">=18.0.0"

package/src/graph.mjs

@@ -14,7 +14,8 @@
  *   - extraction: extract(text) → facts (optional)
  */
 
-import { cosineSimilarity } from './embeddings.mjs';
+import { cosineSimilarity } from './embeddings.mjs';
+import { createWalMutationEvent } from './wal.mjs';
 
 /** @typedef {{ id: string, agent: string, memory: string, category: string, importance: number, tags: string[], embedding: number[]|null, links: {id: string, similarity: number, type?: string}[], created_at: string, updated_at: string, evolution?: object[], accessCount?: number }} Memory */
 
@@ -207,11 +208,12 @@ export class MemoryGraph {
    * @param {number} [opts.config.archiveThreshold=0.15] - Strength below this → archive
    * @param {number} [opts.config.deleteThreshold=0.05] - Strength below this → delete
    */
-  constructor({ storage, embeddings, extraction, llm, config = {} }) {
-    this.storage = storage;
-    this.embeddings = embeddings;
-    this.extraction = extraction || null;
-    this.llm = llm || null;
+  constructor({ storage, embeddings, extraction, llm, wal = null, config = {} }) {
+    this.storage = storage;
+    this.embeddings = embeddings;
+    this.extraction = extraction || null;
+    this.llm = llm || null;
+    this.wal = wal;
     this.memories = [];
     this.loaded = false;
     this._listeners = {};
@@ -314,11 +316,24 @@ export class MemoryGraph {
   }
 
   /** Emit an event to all registered listeners. */
-  emit(event, data) {
-    for (const fn of (this._listeners[event] || [])) {
-      try { fn(data); } catch { /* listener errors don't break the engine */ }
-    }
-  }
+  emit(event, data) {
+    for (const fn of (this._listeners[event] || [])) {
+      try { fn(data); } catch { /* listener errors don't break the engine */ }
+    }
+  }
+
+  async _appendWal(op, { memoryId, actor = null, data = {} } = {}) {
+    if (!this.wal) return;
+    if (typeof this.wal.appendMutation === 'function') {
+      await this.wal.appendMutation({ op, memoryId, actor, data });
+      return;
+    }
+    if (typeof this.wal.append === 'function') {
+      await this.wal.append(createWalMutationEvent({ op, memoryId, actor, data }));
+      return;
+    }
+    throw new Error('wal backend must implement appendMutation() or append()');
+  }
 
   /** Load memories from storage (lazy, called once). */
   async init() {
@@ -697,10 +712,20 @@ export class MemoryGraph {
     } else {
       await this.save();
     }
-    if (pendingConflictsChanged) await this._savePendingConflicts();
-
-    // Emit store event
-    this.emit('store', { id, agent, content: text, category, importance, links: topLinks.length });
+    if (pendingConflictsChanged) await this._savePendingConflicts();
+    await this._appendWal('store', {
+      memoryId: id,
+      actor: agent,
+      data: {
+        category,
+        importance,
+        status: newMem.status,
+        links: topLinks.length,
+      },
+    });
+
+    // Emit store event
+    this.emit('store', { id, agent, content: text, category, importance, links: topLinks.length });
 
     return {
       id,
@@ -1824,13 +1849,23 @@ export class MemoryGraph {
 
     mem.updated_at = now.toISOString();
 
-    if (this.storage.incremental) {
-      await this.storage.upsert(mem);
-    } else {
-      await this.save();
-    }
-
-    const { strength } = this.calcStrength(mem);
+    if (this.storage.incremental) {
+      await this.storage.upsert(mem);
+    } else {
+      await this.save();
+    }
+    await this._appendWal('reinforce', {
+      memoryId: mem.id,
+      actor: mem.agent || null,
+      data: {
+        boost,
+        importance: mem.importance,
+        accessCount: mem.accessCount,
+        reinforcements: mem.reinforcements,
+      },
+    });
+
+    const { strength } = this.calcStrength(mem);
     return {
       id: mem.id, memory: mem.memory, oldImportance, newImportance: mem.importance,
       accessCount: mem.accessCount, reinforcements: mem.reinforcements,
@@ -2027,10 +2062,20 @@ Respond ONLY with a JSON object:
 
     mem.updated_at = new Date().toISOString();
 
-    if (this.storage.incremental) await this.storage.upsert(mem);
-    else await this.save();
-
-    this.emit('dispute', { id: memoryId, disputes: mem.disputes, trust: mem.provenance.trust, status: mem.status, reason });
+    if (this.storage.incremental) await this.storage.upsert(mem);
+    else await this.save();
+    await this._appendWal('dispute', {
+      memoryId,
+      actor: mem.agent || null,
+      data: {
+        disputes: mem.disputes,
+        trust: +mem.provenance.trust.toFixed(4),
+        status: mem.status,
+        reason: reason || null,
+      },
+    });
+
+    this.emit('dispute', { id: memoryId, disputes: mem.disputes, trust: mem.provenance.trust, status: mem.status, reason });
     return {
       id: memoryId, disputes: mem.disputes, trust: +mem.provenance.trust.toFixed(4),
       confidence: mem.confidence, status: mem.status,
@@ -2482,12 +2527,21 @@ Respond ONLY with a JSON object:
     const mem = this._byId(memoryId);
     if (!mem) throw new Error(`Memory not found: ${memoryId}`);
     if (mem.status !== 'active') throw new Error('Only active memories can be quarantined');
-    this._markQuarantined(mem, { reason, details });
-    mem.updated_at = new Date().toISOString();
-    if (this.storage.incremental) await this.storage.upsert(mem);
-    else await this.save();
-    return { id: mem.id, status: mem.status, quarantine: mem.quarantine };
-  }
+    this._markQuarantined(mem, { reason, details });
+    mem.updated_at = new Date().toISOString();
+    if (this.storage.incremental) await this.storage.upsert(mem);
+    else await this.save();
+    await this._appendWal('quarantine', {
+      memoryId: mem.id,
+      actor: mem.agent || null,
+      data: {
+        reason,
+        details: details || null,
+        status: mem.status,
+      },
+    });
+    return { id: mem.id, status: mem.status, quarantine: mem.quarantine };
+  }
 
   async reviewQuarantine(memoryId, { action, reason } = {}) {
     await this.init();

package/src/index.mjs

@@ -19,10 +19,11 @@
 
 import { MemoryGraph } from './graph.mjs';
 import { openaiEmbeddings, noopEmbeddings } from './embeddings.mjs';
-import { jsonStorage, memoryStorage } from './storage.mjs';
-import { supabaseStorage } from './supabase-storage.mjs';
-import { llmExtraction, passthroughExtraction } from './extraction.mjs';
-import { openaiChat, openclawChat } from './llm.mjs';
+import { jsonStorage, memoryStorage } from './storage.mjs';
+import { supabaseStorage } from './supabase-storage.mjs';
+import { jsonlWal } from './wal.mjs';
+import { llmExtraction, passthroughExtraction } from './extraction.mjs';
+import { openaiChat, openclawChat } from './llm.mjs';
 
 /**
  * Create a configured MemoryGraph instance.
@@ -130,8 +131,8 @@ export function createMemory(opts = {}) {
   }
 
   // LLM (for conflict resolution)
-  let llm = null;
-  const llmOpts = opts.llm || {};
+  let llm = null;
+  const llmOpts = opts.llm || {};
   if (llmOpts.type === 'openai' && llmOpts.apiKey) {
     llm = openaiChat({
       apiKey: llmOpts.apiKey,
@@ -144,16 +145,27 @@ export function createMemory(opts = {}) {
       port: llmOpts.port,
       token: llmOpts.token,
     });
-  }
-
-  return new MemoryGraph({
-    storage,
-    embeddings,
-    extraction,
-    llm,
-    config: {
-      ...(opts.graph || {}),
-      ...(opts.predicateSchemas !== undefined ? { predicateSchemas: opts.predicateSchemas } : {}),
+  }
+
+  // Optional local WAL for mutation logging
+  let wal = null;
+  const walOpts = opts.wal;
+  if (walOpts && walOpts.enabled !== false) {
+    wal = jsonlWal({
+      dir: walOpts.dir,
+      filename: walOpts.filename,
+    });
+  }
+
+  return new MemoryGraph({
+    storage,
+    embeddings,
+    extraction,
+    llm,
+    wal,
+    config: {
+      ...(opts.graph || {}),
+      ...(opts.predicateSchemas !== undefined ? { predicateSchemas: opts.predicateSchemas } : {}),
     },
   });
 }
@@ -161,10 +173,34 @@ export function createMemory(opts = {}) {
 // Re-export everything for advanced usage
 export { MemoryGraph, tokenize, computeTrust, computeConfidence, estimateTokens, normalizeClaim } from './graph.mjs';
 export { openaiEmbeddings, noopEmbeddings, cosineSimilarity } from './embeddings.mjs';
-export { jsonStorage, memoryStorage } from './storage.mjs';
-export { supabaseStorage } from './supabase-storage.mjs';
-export { markdownWritethrough, webhookWritethrough } from './writethrough.mjs';
+export { jsonStorage, memoryStorage } from './storage.mjs';
+export { supabaseStorage } from './supabase-storage.mjs';
+export { jsonlWal, createWalMutationEvent, validateWalMutationEvent, WAL_EVENT_VERSION, WAL_MUTATION_OPS } from './wal.mjs';
+export { buildWalReplayTimeline, replayMutationSubset, readAndReplayWal } from './wal-replay.mjs';
+export {
+  WAL_SNAPSHOT_VERSION,
+  WAL_SNAPSHOT_KIND,
+  WAL_SNAPSHOT_FIELDS,
+  validateWalSnapshot,
+  createWalSnapshot,
+  selectWalEventsAfterSnapshot,
+  replaySnapshotAndWal,
+} from './wal-snapshot.mjs';
+export {
+  projectLiveMutationSubsetState,
+  compareMutationSubsetStates,
+  compareLiveStateToSnapshotReplay,
+} from './wal-equivalence.mjs';
+export {
+  WAL_RECOVERY_MANIFEST_VERSION,
+  buildLocalArtifactManifest,
+  verifyLocalArtifacts,
+  validateRecoveryDryRun,
+  formatRecoveryCheckReport,
+} from './wal-recovery.mjs';
+export { markdownWritethrough, webhookWritethrough } from './writethrough.mjs';
 export { llmExtraction, passthroughExtraction } from './extraction.mjs';
 export { openaiChat, openclawChat } from './llm.mjs';
 export { validateBaseUrl } from './validate.mjs';
 export { detectKeyMoments, heartbeatStore, extractTopicSlug, contextualRecall, preCompactionDump } from './runtime.mjs';
+

package/src/supabase-storage.mjs

@@ -490,13 +490,18 @@ export function supabaseStorage({
     async upsertLinks(sourceId, links) {
       assertUUID(sourceId, 'upsertLinks: sourceId');
       if (!links.length) return;
-      const rows = links.map(l => ({
-        id: randomUUID(),
-        source_id: sourceId,
-        target_id: l.id,
-        strength: l.similarity,
-        created_at: new Date().toISOString(),
-      }));
+      // Normalize to canonical direction (sorted pair) to prevent bidirectional dupes.
+      // loadLinks() already treats each row as bidirectional, so one row per pair suffices.
+      const rows = links.map(l => {
+        const [a, b] = [sourceId, l.id].sort();
+        return {
+          id: randomUUID(),
+          source_id: a,
+          target_id: b,
+          strength: l.similarity,
+          created_at: new Date().toISOString(),
+        };
+      });
       await request('POST', `/rest/v1/${linksTable}`, rows, {
         'Prefer': 'return=minimal,resolution=merge-duplicates',
       });

package/src/wal-equivalence.mjs

@@ -0,0 +1,128 @@
+import { normalizeReplayState } from './wal-replay.mjs';
+import { replaySnapshotAndWal } from './wal-snapshot.mjs';
+
+function roundTrust(value) {
+  return typeof value === 'number' && Number.isFinite(value) ? +value.toFixed(4) : null;
+}
+
+function normalizeLiveMemory(memory = {}) {
+  const status = typeof memory.status === 'string' && memory.status ? memory.status : 'active';
+  const disputes = typeof memory.disputes === 'number' ? Math.max(0, Math.trunc(memory.disputes)) : 0;
+  return {
+    id: memory.id,
+    hasStore: true,
+    status,
+    category: typeof memory.category === 'string' ? memory.category : null,
+    importance:
+      typeof memory.importance === 'number' && Number.isFinite(memory.importance)
+        ? memory.importance
+        : null,
+    links: Array.isArray(memory.links) ? memory.links.length : 0,
+    reinforcements:
+      typeof memory.reinforcements === 'number' && Number.isFinite(memory.reinforcements)
+        ? Math.max(0, Math.trunc(memory.reinforcements))
+        : 0,
+    disputes,
+    // Narrow-scope replay only carries trust via dispute events in current WAL subset.
+    trust: disputes > 0 ? roundTrust(memory?.provenance?.trust) : null,
+    accessCount:
+      typeof memory.accessCount === 'number' && Number.isFinite(memory.accessCount)
+        ? Math.max(0, Math.trunc(memory.accessCount))
+        : 0,
+    lastActor: typeof memory.agent === 'string' ? memory.agent : null,
+    lastAt:
+      typeof memory.updated_at === 'string'
+        ? memory.updated_at
+        : (typeof memory.created_at === 'string' ? memory.created_at : null),
+    quarantine: status === 'quarantined'
+      ? {
+        reason: memory?.quarantine?.reason ?? null,
+        details: memory?.quarantine?.details ?? null,
+      }
+      : null,
+  };
+}
+
+export function projectLiveMutationSubsetState(memories = []) {
+  if (!Array.isArray(memories)) {
+    throw new Error('memories must be an array');
+  }
+  const byMemoryId = {};
+  const ordered = [...memories]
+    .filter((memory) => memory && typeof memory.id === 'string' && memory.id)
+    .sort((a, b) => a.id.localeCompare(b.id));
+  for (const memory of ordered) {
+    byMemoryId[memory.id] = normalizeLiveMemory(memory);
+  }
+  return normalizeReplayState({ byMemoryId, applied: 0 });
+}
+
+function comparableRecord(record = {}, { includeTemporal = false } = {}) {
+  return {
+    id: record.id,
+    hasStore: record.hasStore,
+    status: record.status,
+    category: record.category,
+    importance: record.importance,
+    links: record.links,
+    reinforcements: record.reinforcements,
+    disputes: record.disputes,
+    trust: record.trust,
+    accessCount: record.accessCount,
+    lastActor: record.lastActor,
+    ...(includeTemporal ? { lastAt: record.lastAt } : {}),
+    quarantine: record.quarantine
+      ? { reason: record.quarantine.reason ?? null, details: record.quarantine.details ?? null }
+      : null,
+  };
+}
+
+export function compareMutationSubsetStates(liveState, rebuiltState, { includeTemporal = false } = {}) {
+  const live = normalizeReplayState(liveState);
+  const rebuilt = normalizeReplayState(rebuiltState);
+
+  const memoryIds = [...new Set([
+    ...Object.keys(live.byMemoryId || {}),
+    ...Object.keys(rebuilt.byMemoryId || {}),
+  ])].sort();
+
+  const differences = [];
+  for (const memoryId of memoryIds) {
+    const left = comparableRecord(live.byMemoryId[memoryId] || {}, { includeTemporal });
+    const right = comparableRecord(rebuilt.byMemoryId[memoryId] || {}, { includeTemporal });
+    const fields = Object.keys({ ...left, ...right });
+    for (const field of fields) {
+      const l = left[field];
+      const r = right[field];
+      if (JSON.stringify(l) !== JSON.stringify(r)) {
+        differences.push({ memoryId, field, live: l, rebuilt: r });
+      }
+    }
+  }
+
+  return {
+    equivalent: differences.length === 0,
+    compared: memoryIds.length,
+    differences,
+  };
+}
+
+export function compareLiveStateToSnapshotReplay({
+  liveMemories = [],
+  snapshot,
+  walEvents = [],
+  malformed = [],
+  includeTemporal = false,
+} = {}) {
+  const liveState = projectLiveMutationSubsetState(liveMemories);
+  const rebuilt = replaySnapshotAndWal(snapshot, walEvents, { malformed });
+  const rebuiltState = normalizeReplayState(rebuilt);
+  const comparison = compareMutationSubsetStates(liveState, rebuiltState, { includeTemporal });
+
+  return {
+    liveState,
+    rebuiltState,
+    comparison,
+    replay: rebuilt,
+  };
+}