@memstack/core 0.6.4 → 0.7.1

package/README.md

@@ -1,9 +1,13 @@
 # MemStack
 
+> Implementation priority is maintained in the [canonical roadmap](docs/ROADMAP.md).
+
 > The open-source memory layer for AI agents — store, retrieve, summarize, and prune.
 
 [![npm version](https://img.shields.io/npm/v/@memstack/core)](https://www.npmjs.com/package/@memstack/core)
+[![CI](https://github.com/isiomaC/memstack/actions/workflows/ci.yml/badge.svg)](https://github.com/isiomaC/memstack/actions/workflows/ci.yml)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![MCP Reference](https://img.shields.io/badge/MCP-LLM%20Reference-blue)](https://gitmcp.io/isiomaC/memstack)
 
 ```bash
 npm install @memstack/core
@@ -671,9 +675,16 @@ new OpenAIEmbeddingAdapter({ apiKey: "...", baseURL: "https://api.voyageai.com/v
 
 ### Storage Adapters
 
-MemStack ships with **11 production-ready storage adapters (7 experimental)** — every major backend, zero peer dependencies, all client-injected.
+MemStack contains 18 storage-adapter implementations. Twelve are exported from `@memstack/core`; six remain experimental source implementations. Core has no runtime dependencies, and database clients are injected by callers.
+
+**Support levels:**
 
-### Production (e2e verified against real instances)
+- **Production-ready** means exported from the public package, covered by unit tests, and supported as part of the public API.
+- **Real-service E2E verified** means the adapter also passes against its actual database implementation in `pnpm test:e2e`.
+- **Mock-tested** means unit coverage uses an injected fake client rather than a live cloud service.
+- **Experimental** means implemented in source but not exported from the published package.
+
+### Public package exports
 
 **Built-in (zero external deps):**
 | Adapter | Backend | Use case |
@@ -687,6 +698,7 @@ MemStack ships with **11 production-ready storage adapters (7 experimental)**
 | Adapter | Backend | Vector search |
 |---|---|---|
 | `PostgresStorageAdapter` | PostgreSQL + pgvector | HNSW native |
+| `SQLiteStorageAdapter` | SQLite (better-sqlite3) | Cosine in-memory |
 
 **Vector databases:**
 | Adapter | Backend |
@@ -706,13 +718,12 @@ MemStack ships with **11 production-ready storage adapters (7 experimental)**
 |---|---|
 | `Neo4jStorageAdapter` | Neo4j |
 
-### Experimental (mock-tested, blocked by cloud deps or platform constraints)
+### Experimental (mock-tested or missing an optional E2E capability)
 
-Available via direct source import. Not yet in the barrel export — uncomment in `src/index.ts` when e2e verified.
+These implementations are available to source contributors but are not part of the published package API.
 
 | Adapter | Backend | Blocker |
 |---|---|---|
-| `SQLiteStorageAdapter` | SQLite (better-sqlite3) | Native binary for Node 24 |
 | `TursoStorageAdapter` | Turso (libsql) | Cloud-only (needs Turso account) |
 | `ChromaStorageAdapter` | ChromaDB | Embedding function dependency |
 | `PineconeStorageAdapter` | Pinecone | Cloud-only (needs API key) |
@@ -720,7 +731,7 @@ Available via direct source import. Not yet in the barrel export — uncomment i
 | `Mem0StorageAdapter` | Mem0 OSS or Cloud | Cloud-only (needs API key) |
 | `ZepStorageAdapter` | Zep Cloud or CE | Cloud-only (needs API key) |
 
-> **Direct import:** `import { ChromaStorageAdapter } from "@memstack/core/src/adapters/storage/chroma.js"`
+Live cloud compatibility remains unverified for Pinecone, Upstash, Mem0, Zep, and Turso. Chroma's real-client E2E suite is skipped when its optional default embedding function is unavailable. LLM and embedding-provider tests use mocks; live-provider testing is opt-in and is not part of CI.
 
 **Quick-start per backend:**
 
@@ -781,7 +792,6 @@ class MyStorage implements StorageProvider {
 | Neo4j | Neo4j vector index | No | ✅ Production |
 | Hybrid | Delegates to cache/durable | If durable supports | ✅ Production |
 | SQLite | Cosine in-memory | Yes | ✅ Production |
-| Hybrid | Delegates to cache/durable | If durable supports | ✅ Production |
 
 ---
 
@@ -797,18 +807,28 @@ const ms = new MemStack({
   embedding?: EmbeddingProvider,       // Optional — for semantic search
   storage?: StorageProvider,           // Optional — defaults to InMemoryStorageAdapter
   defaults?: {
-    summarizationThreshold?: number,   // Auto-summarize every N interactions. Default: 100
+    summarizationThreshold?: number,   // Auto-summarize every N process() calls. Default: 100
     embedOnStore?: boolean,            // Auto-embed on store(). Default: true
-    pruneStrategy?: PruneStrategy,     // Auto-prune on every store(). Default: disabled
+    pruneStrategy?: PruneStrategy,     // Auto-prune during process() (throttled). Default: disabled
+    pruneInterval?: number,            // Run auto-prune every N process() calls. Default: 100
+    autoImportance?: boolean,          // LLM-score importance in process() when not provided. Default: false
+    autoTags?: boolean,                // LLM-extract tags in process() when not provided. Default: false
+    summarizationPrompt?: string,      // Custom prompt for the summarizer
   },
   hooks?: {
     onMemoryStored?: (memory: Memory) => void;
     onMemoryPruned?: (ids: string[]) => void;
     onSummaryCreated?: (summary: Memory, deletedCount: number) => void;
+    onError?: (error: Error, context: string) => void;
   },
 });
 ```
 
+> **Auto-behaviors run inside `process()`, not `store()`.** `process()` tracks a
+> per-actor call count: summarization fires every `summarizationThreshold` calls,
+> and pruning fires every `pruneInterval` calls (when `pruneStrategy` is set).
+> `store()` is the low-level write and never triggers these.
+
 ### Memory Subsystem
 
 All methods accessible via `ms.memory.*`:
@@ -857,6 +877,8 @@ const data = JSON.parse(fs.readFileSync("state.json", "utf-8"));
 await ms2.import(data);
 ```
 
+Each memory's original `createdAt` is preserved on import, so `export` → `import` is a lossless round-trip — safe for backups and cross-backend migration (e.g. disk → Postgres). All storage adapters honor a `createdAt` supplied on `store()`/`storeBatch()`; when omitted, they default to the current time.
+
 ### Health & Close
 
 ```typescript
@@ -874,14 +896,17 @@ await ms.close(); // graceful shutdown
 const ms = new MemStack({
   llm: new OpenAILLMAdapter({ apiKey: "..." }),
 
-  // Defaults control auto-behavior
+  // Defaults control auto-behavior (all applied during process())
   defaults: {
-    summarizationThreshold: 50,      // Summarize every 50 interactions (default: 100)
+    summarizationThreshold: 50,      // Summarize every 50 process() calls (default: 100)
     embedOnStore: false,             // Don't auto-embed — saves API costs
-    pruneStrategy: {                 // Auto-clean on every store()
+    pruneStrategy: {                 // Auto-clean during process(), throttled by pruneInterval
       type: "byAge",
       maxAge: 90 * 86400000,         // 90 days
     },
+    pruneInterval: 100,              // Run the prune check every 100 process() calls (default: 100)
+    autoImportance: true,            // Let the LLM score importance when you don't pass one
+    autoTags: true,                  // Let the LLM extract tags when you don't pass any
   },
 
   // Hooks for observability
@@ -889,6 +914,7 @@ const ms = new MemStack({
     onMemoryStored: (m) => logger.debug("memory:stored", { id: m.id, actor: m.actorId }),
     onMemoryPruned: (ids) => logger.info("memory:pruned", { count: ids.length }),
     onSummaryCreated: (summary, n) => logger.info("memory:summarized", { count: n }),
+    onError: (err, context) => logger.error("memory:error", { context, message: err.message }),
   },
 });
 ```
@@ -948,13 +974,21 @@ git clone https://github.com/isiomaC/memstack.git
 cd memstack
 pnpm install
 
-pnpm test           # 393 tests, no external services needed
-pnpm test:e2e        # 82 E2E tests (requires Docker)
-pnpm test:watch     # Watch mode
-pnpm build          # CJS + ESM + type declarations
-pnpm check          # TypeScript type-check only
+pnpm test             # 407 core tests, no external services needed
+pnpm test:packages    # 78 package tests after dependency-ordered builds
+pnpm test:e2e         # 80 pass, 1 optional Chroma skip (requires Docker services)
+pnpm test:e2e:run     # Start services, run E2E once, preserve failure logs, clean up
+pnpm smoke:artifacts  # Built core, CLI, MCP, and server black-box checks
+pnpm smoke:packages   # Pack and install publishable tarballs in a clean project
+pnpm smoke:docker     # Build and exercise the server image
+pnpm verify           # Complete local verification pipeline
+pnpm test:watch       # Watch core tests
+pnpm build:all        # Build core and all workspace packages
+pnpm check:all        # Type-check core and all workspace packages
 ```
 
+CI exposes a stable `verification` job. Configure that job as a required status check in GitHub branch protection for `main`.
+
 ### Debugging
 
 Use hooks for observability — MemStack has no built-in logging:

package/dist/index.cjs

@@ -600,7 +600,7 @@ var MemoryStore = class {
   }
   async prune(strategy) {
     await this._ensureInit();
-    const allMemories = await this.storage.retrieve({ limit: this.limits.pruneScan });
+    const allMemories = await this.storage.retrieve({ actorId: strategy.actorId, limit: this.limits.pruneScan });
     const kept = this.pruner.execute(allMemories, strategy);
     const keptIds = new Set(kept.map((k) => k.id));
     const prunedIds = allMemories.filter((m) => !keptIds.has(m.id)).map((m) => m.id);
@@ -612,7 +612,7 @@ var MemoryStore = class {
   }
   async dryRunPrune(strategy) {
     await this._ensureInit();
-    const allMemories = await this.storage.retrieve({ limit: this.limits.pruneScan });
+    const allMemories = await this.storage.retrieve({ actorId: strategy.actorId, limit: this.limits.pruneScan });
     const kept = this.pruner.execute(allMemories, strategy);
     const keptIds = new Set(kept.map((k) => k.id));
     const wouldPrune = allMemories.filter((m) => !keptIds.has(m.id)).map((m) => m.id);
@@ -746,7 +746,7 @@ var InMemoryStorageAdapter = class {
       sourceId: input.sourceId,
       metadata: input.metadata ?? {},
       expiresAt: input.expiresAt,
-      createdAt: now,
+      createdAt: input.createdAt ?? now,
       _touchedAt: now
     };
     this.memories.set(memory.id, memory);
@@ -796,8 +796,11 @@ var InMemoryStorageAdapter = class {
       results = results.filter((m) => m.tags?.some((t) => query.tags.includes(t)));
     }
     if (query.query) {
-      const q = query.query.toLowerCase();
-      results = results.filter((m) => m.content.toLowerCase().includes(q));
+      const terms = query.query.toLowerCase().split(/\s+/).filter(Boolean);
+      results = results.filter((m) => {
+        const content = m.content.toLowerCase();
+        return terms.some((t) => content.includes(t));
+      });
     }
     if (query.createdAfter) {
       results = results.filter((m) => m.createdAt >= query.createdAfter);
@@ -983,7 +986,19 @@ var MemStack = class {
     if (snapshot.version !== 1) {
       throw validationError(`Unsupported snapshot version: ${snapshot.version}`);
     }
-    await this.memory.storeBatch(snapshot.memories);
+    const memories = snapshot.memories.map((m, i) => {
+      const createdAt = typeof m.createdAt === "string" ? new Date(m.createdAt) : m.createdAt;
+      const expiresAt = typeof m.expiresAt === "string" ? new Date(m.expiresAt) : m.expiresAt;
+      const label = m.id ? `id: ${m.id}` : `index ${i}`;
+      if (createdAt && isNaN(createdAt.getTime())) {
+        throw validationError(`Invalid createdAt for memory (${label}): ${String(m.createdAt)}`);
+      }
+      if (expiresAt && isNaN(expiresAt.getTime())) {
+        throw validationError(`Invalid expiresAt for memory (${label}): ${String(m.expiresAt)}`);
+      }
+      return { ...m, createdAt, expiresAt };
+    });
+    await this.memory.storeBatch(memories);
   }
   async health() {
     const status = { storage: false, llm: false, embedding: false };
@@ -1070,7 +1085,7 @@ var DiskStorageAdapter = class {
       sourceId: input.sourceId,
       metadata: input.metadata ?? {},
       expiresAt: input.expiresAt,
-      createdAt: now,
+      createdAt: input.createdAt ?? now,
       _touchedAt: now.toISOString()
     };
     await this._withWriteLock(input.actorId, async () => {
@@ -1103,7 +1118,7 @@ var DiskStorageAdapter = class {
         sourceId: input.sourceId,
         metadata: input.metadata ?? {},
         expiresAt: input.expiresAt,
-        createdAt: now,
+        createdAt: input.createdAt ?? now,
         _touchedAt: now.toISOString()
       };
       all.push(memory);
@@ -1210,8 +1225,11 @@ var DiskStorageAdapter = class {
         records = records.filter((r) => r.tags?.some((t) => query.tags.includes(t)));
       }
       if (query.query) {
-        const q = query.query.toLowerCase();
-        records = records.filter((r) => r.content.toLowerCase().includes(q));
+        const terms = query.query.toLowerCase().split(/\s+/).filter(Boolean);
+        records = records.filter((r) => {
+          const content = r.content.toLowerCase();
+          return terms.some((t) => content.includes(t));
+        });
       }
       results.push(...records);
     }
@@ -1302,9 +1320,14 @@ var DiskStorageAdapter = class {
     }
   }
   async _writeFile(actorId, records) {
+    const filePath = this._filePath(actorId);
+    const tmpPath = `${filePath}.${process.pid}.${Date.now().toString(36)}.tmp`;
     try {
-      await promises.writeFile(this._filePath(actorId), JSON.stringify(records, null, 2), "utf-8");
+      await promises.writeFile(tmpPath, JSON.stringify(records, null, 2), "utf-8");
+      await promises.rename(tmpPath, filePath);
     } catch (err) {
+      await promises.unlink(tmpPath).catch(() => {
+      });
       throw storageError(`Failed to write memories for actor ${actorId}: ${err.message}`);
     }
   }
@@ -1316,15 +1339,57 @@ var DiskStorageAdapter = class {
     });
     this._writeLocks.set(actorId, next);
     await prev;
+    const releaseFileLock = await this._acquireFileLock(actorId);
     try {
       await fn();
     } finally {
+      await releaseFileLock();
       resolve();
       if (this._writeLocks.get(actorId) === next) {
         this._writeLocks.delete(actorId);
       }
     }
   }
+  /**
+   * Acquires an exclusive cross-process lock for `actorId` via `open(path, "wx")`
+   * (atomic create-if-absent). Retries with jittered backoff while the lock is
+   * held elsewhere, and steals locks left behind by a crashed process once
+   * they're older than `staleMs`.
+   */
+  async _acquireFileLock(actorId) {
+    const lockPath = `${this._filePath(actorId)}.lock`;
+    const staleMs = 1e4;
+    const timeoutMs = 3e4;
+    const start = Date.now();
+    while (true) {
+      try {
+        const handle = await promises.open(lockPath, "wx");
+        await handle.close();
+        return async () => {
+          await promises.unlink(lockPath).catch(() => {
+          });
+        };
+      } catch (err) {
+        if (err.code !== "EEXIST") {
+          throw storageError(`Failed to acquire lock for actor ${actorId}: ${err.message}`);
+        }
+        try {
+          const st = await promises.stat(lockPath);
+          if (Date.now() - st.mtimeMs > staleMs) {
+            await promises.unlink(lockPath).catch(() => {
+            });
+            continue;
+          }
+        } catch {
+          continue;
+        }
+        if (Date.now() - start > timeoutMs) {
+          throw storageError(`Timed out waiting for disk storage lock: ${lockPath}`);
+        }
+        await new Promise((r) => setTimeout(r, 15 + Math.random() * 35));
+      }
+    }
+  }
   async _listActors() {
     try {
       const entries = await promises.readdir(this.dir);
@@ -1382,7 +1447,7 @@ var MarkdownStorageAdapter = class {
       sourceId: input.sourceId,
       metadata: input.metadata ?? {},
       expiresAt: input.expiresAt,
-      createdAt: now
+      createdAt: input.createdAt ?? now
     };
     const filePath = this._filePath(input.actorId);
     const block = this._formatBlock(memory);
@@ -1459,8 +1524,11 @@ var MarkdownStorageAdapter = class {
       results = results.filter((m) => m.tags?.some((t) => query.tags.includes(t)));
     }
     if (query.query) {
-      const q = query.query.toLowerCase();
-      results = results.filter((m) => m.content.toLowerCase().includes(q));
+      const terms = query.query.toLowerCase().split(/\s+/).filter(Boolean);
+      results = results.filter((m) => {
+        const content = m.content.toLowerCase();
+        return terms.some((t) => content.includes(t));
+      });
     }
     switch (query.strategy) {
       case "recent":
@@ -2011,7 +2079,7 @@ var PostgresStorageAdapter = class {
       sourceId: input.sourceId,
       metadata: input.metadata ?? {},
       expiresAt: input.expiresAt,
-      createdAt: new Date(now)
+      createdAt: input.createdAt ?? new Date(now)
     };
   }
   async storeBatch(inputs) {
@@ -2214,7 +2282,7 @@ var RedisStorageAdapter = class {
       sourceId: input.sourceId,
       metadata: input.metadata ?? {},
       expiresAt: input.expiresAt,
-      createdAt: now,
+      createdAt: input.createdAt ?? now,
       _touchedAt: now.toISOString()
     };
     await this.redis.set(this._memKey(memory.id), JSON.stringify(memory));
@@ -2238,7 +2306,7 @@ var RedisStorageAdapter = class {
         sourceId: input.sourceId,
         metadata: input.metadata ?? {},
         expiresAt: input.expiresAt,
-        createdAt: now,
+        createdAt: input.createdAt ?? now,
         _touchedAt: now.toISOString()
       };
       await this.redis.set(this._memKey(memory.id), JSON.stringify(memory));
@@ -2434,7 +2502,7 @@ var QdrantStorageAdapter = class {
       sourceId: input.sourceId,
       metadata: input.metadata ?? {},
       expiresAt: input.expiresAt,
-      createdAt: now
+      createdAt: input.createdAt ?? now
     };
     const qdrantId = crypto.randomUUID();
     await this.client.upsert(this.collectionName, {
@@ -2466,7 +2534,7 @@ var QdrantStorageAdapter = class {
         sourceId: input.sourceId,
         metadata: input.metadata ?? {},
         expiresAt: input.expiresAt,
-        createdAt: now
+        createdAt: input.createdAt ?? now
       };
       results.push(memory);
       const qid = crypto.randomUUID();
@@ -2644,10 +2712,13 @@ var Neo4jStorageAdapter = class {
       sourceId: input.sourceId ?? null,
       metadata: JSON.stringify(input.metadata ?? {}),
       expiresAt: input.expiresAt?.toISOString() ?? null,
-      createdAt: now,
+      createdAt: input.createdAt?.toISOString() ?? now,
       embedding: input.embedding ?? null
     };
-    const { createdAt: _, ...updateProps } = props;
+    const updateProps = { ...props };
+    if (!input.createdAt) {
+      delete updateProps.createdAt;
+    }
     const session = this.driver.session({ database: this.database });
     try {
       await session.run(
@@ -2669,7 +2740,7 @@ var Neo4jStorageAdapter = class {
       sourceId: input.sourceId,
       metadata: input.metadata ?? {},
       expiresAt: input.expiresAt,
-      createdAt: new Date(now)
+      createdAt: input.createdAt ?? new Date(now)
     };
   }
   async storeBatch(inputs) {
@@ -2906,7 +2977,7 @@ var WeaviateStorageAdapter = class {
       sourceId: input.sourceId,
       metadata: input.metadata ?? {},
       expiresAt: input.expiresAt,
-      createdAt: now
+      createdAt: input.createdAt ?? now
     };
     const wvId = await this.collection().data.insert({
       id: crypto.randomUUID(),
@@ -3139,7 +3210,7 @@ var LanceDBStorageAdapter = class {
       sourceId: input.sourceId,
       metadata: input.metadata ?? {},
       expiresAt: input.expiresAt,
-      createdAt: now
+      createdAt: input.createdAt ?? now
     };
   }
   async storeBatch(inputs) {
@@ -3150,7 +3221,7 @@ var LanceDBStorageAdapter = class {
     return results;
   }
   async get(id) {
-    const results = await this.table.search(this._zeroVector).limit(1).where(`id = "${id}"`).execute();
+    const results = await this.table.search(this._zeroVector).limit(1).where(`id = "${id}"`).prefilter(true).execute();
     if (results.length === 0) return null;
     const row = results[0];
     if (row.expires_at) {
@@ -3307,7 +3378,7 @@ var MongoDBStorageAdapter = class {
       sourceId: input.sourceId,
       metadata: input.metadata ?? {},
       expiresAt: input.expiresAt,
-      createdAt: now
+      createdAt: input.createdAt ?? now
     };
     await this.collection.insertOne(this.toDoc(memory));
     return memory;
@@ -3536,7 +3607,7 @@ var SQLiteStorageAdapter = class {
       sourceId: input.sourceId,
       metadata: input.metadata ?? {},
       expiresAt: input.expiresAt,
-      createdAt: new Date(now)
+      createdAt: input.createdAt ?? new Date(now)
     };
   }
   async storeBatch(inputs) {
@@ -3809,11 +3880,12 @@ var OpenAILLMAdapter = class {
           try {
             const parsed = JSON.parse(data);
             const content = parsed.choices?.[0]?.delta?.content ?? "";
-            if (parsed.usage) {
-              promptTokens = parsed.usage.prompt_tokens;
-              completionTokens = parsed.usage.completion_tokens;
+            const usage = parsed.usage;
+            if (usage) {
+              promptTokens = usage.prompt_tokens;
+              completionTokens = usage.completion_tokens;
             }
-            if (content) {
+            if (content || usage) {
               yield {
                 text: content,
                 tokens: { prompt: promptTokens, completion: completionTokens, total: promptTokens + completionTokens }
@@ -3869,7 +3941,15 @@ var AnthropicLLMAdapter = class {
       };
     } catch (err) {
       if (err instanceof MemStackError) throw err;
-      throw new MemStackError("LLM_ERROR", `Anthropic request failed: ${err instanceof Error ? err.message : String(err)}`, {
+      const msg = err instanceof Error ? err.message : String(err);
+      const code = err?.code;
+      const isMissingSdk = msg.includes("@anthropic-ai/sdk") && (code === "ERR_MODULE_NOT_FOUND" || code === "MODULE_NOT_FOUND" || /cannot find (package|module)/i.test(msg) || /failed to (load|resolve)/i.test(msg));
+      if (isMissingSdk) {
+        throw new MemStackError("LLM_ERROR", "Anthropic adapter requires @anthropic-ai/sdk. Install it: npm install @anthropic-ai/sdk", {
+          retryable: false
+        });
+      }
+      throw new MemStackError("LLM_ERROR", `Anthropic request failed: ${msg}`, {
         retryable: true
       });
     }
@@ -4158,7 +4238,7 @@ var GroqLLMAdapter = class {
               promptTokens = usage.prompt_tokens;
               completionTokens = usage.completion_tokens;
             }
-            if (content) {
+            if (content || usage) {
               yield {
                 text: content,
                 tokens: { prompt: promptTokens, completion: completionTokens, total: promptTokens + completionTokens }