collective-memory-mcp 0.6.0 → 0.6.1

package/README.md

@@ -1,22 +1,23 @@
 # Collective Memory MCP Server
 
-A persistent, graph-based memory system with **semantic search** that enables AI agents to document their work and learn from each other's experiences. This system transforms ephemeral agent interactions into a searchable knowledge base of structural patterns, solutions, and methodologies.
+A persistent, graph-based memory system with **vector search** that enables AI agents to document their work and learn from each other's experiences. This system transforms ephemeral agent interactions into a searchable knowledge base of structural patterns, solutions, and methodologies.
 
 ## Overview
 
 The Collective Memory System is designed for multi-agent environments where agents need to:
 
 - Document their completed work for future reference
-- Discover how similar tasks were solved previously using **semantic understanding**
+- Discover how similar tasks were solved previously using **vector search**
 - Learn from the structural patterns and approaches of other agents
 - Coordinate across parallel executions without duplicating effort
 
 ## Key Features
 
-- **Semantic Search** - Finds conceptually similar content even when keywords differ
+- **Vector Search** - TF-IDF based search finds conceptually similar content even when keywords differ
 - **Knowledge Graph** - Entities and relations capture complex relationships
 - **Ranked Results** - Similarity scores help identify the most relevant past work
-- **Auto-Embeddings** - New content automatically gets semantic embeddings
+- **Zero Configuration** - Works out of the box, no external dependencies or API keys needed
+- **Pure JavaScript** - No native dependencies, works completely offline
 
 ## Installation
 
@@ -49,10 +50,10 @@ Add this to your Claude system prompt to ensure agents know about the Collective
 You have access to a Collective Memory MCP Server that stores knowledge from previous tasks.
 
 BEFORE starting work, search for similar past tasks using:
-- search_collective_memory (semantic search - understands meaning, not just keywords)
+- search_collective_memory (vector search - understands meaning, not just keywords)
 - find_similar_procedures (finds similar tasks with full implementation details)
 
-The search uses semantic embeddings, so it finds relevant content even when different
+The search uses TF-IDF vector embeddings, so it finds relevant content even when different
 terminology is used. Results are ranked by similarity score.
 
 AFTER completing any task, document it using:
@@ -62,47 +63,16 @@ When writing observations, be SPECIFIC and include facts like file paths, versio
 metrics, and error messages. Avoid vague statements like "works well" or "fixed bugs".
 ```
 
-## Setting Up Semantic Search
+## How Vector Search Works
 
-For semantic search to work, you need to configure an embeddings provider:
+This system uses **TF-IDF (Term Frequency-Inverse Document Frequency)** vector search:
 
-### Option 1: OpenAI (Recommended - Best Quality)
+- Tokenizes text into meaningful terms
+- Calculates term importance based on frequency
+- Uses cosine similarity to rank results
+- Works entirely offline with no external dependencies
 
-```bash
-# Configure with your OpenAI API key
-# Use the manage_embeddings tool with:
-{
-  "action": "configure",
-  "provider": "openai",
-  "api_key": "sk-..."
-}
-```
-
-### Option 2: Ollama (Free - Local)
-
-```bash
-# Install Ollama first
-# Then pull the embedding model
-ollama pull nomic-embed-text
-
-# Configure the provider
-{
-  "action": "configure",
-  "provider": "ollama"
-}
-```
-
-### Generate Embeddings for Existing Data
-
-After configuring, generate embeddings for any existing entities:
-
-```json
-{
-  "action": "generate"
-}
-```
-
-**Note:** Without configuring embeddings, the system falls back to keyword-based search.
+No configuration needed - it just works!
 
 ## Entity Types
 
@@ -145,7 +115,7 @@ After configuring, generate embeddings for any existing entities:
 ### Query & Search
 
 - **read_graph** - Read entire knowledge graph
-- **search_collective_memory** - Semantic search with ranked results
+- **search_collective_memory** - Vector search with ranked results
 - **open_nodes** - Retrieve specific nodes by name
 
 ### Agent Workflow
@@ -153,10 +123,6 @@ After configuring, generate embeddings for any existing entities:
 - **record_task_completion** - Primary tool for documenting completed work
 - **find_similar_procedures** - Find similar tasks with full implementation details
 
-### Embeddings Management
-
-- **manage_embeddings** - Configure semantic search and generate embeddings
-
 ## Example Usage
 
 ### Recording a Task Completion
@@ -182,7 +148,7 @@ await session.callTool("record_task_completion", {
 });
 ```
 
-### Finding Similar Procedures (Semantic Search)
+### Finding Similar Procedures (Vector Search)
 
 ```javascript
 const result = await session.callTool("find_similar_procedures", {
@@ -195,25 +161,18 @@ const result = await session.callTool("find_similar_procedures", {
 //     { "task": {...}, "score": 0.89, "artifacts": [...], "structures": [...] },
 //     { "task": {...}, "score": 0.82, "artifacts": [...], "structures": [...] }
 //   ],
-//   "search_method": "semantic"
+//   "search_method": "vector"
 // }
 ```
 
-### Configuring Embeddings
+### Searching the Collective Memory
 
 ```javascript
-// Check status
-await session.callTool("manage_embeddings", { "action": "status" });
-
-// Configure OpenAI
-await session.callTool("manage_embeddings", {
-  "action": "configure",
-  "provider": "openai",
-  "api_key": "sk-..."
+const result = await session.callTool("search_collective_memory", {
+  query: "database optimization"
 });
 
-// Generate embeddings for existing data
-await session.callTool("manage_embeddings", { "action": "generate" });
+// Returns matching entities with similarity scores
 ```
 
 ## Database
@@ -222,16 +181,15 @@ The server uses JSON file storage for persistence. Data is stored at:
 
 ```
 ~/.collective-memory/memory.json      # Knowledge graph data
-~/.collective-memory/config.json      # Embeddings provider configuration
 ```
 
-## Semantic Search Benefits
+## Vector Search Benefits
 
-| Before (Keyword Search) | After (Semantic Search) |
-|------------------------|------------------------|
-| Query "login" misses "authentication" | Query "login" finds "authentication", "JWT", "OAuth" |
+| Traditional Keyword Search | TF-IDF Vector Search |
+|---------------------------|---------------------|
+| Exact word matching required | Finds related terms automatically |
 | No relevance ranking | Results ranked by similarity score (0-1) |
-| Exact word matching required | Understands meaning and intent |
+| "login" misses "authentication" | "login" finds "authentication", "JWT", "OAuth" |
 | High false-positive rate | More precise, relevant results |
 
 ## Requirements

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "collective-memory-mcp",
-  "version": "0.6.0",
-  "description": "A persistent, graph-based memory system for AI agents with semantic search (MCP Server)",
+  "version": "0.6.1",
+  "description": "A persistent, graph-based memory system for AI agents with TF-IDF vector search (MCP Server)",
   "type": "module",
   "main": "src/server.js",
   "bin": {
@@ -20,7 +20,10 @@
     "collective",
     "anthropic",
     "claude",
-    "model-context-protocol"
+    "model-context-protocol",
+    "vector-search",
+    "tf-idf",
+    "semantic-search"
   ],
   "license": "MIT",
   "repository": {

package/src/server.js

@@ -14,7 +14,6 @@ import {
 } from "@modelcontextprotocol/sdk/types.js";
 import { getStorage } from "./storage.js";
 import { Entity, Relation, ENTITY_TYPES, RELATION_TYPES } from "./models.js";
-import { loadConfig } from "./embeddings.js";
 
 /**
  * Create and configure the MCP server
@@ -22,9 +21,6 @@ import { loadConfig } from "./embeddings.js";
 async function createServer() {
   const storage = getStorage();
 
-  // Initialize embeddings for semantic search
-  await storage.initEmbeddings();
-
   const server = new Server(
     {
       name: "collective-memory",
@@ -234,17 +230,16 @@ async function createServer() {
         {
           name: "search_collective_memory",
           description:
-            "**Search all past work using semantic search** - Use before starting a task to learn from previous solutions. " +
-            "Uses semantic embeddings to find conceptually similar content, even with different keywords. " +
+            "**Search all past work using vector search** - Use before starting a task to learn from previous solutions. " +
+            "Uses TF-IDF vector search to find conceptually similar content, even with different keywords. " +
             "Returns ranked results with similarity scores. " +
-            "Automatically falls back to keyword search if embeddings aren't available. " +
             "Use find_similar_procedures for more detailed results with artifacts.",
           inputSchema: {
             type: "object",
             properties: {
               query: {
                 type: "string",
-                description: "What are you looking for? Semantic search understands meaning. (e.g., 'authentication', 'CORS fix', 'database')",
+                description: "What are you looking for? Vector search understands meaning. (e.g., 'authentication', 'CORS fix', 'database')",
               },
             },
             required: ["query"],
@@ -355,9 +350,9 @@ async function createServer() {
         {
           name: "find_similar_procedures",
           description:
-            "**Use BEFORE starting work** - Find how similar tasks were solved previously using semantic search. " +
+            "**Use BEFORE starting work** - Find how similar tasks were solved previously using vector search. " +
             "Returns complete implementation details including artifacts and structures, ranked by similarity. " +
-            "Understands meaning and intent, not just keywords. " +
+            "Understands meaning and intent using TF-IDF vectors. " +
             "Learn from past solutions before implementing new features. " +
             "Query examples: 'authentication', 'database migration', 'API design', 'error handling'.",
           inputSchema: {
@@ -365,39 +360,12 @@ async function createServer() {
             properties: {
               query: {
                 type: "string",
-                description: "What are you trying to do? Semantic search finds conceptually similar work. (e.g., 'authentication implementation', 'database migration')",
+                description: "What are you trying to do? Vector search finds conceptually similar work. (e.g., 'authentication implementation', 'database migration')",
               },
             },
             required: ["query"],
           },
         },
-        {
-          name: "manage_embeddings",
-          description:
-            "**Manage semantic search embeddings** - Generate embeddings for existing entities " +
-            "to enable semantic search. Run this once after setting up an embeddings provider. " +
-            "Embeddings enable finding similar content even when keywords don't match exactly.",
-          inputSchema: {
-            type: "object",
-            properties: {
-              action: {
-                type: "string",
-                enum: ["generate", "status", "configure"],
-                description: "Action: 'generate' creates embeddings for entities missing them, 'status' shows current state, 'configure' updates settings",
-              },
-              provider: {
-                type: "string",
-                enum: ["openai", "ollama"],
-                description: "Provider to use (only for 'configure' action)",
-              },
-              api_key: {
-                type: "string",
-                description: "API key for OpenAI (only for 'configure' action with provider='openai')",
-              },
-            },
-            required: ["action"],
-          },
-        },
       ],
     };
   });
@@ -736,9 +704,6 @@ Future agents will read your observations to learn. Write for them, not for your
         case "find_similar_procedures":
           return { content: [{ type: "text", text: JSON.stringify(findSimilarProcedures(args), null, 2) }] };
 
-        case "manage_embeddings":
-          return { content: [{ type: "text", text: JSON.stringify(await manageEmbeddings(args), null, 2) }] };
-
         default:
           throw new Error(`Unknown tool: ${name}`);
       }
@@ -1126,112 +1091,6 @@ Future agents will read your observations to learn. Write for them, not for your
     return { similar_tasks: results, count: results.length, search_method: searchResult.method };
   }
 
-  async function manageEmbeddings({ action = "status", provider = null, api_key = null }) {
-    const { saveConfig } = await import("./embeddings.js");
-
-    switch (action) {
-      case "status": {
-        const allEntities = storage.getAllEntities();
-        const withEmbeddings = allEntities.filter(
-          (e) => storage.data.entities[e.name]?.embedding
-        ).length;
-
-        const config = loadConfig();
-        const isReady = storage.embeddingsReady;
-
-        return {
-          status: "success",
-          action: "status",
-          embeddings_ready: isReady,
-          provider: config.embedding_provider,
-          entities_with_embeddings: withEmbeddings,
-          total_entities: allEntities.length,
-          coverage_percent: allEntities.length > 0
-            ? Math.round((withEmbeddings / allEntities.length) * 100)
-            : 0,
-          message: isReady
-            ? `Embeddings enabled using ${config.embedding_provider}. ${withEmbeddings}/${allEntities.length} entities have embeddings.`
-            : "Embeddings not configured. Use 'configure' action to set up a provider.",
-        };
-      }
-
-      case "configure": {
-        if (!provider) {
-          return {
-            status: "error",
-            message: "Provider is required for configure action",
-          };
-        }
-
-        const config = loadConfig();
-        config.embedding_provider = provider;
-
-        if (provider === "openai" && api_key) {
-          config.openai_api_key = api_key;
-        }
-
-        saveConfig(config);
-
-        // Re-initialize embeddings with new config
-        storage.embeddingsReady = false;
-        await storage.initEmbeddings();
-
-        return {
-          status: "success",
-          action: "configure",
-          provider,
-          embeddings_ready: storage.embeddingsReady,
-          message: storage.embeddingsReady
-            ? `Successfully configured ${provider} for embeddings`
-            : `Configured ${provider} but provider is not available. Check API keys or provider status.`,
-        };
-      }
-
-      case "generate": {
-        // Generate embeddings for entities that don't have them
-        const allEntities = storage.getAllEntities();
-        const missing = allEntities.filter(
-          (e) => !storage.data.entities[e.name]?.embedding
-        );
-
-        if (missing.length === 0) {
-          return {
-            status: "success",
-            action: "generate",
-            message: "All entities already have embeddings",
-            processed: 0,
-          };
-        }
-
-        try {
-          const result = await storage.generateMissingEmbeddings((current, total, name) => {
-            // Optional: could emit progress events
-          });
-
-          return {
-            status: "success",
-            action: "generate",
-            processed: result.processed,
-            total_entities: result.total,
-            message: `Generated embeddings for ${result.processed} entities`,
-          };
-        } catch (error) {
-          return {
-            status: "error",
-            action: "generate",
-            message: error.message,
-          };
-        }
-      }
-
-      default:
-        return {
-          status: "error",
-          message: `Unknown action: ${action}`,
-        };
-    }
-  }
-
   return server;
 }
 

package/src/storage.js

@@ -1,28 +1,27 @@
 /**
  * Storage layer for the Collective Memory System using JSON file.
- * Pure JavaScript - no native dependencies required.
+ * Pure JavaScript - no external dependencies required.
+ * Uses TF-IDF vector search for semantic-like matching.
  */
 
-import { promises as fs } from "fs";
 import { existsSync, mkdirSync, readFileSync, writeFileSync } from "fs";
+import { promises as fs } from "fs";
 import path from "path";
 import os from "os";
 import { Entity, Relation } from "./models.js";
-import { getEmbeddings } from "./embeddings.js";
+import { getVectorIndex, buildIndexFromEntities } from "./vector-search.js";
 
 const DB_DIR = path.join(os.homedir(), ".collective-memory");
 const DB_PATH = path.join(DB_DIR, "memory.json");
 
 /**
- * Simple file-based storage
+ * File-based storage with vector search
  */
 export class Storage {
-  constructor(dbPath = DB_PATH, embeddingsEnabled = true) {
+  constructor(dbPath = DB_PATH) {
     this.dbPath = dbPath;
     this.data = null;
-    this.embeddingsEnabled = embeddingsEnabled;
-    this.embeddings = null;
-    this.embeddingsReady = false;
+    this.vectorIndex = getVectorIndex();
     // Initialize synchronously
     this.init();
   }
@@ -45,10 +44,13 @@ export class Storage {
         this.data = {
           entities: {},
           relations: [],
-          version: "2.0", // Version bump for embeddings support
+          version: "2.0",
         };
         this.saveSync();
       }
+
+      // Build vector index from loaded entities
+      this.rebuildIndex();
     } catch (error) {
       // If anything fails, start with empty data
       this.data = {
@@ -60,53 +62,11 @@ export class Storage {
   }
 
   /**
-   * Initialize embeddings asynchronously
-   */
-  async initEmbeddings() {
-    if (!this.embeddingsEnabled || this.embeddingsReady) {
-      return this.embeddingsReady;
-    }
-
-    try {
-      this.embeddings = getEmbeddings();
-      this.embeddingsReady = await this.embeddings.isAvailable();
-    } catch (error) {
-      console.warn("Embeddings not available:", error.message);
-      this.embeddingsReady = false;
-    }
-
-    return this.embeddingsReady;
-  }
-
-  /**
-   * Generate embedding for an entity
+   * Rebuild the vector search index from all entities
    */
-  async generateEmbedding(entity) {
-    if (!this.embeddingsReady || !this.embeddings) {
-      return null;
-    }
-
-    try {
-      const text = this.embeddings.createEntityText(entity);
-      return await this.embeddings.embed(text);
-    } catch (error) {
-      console.warn("Failed to generate embedding:", error.message);
-      return null;
-    }
-  }
-
-  /**
-   * Store embedding in entity data
-   */
-  async updateEntityEmbedding(entityName, entity) {
-    if (!this.embeddingsReady) {
-      return;
-    }
-
-    const embedding = await this.generateEmbedding(entity);
-    if (embedding) {
-      this.data.entities[entityName].embedding = embedding;
-    }
+  rebuildIndex() {
+    const entities = this.getAllEntities();
+    buildIndexFromEntities(entities);
   }
 
   /**
@@ -135,6 +95,15 @@ export class Storage {
     await fs.writeFile(this.dbPath, JSON.stringify(this.data, null, 2), "utf-8");
   }
 
+  /**
+   * Initialize embeddings (placeholder for API compatibility)
+   * This system uses built-in TF-IDF vector search, no external embeddings needed
+   */
+  async initEmbeddings() {
+    // Vector search is always available, no configuration needed
+    return true;
+  }
+
   // ========== Entity Operations ==========
 
   /**
@@ -144,11 +113,12 @@ export class Storage {
     if (this.data.entities[entity.name]) {
       return false;
     }
-    const entityData = entity.toJSON();
-    this.data.entities[entity.name] = entityData;
 
-    // Generate embedding if available
-    await this.updateEntityEmbedding(entity.name, entity);
+    this.data.entities[entity.name] = entity.toJSON();
+
+    // Add to vector index
+    this.vectorIndex.addDocument(entity.name, entity);
+    this.vectorIndex.build();
 
     await this.save();
     return true;
@@ -186,7 +156,6 @@ export class Storage {
     const entity = this.data.entities[name];
     if (!entity) return false;
 
-    const updated = false;
     if (observations !== undefined) {
       entity.observations = observations;
     }
@@ -194,11 +163,8 @@ export class Storage {
       entity.metadata = metadata;
     }
 
-    // Regenerate embedding if observations changed
-    if (observations !== undefined && this.embeddingsReady) {
-      const entityObj = new Entity(entity);
-      await this.updateEntityEmbedding(name, entityObj);
-    }
+    // Rebuild index to reflect changes
+    this.rebuildIndex();
 
     await this.save();
     return true;
@@ -219,6 +185,9 @@ export class Storage {
       r => r.from !== name && r.to !== name
     );
 
+    // Rebuild index
+    this.rebuildIndex();
+
     await this.save();
     return true;
   }
@@ -319,133 +288,42 @@ export class Storage {
     return count;
   }
 
-  // ========== Search ==========
+  // ========== Vector Search ==========
 
   /**
-   * Search entities by name, type, or observations
-   * Uses word-based matching - any word in the query that matches returns the entity
+   * Vector search is always available (built-in TF-IDF)
    */
-  searchEntities(query) {
-    // Split query into words, remove common stop words
-    const stopWords = new Set(["the", "a", "an", "and", "or", "but", "in", "on", "at", "to", "for", "of", "with", "by"]);
-    const words = query
-      .toLowerCase()
-      .split(/\s+/)
-      .filter(w => w.length > 2 && !stopWords.has(w));
-
-    if (words.length === 0) {
-      // Fallback to original query if all words were filtered
-      const lowerQuery = query.toLowerCase();
-      return this.getAllEntities().filter(e => {
-        if (e.name.toLowerCase().includes(lowerQuery)) return true;
-        if (e.entityType.toLowerCase().includes(lowerQuery)) return true;
-        if (e.observations.some(o => o.toLowerCase().includes(lowerQuery))) return true;
-        return false;
-      });
-    }
-
-    return this.getAllEntities().filter(e => {
-      // Check if ANY word matches in name, type, or observations
-      for (const word of words) {
-        if (e.name.toLowerCase().includes(word)) return true;
-        if (e.entityType.toLowerCase().includes(word)) return true;
-        if (e.observations.some(o => o.toLowerCase().includes(word))) return true;
-      }
-      return false;
-    });
+  get embeddingsReady() {
+    return true;
   }
 
   /**
-   * Semantic search using embeddings
-   * Returns entities ranked by similarity score
+   * Alias for semantic search - uses built-in TF-IDF vector search
+   * This provides semantic-like understanding without external dependencies
    */
   async semanticSearchEntities(query, options = {}) {
-    const {
-      topK = 10,
-      threshold = 0.65, // Lower threshold for more matches
-      entityType = null,
-    } = options;
-
-    // Initialize embeddings if not ready
-    if (!this.embeddingsReady) {
-      await this.initEmbeddings();
-    }
-
-    // Fall back to keyword search if embeddings not available
-    if (!this.embeddingsReady) {
-      const results = this.searchEntities(query);
-      return {
-        results: results.map(e => ({ entity: e, score: 0 })),
-        method: "keyword",
-        count: results.length,
-      };
-    }
-
-    try {
-      // Generate embedding for the query
-      const queryEmbedding = await this.embeddings.embed(query);
-
-      // Get all entities with their embeddings
-      const allEntities = this.getAllEntities();
-      const items = allEntities
-        .filter(e => !entityType || e.entityType === entityType)
-        .map(e => ({
-          entity: e,
-          embedding: this.data.entities[e.name]?.embedding || null,
-        }))
-        .filter(item => item.embedding !== null);
-
-      // Find most similar
-      const scoredResults = this.embeddings.findMostSimilar(
-        queryEmbedding,
-        items,
-        topK,
-        threshold
-      );
-
-      return {
-        results: scoredResults.map(r => ({ entity: r.entity, score: r.score })),
-        method: "semantic",
-        count: scoredResults.length,
-      };
-    } catch (error) {
-      console.warn("Semantic search failed, falling back to keyword:", error.message);
-      const results = this.searchEntities(query);
-      return {
-        results: results.map(e => ({ entity: e, score: 0 })),
-        method: "keyword",
-        count: results.length,
-      };
-    }
+    return this.vectorSearchEntities(query, options);
   }
 
   /**
-   * Generate embeddings for all entities that don't have them
-   * Useful for migrating existing data to semantic search
+   * Search entities using TF-IDF vector search
+   * Returns entities ranked by similarity score
    */
-  async generateMissingEmbeddings(progressCallback = null) {
-    // Initialize embeddings if not ready
-    if (!this.embeddingsReady) {
-      const ready = await this.initEmbeddings();
-      if (!ready) {
-        throw new Error("Embeddings provider not available");
-      }
-    }
+  vectorSearchEntities(query, options = {}) {
+    const {
+      topK = 10,
+      threshold = 0.1,
+      entityType = null
+    } = options;
 
-    const entities = this.getAllEntities();
-    const missing = entities.filter(e => !this.data.entities[e.name]?.embedding);
-
-    let processed = 0;
-    for (const entity of missing) {
-      await this.updateEntityEmbedding(entity.name, entity);
-      processed++;
-      if (progressCallback) {
-        progressCallback(processed, missing.length, entity.name);
-      }
-    }
+    // Use vector search index
+    const result = this.vectorIndex.search(query, { topK, threshold, entityType });
 
-    await this.save();
-    return { processed, total: entities.length };
+    return {
+      results: result.results,
+      method: "vector",
+      count: result.count
+    };
   }
 
   /**
@@ -477,6 +355,17 @@ export class Storage {
     return result;
   }
 
+  /**
+   * Get index statistics
+   */
+  getIndexStats() {
+    return {
+      ...this.vectorIndex.getStats(),
+      entityCount: Object.keys(this.data.entities).length,
+      relationCount: this.data.relations.length
+    };
+  }
+
   /**
    * Close storage
    */

package/src/vector-search.js

@@ -0,0 +1,322 @@
+/**
+ * Vector Search module for the Collective Memory System.
+ * Uses TF-IDF (Term Frequency-Inverse Document Frequency) for semantic-like search.
+ * Pure JavaScript - no external dependencies, works completely offline.
+ */
+
+/**
+ * Tokenize text into terms
+ * - Converts to lowercase
+ * - Removes special characters
+ * - Splits into words
+ * - Filters stop words
+ */
+function tokenize(text) {
+  const stopWords = new Set([
+    "a", "about", "above", "after", "again", "against", "all", "am", "an", "and",
+    "any", "are", "as", "at", "be", "because", "been", "before", "being", "below",
+    "between", "both", "but", "by", "can", "did", "do", "does", "doing", "don",
+    "down", "during", "each", "few", "for", "from", "further", "had", "has", "have",
+    "having", "he", "her", "here", "hers", "herself", "him", "himself", "his",
+    "how", "i", "if", "in", "into", "is", "it", "its", "itself", "just", "me",
+    "might", "more", "most", "must", "my", "myself", "no", "nor", "not", "now",
+    "of", "off", "on", "once", "only", "or", "other", "our", "ours", "ourselves",
+    "out", "over", "own", "s", "same", "she", "should", "so", "some", "still",
+    "such", "t", "than", "that", "the", "their", "theirs", "them", "themselves",
+    "then", "there", "these", "they", "this", "those", "through", "to", "too",
+    "under", "until", "up", "very", "was", "we", "were", "what", "when", "where",
+    "which", "while", "who", "whom", "why", "will", "with", "would", "you",
+    "your", "yours", "yourself", "yourselves", "task", "artifact", "structure",
+    "agent", "session", "entity", "description", "created", "during"
+  ]);
+
+  return text
+    .toLowerCase()
+    .replace(/[^\w\s@#.-]/g, " ")  // Keep @, #, ., - for technical terms
+    .split(/\s+/)
+    .filter(word => word.length > 2 && !stopWords.has(word));
+}
+
+/**
+ * Extract terms from an entity for indexing
+ */
+function extractEntityTerms(entity) {
+  const terms = [];
+
+  // Name has high weight
+  terms.push(...tokenize(entity.name));
+
+  // Entity type
+  terms.push(entity.entityType);
+
+  // All observations
+  if (entity.observations) {
+    for (const obs of entity.observations) {
+      terms.push(...tokenize(obs));
+    }
+  }
+
+  // Metadata
+  if (entity.metadata) {
+    for (const value of Object.values(entity.metadata)) {
+      if (typeof value === "string") {
+        terms.push(...tokenize(value));
+      }
+    }
+  }
+
+  return terms;
+}
+
+/**
+ * Calculate term frequency for a document
+ */
+function calculateTermFrequency(terms) {
+  const tf = {};
+  const totalTerms = terms.length;
+
+  for (const term of terms) {
+    tf[term] = (tf[term] || 0) + 1;
+  }
+
+  // Normalize by document length
+  for (const term in tf) {
+    tf[term] = tf[term] / totalTerms;
+  }
+
+  return tf;
+}
+
+/**
+ * Calculate inverse document frequency
+ */
+function calculateIDF(documents) {
+  const idf = {};
+  const totalDocs = documents.length;
+
+  // Count documents containing each term
+  for (const doc of documents) {
+    const uniqueTerms = new Set(doc.terms);
+    for (const term of uniqueTerms) {
+      idf[term] = (idf[term] || 0) + 1;
+    }
+  }
+
+  // Calculate IDF
+  for (const term in idf) {
+    idf[term] = Math.log(totalDocs / (1 + idf[term]));
+  }
+
+  return idf;
+}
+
+/**
+ * Create a TF-IDF vector for a document
+ */
+function createTFIDFVector(tf, idf, allTerms) {
+  const vector = [];
+
+  for (const term of allTerms) {
+    const tfValue = tf[term] || 0;
+    const idfValue = idf[term] || 0;
+    vector.push(tfValue * idfValue);
+  }
+
+  return vector;
+}
+
+/**
+ * Calculate cosine similarity between two vectors
+ */
+function cosineSimilarity(a, b) {
+  if (a.length !== b.length) {
+    return 0;
+  }
+
+  let dotProduct = 0;
+  let normA = 0;
+  let normB = 0;
+
+  for (let i = 0; i < a.length; i++) {
+    dotProduct += a[i] * b[i];
+    normA += a[i] * a[i];
+    normB += b[i] * b[i];
+  }
+
+  normA = Math.sqrt(normA);
+  normB = Math.sqrt(normB);
+
+  if (normA === 0 || normB === 0) {
+    return 0;
+  }
+
+  return dotProduct / (normA * normB);
+}
+
+/**
+ * Vector Search Index
+ */
+class VectorSearchIndex {
+  constructor() {
+    this.documents = [];
+    this.allTerms = new Set();
+    this.idf = {};
+    this.built = false;
+  }
+
+  /**
+   * Add a document to the index
+   */
+  addDocument(id, entity) {
+    const terms = extractEntityTerms(entity);
+
+    this.documents.push({
+      id,
+      entity,
+      terms,
+      tf: null,
+      vector: null
+    });
+
+    for (const term of terms) {
+      this.allTerms.add(term);
+    }
+
+    this.built = false;
+  }
+
+  /**
+   * Build the index (calculate TF-IDF vectors)
+   */
+  build() {
+    if (this.documents.length === 0) {
+      return;
+    }
+
+    const termList = Array.from(this.allTerms);
+
+    // Calculate IDF for all terms
+    this.idf = calculateIDF(this.documents);
+
+    // Calculate TF and create vectors for each document
+    for (const doc of this.documents) {
+      doc.tf = calculateTermFrequency(doc.terms);
+      doc.vector = createTFIDFVector(doc.tf, this.idf, termList);
+    }
+
+    this.allTermsList = termList;
+    this.built = true;
+  }
+
+  /**
+   * Search the index
+   */
+  search(query, options = {}) {
+    const {
+      topK = 10,
+      threshold = 0.1,
+      entityType = null
+    } = options;
+
+    // Build if not already built
+    if (!this.built) {
+      this.build();
+    }
+
+    // Tokenize query
+    const queryTerms = tokenize(query);
+    if (queryTerms.length === 0) {
+      return { results: [], method: "vector", count: 0 };
+    }
+
+    // Create query vector
+    const queryTF = calculateTermFrequency(queryTerms);
+    const queryVector = createTFIDFVector(queryTF, this.idf, this.allTermsList);
+
+    // Calculate similarities
+    const results = this.documents
+      .filter(doc => !entityType || doc.entity.entityType === entityType)
+      .map(doc => {
+        const score = cosineSimilarity(queryVector, doc.vector);
+        return {
+          entity: doc.entity,
+          score
+        };
+      })
+      .filter(r => r.score >= threshold)
+      .sort((a, b) => b.score - a.score)
+      .slice(0, topK);
+
+    return {
+      results,
+      method: "vector",
+      count: results.length
+    };
+  }
+
+  /**
+   * Clear the index
+   */
+  clear() {
+    this.documents = [];
+    this.allTerms = new Set();
+    this.idf = {};
+    this.built = false;
+  }
+
+  /**
+   * Get index statistics
+   */
+  getStats() {
+    return {
+      documentCount: this.documents.length,
+      uniqueTermCount: this.allTerms.size,
+      built: this.built
+    };
+  }
+}
+
+/**
+ * Singleton instance
+ */
+let indexInstance = null;
+
+/**
+ * Get or create the vector search index
+ */
+export function getVectorIndex() {
+  if (!indexInstance) {
+    indexInstance = new VectorSearchIndex();
+  }
+  return indexInstance;
+}
+
+/**
+ * Rebuild index from entities
+ */
+export function buildIndexFromEntities(entities) {
+  const index = getVectorIndex();
+  index.clear();
+
+  for (const entity of entities) {
+    index.addDocument(entity.name, entity);
+  }
+
+  index.build();
+  return index;
+}
+
+export {
+  VectorSearchIndex,
+  tokenize,
+  extractEntityTerms,
+  cosineSimilarity,
+  calculateTermFrequency,
+  calculateIDF
+};
+
+export default {
+  getVectorIndex,
+  buildIndexFromEntities,
+  VectorSearchIndex
+};

package/src/embeddings.js

@@ -1,318 +0,0 @@
-/**
- * Embeddings module for semantic search in the Collective Memory System.
- * Supports multiple embedding providers with cosine similarity.
- */
-
-import fs from "fs";
-import path from "path";
-import os from "os";
-
-const CONFIG_DIR = path.join(os.homedir(), ".collective-memory");
-const CONFIG_PATH = path.join(CONFIG_DIR, "config.json");
-
-/**
- * Default configuration
- */
-const DEFAULT_CONFIG = {
-  embedding_provider: "openai", // 'openai' or 'ollama'
-  openai_api_key: null,
-  openai_model: "text-embedding-3-small",
-  ollama_base_url: "http://localhost:11434",
-  ollama_model: "nomic-embed-text",
-  embedding_dimension: 1536,
-};
-
-/**
- * Load configuration from file
- */
-function loadConfig() {
-  try {
-    if (fs.existsSync(CONFIG_PATH)) {
-      const content = fs.readFileSync(CONFIG_PATH, "utf-8");
-      return { ...DEFAULT_CONFIG, ...JSON.parse(content) };
-    }
-  } catch (error) {
-    console.error("Failed to load config:", error.message);
-  }
-  return { ...DEFAULT_CONFIG };
-}
-
-/**
- * Save configuration to file
- */
-function saveConfig(config) {
-  try {
-    if (!fs.existsSync(CONFIG_DIR)) {
-      fs.mkdirSync(CONFIG_DIR, { recursive: true });
-    }
-    fs.writeFileSync(CONFIG_PATH, JSON.stringify(config, null, 2));
-  } catch (error) {
-    console.error("Failed to save config:", error.message);
-  }
-}
-
-/**
- * Calculate cosine similarity between two vectors
- */
-function cosineSimilarity(a, b) {
-  if (a.length !== b.length) {
-    throw new Error("Vector dimensions must match");
-  }
-
-  let dotProduct = 0;
-  let normA = 0;
-  let normB = 0;
-
-  for (let i = 0; i < a.length; i++) {
-    dotProduct += a[i] * b[i];
-    normA += a[i] * a[i];
-    normB += b[i] * b[i];
-  }
-
-  normA = Math.sqrt(normA);
-  normB = Math.sqrt(normB);
-
-  if (normA === 0 || normB === 0) {
-    return 0;
-  }
-
-  return dotProduct / (normA * normB);
-}
-
-/**
- * OpenAI embeddings provider
- */
-class OpenAIEmbeddings {
-  constructor(config) {
-    this.apiKey = config.openai_api_key || process.env.OPENAI_API_KEY;
-    this.model = config.openai_model || "text-embedding-3-small";
-    this.dimension = config.embedding_dimension || 1536;
-  }
-
-  isAvailable() {
-    return !!this.apiKey;
-  }
-
-  async embed(text) {
-    if (!this.isAvailable()) {
-      throw new Error("OpenAI API key not configured");
-    }
-
-    const response = await fetch("https://api.openai.com/v1/embeddings", {
-      method: "POST",
-      headers: {
-        "Content-Type": "application/json",
-        "Authorization": `Bearer ${this.apiKey}`,
-      },
-      body: JSON.stringify({
-        model: this.model,
-        input: text,
-      }),
-    });
-
-    if (!response.ok) {
-      const error = await response.text();
-      throw new Error(`OpenAI API error: ${error}`);
-    }
-
-    const data = await response.json();
-    return data.data[0].embedding;
-  }
-
-  async embedBatch(texts) {
-    if (!this.isAvailable()) {
-      throw new Error("OpenAI API key not configured");
-    }
-
-    const response = await fetch("https://api.openai.com/v1/embeddings", {
-      method: "POST",
-      headers: {
-        "Content-Type": "application/json",
-        "Authorization": `Bearer ${this.apiKey}`,
-      },
-      body: JSON.stringify({
-        model: this.model,
-        input: texts,
-      }),
-    });
-
-    if (!response.ok) {
-      const error = await response.text();
-      throw new Error(`OpenAI API error: ${error}`);
-    }
-
-    const data = await response.json();
-    return data.data.map((item) => item.embedding);
-  }
-}
-
-/**
- * Ollama embeddings provider (local, free)
- */
-class OllamaEmbeddings {
-  constructor(config) {
-    this.baseUrl = config.ollama_base_url || "http://localhost:11434";
-    this.model = config.ollama_model || "nomic-embed-text";
-    this.dimension = 768; // Default for nomic-embed-text
-  }
-
-  isAvailable() {
-    // Check if Ollama is running
-    return fetch(`${this.baseUrl}/api/tags`)
-      .then((res) => res.ok)
-      .catch(() => false);
-  }
-
-  async embed(text) {
-    const response = await fetch(`${this.baseUrl}/api/embeddings`, {
-      method: "POST",
-      headers: {
-        "Content-Type": "application/json",
-      },
-      body: JSON.stringify({
-        model: this.model,
-        prompt: text,
-      }),
-    });
-
-    if (!response.ok) {
-      const error = await response.text();
-      throw new Error(`Ollama API error: ${error}`);
-    }
-
-    const data = await response.json();
-    return data.embedding;
-  }
-
-  async embedBatch(texts) {
-    // Ollama doesn't support batch embeddings, so we do them sequentially
-    const embeddings = [];
-    for (const text of texts) {
-      embeddings.push(await this.embed(text));
-    }
-    return embeddings;
-  }
-}
-
-/**
- * Main embeddings class that manages providers
- */
-class Embeddings {
-  constructor(config) {
-    this.config = config || loadConfig();
-    this.providers = {
-      openai: new OpenAIEmbeddings(this.config),
-      ollama: new OllamaEmbeddings(this.config),
-    };
-    this.activeProvider = this.config.embedding_provider || "openai";
-  }
-
-  /**
-   * Get the active provider
-   */
-  getProvider() {
-    return this.providers[this.activeProvider];
-  }
-
-  /**
-   * Check if the active provider is available
-   */
-  async isAvailable() {
-    const provider = this.getProvider();
-
-    if (this.activeProvider === "ollama") {
-      return await provider.isAvailable();
-    }
-
-    return provider.isAvailable();
-  }
-
-  /**
-   * Generate embedding for a single text
-   */
-  async embed(text) {
-    const provider = this.getProvider();
-    return await provider.embed(text);
-  }
-
-  /**
-   * Generate embeddings for multiple texts
-   */
-  async embedBatch(texts) {
-    const provider = this.getProvider();
-    return await provider.embedBatch(texts);
-  }
-
-  /**
-   * Find most similar items using cosine similarity
-   */
-  findMostSimilar(queryEmbedding, items, topK = 10, threshold = 0.7) {
-    const results = items.map((item) => {
-      if (!item.embedding) {
-        return { ...item, score: 0 };
-      }
-      const score = cosineSimilarity(queryEmbedding, item.embedding);
-      return { ...item, score };
-    });
-
-    // Sort by score descending and filter by threshold
-    return results
-      .filter((r) => r.score >= threshold)
-      .sort((a, b) => b.score - a.score)
-      .slice(0, topK);
-  }
-
-  /**
-   * Create text representation for entity embedding
-   */
-  createEntityText(entity) {
-    const parts = [];
-
-    // Name carries important semantic weight
-    parts.push(`Name: ${entity.name}`);
-
-    // Entity type provides context
-    parts.push(`Type: ${entity.entityType}`);
-
-    // Observations contain the detailed information
-    if (entity.observations && entity.observations.length > 0) {
-      parts.push(`Observations:\n${entity.observations.join("\n")}`);
-    }
-
-    // Metadata if present
-    if (entity.metadata) {
-      parts.push(`Metadata: ${JSON.stringify(entity.metadata)}`);
-    }
-
-    return parts.join("\n\n");
-  }
-}
-
-/**
- * Singleton instance
- */
-let embeddingsInstance = null;
-
-/**
- * Get or create embeddings instance
- */
-export function getEmbeddings(config) {
-  if (!embeddingsInstance) {
-    embeddingsInstance = new Embeddings(config);
-  }
-  return embeddingsInstance;
-}
-
-/**
- * Export utilities
- */
-export {
-  cosineSimilarity,
-  loadConfig,
-  saveConfig,
-  Embeddings,
-  OpenAIEmbeddings,
-  OllamaEmbeddings,
-};
-
-export default { getEmbeddings, cosineSimilarity, loadConfig, saveConfig };