@jeremiaheth/neolata-mem 0.2.0 → 0.3.0

package/README.md

@@ -2,6 +2,8 @@
 
 **Graph-native memory engine for AI agents.** Zettelkasten-inspired linking, biological decay, conflict resolution.
 
+[![npm version](https://img.shields.io/npm/v/@jeremiaheth/neolata-mem.svg)](https://www.npmjs.com/package/@jeremiaheth/neolata-mem)
+[![bundle size](https://img.shields.io/bundlephobia/minzip/@jeremiaheth/neolata-mem)](https://bundlephobia.com/package/@jeremiaheth/neolata-mem)
 [![MIT License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
 [![Node.js](https://img.shields.io/badge/node-%3E%3D18-brightgreen.svg)](https://nodejs.org)
 
@@ -13,6 +15,58 @@ No Python. No Docker. No Neo4j. Just `npm install`.
 npm install @jeremiaheth/neolata-mem
 ```
 
+## Why neolata-mem?
+
+**Because AI agents need memory that works like actual memory** — connected, decaying, evolving. Not a flat vector database with timestamps. A living knowledge graph that links related facts, forgets what doesn't matter, and resolves contradictions automatically.
+
+## Architecture
+
+```
+┌─────────────────────────────────────────────────────────────────────┐
+│                      STORE WORKFLOW (A-MEM)                         │
+├─────────────────────────────────────────────────────────────────────┤
+│                                                                     │
+│  Text Input                                                         │
+│      │                                                              │
+│      ├──► [Embed] ──────────────┐                                 │
+│      │                           ▼                                  │
+│      │                    Find Related Memories                     │
+│      │                    (similarity > threshold)                  │
+│      │                           │                                  │
+│      │                           ▼                                  │
+│      │                    Auto-Link Bidirectionally                 │
+│      │                    (top N similar)                           │
+│      │                           │                                  │
+│      └────────────────────────── ┘                                  │
+│                                  │                                  │
+│                                  ▼                                  │
+│                           [Store in Graph]                          │
+│                          Memory + Links + Embedding                 │
+│                                                                     │
+└─────────────────────────────────────────────────────────────────────┘
+
+┌─────────────────────────────────────────────────────────────────────┐
+│                        DECAY CYCLE                                  │
+├─────────────────────────────────────────────────────────────────────┤
+│                                                                     │
+│  For each Memory:                                                   │
+│      │                                                              │
+│      ├──► Calculate Strength:                                      │
+│      │    • Base: importance (0.0-1.0)                             │
+│      │    • Age decay: exp(-age / half-life)                       │
+│      │    • Link reinforcement: +5% per link (max +30%)            │
+│      │    • Category stickiness: decisions 1.3x, preferences 1.4x  │
+│      │    • Access boost: +2% per touch (max +20%)                 │
+│      │                                                              │
+│      ├──► strength < 0.05 ? ──► DELETE                             │
+│      │                                                              │
+│      ├──► strength < 0.15 ? ──► ARCHIVE (remove embedding, keep)   │
+│      │                                                              │
+│      └──► Clean broken links                                       │
+│                                                                     │
+└─────────────────────────────────────────────────────────────────────┘
+```
+
 ## Quick Start (3 lines)
 
 ```javascript
@@ -292,6 +346,39 @@ Decay Cycle:
 | Zero-config start | ✅ | ❌ | ❌ | ❌ |
 | LLM optional | ✅ | ❌ | ❌ | ❌ |
 
+## Documentation
+
+Full documentation available at [neolata-mem docs](https://github.com/Jeremiaheth/neolata-mem/tree/main/docs-site).
+
+Run locally:
+```bash
+cd docs-site
+npm install
+npm start
+```
+
+## Contributing
+
+Contributions welcome! This is built by people who break things for a living.
+
+**Before submitting:**
+- Run tests: `npm test`
+- Keep the hacker aesthetic — no corporate fluff
+- Code examples must actually work
+- Break it before you ship it
+
+**Areas that need work:**
+- More storage backends (Supabase, Redis, SQLite)
+- Additional embedding providers
+- Performance optimizations for 100k+ memory graphs
+- Visualization tools
+
+Open an issue or PR. No bureaucracy.
+
+## Changelog
+
+See [CHANGELOG.md](CHANGELOG.md) for version history and breaking changes.
+
 ## License
 
 MIT — do whatever you want.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@jeremiaheth/neolata-mem",
-  "version": "0.2.0",
+  "version": "0.3.0",
   "description": "Graph-native memory engine for AI agents with Zettelkasten linking, biological decay, and conflict resolution",
   "type": "module",
   "main": "src/index.mjs",
@@ -22,7 +22,8 @@
     "README.md"
   ],
   "scripts": {
-    "test": "node --test test/*.test.mjs"
+    "test": "node --test test/*.test.mjs",
+    "lint": "echo 'no linter configured'"
   },
   "keywords": [
     "ai",
@@ -35,8 +36,7 @@
     "llm",
     "vector-search",
     "multi-agent",
-    "decay",
-    "conflict-resolution"
+    "decay"
   ],
   "author": "Jeremiaheth",
   "license": "MIT",
@@ -46,5 +46,7 @@
   },
   "engines": {
     "node": ">=18.0.0"
-  }
+  },
+  "dependencies": {},
+  "devDependencies": {}
 }

package/src/embeddings.mjs

@@ -1,13 +1,50 @@
 /**
- * Embedding provider interface and implementations.
- * All providers must implement: embed(texts) → number[][]
+ * @module embeddings
+ * @description Embedding provider interface and implementations.
+ * 
+ * All embedding providers must implement the following interface:
+ * ```typescript
+ * interface EmbeddingProvider {
+ *   name: string;
+ *   model: string | null;
+ *   embed(texts: string | string[]): Promise<(number[] | null)[]>;
+ * }
+ * ```
+ * 
+ * @example
+ * // OpenAI embeddings
+ * import { openaiEmbeddings } from '@jeremiaheth/neolata-mem/embeddings';
+ * const embedder = openaiEmbeddings({
+ *   apiKey: process.env.OPENAI_API_KEY,
+ *   model: 'text-embedding-3-small',
+ * });
+ * const vectors = await embedder.embed(['Hello world', 'Goodbye world']);
+ * 
+ * @example
+ * // No-op embeddings (keyword search only)
+ * import { noopEmbeddings } from '@jeremiaheth/neolata-mem/embeddings';
+ * const embedder = noopEmbeddings();
+ * const vectors = await embedder.embed(['text']); // → [null]
  */
 
 /**
- * Cosine similarity between two vectors.
- * @param {number[]} a
- * @param {number[]} b
- * @returns {number}
+ * @typedef {Object} EmbeddingProvider
+ * @property {string} name - Provider identifier
+ * @property {string|null} model - Model name or null
+ * @property {function(string|string[]): Promise<(number[]|null)[]>} embed - Embed text(s) to vectors
+ */
+
+/**
+ * Calculate cosine similarity between two embedding vectors.
+ * 
+ * @param {number[]} a - First vector
+ * @param {number[]} b - Second vector (must be same length as a)
+ * @returns {number} Similarity score (0.0 to 1.0, higher = more similar)
+ * @throws {Error} If vectors are different lengths or empty
+ * 
+ * @example
+ * const sim = cosineSimilarity([1, 0, 0], [1, 0, 0]); // → 1.0 (identical)
+ * const sim2 = cosineSimilarity([1, 0, 0], [0, 1, 0]); // → 0.0 (orthogonal)
  */
 export function cosineSimilarity(a, b) {
   let dot = 0, normA = 0, normB = 0;
@@ -19,21 +56,66 @@ export function cosineSimilarity(a, b) {
   return dot / (Math.sqrt(normA) * Math.sqrt(normB));
 }
 
-// ─── OpenAI-Compatible Provider ─────────────────────────────
 /**
- * Works with OpenAI, NVIDIA NIM, Ollama, Azure, any OpenAI-compatible API.
- * @param {object} opts
- * @param {string} opts.apiKey
- * @param {string} opts.model - e.g. 'text-embedding-3-small', 'baai/bge-m3'
- * @param {string} [opts.baseUrl='https://api.openai.com/v1'] - API base URL
- * @param {object} [opts.extraBody] - Extra body params (e.g. { input_type: 'passage' })
- * @param {number} [opts.retryMs=2000] - Retry delay on 429
+ * OpenAI-compatible embedding provider.
+ * 
+ * Works with any API that implements the OpenAI `/v1/embeddings` endpoint:
+ * - OpenAI (text-embedding-3-small, text-embedding-3-large)
+ * - NVIDIA NIM (baai/bge-m3, nvidia/nv-embed-v1)
+ * - Ollama (nomic-embed-text, mxbai-embed-large)
+ * - Azure OpenAI
+ * - Groq, Together, etc.
+ * 
+ * **Rate limiting:** Automatically retries on 429 with exponential backoff.
+ * 
+ * @param {Object} opts - Configuration options
+ * @param {string} opts.apiKey - API key for authentication
+ * @param {string} opts.model - Model identifier (e.g. 'text-embedding-3-small', 'baai/bge-m3')
+ * @param {string} [opts.baseUrl='https://api.openai.com/v1'] - API base URL (without /embeddings)
+ * @param {Object} [opts.extraBody={}] - Additional request body parameters (e.g. { input_type: 'passage' } for NIM)
+ * @param {number} [opts.retryMs=2000] - Base retry delay on 429 in milliseconds
+ * @param {number} [opts.maxRetries=3] - Maximum retry attempts on 429
+ * @returns {EmbeddingProvider} Configured embedding provider
+ * @throws {Error} If apiKey or model is missing
+ * 
+ * @example
+ * // OpenAI
+ * const emb = openaiEmbeddings({
+ *   apiKey: process.env.OPENAI_API_KEY,
+ *   model: 'text-embedding-3-small',
+ * });
+ * 
+ * @example
+ * // NVIDIA NIM (free tier)
+ * const emb = openaiEmbeddings({
+ *   apiKey: process.env.NVIDIA_API_KEY,
+ *   model: 'baai/bge-m3',
+ *   baseUrl: 'https://integrate.api.nvidia.com/v1',
+ *   extraBody: { input_type: 'passage', truncate: 'END' },
+ * });
+ * 
+ * @example
+ * // Local Ollama
+ * const emb = openaiEmbeddings({
+ *   apiKey: 'ollama', // Ollama ignores auth
+ *   model: 'nomic-embed-text',
+ *   baseUrl: 'http://localhost:11434/v1',
+ * });
  */
-export function openaiEmbeddings({ apiKey, model, baseUrl = 'https://api.openai.com/v1', extraBody = {}, retryMs = 2000 }) {
+export function openaiEmbeddings({ apiKey, model, baseUrl = 'https://api.openai.com/v1', extraBody = {}, retryMs = 2000, maxRetries = 3 }) {
+  if (!apiKey) throw new Error('openaiEmbeddings: apiKey is required');
+  if (!model) throw new Error('openaiEmbeddings: model is required');
   return {
     name: `openai-compatible(${model})`,
     model,
-    async embed(texts) {
+    /**
+     * Embed one or more texts.
+     * @param {string|string[]} texts - Text(s) to embed
+     * @param {number} [_attempt=0] - Internal retry counter
+     * @returns {Promise<number[][]>} Array of embedding vectors
+     * @throws {Error} On network failure, API error, or rate limit exceeded
+     */
+    async embed(texts, _attempt = 0) {
       const input = Array.isArray(texts) ? texts : [texts];
       const res = await fetch(`${baseUrl}/embeddings`, {
         method: 'POST',
@@ -41,8 +123,9 @@ export function openaiEmbeddings({ apiKey, model, baseUrl = 'https://api.openai.
         body: JSON.stringify({ model, input, ...extraBody }),
       });
       if (res.status === 429) {
-        await new Promise(r => setTimeout(r, retryMs));
-        return this.embed(texts);
+        if (_attempt >= maxRetries) throw new Error(`Embedding 429: rate limited after ${maxRetries} retries`);
+        await new Promise(r => setTimeout(r, retryMs * (_attempt + 1)));
+        return this.embed(texts, _attempt + 1);
       }
       if (!res.ok) throw new Error(`Embedding ${res.status}: ${await res.text()}`);
       const data = await res.json();
@@ -51,15 +134,31 @@ export function openaiEmbeddings({ apiKey, model, baseUrl = 'https://api.openai.
   };
 }
 
-// ─── Noop Provider (keyword-only mode) ──────────────────────
 /**
- * No-op embedding provider. Returns null embeddings.
- * Use this when you don't want/need vector search — keyword matching still works.
+ * No-op embedding provider for keyword-only search.
+ * 
+ * Returns `null` embeddings, which disables semantic search and falls back
+ * to substring matching. Useful for:
+ * - Testing without API dependencies
+ * - Offline/airgapped environments
+ * - Simple exact-match use cases
+ * 
+ * @param {Object} [opts] - Options (currently unused, reserved for future)
+ * @returns {EmbeddingProvider} No-op provider
+ * 
+ * @example
+ * const mem = createMemory({ embeddings: { type: 'noop' } });
+ * await mem.store('a', 'User prefers dark mode');
+ * const results = await mem.search('a', 'dark mode'); // Keyword match
  */
 export function noopEmbeddings() {
   return {
     name: 'noop',
     model: null,
+    /**
+     * @param {string|string[]} texts
+     * @returns {Promise<null[]>} Array of nulls
+     */
     async embed(texts) {
       const input = Array.isArray(texts) ? texts : [texts];
       return input.map(() => null);

package/src/extraction.mjs

@@ -1,22 +1,55 @@
 /**
- * Fact extraction providers.
- * All providers must implement: extract(text) → Fact[]
- * Fact = { fact: string, category: string, importance: number, tags: string[] }
+ * @module extraction
+ * @description Fact extraction providers for bulk ingestion.
+ * 
+ * All extraction providers must implement:
+ * ```typescript
+ * interface ExtractionProvider {
+ *   name: string;
+ *   extract(text: string): Promise<Fact[]>;
+ * }
+ * 
+ * interface Fact {
+ *   fact: string;
+ *   category: string;
+ *   importance: number;
+ *   tags: string[];
+ * }
+ * ```
+ * 
+ * @example
+ * // LLM extraction
+ * import { llmExtraction } from '@jeremiaheth/neolata-mem/extraction';
+ * const extractor = llmExtraction({
+ *   apiKey: process.env.OPENAI_API_KEY,
+ *   model: 'gpt-4.1-nano',
+ * });
+ * const facts = await extractor.extract('User deployed v2.1 with Redis caching on port 6379');
+ * // [{ fact: 'Deployed v2.1', category: 'event', importance: 0.7, tags: ['deployment'] }, ...]
  */
 
-// ─── LLM-Based Extraction (OpenAI-Compatible) ──────────────
+import { openaiChat } from './llm.mjs';
+
 /**
- * Uses any OpenAI-compatible chat API to extract atomic facts.
- * @param {object} opts
- * @param {string} opts.apiKey
- * @param {string} [opts.model='gpt-4.1-nano'] - Chat model
- * @param {string} [opts.baseUrl='https://api.openai.com/v1']
+ * @typedef {Object} Fact
+ * @property {string} fact - Atomic fact statement
+ * @property {string} category - Fact category (decision|finding|fact|insight|task|event|preference)
+ * @property {number} importance - Importance score 0.0-1.0
+ * @property {string[]} tags - Extracted keywords/tags
  */
-export function llmExtraction({ apiKey, model = 'gpt-4.1-nano', baseUrl = 'https://api.openai.com/v1' }) {
-  return {
-    name: `llm(${model})`,
-    async extract(text) {
-      const prompt = `You are a precise fact extractor. Extract discrete, atomic facts from the following text. Each fact should be self-contained and include specific details (names, numbers, dates, decisions, preferences).
+
+/**
+ * @typedef {Object} ExtractionProvider
+ * @property {string} name - Provider identifier
+ * @property {function(string): Promise<Fact[]>} extract - Extract facts from text
+ */
+
+/**
+ * LLM prompt for structured fact extraction.
+ * @const {string}
+ * @private
+ */
+const EXTRACT_PROMPT = `You are a precise fact extractor. Extract discrete, atomic facts from the following text. Each fact should be self-contained and include specific details (names, numbers, dates, decisions, preferences).
 
 Output as a JSON array of objects with fields:
 - "fact": the extracted fact (string)
@@ -25,43 +58,122 @@ Output as a JSON array of objects with fields:
 - "tags": array of relevant keywords
 
 Text to extract from:
-${text}
-
-Respond ONLY with the JSON array, no markdown formatting.`;
+`;
 
+/**
+ * LLM-based fact extraction (OpenAI-compatible API).
+ * 
+ * Sends text to an LLM with structured prompt to extract atomic facts.
+ * Falls back to single-fact passthrough on LLM failure.
+ * 
+ * **Recommended models:**
+ * - `gpt-4.1-nano` (fast, cheap)
+ * - `gpt-4o-mini`
+ * - `claude-3-haiku` (if using Anthropic)
+ * 
+ * **Cost:** ~$0.0001-0.001 per 1000 chars depending on model.
+ * 
+ * @param {Object} opts - Configuration options
+ * @param {string} opts.apiKey - API key for OpenAI-compatible endpoint
+ * @param {string} [opts.model='gpt-4.1-nano'] - Chat model for extraction
+ * @param {string} [opts.baseUrl='https://api.openai.com/v1'] - API base URL
+ * @returns {ExtractionProvider} Configured extraction provider
+ * @throws {Error} If apiKey is missing
+ * 
+ * @example
+ * // OpenAI
+ * const ext = llmExtraction({
+ *   apiKey: process.env.OPENAI_API_KEY,
+ *   model: 'gpt-4.1-nano',
+ * });
+ * 
+ * @example
+ * // Groq (fast inference)
+ * const ext = llmExtraction({
+ *   apiKey: process.env.GROQ_API_KEY,
+ *   model: 'llama-3.3-70b-versatile',
+ *   baseUrl: 'https://api.groq.com/openai/v1',
+ * });
+ * 
+ * @example
+ * // Use with ingest
+ * const mem = createMemory({
+ *   extraction: { type: 'llm', apiKey: KEY, model: 'gpt-4.1-nano' },
+ * });
+ * const result = await mem.ingest('agent', longDocument);
+ * // Extracted facts automatically stored with A-MEM linking
+ */
+export function llmExtraction({ apiKey, model = 'gpt-4.1-nano', baseUrl = 'https://api.openai.com/v1' }) {
+  if (!apiKey) throw new Error('llmExtraction: apiKey is required');
+  const chat = openaiChat({ apiKey, model, baseUrl, maxTokens: 2000, temperature: 0.1 });
+  return {
+    name: `llm(${model})`,
+    
+    /**
+     * Extract structured facts from text using LLM.
+     * @param {string} text - Input text (up to ~4000 tokens)
+     * @returns {Promise<Fact[]>} Array of extracted facts
+     * @throws Never throws — falls back to passthrough on failure
+     * 
+     * @example
+     * const facts = await extractor.extract(`
+     *   Meeting notes 2024-02-22:
+     *   - Decided to migrate to PostgreSQL
+     *   - Redis will handle session cache on port 6379
+     *   - Deploy v2.1 by Friday
+     * `);
+     * // [
+     * //   { fact: 'Decided to migrate to PostgreSQL', category: 'decision', importance: 0.9, tags: ['postgresql', 'migration'] },
+     * //   { fact: 'Redis handles session cache on port 6379', category: 'fact', importance: 0.7, tags: ['redis', 'cache'] },
+     * //   { fact: 'Deploy v2.1 by Friday', category: 'task', importance: 0.8, tags: ['deployment', 'deadline'] },
+     * // ]
+     */
+    async extract(text) {
       try {
-        const res = await fetch(`${baseUrl}/chat/completions`, {
-          method: 'POST',
-          headers: { 'Authorization': `Bearer ${apiKey}`, 'Content-Type': 'application/json' },
-          body: JSON.stringify({
-            model,
-            messages: [{ role: 'user', content: prompt }],
-            max_tokens: 2000,
-            temperature: 0.1,
-          }),
-        });
-        if (!res.ok) throw new Error(`LLM ${res.status}: ${await res.text()}`);
-        const data = await res.json();
-        const content = data.choices?.[0]?.message?.content || '';
+        const content = await chat.chat(`${EXTRACT_PROMPT}${text}\n\nRespond ONLY with the JSON array, no markdown formatting.`);
         const jsonStr = content.replace(/```json\n?/g, '').replace(/```\n?/g, '').trim();
         return JSON.parse(jsonStr);
       } catch (e) {
+        // Fallback: treat entire text as single fact
         return [{ fact: text, category: 'fact', importance: 0.5, tags: [] }];
       }
     },
   };
 }
 
-// ─── Passthrough Extraction (No LLM) ────────────────────────
 /**
- * Treats the entire input as a single fact. No LLM required.
- * @param {object} [opts]
- * @param {string} [opts.defaultCategory='fact']
- * @param {number} [opts.defaultImportance=0.5]
+ * Passthrough extraction (no LLM, treats input as single fact).
+ * 
+ * Wraps the entire input text as a single fact. Useful when:
+ * - Input is already atomic/structured
+ * - No LLM access (offline mode)
+ * - You want manual control over categories/importance
+ * 
+ * @param {Object} [opts] - Configuration
+ * @param {string} [opts.defaultCategory='fact'] - Default category for all facts
+ * @param {number} [opts.defaultImportance=0.5] - Default importance (0.0-1.0)
+ * @returns {ExtractionProvider} Passthrough provider
+ * 
+ * @example
+ * const ext = passthroughExtraction({ defaultCategory: 'event', defaultImportance: 0.7 });
+ * const facts = await ext.extract('Deployed v2.1');
+ * // [{ fact: 'Deployed v2.1', category: 'event', importance: 0.7, tags: [] }]
+ * 
+ * @example
+ * // Use with ingest when you control fact structure
+ * const mem = createMemory({
+ *   extraction: { type: 'passthrough' },
+ * });
+ * await mem.ingest('agent', 'Server runs on port 8080'); // Single fact
  */
 export function passthroughExtraction({ defaultCategory = 'fact', defaultImportance = 0.5 } = {}) {
   return {
     name: 'passthrough',
+    
+    /**
+     * @param {string} text
+     * @returns {Promise<Fact[]>}
+     */
     async extract(text) {
       return [{ fact: text, category: defaultCategory, importance: defaultImportance, tags: [] }];
     },