@jeremiaheth/neolata-mem 0.8.1 → 0.8.2

package/LICENSE

@@ -1,21 +1,89 @@
-MIT License
-
-Copyright (c) 2026 Jeremiaheth
-
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
+Elastic License 2.0 (ELv2)
+
+URL: https://www.elastic.co/licensing/elastic-license
+
+## Acceptance
+
+By using the software, you agree to all of the terms and conditions below.
+
+## Copyright License
+
+The licensor grants you a non-exclusive, royalty-free, worldwide,
+non-sublicensable, non-transferable license to use, copy, distribute, make
+available, and prepare derivative works of the software, in each case subject
+to the limitations and conditions below.
+
+## Limitations
+
+You may not provide the software to third parties as a hosted or managed
+service, where the service provides users with access to any substantial set
+of the features or functionality of the software.
+
+You may not move, change, disable, or circumvent the license key functionality
+in the software, and you may not remove or obscure any functionality in the
+software that is protected by the license key.
+
+You may not alter, remove, or obscure any licensing, copyright, or other
+notices of the licensor in the software. Any use of the licensor's trademarks
+is subject to applicable law.
+
+## Patents
+
+The licensor grants you a license, under any patent claims the licensor can
+license, or becomes able to license, to make, have made, use, sell, offer for
+sale, import and have imported the software, in each case subject to the
+limitations and conditions in this license. This license does not cover any
+patent claims that you cause to be infringed by modifications or additions to
+the software. If you or your company make any written claim that the software
+infringes or contributes to infringement of any patent, your patent license
+for the software granted under these terms ends immediately. If your company
+makes such a claim, your patent license ends immediately for work on behalf
+of your company.
+
+## Notices
+
+You must ensure that anyone who gets a copy of any part of the software from
+you also gets a copy of these terms.
+
+If you modify the software, you must include in any modified copies of the
+software prominent notices stating that you have modified the software.
+
+## No Other Rights
+
+These terms do not imply any licenses other than those expressly granted in
+these terms.
+
+## Termination
+
+If you use the software in violation of these terms, such use is not licensed,
+and your licenses will automatically terminate. If the licensor provides you
+with a notice of your violation, and you cease all violation of this license
+no later than 30 days after you receive that notice, your licenses will be
+reinstated retroactively. However, if you violate these terms after such
+reinstatement, any additional violation of these terms will cause your licenses
+to terminate automatically and permanently.
+
+## No Liability
+
+*As far as the law allows, the software comes as is, without any warranty or
+condition, and the licensor will not be liable to you for any damages arising
+out of these terms or the use or nature of the software, under any kind of
+legal claim.*
+
+## Definitions
+
+The **licensor** is the entity offering these terms, and the **software** is
+the software the licensor makes available under these terms, including any
+portion of it.
+
+**You** refers to the individual or entity agreeing to these terms.
+
+**Your company** is any legal entity, sole proprietorship, or other kind of
+organization that you work for, plus all organizations that have control over,
+are under the control of, or are under common control with that organization.
+**Control** means ownership of substantially all the assets of an entity, or
+the power to direct the management and policies of an entity.
+
+---
+
+Copyright 2025-2026 Jeremiaheth (Jeremiah Ojo)

package/README.md

@@ -2,7 +2,7 @@
 
 **Graph-native memory engine for AI agents.** Zettelkasten-inspired linking, biological decay, conflict resolution.
 
-[![MIT License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
+[![Elastic License 2.0](https://img.shields.io/badge/license-Elastic--2.0-blue.svg)](LICENSE)
 [![Node.js](https://img.shields.io/badge/node-%3E%3D18-brightgreen.svg)](https://nodejs.org)
 
 ---
@@ -37,6 +37,7 @@ const mem = createMemory({
   },
 });
 
+// Agent IDs like 'kuro' and 'maki' below are just examples — use any string to identify your agents.
 await mem.store('kuro', 'Found XSS vulnerability in login form', { category: 'finding', importance: 0.9 });
 await mem.store('kuro', 'OWASP Top 10 audit completed', { category: 'event' });
 
@@ -68,7 +69,7 @@ embeddings: {
 
 ### 🔗 A-MEM Zettelkasten Linking
 
-Every memory automatically links to related memories — bidirectionally. When you store "Redis runs on port 6379", it finds existing memories about Redis, ports, or databases and creates links in both directions.
+Every memory automatically links to related memories - bidirectionally. When you store "Redis runs on port 6379", it finds existing memories about Redis, ports, or databases and creates links in both directions.
 
 ```javascript
 await mem.store('a', 'Redis runs on port 6379');
@@ -110,14 +111,14 @@ await mem.evolve('a', 'Server now runs on port 8080');
 // Old version archived with evolution history
 ```
 
-**Quarantine lane** — low-trust or structurally conflicting memories are quarantined instead of auto-superseding:
+**Quarantine lane** - low-trust or structurally conflicting memories are quarantined instead of auto-superseding:
 
 ```javascript
 // Store with claim metadata and provenance
 await mem.store('a', 'Server runs on port 443', {
   claim: { subject: 'server', predicate: 'port', value: '443' },
   provenance: { source: 'user_explicit', trust: 1.0 },
-  onConflict: 'quarantine',  // default — quarantine low-trust conflicts
+  onConflict: 'quarantine',  // default - quarantine low-trust conflicts
 });
 
 // Review quarantined memories
@@ -207,7 +208,7 @@ const mem = createMemory({
     dir: './my-data',     // Custom directory for JSON storage
   },
 
-  // Embeddings (optional — keyword search works without)
+  // Embeddings (optional - keyword search works without)
   embeddings: {
     type: 'openai',       // 'openai' (any compatible API) | 'noop' (keyword only)
     apiKey: '...',
@@ -216,7 +217,7 @@ const mem = createMemory({
     extraBody: {},        // Extra params (e.g. { input_type: 'passage' } for NIM)
   },
 
-  // Fact extraction (optional — enables ingest())
+  // Fact extraction (optional - enables ingest())
   extraction: {
     type: 'llm',          // 'llm' | 'passthrough'
     apiKey: '...',
@@ -224,7 +225,7 @@ const mem = createMemory({
     baseUrl: 'https://api.openai.com/v1',
   },
 
-  // LLM for conflict resolution (optional — enables evolve())
+  // LLM for conflict resolution (optional - enables evolve())
   llm: {
     type: 'openai',
     apiKey: '...',
@@ -323,7 +324,7 @@ Set `OPENAI_API_KEY` or `NVIDIA_API_KEY` for embedding support. See `npx @jeremi
 
 ### `createMemory(opts?) → MemoryGraph`
 
-Factory function. All options are optional — zero-config returns a working instance with JSON storage and keyword search.
+Factory function. All options are optional - zero-config returns a working instance with JSON storage and keyword search.
 
 ### Core Methods
 
@@ -419,6 +420,48 @@ Factory function. All options are optional — zero-config returns a working ins
 | `pendingConflicts()` | List unresolved structural conflicts |
 | `resolveConflict(conflictId, opts)` | Resolve a pending conflict |
 
+### Runtime Helpers
+
+Convenience functions for agent workflows — heartbeat auto-store, contextual recall, and pre-compaction dumps.
+
+| Function | Description |
+|----------|-------------|
+| `detectKeyMoments(text, opts?)` | Extract decisions, preferences, commitments, and blockers from text |
+| `extractTopicSlug(text, opts?)` | Derive a topic slug from text (with optional synonym mapping) |
+| `heartbeatStore(mem, agent, turns, config?)` | Auto-store key moments from conversation turns on a heartbeat interval |
+| `contextualRecall(mem, agent, seedText, config?)` | Budget-aware recall: merges recent + semantic + high-importance memories by topic |
+| `preCompactionDump(mem, agent, turns, config?)` | Extract and persist takeaways before context window compaction |
+
+```javascript
+import { createMemory, heartbeatStore, contextualRecall, preCompactionDump } from '@jeremiaheth/neolata-mem';
+
+const mem = createMemory({ /* ... */ });
+
+// Heartbeat: auto-store key moments from recent turns
+const result = await heartbeatStore(mem, 'kuro', conversationTurns, {
+  sessionId: 'sess-123',
+  topicSlug: 'deployment',
+  minNewTurns: 3,         // skip if fewer than 3 new turns (default: 3)
+  lastStoredIndex: -1,    // track position across calls
+});
+// → { stored: 2, ids: [...], lastIndex: 14, moments: [...] }
+
+// Contextual recall: topic-aware, budget-capped context retrieval
+const context = await contextualRecall(mem, 'kuro', 'How did we fix the RLS issue?', {
+  maxTokens: 2000,        // token budget (default: 2000)
+  semanticCount: 8,       // semantic search results (default: 8)
+  importanceThreshold: 0.8,
+});
+// → { topicSlug: 'rls', memories: [...], totalTokens: 1823, excluded: 3 }
+
+// Pre-compaction dump: persist session takeaways before context reset
+const dump = await preCompactionDump(mem, 'kuro', conversationTurns, {
+  sessionId: 'sess-123',
+  maxTakeaways: 10,       // max individual moments to store (default: 10)
+});
+// → { takeaways: 4, snapshotId: '...', ids: [...] }
+```
+
 ### Advanced: Bring Your Own Providers
 
 ```javascript
@@ -482,6 +525,7 @@ Decay Cycle:
 | Conflict resolution | ✅ | ✅ | ❌ | ❌ |
 | Quarantine lane | ✅ | ❌ | ❌ | ❌ |
 | Predicate schemas | ✅ | ❌ | ❌ | ❌ |
+| Runtime helpers (heartbeat/recall/dump) | ✅ | ❌ | ❌ | ❌ |
 | Explainability API | ✅ | ❌ | ❌ | ❌ |
 | Episodes & compression | ✅ | ❌ | ❌ | ❌ |
 | Labeled clusters | ✅ | ❌ | ❌ | ❌ |
@@ -496,7 +540,7 @@ neolata-mem includes several hardening measures:
 
 - **Input validation**: Agent names (alphanumeric, max 64 chars), memory text (max 10KB), bounded total memory count (default 50K)
 - **Prompt injection mitigation**: All user content is XML-fenced in LLM prompts with explicit instruction boundaries. LLM output is structurally validated (type checks, index bounds, category whitelists)
-- **SSRF protection**: All provider URLs validated via `validateBaseUrl()` — blocks cloud metadata endpoints, private IP ranges (configurable), non-HTTP protocols
+- **SSRF protection**: All provider URLs validated via `validateBaseUrl()` - blocks cloud metadata endpoints, private IP ranges (configurable), non-HTTP protocols
 - **Supabase hardening**: UUID validation on all query params (prevents PostgREST injection), error text sanitized (strips tokens/keys), safe upsert-based save (no data loss on crash), automatic 429 retry with backoff
 - **Atomic writes**: JSON storage uses write-to-temp + rename to prevent corruption from concurrent access
 - **Path traversal guards**: Storage directories and write-through paths validated with `resolve()` + prefix checks
@@ -504,12 +548,12 @@ neolata-mem includes several hardening measures:
 - **Retry bounds**: Embedding and Supabase API retries are capped at 3 with exponential backoff (no infinite loops)
 - **Error surfacing**: Failed conflict detection returns `{ error }` instead of silently proceeding
 
-**Trust model**: For JSON storage, neolata-mem trusts the filesystem — protect your data directory. For Supabase, use Row Level Security (RLS) policies. Embedding vectors can approximate original text via inversion attacks — treat them as sensitive.
+**Trust model**: For JSON storage, neolata-mem trusts the filesystem - protect your data directory. For Supabase, use Row Level Security (RLS) policies. Embedding vectors can approximate original text via inversion attacks - treat them as sensitive.
 
 ## Documentation
 
-📖 **[Full User Guide](docs/guide.md)** — configuration deep dive, embedding providers, storage backends, recipes, troubleshooting, architecture.
+📖 **[Full User Guide](docs/guide.md)** - configuration deep dive, embedding providers, storage backends, recipes, troubleshooting, architecture.
 
 ## License
 
-MIT — do whatever you want.
+[Elastic License 2.0](LICENSE) — free to use, modify, and distribute. You just can't offer it as a hosted/managed service.

package/docs/guide.md

@@ -25,8 +25,9 @@
 17. [CLI Reference](#cli-reference)
 18. [OpenClaw Integration](#openclaw-integration)
 19. [Recipes](#recipes)
-20. [Troubleshooting](#troubleshooting)
-21. [Architecture](#architecture)
+20. [Runtime Helpers](#runtime-helpers)
+21. [Troubleshooting](#troubleshooting)
+22. [Architecture](#architecture)
 
 ---
 
@@ -260,10 +261,12 @@ First-class Supabase backend with incremental operations and server-side vector
 storage: {
   type: 'supabase',
   url: process.env.SUPABASE_URL,
-  key: process.env.SUPABASE_SERVICE_KEY,
+  key: process.env.SUPABASE_KEY,  // Prefer anon key + RLS; service key bypasses row-level security
 }
 ```
 
+> ⚠️ **Security:** Prefer a Supabase anon/public key with Row-Level Security (RLS) policies. A service key grants full database access and should only be used for admin operations, never in client-facing agents.
+
 **Setup:** Run `sql/schema.sql` in your Supabase Dashboard SQL Editor to create the required tables.
 
 **Features:**
@@ -329,6 +332,8 @@ This lets you back neolata-mem with PostgreSQL, SQLite, Redis, S3 — anything.
 
 ### Storing
 
+> **Note:** Agent IDs like `'kuro'` and `'maki'` used throughout this guide are just examples — use any string to identify your agents.
+
 ```javascript
 const result = await mem.store('kuro', 'Found XSS in login form', {
   category: 'finding',    // finding | decision | fact | insight | task | event | preference
@@ -1125,7 +1130,7 @@ const mem = createMemory({
   storage: {
     type: 'supabase',
     url: process.env.SUPABASE_URL,
-    key: process.env.SUPABASE_SERVICE_KEY,
+    key: process.env.SUPABASE_KEY,  // Prefer anon key + RLS
   },
   embeddings: {
     type: 'openai',
@@ -1148,6 +1153,8 @@ const results = await mem.search('kuro', 'dark mode');
 
 ### Write-through to webhooks
 
+> ⚠️ **Security:** Webhook URLs are an explicit data exfiltration surface. Each store/decay event sends memory content to the configured endpoint. Only configure URLs you trust and control.
+
 ```javascript
 import { createMemory, webhookWritethrough } from '@jeremiaheth/neolata-mem';
 
@@ -1166,6 +1173,172 @@ detach();
 
 ---
 
+## Runtime Helpers
+
+Standalone functions for common agent workflows. These work with any `MemoryGraph` instance and don't require special configuration.
+
+```javascript
+import {
+  detectKeyMoments, extractTopicSlug,
+  heartbeatStore, contextualRecall, preCompactionDump,
+} from '@jeremiaheth/neolata-mem';
+```
+
+### detectKeyMoments(text, opts?)
+
+Scans text for decisions, preferences, commitments, and blockers using pattern matching.
+
+```javascript
+const moments = detectKeyMoments("Decision: we're going with Supabase. Blocked by RLS permissions.");
+// [
+//   { type: 'decision', text: "Decision: we're going with Supabase", importance: 0.9 },
+//   { type: 'blocker', text: "Blocked by RLS permissions", importance: 0.85 },
+// ]
+```
+
+**Moment types and importance:**
+
+| Type | Importance | Trigger patterns |
+|------|-----------|-----------------|
+| `decision` | 0.9 | "Decision:", "We decided", "Going with", "Let's do", "Ship it" |
+| `commitment` | 0.8 | "I will", "We will", "TODO:", "Action item:" |
+| `blocker` | 0.85 | "Blocked by", "Blocker:", "Can't proceed", "Waiting on" |
+| `preference` | 0.7 | "I prefer", "I like", "I want", "Always use" |
+
+### extractTopicSlug(text, opts?)
+
+Derives a topic slug from text by finding the most frequent non-stop-word. Supports synonym mapping.
+
+```javascript
+extractTopicSlug('How did we fix the RLS policy issue?');
+// → 'rls'
+
+extractTopicSlug('Update the neolata package', {
+  synonyms: { 'neolata-mem': ['neolata', 'memory', 'mem'] }
+});
+// → 'neolata-mem'
+```
+
+### heartbeatStore(mem, agent, turns, config?)
+
+Auto-stores key moments from conversation turns. Designed to be called periodically (e.g., on a heartbeat timer). If no key moments are detected, stores a truncated session snapshot instead.
+
+**Config:**
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `sessionId` | string | — | Tag stored memories with session ID |
+| `topicSlug` | string | — | Tag with topic |
+| `projectSlug` | string | — | Tag with project |
+| `minNewTurns` | number | 3 | Skip if fewer new turns since last call |
+| `lastStoredIndex` | number | -1 | Index of last processed turn (track across calls) |
+
+**Returns:** `{ stored, ids, lastIndex, moments, skipped? }`
+
+```javascript
+const turns = [
+  { role: 'user', content: 'Let\'s do the Supabase migration' },
+  { role: 'assistant', content: 'Decision: migrating to Supabase for production storage.' },
+  { role: 'user', content: 'Ship it' },
+  { role: 'assistant', content: 'TODO: run the migration script on OCI.' },
+];
+
+let lastIndex = -1;
+const result = await heartbeatStore(mem, 'kuro', turns, {
+  sessionId: 'sess-abc',
+  topicSlug: 'supabase',
+  lastStoredIndex: lastIndex,
+});
+// → { stored: 3, ids: [...], lastIndex: 3, moments: [...] }
+
+lastIndex = result.lastIndex; // track for next heartbeat call
+```
+
+### contextualRecall(mem, agent, seedText, config?)
+
+Budget-aware context retrieval that merges three recall strategies: recent memories, semantic search results, and high-importance memories filtered by topic. Results are deduplicated, sorted by importance, and capped to a token budget.
+
+**Config:**
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `maxTokens` | number | 2000 | Token budget for returned memories |
+| `recentCount` | number | 5 | Number of recent memories to fetch |
+| `semanticCount` | number | 8 | Number of semantic search results |
+| `importantCount` | number | 10 | Candidates for importance filtering |
+| `importanceThreshold` | number | 0.8 | Minimum importance for the "important" lane |
+| `synonyms` | object | {} | Synonym map for topic extraction |
+
+**Returns:** `{ topicSlug, memories, totalTokens, excluded }`
+
+```javascript
+const ctx = await contextualRecall(mem, 'kuro', 'What was our RLS fix?', {
+  maxTokens: 1500,
+  importanceThreshold: 0.7,
+});
+
+console.log(ctx.topicSlug);    // 'rls'
+console.log(ctx.memories);     // [...] deduplicated, importance-sorted, budget-capped
+console.log(ctx.totalTokens);  // 1342
+console.log(ctx.excluded);     // 5 (didn't fit in budget)
+```
+
+### preCompactionDump(mem, agent, turns, config?)
+
+Extracts key moments from a full conversation, deduplicates them, persists the top takeaways, and stores a structured session snapshot. Call this before context window compaction to preserve important information.
+
+**Config:**
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `sessionId` | string | — | Tag stored memories with session ID |
+| `topicSlug` | string | — | Tag with topic |
+| `projectSlug` | string | — | Tag with project |
+| `maxTakeaways` | number | 10 | Maximum individual moments to persist |
+
+**Returns:** `{ takeaways, snapshotId, ids }`
+
+The session snapshot is a markdown-formatted summary grouped by type:
+
+```
+## Session Snapshot
+**Decisions:** migrating to Supabase for production storage
+**Open threads:** Blocked by RLS permissions on link inserts
+**Commitments:** TODO: run the migration script on OCI
+**Preferences:** I prefer service key over anon key for writes
+```
+
+```javascript
+const dump = await preCompactionDump(mem, 'kuro', allTurns, {
+  sessionId: 'sess-abc',
+  maxTakeaways: 10,
+});
+// → { takeaways: 4, snapshotId: 'uuid-...', ids: [...] }
+```
+
+### Workflow: Combining All Three
+
+A typical agent lifecycle using all three helpers:
+
+```javascript
+// 1. On session start: recall context
+const ctx = await contextualRecall(mem, agent, userMessage);
+// → inject ctx.memories into system prompt
+
+// 2. Periodically during conversation: heartbeat store
+let lastIdx = -1;
+setInterval(async () => {
+  const r = await heartbeatStore(mem, agent, turns, { lastStoredIndex: lastIdx });
+  lastIdx = r.lastIndex;
+}, 60_000);
+
+// 3. Before compaction: dump takeaways
+const dump = await preCompactionDump(mem, agent, turns);
+// → key moments and snapshot persisted to graph
+```
+
+---
+
 ## Troubleshooting
 
 ### "No results from search"