memshell 0.2.1 → 0.4.0

package/README.md

@@ -2,7 +2,7 @@
 
 <h1>mem.sh</h1>
 
-<p><strong>Persistent memory for AI agents.</strong><br>One line to save. One line to recall.</p>
+<p><strong>Persistent memory for AI agents.</strong><br>One line to save. One line to recall. Auto-ingest conversations.</p>
 
 [![npm version](https://img.shields.io/npm/v/memshell.svg?style=flat-square)](https://www.npmjs.com/package/memshell)
 [![license](https://img.shields.io/npm/l/memshell.svg?style=flat-square)](https://github.com/justedv/mem.sh/blob/main/LICENSE)
@@ -10,7 +10,7 @@
 
 <br>
 
-[Quick Start](#quick-start) · [SDK](#sdk) · [API Server](#api-server) · [Architecture](#how-it-works) · [Contributing](CONTRIBUTING.md)
+[Quick Start](#quick-start) · [Auto-Ingest](#auto-ingest) · [OpenClaw Integration](#openclaw-integration) · [SDK](#sdk) · [API Server](#api-server) · [Architecture](#how-it-works)
 
 </div>
 
@@ -29,17 +29,20 @@ Agents forget everything between sessions. **mem.sh** gives them a brain.
 | | mem.sh | LangChain Memory | Roll your own |
 |---|---|---|---|
 | **Setup** | `npx memshell set "..."` | 47 dependencies + config | Hours of boilerplate |
-| **External APIs** | None | OpenAI key required | Depends |
+| **Auto-ingest** | Built-in | No | You build it |
+| **External APIs** | None (optional) | OpenAI key required | Depends |
 | **Semantic search** | Built-in TF-IDF | Embedding models | You build it |
 | **Storage** | SQLite (local) | Varies | You choose |
-| **Lines of code** | ~1 | ~50+ | ~200+ |
 
 ## Features
 
-- **Fast** — TF-IDF vectorization with cosine similarity, instant results
-- **Local-first** — SQLite storage at `~/.mem/mem.db`, no data leaves your machine
-- **Semantic** — Recall by meaning, not exact match
-- **Zero config** — `npx` and go. No API keys, no setup, no dependencies
+- **Fast** -- TF-IDF vectorization with cosine similarity, instant results
+- **Local-first** -- SQLite storage at `~/.mem/mem.db`, no data leaves your machine
+- **Semantic** -- Recall by meaning, not exact match
+- **Auto-ingest** -- Feed raw conversations, auto-extract key facts via LLM
+- **OpenClaw integration** -- Watch session transcripts and auto-learn
+- **Zero config** -- `npx` and go. No API keys needed for core features
+- **Smart recall** -- Shows source, creation time, and recall frequency
 
 ## Quick Start
 
@@ -51,7 +54,7 @@ npx memshell set "user prefers dark mode"
 
 # Recall semantically
 npx memshell recall "what theme does the user like?"
-# → user prefers dark mode (score: 0.87)
+# => user prefers dark mode (score: 0.87)
 
 # List all memories
 npx memshell list
@@ -63,7 +66,92 @@ npx memshell forget <id>
 npx memshell clear
 ```
 
-### SDK
+## Auto-Ingest
+
+Feed raw conversations and let the LLM extract key facts automatically.
+
+Requires `OPENAI_API_KEY` or `ANTHROPIC_API_KEY` (or configure via `memshell config set apiKey <key>`).
+
+### From a file
+
+```bash
+npx memshell ingest conversation.txt
+npx memshell ingest chat.jsonl
+npx memshell ingest notes.md
+```
+
+### From stdin
+
+```bash
+echo "User said they prefer dark mode and use vim" | npx memshell ingest --stdin
+```
+
+### Watch a directory
+
+```bash
+npx memshell ingest --watch ./logs/
+```
+
+Watches for new or changed `.txt`, `.md`, `.json`, and `.jsonl` files. Tracks what has been processed to avoid duplicates.
+
+### Via API
+
+```bash
+curl -X POST http://localhost:3456/mem/ingest \
+  -H "Content-Type: application/json" \
+  -d '{"text": "User mentioned they love Rust and prefer dark themes"}'
+# => {"extracted": 2, "stored": 2, "duplicates": 0}
+```
+
+### How it works
+
+1. Text is split into ~2000-token chunks
+2. Each chunk is sent to an LLM (gpt-4o-mini or claude-3-haiku) to extract standalone facts
+3. Facts are deduplicated against existing memories (Jaccard similarity > 0.85 = skip)
+4. New facts are stored with auto-generated tags and source tracking
+
+## OpenClaw Integration
+
+Automatically learn from your OpenClaw agent conversations:
+
+```bash
+# Start watching OpenClaw session transcripts
+npx memshell connect openclaw
+
+# Or specify a custom path
+npx memshell connect openclaw /path/to/sessions/
+```
+
+This watches the OpenClaw sessions directory (`~/.openclaw/agents/main/sessions/` by default), parses JSONL transcripts, and auto-ingests new conversations.
+
+### Daemon mode
+
+Run continuous ingestion in the background:
+
+```bash
+# Configure watchers first
+npx memshell config set watch.openclaw ~/.openclaw/agents/main/sessions/
+
+# Start the daemon
+npx memshell daemon
+```
+
+### Configuration
+
+```bash
+# Set LLM API key
+npx memshell config set apiKey sk-...
+
+# Set model
+npx memshell config set model gpt-4o-mini
+
+# View config
+npx memshell config get
+```
+
+Config is stored at `~/.mem/config.json`.
+
+## SDK
 
 ```js
 const mem = require('memshell');
@@ -72,9 +160,9 @@ const mem = require('memshell');
 await mem.set('user prefers dark mode');
 await mem.set('favorite language is rust', { agent: 'coder-bot' });
 
-// Recall (semantic search)
+// Recall (semantic search) -- now includes source and recall count
 const results = await mem.recall('what does the user like?');
-// [{ id, text, score, created_at }]
+// [{ id, text, score, created_at, source, recall_count }]
 
 // List all
 const all = await mem.list();
@@ -92,7 +180,7 @@ mem.sh uses **TF-IDF vectorization** with **cosine similarity** for semantic sea
 
 Memories are stored in `~/.mem/mem.db` (SQLite). Each memory is tokenized and vectorized on write. Queries are vectorized at recall time and ranked by cosine similarity against stored vectors.
 
-→ [Full architecture docs](docs/ARCHITECTURE.md)
+Optional: Enable OpenAI embeddings with `--embeddings` flag for higher quality recall (requires `OPENAI_API_KEY`).
 
 ## API Server
 
@@ -107,29 +195,19 @@ npx memshell serve --port 3456 --key my-secret-key
 | Method | Path | Description |
 |--------|------|-------------|
 | `POST` | `/mem` | Store a memory |
+| `POST` | `/mem/ingest` | Auto-ingest raw text |
 | `GET` | `/mem/recall?q=` | Semantic recall |
 | `GET` | `/mem/list` | List all memories |
+| `GET` | `/mem/stats` | Memory statistics |
+| `GET` | `/mem/export` | Export all memories |
+| `POST` | `/mem/import` | Import memories |
 | `DELETE` | `/mem/:id` | Delete a memory |
 | `DELETE` | `/mem` | Clear all memories |
 
 ### Headers
 
-- `X-Mem-Key` — API key (required if `--key` is set)
-- `X-Mem-Agent` — Agent namespace (optional, isolates memories per agent)
-
-### Example
-
-```bash
-# Store
-curl -X POST http://localhost:3456/mem \
-  -H "Content-Type: application/json" \
-  -H "X-Mem-Key: my-secret-key" \
-  -d '{"text": "user prefers dark mode"}'
-
-# Recall
-curl "http://localhost:3456/mem/recall?q=theme+preference" \
-  -H "X-Mem-Key: my-secret-key"
-```
+- `X-Mem-Key` -- API key (required if `--key` is set)
+- `X-Mem-Agent` -- Agent namespace (optional, isolates memories per agent)
 
 ### SDK with API Mode
 
@@ -146,9 +224,27 @@ await mem.set('user prefers dark mode');
 const results = await mem.recall('theme preference');
 ```
 
-## Contributing
+## All CLI Commands
 
-We welcome contributions. See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
+```
+memshell set <text>              Store a memory
+memshell recall <query>          Semantic recall
+memshell list                    List all memories
+memshell forget <id>             Delete a memory by ID
+memshell clear                   Wipe all memories
+memshell important <id>          Boost memory importance
+memshell ingest <file>           Extract facts from a file
+memshell ingest --stdin          Extract facts from piped text
+memshell ingest --watch <dir>    Watch directory for new files
+memshell connect openclaw        Watch OpenClaw transcripts
+memshell daemon                  Run continuous ingestion
+memshell config set <key> <val>  Set config value
+memshell config get [key]        Show config
+memshell stats                   Show memory statistics
+memshell export                  Export all memories as JSON
+memshell import <file.json>      Import memories from JSON
+memshell serve [--port N]        Start API server
+```
 
 ## License
 

package/bin/mem.js

@@ -1,89 +1,300 @@
 #!/usr/bin/env node
 'use strict';
 
+const fs = require('fs');
+const path = require('path');
 const mem = require('../src/index');
+const { LocalStore } = require('../src/index');
 
 const args = process.argv.slice(2);
 const cmd = args[0];
-const rest = args.slice(1).join(' ');
 
 const HELP = `
-  mem.sh — persistent memory for AI agents
+  \x1b[1mmem.sh\x1b[0m — persistent memory for AI agents
 
-  Usage:
-    mem set <text>          Store a memory
-    mem recall <query>      Semantic recall
-    mem list                List all memories
-    mem forget <id>         Delete a memory by ID
-    mem clear               Wipe all memories
-    mem serve [--port N]    Start API server
+  \x1b[36mCore Commands:\x1b[0m
+    memshell set <text>              Store a memory
+    memshell recall <query>          Semantic recall
+    memshell list                    List all memories
+    memshell forget <id>             Delete a memory by ID
+    memshell clear                   Wipe all memories
+    memshell important <id>          Boost memory importance
 
-  Options:
+  \x1b[36mAuto-Ingest:\x1b[0m
+    memshell ingest <file>           Extract facts from a file
+    memshell ingest --stdin          Extract facts from piped text
+    memshell ingest --watch <dir>    Watch a directory for new files
+
+  \x1b[36mIntegrations:\x1b[0m
+    memshell connect openclaw        Watch OpenClaw session transcripts
+    memshell daemon                  Run continuous ingestion daemon
+
+  \x1b[36mManagement:\x1b[0m
+    memshell config set <key> <val>  Set config value
+    memshell config get [key]        Show config
+    memshell stats                   Show memory statistics
+    memshell export                  Export all memories as JSON
+    memshell import <file.json>      Import memories from JSON
+    memshell serve [--port N]        Start API server
+
+  \x1b[36mOptions:\x1b[0m
     --agent <name>          Agent namespace
     --api <url>             Use remote API instead of local
     --key <key>             API key for remote server
+    --tags <t1,t2>          Tags (comma-separated)
+    --top <N>               Return top N results only
+    --embeddings            Enable OpenAI embeddings (needs OPENAI_API_KEY)
 
-  Examples:
-    mem set "user prefers dark mode"
-    mem recall "what theme?"
-    mem list
-    mem forget 3
+  \x1b[36mExamples:\x1b[0m
+    memshell set "user prefers dark mode" --tags preferences,ui
+    memshell recall "what theme?" --tags preferences --top 3
+    echo "User likes vim and dark mode" | memshell ingest --stdin
+    memshell connect openclaw
+    memshell config set apiKey sk-...
 `;
 
 // Parse flags
 function flag(name) {
   const i = args.indexOf('--' + name);
   if (i === -1) return null;
-  return args[i + 1] || true;
+  if (i + 1 < args.length && !args[i + 1].startsWith('--')) return args[i + 1];
+  return true;
+}
+
+function hasFlag(name) {
+  return args.includes('--' + name);
+}
+
+// Smarter text extraction: skip flag values
+function getText() {
+  const skip = new Set(['--agent', '--api', '--key', '--tags', '--top', '--port', '--watch']);
+  const parts = [];
+  let i = 1;
+  while (i < args.length) {
+    if (skip.has(args[i])) { i += 2; continue; }
+    if (args[i] === '--embeddings' || args[i] === '--stdin' || args[i] === '--force') { i++; continue; }
+    if (args[i].startsWith('--')) { i++; continue; }
+    parts.push(args[i]);
+    i++;
+  }
+  return parts.join(' ').replace(/^["']|["']$/g, '');
+}
+
+function readStdin() {
+  return new Promise((resolve) => {
+    let data = '';
+    process.stdin.setEncoding('utf8');
+    process.stdin.on('data', chunk => data += chunk);
+    process.stdin.on('end', () => resolve(data));
+    // If nothing after 100ms and stdin is a TTY, resolve empty
+    if (process.stdin.isTTY) resolve('');
+  });
 }
 
 async function main() {
   const agent = flag('agent') || 'default';
   const api = flag('api');
   const key = flag('key');
+  const tags = flag('tags') || '';
+  const top = flag('top') ? parseInt(flag('top')) : null;
+  const useEmbeddings = hasFlag('embeddings');
+
+  const configOpts = { agent };
+  if (api) { configOpts.api = api; configOpts.key = key; }
+  if (useEmbeddings) { configOpts.openaiKey = process.env.OPENAI_API_KEY; }
 
-  if (api) mem.configure({ api, key, agent });
-  else mem.configure({ agent });
+  mem.configure(configOpts);
 
-  const opts = { agent };
+  const opts = { agent, tags, top };
 
   switch (cmd) {
     case 'set': case 's': case 'save': case 'remember': {
-      const text = args.slice(1).filter(a => !a.startsWith('--')).join(' ').replace(/^["']|["']$/g, '');
-      if (!text) return console.log('Usage: mem set <text>');
-      const r = await mem.set(text, opts);
-      console.log(`✓ Stored (id: ${r.id})`);
+      const text = getText();
+      if (!text) return console.log('Usage: memshell set <text>');
+      const r = await mem.set(text, { ...opts, tags });
+      console.log(`\x1b[32m+\x1b[0m Stored (id: \x1b[1m${r.id}\x1b[0m)${tags ? ` [tags: ${tags}]` : ''}`);
       break;
     }
     case 'recall': case 'r': case 'search': case 'q': {
-      const query = args.slice(1).filter(a => !a.startsWith('--')).join(' ').replace(/^["']|["']$/g, '');
-      if (!query) return console.log('Usage: mem recall <query>');
+      const query = getText();
+      if (!query) return console.log('Usage: memshell recall <query>');
       const results = await mem.recall(query, opts);
-      if (!results.length) return console.log('No memories found.');
+      if (!results.length) return console.log('\x1b[33mNo memories found.\x1b[0m');
       for (const r of results) {
-        console.log(`  [${r.id}] ${r.text} (score: ${r.score})`);
+        const tagStr = r.tags ? ` \x1b[35m[${r.tags}]\x1b[0m` : '';
+        const srcStr = r.source && r.source !== 'manual' ? ` \x1b[2m(src: ${r.source})\x1b[0m` : '';
+        const recallStr = r.recall_count ? ` \x1b[2m(recalled ${r.recall_count}x)\x1b[0m` : '';
+        console.log(`  \x1b[36m[${r.id}]\x1b[0m ${r.text} \x1b[33m(score: ${r.score})\x1b[0m${tagStr}${srcStr}${recallStr}`);
       }
       break;
     }
     case 'list': case 'ls': case 'l': {
       const all = await mem.list(opts);
-      if (!all.length) return console.log('No memories stored.');
+      if (!all.length) return console.log('\x1b[33mNo memories stored.\x1b[0m');
       for (const r of all) {
-        console.log(`  [${r.id}] ${r.text}  (${r.created_at})`);
+        const tagStr = r.tags ? ` \x1b[35m[${r.tags}]\x1b[0m` : '';
+        const imp = r.importance !== 1.0 ? ` \x1b[33m*${r.importance.toFixed(1)}\x1b[0m` : '';
+        const srcStr = r.source && r.source !== 'manual' ? ` \x1b[2m[${r.source}]\x1b[0m` : '';
+        console.log(`  \x1b[36m[${r.id}]\x1b[0m ${r.text}${tagStr}${imp}${srcStr}  \x1b[2m(${r.created_at})\x1b[0m`);
       }
-      console.log(`\n  ${all.length} memor${all.length === 1 ? 'y' : 'ies'}`);
+      console.log(`\n  \x1b[1m${all.length}\x1b[0m memor${all.length === 1 ? 'y' : 'ies'}`);
       break;
     }
     case 'forget': case 'delete': case 'rm': {
       const id = args[1];
-      if (!id) return console.log('Usage: mem forget <id>');
+      if (!id) return console.log('Usage: memshell forget <id>');
       await mem.forget(id);
-      console.log(`✓ Forgotten (id: ${id})`);
+      console.log(`\x1b[32m+\x1b[0m Forgotten (id: ${id})`);
       break;
     }
     case 'clear': case 'wipe': case 'reset': {
       await mem.clear(opts);
-      console.log('✓ All memories cleared');
+      console.log('\x1b[32m+\x1b[0m All memories cleared');
+      break;
+    }
+    case 'important': case 'boost': {
+      const id = args[1];
+      if (!id) return console.log('Usage: memshell important <id>');
+      const r = await mem.important(Number(id));
+      if (!r) return console.log('\x1b[31mMemory not found.\x1b[0m');
+      console.log(`\x1b[32m+\x1b[0m Boosted memory ${r.id} -> importance: \x1b[1m${r.importance.toFixed(1)}\x1b[0m`);
+      break;
+    }
+    case 'stats': {
+      const s = await mem.stats(opts);
+      console.log(`\n  \x1b[1mMemory Stats\x1b[0m`);
+      console.log(`  Total:          \x1b[36m${s.total}\x1b[0m`);
+      console.log(`  Oldest:         ${s.oldest || 'n/a'}`);
+      console.log(`  Newest:         ${s.newest || 'n/a'}`);
+      console.log(`  Avg importance: \x1b[33m${s.avg_importance}\x1b[0m\n`);
+      break;
+    }
+    case 'export': {
+      const data = await mem.exportAll(opts);
+      console.log(JSON.stringify(data, null, 2));
+      break;
+    }
+    case 'import': {
+      const file = args[1];
+      if (!file) return console.log('Usage: memshell import <file.json>');
+      const raw = fs.readFileSync(path.resolve(file), 'utf8');
+      const data = JSON.parse(raw);
+      const r = await mem.importAll(Array.isArray(data) ? data : data.memories || []);
+      console.log(`\x1b[32m+\x1b[0m Imported ${r.imported} memories`);
+      break;
+    }
+    case 'ingest': {
+      const { ingestFile, ingest: ingestText } = require('../src/ingest');
+      const store = new LocalStore(undefined, useEmbeddings ? { openaiKey: process.env.OPENAI_API_KEY } : {});
+      await store.init();
+
+      if (hasFlag('stdin')) {
+        const text = await readStdin();
+        if (!text.trim()) return console.log('No input received via stdin.');
+        console.log('  Extracting facts from stdin...');
+        const result = await ingestText(text, store, { agent });
+        console.log(`\x1b[32m+\x1b[0m Extracted: ${result.extracted}, Stored: ${result.stored}, Duplicates: ${result.duplicates}`);
+      } else if (hasFlag('watch')) {
+        const dir = flag('watch');
+        if (!dir || dir === true) return console.log('Usage: memshell ingest --watch <directory>');
+        const { watchDirectory } = require('../src/ingest');
+        console.log('  Starting directory watcher (Ctrl+C to stop)...');
+        watchDirectory(dir, store, { agent });
+        // Keep process alive
+        process.on('SIGINT', () => { console.log('\n  Stopped.'); process.exit(0); });
+      } else {
+        const file = getText();
+        if (!file) return console.log('Usage: memshell ingest <file> | --stdin | --watch <dir>');
+        console.log(`  Ingesting: ${file}`);
+        const result = await ingestFile(file, store, { agent, force: hasFlag('force') });
+        if (result.skipped) {
+          console.log(`  Skipped: ${result.file} (already processed, use --force to re-ingest)`);
+        } else {
+          console.log(`\x1b[32m+\x1b[0m Extracted: ${result.extracted}, Stored: ${result.stored}, Duplicates: ${result.duplicates}`);
+        }
+      }
+      break;
+    }
+    case 'connect': {
+      const target = args[1];
+      if (target !== 'openclaw') return console.log('Usage: memshell connect openclaw');
+
+      const { watchOpenClaw, defaultOpenClawPath, setConfigValue } = require('../src/ingest');
+      const store = new LocalStore(undefined, useEmbeddings ? { openaiKey: process.env.OPENAI_API_KEY } : {});
+      await store.init();
+
+      const sessionsPath = args[2] || defaultOpenClawPath();
+      setConfigValue('watch.openclaw', sessionsPath);
+      console.log(`  OpenClaw integration configured.`);
+      console.log(`  Sessions path: ${sessionsPath}`);
+      console.log('  Watching for new transcripts (Ctrl+C to stop)...\n');
+      watchOpenClaw(sessionsPath, store, { agent });
+      process.on('SIGINT', () => { console.log('\n  Stopped.'); process.exit(0); });
+      break;
+    }
+    case 'daemon': {
+      const { loadConfig, watchDirectory, watchOpenClaw } = require('../src/ingest');
+      const store = new LocalStore(undefined, useEmbeddings ? { openaiKey: process.env.OPENAI_API_KEY } : {});
+      await store.init();
+
+      const config = loadConfig();
+      const watchers = config.watch || {};
+      let activeWatchers = 0;
+
+      console.log('  \x1b[1mmem.sh daemon\x1b[0m starting...\n');
+
+      if (watchers.openclaw) {
+        watchOpenClaw(watchers.openclaw, store, { agent });
+        activeWatchers++;
+      }
+
+      // Support array of dir watchers
+      if (Array.isArray(watchers.dirs)) {
+        for (const dir of watchers.dirs) {
+          watchDirectory(typeof dir === 'string' ? dir : dir.path, store, { agent });
+          activeWatchers++;
+        }
+      } else if (watchers.dir) {
+        watchDirectory(watchers.dir, store, { agent });
+        activeWatchers++;
+      }
+
+      if (activeWatchers === 0) {
+        console.log('  No watchers configured. Use:');
+        console.log('    memshell config set watch.openclaw ~/.openclaw/agents/main/sessions/');
+        console.log('    memshell config set watch.dir /path/to/watch');
+        process.exit(1);
+      }
+
+      console.log(`\n  ${activeWatchers} watcher(s) active. Ctrl+C to stop.\n`);
+      process.on('SIGINT', () => { console.log('\n  Daemon stopped.'); process.exit(0); });
+      break;
+    }
+    case 'config': {
+      const { loadConfig, setConfigValue } = require('../src/ingest');
+      const subCmd = args[1];
+
+      if (subCmd === 'set') {
+        const configKey = args[2];
+        const configVal = args.slice(3).join(' ');
+        if (!configKey || !configVal) return console.log('Usage: memshell config set <key> <value>');
+        const result = setConfigValue(configKey, configVal);
+        console.log(`\x1b[32m+\x1b[0m Set ${configKey} = ${configVal}`);
+      } else if (subCmd === 'get') {
+        const config = loadConfig();
+        const configKey = args[2];
+        if (configKey) {
+          const parts = configKey.split('.');
+          let val = config;
+          for (const p of parts) val = val?.[p];
+          console.log(val !== undefined ? JSON.stringify(val, null, 2) : 'Not set');
+        } else {
+          console.log(JSON.stringify(config, null, 2));
+        }
+      } else {
+        const config = loadConfig();
+        console.log(JSON.stringify(config, null, 2));
+      }
       break;
     }
     case 'serve': case 'server': {
@@ -91,6 +302,7 @@ async function main() {
       const authKey = flag('key') || process.env.MEM_KEY || '';
       process.env.MEM_PORT = port;
       if (authKey) process.env.MEM_KEY = authKey;
+      if (useEmbeddings) process.env.MEM_USE_EMBEDDINGS = '1';
       require('../server');
       break;
     }
@@ -99,4 +311,4 @@ async function main() {
   }
 }
 
-main().catch(e => { console.error('Error:', e.message); process.exit(1); });
+main().catch(e => { console.error('\x1b[31mError:\x1b[0m', e.message); process.exit(1); });