memshell 0.3.0 → 0.4.0

package/README.md

@@ -2,7 +2,7 @@
 
 <h1>mem.sh</h1>
 
-<p><strong>Persistent memory for AI agents.</strong><br>One line to save. One line to recall.</p>
+<p><strong>Persistent memory for AI agents.</strong><br>One line to save. One line to recall. Auto-ingest conversations.</p>
 
 [![npm version](https://img.shields.io/npm/v/memshell.svg?style=flat-square)](https://www.npmjs.com/package/memshell)
 [![license](https://img.shields.io/npm/l/memshell.svg?style=flat-square)](https://github.com/justedv/mem.sh/blob/main/LICENSE)
@@ -10,7 +10,7 @@
 
 <br>
 
-[Quick Start](#quick-start) · [SDK](#sdk) · [API Server](#api-server) · [Architecture](#how-it-works) · [Contributing](CONTRIBUTING.md)
+[Quick Start](#quick-start) · [Auto-Ingest](#auto-ingest) · [OpenClaw Integration](#openclaw-integration) · [SDK](#sdk) · [API Server](#api-server) · [Architecture](#how-it-works)
 
 </div>
 
@@ -29,17 +29,20 @@ Agents forget everything between sessions. **mem.sh** gives them a brain.
 | | mem.sh | LangChain Memory | Roll your own |
 |---|---|---|---|
 | **Setup** | `npx memshell set "..."` | 47 dependencies + config | Hours of boilerplate |
-| **External APIs** | None | OpenAI key required | Depends |
+| **Auto-ingest** | Built-in | No | You build it |
+| **External APIs** | None (optional) | OpenAI key required | Depends |
 | **Semantic search** | Built-in TF-IDF | Embedding models | You build it |
 | **Storage** | SQLite (local) | Varies | You choose |
-| **Lines of code** | ~1 | ~50+ | ~200+ |
 
 ## Features
 
-- **Fast** — TF-IDF vectorization with cosine similarity, instant results
-- **Local-first** — SQLite storage at `~/.mem/mem.db`, no data leaves your machine
-- **Semantic** — Recall by meaning, not exact match
-- **Zero config** — `npx` and go. No API keys, no setup, no dependencies
+- **Fast** -- TF-IDF vectorization with cosine similarity, instant results
+- **Local-first** -- SQLite storage at `~/.mem/mem.db`, no data leaves your machine
+- **Semantic** -- Recall by meaning, not exact match
+- **Auto-ingest** -- Feed raw conversations, auto-extract key facts via LLM
+- **OpenClaw integration** -- Watch session transcripts and auto-learn
+- **Zero config** -- `npx` and go. No API keys needed for core features
+- **Smart recall** -- Shows source, creation time, and recall frequency
 
 ## Quick Start
 
@@ -51,7 +54,7 @@ npx memshell set "user prefers dark mode"
 
 # Recall semantically
 npx memshell recall "what theme does the user like?"
-# → user prefers dark mode (score: 0.87)
+# => user prefers dark mode (score: 0.87)
 
 # List all memories
 npx memshell list
@@ -63,7 +66,92 @@ npx memshell forget <id>
 npx memshell clear
 ```
 
-### SDK
+## Auto-Ingest
+
+Feed raw conversations and let the LLM extract key facts automatically.
+
+Requires `OPENAI_API_KEY` or `ANTHROPIC_API_KEY` (or configure via `memshell config set apiKey <key>`).
+
+### From a file
+
+```bash
+npx memshell ingest conversation.txt
+npx memshell ingest chat.jsonl
+npx memshell ingest notes.md
+```
+
+### From stdin
+
+```bash
+echo "User said they prefer dark mode and use vim" | npx memshell ingest --stdin
+```
+
+### Watch a directory
+
+```bash
+npx memshell ingest --watch ./logs/
+```
+
+Watches for new or changed `.txt`, `.md`, `.json`, and `.jsonl` files. Tracks what has been processed to avoid duplicates.
+
+### Via API
+
+```bash
+curl -X POST http://localhost:3456/mem/ingest \
+  -H "Content-Type: application/json" \
+  -d '{"text": "User mentioned they love Rust and prefer dark themes"}'
+# => {"extracted": 2, "stored": 2, "duplicates": 0}
+```
+
+### How it works
+
+1. Text is split into ~2000-token chunks
+2. Each chunk is sent to an LLM (gpt-4o-mini or claude-3-haiku) to extract standalone facts
+3. Facts are deduplicated against existing memories (Jaccard similarity > 0.85 = skip)
+4. New facts are stored with auto-generated tags and source tracking
+
+## OpenClaw Integration
+
+Automatically learn from your OpenClaw agent conversations:
+
+```bash
+# Start watching OpenClaw session transcripts
+npx memshell connect openclaw
+
+# Or specify a custom path
+npx memshell connect openclaw /path/to/sessions/
+```
+
+This watches the OpenClaw sessions directory (`~/.openclaw/agents/main/sessions/` by default), parses JSONL transcripts, and auto-ingests new conversations.
+
+### Daemon mode
+
+Run continuous ingestion in the background:
+
+```bash
+# Configure watchers first
+npx memshell config set watch.openclaw ~/.openclaw/agents/main/sessions/
+
+# Start the daemon
+npx memshell daemon
+```
+
+### Configuration
+
+```bash
+# Set LLM API key
+npx memshell config set apiKey sk-...
+
+# Set model
+npx memshell config set model gpt-4o-mini
+
+# View config
+npx memshell config get
+```
+
+Config is stored at `~/.mem/config.json`.
+
+## SDK
 
 ```js
 const mem = require('memshell');
@@ -72,9 +160,9 @@ const mem = require('memshell');
 await mem.set('user prefers dark mode');
 await mem.set('favorite language is rust', { agent: 'coder-bot' });
 
-// Recall (semantic search)
+// Recall (semantic search) -- now includes source and recall count
 const results = await mem.recall('what does the user like?');
-// [{ id, text, score, created_at }]
+// [{ id, text, score, created_at, source, recall_count }]
 
 // List all
 const all = await mem.list();
@@ -92,7 +180,7 @@ mem.sh uses **TF-IDF vectorization** with **cosine similarity** for semantic sea
 
 Memories are stored in `~/.mem/mem.db` (SQLite). Each memory is tokenized and vectorized on write. Queries are vectorized at recall time and ranked by cosine similarity against stored vectors.
 
-→ [Full architecture docs](docs/ARCHITECTURE.md)
+Optional: Enable OpenAI embeddings with `--embeddings` flag for higher quality recall (requires `OPENAI_API_KEY`).
 
 ## API Server
 
@@ -107,29 +195,19 @@ npx memshell serve --port 3456 --key my-secret-key
 | Method | Path | Description |
 |--------|------|-------------|
 | `POST` | `/mem` | Store a memory |
+| `POST` | `/mem/ingest` | Auto-ingest raw text |
 | `GET` | `/mem/recall?q=` | Semantic recall |
 | `GET` | `/mem/list` | List all memories |
+| `GET` | `/mem/stats` | Memory statistics |
+| `GET` | `/mem/export` | Export all memories |
+| `POST` | `/mem/import` | Import memories |
 | `DELETE` | `/mem/:id` | Delete a memory |
 | `DELETE` | `/mem` | Clear all memories |
 
 ### Headers
 
-- `X-Mem-Key` — API key (required if `--key` is set)
-- `X-Mem-Agent` — Agent namespace (optional, isolates memories per agent)
-
-### Example
-
-```bash
-# Store
-curl -X POST http://localhost:3456/mem \
-  -H "Content-Type: application/json" \
-  -H "X-Mem-Key: my-secret-key" \
-  -d '{"text": "user prefers dark mode"}'
-
-# Recall
-curl "http://localhost:3456/mem/recall?q=theme+preference" \
-  -H "X-Mem-Key: my-secret-key"
-```
+- `X-Mem-Key` -- API key (required if `--key` is set)
+- `X-Mem-Agent` -- Agent namespace (optional, isolates memories per agent)
 
 ### SDK with API Mode
 
@@ -146,9 +224,27 @@ await mem.set('user prefers dark mode');
 const results = await mem.recall('theme preference');
 ```
 
-## Contributing
+## All CLI Commands
 
-We welcome contributions. See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
+```
+memshell set <text>              Store a memory
+memshell recall <query>          Semantic recall
+memshell list                    List all memories
+memshell forget <id>             Delete a memory by ID
+memshell clear                   Wipe all memories
+memshell important <id>          Boost memory importance
+memshell ingest <file>           Extract facts from a file
+memshell ingest --stdin          Extract facts from piped text
+memshell ingest --watch <dir>    Watch directory for new files
+memshell connect openclaw        Watch OpenClaw transcripts
+memshell daemon                  Run continuous ingestion
+memshell config set <key> <val>  Set config value
+memshell config get [key]        Show config
+memshell stats                   Show memory statistics
+memshell export                  Export all memories as JSON
+memshell import <file.json>      Import memories from JSON
+memshell serve [--port N]        Start API server
+```
 
 ## License
 

package/bin/mem.js

@@ -4,6 +4,7 @@
 const fs = require('fs');
 const path = require('path');
 const mem = require('../src/index');
+const { LocalStore } = require('../src/index');
 
 const args = process.argv.slice(2);
 const cmd = args[0];
@@ -11,13 +12,26 @@ const cmd = args[0];
 const HELP = `
   \x1b[1mmem.sh\x1b[0m — persistent memory for AI agents
 
-  \x1b[36mUsage:\x1b[0m
+  \x1b[36mCore Commands:\x1b[0m
     memshell set <text>              Store a memory
     memshell recall <query>          Semantic recall
     memshell list                    List all memories
     memshell forget <id>             Delete a memory by ID
     memshell clear                   Wipe all memories
     memshell important <id>          Boost memory importance
+
+  \x1b[36mAuto-Ingest:\x1b[0m
+    memshell ingest <file>           Extract facts from a file
+    memshell ingest --stdin          Extract facts from piped text
+    memshell ingest --watch <dir>    Watch a directory for new files
+
+  \x1b[36mIntegrations:\x1b[0m
+    memshell connect openclaw        Watch OpenClaw session transcripts
+    memshell daemon                  Run continuous ingestion daemon
+
+  \x1b[36mManagement:\x1b[0m
+    memshell config set <key> <val>  Set config value
+    memshell config get [key]        Show config
     memshell stats                   Show memory statistics
     memshell export                  Export all memories as JSON
     memshell import <file.json>      Import memories from JSON
@@ -34,10 +48,9 @@ const HELP = `
   \x1b[36mExamples:\x1b[0m
     memshell set "user prefers dark mode" --tags preferences,ui
     memshell recall "what theme?" --tags preferences --top 3
-    memshell important 5
-    memshell stats
-    memshell export > backup.json
-    memshell import backup.json
+    echo "User likes vim and dark mode" | memshell ingest --stdin
+    memshell connect openclaw
+    memshell config set apiKey sk-...
 `;
 
 // Parse flags
@@ -52,18 +65,14 @@ function hasFlag(name) {
   return args.includes('--' + name);
 }
 
-function getTextArgs() {
-  return args.slice(1).filter(a => !a.startsWith('--') && (args.indexOf(a) === 0 || !['--agent', '--api', '--key', '--tags', '--top', '--port'].includes(args[args.indexOf(a) - 1]))).join(' ').replace(/^["']|["']$/g, '');
-}
-
 // Smarter text extraction: skip flag values
 function getText() {
-  const skip = new Set(['--agent', '--api', '--key', '--tags', '--top', '--port']);
+  const skip = new Set(['--agent', '--api', '--key', '--tags', '--top', '--port', '--watch']);
   const parts = [];
-  let i = 1; // skip command
+  let i = 1;
   while (i < args.length) {
     if (skip.has(args[i])) { i += 2; continue; }
-    if (args[i] === '--embeddings') { i++; continue; }
+    if (args[i] === '--embeddings' || args[i] === '--stdin' || args[i] === '--force') { i++; continue; }
     if (args[i].startsWith('--')) { i++; continue; }
     parts.push(args[i]);
     i++;
@@ -71,6 +80,17 @@ function getText() {
   return parts.join(' ').replace(/^["']|["']$/g, '');
 }
 
+function readStdin() {
+  return new Promise((resolve) => {
+    let data = '';
+    process.stdin.setEncoding('utf8');
+    process.stdin.on('data', chunk => data += chunk);
+    process.stdin.on('end', () => resolve(data));
+    // If nothing after 100ms and stdin is a TTY, resolve empty
+    if (process.stdin.isTTY) resolve('');
+  });
+}
+
 async function main() {
   const agent = flag('agent') || 'default';
   const api = flag('api');
@@ -92,7 +112,7 @@ async function main() {
       const text = getText();
       if (!text) return console.log('Usage: memshell set <text>');
       const r = await mem.set(text, { ...opts, tags });
-      console.log(`\x1b[32m✓\x1b[0m Stored (id: \x1b[1m${r.id}\x1b[0m)${tags ? ` [tags: ${tags}]` : ''}`);
+      console.log(`\x1b[32m+\x1b[0m Stored (id: \x1b[1m${r.id}\x1b[0m)${tags ? ` [tags: ${tags}]` : ''}`);
       break;
     }
     case 'recall': case 'r': case 'search': case 'q': {
@@ -102,7 +122,9 @@ async function main() {
       if (!results.length) return console.log('\x1b[33mNo memories found.\x1b[0m');
       for (const r of results) {
         const tagStr = r.tags ? ` \x1b[35m[${r.tags}]\x1b[0m` : '';
-        console.log(`  \x1b[36m[${r.id}]\x1b[0m ${r.text} \x1b[33m(score: ${r.score})\x1b[0m${tagStr}`);
+        const srcStr = r.source && r.source !== 'manual' ? ` \x1b[2m(src: ${r.source})\x1b[0m` : '';
+        const recallStr = r.recall_count ? ` \x1b[2m(recalled ${r.recall_count}x)\x1b[0m` : '';
+        console.log(`  \x1b[36m[${r.id}]\x1b[0m ${r.text} \x1b[33m(score: ${r.score})\x1b[0m${tagStr}${srcStr}${recallStr}`);
       }
       break;
     }
@@ -111,8 +133,9 @@ async function main() {
       if (!all.length) return console.log('\x1b[33mNo memories stored.\x1b[0m');
       for (const r of all) {
         const tagStr = r.tags ? ` \x1b[35m[${r.tags}]\x1b[0m` : '';
-        const imp = r.importance !== 1.0 ? ` \x1b[33m★${r.importance.toFixed(1)}\x1b[0m` : '';
-        console.log(`  \x1b[36m[${r.id}]\x1b[0m ${r.text}${tagStr}${imp}  \x1b[2m(${r.created_at})\x1b[0m`);
+        const imp = r.importance !== 1.0 ? ` \x1b[33m*${r.importance.toFixed(1)}\x1b[0m` : '';
+        const srcStr = r.source && r.source !== 'manual' ? ` \x1b[2m[${r.source}]\x1b[0m` : '';
+        console.log(`  \x1b[36m[${r.id}]\x1b[0m ${r.text}${tagStr}${imp}${srcStr}  \x1b[2m(${r.created_at})\x1b[0m`);
       }
       console.log(`\n  \x1b[1m${all.length}\x1b[0m memor${all.length === 1 ? 'y' : 'ies'}`);
       break;
@@ -121,12 +144,12 @@ async function main() {
       const id = args[1];
       if (!id) return console.log('Usage: memshell forget <id>');
       await mem.forget(id);
-      console.log(`\x1b[32m✓\x1b[0m Forgotten (id: ${id})`);
+      console.log(`\x1b[32m+\x1b[0m Forgotten (id: ${id})`);
       break;
     }
     case 'clear': case 'wipe': case 'reset': {
       await mem.clear(opts);
-      console.log('\x1b[32m✓\x1b[0m All memories cleared');
+      console.log('\x1b[32m+\x1b[0m All memories cleared');
       break;
     }
     case 'important': case 'boost': {
@@ -134,15 +157,15 @@ async function main() {
       if (!id) return console.log('Usage: memshell important <id>');
       const r = await mem.important(Number(id));
       if (!r) return console.log('\x1b[31mMemory not found.\x1b[0m');
-      console.log(`\x1b[32m✓\x1b[0m Boosted memory ${r.id} → importance: \x1b[1m${r.importance.toFixed(1)}\x1b[0m`);
+      console.log(`\x1b[32m+\x1b[0m Boosted memory ${r.id} -> importance: \x1b[1m${r.importance.toFixed(1)}\x1b[0m`);
       break;
     }
     case 'stats': {
       const s = await mem.stats(opts);
-      console.log(`\n  \x1b[1m🧠 Memory Stats\x1b[0m`);
-      console.log(`  Total:        \x1b[36m${s.total}\x1b[0m`);
-      console.log(`  Oldest:       ${s.oldest || 'n/a'}`);
-      console.log(`  Newest:       ${s.newest || 'n/a'}`);
+      console.log(`\n  \x1b[1mMemory Stats\x1b[0m`);
+      console.log(`  Total:          \x1b[36m${s.total}\x1b[0m`);
+      console.log(`  Oldest:         ${s.oldest || 'n/a'}`);
+      console.log(`  Newest:         ${s.newest || 'n/a'}`);
       console.log(`  Avg importance: \x1b[33m${s.avg_importance}\x1b[0m\n`);
       break;
     }
@@ -157,7 +180,121 @@ async function main() {
       const raw = fs.readFileSync(path.resolve(file), 'utf8');
       const data = JSON.parse(raw);
       const r = await mem.importAll(Array.isArray(data) ? data : data.memories || []);
-      console.log(`\x1b[32m✓\x1b[0m Imported ${r.imported} memories`);
+      console.log(`\x1b[32m+\x1b[0m Imported ${r.imported} memories`);
+      break;
+    }
+    case 'ingest': {
+      const { ingestFile, ingest: ingestText } = require('../src/ingest');
+      const store = new LocalStore(undefined, useEmbeddings ? { openaiKey: process.env.OPENAI_API_KEY } : {});
+      await store.init();
+
+      if (hasFlag('stdin')) {
+        const text = await readStdin();
+        if (!text.trim()) return console.log('No input received via stdin.');
+        console.log('  Extracting facts from stdin...');
+        const result = await ingestText(text, store, { agent });
+        console.log(`\x1b[32m+\x1b[0m Extracted: ${result.extracted}, Stored: ${result.stored}, Duplicates: ${result.duplicates}`);
+      } else if (hasFlag('watch')) {
+        const dir = flag('watch');
+        if (!dir || dir === true) return console.log('Usage: memshell ingest --watch <directory>');
+        const { watchDirectory } = require('../src/ingest');
+        console.log('  Starting directory watcher (Ctrl+C to stop)...');
+        watchDirectory(dir, store, { agent });
+        // Keep process alive
+        process.on('SIGINT', () => { console.log('\n  Stopped.'); process.exit(0); });
+      } else {
+        const file = getText();
+        if (!file) return console.log('Usage: memshell ingest <file> | --stdin | --watch <dir>');
+        console.log(`  Ingesting: ${file}`);
+        const result = await ingestFile(file, store, { agent, force: hasFlag('force') });
+        if (result.skipped) {
+          console.log(`  Skipped: ${result.file} (already processed, use --force to re-ingest)`);
+        } else {
+          console.log(`\x1b[32m+\x1b[0m Extracted: ${result.extracted}, Stored: ${result.stored}, Duplicates: ${result.duplicates}`);
+        }
+      }
+      break;
+    }
+    case 'connect': {
+      const target = args[1];
+      if (target !== 'openclaw') return console.log('Usage: memshell connect openclaw');
+
+      const { watchOpenClaw, defaultOpenClawPath, setConfigValue } = require('../src/ingest');
+      const store = new LocalStore(undefined, useEmbeddings ? { openaiKey: process.env.OPENAI_API_KEY } : {});
+      await store.init();
+
+      const sessionsPath = args[2] || defaultOpenClawPath();
+      setConfigValue('watch.openclaw', sessionsPath);
+      console.log(`  OpenClaw integration configured.`);
+      console.log(`  Sessions path: ${sessionsPath}`);
+      console.log('  Watching for new transcripts (Ctrl+C to stop)...\n');
+      watchOpenClaw(sessionsPath, store, { agent });
+      process.on('SIGINT', () => { console.log('\n  Stopped.'); process.exit(0); });
+      break;
+    }
+    case 'daemon': {
+      const { loadConfig, watchDirectory, watchOpenClaw } = require('../src/ingest');
+      const store = new LocalStore(undefined, useEmbeddings ? { openaiKey: process.env.OPENAI_API_KEY } : {});
+      await store.init();
+
+      const config = loadConfig();
+      const watchers = config.watch || {};
+      let activeWatchers = 0;
+
+      console.log('  \x1b[1mmem.sh daemon\x1b[0m starting...\n');
+
+      if (watchers.openclaw) {
+        watchOpenClaw(watchers.openclaw, store, { agent });
+        activeWatchers++;
+      }
+
+      // Support array of dir watchers
+      if (Array.isArray(watchers.dirs)) {
+        for (const dir of watchers.dirs) {
+          watchDirectory(typeof dir === 'string' ? dir : dir.path, store, { agent });
+          activeWatchers++;
+        }
+      } else if (watchers.dir) {
+        watchDirectory(watchers.dir, store, { agent });
+        activeWatchers++;
+      }
+
+      if (activeWatchers === 0) {
+        console.log('  No watchers configured. Use:');
+        console.log('    memshell config set watch.openclaw ~/.openclaw/agents/main/sessions/');
+        console.log('    memshell config set watch.dir /path/to/watch');
+        process.exit(1);
+      }
+
+      console.log(`\n  ${activeWatchers} watcher(s) active. Ctrl+C to stop.\n`);
+      process.on('SIGINT', () => { console.log('\n  Daemon stopped.'); process.exit(0); });
+      break;
+    }
+    case 'config': {
+      const { loadConfig, setConfigValue } = require('../src/ingest');
+      const subCmd = args[1];
+
+      if (subCmd === 'set') {
+        const configKey = args[2];
+        const configVal = args.slice(3).join(' ');
+        if (!configKey || !configVal) return console.log('Usage: memshell config set <key> <value>');
+        const result = setConfigValue(configKey, configVal);
+        console.log(`\x1b[32m+\x1b[0m Set ${configKey} = ${configVal}`);
+      } else if (subCmd === 'get') {
+        const config = loadConfig();
+        const configKey = args[2];
+        if (configKey) {
+          const parts = configKey.split('.');
+          let val = config;
+          for (const p of parts) val = val?.[p];
+          console.log(val !== undefined ? JSON.stringify(val, null, 2) : 'Not set');
+        } else {
+          console.log(JSON.stringify(config, null, 2));
+        }
+      } else {
+        const config = loadConfig();
+        console.log(JSON.stringify(config, null, 2));
+      }
       break;
     }
     case 'serve': case 'server': {

package/bin/memshell.js

@@ -4,6 +4,7 @@
 const fs = require('fs');
 const path = require('path');
 const mem = require('../src/index');
+const { LocalStore } = require('../src/index');
 
 const args = process.argv.slice(2);
 const cmd = args[0];
@@ -11,13 +12,26 @@ const cmd = args[0];
 const HELP = `
   \x1b[1mmem.sh\x1b[0m — persistent memory for AI agents
 
-  \x1b[36mUsage:\x1b[0m
+  \x1b[36mCore Commands:\x1b[0m
     memshell set <text>              Store a memory
     memshell recall <query>          Semantic recall
     memshell list                    List all memories
     memshell forget <id>             Delete a memory by ID
     memshell clear                   Wipe all memories
     memshell important <id>          Boost memory importance
+
+  \x1b[36mAuto-Ingest:\x1b[0m
+    memshell ingest <file>           Extract facts from a file
+    memshell ingest --stdin          Extract facts from piped text
+    memshell ingest --watch <dir>    Watch a directory for new files
+
+  \x1b[36mIntegrations:\x1b[0m
+    memshell connect openclaw        Watch OpenClaw session transcripts
+    memshell daemon                  Run continuous ingestion daemon
+
+  \x1b[36mManagement:\x1b[0m
+    memshell config set <key> <val>  Set config value
+    memshell config get [key]        Show config
     memshell stats                   Show memory statistics
     memshell export                  Export all memories as JSON
     memshell import <file.json>      Import memories from JSON
@@ -34,10 +48,9 @@ const HELP = `
   \x1b[36mExamples:\x1b[0m
     memshell set "user prefers dark mode" --tags preferences,ui
     memshell recall "what theme?" --tags preferences --top 3
-    memshell important 5
-    memshell stats
-    memshell export > backup.json
-    memshell import backup.json
+    echo "User likes vim and dark mode" | memshell ingest --stdin
+    memshell connect openclaw
+    memshell config set apiKey sk-...
 `;
 
 // Parse flags
@@ -52,18 +65,14 @@ function hasFlag(name) {
   return args.includes('--' + name);
 }
 
-function getTextArgs() {
-  return args.slice(1).filter(a => !a.startsWith('--') && (args.indexOf(a) === 0 || !['--agent', '--api', '--key', '--tags', '--top', '--port'].includes(args[args.indexOf(a) - 1]))).join(' ').replace(/^["']|["']$/g, '');
-}
-
 // Smarter text extraction: skip flag values
 function getText() {
-  const skip = new Set(['--agent', '--api', '--key', '--tags', '--top', '--port']);
+  const skip = new Set(['--agent', '--api', '--key', '--tags', '--top', '--port', '--watch']);
   const parts = [];
-  let i = 1; // skip command
+  let i = 1;
   while (i < args.length) {
     if (skip.has(args[i])) { i += 2; continue; }
-    if (args[i] === '--embeddings') { i++; continue; }
+    if (args[i] === '--embeddings' || args[i] === '--stdin' || args[i] === '--force') { i++; continue; }
     if (args[i].startsWith('--')) { i++; continue; }
     parts.push(args[i]);
     i++;
@@ -71,6 +80,17 @@ function getText() {
   return parts.join(' ').replace(/^["']|["']$/g, '');
 }
 
+function readStdin() {
+  return new Promise((resolve) => {
+    let data = '';
+    process.stdin.setEncoding('utf8');
+    process.stdin.on('data', chunk => data += chunk);
+    process.stdin.on('end', () => resolve(data));
+    // If nothing after 100ms and stdin is a TTY, resolve empty
+    if (process.stdin.isTTY) resolve('');
+  });
+}
+
 async function main() {
   const agent = flag('agent') || 'default';
   const api = flag('api');
@@ -92,7 +112,7 @@ async function main() {
       const text = getText();
       if (!text) return console.log('Usage: memshell set <text>');
       const r = await mem.set(text, { ...opts, tags });
-      console.log(`\x1b[32m✓\x1b[0m Stored (id: \x1b[1m${r.id}\x1b[0m)${tags ? ` [tags: ${tags}]` : ''}`);
+      console.log(`\x1b[32m+\x1b[0m Stored (id: \x1b[1m${r.id}\x1b[0m)${tags ? ` [tags: ${tags}]` : ''}`);
       break;
     }
     case 'recall': case 'r': case 'search': case 'q': {
@@ -102,7 +122,9 @@ async function main() {
       if (!results.length) return console.log('\x1b[33mNo memories found.\x1b[0m');
       for (const r of results) {
         const tagStr = r.tags ? ` \x1b[35m[${r.tags}]\x1b[0m` : '';
-        console.log(`  \x1b[36m[${r.id}]\x1b[0m ${r.text} \x1b[33m(score: ${r.score})\x1b[0m${tagStr}`);
+        const srcStr = r.source && r.source !== 'manual' ? ` \x1b[2m(src: ${r.source})\x1b[0m` : '';
+        const recallStr = r.recall_count ? ` \x1b[2m(recalled ${r.recall_count}x)\x1b[0m` : '';
+        console.log(`  \x1b[36m[${r.id}]\x1b[0m ${r.text} \x1b[33m(score: ${r.score})\x1b[0m${tagStr}${srcStr}${recallStr}`);
       }
       break;
     }
@@ -111,8 +133,9 @@ async function main() {
       if (!all.length) return console.log('\x1b[33mNo memories stored.\x1b[0m');
       for (const r of all) {
         const tagStr = r.tags ? ` \x1b[35m[${r.tags}]\x1b[0m` : '';
-        const imp = r.importance !== 1.0 ? ` \x1b[33m★${r.importance.toFixed(1)}\x1b[0m` : '';
-        console.log(`  \x1b[36m[${r.id}]\x1b[0m ${r.text}${tagStr}${imp}  \x1b[2m(${r.created_at})\x1b[0m`);
+        const imp = r.importance !== 1.0 ? ` \x1b[33m*${r.importance.toFixed(1)}\x1b[0m` : '';
+        const srcStr = r.source && r.source !== 'manual' ? ` \x1b[2m[${r.source}]\x1b[0m` : '';
+        console.log(`  \x1b[36m[${r.id}]\x1b[0m ${r.text}${tagStr}${imp}${srcStr}  \x1b[2m(${r.created_at})\x1b[0m`);
       }
       console.log(`\n  \x1b[1m${all.length}\x1b[0m memor${all.length === 1 ? 'y' : 'ies'}`);
       break;
@@ -121,12 +144,12 @@ async function main() {
       const id = args[1];
       if (!id) return console.log('Usage: memshell forget <id>');
       await mem.forget(id);
-      console.log(`\x1b[32m✓\x1b[0m Forgotten (id: ${id})`);
+      console.log(`\x1b[32m+\x1b[0m Forgotten (id: ${id})`);
       break;
     }
     case 'clear': case 'wipe': case 'reset': {
       await mem.clear(opts);
-      console.log('\x1b[32m✓\x1b[0m All memories cleared');
+      console.log('\x1b[32m+\x1b[0m All memories cleared');
       break;
     }
     case 'important': case 'boost': {
@@ -134,15 +157,15 @@ async function main() {
       if (!id) return console.log('Usage: memshell important <id>');
       const r = await mem.important(Number(id));
       if (!r) return console.log('\x1b[31mMemory not found.\x1b[0m');
-      console.log(`\x1b[32m✓\x1b[0m Boosted memory ${r.id} → importance: \x1b[1m${r.importance.toFixed(1)}\x1b[0m`);
+      console.log(`\x1b[32m+\x1b[0m Boosted memory ${r.id} -> importance: \x1b[1m${r.importance.toFixed(1)}\x1b[0m`);
       break;
     }
     case 'stats': {
       const s = await mem.stats(opts);
-      console.log(`\n  \x1b[1m🧠 Memory Stats\x1b[0m`);
-      console.log(`  Total:        \x1b[36m${s.total}\x1b[0m`);
-      console.log(`  Oldest:       ${s.oldest || 'n/a'}`);
-      console.log(`  Newest:       ${s.newest || 'n/a'}`);
+      console.log(`\n  \x1b[1mMemory Stats\x1b[0m`);
+      console.log(`  Total:          \x1b[36m${s.total}\x1b[0m`);
+      console.log(`  Oldest:         ${s.oldest || 'n/a'}`);
+      console.log(`  Newest:         ${s.newest || 'n/a'}`);
       console.log(`  Avg importance: \x1b[33m${s.avg_importance}\x1b[0m\n`);
       break;
     }
@@ -157,7 +180,121 @@ async function main() {
       const raw = fs.readFileSync(path.resolve(file), 'utf8');
       const data = JSON.parse(raw);
       const r = await mem.importAll(Array.isArray(data) ? data : data.memories || []);
-      console.log(`\x1b[32m✓\x1b[0m Imported ${r.imported} memories`);
+      console.log(`\x1b[32m+\x1b[0m Imported ${r.imported} memories`);
+      break;
+    }
+    case 'ingest': {
+      const { ingestFile, ingest: ingestText } = require('../src/ingest');
+      const store = new LocalStore(undefined, useEmbeddings ? { openaiKey: process.env.OPENAI_API_KEY } : {});
+      await store.init();
+
+      if (hasFlag('stdin')) {
+        const text = await readStdin();
+        if (!text.trim()) return console.log('No input received via stdin.');
+        console.log('  Extracting facts from stdin...');
+        const result = await ingestText(text, store, { agent });
+        console.log(`\x1b[32m+\x1b[0m Extracted: ${result.extracted}, Stored: ${result.stored}, Duplicates: ${result.duplicates}`);
+      } else if (hasFlag('watch')) {
+        const dir = flag('watch');
+        if (!dir || dir === true) return console.log('Usage: memshell ingest --watch <directory>');
+        const { watchDirectory } = require('../src/ingest');
+        console.log('  Starting directory watcher (Ctrl+C to stop)...');
+        watchDirectory(dir, store, { agent });
+        // Keep process alive
+        process.on('SIGINT', () => { console.log('\n  Stopped.'); process.exit(0); });
+      } else {
+        const file = getText();
+        if (!file) return console.log('Usage: memshell ingest <file> | --stdin | --watch <dir>');
+        console.log(`  Ingesting: ${file}`);
+        const result = await ingestFile(file, store, { agent, force: hasFlag('force') });
+        if (result.skipped) {
+          console.log(`  Skipped: ${result.file} (already processed, use --force to re-ingest)`);
+        } else {
+          console.log(`\x1b[32m+\x1b[0m Extracted: ${result.extracted}, Stored: ${result.stored}, Duplicates: ${result.duplicates}`);
+        }
+      }
+      break;
+    }
+    case 'connect': {
+      const target = args[1];
+      if (target !== 'openclaw') return console.log('Usage: memshell connect openclaw');
+
+      const { watchOpenClaw, defaultOpenClawPath, setConfigValue } = require('../src/ingest');
+      const store = new LocalStore(undefined, useEmbeddings ? { openaiKey: process.env.OPENAI_API_KEY } : {});
+      await store.init();
+
+      const sessionsPath = args[2] || defaultOpenClawPath();
+      setConfigValue('watch.openclaw', sessionsPath);
+      console.log(`  OpenClaw integration configured.`);
+      console.log(`  Sessions path: ${sessionsPath}`);
+      console.log('  Watching for new transcripts (Ctrl+C to stop)...\n');
+      watchOpenClaw(sessionsPath, store, { agent });
+      process.on('SIGINT', () => { console.log('\n  Stopped.'); process.exit(0); });
+      break;
+    }
+    case 'daemon': {
+      const { loadConfig, watchDirectory, watchOpenClaw } = require('../src/ingest');
+      const store = new LocalStore(undefined, useEmbeddings ? { openaiKey: process.env.OPENAI_API_KEY } : {});
+      await store.init();
+
+      const config = loadConfig();
+      const watchers = config.watch || {};
+      let activeWatchers = 0;
+
+      console.log('  \x1b[1mmem.sh daemon\x1b[0m starting...\n');
+
+      if (watchers.openclaw) {
+        watchOpenClaw(watchers.openclaw, store, { agent });
+        activeWatchers++;
+      }
+
+      // Support array of dir watchers
+      if (Array.isArray(watchers.dirs)) {
+        for (const dir of watchers.dirs) {
+          watchDirectory(typeof dir === 'string' ? dir : dir.path, store, { agent });
+          activeWatchers++;
+        }
+      } else if (watchers.dir) {
+        watchDirectory(watchers.dir, store, { agent });
+        activeWatchers++;
+      }
+
+      if (activeWatchers === 0) {
+        console.log('  No watchers configured. Use:');
+        console.log('    memshell config set watch.openclaw ~/.openclaw/agents/main/sessions/');
+        console.log('    memshell config set watch.dir /path/to/watch');
+        process.exit(1);
+      }
+
+      console.log(`\n  ${activeWatchers} watcher(s) active. Ctrl+C to stop.\n`);
+      process.on('SIGINT', () => { console.log('\n  Daemon stopped.'); process.exit(0); });
+      break;
+    }
+    case 'config': {
+      const { loadConfig, setConfigValue } = require('../src/ingest');
+      const subCmd = args[1];
+
+      if (subCmd === 'set') {
+        const configKey = args[2];
+        const configVal = args.slice(3).join(' ');
+        if (!configKey || !configVal) return console.log('Usage: memshell config set <key> <value>');
+        const result = setConfigValue(configKey, configVal);
+        console.log(`\x1b[32m+\x1b[0m Set ${configKey} = ${configVal}`);
+      } else if (subCmd === 'get') {
+        const config = loadConfig();
+        const configKey = args[2];
+        if (configKey) {
+          const parts = configKey.split('.');
+          let val = config;
+          for (const p of parts) val = val?.[p];
+          console.log(val !== undefined ? JSON.stringify(val, null, 2) : 'Not set');
+        } else {
+          console.log(JSON.stringify(config, null, 2));
+        }
+      } else {
+        const config = loadConfig();
+        console.log(JSON.stringify(config, null, 2));
+      }
       break;
     }
     case 'serve': case 'server': {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "memshell",
-  "version": "0.3.0",
+  "version": "0.4.0",
   "description": "Persistent memory for AI agents. Like localStorage but for LLMs.",
   "main": "src/index.js",
   "bin": {

package/server.js

@@ -27,6 +27,19 @@ app.use('/mem', async (req, res, next) => {
   next();
 });
 
+// Ingest raw text
+app.post('/mem/ingest', async (req, res) => {
+  const { text, source } = req.body;
+  if (!text) return res.status(400).json({ error: 'text is required' });
+  try {
+    const { ingest } = require('./src/ingest');
+    const result = await ingest(text, store, { agent: req.agent, source: source || 'api' });
+    res.json(result);
+  } catch (e) {
+    res.status(500).json({ error: e.message });
+  }
+});
+
 // Store a memory
 app.post('/mem', async (req, res) => {
   const { text, tags, importance, metadata } = req.body;

package/src/index.js

@@ -141,8 +141,13 @@ class LocalStore {
       embedding TEXT,
       tags TEXT DEFAULT '',
       created_at TEXT NOT NULL,
-      importance REAL DEFAULT 1.0
+      importance REAL DEFAULT 1.0,
+      source TEXT DEFAULT 'manual',
+      recall_count INTEGER DEFAULT 0
     )`);
+    // Migration: add columns if they don't exist (for existing DBs)
+    try { this._db.run('ALTER TABLE memories ADD COLUMN source TEXT DEFAULT "manual"'); } catch {}
+    try { this._db.run('ALTER TABLE memories ADD COLUMN recall_count INTEGER DEFAULT 0'); } catch {}
     this._save();
   }
 
@@ -169,6 +174,7 @@ class LocalStore {
     const agent = opts.agent || 'default';
     const tags = opts.tags || '';
     const importance = opts.importance || 1.0;
+    const source = opts.source || 'manual';
     const created_at = new Date().toISOString();
 
     let embedding = null;
@@ -182,8 +188,8 @@ class LocalStore {
     }
 
     this._db.run(
-      'INSERT INTO memories (text, agent, embedding, tags, created_at, importance) VALUES (?, ?, ?, ?, ?, ?)',
-      [text, agent, embedding, tags, created_at, importance]
+      'INSERT INTO memories (text, agent, embedding, tags, created_at, importance, source) VALUES (?, ?, ?, ?, ?, ?, ?)',
+      [text, agent, embedding, tags, created_at, importance, source]
     );
     const id = this._db.exec('SELECT last_insert_rowid() as id')[0].values[0][0];
     this._save();
@@ -198,7 +204,7 @@ class LocalStore {
     const filterTags = opts.tags ? opts.tags.split(',').map(t => t.trim()) : null;
 
     const stmt = this._db.exec(
-      'SELECT id, text, agent, embedding, tags, created_at, importance FROM memories WHERE agent = ?',
+      'SELECT id, text, agent, embedding, tags, created_at, importance, source, recall_count FROM memories WHERE agent = ?',
       [agent]
     );
     if (!stmt.length) return [];
@@ -257,9 +263,9 @@ class LocalStore {
     const resultLimit = top || limit;
     const results = scored.slice(0, resultLimit);
 
-    // Bump importance for recalled memories
+    // Bump importance and recall_count for recalled memories
     for (const r of results) {
-      this._db.run('UPDATE memories SET importance = importance + 0.1 WHERE id = ?', [r.id]);
+      this._db.run('UPDATE memories SET importance = importance + 0.1, recall_count = recall_count + 1 WHERE id = ?', [r.id]);
     }
     this._save();
 
@@ -270,7 +276,7 @@ class LocalStore {
     await this.init();
     const agent = opts.agent || 'default';
     const stmt = this._db.exec(
-      'SELECT id, text, agent, tags, created_at, importance FROM memories WHERE agent = ? ORDER BY id DESC',
+      'SELECT id, text, agent, tags, created_at, importance, source, recall_count FROM memories WHERE agent = ? ORDER BY id DESC',
       [agent]
     );
     if (!stmt.length) return [];

package/src/ingest.js

@@ -0,0 +1,348 @@
+'use strict';
+
+const fs = require('fs');
+const path = require('path');
+const os = require('os');
+
+// ── LLM Extraction ────────────────────────────────────────────
+async function callLLM(text, config = {}) {
+  const anthropicKey = config.anthropicKey || process.env.ANTHROPIC_API_KEY;
+  const openaiKey = config.apiKey || config.openaiKey || process.env.OPENAI_API_KEY;
+  const model = config.model || 'gpt-4o-mini';
+
+  const systemPrompt = 'Extract key facts, user preferences, decisions, and important context from this conversation. Return as a JSON array of strings, each a standalone fact. Only return the JSON array, nothing else.';
+
+  if (anthropicKey && (model.startsWith('claude') || !openaiKey)) {
+    const res = await fetch('https://api.anthropic.com/v1/messages', {
+      method: 'POST',
+      headers: {
+        'Content-Type': 'application/json',
+        'x-api-key': anthropicKey,
+        'anthropic-version': '2023-06-01'
+      },
+      body: JSON.stringify({
+        model: model.startsWith('claude') ? model : 'claude-3-haiku-20240307',
+        max_tokens: 2048,
+        system: systemPrompt,
+        messages: [{ role: 'user', content: text }]
+      })
+    });
+    if (!res.ok) throw new Error(`Anthropic API error: ${res.status} ${await res.text()}`);
+    const data = await res.json();
+    const content = data.content[0].text;
+    return JSON.parse(content);
+  }
+
+  if (openaiKey) {
+    const res = await fetch('https://api.openai.com/v1/chat/completions', {
+      method: 'POST',
+      headers: {
+        'Content-Type': 'application/json',
+        'Authorization': `Bearer ${openaiKey}`
+      },
+      body: JSON.stringify({
+        model: model.startsWith('claude') ? 'gpt-4o-mini' : model,
+        messages: [
+          { role: 'system', content: systemPrompt },
+          { role: 'user', content: text }
+        ],
+        temperature: 0.3
+      })
+    });
+    if (!res.ok) throw new Error(`OpenAI API error: ${res.status} ${await res.text()}`);
+    const data = await res.json();
+    const content = data.choices[0].message.content;
+    // Extract JSON array from response
+    const match = content.match(/\[[\s\S]*\]/);
+    if (!match) throw new Error('LLM did not return a valid JSON array');
+    return JSON.parse(match[0]);
+  }
+
+  throw new Error('No API key found. Set OPENAI_API_KEY or ANTHROPIC_API_KEY, or run: memshell config set apiKey <key>');
+}
+
+// ── Chunking ───────────────────────────────────────────────────
+function chunkText(text, maxTokens = 2000) {
+  // Rough estimate: 1 token ≈ 4 chars
+  const maxChars = maxTokens * 4;
+  if (text.length <= maxChars) return [text];
+
+  const chunks = [];
+  const lines = text.split('\n');
+  let current = '';
+
+  for (const line of lines) {
+    if ((current + '\n' + line).length > maxChars && current.length > 0) {
+      chunks.push(current.trim());
+      current = line;
+    } else {
+      current += (current ? '\n' : '') + line;
+    }
+  }
+  if (current.trim()) chunks.push(current.trim());
+  return chunks;
+}
+
+// ── Similarity (simple word overlap for dedup) ─────────────────
+function wordSet(text) {
+  return new Set(text.toLowerCase().replace(/[^a-z0-9\s]/g, '').split(/\s+/).filter(Boolean));
+}
+
+function jaccardSimilarity(a, b) {
+  const setA = wordSet(a);
+  const setB = wordSet(b);
+  let intersection = 0;
+  for (const w of setA) if (setB.has(w)) intersection++;
+  const union = setA.size + setB.size - intersection;
+  return union === 0 ? 0 : intersection / union;
+}
+
+// ── Main Ingest Function ──────────────────────────────────────
+async function ingest(text, store, opts = {}) {
+  const config = loadConfig();
+  const mergedConfig = { ...config, ...opts };
+  const source = opts.source || 'auto-ingest';
+  const agent = opts.agent || 'default';
+
+  const chunks = chunkText(text);
+  let totalExtracted = 0;
+  let totalStored = 0;
+  let totalDuplicates = 0;
+
+  // Get existing memories for dedup
+  const existing = await store.list({ agent });
+  const existingTexts = existing.map(m => m.text);
+
+  for (const chunk of chunks) {
+    if (chunk.trim().length < 20) continue; // skip tiny chunks
+
+    let facts;
+    try {
+      facts = await callLLM(chunk, mergedConfig);
+    } catch (e) {
+      console.error(`  Warning: LLM extraction failed for chunk: ${e.message}`);
+      continue;
+    }
+
+    if (!Array.isArray(facts)) continue;
+    totalExtracted += facts.length;
+
+    for (const fact of facts) {
+      if (typeof fact !== 'string' || fact.trim().length < 5) continue;
+
+      // Dedup check
+      let isDuplicate = false;
+      for (const existing of existingTexts) {
+        if (jaccardSimilarity(fact, existing) > 0.85) {
+          isDuplicate = true;
+          break;
+        }
+      }
+
+      if (isDuplicate) {
+        totalDuplicates++;
+        continue;
+      }
+
+      // Auto-generate tags from fact
+      const tags = [source, 'auto'].join(',');
+      await store.set(fact, { agent, tags, source });
+      existingTexts.push(fact); // prevent self-duplication within batch
+      totalStored++;
+    }
+  }
+
+  return { extracted: totalExtracted, stored: totalStored, duplicates: totalDuplicates };
+}
+
+// ── JSONL Parser (OpenClaw format) ─────────────────────────────
+function parseJSONL(content) {
+  const lines = content.split('\n').filter(l => l.trim());
+  const messages = [];
+
+  for (const line of lines) {
+    try {
+      const obj = JSON.parse(line);
+      if (obj.role && obj.content) {
+        if (obj.role === 'user' || obj.role === 'assistant') {
+          const text = typeof obj.content === 'string' ? obj.content : JSON.stringify(obj.content);
+          messages.push(`${obj.role}: ${text}`);
+        }
+      }
+    } catch {
+      // skip invalid lines
+    }
+  }
+
+  return messages.join('\n');
+}
+
+// ── Config Management ──────────────────────────────────────────
+function configPath() {
+  return path.join(os.homedir(), '.mem', 'config.json');
+}
+
+function loadConfig() {
+  try {
+    return JSON.parse(fs.readFileSync(configPath(), 'utf8'));
+  } catch {
+    return {};
+  }
+}
+
+function saveConfig(config) {
+  const dir = path.dirname(configPath());
+  fs.mkdirSync(dir, { recursive: true });
+  fs.writeFileSync(configPath(), JSON.stringify(config, null, 2));
+}
+
+function setConfigValue(key, value) {
+  const config = loadConfig();
+  // Support dotted keys like watch.openclaw
+  const parts = key.split('.');
+  let obj = config;
+  for (let i = 0; i < parts.length - 1; i++) {
+    if (!obj[parts[i]] || typeof obj[parts[i]] !== 'object') obj[parts[i]] = {};
+    obj = obj[parts[i]];
+  }
+  obj[parts[parts.length - 1]] = value;
+  saveConfig(config);
+  return config;
+}
+
+// ── Processed Tracker ──────────────────────────────────────────
+function processedPath() {
+  return path.join(os.homedir(), '.mem', 'processed.json');
+}
+
+function loadProcessed() {
+  try {
+    return JSON.parse(fs.readFileSync(processedPath(), 'utf8'));
+  } catch {
+    return { files: {} };
+  }
+}
+
+function saveProcessed(data) {
+  const dir = path.dirname(processedPath());
+  fs.mkdirSync(dir, { recursive: true });
+  fs.writeFileSync(processedPath(), JSON.stringify(data, null, 2));
+}
+
+function markProcessed(filePath, mtime) {
+  const data = loadProcessed();
+  data.files[filePath] = { mtime: mtime || Date.now(), processedAt: new Date().toISOString() };
+  saveProcessed(data);
+}
+
+function isProcessed(filePath, mtime) {
+  const data = loadProcessed();
+  const entry = data.files[filePath];
+  if (!entry) return false;
+  if (mtime && entry.mtime < mtime) return false; // file was modified
+  return true;
+}
+
+// ── File Ingestion ─────────────────────────────────────────────
+async function ingestFile(filePath, store, opts = {}) {
+  const absPath = path.resolve(filePath);
+  const stat = fs.statSync(absPath);
+  const mtime = stat.mtimeMs;
+
+  if (!opts.force && isProcessed(absPath, mtime)) {
+    return { skipped: true, file: absPath };
+  }
+
+  const content = fs.readFileSync(absPath, 'utf8');
+  let text;
+
+  const ext = path.extname(absPath).toLowerCase();
+  if (ext === '.jsonl') {
+    text = parseJSONL(content);
+  } else if (ext === '.json') {
+    try {
+      const data = JSON.parse(content);
+      if (Array.isArray(data)) {
+        text = data.map(d => typeof d === 'string' ? d : JSON.stringify(d)).join('\n');
+      } else {
+        text = JSON.stringify(data);
+      }
+    } catch {
+      text = content;
+    }
+  } else {
+    text = content;
+  }
+
+  if (!text || text.trim().length < 20) {
+    return { skipped: true, file: absPath, reason: 'too short' };
+  }
+
+  const source = opts.source || `file:${path.basename(absPath)}`;
+  const result = await ingest(text, store, { ...opts, source });
+  markProcessed(absPath, mtime);
+  return { ...result, file: absPath };
+}
+
+// ── Directory Watcher (polling) ────────────────────────────────
+function watchDirectory(dir, store, opts = {}) {
+  const interval = opts.interval || 10000;
+  const absDir = path.resolve(dir);
+
+  console.log(`  Watching: ${absDir} (every ${interval / 1000}s)`);
+
+  async function scan() {
+    try {
+      const files = fs.readdirSync(absDir).filter(f => {
+        const ext = path.extname(f).toLowerCase();
+        return ['.txt', '.md', '.json', '.jsonl'].includes(ext);
+      });
+
+      for (const file of files) {
+        const filePath = path.join(absDir, file);
+        try {
+          const result = await ingestFile(filePath, store, opts);
+          if (!result.skipped) {
+            console.log(`  Ingested: ${file} (${result.extracted} extracted, ${result.stored} stored, ${result.duplicates} duplicates)`);
+          }
+        } catch (e) {
+          console.error(`  Error processing ${file}: ${e.message}`);
+        }
+      }
+    } catch (e) {
+      console.error(`  Watch error: ${e.message}`);
+    }
+  }
+
+  scan(); // initial scan
+  return setInterval(scan, interval);
+}
+
+// ── OpenClaw Connector ─────────────────────────────────────────
+function defaultOpenClawPath() {
+  return path.join(os.homedir(), '.openclaw', 'agents', 'main', 'sessions');
+}
+
+function watchOpenClaw(sessionsPath, store, opts = {}) {
+  const dir = sessionsPath || defaultOpenClawPath();
+  console.log(`  Connecting to OpenClaw sessions: ${dir}`);
+  return watchDirectory(dir, store, { ...opts, source: 'openclaw' });
+}
+
+module.exports = {
+  ingest,
+  ingestFile,
+  callLLM,
+  chunkText,
+  jaccardSimilarity,
+  parseJSONL,
+  loadConfig,
+  saveConfig,
+  setConfigValue,
+  watchDirectory,
+  watchOpenClaw,
+  defaultOpenClawPath,
+  loadProcessed,
+  isProcessed,
+  markProcessed
+};