@shadowforge0/aquifer-memory 1.0.3 → 1.3.0

package/README.md

@@ -63,43 +63,52 @@ Sessions, summaries, turn-level embeddings, entity graph — all live in one dat
 
 ## Quick Start (MCP Server)
 
-This gets you from zero to a working MCP memory server. For library API usage, see [API Reference](#api-reference) below.
+Two commands from zero to a working MCP memory server — no env vars to set. For library API usage, see [API Reference](#api-reference) below.
 
 ### 1. Start the stack
 
 ```bash
 docker compose up -d
-# Starts PostgreSQL 16 + pgvector and Ollama with bge-m3 (auto-pulled).
-# First run takes a few minutes while Ollama downloads the model.
+# PostgreSQL 16 + pgvector and Ollama with bge-m3 (auto-pulled).
+# First run pulls the model — `docker compose logs -f ollama-pull` to watch.
 ```
 
-Already have PostgreSQL + pgvector and an embedding endpoint? Skip this step.
+Already running PostgreSQL + pgvector and an embedding endpoint? Skip this step — `quickstart` picks up `DATABASE_URL` / `EMBED_PROVIDER` from your environment if you've set them.
 
-### 2. Install
+### 2. Verify
 
 ```bash
-npm install @shadowforge0/aquifer-memory
+npx --yes @shadowforge0/aquifer-memory quickstart
 ```
 
-### 3. Configure + verify
+That's it. `quickstart` autodetects `localhost:5432` PostgreSQL and `localhost:11434` Ollama (from step 1 or your own), runs migrations, embeds a test session, recalls it, and cleans up. If it prints `✓ Aquifer is working`, you're done.
 
-```bash
-export DATABASE_URL="postgresql://aquifer:aquifer@localhost:5432/aquifer"
-export AQUIFER_EMBED_BASE_URL="http://localhost:11434/v1"
-export AQUIFER_EMBED_MODEL="bge-m3"
+For ongoing use, install it into your project so you skip the `npx` resolution cost: `npm install @shadowforge0/aquifer-memory` then `npx aquifer quickstart`.
 
-npx aquifer quickstart
-```
+Using OpenAI instead of Ollama? `export EMBED_PROVIDER=openai` + `OPENAI_API_KEY=sk-...` before `quickstart` — model defaults to `text-embedding-3-small`.
 
-`quickstart` runs migrations, commits a test session, embeds it, recalls it, and cleans up. If it prints `✓ Aquifer is working`, you're done.
+### 3. Wire into your MCP client
 
-### 4. Start the MCP server
+Claude Code, Claude Desktop, or any MCP-capable client — drop this into `.mcp.json` (project-level) or `claude_desktop_config.json`:
 
-```bash
-npx aquifer mcp
+```jsonc
+{
+  "mcpServers": {
+    "aquifer": {
+      "command": "npx",
+      "args": ["--yes", "@shadowforge0/aquifer-memory", "mcp"],
+      "env": {
+        "DATABASE_URL": "postgresql://aquifer:aquifer@localhost:5432/aquifer",
+        "EMBED_PROVIDER": "ollama"
+      }
+    }
+  }
+}
 ```
 
-See [.env.example](.env.example) for all env vars, or [docs/setup.md](docs/setup.md) for the full setup guide.
+Or run it directly: `DATABASE_URL=... EMBED_PROVIDER=ollama npx aquifer mcp`. (MCP server itself stays strict about env — `quickstart`'s autodetect is the try-it path, not the production one.)
+
+Need LLM summarization, the knowledge graph, OpenAI embeddings, or the reranker? See [Environment Variables](#environment-variables) below and [docs/setup.md](docs/setup.md).
 
 ---
 
@@ -236,7 +245,7 @@ Any host that supports MCP stdio can connect the same way — point it at `node
 
 | File | Purpose |
 |------|---------|
-| `index.js` | Entry point — exports `createAquifer`, `createEmbedder`, `createReranker`, `normalizeSession` |
+| `index.js` | Entry point — exports `createAquifer`, `createEmbedder`, `createReranker` |
 | `core/aquifer.js` | Main facade: `migrate()`, `ingest()`, `recall()`, `enrich()` |
 | `core/storage.js` | Session/summary/turn CRUD, FTS search, embedding search |
 | `core/entity.js` | Entity upsert, mention tracking, relation graph, normalization |
@@ -324,18 +333,17 @@ Built-in entity extraction and relationship tracking:
 
 ## Benchmark: LongMemEval
 
-We tested Aquifer's retrieval pipeline on [LongMemEval_S](https://github.com/xiaowu0162/LongMemEval) — 470 questions across 19,195 sessions (98,845 turn embeddings).
+We tested Aquifer's retrieval pipeline on [LongMemEval_S](https://github.com/xiaowu0162/LongMemEval) — 470 questions across 19,195 sessions with 98,795 turn embeddings. Per-question haystack scoping (matching the official protocol), bge-m3 embeddings via OpenRouter.
 
-**Setup:** Per-question haystack scoping (matching official methodology), bge-m3 embeddings via OpenRouter, turn-level user-only embedding.
+| Pipeline | R@1 | R@3 | R@5 | R@10 |
+|----------|-----|-----|-----|------|
+| Turn-only (cosine) | 89.5% | 96.6% | 98.1% | 98.9% |
+| Three-way hybrid (FTS + session_emb + turn_emb → RRF) | 79.2% | 94.0% | 97.7% | 98.9% |
+| **Hybrid + Cohere Rerank v3.5 (top-30)** | **96.0%** | **98.5%** | **99.3%** | **99.8%** |
 
-| Metric | Aquifer (bge-m3) |
-|--------|-----------------|
-| R@1 | 89.6% |
-| R@3 | 96.6% |
-| R@5 | 98.1% |
-| R@10 | 98.9% |
+Measured 2026-04-19 on Aquifer 1.2.1.
 
-**Key finding:** Turn-level embedding is the main driver — going from session-level (R@1=26.8%) to turn-level (R@1=89.6%) is a 3x improvement.
+**Key findings.** Turn-level embedding alone beats session-level (26.8% → 89.5% R@1, a 3× improvement). Hybrid fusion adds robustness at R@3-R@10 but trades R@1 because FTS + session-level signals spread the top candidate across adjacent sessions. Re-ranking the hybrid top-30 with a cross-encoder (Cohere Rerank v3.5) wins back the top-1 precision and then some — +16.9pt R@1 over hybrid baseline, and 6.5pt above pure turn-level cosine. That's the production pipeline Aquifer ships by default when a reranker is configured.
 
 ### Multi-Tenant
 
@@ -408,7 +416,7 @@ const result = await aquifer.enrich('session-001', {
 // Returns: { summary, turnsEmbedded, entitiesFound, warnings, effectiveModel, postProcessError }
 ```
 
-**postProcess hook**: runs after transaction commit, receives full context (session, summary, embedding, parsedEntities, etc.). Best-effort, at-most-once.
+**postProcess hook**: runs after transaction commit, receives full context (session, summary, embedding, parsedEntities, etc.). Best-effort, at-most-once. If the hook throws, the error is captured and returned as `postProcessError` on the enrich result — the session itself remains committed and is not retried.
 
 #### `aquifer.recall(query, opts)`
 

package/consumers/claude-code.js

@@ -0,0 +1,117 @@
+'use strict';
+
+// ---------------------------------------------------------------------------
+// Claude Code host adapter.
+//
+// Generic entry points for CC-side afterburn hooks. No persona logic — the
+// caller (typically cc-afterburn.js) constructs the Miranda persona hooks
+// via consumers/miranda and injects them via `postProcess`, `summaryFn`,
+// `entityParseFn`.
+//
+// API:
+//   runEnrich({ aquifer, sessionId, agentId, ... })
+//       Enrich an already-committed session. Used by cc-afterburn after
+//       cc-session-to-pg has written the session row.
+//
+//   runBackfill({ aquifer, sessionIds, ... })
+//       Iterate enrich() over pending sessions (for catch-up after a gap).
+//
+//   runContextInject({ aquifer, pool, agentId })
+//       Return the Miranda-flavored system context string for a CC session
+//       start hook. (Delegates to consumers/miranda/context-inject.)
+// ---------------------------------------------------------------------------
+
+/**
+ * Enrich one committed session. Caller supplies the summaryFn / entityParseFn /
+ * postProcess they want (persona-specific hooks).
+ *
+ * @param {object} opts
+ * @param {object} opts.aquifer
+ * @param {string} opts.sessionId
+ * @param {string} opts.agentId
+ * @param {function} [opts.summaryFn]
+ * @param {function} [opts.entityParseFn]
+ * @param {function} [opts.postProcess]
+ * @param {object}  [opts.logger]
+ * @returns {Promise<object>} The enrich result.
+ */
+async function runEnrich({
+    aquifer, sessionId, agentId,
+    summaryFn = null, entityParseFn = null, postProcess = null,
+    logger = console,
+} = {}) {
+    if (!aquifer) throw new Error('runEnrich: aquifer is required');
+    if (!sessionId) throw new Error('runEnrich: sessionId is required');
+    if (!agentId) throw new Error('runEnrich: agentId is required');
+
+    const result = await aquifer.enrich(sessionId, {
+        agentId,
+        summaryFn: summaryFn || undefined,
+        entityParseFn: entityParseFn || undefined,
+        postProcess: postProcess || undefined,
+    });
+
+    if (result.postProcessError && logger.warn) {
+        logger.warn(`[cc-adapter] postProcess error for ${sessionId}: ${result.postProcessError.message}`);
+    }
+    if (logger.info) {
+        logger.info(`[cc-adapter] enriched ${sessionId} (turns=${result.turnsEmbedded}, entities=${result.entitiesFound})`);
+    }
+    return result;
+}
+
+/**
+ * Enrich a batch of sessions sequentially. Errors on one session don't stop
+ * the batch; they're captured and returned alongside successes.
+ *
+ * @param {object} opts
+ * @param {object} opts.aquifer
+ * @param {string[]} opts.sessionIds
+ * @param {function} opts.buildHooks — (sessionId) => { summaryFn?, entityParseFn?, postProcess? }
+ *        Called per session; lets the caller rebuild persona hooks with the
+ *        right sessionId / agentId / now.
+ * @param {string} [opts.agentId='main']
+ * @param {object} [opts.logger]
+ * @returns {Promise<{ succeeded: object[], failed: object[] }>}
+ */
+async function runBackfill({
+    aquifer, sessionIds, buildHooks,
+    agentId = 'main', logger = console,
+} = {}) {
+    if (!aquifer) throw new Error('runBackfill: aquifer is required');
+    if (!Array.isArray(sessionIds)) throw new Error('runBackfill: sessionIds must be an array');
+    if (typeof buildHooks !== 'function') throw new Error('runBackfill: buildHooks must be a function');
+
+    const succeeded = [];
+    const failed = [];
+
+    for (const sessionId of sessionIds) {
+        try {
+            const hooks = await buildHooks(sessionId, agentId);
+            const result = await runEnrich({
+                aquifer, sessionId, agentId,
+                summaryFn: hooks?.summaryFn,
+                entityParseFn: hooks?.entityParseFn,
+                postProcess: hooks?.postProcess,
+                logger,
+            });
+            succeeded.push({ sessionId, result });
+        } catch (err) {
+            if (logger.warn) logger.warn(`[cc-adapter] backfill failed for ${sessionId}: ${err.message}`);
+            failed.push({ sessionId, error: err.message });
+        }
+    }
+
+    return { succeeded, failed };
+}
+
+/**
+ * Build the Miranda-flavored system context for a CC SessionStart hook.
+ * Delegates to consumers/miranda/context-inject.computeInjection.
+ */
+async function runContextInject(opts = {}) {
+    const { computeInjection } = require('./miranda/context-inject');
+    return computeInjection(opts);
+}
+
+module.exports = { runEnrich, runBackfill, runContextInject };

package/consumers/cli.js

@@ -43,7 +43,7 @@ function parsePositiveInt(value, fallback) {
 function parseArgs(argv) {
   const args = { _: [], flags: {} };
   // Flags that take a value (not boolean)
-  const VALUE_FLAGS = new Set(['limit', 'agent-id', 'source', 'date-from', 'date-to', 'output', 'format', 'config', 'status', 'concurrency', 'entities', 'entity-mode', 'session-id', 'verdict', 'note', 'db', 'since', 'min-messages', 'lookback-days', 'max-chars']);
+  const VALUE_FLAGS = new Set(['limit', 'agent-id', 'source', 'date-from', 'date-to', 'output', 'format', 'config', 'status', 'concurrency', 'entities', 'entity-mode', 'session-id', 'verdict', 'note', 'db', 'since', 'min-messages', 'lookback-days', 'max-chars', 'out']);
   for (let i = 0; i < argv.length; i++) {
     if (argv[i] === '--') { args._.push(...argv.slice(i + 1)); break; }
     if (argv[i].startsWith('--')) {
@@ -360,6 +360,16 @@ Options:
     return;
   }
 
+  // mcp-contract: write canonical MCP tool manifest to disk. No Aquifer
+  // instance needed — manifest is static. Default path /tmp/aquifer-mcp-contract.json.
+  if (command === 'mcp-contract') {
+    const { writeMcpManifestFile } = require('../index');
+    const outPath = args.flags.out || '/tmp/aquifer-mcp-contract.json';
+    const written = writeMcpManifestFile(outPath);
+    console.log(`Wrote MCP manifest to ${written}`);
+    return;
+  }
+
   // All other commands need an Aquifer instance
   const configOverrides = {};
   if (args.flags.config) {
@@ -367,6 +377,23 @@ Options:
     process.env.AQUIFER_CONFIG = args.flags.config;
   }
 
+  // quickstart is the try-it path: autodetect docker-compose defaults so a
+  // fresh `docker compose up -d && npx aquifer quickstart` works with zero env.
+  // Production commands (migrate, mcp, recall, ...) stay strict — they expect
+  // the operator to have set env explicitly.
+  if (command === 'quickstart') {
+    const { autodetectForQuickstart } = require('./shared/autodetect');
+    const detected = await autodetectForQuickstart(process.env);
+    if (Object.keys(detected).length > 0) {
+      console.log('Autodetected localhost services (env not set):');
+      for (const [k, v] of Object.entries(detected)) {
+        console.log(`  ${k}=${v}`);
+        process.env[k] = v;
+      }
+      console.log('  Export these in your shell (or MCP client env) to make them permanent.\n');
+    }
+  }
+
   const aquifer = createAquiferFromConfig(configOverrides);
 
   try {

package/consumers/default/daily-entries.js

@@ -0,0 +1,196 @@
+'use strict';
+
+// Aquifer default persona — parameterized daily_entries writer.
+// Schema matches miranda.daily_entries (id / event_at / source / tag / text /
+// agent_id / session_id / metadata / dedupe_key) — hosts clone that DDL into
+// their own schema and set persona.dailyTable = '<schema>.daily_entries'.
+//
+// Host must create the table before use:
+//   CREATE TABLE jenny.daily_entries (LIKE miranda.daily_entries INCLUDING ALL);
+
+const crypto = require('crypto');
+const { parseHandoffSection } = require('../miranda/prompts/summary');
+
+const UPSERT_TAGS = new Set(['[FOCUS]', '[TODO]', '[STATS]', '[HIGHLIGHT]', '[SYSTEM]', '[HANDOFF]']);
+
+function taipeiDateString(now) {
+  if (!now) now = new Date();
+  return new Intl.DateTimeFormat('sv-SE', {
+    timeZone: 'Asia/Taipei',
+    year: 'numeric', month: '2-digit', day: '2-digit',
+  }).format(now);
+}
+
+function textHash6(text) {
+  const normalized = (text || '').normalize('NFKC').replace(/\s+/g, ' ').trim().toLowerCase();
+  if (!normalized) return 'empty';
+  return crypto.createHash('sha256').update(normalized).digest('hex').slice(0, 6);
+}
+
+async function insertDailyEntry(pool, tableName, { eventAt, source, tag, text, agentId, sessionId, metadata, dedupeKey }) {
+  const shouldUpsert = dedupeKey && UPSERT_TAGS.has(tag);
+  const sql = shouldUpsert
+    ? `INSERT INTO ${tableName}
+         (event_at, source, tag, text, agent_id, session_id, metadata, dedupe_key)
+       VALUES ($1, $2, $3, $4, $5, $6, $7, $8)
+       ON CONFLICT (dedupe_key) DO UPDATE SET
+         text = EXCLUDED.text,
+         event_at = EXCLUDED.event_at,
+         metadata = EXCLUDED.metadata
+       RETURNING id, event_at, source, tag, text`
+    : `INSERT INTO ${tableName}
+         (event_at, source, tag, text, agent_id, session_id, metadata, dedupe_key)
+       VALUES ($1, $2, $3, $4, $5, $6, $7, $8)
+       ON CONFLICT (dedupe_key) DO NOTHING
+       RETURNING id, event_at, source, tag, text`;
+  const result = await pool.query(sql, [
+    eventAt, source, tag || null, text,
+    agentId || 'main', sessionId || null,
+    metadata ? JSON.stringify(metadata) : '{}',
+    dedupeKey || null,
+  ]);
+  return result.rows[0] || null;
+}
+
+async function getDailyEntries(pool, tableName, date, agentId) {
+  const result = await pool.query(
+    `SELECT * FROM ${tableName}
+      WHERE (event_at AT TIME ZONE 'Asia/Taipei')::date = $1
+        AND ($2::text IS NULL OR agent_id = $2)
+      ORDER BY event_at ASC`,
+    [date, agentId || null]
+  );
+  return result.rows;
+}
+
+async function fetchDailyContext(pool, date, agentId, tableName = null) {
+  if (!tableName) return '';
+  const rows = await getDailyEntries(pool, tableName, date, agentId);
+  if (!rows || rows.length === 0) return '';
+  let currentFocus = '', currentTodo = '';
+  const entries = [];
+  for (const row of rows) {
+    if (row.tag === '[FOCUS]') currentFocus = row.text;
+    else if (row.tag === '[TODO]') currentTodo = row.text;
+    else if (!row.tag || row.tag === '[CLI]') {
+      const time = new Date(row.event_at).toLocaleTimeString('sv-SE', {
+        timeZone: 'Asia/Taipei', hour: '2-digit', minute: '2-digit',
+      });
+      entries.push(`- (${time}) ${row.text}`);
+    }
+  }
+  const recentEntries = entries.slice(-20);
+  const parts = [];
+  if (currentFocus) parts.push(`當前焦點: ${currentFocus}`);
+  if (currentTodo) parts.push(`當前待辦:\n${currentTodo}`);
+  if (recentEntries.length > 0) parts.push(`今日紀錄:\n${recentEntries.join('\n')}`);
+  let text = parts.join('\n\n');
+  if (text.length > 3000) text = text.slice(0, 3000) + '\n...(truncated)';
+  return text;
+}
+
+async function writeDailyEntries({
+  sections, recap, pool, sessionId, agentId, logger = console,
+  source = 'afterburn', tag = null, now, dailyTable,
+}) {
+  if (!dailyTable) return { inserted: 0, focusUpdated: false, todoUpdated: false };
+  if (!now) now = new Date();
+  const date = taipeiDateString(now);
+  let inserted = 0, focusUpdated = false, todoUpdated = false;
+
+  // Session bullets
+  if (sections?.session_entries) {
+    const entryLines = sections.session_entries.split('\n');
+    const bullets = entryLines.filter(l => l.trim().startsWith('- ')).map(l => l.trim().slice(2));
+    for (const bullet of bullets) {
+      const timeMatch = bullet.match(/^\((\d{2}:\d{2})\)\s*(.*)/);
+      const text = timeMatch ? timeMatch[2] : bullet;
+      const eventAt = now.toISOString();
+      const row = await insertDailyEntry(pool, dailyTable, {
+        eventAt, source, tag, text,
+        agentId, sessionId, metadata: {},
+        dedupeKey: `daily:${date}:${textHash6(text)}`,
+      });
+      if (row) inserted++;
+    }
+    if (logger.info) logger.info(`[default-persona] wrote ${inserted} daily entries to ${dailyTable}`);
+  }
+
+  // Focus
+  if (recap?.focus_decision === 'update' && recap.focus) {
+    await insertDailyEntry(pool, dailyTable, {
+      eventAt: now.toISOString(), source, tag: '[FOCUS]',
+      text: recap.focus, agentId, sessionId,
+      metadata: { proposed_by: source },
+      dedupeKey: `daily:${date}:focus:${source}`,
+    });
+    focusUpdated = true;
+    if (logger.info) logger.info(`[default-persona] focus updated: ${recap.focus.slice(0, 60)}`);
+  }
+
+  // TODO
+  if (recap?.todo_new?.length > 0 || recap?.todo_done?.length > 0) {
+    const todayEntries = await getDailyEntries(pool, dailyTable, date, agentId);
+    let currentItems = [];
+    for (const row of todayEntries) {
+      if (row.tag === '[TODO]') {
+        currentItems = row.text.split('\n').map(s => s.replace(/^[-•]\s*/, '').trim()).filter(Boolean);
+      }
+    }
+    if (recap.todo_done?.length > 0) {
+      for (const done of recap.todo_done) {
+        const dl = done.toLowerCase();
+        currentItems = currentItems.filter(item => {
+          const il = item.toLowerCase();
+          return il !== dl && !il.includes(dl) && !dl.includes(il);
+        });
+      }
+    }
+    if (recap.todo_new?.length > 0) {
+      for (const n of recap.todo_new) {
+        if (!currentItems.some(i => i.toLowerCase() === n.toLowerCase())) currentItems.push(n);
+      }
+    }
+    await insertDailyEntry(pool, dailyTable, {
+      eventAt: now.toISOString(), source, tag: '[TODO]',
+      text: currentItems.map(i => `- ${i}`).join('\n') || '（全部完成）',
+      agentId, sessionId,
+      metadata: { proposed_by: source, todo_new: recap.todo_new, todo_done: recap.todo_done },
+      dedupeKey: `daily:${date}:todo:${source}`,
+    });
+    todoUpdated = true;
+    if (logger.info) logger.info(`[default-persona] todo updated: ${currentItems.length} items`);
+  }
+
+  // Handoff
+  if (sections?.handoff) {
+    const handoff = parseHandoffSection(sections.handoff);
+    if (handoff) {
+      let handoffText;
+      switch (handoff.status) {
+        case 'completed': handoffText = `上一段已完成 ${handoff.lastStep}`; break;
+        case 'blocked': handoffText = `上一段卡在 ${handoff.lastStep}`; break;
+        default: handoffText = `上一段停在 ${handoff.lastStep}`;
+      }
+      if (handoff.next && handoff.next !== '無') handoffText += `，下一步建議 ${handoff.next}`;
+      if (handoff.decided) handoffText += `，已決定 ${handoff.decided}`;
+      if (handoff.blocker && handoff.status !== 'blocked') handoffText += `，卡在 ${handoff.blocker}`;
+      handoffText += '。';
+      await insertDailyEntry(pool, dailyTable, {
+        eventAt: now.toISOString(), source, tag: '[HANDOFF]',
+        text: handoffText, agentId, sessionId,
+        metadata: { ...handoff, proposed_by: source },
+        dedupeKey: `daily:${date}:handoff:${source}`,
+      });
+      if (logger.info) logger.info(`[default-persona] handoff written: ${handoffText.slice(0, 80)}`);
+    }
+  }
+
+  return { inserted, focusUpdated, todoUpdated };
+}
+
+module.exports = {
+  taipeiDateString, textHash6,
+  insertDailyEntry, getDailyEntries, fetchDailyContext, writeDailyEntries,
+  UPSERT_TAGS,
+};