@shadowforge0/aquifer-memory 1.5.9 → 1.6.0

package/.env.example

@@ -0,0 +1,23 @@
+DATABASE_URL=postgresql://aquifer:aquifer@localhost:5432/aquifer
+AQUIFER_SCHEMA=aquifer
+AQUIFER_TENANT_ID=default
+
+# Legacy is the default for backward compatibility. Use curated only after
+# finalization and scoped serving have been verified for your host.
+AQUIFER_MEMORY_SERVING_MODE=legacy
+# AQUIFER_MEMORY_SERVING_MODE=curated
+# AQUIFER_MEMORY_ACTIVE_SCOPE_KEY=project:example
+# AQUIFER_MEMORY_ACTIVE_SCOPE_PATH=global,project:example
+
+AQUIFER_EMBED_BASE_URL=http://localhost:11434/v1
+AQUIFER_EMBED_MODEL=bge-m3
+# EMBED_PROVIDER=ollama
+# EMBED_PROVIDER=openai
+# OPENAI_API_KEY=sk-...
+
+# Optional built-in summarization.
+# AQUIFER_LLM_BASE_URL=http://localhost:11434/v1
+# AQUIFER_LLM_MODEL=llama3.1
+
+# Startup migration behavior: apply | check | off.
+AQUIFER_MIGRATIONS_MODE=apply

package/README.md

@@ -2,9 +2,9 @@
 
 # 🌊 Aquifer
 
-**PG-native long-term memory for AI agents**
+**Long-term memory for AI agents, backed by PostgreSQL.**
 
-*Turn-level embedding, hybrid RRF ranking, trust scoring, entity intersection, knowledge graph, entity scoping — all on PostgreSQL + pgvector.*
+*Store sessions, enrich them, and recall the exact turn where a decision happened — without adding a separate vector database.*
 
 [![npm version](https://img.shields.io/npm/v/@shadowforge0/aquifer-memory)](https://www.npmjs.com/package/@shadowforge0/aquifer-memory)
 [![PostgreSQL 15+](https://img.shields.io/badge/PostgreSQL-15%2B-336791)](https://www.postgresql.org/)
@@ -17,6 +17,73 @@
 
 ---
 
+## Start Here
+
+Aquifer is designed to have a short default path: start PostgreSQL + embeddings, run `quickstart`, then point your MCP client at `aquifer mcp`.
+
+For library API usage, skip to [API Reference](#api-reference). For a slightly more guided first run, see [docs/getting-started.md](docs/getting-started.md).
+
+### 1. Start the local stack
+
+```bash
+docker compose up -d
+# PostgreSQL 16 + pgvector and Ollama with bge-m3 (auto-pulled).
+# First run pulls the model — `docker compose logs -f ollama-pull` to watch.
+```
+
+Already running PostgreSQL + pgvector and an embedding endpoint? Skip this step. `quickstart` picks up `DATABASE_URL` and embed settings from your environment if you already have them.
+
+### 2. Verify end-to-end
+
+```bash
+npx --yes @shadowforge0/aquifer-memory quickstart
+```
+
+`quickstart` autodetects `localhost:5432` PostgreSQL and `localhost:11434` Ollama (from step 1 or your own), runs migrations, embeds a test session, recalls it, and cleans up. If it prints `✓ Aquifer is working`, you're done.
+
+For ongoing use, install it into your project so you skip the `npx` resolution cost: `npm install @shadowforge0/aquifer-memory` then `npx aquifer quickstart`.
+
+Using OpenAI instead of Ollama? `export EMBED_PROVIDER=openai` + `OPENAI_API_KEY=sk-...` before `quickstart` — model defaults to `text-embedding-3-small`.
+
+### 3. Connect your MCP client
+
+Claude Code, Claude Desktop, or any MCP-capable client — drop this into `.mcp.json` (project-level) or `claude_desktop_config.json`:
+
+```jsonc
+{
+  "mcpServers": {
+    "aquifer": {
+      "command": "npx",
+      "args": ["--yes", "@shadowforge0/aquifer-memory", "mcp"],
+      "env": {
+        "DATABASE_URL": "postgresql://aquifer:aquifer@localhost:5432/aquifer",
+        "EMBED_PROVIDER": "ollama",
+        "AQUIFER_MEMORY_SERVING_MODE": "legacy"
+      }
+    }
+  }
+}
+```
+
+Or run it directly: `DATABASE_URL=... EMBED_PROVIDER=ollama npx aquifer mcp`. The MCP server itself stays strict about env; `quickstart` autodetect is the try-it path, not the production one.
+
+Keep `AQUIFER_MEMORY_SERVING_MODE=legacy` for first rollout. Switch to `curated` only when you want `session_recall` and `session_bootstrap` to serve active curated memory; `evidence_recall` stays the explicit audit/debug lane. Rollback is just flipping env or config back to `legacy`.
+
+### Common commands
+
+| Goal | Command |
+|---|---|
+| Verify setup | `npx aquifer quickstart` |
+| Start the MCP server | `npx aquifer mcp` |
+| Search memory manually | `npx aquifer recall "auth middleware"` |
+| Plan curated memory compaction | `npx aquifer compact --cadence daily --period-start 2026-04-27T00:00:00Z --period-end 2026-04-28T00:00:00Z` |
+| Inspect storage health | `npx aquifer stats` |
+| Enrich pending sessions | `npx aquifer backfill` |
+
+Need LLM summarization, the knowledge graph, OpenAI embeddings, reranking, or operations details? See [docs/setup.md](docs/setup.md) and [Environment Variables](#environment-variables).
+
+---
+
 ## Why Aquifer?
 
 Most AI memory systems bolt a vector DB on the side. Aquifer takes a different approach: **PostgreSQL is the memory**.
@@ -61,57 +128,6 @@ Sessions, summaries, turn-level embeddings, entity graph — all live in one dat
 
 ---
 
-## Quick Start (MCP Server)
-
-Two commands from zero to a working MCP memory server — no env vars to set. For library API usage, see [API Reference](#api-reference) below.
-
-### 1. Start the stack
-
-```bash
-docker compose up -d
-# PostgreSQL 16 + pgvector and Ollama with bge-m3 (auto-pulled).
-# First run pulls the model — `docker compose logs -f ollama-pull` to watch.
-```
-
-Already running PostgreSQL + pgvector and an embedding endpoint? Skip this step — `quickstart` picks up `DATABASE_URL` / `EMBED_PROVIDER` from your environment if you've set them.
-
-### 2. Verify
-
-```bash
-npx --yes @shadowforge0/aquifer-memory quickstart
-```
-
-That's it. `quickstart` autodetects `localhost:5432` PostgreSQL and `localhost:11434` Ollama (from step 1 or your own), runs migrations, embeds a test session, recalls it, and cleans up. If it prints `✓ Aquifer is working`, you're done.
-
-For ongoing use, install it into your project so you skip the `npx` resolution cost: `npm install @shadowforge0/aquifer-memory` then `npx aquifer quickstart`.
-
-Using OpenAI instead of Ollama? `export EMBED_PROVIDER=openai` + `OPENAI_API_KEY=sk-...` before `quickstart` — model defaults to `text-embedding-3-small`.
-
-### 3. Wire into your MCP client
-
-Claude Code, Claude Desktop, or any MCP-capable client — drop this into `.mcp.json` (project-level) or `claude_desktop_config.json`:
-
-```jsonc
-{
-  "mcpServers": {
-    "aquifer": {
-      "command": "npx",
-      "args": ["--yes", "@shadowforge0/aquifer-memory", "mcp"],
-      "env": {
-        "DATABASE_URL": "postgresql://aquifer:aquifer@localhost:5432/aquifer",
-        "EMBED_PROVIDER": "ollama"
-      }
-    }
-  }
-}
-```
-
-Or run it directly: `DATABASE_URL=... EMBED_PROVIDER=ollama npx aquifer mcp`. (MCP server itself stays strict about env — `quickstart`'s autodetect is the try-it path, not the production one.)
-
-Need LLM summarization, the knowledge graph, OpenAI embeddings, or the reranker? See [Environment Variables](#environment-variables) below and [docs/setup.md](docs/setup.md).
-
----
-
 ## Environment Variables
 
 | Variable | Required? | Purpose | Example |
@@ -132,16 +148,39 @@ Need LLM summarization, the knowledge graph, OpenAI embeddings, or the reranker?
 | `AQUIFER_RERANK_PROVIDER` | No | Reranker provider: `tei`, `jina`, `openrouter` | `tei` |
 | `AQUIFER_RERANK_BASE_URL` | No | Reranker endpoint | `http://localhost:8080` |
 | `AQUIFER_AGENT_ID` | No | Default agent ID | `main` |
+| `AQUIFER_MEMORY_SERVING_MODE` | No | Public serving mode: `legacy` default, or opt-in `curated` | `curated` |
+| `AQUIFER_MEMORY_ACTIVE_SCOPE_KEY` | No | Default active curated scope for recall/bootstrap | `project:aquifer` |
+| `AQUIFER_MEMORY_ACTIVE_SCOPE_PATH` | No | Ordered curated scope path for inheritance | `global,project:aquifer` |
 | `AQUIFER_MIGRATIONS_MODE` | No | Startup handshake mode: `apply` (default), `check`, `off` | `apply` |
 | `AQUIFER_MIGRATION_LOCK_TIMEOUT_MS` | No | Advisory-lock wait before `AQ_MIGRATION_LOCK_TIMEOUT` (default 30000) | `30000` |
+| `AQUIFER_INSIGHTS_DEDUP_MODE` | No | Insights semantic dedup mode: `off` (default), `shadow`, `enforce` — env wins over code for this field only, so operators can kill-switch without redeploy | `shadow` |
+| `AQUIFER_INSIGHTS_DEDUP_COSINE` | No | Cosine threshold for semantic merge (default `0.88`; warn outside `[0.75, 0.95]`) | `0.90` |
+| `AQUIFER_INSIGHTS_DEDUP_CLOSE_BAND_FROM` | No | Lower bound for close-band logging (`dedupNear`); must be below threshold (default `0.85`) | `0.82` |
 
 Full env-to-config mapping is in [consumers/shared/config.js](consumers/shared/config.js).
 
+Curated serving is opt-in. If a host needs rollback during rollout, set `AQUIFER_MEMORY_SERVING_MODE=legacy` and restart the MCP/CLI process; no destructive DB rollback is required.
+
+### Insights semantic dedup (1.5.10)
+
+When a cron extractor (`scripts/extract-insights-from-recent-sessions.js`) or any other caller writes insights via `commitInsight`, the canonical-key layer (1.5.3+) dedupes rows whose `canonicalClaim + entities` hash to the same value. But LLMs don't always produce the same `canonicalClaim` across runs, so 1.5.10 adds a second tier: `title + body` are embedded, matched against `(tenant, agent, type)`-scoped active rows, and a top cosine above `AQUIFER_INSIGHTS_DEDUP_COSINE` triggers supersede (enforce) or metadata-only would-merge logging (shadow). Close-band hits (`closeBandFrom ≤ cos < threshold`) write `metadata.dedupNear` without supersede so operators can tune thresholds without committing.
+
+Recommended rollout: `shadow` for one weekly cycle, inspect `SELECT metadata->>'shadowMatch' FROM insights WHERE metadata ? 'shadowMatch'`, then flip to `enforce`. Kill-switch: `AQUIFER_INSIGHTS_DEDUP_MODE=off` and restart.
+
+Pre-1.5.3 rows with `canonical_key_v2 IS NULL` are caught by the semantic tier but skip the canonical path; a startup warn points at the one-shot backfill:
+
+```bash
+DATABASE_URL=... \
+  node scripts/backfill-canonical-key.js --schema <schema> --agent <id>
+```
+
+The script is idempotent (`WHERE canonical_key_v2 IS NULL` guard) and race-safe with live writers.
+
 ---
 
 ## Host Integration
 
-MCP is the primary integration surface. Agent hosts connect to the Aquifer MCP server, which exposes five tools: `session_recall`, `session_feedback`, `session_bootstrap`, `memory_stats`, `memory_pending`.
+MCP is the primary integration surface. Agent hosts connect to the Aquifer MCP server, which exposes eight tools: `session_recall`, `evidence_recall`, `session_feedback`, `memory_feedback`, `feedback_stats`, `session_bootstrap`, `memory_stats`, `memory_pending`.
 
 | Integration | Route | Status | When to use |
 |-------------|-------|--------|-------------|
@@ -172,7 +211,7 @@ Add to your project's `.claude.json` or user-level MCP config:
 }
 ```
 
-Tools appear as `mcp__aquifer__session_recall`, `mcp__aquifer__session_feedback`, `mcp__aquifer__session_bootstrap`, etc.
+Tools appear as `mcp__aquifer__session_recall`, `mcp__aquifer__evidence_recall`, `mcp__aquifer__session_bootstrap`, `mcp__aquifer__session_feedback`, `mcp__aquifer__memory_feedback`, `mcp__aquifer__feedback_stats`, `mcp__aquifer__memory_stats`, `mcp__aquifer__memory_pending`.
 
 ### OpenClaw
 
@@ -196,7 +235,7 @@ Add to `openclaw.json` under `mcp.servers`:
 }
 ```
 
-Tools materialize as `aquifer__session_recall`, `aquifer__session_feedback`, `aquifer__session_bootstrap`, `aquifer__memory_stats`, `aquifer__memory_pending` (server name prefix added by the host).
+Tools materialize as `aquifer__session_recall`, `aquifer__evidence_recall`, `aquifer__session_feedback`, `aquifer__memory_feedback`, `aquifer__feedback_stats`, `aquifer__session_bootstrap`, `aquifer__memory_stats`, `aquifer__memory_pending` (server name prefix added by the host).
 
 The OpenClaw plugin (`consumers/openclaw-plugin.js`) is retained for session capture via `before_reset` but is **not** the recommended tool delivery path. Use MCP.
 
@@ -331,22 +370,6 @@ Built-in entity extraction and relationship tracking:
 - **Entity-session mapping**: which entities appear in which sessions
 - **Entity boost in ranking**: sessions with relevant entities score higher
 
----
-
-## Benchmark: LongMemEval
-
-We tested Aquifer's retrieval pipeline on [LongMemEval_S](https://github.com/xiaowu0162/LongMemEval) — 470 questions across 19,195 sessions with 98,795 turn embeddings. Per-question haystack scoping (matching the official protocol), bge-m3 embeddings via OpenRouter.
-
-| Pipeline | R@1 | R@3 | R@5 | R@10 |
-|----------|-----|-----|-----|------|
-| Turn-only (cosine) | 89.5% | 96.6% | 98.1% | 98.9% |
-| Three-way hybrid (FTS + session_emb + turn_emb → RRF) | 79.2% | 94.0% | 97.7% | 98.9% |
-| **Hybrid + Cohere Rerank v3.5 (top-30)** | **96.0%** | **98.5%** | **99.3%** | **99.8%** |
-
-Measured 2026-04-19 on Aquifer 1.2.1.
-
-**Key findings.** Turn-level embedding alone beats session-level (26.8% → 89.5% R@1, a 3× improvement). Hybrid fusion adds robustness at R@3-R@10 but trades R@1 because FTS + session-level signals spread the top candidate across adjacent sessions. Re-ranking the hybrid top-30 with a cross-encoder (Cohere Rerank v3.5) wins back the top-1 precision and then some — +16.9pt R@1 over hybrid baseline, and 6.5pt above pure turn-level cosine. That's the production pipeline Aquifer ships by default when a reranker is configured.
-
 ### Multi-Tenant
 
 Every table includes `tenant_id` (default: `'default'`). Isolation is enforced at the query level — no cross-tenant data leakage by design.
@@ -400,7 +423,7 @@ The MCP consumer (`consumers/mcp.js`) already wires `aquifer.init()` before `ser
 
 #### `aquifer.listPendingMigrations()` / `aquifer.getMigrationStatus()`
 
-Returns `{ required, applied, pending, lastRunAt }` via a `pg_tables` signature probe. No DDL runs. Use it from a health check or from a consumer that wants to surface drift before calling `init()`.
+Returns `{ required, applied, pending, lastRunAt }` via table and column signature probes (`pg_tables` plus `information_schema.columns` for alter-only migrations). No DDL runs. Use it from a health check or from a consumer that wants to surface drift before calling `init()`.
 
 #### `aquifer.migrate()`