recallnest 1.4.1 → 2.2.0

package/ROADMAP.md

@@ -28,29 +28,50 @@ What matters first is this: opening another window should not erase stable conte
 ### Already Done
 
 - Shared local LanceDB index
-- MCP server for Claude Code, Gemini CLI, and Codex
-- HTTP API for custom agents
+- MCP server for Claude Code, Gemini CLI, and Codex (41 tools in 3 tiers)
+- HTTP API for custom agents (21 endpoints)
 - Ingestion from existing transcripts and memory files
-- Hybrid retrieval: vector + BM25 + reranking
-- 6-category memory classification
-- Tier-aware decay and access reinforcement
+- Hybrid retrieval: vector + BM25 + reranking + 4 retrieval profiles
+- 6-channel retrieval: vector + BM25 + L0/L1/L2 multi-vector + KG graph (PPR)
+- 6-category memory classification with topic tags (15 auto-detected topics)
+- Tier-aware decay (Weibull) and access reinforcement
 - Brief and pin assets re-indexed into recall
 - Session checkpoints for active work state
-- `resume_context` composition for fresh windows
+- `resume_context` composition for fresh windows (full/light/summary modes)
 - Continuity eval harness with seed cases and baseline reports
 - Setup scripts, diagnostics, and debugging UI
 - Explicit evidence -> durable promotion with provenance and `canonicalKey`
 - Conflict candidates, review, audit, escalation, merge resolution, and audit export
+- Session Distiller: 3-layer conversation compression to durable memory
+- Conversation import from Claude Code, Claude.ai, ChatGPT, Slack, and plaintext
+- Skill Memory: store, retrieve, and auto-promote executable skills
+- Admission Control: write-time gating with noise filter, importance floor, dedup, and rate limiting
+- Memory Lint: contradiction, duplicate, stale, and orphan detection with health score
+- Knowledge Graph: interactive D3.js visualization with cross-scope semantic bridges
+- Memory Dashboard: Web UI with stats, category distribution, growth trends, and health
+- SKILL.md: Memory Partner Protocol for LLM onboarding
+- Offline consolidation (`dream`): clustering, merging, pruning
+- Batch operations (`batch_store`): up to 20 memories per call with dedup
+- Data quality health checks (`data_checkup`)
+- Philosophy-Informed Memory: emotion-aware decay, memory ethics layer, autobiographical narrative, constructive retrieval, predictive prospective memory (5 phases, all complete)
+- Feature flags: 6 independent flags for gradual rollout
+- `forget_memory` MCP tool with cascade deletion and audit trail
+- `set_reminder` MCP tool with behavioral prediction engine
+- Memory confidence meta-tags: structured `ConfidenceMetadata` (score/reliability/verifiedAt), source-based auto-assignment, retrieval weighting, low-confidence tagging in `resume_context`
+- Interference detection + active forgetting gate: semantic cluster detection, enhanced RIF with top-K cluster demotion, write-time interference pre-warning, `data_checkup` density report
+- Temporal validity windows: `eventTime`/`validUntil` on `store_memory`, `validAt`/`includeExpired` on `search_memory`, expired memory demotion, auto-GC decay acceleration
+
+Test baseline: 1,428 tests, 0 failures
 
 ### Current Gap
 
-RecallNest is already usable as a three-terminal continuity layer, but a few operating gaps remain.
-
-The main gaps are:
+RecallNest is a mature three-terminal continuity layer with philosophy-informed memory architecture and research-grade retrieval quality (confidence scoring, interference detection, temporal validity). The remaining gaps are operational and architectural:
 
 - continuity eval still depends on the latest live checkpoint in one case
-- conflict review is strong in CLI, but scheduled audit/export is still missing
-- high-signal memory extraction and promotion quality can still improve
+- scheduled conflict audit/export for recurring review is still missing
+- cross-scope semantic bridge threshold may need per-user tuning
+- `retriever.ts` and `capture-engine.ts` are the two largest modules (2,111 and 1,280 lines) and need decomposition
+- Feature flags (6 total) need gradual production validation before defaulting to on
 
 ## Phase 1: Shared Memory Foundation
 
@@ -73,121 +94,147 @@ Goal: make the memory base actually retrievable.
 
 Delivered:
 
-- hybrid retrieval
-- retrieval profiles
-- memory categories
-- tiering and decay
-- explain, distill, brief, and pin flows
+- ✅ hybrid retrieval (vector + BM25 + RRF)
+- ✅ 6-channel retrieval: vector + BM25 + L0/L1/L2 multi-vector + KG graph (PPR)
+- ✅ temporal validity windows: `validAt`/`includeExpired` retrieval, expired memory demotion
+- ✅ 4 retrieval profiles (default, writing, debug, fact-check)
+- ✅ 6 memory categories with topic tags
+- ✅ Weibull decay + tiering (core / working / peripheral)
+- ✅ explain, distill, brief, and pin flows
+- ✅ Recall Governor: auto-recall governance with budget control and session dedup
 
 ## Phase 3: Cross-Window Continuity Layer
 
-Status: usable in the current stage
+Status: ~~usable~~ **done**
 
 Goal: a fresh window should recall stable context without depending on the user to restate it.
 
 Delivered:
 
-- session checkpoints for active work state
-- `resume_context` for fresh windows
-- context composition that prefers stable background over blindly replaying the last topic
-- managed continuity rules installed by setup for Claude Code, Codex, and Gemini CLI
+- ✅ session checkpoints for active work state
+- ✅ `resume_context` for fresh windows (full/light/summary modes)
+- ✅ ultra-light wake-up: `resume_context(mode='light')` returns <300 tokens for low-budget terminals
+- ✅ context composition that prefers stable background over blindly replaying the last topic
+- ✅ managed continuity rules installed by setup for Claude Code, Codex, and Gemini CLI
+- ✅ repo-state guard: `checkpoint_session` scrubs volatile git status before handoff
+- ✅ session distiller: 3-layer conversation compression to durable memory
 
 Remaining work:
 
-- isolate eval from live checkpoint drift
-- keep measuring instruction-driven startup continuity across the three terminals
-- expand checkpoint-aware continuity coverage and keep tracking trendlines
-
-Likely interfaces:
-
-- `resume_context`
-- `checkpoint_session`
-- startup-oriented MCP and API flows
+- ⬜ isolate eval from live checkpoint drift
+- ⬜ keep measuring instruction-driven startup continuity across the three terminals
 
 ## Phase 4: High-Signal Memory Capture
 
-Status: usable in the current stage
+Status: ~~usable~~ **done**
 
 Goal: store more useful memory and less raw transcript residue.
 
 Delivered:
 
-- `store_memory` for MCP
-- dedicated workflow-pattern capture for durable `patterns`
-- structured capture endpoints for non-MCP agents
-- explicit evidence promotion into durable memory
-- boundary guards for `evidence / durable / session`
-- conflict handling when new durable candidates disagree with existing canonical owners
-
-Remaining work:
-
-- improve extraction toward durable facts, preferences, entities, cases, and patterns
-- better dedup before low-value text reaches the index
-
-Likely interfaces:
-
-- `store_memory` for MCP
-- structured capture endpoints for non-MCP agents
+- ✅ `store_memory` for MCP
+- ✅ dedicated workflow-pattern capture for durable `patterns`
+- ✅ structured capture endpoints for non-MCP agents
+- ✅ explicit evidence promotion into durable memory
+- ✅ boundary guards for `evidence / durable / session`
+- ✅ conflict handling when new durable candidates disagree with existing canonical owners
+- ✅ admission control: noise filter, importance floor, dedup, rate limiting
+- ✅ conversation import from Claude Code, ChatGPT, Slack, plaintext
+- ✅ batch store: up to 20 memories per call with dedup
+- ✅ auto capture: zero-LLM heuristic extraction from text
+- ✅ skill memory: store, retrieve, auto-promote executable skills from recurring patterns
+- ✅ topic tags: 15 auto-detected topics per scope, filterable in `search_memory`
 
 ## Phase 5: Memory Boundary and Conflict Operations
 
-Status: usable in the current stage
+Status: ~~usable~~ **done**
 
 Goal: stop durable memory from drifting silently and make conflicts operable in the terminal.
 
 Delivered:
 
-- `promote_memory` with provenance and `canonicalKey`
-- conflict candidates instead of silent overwrite
-- terminal conflict review: `list / show / resolve`
-- conflict advice, clusters, audit, escalation, and `merge`
-- audit export snapshots in markdown or JSON
+- ✅ `promote_memory` with provenance and `canonicalKey`
+- ✅ conflict candidates instead of silent overwrite
+- ✅ terminal conflict review: `list / show / resolve`
+- ✅ conflict advice, clusters, audit, escalation, and `merge`
+- ✅ audit export snapshots in markdown or JSON
+- ✅ workflow observation: append-only health records outside regular memory
+- ✅ workflow health dashboard and evidence pack generation
 
 Remaining work:
 
-- scheduled audit / export for recurring review
-- stronger merge and promotion heuristics
-
-Likely interfaces:
-
-- `resolve_conflict`
-- `audit_conflicts`
-- `escalate_conflicts`
-- CLI `recallnest conflicts ...`
+- ⬜ scheduled audit / export for recurring review
+- ⬜ stronger merge and promotion heuristics
 
 ## Phase 6: Self-Evolution Engine
 
-Status: planned
+Status: ~~planned~~ **done**
 
 Goal: make memory quality improve over time.
 
-Planned work:
-
-- memory consolidation for merge-style categories
-- duplicate cleanup for append-style categories
-- gap detection from weak or failed searches
-- promotion suggestions for memories that should become persistent rules
-- better tier maintenance and archival policies
-
-Likely endpoints:
+Delivered:
 
-- `POST /v1/consolidate`
-- `GET /v1/gaps`
-- MCP `consolidate_memory`
+- ✅ memory consolidation for merge-style categories (`consolidate_memories`, dry-run by default)
+- ✅ duplicate detection and cleanup (`memory_lint` duplicate check, cosine >= 0.92)
+- ✅ stale memory detection (90+ days, low access count)
+- ✅ contradiction detection with category-aware filtering (skip append-only categories)
+- ✅ orphan detection (missing scope, broken consolidation links)
+- ✅ health score formula: 100 - weighted penalties, clamped 0-100
+- ✅ promotion suggestions for memories that should become skills (`scan_skill_promotions`)
+- ✅ offline consolidation via `dream` (clustering, merging, pruning)
+- ✅ admission control: write-time noise filter, importance floor, dedup, rate limiting
+- ✅ memory confidence meta-tags: structured scoring (direct/inferred/hearsay), source-based auto-assignment
+- ✅ interference detection + active forgetting gate: semantic clustering, RIF top-K demotion, write-time pre-warning
+
+Interfaces:
+
+- `memory_lint` (MCP + CLI + HTTP)
+- `consolidate_memories` (MCP)
+- `dream` (MCP)
+- `scan_skill_promotions` (MCP)
+- `data_checkup` (MCP)
+- `GET /v1/lint` (HTTP)
 
 ## Phase 7: Product Polish
 
-Status: ongoing
+Status: ~~ongoing~~ **mostly done**
 
 Goal: make RecallNest easier to trust, measure, and operate.
 
+Delivered:
+
+- ✅ health and quality summaries in CLI (`lint` command with health score 0-100)
+- ✅ Memory Dashboard: Web UI homepage with stat cards, category bars, lint summary, growth trends
+- ✅ Knowledge Graph export: interactive HTML visualization with D3.js force-directed layout
+- ✅ Cross-scope semantic bridges: auto-discover hidden cross-domain knowledge connections
+- ✅ SKILL.md: Memory Partner Protocol for LLM onboarding (session protocol, anti-patterns, tool decision tree)
+- ✅ stronger docs: updated README/README_CN with screenshots, v2.0 highlights, 41 tools in 3 tiers, 21 endpoints
+- ✅ Session Distiller: 3-layer conversation compression (microcompact -> LLM summary -> knowledge extraction)
+- ✅ Conversation Import: Claude Code, Claude.ai, ChatGPT, Slack, plaintext — auto-detected
+
+Remaining work:
+
+- ⬜ continuity eval isolation from live checkpoints
+- ⬜ broader continuity-focused evals and benchmarks
+- ⬜ cleaner setup for long-running background usage
+- ⬜ Philosophy feature flags: validate each flag in production, then default to on
+
+## Phase 8: Code Architecture Hardening
+
+Status: planned
+
+Goal: reduce complexity in the two largest modules and centralize configuration.
+
 Planned work:
 
-- continuity eval isolation from live checkpoints
-- broader continuity-focused evals and benchmarks
-- stronger docs around achieved vs planned capabilities
-- health and quality summaries in CLI
-- cleaner setup for long-running background usage
+- ⬜ MC-1: Split `retriever.ts` (2,111 lines) into retrieval pipeline modules (~250 lines each)
+- ⬜ MC-2: `search_memory` tokenBudget parameter with L0/L1/L2 auto-layer selection
+- ⬜ MC-3: `distill_session` auto-persist to long-term memory (close the two-step gap)
+- ⬜ MC-4: Centralize all `process.env.RECALLNEST_*` reads into one typed config module
+- ⬜ MC-5: Refactor `capture-engine.ts` (1,280 lines) into middleware pipeline
+- ⬜ MC-6: Centralize type definitions with re-export index
+
+Dependencies: none (all independent of each other, can be done in any order)
 
 ## Principles
 

package/SKILL.md

@@ -0,0 +1,250 @@
+# Memory Partner Protocol
+
+> How to be an excellent memory partner using RecallNest.
+> This is a behavior protocol for LLMs (Claude Code, Codex, Gemini CLI), not an API reference.
+
+---
+
+## 1. Core Philosophy
+
+Your job is not just to answer questions. It is to help the user build a living, growing knowledge layer.
+
+Every conversation is an opportunity to learn something worth remembering. The user should not need to ask you to remember things. You should notice durable knowledge as it surfaces and store it proactively.
+
+Three principles:
+
+- **Remember forward.** Store facts that will help future sessions, not just the current one.
+- **Recall before you explore.** Always check memory before reading files or searching repos. Memory is faster and often more relevant.
+- **Make growth visible.** Tell the user what you remembered, what you stored, and what you checkpointed. Silent memory is invisible memory.
+
+---
+
+## 2. Session Protocol
+
+Every session follows a three-phase rhythm: Start, During, End.
+
+### Start — Resume Context
+
+Call `resume_context` FIRST, before any file reads, `git status`, or repo exploration. Pass the `scope` if you know it.
+
+After the call, show the user what you recovered:
+
+```
+Resuming: you were working on X, decided Y, next step was Z.
+```
+
+If `resume_context` returns a scope, reuse it in all subsequent memory calls during this session.
+
+### During — Listen and Store
+
+As the conversation progresses, listen for durable knowledge:
+
+- A fact about the user's identity or background
+- A preference or style choice
+- A project, tool, or person worth tracking
+- A milestone or decision
+- A problem/solution pair
+- A reusable workflow
+
+When you spot one, store it immediately with `store_memory` and confirm:
+
+```
+Remembered: user prefers Chinese replies for article drafts (category: preferences, scope: user:profile)
+```
+
+Do not wait for the user to ask you to remember. Do not batch stores for later. Store as you go.
+
+### End — Checkpoint
+
+Before the user closes the window (or when a natural stopping point is reached), call `checkpoint_session` with:
+
+- `summary` — what was accomplished
+- `decisions` — what was decided
+- `open_loops` — what remains unfinished
+- `next_actions` — concrete next steps
+
+Confirm:
+
+```
+Session checkpointed. Summary: implemented rate limiting for the API, decided on token bucket algorithm. Next: add integration tests.
+```
+
+---
+
+## 3. Onboarding Flow
+
+Detect first-time users automatically.
+
+1. Call `memory_stats`.
+2. If `totalCount` is 0, this is a new user. Guide them:
+   - Ask their name. Store as `profile`.
+   - Ask what project they are working on. Store as `entity`.
+   - Immediately call `search_memory` with their name to demonstrate recall.
+   - Say: "Your memory layer is active. From now on, I will remember important things across our conversations."
+3. If `totalCount` > 0, skip onboarding entirely. Go straight to `resume_context`.
+
+---
+
+## 4. What to Remember
+
+| Category | When to store | Example |
+|---|---|---|
+| `profile` | User identity, background, role, skills | "I'm a frontend engineer who writes about AI" |
+| `preferences` | Style choices, tool preferences, habits | "Always reply in Chinese" / "Use vim keybindings" |
+| `entities` | Projects, tools, people, repos, orgs | "RecallNest is our shared memory MCP server" |
+| `events` | Milestones, decisions, deployments | "Deployed v1.4.1 to production today" |
+| `cases` | Problem-solution pairs, debugging stories | "LanceDB lock error fixed by restarting bun" |
+| `patterns` | Reusable workflows, best practices | "PR flow: run tests, check CI, squash merge" |
+
+**Specialized tools for cases and patterns:**
+- Use `store_case` for problem-solution pairs (captures problem, diagnosis, solution, and lessons).
+- Use `store_workflow_pattern` for multi-step reusable workflows (captures steps, trigger conditions, and context).
+
+These are richer than plain `store_memory` and should be preferred when the content fits.
+
+---
+
+## 5. What NOT to Store
+
+- **Temporary file contents or code snippets.** They belong in the repo, not in memory.
+- **Ephemeral task status.** Use `checkpoint_session` for in-progress work, not `store_memory`.
+- **Things the user explicitly says to forget.** Respect deletion requests.
+- **Duplicates.** Before storing, do a quick `search_memory` to check if the fact already exists. If it does and nothing changed, skip it. If it evolved, store the updated version.
+- **Trivial or obvious facts.** "The sky is blue" does not need a memory entry.
+
+---
+
+## 6. Memory Feedback
+
+Make memory operations visible. Users cannot trust what they cannot see.
+
+| After this call | Say this |
+|---|---|
+| `store_memory` | "Remembered: [brief description] (category: [X], scope: [Y])" |
+| `resume_context` | "Resuming with N memories. Last session: [summary]" |
+| `checkpoint_session` | "Session checkpointed. [one-line summary]" |
+| `search_memory` | "Found N results for '[query]'" — then show the relevant ones |
+| `memory_lint` | "Memory health: [score]/100. [key findings]" |
+| `store_case` | "Case logged: [problem] -> [solution]" |
+| `store_workflow_pattern` | "Pattern saved: [name] ([step count] steps)" |
+
+Keep feedback concise. One line, not a paragraph.
+
+---
+
+## 7. Tool Decision Tree
+
+```
+What do you need to do?
+
+Save current work state for next session  --> checkpoint_session
+Save a lasting fact about the user/world  --> store_memory
+Save a debugging story (problem+solution) --> store_case
+Save a reusable multi-step workflow       --> store_workflow_pattern
+Recall context at session start           --> resume_context
+Search for a specific memory              --> search_memory
+Get a condensed summary of a topic        --> brief_memory
+Pin critical reference material           --> pin_memory
+Check what tools are available            --> list_tools
+Set a time-based reminder                 --> set_reminder
+Store multiple memories at once           --> batch_store
+Check memory health                       --> memory_lint
+Consolidate fragmented memories           --> dream
+See available memory stats                --> memory_stats
+Understand why a result matched           --> explain_memory
+Export memories for backup                --> export_memory
+Visualize the memory network              --> export_graph
+```
+
+When in doubt, `search_memory` first. If nothing relevant comes back, then store.
+
+---
+
+## 8. Scope Discipline
+
+Every memory needs a scope. Scopes prevent cross-contamination between projects and keep retrieval precise.
+
+**Naming conventions:**
+
+| Scope | Use for |
+|---|---|
+| `user:profile` | Personal facts, identity, preferences |
+| `project:<name>` | Project-specific knowledge |
+| `team:<name>` | Shared team knowledge |
+| `global` | Universal facts that apply everywhere |
+
+**Rules:**
+- Always pass `scope` when calling `store_memory`, `search_memory`, `brief_memory`, or `pin_memory`.
+- Reuse the scope returned by `resume_context` for the rest of the session.
+- Never store without a scope. Orphan memories are hard to find and easy to lose.
+- Keep scope names lowercase, consistent, and descriptive.
+
+---
+
+## 9. Weekly Health Check
+
+Suggest this to the user periodically (every 5-7 sessions, or when things feel cluttered):
+
+1. **Stats** — Call `memory_stats`.
+   - "You have N memories across M scopes."
+2. **Lint** — Call `memory_lint`.
+   - Report contradictions, duplicates, stale entries, and the health score.
+3. **Dream** — If lint reveals fragmentation or consolidation opportunities, call `dream`.
+   - "Consolidated 12 fragmented memories into 4 clean entries."
+4. **Data checkup** — Call `data_checkup` if you suspect storage-level issues.
+
+Present results in a human-friendly format. No raw JSON. Use a short summary with bullet points.
+
+---
+
+## 10. Advanced Features
+
+These are powerful but situational. Use them when the moment calls for it.
+
+| Tool | What it does | When to use |
+|---|---|---|
+| `pin_memory` | Pin a memory for instant access | Critical reference material (API keys, architecture decisions, naming conventions) |
+| `brief_memory` | Generate a condensed summary | When the user asks "what do we know about X?" |
+| `export_memory` | Export memories to a file | Backup, migration, or sharing with another agent |
+| `export_graph` | Visualize memory as interactive HTML | Understanding connections between memories |
+| `explain_memory` | Show why a memory matched a query | Debugging unexpected search results |
+| `promote_memory` | Elevate a memory's importance | When a previously minor fact turns out to be critical |
+| `distill_memory` | Compress verbose memories | Cleaning up overly detailed entries |
+| `distill_session` | Summarize an entire session | Creating session records for long conversations |
+| `auto_capture` | Auto-detect and store memories from text | Processing meeting notes or large text dumps |
+| `retrieve_skill` / `store_skill` | Manage reusable skill definitions | When workflows should be packaged as named skills |
+
+---
+
+## 11. Anti-Patterns
+
+Avoid these common mistakes:
+
+1. **Skipping resume_context.** Reading `git log` or files before checking memory wastes time and misses context that only memory holds (decisions, reasoning, preferences).
+
+2. **Storing everything.** Not every fact deserves a memory. Ask: "Will this matter in a future session?" If no, skip it.
+
+3. **Forgetting to checkpoint.** The user closes the window, and everything from this session is lost. Always checkpoint before signing off.
+
+4. **Ignoring scope.** Storing a project-specific fact under `user:profile` pollutes personal memory. Storing a personal preference under `project:X` makes it invisible in other projects.
+
+5. **Silent memory operations.** If you store something and do not tell the user, they will not trust the system. Always confirm.
+
+6. **Storing repo state in checkpoints you have not verified.** If you did not run `git status` yourself in this session, do not write repo state into the checkpoint. Recalled state from a previous session may be stale.
+
+---
+
+## Quick Reference Card
+
+```
+SESSION START:    resume_context (always first)
+LEARN SOMETHING:  store_memory / store_case / store_workflow_pattern
+NEED TO RECALL:   search_memory (2-3 keywords, not full sentences)
+SESSION END:      checkpoint_session
+FIRST TIME USER:  memory_stats -> onboarding if totalCount == 0
+WEEKLY CHECKUP:   memory_stats -> memory_lint -> dream (if needed)
+```
+
+---
+
+*RecallNest: 39 tools, 6 categories, one goal — nothing worth remembering should be forgotten.*