recallnest 1.4.1 → 2.2.1

package/docs/architecture.md

@@ -1,168 +0,0 @@
-# Architecture
-
-> 架构总览：RecallNest 的分层设计，从接入层到存储层。
-
-边界说明：
-
-- structured capture 写 durable memory
-- checkpoint store 写 session state
-- raw ingest 写 evidence
-
-See [memory-boundary-contract.md](./memory-boundary-contract.md).
-
-## System Architecture
-
-```
-┌─────────────────────────────────────────────────────────────┐
-│                       Client Layer                           │
-├──────────┬──────────┬──────────┬──────────┬─────────────────┤
-│ Claude   │ Gemini   │ Codex    │ Custom   │ curl / any      │
-│ Code     │ CLI      │          │ Agents   │ HTTP client     │
-└────┬─────┴────┬─────┴────┬─────┴────┬─────┴────┬────────────┘
-     │          │          │          │          │
-     └──── MCP (stdio) ───┘          └── HTTP ──┘
-                │                        │
-                ▼                        ▼
-┌─────────────────────────────────────────────────────────────┐
-│                     Integration Layer                        │
-├────────────────────────┬────────────────────────────────────┤
-│    MCP Server          │       HTTP API Server               │
-│    (mcp-server.ts)     │       (api-server.ts)               │
-│                        │       port 4318                     │
-│  Tools:                │                                     │
-│  - search_memory       │  Endpoints:                         │
-│  - memory_stats        │  - POST /v1/recall                  │
-│  - brief_memory        │  - POST /v1/store                   │
-│  - distill_memory      │  - POST /v1/search                  │
-│  - pin_memory          │  - GET  /v1/stats                   │
-│  - explain_memory      │  - GET  /v1/health                  │
-│  - export_memory       │  - POST /v1/consolidate             │
-│                        │  - GET  /v1/gaps                    │
-└────────────┬───────────┴──────────────┬─────────────────────┘
-             │                          │
-             ▼                          ▼
-┌─────────────────────────────────────────────────────────────┐
-│                      Core Engine                             │
-├─────────────────────────────────────────────────────────────┤
-│                                                              │
-│  ┌─────────────┐  ┌──────────────┐  ┌────────────────────┐  │
-│  │  Retriever   │  │  Classifier  │  │  Query Expander    │  │
-│  │ (hybrid:     │  │ (6 categories│  │  (synonym +        │  │
-│  │  vector +    │  │  auto-assign)│  │   semantic expand) │  │
-│  │  BM25 + RRF) │  │              │  │                    │  │
-│  └──────┬───────┘  └──────────────┘  └────────────────────┘  │
-│         │                                                    │
-│  ┌──────┴───────┐  ┌──────────────┐  ┌────────────────────┐  │
-│  │  Decay       │  │  Access      │  │  Noise Filter      │  │
-│  │  Engine      │  │  Tracker     │  │  (relevance        │  │
-│  │  (Weibull)   │  │  (use it or  │  │   threshold)       │  │
-│  │              │  │   lose it)   │  │                    │  │
-│  └──────────────┘  └──────────────┘  └────────────────────┘  │
-│                                                              │
-│  ┌──────────────┐  ┌──────────────┐                          │
-│  │ Consolidator │  │ Gap Detector │   ← Self-Evolution       │
-│  │ (merge/dedup)│  │ (find blind  │                          │
-│  │              │  │  spots)      │                          │
-│  └──────────────┘  └──────────────┘                          │
-│                                                              │
-└──────────────────────────┬──────────────────────────────────┘
-                           │
-                           ▼
-┌─────────────────────────────────────────────────────────────┐
-│                     Storage Layer                            │
-├─────────────────────────┬───────────────────────────────────┤
-│  LanceDB               │  Jina Embeddings                   │
-│  (vector + columnar)    │  (v5, 1024-dim)                    │
-│                         │                                    │
-│  Tables:                │  Task-aware:                       │
-│  - memories (main)      │  - retrieval.query (for searches)  │
-│  - access_log           │  - retrieval.passage (for docs)    │
-│  - search_log           │                                    │
-└─────────────────────────┴───────────────────────────────────┘
-```
-
-## Data Flow
-
-### Ingestion Pipeline
-
-```
-Source Files                    Processing                  Storage
-─────────────                  ──────────                  ───────
-
-CC transcripts ─┐
-                │   ┌──────────────────┐   ┌───────────┐
-Codex sessions ─┼──►│  Chunker         │──►│ Embedder  │──► LanceDB
-                │   │  (split by turn, │   │ (Jina v5) │
-Gemini chats ───┤   │   noise filter)  │   └───────────┘
-                │   └──────────────────┘
-Memory .md ─────┘         │
-                          ▼
-                   ┌──────────────┐
-                   │ Classifier   │
-                   │ (6 categories│
-                   │  + tier)     │
-                   └──────────────┘
-```
-
-> 入库流程：多源文件 → 分块 → 过滤噪音 → 分类 → 嵌入向量 → 存入 LanceDB
-
-### Search Pipeline
-
-```
-Query                                               Results
-─────                                               ───────
-
-"Docker debugging"
-    │
-    ▼
-┌──────────────┐    ┌──────────────┐    ┌───────────────┐
-│ Query        │───►│ Hybrid       │───►│ Post-process  │──► Top-K
-│ Expander     │    │ Retrieval    │    │               │   Results
-│ (synonyms)   │    │              │    │ - Decay       │
-└──────────────┘    │ Vector: 0.7  │    │ - Access boost│
-                    │ BM25:   0.3  │    │ - Score floor │
-                    │ RRF merge    │    │ - Dedup       │
-                    └──────────────┘    └───────────────┘
-```
-
-## Key Design Decisions
-
-| Decision | Rationale |
-|----------|-----------|
-| **LanceDB** (not SQLite/Postgres) | Native vector search, columnar storage, zero-config, single-file DB |
-| **Jina v5** (not OpenAI embeddings) | Task-aware embeddings (query vs passage), better multilingual, 1024-dim sweet spot |
-| **Hybrid retrieval** (vector + BM25) | Vector alone misses keyword matches; BM25 alone misses semantic similarity |
-| **RRF fusion** | Reciprocal Rank Fusion is parameter-free and robust across score distributions |
-| **Weibull decay** (not exponential) | Better models human forgetting: slow start, accelerating fade |
-| **6 categories** (not free-form tags) | Structured enough for filtering and lifecycle rules, simple enough to auto-classify |
-| **HTTP API + MCP** (not just MCP) | MCP is great for CLI tools, but HTTP API works with any language/framework |
-| **Bun runtime** | Fast startup, native TypeScript, good for CLI tools and local servers |
-
-## File Map
-
-```
-src/
-├── api-server.ts          # HTTP API server (port 4318)
-├── mcp-server.ts          # MCP server (stdio transport)
-├── cli.ts                 # CLI entry point (lm command)
-├── store.ts               # LanceDB storage layer
-├── retriever.ts           # Hybrid retrieval (vector + BM25 + RRF)
-├── embedder.ts            # Jina embedding client
-├── ingest.ts              # Multi-source ingestion pipeline
-├── chunker.ts             # Text chunking with noise filtering
-├── decay-engine.ts        # Weibull time decay
-├── access-tracker.ts      # "Use it or lose it" tracking
-├── noise-filter.ts        # Low-quality content filter
-├── query-expander.ts      # Query expansion (synonyms)
-├── retrieval-profiles.ts  # 4 search profiles (precision/balanced/exploratory/recent)
-├── memory-output.ts       # Output formatting
-├── memory-assets.ts       # Brief/pin/export asset management
-├── asset-sync.ts          # Asset indexing
-├── runtime-config.ts      # Config loading
-├── llm-client.ts          # LLM client for smart extraction
-├── doctor.ts              # Health check diagnostics
-├── adaptive-retrieval.ts  # Adaptive retrieval strategies
-├── stderr-log.ts          # Logging
-├── tracker.ts             # Ingestion tracker (incremental)
-└── __tests__/             # Test suite
-```

package/docs/memory-boundary-contract.md

@@ -1,226 +0,0 @@
-# Memory Boundary Contract
-
-> RecallNest 先解决“谁能写什么、谁说了算”，再继续堆功能。没有边界的记忆系统，只会把更多垃圾写进库里。
-
-## Why This Exists
-
-The same fact should not be authored by every memory path.
-
-RecallNest now distinguishes between:
-
-- `canonical`: stable facts maintained in curated docs outside the raw memory index
-- `durable`: reusable structured memory that RecallNest is allowed to recall across windows
-- `session`: current task state captured by `checkpoint_session`
-- `observation`: append-only workflow health records used for self-evolution, not for stable recall
-- `evidence`: raw transcripts and unstructured ingest that are useful as hints, but not authority
-
-The core rule is:
-
-**One fact may have multiple mirrors, but only one authority write path.**
-
----
-
-## The Five Layers
-
-| Layer | What it stores | Authority write path | Can be mirrored into LanceDB? | Conflict policy |
-|-------|----------------|----------------------|-------------------------------|-----------------|
-| `canonical` | Stable identity, user profile, long-lived project rules | Curated docs such as `USER.md`, `PROJECT.md`, hand-maintained notes | Yes, as read-only mirrors | Manual review / canonical wins |
-| `durable` | Reusable `profile`, `preferences`, `entities`, `cases`, `patterns`, `events` | Structured writes such as `store_memory`, `store_case`, `store_workflow_pattern` | Yes | Category-specific: merge or append |
-| `session` | Current objective, summary, decisions, open loops, next actions | `checkpoint_session` | No, not as durable memory | Latest checkpoint wins |
-| `observation` | Workflow success/failure/correction/missed records for self-evolution | `workflow_observe`, `POST /v1/workflow-observe`, `recallnest workflow-observe` | No | Append-only |
-| `evidence` | Raw transcripts, loose snippets, import artifacts, candidate facts | `ingest.ts` and other unstructured importers | Already stored in the main index, but treated as low-trust evidence | Never overrides higher layers |
-
----
-
-## Authority Rules
-
-### 1. Structured memory creates durable authority
-
-These write paths are allowed to author durable memory:
-
-- `store_memory`
-- `store_case`
-- `store_workflow_pattern`
-- `POST /v1/store`
-- `POST /v1/case`
-- `POST /v1/pattern`
-- `POST /v1/capture`
-
-They write metadata with:
-
-- `boundary.layer = "durable"`
-- `boundary.authority = "structured-memory"`
-
-### 2. Session state stays in the checkpoint store
-
-`checkpoint_session` is the authority for current task state. It must not be treated as durable memory.
-
-That means:
-
-- session summaries do not go back into the durable memory index automatically
-- `resume_context` may read the latest checkpoint
-- durable memory and checkpoints are combined at read time, not merged into one store
-
-### 3. Workflow observation stays in its own store
-
-Workflow observations are operational records, not regular memory.
-
-That means:
-
-- `workflow_observe` writes append-only records to a dedicated observation store
-- observations are not one of the six memory categories
-- observations do not get promoted into stable recall by default
-- `workflow_health` and `workflow_evidence` read the observation store directly instead of searching LanceDB memory
-
-This boundary matters because self-evolution signals should not pollute `resume_context`.
-
-### 4. Transcript ingest is evidence, not authority
-
-Raw conversations from:
-
-- Claude Code
-- Codex
-- Gemini CLI
-
-are useful, but they are not allowed to mint authoritative stable facts.
-
-Current enforcement:
-
-- transcript-derived `profile` and `preferences` are downgraded to stored `events`
-- transcript-derived records are marked with `boundary.layer = "evidence"`
-- `resume_context` does not use transcript/evidence records as stable context
-- evidence only becomes durable through explicit structured write or `promote_memory`
-
-### 5. Durable beats evidence at read time
-
-When RecallNest composes startup context:
-
-- `canonical` beats `durable`
-- `durable` beats `evidence`
-- `session` is composed alongside them, not flattened into durable memory
-- `observation` is excluded from startup context unless a caller explicitly asks for workflow health/evidence
-
-If durable `cases` or `patterns` exist, they should be preferred over raw transcript fragments.
-
----
-
-## Category Ownership
-
-| Category | Preferred authority | Evidence mirror allowed? | Notes |
-|----------|---------------------|--------------------------|-------|
-| `profile` | curated docs or structured `store_memory` | Yes | Raw transcript profile statements are not authoritative |
-| `preferences` | curated docs or structured `store_memory` | Yes | Raw preference chatter is especially easy to overfit |
-| `entities` | structured memory or curated docs | Yes | Mentions in transcripts are hints, not stable truth |
-| `events` | structured memory or ingest | Yes | Safe append-only history bucket |
-| `cases` | `store_case` | Yes | Transcript debug stories can exist as evidence, but durable cases should be explicit |
-| `patterns` | `store_workflow_pattern` or structured `store_memory` | Yes | Reusable workflows should be promoted deliberately |
-
-Workflow observations are intentionally outside this table. They are not a seventh memory category.
-
----
-
-## Conflict Policy
-
-| Category type | Policy |
-|---------------|--------|
-| `profile`, `preferences`, `entities`, `patterns` | `latest-wins` within the same authority layer |
-| `events`, `cases` | `append-only` |
-| cross-layer conflicts | Higher-authority layer wins; lower layer stays as evidence |
-
-Cross-layer conflicts must not silently overwrite higher-authority data.
-
-Current implementation detail:
-
-- if an evidence promotion targets a `latest-wins` durable category and the same `canonicalKey` already exists with different text, RecallNest creates an open conflict candidate instead of silently overwriting the durable record
-- narrow exception: if the occupied durable memory and the incoming promotion resolve to the same preference slot (currently brand-item preferences such as `preferences:brand-item:<brand>:<item>`, reply-style preferences such as `preferences:reply-style:concise:direct`, or tool-choice preferences such as `preferences:tool-choice:bun:over:node`), RecallNest collapses the promotion onto the existing durable owner instead of opening a conflict
-- if a durable write reuses an existing `canonicalKey` under a different durable category, RecallNest creates an open conflict candidate instead of silently creating a second durable owner
-- conflicts can then be resolved explicitly by keeping the existing durable memory or accepting the incoming promoted text
-- if the exact same conflict fingerprint appears again after a previous review, RecallNest reopens the existing conflict record instead of creating a duplicate review item
-
----
-
-## Canonical Keys
-
-Durable writes can carry a `canonicalKey`.
-
-Use it when different observations should map to the same durable fact or workflow, for example:
-
-- `user-reply-style`
-- `project-recallnest-positioning`
-- `workflow-cross-window-handoff`
-
-Current behavior:
-
-- exact same `canonicalKey` + same durable text => dedupe
-- same `canonicalKey` on merge-type categories => update latest durable record
-- same `canonicalKey` on append-type categories => keep append behavior unless the text is identical
-- evidence promotion into an occupied merge-type `canonicalKey` => open conflict candidate for manual review, unless the occupied durable owner and incoming promotion are the same atomic preference slot
-- reusing a `canonicalKey` across different durable categories => open conflict candidate for manual review
-
-This gives RecallNest an explicit way to say “these two writes refer to the same durable memory”.
-
----
-
-## Promotion Path
-
-Raw evidence should not silently become durable memory.
-
-Instead, RecallNest now supports an explicit promotion path:
-
-- MCP: `promote_memory`
-- HTTP: `POST /v1/promote`
-
-Promotion rules:
-
-- source memory must be evidence or transcript-derived
-- target durable memory gets a new durable boundary
-- promoted record keeps first-hop provenance via `promotedFrom.memoryId`
-- same-slot auto-collapse and repeated exact promotions append observation trail under `provenanceHistory[*]` without silently overwriting durable wording
-- optional `text` lets the caller replace raw transcript wording with a clean durable statement
-- if the promoted text disagrees with an existing durable record for the same `canonicalKey`, RecallNest stores an open conflict candidate instead of silently overwriting the durable record
-- if the promotion would reuse a `canonicalKey` that is already owned by another durable category, RecallNest stores an open conflict candidate instead of creating a second durable owner
-
-Conflict handling entry points:
-
-- MCP: `list_conflicts`, `resolve_conflict`
-- HTTP: `GET /v1/conflicts`, `POST /v1/conflicts/resolve`
-- CLI: `recallnest conflicts list/show/resolve`
-
-Conflict review helpers:
-
-- list and inspect views expose derived advice (`keep_existing`, `accept_incoming`, `manual_review`) with conservative confidence levels
-- when a same-category promotion conflict is close enough to merge, advice may also include a conservative `mergeSuggestion` for human review
-- conflict resolution now supports `merge`, which rewrites the existing durable owner with either an explicit `mergedText` or the derived `mergeSuggestion`
-- list views can also be grouped by cluster so repeated conflicts on the same `canonicalKey` are reviewed as one review stream instead of many isolated rows
-- audit views summarize `stale` / `escalated` clusters first so terminal operators can review the highest-risk conflicts without manually scanning every row
-- escalation views can explicitly stamp `stale` / `escalated` conflicts with review metadata so operators know which aging conflicts were already triaged
-- CLI audit now supports `--export --format md|json`, so operators can persist a conflict review snapshot for handoff or later triage
-
----
-
-## Enforcement Points In Code
-
-Current implementation:
-
-- [ingest.ts](../src/ingest.ts): transcript ingest writes evidence metadata and downgrades risky stable categories
-- [capture-engine.ts](../src/capture-engine.ts): structured writes stamp durable boundary metadata and create promotion conflict candidates when needed
-- [conflict-engine.ts](../src/conflict-engine.ts): builds and resolves explicit conflict candidates
-- [conflict-store.ts](../src/conflict-store.ts): persists conflict candidates outside the main LanceDB index
-- [context-composer.ts](../src/context-composer.ts): startup continuity ignores evidence-only stable recall
-- [memory-boundaries.ts](../src/memory-boundaries.ts): shared boundary resolution and read guards
-- [workflow-observation-store.ts](../src/workflow-observation-store.ts): persists dedicated append-only workflow observations outside regular memory
-- [workflow-observation-engine.ts](../src/workflow-observation-engine.ts): computes workflow health reports and evidence packs without touching durable recall
-
----
-
-## Practical Consequences
-
-This contract is intentionally conservative:
-
-- transcript recall can still help search and debugging
-- structured memory is how facts graduate into durable continuity
-- checkpoints carry active work state
-- workflow observations can guide repair and evaluation, but they do not become stable context automatically
-- startup continuity should prefer fewer high-trust facts over more noisy fragments
-
-That tradeoff is deliberate. A smaller but cleaner memory layer is more useful than a larger index with no ownership rules.

package/docs/memory-categories.md

@@ -1,119 +0,0 @@
-# Memory Categories
-
-> 6 类分类体系：RecallNest 自动将每条记忆归入 6 个类别之一，便于过滤和差异化管理。
-
-## The Six Categories
-
-RecallNest classifies every memory into one of six categories. Classification happens automatically during ingestion using keyword heuristics and (optionally) LLM-based analysis.
-
-Category is not the same thing as authority.
-
-- Structured writes can author durable memory.
-- Raw transcript ingest is treated as evidence-first input.
-- Stable transcript claims should not override curated or structured memory.
-- Workflow observations are not a seventh memory category; they live in a separate observation store and never become durable recall by default.
-
-See [memory-boundary-contract.md](./memory-boundary-contract.md) for the layer and authority rules.
-
-| Category | What It Stores | Examples | Consolidation |
-|----------|---------------|----------|---------------|
-| **profile** | Who the user is — role, background, identity | "Data scientist with 5 years experience", "Works at Acme Corp" | Merge (latest wins) |
-| **preferences** | How the user likes things done | "Prefers TDD", "Uses Bun over Node", "Commit after every change" | Merge (slot-aware; item facts stay atomic) |
-| **entities** | Named things — projects, tools, people, repos | "Project X uses LanceDB", "Bob handles DevOps" | Merge (accumulate) |
-| **events** | Things that happened — incidents, decisions, milestones | "Migrated to AWS on 2026-01-15", "Bot crashed due to OOM" | Append (dedup only) |
-| **cases** | Debugging stories, troubleshooting workflows | "Fixed Docker crash by increasing memory limit", "API timeout was caused by DNS" | Append (dedup only) |
-| **patterns** | Recurring techniques, best practices, anti-patterns | "Always grep callers after API change", "Use mv to Trash, never rm" | Merge (refine) |
-
----
-
-## Why Categories Matter
-
-### 1. Smarter Search
-
-Filter by category to reduce noise:
-
-```bash
-# MCP: search only debugging cases
-search_memory("Docker crash", category="cases")
-
-# HTTP API: same thing
-curl -X POST localhost:4318/v1/search \
-  -d '{"query": "Docker crash", "category": "cases"}'
-```
-
-> 按类别过滤能显著减少噪音：搜 debug 问题时只看 cases，搜用户偏好时只看 preferences。
-
-### 2. Different Lifecycle Rules
-
-Not all memories age the same way:
-
-- **profile** and **preferences**: Slow decay — identity and habits change slowly
-- **events**: Medium decay — historical facts stay relevant for months
-- **cases**: Medium decay — debugging patterns remain useful
-- **patterns**: Slow decay — best practices persist
-- **entities**: Fast update — project details change frequently
-
-### 3. Consolidation Strategy
-
-Merge-type categories (profile, preferences, entities, patterns) benefit from periodic consolidation — combining 5 similar entries into 1 refined one.
-
-For `preferences`, merge should be slot-aware, not topic-wide. Rewordings of the same preference can collapse, but concrete item/object preferences under the same brand or theme should stay as separate facts, and higher-value slots such as reply style or tool choice should keep their own stable owners.
-
-Append-type categories (events, cases) keep distinct entries but remove exact duplicates.
-
----
-
-## Classification Rules
-
-### Automatic (Keyword Heuristics)
-
-During ingestion, RecallNest scans for signal words:
-
-| Signal | Category |
-|--------|----------|
-| "I am", "my role", "background", "experience" | `profile` |
-| "prefer", "always", "never", "like to", "style" | `preferences` |
-| Names, URLs, project names, tool names | `entities` |
-| Dates, "happened", "decided", "migrated", "shipped" | `events` |
-| "fixed", "debugged", "root cause", "workaround", "error" | `cases` |
-| "pattern", "best practice", "rule", "always do", "anti-pattern" | `patterns` |
-
-Transcript-derived `profile` / `preferences` may still be detected during extraction, but RecallNest can downgrade them to evidence-only storage rather than treating them as durable truth.
-
-### Manual Override
-
-When storing via API, you can specify the category explicitly:
-
-```json
-POST /v1/store
-{
-  "text": "Always run tests before pushing",
-  "category": "patterns"
-}
-```
-
-If omitted, defaults to `"events"`.
-
----
-
-## Category Distribution
-
-A healthy memory index typically looks like:
-
-```
-events     ████████████████████████  45%
-cases      ████████████             20%
-entities   ████████                 15%
-patterns   ██████                   10%
-preferences ███                      5%
-profile    ██                        5%
-```
-
-Check yours with:
-
-```bash
-curl http://localhost:4318/v1/stats
-# or via MCP: memory_stats
-```
-
-> 如果某个类别占比异常（比如 events 占了 90%），说明分类规则可能需要调优。