recallnest 3.0.0 → 4.0.0

package/README.md

@@ -10,13 +10,13 @@ A local-first memory system backed by LanceDB that turns scattered conversation
 
 [![GitHub](https://img.shields.io/github/stars/AliceLJY/recallnest?style=social)](https://github.com/AliceLJY/recallnest)
 [![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)
-[![Runtime](https://img.shields.io/badge/Runtime-Bun_|_Node.js_22+-f9f1e1?logo=bun)](https://bun.sh)
+[![Runtime](https://img.shields.io/badge/Runtime-Bun-f9f1e1?logo=bun)](https://bun.sh)
 [![LanceDB](https://img.shields.io/badge/LanceDB-Vector+FTS-orange)](https://lancedb.com)
 [![MCP](https://img.shields.io/badge/MCP-44_tools-blue)](https://modelcontextprotocol.io)
 [![CI](https://github.com/AliceLJY/recallnest/actions/workflows/ci.yml/badge.svg)](https://github.com/AliceLJY/recallnest/actions/workflows/ci.yml)
 [![CC Plugin](https://img.shields.io/badge/Claude_Code-Plugin-blueviolet)](https://github.com/AliceLJY/recallnest)
 
-**English** | [简体中文](README_CN.md) | [Roadmap](ROADMAP.md)
+**English** | [简体中文](README_CN.md) | [Roadmap](ROADMAP.md) | [Website](https://aliceljy.github.io/recallnest/)
 
 </div>
 
@@ -26,8 +26,139 @@ A local-first memory system backed by LanceDB that turns scattered conversation
 
 Coding agents forget everything between windows. Your context — project configs, debugging decisions, entity mappings — is scattered across Claude Code, Codex, Kimi, Antigravity — and every other terminal you open — with no shared memory.
 
-RecallNest solves this: **a single LanceDB-backed memory layer that your coding agents read and write**. Context stored in one window is auto-recalled in another. Sessions checkpoint on exit and resume on start. Memory decays, evolves, and self-organizes — not just raw log storage.
+RecallNest is **one LanceDB-backed memory layer that all of them read and write**. Context stored in one window is recalled in another. Sessions checkpoint on exit and resume on start. Memory decays, evolves, and self-organizes — it is not a log you grep.
 
+### What a recall actually looks like
+
+```text
+Query   : deploy rollback
+Hits    : 5
+
+#  ID       Score Category  Tier        Source  Date        Age  Retrieval Path
+1  ee79037a 46.1% cases     peripheral  cc      2026-08-25  2d   vector
+   [assistant] Rolled back to the previous image and pinned the digest so the next…
+   prov : evidence/transcript-ingest
+   imgs : 52 agent-made in this session · read sess=dca70d4a
+```
+
+Three things in that block carry most of the design:
+
+- **`Source cc` · `Age 2d`** — this came out of a Claude Code window two days ago and you
+  are reading it from a different terminal, possibly on a different machine. That is the
+  premise the whole project is built on.
+- **`prov : evidence/…`** — every row states which layer it sits on. A fragment scraped out
+  of a transcript never gets to pose as a decision you actually made; moving to durable
+  memory is a separate, gated step with its own evidence requirement.
+- **`imgs : …`** — that session contained 52 images. **Not one of them is in the database.**
+  The line exists so you know there is something to go look at, and producing it cost no
+  model call, no vector, and no storage.
+
+> The row above comes from transcript ingest. Since 2026-10-06 that step is off by default:
+> `ingest` no longer chunks conversation records into the store, so a default setup holds only
+> the memories you stored, distilled entries and memory files, and raw sessions are left to a
+> dedicated session-search tool. To get rows like this one back, see "Index existing
+> conversations" below.
+
+That last one is the approach in miniature: **store what makes a thing findable, not
+everything that could ever be asked about it.** The full reasoning — including the two
+places where the obvious implementation was wrong — is in
+[Images: addressable, not embedded](#images-addressable-not-embedded).
+
+
+## Core Capabilities
+
+### Access & Setup
+
+| Capability | Description |
+|---|---|
+| **CC Plugin** | Install in Claude Code with one command — no manual config |
+| **Shared Index** | One LanceDB store shared by every terminal that speaks MCP |
+| **Dual Interface** | MCP (stdio) for CLI tools + HTTP API for custom agents |
+| **One-Click Setup** | Integration scripts install MCP access and continuity rules |
+
+### Recall & Continuity
+
+| Capability | Description |
+|---|---|
+| **Hybrid Retrieval** | 6-channel: vector + BM25 + L0/L1/L2 multi-vector + KG graph (PPR). The default mode is vector-only (since 2026-06); BM25 fusion and graph expansion are opt-in |
+| **Write-time Triggers** | Each memory can carry 2–6 "how will this be asked about later" phrasings (`triggers`), embedded separately and used only for recall — never rendered as evidence. Borrowed from Tencent T-Mem (EMNLP 2026). |
+| **4 Retrieval Profiles** | default, writing, debug, fact-check — tuned for different tasks |
+| **Session Continuity** | `checkpoint_session` + `resume_context` (full/light/summary modes) with repo-state guard |
+| **Session Distiller** | 3-layer conversation compression: microcompact → LLM summary → knowledge extraction |
+| **Conversation Import** | Import from Claude Code, Claude.ai, ChatGPT, Slack, and plaintext |
+| **Topic Tags** | Intra-scope topic partitioning — auto-detected, filterable in search |
+| **Related Scope Sidecar** | Opt-in `includeRelatedScopes` search over configured `scopeRelations`, shown separately from the main scoped ranking |
+
+### Memory Lifecycle & Governance
+
+| Capability | Description |
+|---|---|
+| **Memory Evolution** | Supersede chains, decay scoring, LLM importance, consolidation, archival |
+| **Smart Promotion** | Evidence → durable memory with conflict guards, merge resolution, and audit trail |
+| **Privacy Tiers** | 4-tier (`ephemeral` / `private` / `durable` / `shared`) with cascade forgetting |
+| **Admission Control** | Write-time gating: noise filter, importance floor, dedup, rate limiting |
+| **Memory Lint** | Contradiction, duplicate, stale, and orphan detection with health score |
+| **Offline Consolidation** | `dream` command: clustering, merging, pruning of accumulated memories |
+
+### Reasoning & Structure
+
+| Capability | Description |
+|---|---|
+| **Knowledge Graph** | Entity relation graph with PPR algorithm for multi-hop questions |
+| **Constructive Retrieval** | Multi-source candidate expansion + grounded context reconstruction |
+| **Narrative Architecture** | 3-layer autobiographical metadata (life-period → general-event → specific-event) |
+| **Skill Memory** | Store, retrieve, and promote executable skills from recurring patterns |
+| **Predictive Reminders** | Behavioral-signal prediction engine surfaces "you might need this" suggestions |
+| **6 Categories** | profile, preferences, entities, events, cases, patterns — with category-aware merge strategies |
+
+### Visibility & Operations
+
+| Capability | Description |
+|---|---|
+| **Dashboard** | Web UI with stats, category distribution, growth trends, and health |
+| **Workflow Observation** | Dedicated append-only workflow health records, outside regular memory |
+| **Structured Assets** | Pins, briefs, and distilled summaries — not just raw logs |
+| **Data Checkup** | Data quality health checks on the memory store (including source health) |
+| **Source Heartbeats** | Automatic ingest health tracking per data source with staleness alerts |
+| **Export Graph** | Export interactive HTML knowledge graph visualization |
+| **Batch Operations** | Store up to 20 memories in a single call with dedup |
+| **Connector Framework** | Standard connector-v1 format for external data sources with example adapters |
+
+---
+
+## Architecture
+
+```
+  CLIENTS                    ACCESS                      CORE ENGINE                    STORAGE
+  ──────────────────────     ───────────────────────     ────────────────────────────   ──────────────────────
+
+  Claude Code                MCP over stdio              Retriever                      LanceDB
+  Codex                ───▶  44 tools, 3 tiers    ───▶   vector · hybrid opt-in  ───▶   vector + columnar
+  Kimi · Antigravity                                     Classifier · 6 categories
+  Doubao desktop                                         Context composer
+                             HTTP API :4318              resume_context                 Jina embeddings v5
+  your scripts · cron  ───▶  21 endpoints          ───▶  Decay · Weibull half-life ─▶   1024-dim, task-aware
+                                                         Conflict · audit + merge
+  phone app            ───▶  read-only gateway     ───▶  Capture: evidence → durable
+                             :8791, token-gated
+```
+
+> Retriever: `retrieval.mode` defaults to `"vector"` (vector-only search). Setting it to `"hybrid"` in
+> `config.json` adds BM25 full-text search, merged with the vector results by weighted score fusion
+> (`vectorWeight` 0.7 / `bm25Weight` 0.3 by default).
+
+### Internal Design
+
+- **L0 / L1 / L2 Dynamic Folding** — every memory stores 3 granularity layers (one-liner / bullet summary / full content); retrieval dynamically selects which layer to return based on relevance score and token budget
+- **Weibull Decay + Emotion Modulation** — memories decay along a parametric Weibull curve; importance scores modulate the half-life, and emotional salience extends it further (up to 30%)
+- **Vector Pre-filter + LLM Dedup** — 90% of dedup decisions use cheap cosine similarity (>= 0.92); only borderline cases invoke LLM judgment, keeping costs low without sacrificing accuracy
+- **Category-Aware Merge Strategies** — `profile` and `preferences` use merge-on-conflict (latest wins); `events` and `cases` use append-only (history preserved)
+- **Write-time Rehearsal (Triggers)** — a memory stored with `triggers: ["别人的榜我们要不要也去排个名", ...]` gets each phrasing embedded on the query side into a side table (`memory_triggers`); at recall time the query is matched against triggers, the best-matching trigger pulls its host memory into the candidate pool (max-over-triggers attribution, cosine hard gate + lexical soft gate), and the host then goes through the normal scoring chain. Triggers never appear in results — they are an entry point, not evidence. `RECALLNEST_TRIGGER_RECALL=false` turns it off; `triggers-backfill` / `triggers-calibrate` CLI manage existing data
+- **Display Score vs Elimination Score** — dual-track retrieval: tier floor prevents core memories from ever dropping out, while decay boost lets fresh memories surface temporarily without permanently displacing stable ones
+
+> Full architecture deep-dive: [`docs/architecture.md`](docs/architecture.md)
+
+---
 
 ## Who Can Connect
 
@@ -104,7 +235,7 @@ npm install -g recallnest      # install globally
 recallnest doctor
 ```
 
-Works with Node.js 22+ (via tsx) or Bun. No git clone needed.
+Requires [Bun](https://bun.sh): RecallNest does not run on Node.js. The npm launcher only hands off to Bun, and exits with install instructions if `bun` is not on your PATH. No git clone needed.
 
 ### Option C: Manual setup
 
@@ -159,279 +290,94 @@ bun run seed:continuity
 bun run src/cli.ts doctor
 ```
 
----
-
-## Web UI
-
-<p align="center">
-  <img src="assets/dashboard.png" alt="RecallNest Dashboard" width="800" />
-  <br><em>Dashboard — total count, category distribution, health score, and growth trends at a glance.</em>
-</p>
-
-<p align="center">
-  <img src="assets/screenshots/ui-full.png" alt="RecallNest Search Workbench" width="800" />
-  <br><em>Search Workbench — hybrid search with topic tag filtering, 4 retrieval profiles, Skills browser, and asset management.</em>
-</p>
-
-<p align="center">
-  <img src="assets/knowledge-graph.png" alt="RecallNest Knowledge Graph" width="800" />
-  <br><em>Knowledge Graph — interactive force-directed visualization with semantic bridges revealing cross-domain connections.</em>
-</p>
-
-```bash
-bun run src/ui-server.ts
-# → http://localhost:4317
-```
-
----
-
-## Core Capabilities
-
-### Access & Setup
-
-| Capability | Description |
-|---|---|
-| **CC Plugin** | Install in Claude Code with one command — no manual config |
-| **Shared Index** | One LanceDB store shared by every terminal that speaks MCP |
-| **Dual Interface** | MCP (stdio) for CLI tools + HTTP API for custom agents |
-| **One-Click Setup** | Integration scripts install MCP access and continuity rules |
-
-### Recall & Continuity
-
-| Capability | Description |
-|---|---|
-| **Hybrid Retrieval** | 6-channel: vector + BM25 + L0/L1/L2 multi-vector + KG graph (PPR) |
-| **4 Retrieval Profiles** | default, writing, debug, fact-check — tuned for different tasks |
-| **Session Continuity** | `checkpoint_session` + `resume_context` (full/light/summary modes) with repo-state guard |
-| **Session Distiller** | 3-layer conversation compression: microcompact → LLM summary → knowledge extraction |
-| **Conversation Import** | Import from Claude Code, Claude.ai, ChatGPT, Slack, and plaintext |
-| **Topic Tags** | Intra-scope topic partitioning — auto-detected, filterable in search |
-| **Related Scope Sidecar** | Opt-in `includeRelatedScopes` search over configured `scopeRelations`, shown separately from the main scoped ranking |
-
-### Memory Lifecycle & Governance
-
-| Capability | Description |
-|---|---|
-| **Memory Evolution** | Supersede chains, decay scoring, LLM importance, consolidation, archival |
-| **Smart Promotion** | Evidence → durable memory with conflict guards, merge resolution, and audit trail |
-| **Privacy Tiers** | 4-tier (`ephemeral` / `private` / `durable` / `shared`) with cascade forgetting |
-| **Admission Control** | Write-time gating: noise filter, importance floor, dedup, rate limiting |
-| **Memory Lint** | Contradiction, duplicate, stale, and orphan detection with health score |
-| **Offline Consolidation** | `dream` command: clustering, merging, pruning of accumulated memories |
-
-### Reasoning & Structure
-
-| Capability | Description |
-|---|---|
-| **Knowledge Graph** | Entity relation graph with PPR algorithm for multi-hop questions |
-| **Constructive Retrieval** | Multi-source candidate expansion + grounded context reconstruction |
-| **Narrative Architecture** | 3-layer autobiographical metadata (life-period → general-event → specific-event) |
-| **Skill Memory** | Store, retrieve, and promote executable skills from recurring patterns |
-| **Predictive Reminders** | Behavioral-signal prediction engine surfaces "you might need this" suggestions |
-| **6 Categories** | profile, preferences, entities, events, cases, patterns — with category-aware merge strategies |
-
-### Visibility & Operations
-
-| Capability | Description |
-|---|---|
-| **Dashboard** | Web UI with stats, category distribution, growth trends, and health |
-| **Workflow Observation** | Dedicated append-only workflow health records, outside regular memory |
-| **Structured Assets** | Pins, briefs, and distilled summaries — not just raw logs |
-| **Data Checkup** | Data quality health checks on the memory store (including source health) |
-| **Source Heartbeats** | Automatic ingest health tracking per data source with staleness alerts |
-| **Export Graph** | Export interactive HTML knowledge graph visualization |
-| **Batch Operations** | Store up to 20 memories in a single call with dedup |
-| **Connector Framework** | Standard connector-v1 format for external data sources with example adapters |
-
----
-
-## New in v3.0: A Supported Runtime, and Conclusions That Can Be Used
-
-v3.0 is a major release for one reason that shows up on install and one that shows up in
-how memory behaves.
-
-**The runtime boundary moved to Node 22.** RecallNest had been carrying `openai@4`, which
-pulls in the deprecated `formdata-node` → `node-domexception` chain. Every `openai` release
-since v5 has zero dependencies, so the chain disappears on any upgrade — but v7 declares
-`engines.node >= 22.0.0`, which makes "raise the Node floor" and "move off a deprecated
-dependency chain" the same piece of work rather than two. `engines.node` is now `>=22`.
-This is the breaking part of the major.
-
-**A synthesized conclusion can now reach stable memory.** `dream` writes cluster insights
-and cross-memory patterns on the evidence layer deliberately — a model re-reading its own
-memories is a lead to its sources, not authority over them — but stable-memory selection
-refuses the evidence layer outright. The consequence was absolute: no matter how well
-supported a synthesized conclusion was, nothing downstream could lean on it. `promote_synthesis`
-(MCP) and `recallnest promote-synthesis` (CLI) give it a road, gated on the conclusion's own
-validated evidence set rather than on repetition, because a synthesis is already a
-cross-entry aggregate. The synthesized row is never modified; promotion writes a separate
-durable entry carrying `promotedFrom` back to it, through the same path and the same
-`canonicalKey` dedup as every other promotion.
-
-Also in this release:
-
-- **Retrieve audit rows say what was served.** They used to record the query and a hit
-  count. Because beliefs are revised in place — the id stays, the version increments, the
-  old text survives as a `superseded` row — "memory X was retrieved" was ambiguous across
-  every belief change. Rows now carry each result's id, revision, lifecycle status, and
-  boundary. The list is capped, and a capped list says so.
-- **HTTP contract tests for embeddings and chat completions.** Every previous test stubbed
-  the SDK client, so nothing exercised a socket, and a transport-level regression would
-  have passed all of them. These drive the real classes through the real SDK against a
-  loopback server, covering success, error, and timeout for the default OpenAI shape plus
-  the Jina and Qwen-compatible profiles. No network egress, no vendor credentials.
-- **Fixed: a rate-limit reply could trigger an unbounded request storm.** Found by those
-  tests. The embedder retried a context-length error by chunking, the chunker returns short
-  text unchanged, and the gate that decided "this is a context error" also matched
-  rate-limit wording — so a 429 re-embedded the same text forever. Measured at over 61,000
-  requests in five seconds against an endpoint asking us to slow down. Failure is bounded now.
-- **A known recall gap is now a runnable regression.** A memory that is long, older, and
-  rarely read can fail to be recalled by words copied out of its own body, while short,
-  fresh, frequently-read entries outrank it. It is a ranking problem, not a filtering one —
-  the entry clears every threshold and then places last. The reproduction is checked in; the
-  ranking fix is not in this release.
-- **44 MCP tools** across three tiers, up from 43.
-
-### Upgrading from v2.6
-
-- **Node 22 or newer is required.** Bun users are unaffected. This is the only breaking
-  change in the release.
-- Existing LanceDB data is opened in place. No export or import step.
-- `promote_synthesis` defaults to dry-run and writes nothing until `dryRun=false`.
-- `audit.jsonl` rows for retrievals are larger now that they list what was served. Rotation
-  is still manual; archive it if it approaches 50 MB.
-
-### A note on npm history
-
-`recallnest@2.6.1` on npm was a maintenance release published before Trusted Publishing was
-configured for this package, so it carries no build provenance. That is a fact about how it
-was published, not a defect in the package, and it is not retroactively fixable — the same
-npm version cannot be republished to add provenance.
-
----
-
-## New in v2.6: Reliable Cross-Process Memory and Distribution
-
-v2.6 turns the development since v2.5.4 into a release-ready upgrade:
-
-- **Cross-process visibility** — LanceDB now checks for external commits on every read by default, so resident MCP/API/UI processes see writes from CLI ingestion without a restart. Set `RECALLNEST_READ_CONSISTENCY_INTERVAL=<seconds>` for bounded staleness or `off` for the legacy unchecked behavior.
-- **Safer memory evolution** — belief changes preserve the old row as `superseded`; procedural memories avoid time decay; cold-start and length-normalization behavior no longer suppress short entity queries or stable memories.
-- **More reliable consolidation** — `dream` now distinguishes failure classes, honors wall-clock budgets, refills vectors before semantic clustering, and asserts that successful runs actually produced work.
-- **Broader conversation coverage** — Kimi, AGY/Antigravity, and minis sources are recognized across ingestion, scope boundaries, and term resolution.
-- **One public version** — npm metadata, CLI `--version`, MCP handshakes, HTTP health responses, and the Claude Code marketplace all follow the `2.6.0` release contract.
-- **Installable Claude Code plugin** — installation now registers the 43-tool MCP server and continuity skill, asks for the Jina key through sensitive plugin configuration, and keeps config plus LanceDB data in Claude Code's persistent plugin data directory.
-
-### Upgrading from v2.5.4
-
-- Existing LanceDB data is opened in place. Legacy tables missing newer fields are migrated automatically, including empty tables.
-- Plugin data survives plugin updates because it is stored outside the versioned cache. A manual clone continues to use its existing `config.json` and database path.
-- Retrieval auditing now includes read operations. `audit.jsonl` has no automatic rotation yet; consider archiving it if it approaches 50 MB.
-- `RECALLNEST_LAYER_ADMISSION` remains opt-in (`observe` or `on`); the default is `off`.
+Since 2026-10-06, `ingest` does two things by default: it reconciles memory files
+(`sources.memory`) and moves files from the Minis drop folder into `data/minis-archive`. The six
+conversation sources (cc / codex / kimi / gemini / desktop / minis) are not chunked, embedded or
+written. To index transcripts as well, set `RECALLNEST_TRANSCRIPT_INGEST=on` in the environment
+that runs `ingest` (only the exact value `on` counts). Minis conversations archived while the
+switch was off are not picked up automatically; copy them from `data/minis-archive` back into
+the drop folder and run `ingest` again.
 
 ---
 
-## New in v2.1: Philosophy-Informed Memory
-
-v2.0 built the operational memory platform; v2.1 added philosophy-informed memory behavior.
+## Images: addressable, not embedded
 
-Five upgrades derived from 9 research dimensions in philosophy of memory, each mapped to concrete engineering:
+Conversations contain images. A text memory layer does not. The usual answer is a
+multimodal embedding model — encode every image into the same space as the text. That is
+right for photo libraries. It is the wrong shape here, for a cheap reason: **in a
+conversation an image almost never arrives alone.** It comes wrapped in "look at this
+error", and the reply right after it usually describes what was in the picture. The words
+around the image are already an index of it. What was missing was never semantic search
+over pixels — it was knowing a picture is sitting there at all.
 
-- **Emotion-Aware Decay** *(Affective Memory Theory)* — Memories with strong emotional content decay 20-30% slower. Keyword-based emotion detection computes `salience` (mnemonic significance), which feeds into the Weibull half-life formula and a rebalanced 4-factor evolution score. Zero LLM cost.
+So RecallNest does not encode images. It records **how many images are in the session a
+memory came from**, and lets you decide whether to open the original transcript. Meaning
+is resolved on demand, by whatever model is asking, at the moment it matters.
 
-- **Memory Ethics Layer** *(Right to Be Forgotten / GDPR Art. 17)* — Four privacy tiers (`ephemeral` / `private` / `durable` / `shared`). Cascade forgetting engine that propagates deletion through KG triples, evolution chains, pin assets, and briefs. Full audit trail. `forget_memory` MCP tool for agent-driven deletion.
+The cost is worth stating plainly: **no multimodal model, no re-embedding, no image
+storage, no change to any vector.** Backfilling 21,319 existing memories touched metadata
+only.
 
-- **Autobiographical Narrative** *(Narrative Identity Theory / Conway's 3-layer model)* — Memories are tagged with `lifePeriod → generalEvent → specificEvent` hierarchy, orthogonal to existing 6 categories. Retrieval pulls narrative siblings. Context rendering groups by life period. Rule-based tagger with EN+CN support.
-
-- **Constructive Retrieval** *(Simulation Theory / Michaelian)* — Instead of returning raw stored text, RecallNest now reconstructs context from an expanded candidate set: KG neighbors + evolution chains + cluster members + narrative siblings. Source-map grounded coverage replaces lexical overlap. Contradictions are detected and flagged.
-
-- **Predictive Prospective Memory** *(Mental Time Travel / Tulving)* — Heuristic prediction engine that surfaces "you might need this" reminders from behavioral signals: stale checkpoint open loops, corrected workflow observations, high-frequency dormant memories, and uncovered query topics. Zero LLM cost. Auto-expire in 7 days if unaccepted.
-
----
+Two design choices in it were not obvious, and both were wrong on the first attempt.
 
-## New in v2.2: Retrieval Quality Hardening
+### Session-level, on purpose
 
-v2.1 added philosophy-informed behavior; v2.2 closes the last three engine-layer gaps identified by a frontier research scan (ACC, PI-LLM, TSM).
+The marker counts the whole session, not the turn — coarser than it first looks like it
+should be, and the coarseness is the point.
 
-- **Memory Confidence Meta-tags** *(ACC / Dual-Process UQ)* — Each memory now carries structured `ConfidenceMetadata` (score, reliability tier: `direct` / `inferred` / `hearsay`). Auto-assigned from source on write (`manual` = 0.9, `agent` = 0.7, `conversation_import` = 0.5). Retrieval scores are weighted by confidence. `resume_context` tags low-confidence items with `[低置信]`.
+A turn that is nothing but a pasted screenshot has almost no text, so it never cleared the
+length gate and never entered the store. Measured on real transcripts, **12.5% of turns
+containing a pasted image were dropped whole** — including the ones worth the most, like
+seven screenshots with no caption, or "here are the steps" attached to a picture that *is*
+the steps. A turn-level marker has nothing to attach to for exactly those. A session-level
+marker lands on that session's *other* memories, which did get stored, and those are what
+a search surfaces.
 
-- **Interference Detection + Active Forgetting Gate** *(PI-LLM / SleepGate)* — Semantic cluster detection identifies groups of near-duplicate memories competing for retrieval. Enhanced RIF keeps only top-K (default 3) per cluster; extras are demoted 50% instead of removed. Write-time pre-warning: when a scope accumulates ≥5 high-similarity active memories, the weakest is flagged `pending_review`. `data_checkup` reports interference density.
+The trade-off is undisguised: every memory from a session carries the same count, so the
+images may have nothing to do with the row you are looking at. The line says
+`in this session`, not `in this memory`, for that reason.
 
-- **Temporal Validity Windows** *(TSM / TiMem / Zep)* — `store_memory` accepts `validUntil` (expiration) and `eventTime` (when the event actually happened). `search_memory` supports `validAt` (point-in-time query) and `includeExpired` (demote 80% instead of hide). Auto-GC applies 2× decay acceleration to expired memories.
+### Two classes, because they answer different questions
 
-- **Usage-Adjusted Auto-GC** *(off by default)* — `RECALLNEST_USAGE_DECAY=true` enables a GC-only cold-memory penalty when constructive retrieval is also active. Cold memories discount the frequency component instead of changing online retrieval ranking.
-
----
-
-## New in v2.3: Connector Ecosystem + Source Health
-
-v2.2 hardened retrieval quality; v2.3 opens RecallNest to external data sources with a standard connector framework and operational health monitoring.
+| Bucket | What it is | The question it answers |
+|---|---|---|
+| user-pasted | Pictures a human put into a message | *Where is that screenshot I sent?* |
+| agent-made | Everything else the session produced | *What did the page look like? What did I generate?* |
 
-- **Connector-v1 Standard** *(GB-2)* — A JSON format (`ConnectorOutputV1`) that any external script can produce. Obsidian vaults, emails, RSS feeds, log files — normalize once, ingest through the full dedup/embed/extract pipeline. See [`docs/connector-spec.md`](docs/connector-spec.md) for the specification and [`connectors/examples/`](connectors/examples/) for adapter skeletons (email, logs, RSS).
+Keeping only the first is tempting — a person searching their own memory wants their own
+screenshots. But an agent reconstructing its past work wants the other: the diagram it
+drew, the rendering it captured, the illustration it made for a post. **Of 1,767 sessions
+carrying images, 1,103 contain no human-pasted image at all.** Keep one bucket and those
+sessions go silent — precisely the ones where the agent did visual work.
 
-- **Obsidian Vault Ingestion** *(GB-1)* — First-party Obsidian connector: scans `.md` files, extracts frontmatter + wikilinks, maps folder structure to tags. One command: `lm ingest --obsidian /path/to/vault`.
+### The second bucket is a complement, not a list
 
-- **Source Health Monitoring** *(GB-3)* — Every connector ingest writes a heartbeat to `data/source-heartbeat.json`. `data_checkup` flags stale sources (>7d warning, >30d error). `doctor --ci` shows a per-source heartbeat summary with human-readable age.
+The first implementation defined agent-made images by enumeration: inside `tool_result`,
+inside `payload.output`, inside `tool.result`. Every location was real. The list was still
+wrong, because **the set of ways an image can appear only grows**, and an enumeration
+silently drops whatever it did not anticipate.
 
----
+So the second bucket is a complement: count every image signal in the record, subtract the
+ones positively identified as human-pasted, attribute the rest without asking where it came
+from. Across 9,619 transcripts:
 
-## Architecture
-
-```
-┌──────────────────────────────────────────────────────────┐
-│                     Client Layer                          │
-├──────────┬──────────┬──────────┬──────────────────────────┤
-│ Claude   │ Anti-    │ Codex    │ Custom Agents / curl     │
-│ Code     │ gravity  │          │                          │
-└────┬─────┴────┬─────┴────┬─────┴──────┬──────────────────┘
-     │          │          │            │
-     └──── MCP (stdio) ───┘     HTTP API (port 4318)
-                │                       │
-                ▼                       ▼
-┌──────────────────────────────────────────────────────────┐
-│                   Integration Layer                       │
-│  ┌─────────────────────┐  ┌────────────────────────────┐ │
-│  │  MCP Server         │  │  HTTP API Server           │ │
-│  │  44 tools           │  │  21 endpoints              │ │
-│  └─────────┬───────────┘  └──────────┬─────────────────┘ │
-└────────────┼─────────────────────────┼───────────────────┘
-             └──────────┬──────────────┘
-                        ▼
-┌──────────────────────────────────────────────────────────┐
-│                     Core Engine                           │
-│                                                           │
-│  ┌────────────┐  ┌────────────┐  ┌─────────────────────┐ │
-│  │ Retriever  │  │ Classifier │  │ Context Composer     │ │
-│  │ (vector +  │  │ (6 cats)   │  │ (resume_context)     │ │
-│  │ BM25 + RRF)│  │            │  │                      │ │
-│  └────────────┘  └────────────┘  └──────────────────────┘ │
-│  ┌────────────┐  ┌────────────┐  ┌─────────────────────┐ │
-│  │ Decay      │  │ Conflict   │  │ Capture Engine       │ │
-│  │ Engine     │  │ Engine     │  │ (evidence → durable) │ │
-│  │ (Weibull)  │  │ (audit +   │  │                      │ │
-│  │            │  │  merge)    │  │                      │ │
-│  └────────────┘  └────────────┘  └──────────────────────┘ │
-└──────────────────────────┬───────────────────────────────┘
-                           ▼
-┌──────────────────────────────────────────────────────────┐
-│                    Storage Layer                          │
-│  ┌─────────────────────┐  ┌────────────────────────────┐ │
-│  │ LanceDB             │  │ Jina Embeddings v5         │ │
-│  │ (vector + columnar) │  │ (1024-dim, task-aware)     │ │
-│  └─────────────────────┘  └────────────────────────────┘ │
-└──────────────────────────────────────────────────────────┘
-```
-
-### Internal Design
+| | Enumerated | Complement |
+|---|---|---|
+| Agent-made images | 5,812 | **10,938** |
+| Sessions with any image | 1,507 | **1,767** |
+| Human-pasted images | 1,629 | 1,629 |
 
-- **L0 / L1 / L2 Dynamic Folding** — every memory stores 3 granularity layers (one-liner / bullet summary / full content); retrieval dynamically selects which layer to return based on relevance score and token budget
-- **Weibull Decay + Emotion Modulation** — memories decay along a parametric Weibull curve; importance scores modulate the half-life, and emotional salience extends it further (up to 30%)
-- **Vector Pre-filter + LLM Dedup** — 90% of dedup decisions use cheap cosine similarity (>= 0.92); only borderline cases invoke LLM judgment, keeping costs low without sacrificing accuracy
-- **Category-Aware Merge Strategies** — `profile` and `preferences` use merge-on-conflict (latest wins); `events` and `cases` use append-only (history preserved)
-- **Display Score vs Elimination Score** — dual-track retrieval: tier floor prevents core memories from ever dropping out, while decay boost lets fresh memories surface temporarily without permanently displacing stable ones
+**The enumeration missed 5,126 images and 376 sessions** — nearly half. The largest class
+it dropped was image *generation*, which lives in neither container the list knew about.
+Human-pasted counts are identical under both definitions, which is the check that matters:
+widening the second bucket did not contaminate the precise one. A regression test feeds the
+parser an `image_generation_call` — a shape the source never names — and asserts it lands
+in the second bucket; under the enumerated implementation that test fails.
 
-> Full architecture deep-dive: [`docs/architecture.md`](docs/architecture.md)
+One caveat: the complement counts *signals*, not certified pictures. A single generation can
+leave both a call and a completion record and be counted twice. That direction was chosen
+deliberately — the question is "is there anything here to look at", not "exactly how many".
 
----
 
 ## Interfaces
 
@@ -499,7 +445,7 @@ Examples live in [`integrations/examples/`](integrations/examples/):
 | `data_checkup` | Run data quality health checks on the memory store |
 | `dream` | Run offline memory consolidation (clustering, merging, pruning) |
 | `memory_lint` | Run memory quality checks: contradictions, duplicates, stale entries, orphans |
-| `forget_memory` | Cascade-delete a memory with KG cleanup, pin archival, and audit trail |
+| `forget_memory` | Delete a memory together with its KG triples, trigger rows and the indexed copies of pins made from it, archive the pin files, read back to confirm, and write an audit entry |
 | `export_graph` | Export memories as an interactive HTML knowledge graph |
 
 </details>
@@ -569,7 +515,7 @@ bun run src/cli.ts lint --scope project:myapp   # lint a specific scope
 bun run src/cli.ts graph --open                 # export & open knowledge graph
 bun run src/cli.ts graph --max-nodes 50         # smaller graph
 
-# Ingestion & diagnostics
+# Ingestion & diagnostics (transcripts are not ingested by default; see "Index existing conversations")
 bun run src/cli.ts ingest --source all
 bun run src/cli.ts doctor
 ```
@@ -578,6 +524,53 @@ bun run src/cli.ts doctor
 
 ---
 
+## Web UI
+
+<p align="center">
+  <img src="assets/dashboard.png" alt="RecallNest Dashboard" width="800" />
+  <br><em>Dashboard — total count, category distribution, health score, and growth trends at a glance.</em>
+</p>
+
+<p align="center">
+  <img src="assets/screenshots/ui-full.png" alt="RecallNest Search Workbench" width="800" />
+  <br><em>Search Workbench — hybrid search with topic tag filtering, 4 retrieval profiles, Skills browser, and asset management.</em>
+</p>
+
+<p align="center">
+  <img src="assets/knowledge-graph.png" alt="RecallNest Knowledge Graph" width="800" />
+  <br><em>Knowledge Graph — interactive force-directed visualization with semantic bridges revealing cross-domain connections.</em>
+</p>
+
+```bash
+bun run src/ui-server.ts
+# → http://localhost:4317
+```
+
+---
+
+## What's new
+
+**v4.0** runs on **Bun only** and hands its cross-process locks to the kernel. RecallNest
+used to decide a lock file was stale from its pid and timestamp and then delete it — two
+processes that saw the same stale file could both end up holding the lock (more than one
+winner in up to 30 of 40 rounds of an eight-process race). Locks are now `flock(2)`: the
+kernel releases them when a process exits or is killed, and a stress test that kills a
+writer every 15–60 ms for six seconds never sees two writers inside.
+
+The rest of the release fixes data that could go wrong quietly: a deleted memory written
+back by a concurrent update, history rows left pointing at nothing, secrets kept in
+metadata and sent to the embedding API inside triggers, explicit searches such as
+`git push rules` or `#22` returning nothing, and write receipts that reported rejected
+writes as stored.
+
+**Upgrading:** install Bun, then stop every RecallNest process before starting v4 — old and
+new lock code must not run together. A few defaults and HTTP replies changed; the list is
+at the top of the v4.0.0 entry in the changelog. Existing LanceDB data opens in place.
+
+**Full history — v4.0 through v1.0, with the upgrade notes for each — is in
+[CHANGELOG.md](CHANGELOG.md).**
+
+
 ## Multilingual Support
 
 RecallNest works out of the box with English. For multilingual memory (Chinese, Japanese, Thai, and 20+ more), install [babel-memory](https://github.com/AliceLJY/babel-memory) with the language packs you need: