recallnest 3.0.0 → 3.0.1

package/CHANGELOG.md

@@ -1,5 +1,63 @@
 # Changelog
 
+<!-- Before cutting a release, run every entry past these five:
+     1. before / after — a number, not an adjective. No number, no claim.
+        (Nothing measurable? Then one plain sentence. Do not manufacture a figure.)
+     2. under what conditions was it measured — version, sample size, environment.
+     3. what does this cost and who gets hit by it — state it, do not bury it.
+     4. what alternative was rejected and why — the reason must be specific
+        enough that someone could refute it.
+     5. what looks like it belongs here but deliberately does not, and on what evidence.
+
+     When condensing this file into a GitHub Release body, 2 / 4 / 5 are the first
+     things to go: a conclusion reads as information, a reason reads as filler.
+     Protect them by name. The v3.0.0 Release body dropped 11 of 11 such items in
+     exactly that step, while this file had every one of them. -->
+
+## v3.0.1 — Publication hygiene (2026-09-06)
+
+### Added
+
+- **`embedding.timeoutMs` — an opt-in per-request timeout for the embedding client**
+  (config.json, or `RECALLNEST_EMBEDDING_TIMEOUT_MS`, which wins over the file so one
+  deployment can bound itself without editing a shared config). **Default behavior is
+  unchanged**: with neither set, no `timeout` is passed to the SDK at all and the client
+  keeps its own default — 600s in `openai` v7 — exactly as in every release through
+  v3.0.0. The knob exists because that default stacks badly: the SDK retries twice on its
+  own and `Embedder` retries twice more, so a black-holed endpoint issues **9 requests for
+  a single `embedPassage` call** (measured against a loopback endpoint that never answers),
+  putting the worst case at roughly 90 minutes. Lowering the default for everyone was
+  rejected instead of shipped — bulk embedding of long documents is legitimately slow, and
+  a global cut would turn working ingests into failures — so the timeout is only ever
+  shortened by a caller that asks. Non-finite, zero and negative values are ignored rather
+  than throwing, so a typo degrades to the old behavior instead of to an embedder that
+  aborts every request. Covered by the loopback suite (abort within the configured budget)
+  and by `runtime-config-embedding.test.ts` (config.json → client, env precedence, and a
+  whole-client comparison against the pre-change construction expression proving the unset
+  path is unchanged).
+
+### Changed
+
+- **Publication hygiene: no machine-local identifiers in the tree, no tests in the tarball.**
+  Three eval scripts (`eval/ghost-scan.ts`, `eval/lengthnorm-shadow.ts`,
+  `eval/term-registry-shadow.ts`) imported the runtime through an absolute path that only
+  existed on the author's machine; they now resolve the repository root from
+  `RECALLNEST_ROOT`, falling back to `~/recallnest` — the identical path on the original
+  install, a working one everywhere else. `scripts/pivot-distill-supervisor.ts` derives its
+  user-level launchd labels from the current account (`com.<user>.<task>`, overridable via
+  `PIVOT_DISTILL_LAUNCHD_PREFIX`) instead of a hardcoded username; the supervised set is
+  unchanged for the original account, and the suite now asserts the derivation rule and the
+  override. `package.json#files` excludes `src/**/__tests__/**` — `.npmignore` alone never
+  could, because `files` wins — so the dry-run tarball goes from 357 files / 965.8 kB
+  (3.8 MB unpacked) to 189 files / 620.0 kB (2.1 MB unpacked), measured with npm 11.19.0
+  on this commit; `scripts/verify-package-contents.mjs` now rejects any `__tests__` directory,
+  so the CI gate turns red if this regresses. A new `privacy-regression.test.ts` scans
+  `src/`, `scripts/`, `eval/`, `bin/`, root-level `*.md` and shipped `data/*.example.*` for the
+  removed identifiers and for macOS home paths (obvious placeholder accounts excepted).
+  Rejected: relative imports in the eval scripts, because these are reproduction tools for
+  figures measured against the production install and must keep pointing at it even when
+  invoked from another checkout. Deliberately not here: the README acknowledgements are unchanged.
+
 ## v3.0.0 — Node 22, and conclusions that can be used (2026-08-24)
 
 Two things make this a major release: the runtime floor moves, and a synthesized
@@ -327,6 +385,49 @@ P0 反馈闭环：让 skill 的 `successCount` / `failureCount` 真有真实使
 
 - 基线 1486 → 1520 / 0 fail（+13 新 case 覆盖 outcome 映射 + 错误路径 + 元数据完整性 + 时间戳；+1 顺手修 workflow-observation.test.ts dashboard 漏传 now 参数 pre-existing bug）
 
+## v2.3 — Connector ecosystem + source health
+
+> Migrated out of README on 2026-08-28. These three entries lived in the README as
+> "New in vX" sections and were never in this file; release dates were not recorded at
+> the time, so none is stated here rather than guessed. They sit between v2.5.0-pre
+> (2026-05-27) and v1.3.1 (2026-03-12) by position in the sequence.
+
+v2.2 hardened retrieval quality; v2.3 opens RecallNest to external data sources with a standard connector framework and operational health monitoring.
+
+- **Connector-v1 Standard** *(GB-2)* — A JSON format (`ConnectorOutputV1`) that any external script can produce. Obsidian vaults, emails, RSS feeds, log files — normalize once, ingest through the full dedup/embed/extract pipeline. See [`docs/connector-spec.md`](docs/connector-spec.md) for the specification and [`connectors/examples/`](connectors/examples/) for adapter skeletons (email, logs, RSS).
+
+- **Obsidian Vault Ingestion** *(GB-1)* — First-party Obsidian connector: scans `.md` files, extracts frontmatter + wikilinks, maps folder structure to tags. One command: `lm ingest --obsidian /path/to/vault`.
+
+- **Source Health Monitoring** *(GB-3)* — Every connector ingest writes a heartbeat to `data/source-heartbeat.json`. `data_checkup` flags stale sources (>7d warning, >30d error). `doctor --ci` shows a per-source heartbeat summary with human-readable age.
+
+## v2.2 — Retrieval quality hardening
+
+v2.1 added philosophy-informed behavior; v2.2 closes the last three engine-layer gaps identified by a frontier research scan (ACC, PI-LLM, TSM).
+
+- **Memory Confidence Meta-tags** *(ACC / Dual-Process UQ)* — Each memory now carries structured `ConfidenceMetadata` (score, reliability tier: `direct` / `inferred` / `hearsay`). Auto-assigned from source on write (`manual` = 0.9, `agent` = 0.7, `conversation_import` = 0.5). Retrieval scores are weighted by confidence. `resume_context` tags low-confidence items with `[低置信]`.
+
+- **Interference Detection + Active Forgetting Gate** *(PI-LLM / SleepGate)* — Semantic cluster detection identifies groups of near-duplicate memories competing for retrieval. Enhanced RIF keeps only top-K (default 3) per cluster; extras are demoted 50% instead of removed. Write-time pre-warning: when a scope accumulates ≥5 high-similarity active memories, the weakest is flagged `pending_review`. `data_checkup` reports interference density.
+
+- **Temporal Validity Windows** *(TSM / TiMem / Zep)* — `store_memory` accepts `validUntil` (expiration) and `eventTime` (when the event actually happened). `search_memory` supports `validAt` (point-in-time query) and `includeExpired` (demote 80% instead of hide). Auto-GC applies 2× decay acceleration to expired memories.
+
+- **Usage-Adjusted Auto-GC** *(off by default)* — `RECALLNEST_USAGE_DECAY=true` enables a GC-only cold-memory penalty when constructive retrieval is also active. Cold memories discount the frequency component instead of changing online retrieval ranking.
+
+## v2.1 — Philosophy-informed memory
+
+v2.0 built the operational memory platform; v2.1 added philosophy-informed memory behavior.
+
+Five upgrades derived from 9 research dimensions in philosophy of memory, each mapped to concrete engineering:
+
+- **Emotion-Aware Decay** *(Affective Memory Theory)* — Memories with strong emotional content decay 20-30% slower. Keyword-based emotion detection computes `salience` (mnemonic significance), which feeds into the Weibull half-life formula and a rebalanced 4-factor evolution score. Zero LLM cost.
+
+- **Memory Ethics Layer** *(Right to Be Forgotten / GDPR Art. 17)* — Four privacy tiers (`ephemeral` / `private` / `durable` / `shared`). Cascade forgetting engine that propagates deletion through KG triples, evolution chains, pin assets, and briefs. Full audit trail. `forget_memory` MCP tool for agent-driven deletion.
+
+- **Autobiographical Narrative** *(Narrative Identity Theory / Conway's 3-layer model)* — Memories are tagged with `lifePeriod → generalEvent → specificEvent` hierarchy, orthogonal to existing 6 categories. Retrieval pulls narrative siblings. Context rendering groups by life period. Rule-based tagger with EN+CN support.
+
+- **Constructive Retrieval** *(Simulation Theory / Michaelian)* — Instead of returning raw stored text, RecallNest now reconstructs context from an expanded candidate set: KG neighbors + evolution chains + cluster members + narrative siblings. Source-map grounded coverage replaces lexical overlap. Contradictions are detected and flagged.
+
+- **Predictive Prospective Memory** *(Mental Time Travel / Tulving)* — Heuristic prediction engine that surfaces "you might need this" reminders from behavioral signals: stale checkpoint open loops, corrected workflow observations, high-frequency dormant memories, and uncovered query topics. Zero LLM cost. Auto-expire in 7 days if unaccepted.
+
 ## v1.3.1 — Upstream Sync (2026-03-12)
 
 Synced with [CortexReach/memory-lancedb-pro](https://github.com/CortexReach/memory-lancedb-pro) master (v1.1.0-beta.6+).

package/README.md

@@ -26,8 +26,127 @@ A local-first memory system backed by LanceDB that turns scattered conversation
 
 Coding agents forget everything between windows. Your context — project configs, debugging decisions, entity mappings — is scattered across Claude Code, Codex, Kimi, Antigravity — and every other terminal you open — with no shared memory.
 
-RecallNest solves this: **a single LanceDB-backed memory layer that your coding agents read and write**. Context stored in one window is auto-recalled in another. Sessions checkpoint on exit and resume on start. Memory decays, evolves, and self-organizes — not just raw log storage.
+RecallNest is **one LanceDB-backed memory layer that all of them read and write**. Context stored in one window is recalled in another. Sessions checkpoint on exit and resume on start. Memory decays, evolves, and self-organizes — it is not a log you grep.
 
+### What a recall actually looks like
+
+```text
+Query   : deploy rollback
+Hits    : 5
+
+#  ID       Score Category  Tier        Source  Date        Age  Retrieval Path
+1  ee79037a 46.1% cases     peripheral  cc      2026-08-25  2d   vector
+   [assistant] Rolled back to the previous image and pinned the digest so the next…
+   prov : evidence/transcript-ingest
+   imgs : 52 agent-made in this session · read sess=dca70d4a
+```
+
+Three things in that block carry most of the design:
+
+- **`Source cc` · `Age 2d`** — this came out of a Claude Code window two days ago and you
+  are reading it from a different terminal, possibly on a different machine. That is the
+  premise the whole project is built on.
+- **`prov : evidence/…`** — every row states which layer it sits on. A fragment scraped out
+  of a transcript never gets to pose as a decision you actually made; moving to durable
+  memory is a separate, gated step with its own evidence requirement.
+- **`imgs : …`** — that session contained 52 images. **Not one of them is in the database.**
+  The line exists so you know there is something to go look at, and producing it cost no
+  model call, no vector, and no storage.
+
+That last one is the approach in miniature: **store what makes a thing findable, not
+everything that could ever be asked about it.** The full reasoning — including the two
+places where the obvious implementation was wrong — is in
+[Images: addressable, not embedded](#images-addressable-not-embedded).
+
+
+## Core Capabilities
+
+### Access & Setup
+
+| Capability | Description |
+|---|---|
+| **CC Plugin** | Install in Claude Code with one command — no manual config |
+| **Shared Index** | One LanceDB store shared by every terminal that speaks MCP |
+| **Dual Interface** | MCP (stdio) for CLI tools + HTTP API for custom agents |
+| **One-Click Setup** | Integration scripts install MCP access and continuity rules |
+
+### Recall & Continuity
+
+| Capability | Description |
+|---|---|
+| **Hybrid Retrieval** | 6-channel: vector + BM25 + L0/L1/L2 multi-vector + KG graph (PPR) |
+| **4 Retrieval Profiles** | default, writing, debug, fact-check — tuned for different tasks |
+| **Session Continuity** | `checkpoint_session` + `resume_context` (full/light/summary modes) with repo-state guard |
+| **Session Distiller** | 3-layer conversation compression: microcompact → LLM summary → knowledge extraction |
+| **Conversation Import** | Import from Claude Code, Claude.ai, ChatGPT, Slack, and plaintext |
+| **Topic Tags** | Intra-scope topic partitioning — auto-detected, filterable in search |
+| **Related Scope Sidecar** | Opt-in `includeRelatedScopes` search over configured `scopeRelations`, shown separately from the main scoped ranking |
+
+### Memory Lifecycle & Governance
+
+| Capability | Description |
+|---|---|
+| **Memory Evolution** | Supersede chains, decay scoring, LLM importance, consolidation, archival |
+| **Smart Promotion** | Evidence → durable memory with conflict guards, merge resolution, and audit trail |
+| **Privacy Tiers** | 4-tier (`ephemeral` / `private` / `durable` / `shared`) with cascade forgetting |
+| **Admission Control** | Write-time gating: noise filter, importance floor, dedup, rate limiting |
+| **Memory Lint** | Contradiction, duplicate, stale, and orphan detection with health score |
+| **Offline Consolidation** | `dream` command: clustering, merging, pruning of accumulated memories |
+
+### Reasoning & Structure
+
+| Capability | Description |
+|---|---|
+| **Knowledge Graph** | Entity relation graph with PPR algorithm for multi-hop questions |
+| **Constructive Retrieval** | Multi-source candidate expansion + grounded context reconstruction |
+| **Narrative Architecture** | 3-layer autobiographical metadata (life-period → general-event → specific-event) |
+| **Skill Memory** | Store, retrieve, and promote executable skills from recurring patterns |
+| **Predictive Reminders** | Behavioral-signal prediction engine surfaces "you might need this" suggestions |
+| **6 Categories** | profile, preferences, entities, events, cases, patterns — with category-aware merge strategies |
+
+### Visibility & Operations
+
+| Capability | Description |
+|---|---|
+| **Dashboard** | Web UI with stats, category distribution, growth trends, and health |
+| **Workflow Observation** | Dedicated append-only workflow health records, outside regular memory |
+| **Structured Assets** | Pins, briefs, and distilled summaries — not just raw logs |
+| **Data Checkup** | Data quality health checks on the memory store (including source health) |
+| **Source Heartbeats** | Automatic ingest health tracking per data source with staleness alerts |
+| **Export Graph** | Export interactive HTML knowledge graph visualization |
+| **Batch Operations** | Store up to 20 memories in a single call with dedup |
+| **Connector Framework** | Standard connector-v1 format for external data sources with example adapters |
+
+---
+
+## Architecture
+
+```
+  CLIENTS                    ACCESS                      CORE ENGINE                    STORAGE
+  ──────────────────────     ───────────────────────     ────────────────────────────   ──────────────────────
+
+  Claude Code                MCP over stdio              Retriever                      LanceDB
+  Codex                ───▶  44 tools, 3 tiers    ───▶   vector + BM25 + RRF     ───▶   vector + columnar
+  Kimi · Antigravity                                     Classifier · 6 categories
+  Doubao desktop                                         Context composer
+                             HTTP API :4318              resume_context                 Jina embeddings v5
+  your scripts · cron  ───▶  21 endpoints          ───▶  Decay · Weibull half-life ─▶   1024-dim, task-aware
+                                                         Conflict · audit + merge
+  phone app            ───▶  read-only gateway     ───▶  Capture: evidence → durable
+                             :8791, token-gated
+```
+
+### Internal Design
+
+- **L0 / L1 / L2 Dynamic Folding** — every memory stores 3 granularity layers (one-liner / bullet summary / full content); retrieval dynamically selects which layer to return based on relevance score and token budget
+- **Weibull Decay + Emotion Modulation** — memories decay along a parametric Weibull curve; importance scores modulate the half-life, and emotional salience extends it further (up to 30%)
+- **Vector Pre-filter + LLM Dedup** — 90% of dedup decisions use cheap cosine similarity (>= 0.92); only borderline cases invoke LLM judgment, keeping costs low without sacrificing accuracy
+- **Category-Aware Merge Strategies** — `profile` and `preferences` use merge-on-conflict (latest wins); `events` and `cases` use append-only (history preserved)
+- **Display Score vs Elimination Score** — dual-track retrieval: tier floor prevents core memories from ever dropping out, while decay boost lets fresh memories surface temporarily without permanently displacing stable ones
+
+> Full architecture deep-dive: [`docs/architecture.md`](docs/architecture.md)
+
+---
 
 ## Who Can Connect
 
@@ -161,277 +280,84 @@ bun run src/cli.ts doctor
 
 ---
 
-## Web UI
+## Images: addressable, not embedded
 
-<p align="center">
-  <img src="assets/dashboard.png" alt="RecallNest Dashboard" width="800" />
-  <br><em>Dashboard — total count, category distribution, health score, and growth trends at a glance.</em>
-</p>
+Conversations contain images. A text memory layer does not. The usual answer is a
+multimodal embedding model — encode every image into the same space as the text. That is
+right for photo libraries. It is the wrong shape here, for a cheap reason: **in a
+conversation an image almost never arrives alone.** It comes wrapped in "look at this
+error", and the reply right after it usually describes what was in the picture. The words
+around the image are already an index of it. What was missing was never semantic search
+over pixels — it was knowing a picture is sitting there at all.
 
-<p align="center">
-  <img src="assets/screenshots/ui-full.png" alt="RecallNest Search Workbench" width="800" />
-  <br><em>Search Workbench — hybrid search with topic tag filtering, 4 retrieval profiles, Skills browser, and asset management.</em>
-</p>
+So RecallNest does not encode images. It records **how many images are in the session a
+memory came from**, and lets you decide whether to open the original transcript. Meaning
+is resolved on demand, by whatever model is asking, at the moment it matters.
 
-<p align="center">
-  <img src="assets/knowledge-graph.png" alt="RecallNest Knowledge Graph" width="800" />
-  <br><em>Knowledge Graph — interactive force-directed visualization with semantic bridges revealing cross-domain connections.</em>
-</p>
+The cost is worth stating plainly: **no multimodal model, no re-embedding, no image
+storage, no change to any vector.** Backfilling 21,319 existing memories touched metadata
+only.
 
-```bash
-bun run src/ui-server.ts
-# → http://localhost:4317
-```
+Two design choices in it were not obvious, and both were wrong on the first attempt.
 
----
+### Session-level, on purpose
 
-## Core Capabilities
+The marker counts the whole session, not the turn — coarser than it first looks like it
+should be, and the coarseness is the point.
 
-### Access & Setup
+A turn that is nothing but a pasted screenshot has almost no text, so it never cleared the
+length gate and never entered the store. Measured on real transcripts, **12.5% of turns
+containing a pasted image were dropped whole** — including the ones worth the most, like
+seven screenshots with no caption, or "here are the steps" attached to a picture that *is*
+the steps. A turn-level marker has nothing to attach to for exactly those. A session-level
+marker lands on that session's *other* memories, which did get stored, and those are what
+a search surfaces.
 
-| Capability | Description |
-|---|---|
-| **CC Plugin** | Install in Claude Code with one command — no manual config |
-| **Shared Index** | One LanceDB store shared by every terminal that speaks MCP |
-| **Dual Interface** | MCP (stdio) for CLI tools + HTTP API for custom agents |
-| **One-Click Setup** | Integration scripts install MCP access and continuity rules |
+The trade-off is undisguised: every memory from a session carries the same count, so the
+images may have nothing to do with the row you are looking at. The line says
+`in this session`, not `in this memory`, for that reason.
 
-### Recall & Continuity
+### Two classes, because they answer different questions
 
-| Capability | Description |
-|---|---|
-| **Hybrid Retrieval** | 6-channel: vector + BM25 + L0/L1/L2 multi-vector + KG graph (PPR) |
-| **4 Retrieval Profiles** | default, writing, debug, fact-check — tuned for different tasks |
-| **Session Continuity** | `checkpoint_session` + `resume_context` (full/light/summary modes) with repo-state guard |
-| **Session Distiller** | 3-layer conversation compression: microcompact → LLM summary → knowledge extraction |
-| **Conversation Import** | Import from Claude Code, Claude.ai, ChatGPT, Slack, and plaintext |
-| **Topic Tags** | Intra-scope topic partitioning — auto-detected, filterable in search |
-| **Related Scope Sidecar** | Opt-in `includeRelatedScopes` search over configured `scopeRelations`, shown separately from the main scoped ranking |
-
-### Memory Lifecycle & Governance
-
-| Capability | Description |
-|---|---|
-| **Memory Evolution** | Supersede chains, decay scoring, LLM importance, consolidation, archival |
-| **Smart Promotion** | Evidence → durable memory with conflict guards, merge resolution, and audit trail |
-| **Privacy Tiers** | 4-tier (`ephemeral` / `private` / `durable` / `shared`) with cascade forgetting |
-| **Admission Control** | Write-time gating: noise filter, importance floor, dedup, rate limiting |
-| **Memory Lint** | Contradiction, duplicate, stale, and orphan detection with health score |
-| **Offline Consolidation** | `dream` command: clustering, merging, pruning of accumulated memories |
-
-### Reasoning & Structure
-
-| Capability | Description |
-|---|---|
-| **Knowledge Graph** | Entity relation graph with PPR algorithm for multi-hop questions |
-| **Constructive Retrieval** | Multi-source candidate expansion + grounded context reconstruction |
-| **Narrative Architecture** | 3-layer autobiographical metadata (life-period → general-event → specific-event) |
-| **Skill Memory** | Store, retrieve, and promote executable skills from recurring patterns |
-| **Predictive Reminders** | Behavioral-signal prediction engine surfaces "you might need this" suggestions |
-| **6 Categories** | profile, preferences, entities, events, cases, patterns — with category-aware merge strategies |
-
-### Visibility & Operations
-
-| Capability | Description |
-|---|---|
-| **Dashboard** | Web UI with stats, category distribution, growth trends, and health |
-| **Workflow Observation** | Dedicated append-only workflow health records, outside regular memory |
-| **Structured Assets** | Pins, briefs, and distilled summaries — not just raw logs |
-| **Data Checkup** | Data quality health checks on the memory store (including source health) |
-| **Source Heartbeats** | Automatic ingest health tracking per data source with staleness alerts |
-| **Export Graph** | Export interactive HTML knowledge graph visualization |
-| **Batch Operations** | Store up to 20 memories in a single call with dedup |
-| **Connector Framework** | Standard connector-v1 format for external data sources with example adapters |
-
----
-
-## New in v3.0: A Supported Runtime, and Conclusions That Can Be Used
-
-v3.0 is a major release for one reason that shows up on install and one that shows up in
-how memory behaves.
-
-**The runtime boundary moved to Node 22.** RecallNest had been carrying `openai@4`, which
-pulls in the deprecated `formdata-node` → `node-domexception` chain. Every `openai` release
-since v5 has zero dependencies, so the chain disappears on any upgrade — but v7 declares
-`engines.node >= 22.0.0`, which makes "raise the Node floor" and "move off a deprecated
-dependency chain" the same piece of work rather than two. `engines.node` is now `>=22`.
-This is the breaking part of the major.
-
-**A synthesized conclusion can now reach stable memory.** `dream` writes cluster insights
-and cross-memory patterns on the evidence layer deliberately — a model re-reading its own
-memories is a lead to its sources, not authority over them — but stable-memory selection
-refuses the evidence layer outright. The consequence was absolute: no matter how well
-supported a synthesized conclusion was, nothing downstream could lean on it. `promote_synthesis`
-(MCP) and `recallnest promote-synthesis` (CLI) give it a road, gated on the conclusion's own
-validated evidence set rather than on repetition, because a synthesis is already a
-cross-entry aggregate. The synthesized row is never modified; promotion writes a separate
-durable entry carrying `promotedFrom` back to it, through the same path and the same
-`canonicalKey` dedup as every other promotion.
-
-Also in this release:
-
-- **Retrieve audit rows say what was served.** They used to record the query and a hit
-  count. Because beliefs are revised in place — the id stays, the version increments, the
-  old text survives as a `superseded` row — "memory X was retrieved" was ambiguous across
-  every belief change. Rows now carry each result's id, revision, lifecycle status, and
-  boundary. The list is capped, and a capped list says so.
-- **HTTP contract tests for embeddings and chat completions.** Every previous test stubbed
-  the SDK client, so nothing exercised a socket, and a transport-level regression would
-  have passed all of them. These drive the real classes through the real SDK against a
-  loopback server, covering success, error, and timeout for the default OpenAI shape plus
-  the Jina and Qwen-compatible profiles. No network egress, no vendor credentials.
-- **Fixed: a rate-limit reply could trigger an unbounded request storm.** Found by those
-  tests. The embedder retried a context-length error by chunking, the chunker returns short
-  text unchanged, and the gate that decided "this is a context error" also matched
-  rate-limit wording — so a 429 re-embedded the same text forever. Measured at over 61,000
-  requests in five seconds against an endpoint asking us to slow down. Failure is bounded now.
-- **A known recall gap is now a runnable regression.** A memory that is long, older, and
-  rarely read can fail to be recalled by words copied out of its own body, while short,
-  fresh, frequently-read entries outrank it. It is a ranking problem, not a filtering one —
-  the entry clears every threshold and then places last. The reproduction is checked in; the
-  ranking fix is not in this release.
-- **44 MCP tools** across three tiers, up from 43.
-
-### Upgrading from v2.6
-
-- **Node 22 or newer is required.** Bun users are unaffected. This is the only breaking
-  change in the release.
-- Existing LanceDB data is opened in place. No export or import step.
-- `promote_synthesis` defaults to dry-run and writes nothing until `dryRun=false`.
-- `audit.jsonl` rows for retrievals are larger now that they list what was served. Rotation
-  is still manual; archive it if it approaches 50 MB.
-
-### A note on npm history
-
-`recallnest@2.6.1` on npm was a maintenance release published before Trusted Publishing was
-configured for this package, so it carries no build provenance. That is a fact about how it
-was published, not a defect in the package, and it is not retroactively fixable — the same
-npm version cannot be republished to add provenance.
-
----
-
-## New in v2.6: Reliable Cross-Process Memory and Distribution
-
-v2.6 turns the development since v2.5.4 into a release-ready upgrade:
-
-- **Cross-process visibility** — LanceDB now checks for external commits on every read by default, so resident MCP/API/UI processes see writes from CLI ingestion without a restart. Set `RECALLNEST_READ_CONSISTENCY_INTERVAL=<seconds>` for bounded staleness or `off` for the legacy unchecked behavior.
-- **Safer memory evolution** — belief changes preserve the old row as `superseded`; procedural memories avoid time decay; cold-start and length-normalization behavior no longer suppress short entity queries or stable memories.
-- **More reliable consolidation** — `dream` now distinguishes failure classes, honors wall-clock budgets, refills vectors before semantic clustering, and asserts that successful runs actually produced work.
-- **Broader conversation coverage** — Kimi, AGY/Antigravity, and minis sources are recognized across ingestion, scope boundaries, and term resolution.
-- **One public version** — npm metadata, CLI `--version`, MCP handshakes, HTTP health responses, and the Claude Code marketplace all follow the `2.6.0` release contract.
-- **Installable Claude Code plugin** — installation now registers the 43-tool MCP server and continuity skill, asks for the Jina key through sensitive plugin configuration, and keeps config plus LanceDB data in Claude Code's persistent plugin data directory.
-
-### Upgrading from v2.5.4
-
-- Existing LanceDB data is opened in place. Legacy tables missing newer fields are migrated automatically, including empty tables.
-- Plugin data survives plugin updates because it is stored outside the versioned cache. A manual clone continues to use its existing `config.json` and database path.
-- Retrieval auditing now includes read operations. `audit.jsonl` has no automatic rotation yet; consider archiving it if it approaches 50 MB.
-- `RECALLNEST_LAYER_ADMISSION` remains opt-in (`observe` or `on`); the default is `off`.
-
----
-
-## New in v2.1: Philosophy-Informed Memory
-
-v2.0 built the operational memory platform; v2.1 added philosophy-informed memory behavior.
-
-Five upgrades derived from 9 research dimensions in philosophy of memory, each mapped to concrete engineering:
-
-- **Emotion-Aware Decay** *(Affective Memory Theory)* — Memories with strong emotional content decay 20-30% slower. Keyword-based emotion detection computes `salience` (mnemonic significance), which feeds into the Weibull half-life formula and a rebalanced 4-factor evolution score. Zero LLM cost.
-
-- **Memory Ethics Layer** *(Right to Be Forgotten / GDPR Art. 17)* — Four privacy tiers (`ephemeral` / `private` / `durable` / `shared`). Cascade forgetting engine that propagates deletion through KG triples, evolution chains, pin assets, and briefs. Full audit trail. `forget_memory` MCP tool for agent-driven deletion.
-
-- **Autobiographical Narrative** *(Narrative Identity Theory / Conway's 3-layer model)* — Memories are tagged with `lifePeriod → generalEvent → specificEvent` hierarchy, orthogonal to existing 6 categories. Retrieval pulls narrative siblings. Context rendering groups by life period. Rule-based tagger with EN+CN support.
-
-- **Constructive Retrieval** *(Simulation Theory / Michaelian)* — Instead of returning raw stored text, RecallNest now reconstructs context from an expanded candidate set: KG neighbors + evolution chains + cluster members + narrative siblings. Source-map grounded coverage replaces lexical overlap. Contradictions are detected and flagged.
-
-- **Predictive Prospective Memory** *(Mental Time Travel / Tulving)* — Heuristic prediction engine that surfaces "you might need this" reminders from behavioral signals: stale checkpoint open loops, corrected workflow observations, high-frequency dormant memories, and uncovered query topics. Zero LLM cost. Auto-expire in 7 days if unaccepted.
-
----
-
-## New in v2.2: Retrieval Quality Hardening
-
-v2.1 added philosophy-informed behavior; v2.2 closes the last three engine-layer gaps identified by a frontier research scan (ACC, PI-LLM, TSM).
-
-- **Memory Confidence Meta-tags** *(ACC / Dual-Process UQ)* — Each memory now carries structured `ConfidenceMetadata` (score, reliability tier: `direct` / `inferred` / `hearsay`). Auto-assigned from source on write (`manual` = 0.9, `agent` = 0.7, `conversation_import` = 0.5). Retrieval scores are weighted by confidence. `resume_context` tags low-confidence items with `[低置信]`.
-
-- **Interference Detection + Active Forgetting Gate** *(PI-LLM / SleepGate)* — Semantic cluster detection identifies groups of near-duplicate memories competing for retrieval. Enhanced RIF keeps only top-K (default 3) per cluster; extras are demoted 50% instead of removed. Write-time pre-warning: when a scope accumulates ≥5 high-similarity active memories, the weakest is flagged `pending_review`. `data_checkup` reports interference density.
-
-- **Temporal Validity Windows** *(TSM / TiMem / Zep)* — `store_memory` accepts `validUntil` (expiration) and `eventTime` (when the event actually happened). `search_memory` supports `validAt` (point-in-time query) and `includeExpired` (demote 80% instead of hide). Auto-GC applies 2× decay acceleration to expired memories.
-
-- **Usage-Adjusted Auto-GC** *(off by default)* — `RECALLNEST_USAGE_DECAY=true` enables a GC-only cold-memory penalty when constructive retrieval is also active. Cold memories discount the frequency component instead of changing online retrieval ranking.
-
----
-
-## New in v2.3: Connector Ecosystem + Source Health
-
-v2.2 hardened retrieval quality; v2.3 opens RecallNest to external data sources with a standard connector framework and operational health monitoring.
+| Bucket | What it is | The question it answers |
+|---|---|---|
+| user-pasted | Pictures a human put into a message | *Where is that screenshot I sent?* |
+| agent-made | Everything else the session produced | *What did the page look like? What did I generate?* |
 
-- **Connector-v1 Standard** *(GB-2)* — A JSON format (`ConnectorOutputV1`) that any external script can produce. Obsidian vaults, emails, RSS feeds, log files — normalize once, ingest through the full dedup/embed/extract pipeline. See [`docs/connector-spec.md`](docs/connector-spec.md) for the specification and [`connectors/examples/`](connectors/examples/) for adapter skeletons (email, logs, RSS).
+Keeping only the first is tempting — a person searching their own memory wants their own
+screenshots. But an agent reconstructing its past work wants the other: the diagram it
+drew, the rendering it captured, the illustration it made for a post. **Of 1,767 sessions
+carrying images, 1,103 contain no human-pasted image at all.** Keep one bucket and those
+sessions go silent — precisely the ones where the agent did visual work.
 
-- **Obsidian Vault Ingestion** *(GB-1)* — First-party Obsidian connector: scans `.md` files, extracts frontmatter + wikilinks, maps folder structure to tags. One command: `lm ingest --obsidian /path/to/vault`.
+### The second bucket is a complement, not a list
 
-- **Source Health Monitoring** *(GB-3)* — Every connector ingest writes a heartbeat to `data/source-heartbeat.json`. `data_checkup` flags stale sources (>7d warning, >30d error). `doctor --ci` shows a per-source heartbeat summary with human-readable age.
+The first implementation defined agent-made images by enumeration: inside `tool_result`,
+inside `payload.output`, inside `tool.result`. Every location was real. The list was still
+wrong, because **the set of ways an image can appear only grows**, and an enumeration
+silently drops whatever it did not anticipate.
 
----
+So the second bucket is a complement: count every image signal in the record, subtract the
+ones positively identified as human-pasted, attribute the rest without asking where it came
+from. Across 9,619 transcripts:
 
-## Architecture
-
-```
-┌──────────────────────────────────────────────────────────┐
-│                     Client Layer                          │
-├──────────┬──────────┬──────────┬──────────────────────────┤
-│ Claude   │ Anti-    │ Codex    │ Custom Agents / curl     │
-│ Code     │ gravity  │          │                          │
-└────┬─────┴────┬─────┴────┬─────┴──────┬──────────────────┘
-     │          │          │            │
-     └──── MCP (stdio) ───┘     HTTP API (port 4318)
-                │                       │
-                ▼                       ▼
-┌──────────────────────────────────────────────────────────┐
-│                   Integration Layer                       │
-│  ┌─────────────────────┐  ┌────────────────────────────┐ │
-│  │  MCP Server         │  │  HTTP API Server           │ │
-│  │  44 tools           │  │  21 endpoints              │ │
-│  └─────────┬───────────┘  └──────────┬─────────────────┘ │
-└────────────┼─────────────────────────┼───────────────────┘
-             └──────────┬──────────────┘
-                        ▼
-┌──────────────────────────────────────────────────────────┐
-│                     Core Engine                           │
-│                                                           │
-│  ┌────────────┐  ┌────────────┐  ┌─────────────────────┐ │
-│  │ Retriever  │  │ Classifier │  │ Context Composer     │ │
-│  │ (vector +  │  │ (6 cats)   │  │ (resume_context)     │ │
-│  │ BM25 + RRF)│  │            │  │                      │ │
-│  └────────────┘  └────────────┘  └──────────────────────┘ │
-│  ┌────────────┐  ┌────────────┐  ┌─────────────────────┐ │
-│  │ Decay      │  │ Conflict   │  │ Capture Engine       │ │
-│  │ Engine     │  │ Engine     │  │ (evidence → durable) │ │
-│  │ (Weibull)  │  │ (audit +   │  │                      │ │
-│  │            │  │  merge)    │  │                      │ │
-│  └────────────┘  └────────────┘  └──────────────────────┘ │
-└──────────────────────────┬───────────────────────────────┘
-                           ▼
-┌──────────────────────────────────────────────────────────┐
-│                    Storage Layer                          │
-│  ┌─────────────────────┐  ┌────────────────────────────┐ │
-│  │ LanceDB             │  │ Jina Embeddings v5         │ │
-│  │ (vector + columnar) │  │ (1024-dim, task-aware)     │ │
-│  └─────────────────────┘  └────────────────────────────┘ │
-└──────────────────────────────────────────────────────────┘
-```
-
-### Internal Design
+| | Enumerated | Complement |
+|---|---|---|
+| Agent-made images | 5,812 | **10,938** |
+| Sessions with any image | 1,507 | **1,767** |
+| Human-pasted images | 1,629 | 1,629 |
 
-- **L0 / L1 / L2 Dynamic Folding** — every memory stores 3 granularity layers (one-liner / bullet summary / full content); retrieval dynamically selects which layer to return based on relevance score and token budget
-- **Weibull Decay + Emotion Modulation** — memories decay along a parametric Weibull curve; importance scores modulate the half-life, and emotional salience extends it further (up to 30%)
-- **Vector Pre-filter + LLM Dedup** — 90% of dedup decisions use cheap cosine similarity (>= 0.92); only borderline cases invoke LLM judgment, keeping costs low without sacrificing accuracy
-- **Category-Aware Merge Strategies** — `profile` and `preferences` use merge-on-conflict (latest wins); `events` and `cases` use append-only (history preserved)
-- **Display Score vs Elimination Score** — dual-track retrieval: tier floor prevents core memories from ever dropping out, while decay boost lets fresh memories surface temporarily without permanently displacing stable ones
+**The enumeration missed 5,126 images and 376 sessions** — nearly half. The largest class
+it dropped was image *generation*, which lives in neither container the list knew about.
+Human-pasted counts are identical under both definitions, which is the check that matters:
+widening the second bucket did not contaminate the precise one. A regression test feeds the
+parser an `image_generation_call` — a shape the source never names — and asserts it lands
+in the second bucket; under the enumerated implementation that test fails.
 
-> Full architecture deep-dive: [`docs/architecture.md`](docs/architecture.md)
+One caveat: the complement counts *signals*, not certified pictures. A single generation can
+leave both a call and a completion record and be counted twice. That direction was chosen
+deliberately — the question is "is there anything here to look at", not "exactly how many".
 
----
 
 ## Interfaces
 
@@ -578,6 +504,48 @@ bun run src/cli.ts doctor
 
 ---
 
+## Web UI
+
+<p align="center">
+  <img src="assets/dashboard.png" alt="RecallNest Dashboard" width="800" />
+  <br><em>Dashboard — total count, category distribution, health score, and growth trends at a glance.</em>
+</p>
+
+<p align="center">
+  <img src="assets/screenshots/ui-full.png" alt="RecallNest Search Workbench" width="800" />
+  <br><em>Search Workbench — hybrid search with topic tag filtering, 4 retrieval profiles, Skills browser, and asset management.</em>
+</p>
+
+<p align="center">
+  <img src="assets/knowledge-graph.png" alt="RecallNest Knowledge Graph" width="800" />
+  <br><em>Knowledge Graph — interactive force-directed visualization with semantic bridges revealing cross-domain connections.</em>
+</p>
+
+```bash
+bun run src/ui-server.ts
+# → http://localhost:4317
+```
+
+---
+
+## What's new
+
+**v3.0** raised the runtime floor to **Node 22** (the only breaking change — Bun users are
+unaffected) and gave synthesized conclusions a road into stable memory: a `dream` insight
+can now be promoted on the strength of its own validated evidence set, instead of being
+permanently stuck on the evidence layer where nothing downstream could lean on it.
+
+It also fixed a rate-limit reply that could trigger an unbounded request storm — measured
+at over 61,000 requests in five seconds against an endpoint asking us to slow down. Found
+by the new HTTP contract tests, which drive the real client classes against a loopback
+server instead of stubbing the SDK.
+
+Existing LanceDB data opens in place; there is no export or import step.
+
+**Full history — v3.0 through v1.0, with the upgrade notes for each — is in
+[CHANGELOG.md](CHANGELOG.md).**
+
+
 ## Multilingual Support
 
 RecallNest works out of the box with English. For multilingual memory (Chinese, Japanese, Thai, and 20+ more), install [babel-memory](https://github.com/AliceLJY/babel-memory) with the language packs you need: