recallnest 2.6.1 → 3.0.1

package/CHANGELOG.md

@@ -1,5 +1,254 @@
 # Changelog
 
+<!-- Before cutting a release, run every entry past these five:
+     1. before / after — a number, not an adjective. No number, no claim.
+        (Nothing measurable? Then one plain sentence. Do not manufacture a figure.)
+     2. under what conditions was it measured — version, sample size, environment.
+     3. what does this cost and who gets hit by it — state it, do not bury it.
+     4. what alternative was rejected and why — the reason must be specific
+        enough that someone could refute it.
+     5. what looks like it belongs here but deliberately does not, and on what evidence.
+
+     When condensing this file into a GitHub Release body, 2 / 4 / 5 are the first
+     things to go: a conclusion reads as information, a reason reads as filler.
+     Protect them by name. The v3.0.0 Release body dropped 11 of 11 such items in
+     exactly that step, while this file had every one of them. -->
+
+## v3.0.1 — Publication hygiene (2026-09-06)
+
+### Added
+
+- **`embedding.timeoutMs` — an opt-in per-request timeout for the embedding client**
+  (config.json, or `RECALLNEST_EMBEDDING_TIMEOUT_MS`, which wins over the file so one
+  deployment can bound itself without editing a shared config). **Default behavior is
+  unchanged**: with neither set, no `timeout` is passed to the SDK at all and the client
+  keeps its own default — 600s in `openai` v7 — exactly as in every release through
+  v3.0.0. The knob exists because that default stacks badly: the SDK retries twice on its
+  own and `Embedder` retries twice more, so a black-holed endpoint issues **9 requests for
+  a single `embedPassage` call** (measured against a loopback endpoint that never answers),
+  putting the worst case at roughly 90 minutes. Lowering the default for everyone was
+  rejected instead of shipped — bulk embedding of long documents is legitimately slow, and
+  a global cut would turn working ingests into failures — so the timeout is only ever
+  shortened by a caller that asks. Non-finite, zero and negative values are ignored rather
+  than throwing, so a typo degrades to the old behavior instead of to an embedder that
+  aborts every request. Covered by the loopback suite (abort within the configured budget)
+  and by `runtime-config-embedding.test.ts` (config.json → client, env precedence, and a
+  whole-client comparison against the pre-change construction expression proving the unset
+  path is unchanged).
+
+### Changed
+
+- **Publication hygiene: no machine-local identifiers in the tree, no tests in the tarball.**
+  Three eval scripts (`eval/ghost-scan.ts`, `eval/lengthnorm-shadow.ts`,
+  `eval/term-registry-shadow.ts`) imported the runtime through an absolute path that only
+  existed on the author's machine; they now resolve the repository root from
+  `RECALLNEST_ROOT`, falling back to `~/recallnest` — the identical path on the original
+  install, a working one everywhere else. `scripts/pivot-distill-supervisor.ts` derives its
+  user-level launchd labels from the current account (`com.<user>.<task>`, overridable via
+  `PIVOT_DISTILL_LAUNCHD_PREFIX`) instead of a hardcoded username; the supervised set is
+  unchanged for the original account, and the suite now asserts the derivation rule and the
+  override. `package.json#files` excludes `src/**/__tests__/**` — `.npmignore` alone never
+  could, because `files` wins — so the dry-run tarball goes from 357 files / 965.8 kB
+  (3.8 MB unpacked) to 189 files / 620.0 kB (2.1 MB unpacked), measured with npm 11.19.0
+  on this commit; `scripts/verify-package-contents.mjs` now rejects any `__tests__` directory,
+  so the CI gate turns red if this regresses. A new `privacy-regression.test.ts` scans
+  `src/`, `scripts/`, `eval/`, `bin/`, root-level `*.md` and shipped `data/*.example.*` for the
+  removed identifiers and for macOS home paths (obvious placeholder accounts excepted).
+  Rejected: relative imports in the eval scripts, because these are reproduction tools for
+  figures measured against the production install and must keep pointing at it even when
+  invoked from another checkout. Deliberately not here: the README acknowledgements are unchanged.
+
+## v3.0.0 — Node 22, and conclusions that can be used (2026-08-24)
+
+Two things make this a major release: the runtime floor moves, and a synthesized
+conclusion can finally reach stable memory. Everything else here is either in service of
+those or was found while doing them.
+
+### Changed
+
+- **`engines.node` is now `>=22`** (was `>=18.14.1`). This is the only breaking change in
+  the release; Bun users are unaffected.
+- **`openai` migrated from `^4.0.0` to `^7.5.0`**, which removes the
+  `openai@4 → formdata-node@4.4.1 → node-domexception@1.0.0` deprecated chain from the
+  dependency tree. Every `openai` release since v5 has zero dependencies, so the chain
+  disappears on any upgrade — v7 was chosen because it declares `engines.node >= 22.0.0`,
+  making the Node floor and the dependency cleanup one decision instead of two. Verified
+  against the SDK rather than its migration guide: the guide documents the `httpAgent`
+  removal, the move to built-in `fetch`, and the Node 22 floor, but says nothing about the
+  three things this codebase actually depends on — `embeddings.create`, the second
+  `RequestOptions` argument on `chat.completions.create`, and the error classes exposed as
+  statics on the default export. All three hold.
+- **MCP tool count is 44**, up from 43 (`promote_synthesis`, governance tier).
+
+### Added
+
+- **`promote_synthesis` (MCP) and `recallnest promote-synthesis` (CLI)** — a promotion road
+  for dream-synthesized conclusions. `buildDerivedBoundary` marks every cluster insight and
+  cross-memory pattern `layer: "evidence"` on purpose, and `shouldUseStableMemoryResult`
+  refuses the evidence layer, so a synthesized conclusion could never take part in stable
+  memory regardless of how well supported it was. Eligibility comes from the conclusion's
+  own validated evidence set — `synthesis_contract >= 2`, at least two distinct still-active
+  evidence memories — rather than from repetition, because a synthesis is already a
+  cross-entry aggregate. It abstains when the evidence cannot be resolved, counts every
+  rejection reason, defaults to dry-run, never modifies the synthesized row, and writes
+  through the existing `promoteMemory` path so `canonicalKey` dedup makes a re-run a
+  revision instead of a duplicate.
+- **Retrieve audit rows carry revision and provenance.** `retrieved[]` records each served
+  result's id, `evolution.version`, lifecycle status, and boundary layer/authority;
+  `retrievedTotal` appears when the capped list is shorter than the truth. Beliefs are
+  revised in place — the canonical id survives, the version increments, the old text
+  remains as a `superseded` row — so a row saying only `hits=N` could not distinguish a
+  current answer from a stale one after the fact.
+- **HTTP contract tests for embeddings and chat completions** (`openai-contract.test.ts`).
+  Every prior embedder/LLM test stubbed the SDK client, so none of them touched a socket
+  and a transport-level regression would have passed all of them. These drive the real
+  `Embedder` and `LLMClient` through the real SDK against a loopback `node:http` server,
+  covering success, error, and timeout for the default OpenAI request shape plus the Jina
+  (`task`/`normalized`/`dimensions`) and Qwen-compatible profiles. No network egress, no
+  vendor credentials, no paid calls.
+- **A repeatable regression for verbatim self-recall** (`verbatim-self-recall.test.ts`).
+  A long, older, rarely-read entry can fail to be recalled by words copied out of its own
+  body while short, fresh, frequently-read entries outrank it. The reproduction pins the
+  mechanism: the entry is *not* filtered — `hard_min_score`, `layer_admission` and
+  `noise_filter` each drop zero rows and its score clears the threshold — it is admitted
+  and then ranked last. The ranking fix is deliberately not in this release.
+- **`scripts/tier-exemption-snapshot.ts`** — captures tier and decay-exemption distribution
+  so "we did not change tiers" becomes checkable later rather than asserted.
+
+### Added — synthesis quality (generation side)
+
+Six changes to what `dream` produces, from a full diagnostic on 2026-08-22 whose core
+finding was that **derived memories had a lower conclusion density than their own input**:
+over the same 40–160 character band, with a deterministic regex and no LLM judge,
+hand-written pivots scored 40.4%, raw transcripts 25.1%, insights 14.4%, patterns 12.1%.
+Synthesis was subtracting.
+
+- **`cluster_insight` no longer routes through `generateL0`.** That prompt is a chunk
+  summariser — it asks for a one-line summary for retrieval and explicitly preserves ports,
+  IPs, URLs and file paths verbatim; it never asks for a conclusion, a cause, or a decision.
+  The new `synthesizeClusterInsight` goes through `synthesis-contract.ts`, asks for a
+  reusable conclusion, and **may abstain**. The old path's `if (!insight)` only fired when
+  the LLM call itself failed, so "this cluster is not worth keeping" had no way to be
+  expressed and all 2,427 rows were episode summaries by construction.
+- **`extractPattern` changed target concept and now requires evidence.** The old prompt
+  asked for recurring preferences, behavioural tendencies and values — a personality-profile
+  frame, which produced dispositional attribution in 56.1% of outputs against 1.4% in the
+  source material. The old prompt also claimed to want "at least 2 memories as supporting
+  evidence" while the implementation neither received nor checked any numbering, making the
+  sentence free. Two pieces of evidence is now a hard gate.
+- **`validateSynthesis` runs before anything is written.** The old path checked only
+  `response.length < 5`, which is how 28 memories ended up holding their own system prompt
+  as their body. `has` must be a boolean, length must be in range, the text must not echo
+  the prompt, and evidence numbers must refer to real, distinct cluster members. Failures
+  are counted by reason instead of collapsing into one `null`.
+- **Pattern and insight are decoupled.** `if (!insight) { continue; }` bound two different
+  target concepts to one fate: an abstaining insight meant pattern extraction never ran.
+  They now fail and abstain independently, and "this round produced nothing" is judged on
+  whether the cluster had any effect at all rather than on whether an insight existed.
+- **Derived rows carry an explicit `boundary`.** All 4,749 historical derivatives had none;
+  the retriever's `isEvidenceLayer` fell back to scope matching, so 96.4% were classified
+  correctly only because they happened to sit in a `cc:`/`codex:` scope — by luck of naming,
+  not by design. The remaining 173 sat in `memory`/`project:*` scopes and silently held
+  durable authority. Derived rows are `layer: "evidence"`, `authority: "distillation"`.
+- **`synthesis_contract` version stamp**, so "which code produced this output" stops being
+  a guess from timestamps. v2 adds a single-JSON-object constraint: v1 emitted multiple JSON
+  objects on 3.3% of calls (all with `finish_reason: stop`, unrelated to truncation), which
+  made the whole response unparsable; the same measurement after the constraint is 0.0%.
+
+### Added — memory utility (MemOS value backtracking)
+
+- **Value-backtracking join key (P0).** `workflow_observe` now records `readerId` and
+  `recalledIds`, supplying the `(memory_id, outcome)` pairing the database never had: one
+  side had outcomes without participants (observations did not record memories), the other
+  had participants without outcomes (the access tracker recorded hits but not task success).
+  New `src/recall-ledger.ts` keeps process-level reader identity and a retrieval-hit ledger;
+  managed observations from `resume_context` / `checkpoint_session` carry it automatically.
+  New CLI `memory-utility <id>` reports the success rate of tasks a memory took part in.
+- **`utility` column (P1).** `src/memory-utility.ts` weights outcomes (success +0.8 /
+  corrected +0.2 / missed −0.3 / failure −0.8) with a 30-day half-life; **it can be
+  negative**. It lands in `metadata.utility` and deliberately not in `importance`, which is
+  threshold-valued (≥0.95 grants permanent decay exemption) and already written by structured
+  memory. Retrieval gained `utilityWeight`, default 0 (off), feeding ranking only — not tier,
+  not decay exemption.
+- **Cross-source criterion for promotion (P2).** `scanForPromotions` gained
+  `minDistinctSources` (default 2). Counting "how many similar cases" and "how many distinct
+  episodes" differ in dimension, not degree — hitting the same problem three times is one
+  experience repeated, which belongs on the failure-burst path instead. It **abstains** when
+  source pointers are unavailable: only 1.2% of cases across the database carry a `src:` tag,
+  so judging strictly would mostly punish three different problems solved on one day.
+
+### Fixed
+
+- **A rate-limit reply could trigger an unbounded request storm.** Found by the new contract
+  tests. `embedSingle` retried a context-length error by chunking, and each chunk recursed
+  back into `embedSingle`, able to re-enter the same branch. Two facts made that
+  non-terminating: `smartChunk` returns short text unchanged, so the recursion re-embedded
+  the identical string and reproduced the identical error; and the gate
+  `/context|too long|exceed|length/i` also matches rate-limit wording — both
+  `"Rate limit exceeded"` and `"429 quota exceeded"` trigger chunking. Measured before the
+  fix: **61,724 requests in five seconds** against an endpoint asking us to slow down. The
+  batch path had the same shape. The missing terminating condition is now present: chunking
+  that cannot produce anything smaller than its input throws instead of recursing, and a
+  chunk's own embed call cannot re-enter the branch. After: 4 ms, one request, correct error.
+- **Multiple `AccessTracker` instances in one process no longer invent separate readers.**
+  `createComponentResolver` caches components per profile, so one session opening two
+  profiles produced a phantom second reader, inflating `distinctReaderCount` and the
+  skill-promotion read boost. The implementation now matches what `access-tracker.ts` says
+  about itself ("one stdio MCP server process ≈ one CC session").
+
+### Documentation
+
+- `docs/memory-boundary-contract.md` documents both promotion scans, why their eligibility
+  gates differ, and the abstention rule.
+- `retriever.ts` records that `candidatePoolSize` above 20 has never had any effect — both
+  candidate legs land in `store.ts`'s `clampInt(limit, 1, 20)`. The clamp has been there
+  since the first commit while the 2026-07-16 tuning only touched the retriever, so that
+  `30` was inert from the day it landed. Left alone deliberately: raising it means widening
+  the candidate pool, which was measured and rejected on 2026-08-22.
+
+### Upgrade notes
+
+- **Node 22 or newer is required.** This is the release's only breaking change.
+- Existing LanceDB data opens in place; no export/import step.
+- `promote_synthesis` defaults to dry-run and writes nothing until `dryRun=false`.
+- `audit.jsonl` retrieve rows are larger now that they list what was served. The list is
+  capped at 10 entries with `retrievedTotal` marking truncation. Rotation is still manual;
+  archive it if it approaches 50 MB.
+- `utilityWeight` defaults to 0 and `metadata.utility` is written only by an explicit
+  `memory-utility --apply` run, so the MemOS work above is **available but not yet acting on
+  anything**. The resident MCP server was restarted on 2026-08-24, so `readerId` is now being
+  written (verified end-to-end on a fresh server: a `workflow_observe` call landed an
+  observation carrying `readerId`); `recalledIds` only gets a value when a retrieval hit
+  actually occurs, and no row carries `metadata.utility` yet because `--apply` has not been
+  run. Real data therefore starts accumulating from this release forward, not before it.
+  This is the intended shadow period, not a defect.
+
+### Verification
+
+- 2,324 tests pass across 165 files, 0 fail (2,281 at the 3.0 gate → +14 contract, +6 audit
+  revision, +8 verbatim self-recall, +15 synthesis promotion). Verified under three
+  feature-flag combinations, including the CI default, after CI caught one assertion
+  that had been reading the developer's local `.env` rather than product behaviour.
+- The deprecated chain is absent from `npm ls`, `node_modules/`, `bun.lock` and
+  `package-lock.json`; installed `openai@7.5.0` reports zero dependencies and
+  `engines.node >= 22.0.0`.
+- SDK compatibility was measured, not assumed: 20 sandbox probes against v7.5.0 (9 API
+  surface, 11 loopback-HTTP runtime) plus a `tsc --strict` check of all five call sites,
+  with a deliberately-broken control run to confirm the type check actually reports errors.
+- `promote_synthesis` was exercised read-only against the live database: 52 contract-stamped
+  derivatives exist; `project:codex-self-evolution` yielded 10 candidates from 10 examined,
+  `project:antigravity-cli` 4 from 9 (5 below importance), `learnings` 3 from 4 (1 with too
+  few distinct sources). Nothing was written.
+- The verbatim self-recall and synthesis-promotion suites were both reverse-verified —
+  removing the behaviour each asserts turns the relevant tests red and no others.
+
+### A note on npm history
+
+`recallnest@2.6.1` was a maintenance release published before Trusted Publishing was
+configured for this package and therefore carries no build provenance. That describes how it
+was published; it is not retroactively fixable, since the same npm version cannot be
+republished to add provenance.
+
 ## v2.6.0 — Cross-process consistency and reliable distribution (2026-08-13)
 
 ### Added
@@ -136,6 +385,49 @@ P0 反馈闭环：让 skill 的 `successCount` / `failureCount` 真有真实使
 
 - 基线 1486 → 1520 / 0 fail（+13 新 case 覆盖 outcome 映射 + 错误路径 + 元数据完整性 + 时间戳；+1 顺手修 workflow-observation.test.ts dashboard 漏传 now 参数 pre-existing bug）
 
+## v2.3 — Connector ecosystem + source health
+
+> Migrated out of README on 2026-08-28. These three entries lived in the README as
+> "New in vX" sections and were never in this file; release dates were not recorded at
+> the time, so none is stated here rather than guessed. They sit between v2.5.0-pre
+> (2026-05-27) and v1.3.1 (2026-03-12) by position in the sequence.
+
+v2.2 hardened retrieval quality; v2.3 opens RecallNest to external data sources with a standard connector framework and operational health monitoring.
+
+- **Connector-v1 Standard** *(GB-2)* — A JSON format (`ConnectorOutputV1`) that any external script can produce. Obsidian vaults, emails, RSS feeds, log files — normalize once, ingest through the full dedup/embed/extract pipeline. See [`docs/connector-spec.md`](docs/connector-spec.md) for the specification and [`connectors/examples/`](connectors/examples/) for adapter skeletons (email, logs, RSS).
+
+- **Obsidian Vault Ingestion** *(GB-1)* — First-party Obsidian connector: scans `.md` files, extracts frontmatter + wikilinks, maps folder structure to tags. One command: `lm ingest --obsidian /path/to/vault`.
+
+- **Source Health Monitoring** *(GB-3)* — Every connector ingest writes a heartbeat to `data/source-heartbeat.json`. `data_checkup` flags stale sources (>7d warning, >30d error). `doctor --ci` shows a per-source heartbeat summary with human-readable age.
+
+## v2.2 — Retrieval quality hardening
+
+v2.1 added philosophy-informed behavior; v2.2 closes the last three engine-layer gaps identified by a frontier research scan (ACC, PI-LLM, TSM).
+
+- **Memory Confidence Meta-tags** *(ACC / Dual-Process UQ)* — Each memory now carries structured `ConfidenceMetadata` (score, reliability tier: `direct` / `inferred` / `hearsay`). Auto-assigned from source on write (`manual` = 0.9, `agent` = 0.7, `conversation_import` = 0.5). Retrieval scores are weighted by confidence. `resume_context` tags low-confidence items with `[低置信]`.
+
+- **Interference Detection + Active Forgetting Gate** *(PI-LLM / SleepGate)* — Semantic cluster detection identifies groups of near-duplicate memories competing for retrieval. Enhanced RIF keeps only top-K (default 3) per cluster; extras are demoted 50% instead of removed. Write-time pre-warning: when a scope accumulates ≥5 high-similarity active memories, the weakest is flagged `pending_review`. `data_checkup` reports interference density.
+
+- **Temporal Validity Windows** *(TSM / TiMem / Zep)* — `store_memory` accepts `validUntil` (expiration) and `eventTime` (when the event actually happened). `search_memory` supports `validAt` (point-in-time query) and `includeExpired` (demote 80% instead of hide). Auto-GC applies 2× decay acceleration to expired memories.
+
+- **Usage-Adjusted Auto-GC** *(off by default)* — `RECALLNEST_USAGE_DECAY=true` enables a GC-only cold-memory penalty when constructive retrieval is also active. Cold memories discount the frequency component instead of changing online retrieval ranking.
+
+## v2.1 — Philosophy-informed memory
+
+v2.0 built the operational memory platform; v2.1 added philosophy-informed memory behavior.
+
+Five upgrades derived from 9 research dimensions in philosophy of memory, each mapped to concrete engineering:
+
+- **Emotion-Aware Decay** *(Affective Memory Theory)* — Memories with strong emotional content decay 20-30% slower. Keyword-based emotion detection computes `salience` (mnemonic significance), which feeds into the Weibull half-life formula and a rebalanced 4-factor evolution score. Zero LLM cost.
+
+- **Memory Ethics Layer** *(Right to Be Forgotten / GDPR Art. 17)* — Four privacy tiers (`ephemeral` / `private` / `durable` / `shared`). Cascade forgetting engine that propagates deletion through KG triples, evolution chains, pin assets, and briefs. Full audit trail. `forget_memory` MCP tool for agent-driven deletion.
+
+- **Autobiographical Narrative** *(Narrative Identity Theory / Conway's 3-layer model)* — Memories are tagged with `lifePeriod → generalEvent → specificEvent` hierarchy, orthogonal to existing 6 categories. Retrieval pulls narrative siblings. Context rendering groups by life period. Rule-based tagger with EN+CN support.
+
+- **Constructive Retrieval** *(Simulation Theory / Michaelian)* — Instead of returning raw stored text, RecallNest now reconstructs context from an expanded candidate set: KG neighbors + evolution chains + cluster members + narrative siblings. Source-map grounded coverage replaces lexical overlap. Contradictions are detected and flagged.
+
+- **Predictive Prospective Memory** *(Mental Time Travel / Tulving)* — Heuristic prediction engine that surfaces "you might need this" reminders from behavioral signals: stale checkpoint open loops, corrected workflow observations, high-frequency dormant memories, and uncovered query topics. Zero LLM cost. Auto-expire in 7 days if unaccepted.
+
 ## v1.3.1 — Upstream Sync (2026-03-12)
 
 Synced with [CortexReach/memory-lancedb-pro](https://github.com/CortexReach/memory-lancedb-pro) master (v1.1.0-beta.6+).