dsh-engram 0.5.1 → 0.7.0

package/README.md

@@ -264,8 +264,8 @@ curl -s -X POST http://127.0.0.1:3080/api/dsh-engram/import \
 <!-- dsh-market compatibility block — keep labels bilingual and in sync with
      docs/RELEASE-NOTES.TEMPLATE.md (market README convention) -->
 
-- **DSH 版本 / DSH versions**: `>=0.1.2-alpha.2 <0.3.0-0` (本版 `0.5.1`; 已发布的 `0.3.7`、`0.4.0` 仍声明旧的 `0.1.x` 范围 `>=0.1.2-alpha.2 <0.2.0-0` — 见下方矩阵)
-- **最后验证日期 / Last verified**: 2026-10-19 — 会话扫描修复（`session.v<N>.jsonl.zstd`，issue #8）+ 数据导出/导入与替换/冲突可视化（issue #3），npm test 334/334 通过
+- **DSH 版本 / DSH versions**: `>=0.1.2-alpha.2 <0.3.0-0` (本版 `0.7.0`; 已发布的 `0.3.7`、`0.4.0` 仍声明旧的 `0.1.x` 范围 `>=0.1.2-alpha.2 <0.2.0-0` — 见下方矩阵)
+- **最后验证日期 / Last verified**: 2026-10-19 — episode / entity pages / anchor revalidation，npm test 381/381 通过
 - **安装方式已验证 / Verified install**: `dsh plugin --profile web add dsh-engram`
 
 > **Short version.** Every **published** release (`0.2.0` … `0.3.6`) supports
@@ -280,7 +280,9 @@ curl -s -X POST http://127.0.0.1:3080/api/dsh-engram/import \
 | `0.2.0` … `0.3.6` (npm, GitHub v0.3.3–v0.3.6) | published | **old DSH only** — `0.1.0-rc.7` / `0.1.0-rc.8` / `0.1.1-rc.1` / `0.1.1-rc.2` | peers `@deepseek-ai/dsh-*@^0.1.0-rc.7` |
 | `0.3.7` / `0.4.0` (npm) | published | **new DSH, settings `installSection`** — `>=0.1.2-alpha.2 <0.2.0-0` | `engines.dsh` + peers `>=0.1.2-alpha.2 <0.2.0-0` |
 | `0.5.0` (npm) | published | **DSH `0.1.2-alpha.2` … `<0.3.0-0`** — 设置卡迁移到 entry-id volatile Config（`0.1.7+`/`0.2.x`），`≤0.1.5` 自动回退 `installSection` | `engines.dsh` + peers `>=0.1.2-alpha.2 <0.3.0-0` |
-| `0.5.1` (本期 / this release) | published | 同 `0.5.0` — 会话扫描修复（`session.v<N>.jsonl.zstd`，issue #8）、数据导出/导入 GUI、替换/冲突可视化（issue #3） | 同 `0.5.0` |
+| `0.5.1` | published | 同 `0.5.0` — 会话扫描修复（`session.v<N>.jsonl.zstd`，issue #8）、数据导出/导入 GUI、替换/冲突可视化（issue #3） | 同 `0.5.0` |
+| `0.6.0` | published | 同 `0.5.0` — Context Ledger、BoundaryPrune/ResultPack 默认开 + 账本否决、JIT 推送、Context GC 改 opt-in | 同 `0.5.0` |
+| `0.7.0` (本期 / this release) | published | 同 `0.5.0` — session episode、resume brief、entity pages/cards、revalidatable anchors + layered TTL | 同 `0.5.0` |
 
 ### Where the API family broke — `0.1.2-alpha.2`
 
@@ -314,7 +316,7 @@ the loader), and the settings form is keyed by the profile entry id (`engram`),
 so nothing needs "mounting". `0.5.0` feature-detects the two generations in
 `installEngramSettings`:
 
-- DSH `≥0.1.7` / `0.2.x` — the 27 GUI knobs are the volatile `Config` fields
+- DSH `≥0.1.7` / `0.2.x` — the 34 GUI knobs are the volatile `Config` fields
   (`lib/settings.js` `makeEngineConfigSchema`, exported as `Config` from
   `lib/index.js`); the card reads/writes entry `engram` through the same
   `/api/dsh-engram/settings` HTTP route, and the `service.settings` seam
@@ -330,14 +332,29 @@ drives mutations).
 So: keep DSH on `0.1.1-rc.2` or older while running a release ≤ `0.3.6`, and
 upgrade the plugin to `>=0.3.7` before moving DSH to `0.1.2-alpha.2`.
 
-### Current release — `0.5.1`
-
-Patch release on top of `0.5.0`: session scanning resolves the **versioned** log
-name (`session.v3/v4.jsonl.zstd`) instead of the hardcoded
-`session.jsonl.zstd` — on current DSH that mismatch made `/stats`, `/toolstats`,
-BoundaryPrune's repayment history and the CJK session search all return zero
-silently (issue #8). The memory viewer also gains data export/import and
-supersede/contradict visualization (issue #3).
+### Current release — `0.7.0`
+
+From event records to **maintainable current truth**: sessions sediment an
+episode at teardown, new sessions get a resume brief, `esr_node` becomes a
+readable page, and file anchors make stale truth detectable.
+
+- **Session episode** — zero-LLM digest at `session/disposed` (files / commands /
+  failures→fixed / decisions / open loops), never indexed, newest 20 per
+  workspace kept.
+- **Resume brief** — the newest episode from the last 7 days is injected as a
+  `[RESUME]` line (≤240 chars + drill pointer) at session start.
+- **Entity pages + cards** — `truth / timeline / open loops / conflicts /
+  relations`; up to 2 one-line cards in the index, full page via
+  `engram_detail(entity=…)`.
+- **Revalidatable anchors** — file-backed memories record `mtime/size/sha256`;
+  a bounded revalidation at session start tags changed/missing sources `stale`
+  (dropped from index/truth, still re-fetchable).
+- **Layered TTL** — preference/procedure permanent, episode 14d, error 60d,
+  others 180d.
+- **Migration** — only new writes are anchored (no historical backfill);
+  restart `dsh web` after upgrading; an episode appears after one session ends,
+  the resume brief on the next new session. The `0.6.0` ledger / defaults / JIT
+  behaviour is unchanged.
 
 The published `0.4.0` declared `>=0.1.2-alpha.2 <0.2.0-0` — which the semver
 gate **rejects on every DSH `0.2.0-0` prerelease** (`<0.2.0-0` excludes
@@ -358,7 +375,7 @@ feature-detected legacy fallback).
 `0.1.2-alpha.2` is the minimum version for this API family: host settings are
 **feature-detected** — on DSH `≤0.1.5` they register the `dsh-engram`
 namespace via `settings.installSection`; on `≥0.1.7` / `0.2.x` the form is the
-plugin's `Config` with the 27 GUI knobs marked `.volatile()` (entry id
+plugin's `Config` with the 34 GUI knobs marked `.volatile()` (entry id
 `engram`). The browser settings card reads and writes through the plugin's own
 HTTP route (`/api/dsh-engram/settings`, see Route B above) on both. The full
 regression suite (npm test 329/329, incl. the new settings-v2 coverage for the
@@ -623,10 +640,12 @@ Triggers follow DSH's own compaction timing (step pressure / context-overflow /
 `/compact`); the lock, replay validation, tool-call/result pairing and token
 pricing are all reused from the basic engine. Any error falls back to the
 default compressor — compaction is never broken; with `gcReplacesCompaction:
-false` (or a host without `dsh-compaction-basic`) a bare `BasicCompactionEngine`
-is mounted instead, so the `compaction` service never disappears while
-dsh-engram is installed. Mounting registers the `compaction` service on the
-plugin's fiber, so unloading/reloading engram restores the default engine.
+false` (the default since v0.6, or a host without `dsh-compaction-basic`) a bare
+`BasicCompactionEngine` is mounted instead, so the `compaction` service never
+disappears while dsh-engram is installed. Mounting registers the `compaction`
+service on the plugin's fiber, so unloading/reloading engram restores the
+default engine. **Taking over compaction is opt-in**: turn on
+`gcReplacesCompaction` (and `autoWebCompaction` for web presets).
 
 > **Shrink gate**: the harness refuses any checkpoint that is not strictly
 > smaller than the evicted span (`summary is not smaller than the shadowed
@@ -641,19 +660,19 @@ plugin's fiber, so unloading/reloading engram restores the default engine.
 
 #### Where it is enabled (the entry point)
 
-Assembly happens on **two planes** — and web is now fully automatic, zero config:
+Assembly happens on **two planes**, but both switches are opt-in (no takeover by
+default):
 
-- **Host plane (headless / TUI / base profiles) — on by default, zero config**:
+- **Host plane (headless / TUI / base profiles)**:
   the main plugin (`lib/index.js`) mounts the engine inside `ctx.effect` via
   `mountCompactionEngine(ctx, resolved, { readWorkspace }) → new Engine()`,
   registering the `compaction` service on its own fiber; the plugin patch
   disables the base `compaction-basic` row, making engram the sole host-plane
-  provider. `gcReplacesCompaction: true` (default) → `ContextGcEngine`;
-  `false` → a bare `BasicCompactionEngine`. Tune via profile patch
-  (`config: { gcReplacesCompaction: false }`) or the settings card's
-  `gcNarrative` knob (narrative off = fully mechanical).
-- **Preset plane (web profile) — fully automatic, zero config, covers every
-  preset**: web keeps compaction inside each agent preset's isolated realm
+  provider. `gcReplacesCompaction: false` (default) → a bare
+  `BasicCompactionEngine` (DSH's own summarizer); `true` → `ContextGcEngine`
+  (pointer summaries). Enable via profile patch
+  (`config: { gcReplacesCompaction: true }`) or the settings card.
+- **Preset plane (web profile) — covers every stock preset once enabled**: web keeps compaction inside each agent preset's isolated realm
   (the shipped presets mount their own `compaction-basic`), unreachable from a
   host-plane row — and unreachable from a profile patch too (`mountPreset`
   builds its `Include.Config` with no patches; verified against the harness
@@ -663,8 +682,9 @@ Assembly happens on **two planes** — and web is now fully automatic, zero conf
   root: `standard`, `code`, `cordis`, and your own presets): it swaps the
   `compaction-basic` row inside the `compaction` group for
   `dsh-engram/compaction`. That auto-wiring is:
-  - **on by default** (`autoWebCompaction: true`, toggle in the settings card's
-    memory-GC section), idempotent — an already-wired preset is left alone;
+  - **off by default (opt-in)** (`autoWebCompaction: false`, toggle in the
+    settings card's memory-GC section); once enabled it is idempotent — an
+    already-wired preset is left alone;
     **what "off" means**: the plugin stops auto-wiring on later boots (the
     headless/TUI/base host plane is unaffected) — it does NOT un-wire presets
     that were already wired; use `npm run web-compaction:revert` for that;
@@ -719,9 +739,10 @@ Assembly happens on **two planes** — and web is now fully automatic, zero conf
 
 #### First impressions after an npm install (perceive / guide / configure)
 
-- **Automatic**: install into a profile and boot once — the host plane is taken
-  over by `mountCompactionEngine`, and the web plane by `autoWebCompaction`
-  (default on) rewriting every stock preset at boot. **Zero manual setup.**
+- **Semi-automatic (opt-in)**: DSH's own summarizer stays the default. Turn on
+  `gcReplacesCompaction` (host plane) and `autoWebCompaction` (web presets) in
+  the settings card, then boot once — `mountCompactionEngine` takes over and
+  the stock presets are rewritten. **No file editing after that.**
 - **Perceivable**: at boot the plugin writes an authoritative snapshot to
   `$DSH_HOME/engram/context-gc.status.json` (`host` plane mode + the per-preset
   `web` results + effective config). At any time run `npx dsh-engram status` or
@@ -730,8 +751,9 @@ Assembly happens on **two planes** — and web is now fully automatic, zero conf
   log lines confirm it too: `engram context-gc: compaction = Context GC …` and
   `engram web-provision: wired Context GC into preset …`.
 - **Configurable**: one settings card (memory-GC section) governs everything:
-  `autoWebCompaction` (web auto-wiring on/off), `gcReplacesCompaction`
-  (Context GC vs default summarizer), `gcNarrative` (narrative safety net).
+  `autoWebCompaction` (web auto-wiring, default off), `gcReplacesCompaction`
+  (Context GC vs host summarizer, default off), `gcNarrative` (narrative
+  safety net).
   Changes take effect after restarting `dsh web`, and the web rows are
   refreshed to match automatically.
 
@@ -748,7 +770,7 @@ means that preset was customized (or has no compaction group) and was left untou
 `dsh-compaction-basic unavailable` means the backend is missing and everything
 falls back to the default compressor.
 
-### BoundaryPrune (subtask-boundary pruning, default OFF)
+### BoundaryPrune (subtask-boundary pruning, default ON + ledger veto)
 
 Context GC owns *what a compaction summary says*; BoundaryPrune owns *when a
 deterministic prune fires*: at todo-completion progress and goal terminal
@@ -783,10 +805,13 @@ later request. The two switches are independent.
   T2048+small-edges lands 2.4× the first-prune savings of the host default on
   the real corpus (10.14M vs 4.23M chars). Lowering the threshold is safe with
   `resultPack` on, which archives the removed middles.
-- Default **off**: `boundaryPrune: false`. When on, the log shows
+- Default **on**: `boundaryPrune: true` (`false` is the opt-out). Adaptive gate
+  `boundaryPruneAdaptive: true` lets the Context Ledger stop further fires once
+  this workspace's own accounting is net negative (≥3 fires) — evidence can only
+  turn pruning off. The log shows
   `engram boundary-prune (todo @N reqs): pruned …`.
 
-### ResultPack L2 (archive + `result_recall`, default OFF)
+### ResultPack L2 (archive + `result_recall`, default ON)
 
 BoundaryPrune makes pruning lossless: the removed middle of every pruned
 tool result is redacted (same rules as the memory write path) and archived to
@@ -824,6 +849,112 @@ Mechanisms must be provably activating, and dormant ones are off by default.
   the loop already measured from session logs (engram_recall queries → hits →
   engram_detail follows) so you can see injection→query→detail activation.
 
+### Context Ledger (net-savings accounting)
+
+Turns "saves tokens" into a checkable number. Every model request is one ledger
+tick: injected `[ENGRAM]`/`[RECALL]` and `[ESR]` blocks are charged per request
+they survive; results removed by BoundaryPrune are credited per later request
+they no longer replay; every prune pays one prefix-rewrite penalty
+`(1 − ledgerCacheDiscount)` (the KV-cache break). `GET /api/dsh-engram/stats`
+and `/overview` return `ledger`:
+
+- `totals` — `requests / injectedChars / avoidedChars / rewriteCount /
+  rewriteChars / netChars / netTokens` with
+  `net = avoided − injected − rewriteChars × (1 − discount)`;
+- `byWorkspace` / `byDay` — the same arithmetic per workspace and calendar day
+  (`byDay` newest first);
+- `openSessions` — sessions still accumulating in memory (a restart loses at
+  most `flushEvery` requests).
+
+The Telemetry tab renders it directly (net-token headline, saved / injected /
+rewrite-penalty cards, a 14-day red/green net chart), and the memory overview
+adds a "net context tokens" stat card. Durable rollup at
+`~/.dsh/engram/ledger.json` (atomic write, daily buckets, delta flush); the hot
+path is an integer add, zero I/O and zero model calls.
+Defaults: `ledgerEnabled: true`, `ledgerCacheDiscount: 0.1`; off = no listeners,
+no writes. **Honest scope**: tokens are `chars / 4` estimates, not provider
+billing, and no quality delta is claimed; Context GC compaction savings are not
+wired yet (the host does not expose before/after sizes), so the ledger only
+counts what it can prove.
+
+### JIT push (v0.6-C)
+
+Pull-only recall measured dormant (24 `engram_recall` / 8 `engram_detail` across
+112 real sessions), so v0.6 pushes on three deterministic events instead. One
+line per event, at most 3 lines per session (`jitMaxLines`, `jitMaxLineChars`=220),
+and silence when there is nothing relevant to point at:
+
+| Event | Trigger | What is pushed |
+|---|---|---|
+| Error recurrence | the same `cmd:` signature fails a second time and a prior error/procedure memory exists | that memory's id + a one-line excerpt + `engram_detail` |
+| File focus | the same file is edited twice in a row and a filePath/dir-anchored memory exists | preference first, then fact/decision |
+| Goal completion | goal complete/block while active ESR tasks remain | the first two open tasks + `esr_status` |
+
+- **Append-only**: a pushed line is never edited or withdrawn, so the prefix
+  changes once per push. Each push charges a rewrite
+  (`estimatePrefixChars + line length`) in the Context Ledger
+  (`rewriteCount` / `rewriteChars`), visible on the telemetry page.
+- **Silent without a hit** — no memory/task to point at, no line.
+- **Zero LLM, pure rules**; `jitEnabled:false` disables it, and the empty
+  `[JIT]` section costs nothing.
+- **Not yet**: the resume brief (next-day continuation) and an explicit
+  remaining-requests ROI gate — today the 3-line budget plus the ledger are the
+  guardrails.
+
+### Session episode + resume brief (v0.7-A/D)
+
+- **episode** — at `session/disposed`, a zero-LLM digest of the event log: files
+  touched, commands run, failures that later recovered (`cmd×N→fixed`), written
+  commit subjects, and still-open todos. Written only when there is substance (a
+  bare `ls` session produces none); **never indexed**; the newest 20 per
+  workspace are kept and older ones archived.
+- **resume brief** — at the first assembly of a new session, the newest episode
+  from the last 7 days is injected as a `[RESUME]` line (≤240 chars) plus an
+  `engram_detail <id>` pointer, frozen with `[ENGRAM]`/`[RECALL]`. No recent
+  episode → no line; returning to a project does not need a re-briefing.
+- Settings: `episodeEnabled`, `resumeBriefEnabled` (both default on), plus
+  `episodeMaxPerWorkspace` / `resumeBriefMaxAgeDays` / `resumeBriefMaxChars`.
+
+### Entity pages + entity cards (v0.7-B)
+
+`esr_node` grows from a dot on the graph into a readable **page**: current
+truth, timeline, open loops, conflicts and relations for one thing
+(package / repo / service / bug…).
+
+- **Entity card (in the index, default on)** — `[ENGRAM]` renders ONE line for
+  each active entity (anchored memories or open tasks), at most 2:
+  `[ENT] dsh-engram · package · 3 mem · 1 open · truth: … #id · page: engram_detail(entity="dsh-engram")`
+  Those entities' flat memory lines are not repeated; other entities stay in a
+  single `nodes:` line.
+- **Full page (on drill)** — `engram_detail(entity="<name|id>")` returns
+  `truth / timeline / open loops / conflicts / relations`, with memory ids and
+  evidence gaps. A name with anchored memories but no registered node still
+  gets a virtual page.
+- Pure, deterministic: preference leads `truth`, superseded rows never enter
+  it, conflicts are de-duplicated by unordered pair (`#a vs #b`).
+- Settings `entityCardsEnabled` (default on); `entityCardsMax` /
+  `entityCardChars` tunable.
+
+### Revalidatable anchors + layered TTL (v0.7-C)
+
+"Treating the past as truth" is the strongest credibility critique against
+memory plugins. v0.7 gives file-backed memories an **anchor**:
+
+- on write, record `{path, mtimeMs, size, hash?}` (sha256 for files ≤256KB,
+  stat only for larger ones) in the memory row; relative paths resolve against
+  the workspace;
+- at session start a bounded revalidation (≤40 rows) tags changed/missing
+  sources `stale`; the `[ENGRAM]` index and entity `truth` stop showing them,
+  but nothing is deleted — `engram_detail` still returns them and the entity
+  timeline marks `· stale`;
+- **layered TTL**: `preference` / `procedure` permanent; `episode` 14 days;
+  `error` 60 days; everything else `expireDays` (180). Hard deletes still do not
+  exist — GC only archives.
+- Settings: `anchorRevalidate` (default on), `episodeTtlDays`, `errorTtlDays`.
+- **Not done**: the offline consolidation job (episode abstraction/merge) —
+  deterministic rules can only roll up, not abstract semantically; deferred
+  until v0.8 alongside playbooks, rather than shipping a lossy pseudo-merge.
+
 ## Auto-capture policy
 
 Capture is deterministic and offline — it only sees tool *results*, never the
@@ -908,12 +1039,13 @@ Defaults are token-conscious; override any key via the profile patch
     autoCapture: true        # zero-LLM tool-result capture
     autoCapturePreferences: true  # zero-LLM user-correction/preference capture (prefer/avoid/switch-to cues)
     preferenceCapturePerSession: 8  # per-session preference memory cap
-    boundaryPrune: false       # ResultPack L1: prune stale over-threshold tool results at todo/goal boundaries (default OFF)
+    boundaryPrune: true        # ResultPack L1: prune stale over-threshold tool results at todo/goal boundaries (default ON)
+    boundaryPruneAdaptive: true   # ledger veto: stop pruning this workspace once its net is negative (>=3 fires)
     boundaryPruneThresholdChars: 0   # threshold override (0 = host 8192; e.g. 2048 prunes earlier)
     boundaryPruneHeadChars: 0  # retained head under an override (0 = host 4096)
     boundaryPruneTailChars: 0  # retained tail under an override (0 = host 1024)
     boundaryPruneIdleCadence: 0   # request-count boundary for todo/goal-less sessions (0 = event-only)
-    resultPack: false          # ResultPack L2: archive pruned middles + result_recall paged recall (default OFF; needs L1 on)
+    resultPack: true           # ResultPack L2: archive pruned middles + result_recall paged recall (default ON; needs L1 on)
     resultPackRetentionDays: 7 # archive retention days (mtime sweep on the GC cadence)
     sessionSearch: true      # engram_recall may also FTS past sessions
     recallScope: workspace   # recall scope: "workspace" (strict isolation, default) | "global" (opt-in cross-workspace recall with [W:<ws>] origin markers)
@@ -932,8 +1064,17 @@ Defaults are token-conscious; override any key via the profile patch
     gcEnabled: true          # scheduled memory GC
     gcIntervalHours: 24      # sweep cadence
     gcStableRetentionDays: 120  # stable tasks leave [ESR] after this
-    gcReplacesCompaction: true # Context GC: take over DSH auto-compact (false = keep the default LLM summarizer)
+    gcReplacesCompaction: false # Context GC: take over DSH auto-compact (default off = host summarizer; true = opt-in pointers)
     gcNarrative: true        # scoped LLM narrative for un-provenanced turns; false = fully mechanical (zero LLM)
+    autoWebCompaction: false # rewrite stock web presets (default off; explicit opt-in, revert with the CLI)
+    jitEnabled: true         # JIT push: error recurrence / file focus / goal completion → one memory line
+    episodeEnabled: true     # session episode: zero-LLM digest at teardown (never indexed)
+    resumeBriefEnabled: true # resume brief: newest episode from the last 7 days at session start
+    entityCardsEnabled: true # entity card: one truth/tasks/drill line per active entity in the index
+    anchorRevalidate: true   # re-stat filePath anchors at session start (changed/missing → stale)
+    episodeTtlDays: 14       # episode TTL (preference/procedure permanent; error 60d; others expireDays)
+    ledgerEnabled: true      # Context Ledger: injected cost / prune savings / rewrite penalty → net tokens
+    ledgerCacheDiscount: 0.1 # cached-prefix price share; rewrites pay (1−discount)
     engramIndexOrder: 40    # systemPrompt section order (before tools band)
     esrOrder: 41
 ```
@@ -941,7 +1082,7 @@ Defaults are token-conscious; override any key via the profile patch
 ## Development
 
 ```sh
-npm test            # 291 tests: core + web API + GC + usage-observability + Context GC + ESR triggers + BoundaryPrune + preference learning (node:test)
+npm test            # 381 tests: core + web API + GC + usage-observability + Context GC + ESR triggers + BoundaryPrune + preference learning + Context Ledger + JIT push (node:test)
 npm run eval        # offline recall + structure benchmark (deterministic)
 npm run build:client
 ```