dsh-continual-evolve 0.15.2 → 0.15.4

package/README.md

@@ -7,7 +7,7 @@ English | [中文](README.zh.md)
 [![CI](https://github.com/ZK-Andy/dsh-continual-evolve/actions/workflows/ci.yml/badge.svg)](https://github.com/ZK-Andy/dsh-continual-evolve/actions/workflows/ci.yml)
 [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
 [![Node](https://img.shields.io/badge/node-%5E22.19%20%7C%7C%20%3E%3D24-339933)](package.json)
-[![Tests](https://img.shields.io/badge/tests-190%20passing-brightgreen)]()
+[![Tests](https://img.shields.io/badge/tests-199%20passing-brightgreen)]()
 [![Coverage · statements](https://img.shields.io/badge/coverage_statements-97%25-brightgreen)]()
 [![Coverage · branches](https://img.shields.io/badge/coverage_branches-91%25-green)]()
 [![Coverage · functions](https://img.shields.io/badge/coverage_functions-100%25-brightgreen)]()
@@ -22,7 +22,7 @@ Reusable experience gathered in one session (user preferences, pitfalls, project
 
 1. **Session-start injection** — the plugin registers a single system-prompt section: the workspace's memory index as a store query (hooks only, whole-line truncated over budget, ordered feedback > user > reference), the central-library path, and a three-line guide. The section is computed once per session and reused byte-for-byte, keeping the system prompt stable and the prompt cache warm; an empty store costs zero tokens.
 2. **Two tools, one writer** — `memory_write` (create/update/delete) is the explicit path for "remember …" / "forget …": every proposal passes the code gates (id shape, type enum, the feedback Why/How contract, secret screening carried over from v1, 64KB body cap) inside one transaction. `memory_read` fetches bodies by id and searches by keyword — whitespace-separated terms are AND-combined, with literal substring matching for short terms and whenever the trigram index misses; the injected index carries hooks only. On hosts without the tools service the plugin degrades to "tell the user" phrasing and stays functional.
-3. **Proposal-based extraction** — the only background automation, audited in the `extraction_log`: every `agent/turn-stopping` schedules that session's boundary, with state kept **per session** (a run in flight keeps only its newest boundary — coalescing, not queueing; `compaction/start` and session close drain it) and **no idle timer**. A run reads the session increment since its cursor, feeds FTS-similar existing memories to one LLM call, and the model's structured proposal lands through the same gates; failures and rejections never advance the cursor. Internal agents, empty increments, turns without real user prose, and turns that already carried an explicit `memory_write` are mechanically skipped — and every skip is ledgered.
+3. **Proposal-based extraction** — the only background automation, audited in the `extraction_log`: every `agent/turn-stopping` schedules that session's boundary, with state kept **per session** (a run in flight keeps only its newest boundary — coalescing, not queueing; `compaction/start` and session close drain it) and **no idle timer**. A run reads the session increment since its cursor, feeds FTS-similar existing memories to one LLM call, and the model's structured proposal lands through the same gates; failures and rejections never advance the cursor. Internal agents, empty increments, turns without real user prose, and turns that already carried an explicit `memory_write` are mechanically skipped — and every skip is ledgered, consecutive quiet turns of one session as a single row carrying how many wake-ups it stands for.
 4. **Rollback is a ledger, not a mechanism** — every mutation lands an `extraction_log` row with full before/after snapshots; undo means rewriting the `before` values. A hygiene pass rides on every applied run: orphan FTS rows are dropped and secret-bearing rows are quarantined (hidden, never deleted). The read-only card surfaces quarantine anomalies.
 5. **Read-only card** — mounted on the plugin's page in the official plugin manager (the `plugins.bundle.config` slot; hosts without it silently degrade to no card), shaped like the market's own settings card and aligned with ZCode's Settings → Memory viewer: a dropdown scope selector over the store's workspace partitions (opening on the host's current workspace), a search box, per-entry relative updated times, click-to-preview bodies (5 MiB cap), and patrol warnings. Localized through the host's `locale/*.json` metadata channel; host theme tokens, the `Button` primitive, and zh/en copy that tracks the DSH language. The card is a request-time projection of the database and has no edit path.
 
@@ -71,7 +71,7 @@ Profile patch example:
 
 ```bash
 pnpm install && pnpm build   # deps + clean build -> lib/
-pnpm test                    # vitest (190 tests)
+pnpm test                    # vitest (199 tests)
 pnpm test:coverage           # v8 coverage, CI-enforced thresholds
 pnpm lint                    # oxlint src test client
 pnpm check:pack              # published-artifact consistency (also runs on prepack)

package/README.zh.md

@@ -7,7 +7,7 @@
 [![CI](https://github.com/ZK-Andy/dsh-continual-evolve/actions/workflows/ci.yml/badge.svg)](https://github.com/ZK-Andy/dsh-continual-evolve/actions/workflows/ci.yml)
 [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
 [![Node](https://img.shields.io/badge/node-%5E22.19%20%7C%7C%20%3E%3D24-339933)](package.json)
-[![Tests](https://img.shields.io/badge/tests-190%20passing-brightgreen)]()
+[![Tests](https://img.shields.io/badge/tests-199%20passing-brightgreen)]()
 [![Coverage · statements](https://img.shields.io/badge/coverage_statements-97%25-brightgreen)]()
 [![Coverage · branches](https://img.shields.io/badge/coverage_branches-91%25-green)]()
 [![Coverage · functions](https://img.shields.io/badge/coverage_functions-100%25-brightgreen)]()
@@ -22,7 +22,7 @@ Agent 每个会话积累的可复用经验（用户偏好、踩坑教训、项
 
 1. **开局注入** —— 插件只注册一个 system-prompt section：注入本工作区的记忆索引（查库结果，只含钩子，超预算按整行截断，feedback > user > reference 排序）、中央库路径、三行指南。该 section 每会话只算一次并逐字节复用，system prompt 稳定、prompt cache 持续命中；空 store 零 token。
 2. **两个工具，一个写者** —— `memory_write`（create/update/delete）承接显式的"记住……"/"忘掉……"：每条提案过代码门禁（id 规范、type 枚举、feedback 的 Why/How 契约、v1 移植的 secret 筛查、64KB 正文上限）后在一个事务内落盘。`memory_read` 按 id 取正文、按关键词检索——空格分隔的关键词取 AND，短词与 trigram 未命中时回退字面子串匹配；注入索引只有钩子。宿主没有工具服务时降级为"告诉用户"措辞，注入不受影响。
-3. **提案制提取** —— 插件唯一的后台自动化，全程 `extraction_log` 可对账：每个 `agent/turn-stopping` 调度**该会话自己**的边界，状态按会话隔离（运行中只保留最新边界——合并而非排队；`compaction/start` 与会话收尾触发 drain），**没有空闲计时器**。一次运行读取该会话游标之后的轨迹增量，用 FTS 取相似既有记忆喂给一次 LLM 调用，模型的结构化提案过同一套门禁落盘；失败与被拒不推游标，下次触发重试。内部 agent、空增量、无真实用户文本的轮次、当轮已有显式 `memory_write` 的轮次机械跳过——每次跳过都记账。
+3. **提案制提取** —— 插件唯一的后台自动化，全程 `extraction_log` 可对账：每个 `agent/turn-stopping` 调度**该会话自己**的边界，状态按会话隔离（运行中只保留最新边界——合并而非排队；`compaction/start` 与会话收尾触发 drain），**没有空闲计时器**。一次运行读取该会话游标之后的轨迹增量，用 FTS 取相似既有记忆喂给一次 LLM 调用，模型的结构化提案过同一套门禁落盘；失败与被拒不推游标，下次触发重试。内部 agent、空增量、无真实用户文本的轮次、当轮已有显式 `memory_write` 的轮次机械跳过——每次跳过都记账，同一会话连续的空转轮次合成一行并累计唤醒次数。
 4. **回滚靠账本，不靠机制** —— 每次变更落一条带 before/after 全文快照的 `extraction_log` 行；undo 就是取 before 重写。每次应用后随跑一次巡检：孤儿 FTS 行清除、含密钥条目隔离（隐藏，不物理删除）。只读卡片显形巡检异常。
 5. **只读卡片** —— 挂在官方插件管理的本 bundle 页面（`plugins.bundle.config` slot，老宿主自动降级为无卡片）：形态对齐市场自家设置卡，内容对齐 ZCode 设置的记忆选项卡——下拉作用域选择器列出库里的工作区分区（默认选中宿主当前工作区）、搜索框、每条记忆的相对更新时间、点击行内展开正文预览（5 MiB 上限）、巡检异常警告。显示名与描述走宿主 `locale/*.json` 通道双语；宿主主题 token、`Button` 原语、中英文案跟随 DSH 语言。卡片是数据库的请求时投影，没有编辑路径。
 
@@ -71,7 +71,7 @@ profile patch 示例：
 
 ```bash
 pnpm install && pnpm build   # 依赖 + 干净构建 -> lib/
-pnpm test                    # vitest（190 例）
+pnpm test                    # vitest（199 例）
 pnpm test:coverage           # v8 覆盖率，CI 强制阈值
 pnpm lint                    # oxlint src test client
 pnpm check:pack              # 发布产物一致性（prepack 也会跑）

package/lib/extraction.d.ts

@@ -34,8 +34,11 @@ export interface RunOptions {
     incrementMaxChars?: number;
 }
 /**
- * Run one extraction pass for a target. Never throws: every terminal path
- * is either a landed application, a ledgered skip, or a ledgered failure.
+ * Run one extraction pass for a target. Never throws unless the ledger
+ * itself is unwritable: every terminal path is either a landed application,
+ * a ledgered skip, or a ledgered failure — anything that escapes the
+ * per-stage handling inside lands a failed `unexpected-error` ledger row
+ * here, so no failure ends as a scheduler warn without an account trace.
  */
 export declare function runExtraction(deps: ExtractionDeps, target: ExtractionTarget, opts: RunOptions): Promise<ExtractionResult>;
 /** The event wiring surface (duck-typed cordis context). */

package/lib/extraction.js

@@ -1,11 +1,59 @@
 import { boundaryCursorOf, evaluateSlice, eventsAfterCursor, isInternalAgent, seqOfCursor, serializeIncrement } from "./extraction-surface.js";
 import { EXTRACTION_SYSTEM_PROMPT, extractionPrompt, manifestLineOf, parseExtractionAnswer } from "./extraction-prompt.js";
 import { cwdOf } from "./memory-section.js";
+/** Head length of a failed run's raw answer kept in the ledger `files` column. */
+const FAILURE_RAW_MAX = 2048;
+/** Head length of an unexpected error's message/stack trace kept in the ledger `files` column. */
+const UNEXPECTED_ERROR_RAW_MAX = 2048;
+/** Head length of an error message embedded in a ledger `skip_reason`. */
+const ERROR_REASON_MAX = 300;
+/** Split an unexpected throw into a ledger-sized reason head and file body. */
+function errorText(error) {
+    const message = error instanceof Error && error.message.length > 0 ? error.message : String(error);
+    const stack = error instanceof Error && typeof error.stack === "string" ? error.stack : message;
+    return { message, detail: `${message}\n${stack}`.slice(0, UNEXPECTED_ERROR_RAW_MAX) };
+}
 /**
- * Run one extraction pass for a target. Never throws: every terminal path
- * is either a landed application, a ledgered skip, or a ledgered failure.
+ * Run one extraction pass for a target. Never throws unless the ledger
+ * itself is unwritable: every terminal path is either a landed application,
+ * a ledgered skip, or a ledgered failure — anything that escapes the
+ * per-stage handling inside lands a failed `unexpected-error` ledger row
+ * here, so no failure ends as a scheduler warn without an account trace.
  */
 export async function runExtraction(deps, target, opts) {
+    const started = Date.now();
+    try {
+        return await runExtractionInner(deps, target, opts);
+    }
+    catch (error) {
+        const { message, detail } = errorText(error);
+        const reason = `unexpected-error: ${message.slice(0, ERROR_REASON_MAX)}`;
+        try {
+            deps.store.logSkipped({
+                sessionId: target.sessionId,
+                workspaceId: target.workspaceId,
+                trigger: opts.trigger,
+                skipReason: reason,
+                status: "failed",
+                model: target.model,
+                durationMs: Date.now() - started,
+                files: detail,
+            });
+        }
+        catch (ledgerError) {
+            // The ledger itself is down — nothing left to write to. Chain
+            // both errors so the scheduler's last-resort row keeps the
+            // triggering failure, not just the ledger fault.
+            throw new Error(`extraction failed (${message}); failure ledger write also failed (${errorText(ledgerError).message})`);
+        }
+        return { status: "failed", skipReason: reason };
+    }
+}
+/**
+ * The pipeline body: one bounded increment of one session, one LLM call, one
+ * gated transactional application, one ledger row, one cursor.
+ */
+async function runExtractionInner(deps, target, opts) {
     const { store } = deps;
     const ledgerBase = {
         sessionId: target.sessionId,
@@ -35,8 +83,14 @@ export async function runExtraction(deps, target, opts) {
         all = Array.isArray(snapshot.events) ? snapshot.events : [];
     }
     catch (error) {
-        store.logSkipped({ ...ledgerBase, skipReason: "surface-read-failed", status: "failed" });
-        return { status: "failed", skipReason: `surface read failed: ${error instanceof Error ? error.message : String(error)}` };
+        const { message, detail } = errorText(error);
+        try {
+            store.logSkipped({ ...ledgerBase, skipReason: "surface-read-failed", status: "failed", files: detail });
+        }
+        catch (ledgerError) {
+            throw new Error(`surface read failed (${message}); failure ledger write also failed (${errorText(ledgerError).message})`);
+        }
+        return { status: "failed", skipReason: `surface read failed: ${message}` };
     }
     const events = eventsAfterCursor(all, cursor === undefined ? undefined : `seq:${cursor}`);
     const slice = evaluateSlice({ events, internalAgent: false });
@@ -70,13 +124,21 @@ export async function runExtraction(deps, target, opts) {
         usage = answer.usage;
     }
     catch (error) {
-        store.logSkipped({ ...ledgerBase, skipReason: "llm-failed", status: "failed", durationMs: Date.now() - started });
-        return { status: "failed", skipReason: `llm call failed: ${error instanceof Error ? error.message : String(error)}` };
+        const { message, detail } = errorText(error);
+        try {
+            store.logSkipped({ ...ledgerBase, skipReason: "llm-failed", status: "failed", durationMs: Date.now() - started, files: detail });
+        }
+        catch (ledgerError) {
+            throw new Error(`llm call failed (${message}); failure ledger write also failed (${errorText(ledgerError).message})`);
+        }
+        return { status: "failed", skipReason: `llm call failed: ${message}` };
     }
     const parsed = parseExtractionAnswer(text);
     if (parsed.decision === "error") {
         // A malformed answer is a failed run: the increment stays unconsumed
-        // and the next trigger retries it.
+        // and the next trigger retries it. The answer's head rides in the
+        // ledger row — the stream is gone after this, so without it the
+        // failure is undiagnosable after the fact.
         store.logSkipped({
             ...ledgerBase,
             skipReason: parsed.reason,
@@ -84,6 +146,7 @@ export async function runExtraction(deps, target, opts) {
             model: target.model,
             usage,
             durationMs: Date.now() - started,
+            files: text.slice(0, FAILURE_RAW_MAX),
         });
         return { status: "failed", skipReason: parsed.reason };
     }
@@ -208,7 +271,26 @@ export function createExtractionScheduler(host, deps) {
             provider: target.provider,
             model: target.model,
         }, { trigger }).catch((error) => {
-            host.logger("continual-evolve").warn(`extraction run failed unexpectedly: ${error instanceof Error ? error.message : String(error)}`);
+            // Last resort: runExtraction ledgers its own unexpected failures,
+            // so this fires only when even that failed (the ledger write
+            // threw) — retry one ledger row here before the warn, so the
+            // failure still has an account trace.
+            const { message, detail } = errorText(error);
+            try {
+                deps.store.logSkipped({
+                    sessionId: target.sessionId,
+                    workspaceId: cwdOf(target.agent),
+                    trigger,
+                    skipReason: `unexpected-error: ${message.slice(0, ERROR_REASON_MAX)}`,
+                    status: "failed",
+                    model: target.model,
+                    files: detail,
+                });
+            }
+            catch {
+                // The ledger is down — the warn below is the last resort.
+            }
+            host.logger("continual-evolve").warn(`extraction run failed unexpectedly: ${message}`);
         }).finally(() => {
             slot.running = false;
             if (slot.pending !== null) {

package/lib/store.d.ts

@@ -30,7 +30,12 @@ export interface MemoryRecord {
     createdAt: string;
     updatedAt: string;
 }
-/** One ledger row (audit + rollback via embedded before/after snapshots). */
+/**
+ * One ledger row (audit + rollback via embedded before/after snapshots).
+ * Consecutive mechanical skips of the same kind share one row, whose
+ * `occurrences` column counts the wake-ups it stands for (`1` for everything
+ * else); the interface never sets it — the store owns that column.
+ */
 export interface LedgerEntry {
     runId?: string | undefined;
     sessionId?: string | undefined;
@@ -42,7 +47,7 @@ export interface LedgerEntry {
     durationMs?: number | undefined;
     model?: string | undefined;
     usage?: unknown;
-    /** Applied/attempted mutations: `{id, action, before?, after?, sourceSeqs}`. */
+    /** Applied/attempted mutations: `{id, action, before?, after?, sourceSeqs}`; a parse-failed run instead carries the raw answer's head. */
     files?: unknown;
 }
 /** Workspace rows the card's selector lists. */

package/lib/store.js

@@ -26,6 +26,15 @@ export const MD_IMPORT_STATE_KEY = "md-imported";
 export const GLOBAL_WORKSPACE = "";
 /** Preview size cap for the card's content endpoint (ZCode's viewer uses 5 MiB). */
 export const MEMORY_FILE_PREVIEW_LIMIT = 5 * 1024 * 1024;
+/** Ledger column counting the wake-ups one coalesced row stands for. */
+const LEDGER_OCCURRENCES = "occurrences";
+/**
+ * Skips a busy-but-quiet session repeats verbatim, turn after turn: the
+ * ledger keeps one counted row instead of one row per wake-up. Fault-shaped
+ * skips (`no-route`, `no-surface`, `no-workspace`) and every failure stay one
+ * row per occurrence — a persistent fault deserves a timeline, not a counter.
+ */
+const COALESCED_SKIP_REASONS = new Set(["no-user-prose", "no-new-events"]);
 const DDL = `
 PRAGMA journal_mode=WAL;
 PRAGMA busy_timeout=5000;CREATE TABLE IF NOT EXISTS memories (
@@ -58,7 +67,7 @@ CREATE TABLE IF NOT EXISTS extraction_log (
   trigger TEXT CHECK (trigger IN ('turn-debounce','compaction','close-drain','explicit','import')),
   decision TEXT, skip_reason TEXT,
   status TEXT, duration_ms INTEGER, model TEXT, usage TEXT,
-  files TEXT
+  files TEXT, occurrences INTEGER NOT NULL DEFAULT 1
 );
 CREATE TABLE IF NOT EXISTS state (
   workspace_id TEXT NOT NULL, key TEXT NOT NULL, value TEXT NOT NULL,
@@ -119,6 +128,7 @@ export async function openMemoryStore(explicitPath) {
         throw new StoreUnavailableError(`node:sqlite unavailable (Node ≥ 22.5 required) — memory store disabled: ${error instanceof Error ? error.message : String(error)}`);
     }
     database.exec(DDL);
+    ensureLedgerOccurrences(database);
     const stmts = {
         get: database.prepare("SELECT * FROM memories WHERE workspace_id = ? AND id = ?"),
         list: database.prepare(`SELECT * FROM memories WHERE workspace_id = ? AND status = 'active' ${LIST_ORDER}`),
@@ -137,6 +147,9 @@ export async function openMemoryStore(explicitPath) {
             "WHERE status = 'active' AND workspace_id <> '' GROUP BY workspace_id ORDER BY ts DESC LIMIT 1"),
         ledger: database.prepare("INSERT INTO extraction_log (run_id, ts, session_id, workspace_id, trigger, decision, skip_reason, status, duration_ms, model, usage, files) " +
             "VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"),
+        latestSkip: database.prepare("SELECT rowid AS rid, skip_reason AS reason, status FROM extraction_log " +
+            "WHERE workspace_id IS ? AND session_id IS ? ORDER BY ts DESC, rowid DESC LIMIT 1"),
+        bumpSkip: database.prepare(`UPDATE extraction_log SET ts = ?, ${LEDGER_OCCURRENCES} = ${LEDGER_OCCURRENCES} + 1 WHERE rowid = ?`),
         prune: database.prepare("DELETE FROM extraction_log WHERE ts < ?"),
         ftsOrphans: database.prepare("SELECT f.rowid AS rid FROM memories_fts f LEFT JOIN memories m ON m.workspace_id = f.workspace_id AND m.id = f.id WHERE m.id IS NULL"),
         ftsDrop: database.prepare("DELETE FROM memories_fts WHERE rowid = ?"),
@@ -161,6 +174,27 @@ export async function openMemoryStore(explicitPath) {
         });
         return stmt.all(workspaceId, ...patterns, limit).map(recordOf);
     };
+    /**
+     * Fold a repeat of the previous row's mechanical skip into that row: the
+     * quiet turns of one session are one cursor position, so they are one
+     * ledger row carrying how many wake-ups it stands for. Only a
+     * *consecutive* repeat coalesces — anything in between (a model decision,
+     * an applied batch, a different reason, another session) starts a new row,
+     * which is exactly what "the same cursor position" means here. The bumped
+     * row takes the newest `ts`, so recency queries still see the session as
+     * active.
+     */
+    const coalesceSkip = (entry) => {
+        if (entry.status !== "skipped" || entry.skipReason === undefined || !COALESCED_SKIP_REASONS.has(entry.skipReason)) {
+            return false;
+        }
+        const latest = stmts.latestSkip.get(entry.workspaceId ?? null, entry.sessionId ?? null);
+        if (latest === undefined || latest.status !== "skipped" || latest.reason !== entry.skipReason) {
+            return false;
+        }
+        stmts.bumpSkip.run(new Date().toISOString(), latest.rid);
+        return true;
+    };
     const store = {
         path,
         get(workspaceId, id) {
@@ -190,6 +224,9 @@ export async function openMemoryStore(explicitPath) {
             return applyProposals({ database, now: () => new Date().toISOString() }, ctx, proposals).outcomes;
         },
         logSkipped(entry) {
+            if (coalesceSkip(entry)) {
+                return;
+            }
             writeLedger(stmts.ledger, entry);
         },
         cursor(workspaceId, sessionId) {
@@ -255,6 +292,18 @@ export async function openMemoryStore(explicitPath) {
     };
     return store;
 }
+/**
+ * Add the ledger's `occurrences` column to a store created before coalescing
+ * existed. The DDL only covers fresh files, and the plugin is the sole writer
+ * of an existing user database — the one migration the ledger needs.
+ */
+function ensureLedgerOccurrences(database) {
+    const columns = database.prepare("SELECT name FROM pragma_table_info('extraction_log')").all();
+    if (columns.some((column) => column.name === LEDGER_OCCURRENCES)) {
+        return;
+    }
+    database.exec(`ALTER TABLE extraction_log ADD COLUMN ${LEDGER_OCCURRENCES} INTEGER NOT NULL DEFAULT 1`);
+}
 /** JSON-encode the optional ledger fields and insert the row (never throws past a warn). */
 function writeLedger(stmt, entry) {
     stmt.run(entry.runId ?? null, new Date().toISOString(), entry.sessionId ?? null, entry.workspaceId ?? null, entry.trigger, entry.decision ?? null, entry.skipReason ?? null, entry.status ?? null, entry.durationMs ?? null, entry.model ?? null, entry.usage === undefined ? null : JSON.stringify(entry.usage), entry.files === undefined ? null : JSON.stringify(entry.files));

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dsh-continual-evolve",
-  "version": "0.15.2",
+  "version": "0.15.4",
   "description": "ZCode-style workspace memory for DeepSeek Harness: one central SQLite store partitioned by workspace, index injection, and sole-writer gated writes.",
   "license": "MIT",
   "repository": {