@logictan/dsh-verbatim-compaction 0.3.0 → 0.4.0

package/README.md

@@ -17,14 +17,16 @@
 ## 摘要长什么样
 
 ```
-## user (turn 37)
+## user
 （用户原话，逐字，永不改写）
 
-### assistant (turn 37)
+### assistant
+## Asked
+- 是否要用 FULL 而不是 INCREMENTAL
 ## Did
 - 实测 `auto_vacuum=FULL` 删除后自动缩容：19.61 MB → 0.01 MB
 ## Concluded
-- 选 FULL 而非 INCREMENTAL：回收发生在宿主自己的 DELETE 里，不受 busy_timeout=0 影响
+- 选 FULL：回收发生在宿主自己的 DELETE 里，不受 busy_timeout=0 影响
 ## Overturned
 - 先前认为必须显式执行 VACUUM 才能回收空间
   tools: bash×280 ssh_exec×44 web_search×6 (408830 bytes)
@@ -32,6 +34,8 @@
   wrote: /Volumes/LogicExt/Git/BookReader/src/video.js
 ```
 
+标题不带轮次号：日志里的 `turn` 号既不唯一也不是一轮一个（一个真实会话把两条请求都编号成 1），印出来就是会撒谎的标题。
+
 - **用户原话逐字**，不重写、不改写、不摘要。
 - **助手正文由提取块替代**，不并列保留——并列就是同一份内容写两遍。
 - 工具结果只留「工具名 × 次数 + 字节数」，内容丢弃。
@@ -51,7 +55,11 @@
 
 只抄正文救不了「正文为空」的轮次。实测一个 13878 事件的会话：1973 条 assistant 消息里只有 317 条带正文，其中一轮跑了 342 次工具调用却一个字没写，纯逐字摘要把它压成一行 `tools: bash×281`——活干了，记录没了。
 
-所以提取的材料是三段：用户请求（逐字）、助手正文（逐字）、工具轨迹（每条调用的 `description`，失败的调用另记一行首行错误）。模型输出固定的三节：`## Did` / `## Concluded` / `## Overturned`。
+所以提取的材料是三段：用户请求（逐字）、助手正文（逐字）、工具轨迹（每条调用的 `description`，失败的调用另记一行首行错误）。模型输出固定的四节：`## Asked` / `## Did` / `## Concluded` / `## Overturned`。
+
+空节直接删掉，不留 `(none)` 占位。实测一份真实提取块里 `## Asked` 和 `## Overturned` 都是 `(none)`，删掉后 312 → 271 字符；按 130 轮算，这些占位合计约 900 token 只是在记录「什么都没发生」。清理在代码里做（`pruneEmptySections`），不只依赖提示词——模型仍写了 `(none)` 也不会白花 token。判据是整节每一行去掉项目符号后都只是 `(none)` / `none` / `n/a` / `无` 之类的占位；`none of the tests passed`、`无法连接，改用代理` 这类真内容照原样保留。
+
+`## Asked` 记的是「这一轮向用户提了什么、给了哪些选项」。实测 123 条真人消息里 43 条（35%）不超过 12 个字符——`3`、`1`、`同意`、`做`、`继续`——而 46% 是在回答上一轮助手的提问。只逐字保留「3」而不保留它在回答什么问题，等于记下一个决定却没记下决定的是什么。
 
 模型用的是会话自己的路由（`session.requestHeader()?.config`），和宿主原生压缩同一个取法，不需要单独配置凭据。
 

package/lib/index.js

@@ -150,12 +150,64 @@ export function isErrorResult(event) {
     const message = messageOf(event);
     return message?.isError === true;
 }
+/**
+ * Drop headings whose body says nothing.
+ *
+ * The instruction asks the model to omit empty sections, but a model that writes
+ * `## Overturned\n- (none)` anyway must not cost tokens on every turn: 130 turns
+ * of that placeholder is ~900 tokens recording that nothing was overturned. So
+ * the shape is enforced here as well, not only requested in the prompt.
+ *
+ * A heading is dropped when every non-blank line under it, once its bullet is
+ * stripped, is a placeholder — `(none)`, `none`, `n/a`, `-`, `无`. A heading with
+ * any real content is kept verbatim.
+ *
+ * @param body - one slot's extracted text.
+ * @returns the same text with empty sections removed, or an empty string when
+ * nothing substantive remains.
+ */
+export function pruneEmptySections(body) {
+    const lines = body.split('\n');
+    const kept = [];
+    let pendingHeading;
+    let pendingBody = [];
+    const flush = () => {
+        if (pendingHeading === undefined) {
+            kept.push(...pendingBody);
+            pendingBody = [];
+            return;
+        }
+        const substantive = pendingBody.some((line) => {
+            const text = line.replace(/^\s*(?:[-*]|\d+\.)\s*/, '').trim();
+            if (text.length === 0)
+                return false;
+            return !/^(?:\(?none\)?|n\/a|na|-{1,2}|无|沒有|没有)$/i.test(text);
+        });
+        if (substantive)
+            kept.push(pendingHeading, ...pendingBody);
+        pendingHeading = undefined;
+        pendingBody = [];
+    };
+    for (const line of lines) {
+        if (/^#{1,6}\s/.test(line)) {
+            flush();
+            pendingHeading = line;
+            continue;
+        }
+        pendingBody.push(line);
+    }
+    flush();
+    return kept.join('\n').replace(/\n{3,}/g, '\n\n').trim();
+}
 /** Directive appended as the final user message of the extraction call. */
 const EXTRACT_INSTRUCTION = [
     'You are summarizing a batch of turns from a coding session for a compaction checkpoint.',
-    'Each turn is introduced by a line `=== TURN <n> ===` and contains the user request, the assistant prose (may be empty), and a trace of the tool calls it made (descriptions only) plus any results the log marked as errors.',
+    'Each item is introduced by a line `=== SLOT <key> (turn <n>) ===` and contains the user request, the assistant prose (may be empty), and a trace of the tool calls it made (descriptions only) plus any results the log marked as errors.',
+    '',
+    'For EVERY item, emit exactly one block, introduced by `=== SLOT <key> ===` using that same key. Under each heading write at most two terse lines. OMIT a heading entirely when it has no real information — do not write "(none)" and do not invent. An item where nothing substantive happened may be omitted altogether.',
     '',
-    'For EVERY turn, emit exactly one block, introduced by the same `=== TURN <n> ===` line, containing these three Markdown headings in order. Under each, at most two terse lines. If a section has no real information, write "(none)" — never invent.',
+    '## Asked',
+    '- the question(s) this turn put to the user, and the options it offered — the next user message is usually the answer',
     '',
     '## Did',
     '- what was actually done, concretely',
@@ -362,13 +414,26 @@ export function buildSections(events) {
  * @returns the same sections with `extracted` set where available.
  */
 function withExtractions(sections, extracted) {
-    return sections.map((section) => ({
-        turns: section.turns.map((turn) => {
-            const block = extracted.get(turn.turn);
+    return sections.map((section, sectionIndex) => ({
+        turns: section.turns.map((turn, turnIndex) => {
+            const block = extracted.get(turnKey(sectionIndex, turnIndex));
             return block === undefined ? turn : { ...turn, extracted: block };
         }),
     }));
 }
+/**
+ * Position-based key for one turn: `<section index>:<turn index>`.
+ *
+ * Not the turn number. `turn/start` is emitted per step in some sessions and only
+ * once in others, so the number in the log is neither unique nor one-per-request.
+ * A real session carried 2 user requests under a single `turn/start` (turn 1), so
+ * both sections were keyed `1`; the second extraction overwrote the first, and
+ * the digest printed the same block twice — once under a request about source
+ * refresh, once under 「模拟器使用硬件加速！」.
+ */
+function turnKey(sectionIndex, turnIndex) {
+    return `${sectionIndex}:${turnIndex}`;
+}
 /**
  * Render one section to digest text, truncating assistant prose to `capChars`.
  *
@@ -383,15 +448,14 @@ function withExtractions(sections, extracted) {
 function renderSection(section, capChars) {
     const lines = [];
     for (const turn of section.turns) {
-        const label = turn.turn === undefined ? '?' : String(turn.turn);
         if (turn.user.length > 0)
-            lines.push(`\n## user (turn ${label})\n${turn.user.join('\n')}`);
+            lines.push(`\n## user\n${turn.user.join('\n')}`);
         if (turn.extracted !== undefined) {
             let block = turn.extracted;
             if (capChars !== undefined && block.length > capChars) {
                 block = `${block.slice(0, capChars)}\n…(+${block.length - capChars} chars)`;
             }
-            lines.push(`### assistant (turn ${label})\n${block}`);
+            lines.push(`### assistant\n${block}`);
         }
         else {
             let body = turn.assistant.join('\n');
@@ -399,7 +463,7 @@ function renderSection(section, capChars) {
                 body = `${body.slice(0, capChars)}\n…(+${body.length - capChars} chars)`;
             }
             if (body.length > 0)
-                lines.push(`### assistant (turn ${label})\n${body}`);
+                lines.push(`### assistant\n${body}`);
         }
         if (turn.tools.size > 0) {
             const names = [...turn.tools.entries()]
@@ -653,17 +717,20 @@ async function extractBatch(ctx, agent, batch, signal) {
     const model = route?.model ?? agent.options.model;
     if (provider === undefined || model === undefined)
         return out;
+    // The label the model echoes back is the slot key, not the turn number: two
+    // requests can share a number, and a duplicate label would collapse two blocks
+    // into one entry.
     const parts = [];
-    for (const [turn, events] of batch) {
-        const prose = events
+    for (const slot of batch) {
+        const prose = slot.events
             .filter((event) => event.type === 'assistant/message')
             .map((event) => textOf(event))
             .filter((text) => text.length > 0)
             .join('\n');
-        const material = buildExtractionMaterial(events, prose);
+        const material = buildExtractionMaterial(slot.events, prose);
         if (material.length === 0)
             continue;
-        parts.push(`=== TURN ${turn} ===\n${material}`);
+        parts.push(`=== SLOT ${slot.key} (turn ${slot.turn ?? '?'}) ===\n${material}`);
     }
     if (parts.length === 0)
         return out;
@@ -675,17 +742,12 @@ async function extractBatch(ctx, agent, batch, signal) {
             messages: [
                 { role: 'user', content: [{ type: 'text', text: `${parts.join('\n\n')}\n\n${EXTRACT_INSTRUCTION}` }] },
             ],
-            // One block per turn, so the cap scales with the batch rather than being
-            // a per-turn 400 that a dozen turns would multiply past the digest ceiling.
             maxTokens: EXTRACT_TOKENS_PER_TURN * parts.length,
             sessionId: agent.session.id,
             purpose: 'compaction',
             ...(signal === undefined ? {} : { signal }),
         });
         for await (const chunk of stream) {
-            // Chunk kinds come from `@deepseek-ai/dsh-llm`'s BlockAssembler: text
-            // arrives as `text-delta`, and `reasoning-delta` is deliberately ignored —
-            // reasoning is 98.8% of a long session's output and is not the record.
             const delta = chunk;
             if (delta.type === 'text-delta' && typeof delta.text === 'string')
                 chunks.push(delta.text);
@@ -694,17 +756,15 @@ async function extractBatch(ctx, agent, batch, signal) {
     catch {
         return out;
     }
-    // Split the answer back per turn. A turn the model skipped simply keeps its
-    // prose, so a partial answer degrades per turn instead of losing the batch.
     const answer = chunks.join('').trim();
     if (answer.length === 0)
         return out;
-    const sections = answer.split(/^=== TURN (\d+) ===$/m);
-    for (let index = 1; index < sections.length; index += 2) {
-        const turn = Number(sections[index]);
-        const body = (sections[index + 1] ?? '').trim();
-        if (Number.isFinite(turn) && body.length > 0)
-            out.set(turn, body);
+    const blocks = answer.split(/^=== SLOT ([^\s(]+)[^=]*===$/m);
+    for (let index = 1; index < blocks.length; index += 2) {
+        const key = blocks[index];
+        const body = pruneEmptySections((blocks[index + 1] ?? '').trim());
+        if (key !== undefined && body.length > 0)
+            out.set(key, body);
     }
     return out;
 }
@@ -779,33 +839,47 @@ export default class VerbatimCompactionEngine extends BasicCompactionEngine {
      * @returns turn number to extracted block, omitting turns that yielded none.
      */
     async extractAll(source, agent, signal) {
-        const turns = new Map();
+        // Walk the same sections the renderer will use, so a key addresses the exact
+        // slot the block will be written into. Grouping by turn number instead lost
+        // blocks whenever the log reused a number.
+        const sections = buildSections(source);
+        const slots = [];
         let lastTurn;
-        for (const event of source) {
-            if (event.type === 'turn/start') {
-                const turn = turnOf(event);
-                if (turn !== undefined)
-                    lastTurn = turn;
-                continue;
+        let cursor = 0;
+        for (const [sectionIndex, section] of sections.entries()) {
+            for (const [turnIndex, turn] of section.turns.entries()) {
+                // Collect this turn's events by walking the source in order; the sections
+                // were built from the same array, so the cursor never rewinds.
+                const collected = [];
+                while (cursor < source.length) {
+                    const event = source[cursor];
+                    cursor += 1;
+                    if (event === undefined)
+                        break;
+                    if (event.type === 'turn/start') {
+                        const turnNumber = turnOf(event);
+                        if (turnNumber !== undefined)
+                            lastTurn = turnNumber;
+                        collected.push(event);
+                        continue;
+                    }
+                    collected.push(event);
+                    if (event.type === 'user/message' || event.type === 'assistant/message') {
+                        if (!isInjection(event) && !isCheckpoint(event))
+                            break;
+                    }
+                }
+                slots.push({ key: turnKey(sectionIndex, turnIndex), turn: turn.turn, events: collected });
             }
-            const turn = turnOf(event) ?? lastTurn;
-            if (turn === undefined)
-                continue;
-            const bucket = turns.get(turn);
-            if (bucket === undefined)
-                turns.set(turn, [event]);
-            else
-                bucket.push(event);
         }
-        const ordered = [...turns.entries()].sort((left, right) => left[0] - right[0]);
         const extracted = new Map();
-        for (let index = 0; index < ordered.length; index += TURNS_PER_CALL) {
+        for (let index = 0; index < slots.length; index += TURNS_PER_CALL) {
             if (signal?.aborted === true)
                 break;
-            const batch = ordered.slice(index, index + TURNS_PER_CALL);
+            const batch = slots.slice(index, index + TURNS_PER_CALL);
             const blocks = await extractBatch(this.ctx, agent, batch, signal);
-            for (const [turn, block] of blocks)
-                extracted.set(turn, block);
+            for (const [key, block] of blocks)
+                extracted.set(key, block);
         }
         return extracted;
     }

package/lib/types/index.d.ts

@@ -80,6 +80,23 @@ export interface SummaryResult {
  * @returns true only when the log itself marks the result as an error.
  */
 export declare function isErrorResult(event: SessionEvent): boolean;
+/**
+ * Drop headings whose body says nothing.
+ *
+ * The instruction asks the model to omit empty sections, but a model that writes
+ * `## Overturned\n- (none)` anyway must not cost tokens on every turn: 130 turns
+ * of that placeholder is ~900 tokens recording that nothing was overturned. So
+ * the shape is enforced here as well, not only requested in the prompt.
+ *
+ * A heading is dropped when every non-blank line under it, once its bullet is
+ * stripped, is a placeholder — `(none)`, `none`, `n/a`, `-`, `无`. A heading with
+ * any real content is kept verbatim.
+ *
+ * @param body - one slot's extracted text.
+ * @returns the same text with empty sections removed, or an empty string when
+ * nothing substantive remains.
+ */
+export declare function pruneEmptySections(body: string): string;
 /**
  * Build the compact per-turn trace fed to the summarizer.
  *
@@ -156,7 +173,7 @@ export declare function buildSections(events: readonly SessionEvent[]): Section[
  * ceiling; the tiers exist for the turns where extraction was unavailable and
  * the prose is all that remains.
  */
-export declare function renderDigest(events: readonly SessionEvent[], budgetTokens: number, extracted?: ReadonlyMap<number | undefined, string>): string;
+export declare function renderDigest(events: readonly SessionEvent[], budgetTokens: number, extracted?: ReadonlyMap<string, string>): string;
 /**
  * The surface seqs this compaction shadows, recovered from the summarization
  * input's message count.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@logictan/dsh-verbatim-compaction",
-  "version": "0.3.0",
+  "version": "0.4.0",
   "description": "Deterministic compaction backend that keeps conversation prose verbatim in the checkpoint instead of re-summarizing it: user and assistant text survive word for word, tool results collapse to one-line pointers, reasoning is never read, and no model call is made.",
   "type": "module",
   "main": "lib/index.js",