dsh-live-compact 1.1.2 → 1.1.4

package/README.md

@@ -14,13 +14,13 @@
 headless：
 
 ```sh
-dsh plugin --profile headless add dsh-live-compact@1.1.2
+dsh plugin --profile headless add dsh-live-compact@1.1.4
 ```
 
 网页端：
 
 ```sh
-dsh plugin --profile web add dsh-live-compact@1.1.2
+dsh plugin --profile web add dsh-live-compact@1.1.4
 ```
 
 想先装到本机看版本：
@@ -30,6 +30,20 @@ npm install -g dsh-live-compact
 dsh-live-compact --version
 ```
 
+## 1.1.4
+
+- 修复 #34：空闲维护的大段 catch-up 改为分批让出事件循环；Windows 阴影文件短暂忙时按 8 次上限和退避自动重试，永久失败保持 fail-closed，不阻塞轮次结束返回。
+- 游标和 retryable 结果按会话缓存，避免长历史重复扫描；工具结果、raw/meta 字节与 SHA256 在重启恢复后继续校验。
+- 本地门禁 `npm run test:gate` 通过 508/508；Windows 2022 x64 与 Windows 11 ARM 的标准用户长会话验收均通过，含 29 轮、进程重启边界和 15,001 事件/356 个工具结果的双阶段捕获恢复。
+
+## 1.1.3
+
+- 修复 #34：工具结果事件不再同步执行 shadow/ACL 文件操作，改为事件循环后排队并按 toolCallId 去重；维护开始前会等待已排队捕获，`captureNew` 仍负责补抓和重试。
+- 这条路径避免在模型收到工具结果前被同步磁盘操作卡住；权限失败仍然只拒绝备份，不让宿主 fatal 退出。
+- Windows hosted 验收：Windows 11 ARM、Windows 2022 x64 的 Node 22/24 core，以及三套 DSH 0.2.0/0.2.1 普通用户 runtime 均通过；官方 Desktop 普通用户安装、启用、设置保存和重启读取在 Windows 11 ARM 与 Windows 2022 均通过。
+- 同一候选又做了 29 轮持久会话回放：Windows 2022 用时 14 分 05 秒，Windows 11 ARM 用时 17 分 08 秒，含第 21 轮进程重启边界和 58 次工具结果捕获。原报告的 93M/第 29 轮冻结没有在这组受控回放中重现；Node 24 首次双进程 cursor 测试的 299/300 竞态重跑通过，作为后续观察项保留。
+- Config、验证/头脑风暴模式和自动主题分类继续后置，不改变 1.1.2 的小切默认参数。
+
 ## 1.1.2
 
 - 常用压缩参数可以在 DSH 插件设置里改，保留原有环境变量用法。
@@ -43,7 +57,7 @@ dsh-live-compact --version
 
 ### Windows 备份权限
 
-Windows 双用户实测（Actions run `42367713007`，Windows 11 Enterprise ARM / Server 2022 x64）确认：1.1.1 的 `/inheritance:r /grant:r` 会留下其他主体的显式 ACE，第二个普通用户仍能读取备份。1.1.2 改为完整替换 DACL。本次运行时代码修改后，最终候选仍须重新完成原生 Windows 与 Desktop 验收，结果随最终候选记录。
+Windows 双用户实测（Actions run `42367713007`，Windows 11 Enterprise ARM / Server 2022 x64）确认：1.1.1 的 `/inheritance:r /grant:r` 会留下其他主体的显式 ACE，第二个普通用户仍能读取备份。1.1.2 改为完整替换 DACL。1.1.3 候选又通过了 hosted Windows core/runtime、官方 Desktop 普通用户安装与设置保存/重启，以及 29 轮持久会话回放；原报告的 93M/第 29 轮冻结未在受控回放中重现，慢存储、超大历史和共享目录压力仍属于后续观察范围。
 
 权限写入改为固定的 `%SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe`（Windows PowerShell 5.1 / .NET Framework），使用 `-NoProfile -NonInteractive -EncodedCommand`。脚本固定内嵌在已打包的 `shadow.js`，路径、SID、文件类型通过子进程环境传递，不拼进脚本。当前用户 SID 仍由固定 System32 `whoami.exe` 每进程获取一次，脚本再对照自身 Windows 身份。新建受保护的 DACL，只允许当前用户、SYSTEM、Administrators 完全控制；目录 ACE 带 OI/CI，直接修正的文件 ACE 不带继承标志。只修改 DACL，不提权，不改 owner、group 或 SACL，不使用临时宽松 ACL。写后读取实际 DACL，核验保护位、完整 ACE 数量、SID、Allow、权限及继承标志，收到确认才缓存。工具失败、超时、SID 非法或读回不符都会熔断本进程后续备份，不回退到 chmod。
 
@@ -75,17 +89,19 @@ Windows 双用户实测（Actions run `42367713007`，Windows 11 Enterprise ARM
 
 这轮连的是插件版。10 组同一套 31 轮填日志，组内串行，两个模型同时开。原版 DSH 压缩对比留到 1.1.2。Coder 390 27B 这轮没跑，后补。测评图在下面，也在 [v1.1.1](https://github.com/Tranquillitatis-Lab/dsh-live-compact/releases/tag/v1.1.1)。
 
+下面三张图沿用已完成的 1.1.1 实测数据；1.1.2 只更新运行时修复和门禁，图标题按当前发布版本标注。
+
 看图时，先看日志题答对多少，再看首步缓存和压缩节省。首步缓存取每轮第一个 `step_end`，把命中 token 加起来，再除以这些请求的总输入 token。压缩节省是 `spine-cuts.jsonl` 里 `saved` 的合计。压缩轮次耗时是发生压缩的各轮总耗时之和，压缩操作本身为毫秒级。任务墙钟是从第 1 轮开始到第 31 轮结束的实际用时。结束上下文取最后一轮的 `totalTokens`，包含此前保留下来的内容。
 
-DS V4.1 Flash 跑了 10 组，填日志 230/230，跳过 0。首步缓存 98.64%，压缩节省 65 180 token，总用时 36 分 18 秒。多数组结束在 3.5 万到 4.6 万 token。第 1 组读了 25 次全文，4 次压缩共节省 21 493 token，结束时仍有 282 871。第 2 组几乎只用 bash，只读了一次全文，未触发压缩，结束时是 37 781。
+DS V4.1 Flash 跑了 10 组，填日志 230/230，跳过 0。首步缓存 98.64%，压缩节省 65 180 token，总墙钟 36 分 18 秒。多数组结束在 3.5 万到 4.6 万 token。第 1 组读了 25 次全文，4 次压缩共节省 21 493 token，结束时仍有 282 871。第 2 组几乎只用 bash，只读了一次全文，未触发压缩，结束时是 37 781。图中的“含切割轮次墙钟”是整轮模型、工具和网络等待，不是插件小切耗时；小切操作本身为毫秒级。
 
-GLM 5.3 Flash 用了同一个插件、同一套题。有 7 组填日志达到 23/23。第 5、6 组全程连接失败。第 7 组前段连接失败，完成 14/23。有用量记录的轮次，首步缓存是 98.97%。第 1、2 组主要用 grep，未触发压缩，结束时分别是 39 511 和 29 987 token。第 4、9 组反复读全文，结束时分别是 285 535 和 281 515，和 DS 第 1 组很像。可比的 7 组里，DS 每组都更快。
+GLM 5.3 Flash 用了同一个插件、同一套题。完整跑完的 7 组为 23/23；第 5、6 组连接失败；第 7 组前段连接失败，完成 14/23。有用量记录的轮次，首步缓存是 98.97%。第 1、2 组主要用 grep，未触发压缩，结束时分别是 39 511 和 29 987 token。第 4、9 组反复读全文，结束时分别是 285 535 和 281 515。GLM 图只展示插件实测；没有对应 stock 配对数据的地方不推断原版差值。
 
-![插件版对比](docs/slide-111-overview.jpg)
+![插件版对比](docs/slide-112-overview.jpg)
 
-![DS V4.1 Flash](docs/slide-111-ds.jpg)
+![DS V4.1 Flash](docs/slide-112-ds.jpg)
 
-![GLM 5.3 Flash](docs/slide-111-glm.jpg)
+![GLM 5.3 Flash](docs/slide-112-glm.jpg)
 
 ## 1.0.9
 
@@ -297,8 +313,8 @@ GLM 5.3 Flash 用了同一个插件、同一套题。有 7 组填日志达到 23
 从 npm 装。headless 和网页是两个 profile，要各装一次。桌面端是另一个 profile，普通 `dsh plugin --profile desktop` 会被拒绝。
 
 ```sh
-dsh plugin --profile headless add dsh-live-compact@1.1.2
-dsh plugin --profile web add dsh-live-compact@1.1.2
+dsh plugin --profile headless add dsh-live-compact@1.1.4
+dsh plugin --profile web add dsh-live-compact@1.1.4
 ```
 
 想先装到本机看版本：
@@ -308,12 +324,12 @@ npm install -g dsh-live-compact
 dsh-live-compact --version
 ```
 
-旧版 1.1.1 的国内镜像安装命令：
+上不了 npm 时的镜像安装命令：
 
 ```sh
-dsh plugin --profile headless add https://download.merkyorlynn.com/downloads/plugins/dsh-live-compact-1.1.1.tgz
-dsh plugin --profile web add https://download.merkyorlynn.com/downloads/plugins/dsh-live-compact-1.1.1.tgz
-npm install -g --force "https://download.merkyorlynn.com/downloads/plugins/dsh-live-compact-1.1.1.tgz"
+dsh plugin --profile headless add https://download.merkyorlynn.com/downloads/plugins/dsh-live-compact-1.1.4.tgz
+dsh plugin --profile web add https://download.merkyorlynn.com/downloads/plugins/dsh-live-compact-1.1.4.tgz
+npm install -g --force "https://download.merkyorlynn.com/downloads/plugins/dsh-live-compact-1.1.4.tgz"
 dsh-live-compact --version
 ```
 

package/index.js

@@ -15,6 +15,7 @@
 import { randomBytes, randomUUID } from "node:crypto";
 import { closeSync, existsSync, openSync, readFileSync, renameSync, unlinkSync, writeFileSync } from "node:fs";
 import { createRequire } from "node:module";
+import { setImmediate as scheduleImmediate } from "node:timers";
 import { dirname, join } from "node:path";
 import { pathToFileURL } from "node:url";
 import { loadBase, loadCompaction, loadLlm } from "./adapter.js";
@@ -52,6 +53,10 @@ const URGENT_TOKENS = 48000;
 const MAX_TAIL = 2200;
 const MAX_TRACKED_SESSIONS = 64;
 const DEFAULT_KEY = "__default__";
+const CAPTURE_EVENT_BATCH = 128;
+const CAPTURE_RESULT_BATCH = 1;
+const RECOVER_RAW_BATCH = 1;
+const CAPTURE_RETRY_LIMIT = 8;
 
 function eventContent(session, seq) {
   const event = session.eventAt(seq);
@@ -459,6 +464,15 @@ export default class SpineCompactionEngine extends BasicCompactionEngine {
   rawSeen = new Map();
   callIndex = new Map();
   captureCursor = new Map();
+  pendingCaptures = new Set();
+  pendingCaptureJobs = new Map();
+  pendingCaptureKeys = new Set();
+  pendingCatchups = new Map();
+  pendingCatchupJobs = new Set();
+  pendingCuts = new Set();
+  captureRetryStats = new Map();
+  retryableCaptureSeqs = new Map();
+  retryableCaptureSeeded = new Set();
   pruneGuarded = false;
 
   constructor(ctx, config, resolvedConfig) {
@@ -468,15 +482,41 @@ export default class SpineCompactionEngine extends BasicCompactionEngine {
     if (process.env.SPINE_ENGINE_FAIL === "after-super") throw new Error("spine: after super");
     this.liveConfig = config;
     try {
-      const pendingCuts = new Set();
       ctx.effect(() => async () => {
-        await Promise.allSettled(pendingCuts);
+        await Promise.allSettled([...this.pendingCuts, ...this.pendingCaptures, ...this.pendingCatchupJobs]);
       });
       ctx.on("session/event", (session, event) => {
         try {
           this.noteEvent(session, event);
           if (event?.type !== "tool/result") return;
-          this.captureOne(session, event);
+          const id = event.data?.message?.toolCallId;
+          if (!id) return;
+          const key = this.rawKey(session, id);
+          const cached = this.rawSeen.get(key);
+          if (cached && !cached.retryable) return;
+          if (this.pendingCaptureKeys.has(key)) return;
+          this.pendingCaptureKeys.add(key);
+          let job;
+          job = new Promise((resolve) => {
+            scheduleImmediate(() => {
+              try {
+                const current = this.rawSeen.get(key);
+                if (!current || current.retryable) {
+                  const record = this.captureOne(session, event);
+                  this.noteCaptureResult(session, event, record);
+                }
+              } catch (error) {
+                noteShadowWarning(ctx.logger, error);
+              } finally {
+                this.pendingCaptureKeys.delete(key);
+                this.pendingCaptures.delete(job);
+                if (this.pendingCaptureJobs.get(key) === job) this.pendingCaptureJobs.delete(key);
+                resolve();
+              }
+            });
+          });
+          this.pendingCaptures.add(job);
+          this.pendingCaptureJobs.set(key, job);
         } catch (error) {
           noteShadowWarning(ctx.logger, error);
         }
@@ -489,13 +529,13 @@ export default class SpineCompactionEngine extends BasicCompactionEngine {
         const done = events.some((event) => event.type === "turn/end" && event.data?.reason?.kind === "completed");
         if (!done) return;
         state.warming = true;
-        const job = this.finishTurn(agent, events)
+        const job = this.finishTurn(agent, events, { deferCapture: true })
           .catch((error) => noteShadowWarning(this.ctx?.logger, error))
           .finally(() => {
             state.warming = false;
-            pendingCuts.delete(job);
+            this.pendingCuts.delete(job);
           });
-        pendingCuts.add(job);
+        this.pendingCuts.add(job);
       });
     } catch (error) {
       const message = error instanceof Error ? error.message : String(error);
@@ -545,12 +585,167 @@ export default class SpineCompactionEngine extends BasicCompactionEngine {
     return planMessages(messages);
   }
 
-  async finishTurn(agent, events) {
+  scheduleCaptureCatchup(agent, events, settings) {
+    this.pendingCatchups ||= new Map();
+    this.pendingCatchupJobs ||= new Set();
+    this.pendingCuts ||= new Set();
+    const key = sessionKey(agent);
+    let task = this.pendingCatchups.get(key);
+    if (task) {
+      task.agent = agent;
+      task.session = agent.session;
+      task.events = events;
+      task.settings = settings;
+      return task;
+    }
+    let resolve;
+    const promise = new Promise((done) => { resolve = done; });
+    task = {
+      key,
+      agent,
+      session: agent.session,
+      events,
+      settings,
+      phase: "events",
+      eventIndex: 0,
+      recoverIndex: 0,
+      visibleIds: null,
+      scheduled: false,
+      retryRequested: false,
+      captureRetryAttempts: 0,
+      progressed: false,
+      eventTail: Number.isInteger(events?.at?.(-1)?.seq) ? events.at(-1).seq : -1,
+      surfaceTail: agent.session.surface?.nodes?.at?.(-1) ?? -1,
+      resolve,
+      promise,
+    };
+    this.pendingCatchups.set(key, task);
+    this.pendingCatchupJobs.add(promise);
+    this.queueCaptureCatchup(task);
+    return task;
+  }
+
+  queueCaptureRetry(task) {
+    if (!task || task.finished || task.scheduled) return;
+    task.scheduled = true;
+    const delay = Math.min(250, 10 * (2 ** Math.max(0, task.captureRetryAttempts - 1)));
+    setTimeout(() => {
+      task.scheduled = false;
+      if (!task.finished) this.queueCaptureCatchup(task);
+    }, delay);
+  }
+
+  queueCaptureCatchup(task) {
+    if (!task || task.scheduled) return;
+    task.scheduled = true;
+    scheduleImmediate(() => {
+      task.scheduled = false;
+      try {
+        const latest = task.session.snapshotEvents?.();
+        const latestTail = Number.isInteger(latest?.at?.(-1)?.seq) ? latest.at(-1).seq : -1;
+        const latestSurfaceTail = task.session.surface?.nodes?.at?.(-1) ?? -1;
+        if (Array.isArray(latest) && (latestTail !== task.eventTail || latest.length !== task.events.length)) {
+          task.events = latest;
+          task.eventTail = latestTail;
+          if (task.phase === "visible") {
+            task.phase = "events";
+            task.visibleIds = null;
+            task.recoverIndex = 0;
+          }
+        }
+        if (latestSurfaceTail !== task.surfaceTail) {
+          task.surfaceTail = latestSurfaceTail;
+          if (task.phase === "visible") {
+            task.visibleIds = [...new Set(this.surfaceToolIds(task.session))];
+            task.recoverIndex = 0;
+          }
+        }
+        if (task.phase === "events") {
+          const result = this.captureNewBatch(task.session, task.events, task.settings, {
+            eventLimit: CAPTURE_EVENT_BATCH,
+            resultLimit: CAPTURE_RESULT_BATCH,
+            startIndex: task.eventIndex,
+          });
+          task.eventIndex = result.nextIndex;
+          task.progressed ||= result.progressed;
+          if (result.retryable) {
+            task.captureRetryAttempts += 1;
+            const previousMax = this.captureRetryStats?.get(task.key) || 0;
+            if (task.captureRetryAttempts > previousMax) this.captureRetryStats.set(task.key, task.captureRetryAttempts);
+            if (task.captureRetryAttempts >= CAPTURE_RETRY_LIMIT) {
+              this.finishCaptureCatchup(task);
+              return;
+            }
+            this.queueCaptureRetry(task);
+            return;
+          }
+          if (result.progressed) task.captureRetryAttempts = 0;
+          if (!result.done) {
+            this.queueCaptureCatchup(task);
+            return;
+          }
+          task.phase = "visible";
+          task.recoverIndex = 0;
+          task.visibleIds = [...new Set(this.surfaceToolIds(task.session))];
+        }
+        const recovered = this.recoverVisibleRawBatch(task.session, {}, task.recoverIndex, RECOVER_RAW_BATCH, task.visibleIds);
+        task.progressed ||= recovered.progressed;
+        task.recoverIndex = recovered.nextIndex;
+        if (!recovered.done) {
+          this.queueCaptureCatchup(task);
+          return;
+        }
+        this.finishCaptureCatchup(task);
+      } catch (error) {
+        noteShadowWarning(this.ctx?.logger, error);
+        this.finishCaptureCatchup(task);
+      }
+    });
+  }
+
+  finishCaptureCatchup(task) {
+    if (!task || task.finished) return;
+    task.finished = true;
+    this.pendingCatchups.delete(task.key);
+    this.pendingCatchupJobs.delete(task.promise);
+    task.resolve();
+    if (task.retryRequested && task.progressed) this.scheduleMaintenanceRetry(task.agent);
+  }
+
+  scheduleMaintenanceRetry(agent) {
+    const state = this.stateFor(agent);
+    if (state.retryScheduled) return;
+    state.retryScheduled = true;
+    scheduleImmediate(() => {
+      state.retryScheduled = false;
+      if (state.warming) return;
+      const events = agent.session.snapshotEvents();
+      const done = events.some((event) => event.type === "turn/end" && event.data?.reason?.kind === "completed");
+      if (!done) return;
+      state.warming = true;
+      const job = this.finishTurn(agent, events, { deferCapture: true })
+        .catch((error) => noteShadowWarning(this.ctx?.logger, error))
+        .finally(() => {
+          state.warming = false;
+          this.pendingCuts.delete(job);
+        });
+      this.pendingCuts.add(job);
+    });
+  }
+
+  async finishTurn(agent, events, options = {}) {
     const settings = readSettings(this.liveConfig);
     const state = this.stateFor(agent);
     const signal = AbortSignal.timeout(30000);
     const ev = events || agent.session.snapshotEvents();
-    this.captureNew(agent.session, ev, settings);
+    const deferCapture = options.deferCapture === true;
+    const catchup = deferCapture
+      ? this.scheduleCaptureCatchup(agent, ev, settings)
+      : { finished: true, progressed: false, retryRequested: false };
+    if (!deferCapture) {
+      await this.flushPendingCaptures();
+      this.captureNew(agent.session, ev, settings);
+    }
     const jobs = collectJobs(ev);
     const calls = indexToolCalls(ev);
     try {
@@ -610,6 +805,7 @@ export default class SpineCompactionEngine extends BasicCompactionEngine {
           try {
             state.localOnly = true;
             state.approvedCard = chunk.card;
+            if (deferCapture) await this.flushPendingCaptures(agent.session, this.chunkToolCallIds(agent.session, chunk));
             await this.commitSelectedSpan(agent, chunk, op);
             measurement = this.ctx.tokenMeter.measure(agent.session);
             const after = measurement.totalTokens;
@@ -630,6 +826,10 @@ export default class SpineCompactionEngine extends BasicCompactionEngine {
             this.ctx.logger?.info?.(`spine cut ${Date.now() - started}ms ${before} -> ${after} chunk=${chunk.tokens} saved=${saved}`);
           } catch (error) {
             const message = error instanceof Error ? error.message : String(error);
+            if (message.includes("raw backup missing before cut")) {
+              catchup.retryRequested = true;
+              if (catchup.finished && catchup.progressed) this.scheduleMaintenanceRetry(agent);
+            }
             this.ctx.logger?.warn?.(`spine cut failed: ${message}`);
             break;
           }
@@ -652,6 +852,35 @@ export default class SpineCompactionEngine extends BasicCompactionEngine {
     }
   }
 
+  async flushPendingCaptures(session = null, ids = null) {
+    if (!session) {
+      if (!this.pendingCaptures?.size) return;
+      await Promise.allSettled([...this.pendingCaptures]);
+      return;
+    }
+    const key = session.id || DEFAULT_KEY;
+    const pendingIds = ids || this.surfaceToolIds(session);
+    const jobs = [...new Set(pendingIds)]
+      .map((id) => this.pendingCaptureJobs?.get?.(`${key}\0${id}`))
+      .filter(Boolean);
+    if (jobs.length) await Promise.allSettled(jobs);
+  }
+
+  chunkToolCallIds(session, chunk) {
+    const nodes = session?.surface?.nodes || [];
+    const start = nodes.indexOf(chunk?.start);
+    const end = nodes.indexOf(chunk?.end);
+    if (start < 0 || end < start) return [];
+    const ids = [];
+    for (const seq of nodes.slice(start, end + 1)) {
+      const event = session.eventAt?.(seq);
+      if (event?.type !== "tool/result") continue;
+      const id = event.data?.message?.toolCallId;
+      if (id) ids.push(id);
+    }
+    return ids;
+  }
+
   async commitSelectedSpan(agent, chunk, signal) {
     signal?.throwIfAborted();
     const session = agent.session;
@@ -685,7 +914,8 @@ export default class SpineCompactionEngine extends BasicCompactionEngine {
       const event = session.eventAt(seq);
       if (event?.type === "tool/result") {
         const id = event.data?.message?.toolCallId;
-        if (!id || !verifiedRaw(this.rawSeen.get(this.rawKey(session, id))).ok) {
+        const raw = id ? this.rawSeen.get(this.rawKey(session, id)) : null;
+        if (!id || !verifiedRaw(raw).ok) {
           throw new Error(`spine: raw backup missing before cut: ${id || "unknown"}`);
         }
       }
@@ -941,19 +1171,109 @@ export default class SpineCompactionEngine extends BasicCompactionEngine {
     }
   }
 
-  captureNew(session, events, settings = readSettings(this.liveConfig)) {
-    if (!Array.isArray(events)) events = [];
+  noteCaptureResult(session, event, record) {
+    const key = session?.id || DEFAULT_KEY;
+    this.retryableCaptureSeqs ||= new Map();
+    this.retryableCaptureSeeded ||= new Set();
+    let seqs = this.retryableCaptureSeqs.get(key);
+    if (!seqs) {
+      seqs = new Set();
+      this.retryableCaptureSeqs.set(key, seqs);
+    }
+    const seq = Number.isInteger(event?.seq) ? event.seq : null;
+    if (seq === null) return;
+    if (record?.retryable) seqs.add(seq);
+    else seqs.delete(seq);
+  }
+
+  captureCursorStart(session, events, settings) {
     const key = session?.id || DEFAULT_KEY;
     let cursor = this.captureCursor.has(key) ? this.captureCursor.get(key) : this.readCaptureCursor(key);
     if (cursor == null) cursor = restartCaptureCursor(events, settings.keepRounds);
     if (!Number.isSafeInteger(cursor) || cursor < 0 || events.length < cursor) cursor = restartCaptureCursor(events, settings.keepRounds);
-    // A live event callback may have failed before the recent-three restart
-    // boundary. Keep those visible originals eligible without replaying history.
-    for (const seq of session?.surface?.nodes || []) {
-      const event = session.eventAt?.(seq);
-      const id = event?.type === "tool/result" && event.data?.message?.toolCallId;
-      if (id && this.rawSeen.get(this.rawKey(session, id))?.retryable) cursor = Math.min(cursor, seq);
+    this.retryableCaptureSeqs ||= new Map();
+    this.retryableCaptureSeeded ||= new Set();
+    let retryable = this.retryableCaptureSeqs.get(key);
+    if (!retryable) {
+      retryable = new Set();
+      this.retryableCaptureSeqs.set(key, retryable);
     }
+    if (!this.retryableCaptureSeeded.has(key)) {
+      for (const seq of session?.surface?.nodes || []) {
+        const event = session.eventAt?.(seq);
+        const id = event?.type === "tool/result" && event.data?.message?.toolCallId;
+        if (id && this.rawSeen.get(this.rawKey(session, id))?.retryable) retryable.add(seq);
+      }
+      this.retryableCaptureSeeded.add(key);
+    }
+    if (retryable.size) cursor = Math.min(cursor, Math.min(...retryable));
+    return cursor;
+  }
+
+  captureNewBatch(session, events, settings = readSettings(this.liveConfig), limits = {}) {
+    if (!Array.isArray(events)) events = [];
+    const key = session?.id || DEFAULT_KEY;
+    const cursor = this.captureCursorStart(session, events, settings);
+    let next = cursor;
+    let retry = null;
+    let processed = 0;
+    let captures = 0;
+    let progressed = false;
+    let exhausted = true;
+    const seen = new Set();
+    const eventLimit = Math.max(1, limits.eventLimit || CAPTURE_EVENT_BATCH);
+    const resultLimit = Math.max(1, limits.resultLimit || CAPTURE_RESULT_BATCH);
+    let index = Math.max(0, Number.isInteger(limits.startIndex) ? limits.startIndex : 0);
+    const indexedSeq = Number.isInteger(events[index]?.seq) ? events[index].seq : null;
+    if (index > 0 && indexedSeq !== null && indexedSeq > cursor) {
+      const rewind = events.findIndex((event) => Number.isInteger(event?.seq) && event.seq >= cursor);
+      index = rewind >= 0 ? rewind : 0;
+    }
+    for (; index < events.length; index += 1) {
+      const event = events[index];
+      const seq = Number.isInteger(event?.seq) ? event.seq : next;
+      if (seq < cursor) continue;
+      this.noteEvent(session, event);
+      if (event?.type === "tool/result") {
+        const id = event.data?.message?.toolCallId;
+        if (id && !seen.has(id)) {
+          seen.add(id);
+          const cached = this.rawSeen.get(this.rawKey(session, id));
+          const record = cached && !cached.retryable ? cached : this.captureOne(session, event);
+          if (!cached || cached.retryable) {
+            if (record?.ok) progressed = true;
+            captures += 1;
+          }
+          this.noteCaptureResult(session, event, record);
+          if (record?.retryable && retry === null) retry = seq;
+        }
+      }
+      next = seq + 1;
+      processed += 1;
+      if (retry !== null || processed >= eventLimit || captures >= resultLimit) {
+        exhausted = false;
+        index += 1;
+        break;
+      }
+    }
+    next = retry ?? next;
+    this.captureCursor.set(key, next);
+    this.writeCaptureCursor(key, next);
+    const lastEvent = events.at(-1);
+    const lastSeq = Number.isInteger(lastEvent?.seq) ? lastEvent.seq : events.length - 1;
+    return {
+      seen,
+      progressed,
+      retryable: retry !== null,
+      done: exhausted && retry === null && next > lastSeq,
+      nextIndex: index,
+    };
+  }
+
+  captureNew(session, events, settings = readSettings(this.liveConfig)) {
+    if (!Array.isArray(events)) events = [];
+    const key = session?.id || DEFAULT_KEY;
+    const cursor = this.captureCursorStart(session, events, settings);
     let next = cursor;
     let retry = null;
     const seen = new Set();
@@ -967,6 +1287,7 @@ export default class SpineCompactionEngine extends BasicCompactionEngine {
           seen.add(id);
           const cached = this.rawSeen.get(this.rawKey(session, id));
           const record = cached && !cached.retryable ? cached : this.captureOne(session, event);
+          this.noteCaptureResult(session, event, record);
           if (record?.retryable && retry === null) retry = seq;
         }
       }
@@ -1044,6 +1365,29 @@ export default class SpineCompactionEngine extends BasicCompactionEngine {
     }
   }
 
+  recoverVisibleRawBatch(session, access = {}, startIndex = 0, limit = RECOVER_RAW_BATCH, visibleIds = null) {
+    const ids = visibleIds || [...new Set(this.surfaceToolIds(session))];
+    const dir = sessionDir(shadowDirectory(), session?.id || DEFAULT_KEY);
+    let index = Math.max(0, startIndex);
+    let processed = 0;
+    let progressed = false;
+    while (index < ids.length && processed < Math.max(1, limit)) {
+      const id = ids[index++];
+      const mapKey = this.rawKey(session, id);
+      if (this.rawSeen.has(mapKey)) continue;
+      try {
+        const record = readRawToolResult({ dir, toolCallId: id, access });
+        this.rawSeen.set(mapKey, record);
+        progressed ||= Boolean(record?.ok);
+      } catch (error) {
+        noteShadowWarning(this.ctx?.logger, error);
+        this.rawSeen.set(mapKey, { ok: false, reason: "unwritable", toolCallId: id });
+      }
+      processed += 1;
+    }
+    return { nextIndex: index, done: index >= ids.length, progressed };
+  }
+
   captureSession(session) {
     return this.captureNew(session, session?.snapshotEvents?.() || []);
   }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dsh-live-compact",
-  "version": "1.1.2",
+  "version": "1.1.4",
   "description": "In-place newest-turn compaction for DeepSeek Harness. Compress while the task runs and keep the prefix.",
   "license": "MIT",
   "author": "MerkyorLynn <268568828+MerkyorLynn@users.noreply.github.com>",
@@ -41,7 +41,7 @@
     "compaction"
   ],
   "scripts": {
-    "test:gate": "node test/p0-gate.mjs && node test/p0-blockers.mjs && node test/excerpt.mjs && node --test test/public-exports.test.mjs test/cooldown.test.mjs test/gate-config.test.mjs test/drain-stopline.mjs test/scan-once.mjs test/issue-19.mjs test/issue-20.mjs test/windows-acl.test.mjs test/issue-27.mjs test/issue-29.mjs test/issue-29-notice.mjs test/client-settings.test.mjs test/settings-integration.test.mjs test/runtime-lifecycle.mjs test/shadow-lifecycle.mjs test/finalize-over-32000.mjs",
+    "test:gate": "node test/p0-gate.mjs && node test/p0-blockers.mjs && node test/excerpt.mjs && node --test test/public-exports.test.mjs test/cooldown.test.mjs test/gate-config.test.mjs test/drain-stopline.mjs test/scan-once.mjs test/issue-19.mjs test/issue-20.mjs test/issue-34.mjs test/windows-acl.test.mjs test/issue-27.mjs test/issue-29.mjs test/issue-29-notice.mjs test/client-settings.test.mjs test/settings-integration.test.mjs test/runtime-lifecycle.mjs test/shadow-lifecycle.mjs test/finalize-over-32000.mjs",
     "test:live-gate": "python3 -B -m unittest discover -s test/live-fill -p test_*.py"
   },
   "engines": {