dsh-fast-mode 1.5.28 → 1.5.30

package/CHANGELOG.md

@@ -1,5 +1,20 @@
 # Changelog
 
+## 1.5.30（把 1.5.28 的修复拿真实流量做 A/B：同一份会话，同一串事件，两个版本）
+
+- **新增 `bench/replay.mjs`**：把一份留存会话的事件流原样喂给插件自己的钩子（`session/event` + `agent/pre-step`），数出「这一版会发几条提醒、落在第几步」。会话 id 是合成的，`metrics`/`benchmarkFile` 关掉，不碰 `~/.dsh` 状态。
+- **量化修复效果**：轮内压缩时宿主用**同一个 turn 号**重入该轮，而这次重入不落盘——所以真实会话记录里看不到它，只看得到「同一轮两条提醒」。`--simulate-reentry` 在每次 `compaction/start` 后按当前 turn 号补发一条 `turn/start`，把这个假设显式化：
+  - 1.5.27 的接线（`noteTurn` 不知道 turn 号）：1529 个合格轮里 **940 轮**会被发两次以上；
+  - 1.5.28 的接线：同样 1529 轮 **0 轮**；
+  - 单会话对照更直观：`session-1ee3a310` 111 个合格轮，旧版 82 轮重复（最长的 121 号轮在第 4/13/38/65/102 步各发一次），新版 0 轮。
+- **真实流量侧读数不变**：`bench/turns.py` 近 72 小时 651 个合格轮里 120 轮真收到两条、其中 110 轮两次之间夹着压缩，与重入假设一致。
+- **没有行为变更**：只加脚本、改文档与版本号；`src/` 与 1.5.29 逐字相同。
+
+## 1.5.29（1.5.28 的补丁：把 `bench/turns.py` 真的放进包里）
+
+- `package.json` 的 `files` 是一份白名单，1.5.28 新增的 `bench/turns.py` 没进去：`npm pack dsh-fast-mode@1.5.28` 里没有这个脚本，而 `CHANGELOG` 和 `bench/README.md` 都在教读者运行它。1.5.29 只改白名单 + 版本号，**代码与 1.5.28 逐字相同**。
+- 复核方式：`npm pack dsh-fast-mode@1.5.29 --silent` 解包后应同时看到 `package/bench/turns.py` 与 `package/index.js` 里的 `VERSION = '1.5.29'`。
+
 ## 1.5.28（稳定性：压缩后同一轮里不再重复发同一条提醒）
 
 - **先量后改**：新增 `bench/turns.py`，把留存会话按 `turn/start` 切成轮，统计「够长却没收到提醒」「同一轮收到两次以上」「多轮会话里后续轮是否重新上膛」。72 小时窗口 68 个会话 / 1446 轮：647 个合格轮**646 个收到**提醒（99.8 %）、后续轮重新上膛 599/600；但**同一轮收到两次的有 114 轮**，其中 **105 轮（92 %）两次之间夹着一次上下文压缩**。

package/bench/README.md

@@ -136,6 +136,18 @@ node bench/verify-arm-reminders.mjs --arms                # 臂 x 场景的提
 
 采集方式：跑这一格时把臂补丁另存为 `<tag>.arm.yml` 并加 `debug: true` + `debugFile: <runs>/<tag>.debug`，跑完从调试日志里按 `step=N ... turnSteps=M ... directive=kind` 抽行，然后删掉日志与补丁。`debug` 只在步边界写一行已经格式化好的文本，不参与任何调度判断、不额外发请求；日志写在 run 目录之外，所以 `leak_scan`/session id 统计不受影响。老 result 行没有这些字段，读取方一律用 `.get()`，缺字段等于「没有证据」，不是「零提醒」。
 
+### 同一份会话，两个版本各会发几条提醒
+
+`bench/replay.mjs` 把留存会话的事件流喂回插件自己的钩子（`session/event` + `agent/pre-step`），输出每一轮的步数、压缩次数、提醒条数与落点。会话 id 是合成的，`metrics`/`benchmarkFile` 关掉，不写任何 `~/.dsh` 状态。
+
+```bash
+node bench/replay.mjs ~/.dsh/sessions/<slug>/session-<uuid>/session.v4.jsonl.zstd
+node bench/replay.mjs <transcript> --json
+node bench/replay.mjs <transcript> --simulate-reentry   # 每次压缩后按当前 turn 号重入
+```
+
+`--simulate-reentry` 补的是会话记录里**不存在**的那条事件：宿主压缩后重入同一轮时用同一个 turn 号再发一次 `turn/start`，而这次重入不落盘。加上它以后，1.5.27 与 1.5.28 的差别就能在同一份数据上量出来（1529 个合格轮：940 → 0）。
+
 ### 提醒是否每轮重新上膛、且每轮只发一次
 
 `bench/turns.py` 读留存会话，按 `turn/start` 切轮，输出：轮数/会话、合格轮（默认 ≥ 4 步）、**没收到提醒**的合格轮、**收到两次以上**的轮、多轮会话里后续轮的重新上膛率。只有 `user/message` / `agent/inbox/spliced` / `system/message` 里的文案算投递，模型在自己正文里复述的不算。

package/bench/replay.mjs

@@ -0,0 +1,154 @@
+#!/usr/bin/env node
+/**
+ * Replay one retained transcript through the plugin's own hooks.
+ *
+ * The reminder is delivered at a step boundary, so a transcript could only ever
+ * count *delivered* reminders, never the ones the plugin would have sent under a
+ * different version. This replays the recorded event stream into `apply()` with a
+ * stand-in context, counts what the scheduler decides to emit, and prints one
+ * line per turn: steps, reminders, and the step each reminder landed on.
+ *
+ * That is how the 1.5.28 latch fix is checked against real traffic instead of
+ * synthetic test events: same transcript, same hook order, two versions of
+ * `noteTurn`, two counts.
+ *
+ * Usage:
+ *   node bench/replay.mjs ~/.dsh/sessions/<slug>/session-<uuid>/session.v4.jsonl.zstd
+ *   node bench/replay.mjs <transcript> --min-steps 4 --json
+ *   node bench/replay.mjs <transcript> --simulate-reentry
+ *
+ * `--simulate-reentry` adds the event the transcript cannot show: after each
+ * `compaction/start` it re-fires `turn/start` with the same turn number, which is
+ * the hypothesis for the doubled reminders (the harness re-enters a turn it
+ * compacted mid-flight, and that re-entry is not persisted). With the event
+ * added, the pre-1.5.28 latch resets and the same turn is reminded twice; with
+ * the turn guard it is not.
+ *
+ * The session id is synthetic, so the run never touches `~/.dsh` state; metrics
+ * and benchmark output are switched off in the config passed to `apply()`.
+ */
+import { execFileSync } from 'node:child_process'
+import { readFileSync } from 'node:fs'
+
+import { apply } from '../index.js'
+
+const NEEDLE = 'prose is now the cost'
+const CONFIG = { enabled: true, gear: 'auto', outputDiscipline: 'late', metrics: false, benchmarkFile: '' }
+
+/** Minimal stand-in for the Cordis context: handlers plus a silent logger. */
+function fakeContext() {
+  const handlers = new Map()
+  return {
+    handlers,
+    on(event, handler) {
+      handlers.set(event, handler)
+      return () => handlers.delete(event)
+    },
+    inject: () => undefined,
+    get: () => undefined,
+    logger: { info: () => {} },
+    fire(event, ...args) {
+      const handler = handlers.get(event)
+      if (!handler) return undefined
+      return handler(...args)
+    },
+  }
+}
+
+/** Decide the way the harness does: enter, and keep whatever the plugin added. */
+const preStep = (ctx, agentId, step) => {
+  const decision = {
+    kind: 'enter',
+    messages: [{ id: 'sys', role: 'system', content: [] }, { id: `last-${step}`, role: 'assistant', content: [] }],
+  }
+  return ctx.fire('agent/pre-step', { agent: { id: agentId }, messages: [] }, async () => decision)
+}
+
+const events = path => {
+  const raw = path.endsWith('.zstd')
+    ? execFileSync('zstd', ['-dc', path], { maxBuffer: 1 << 30 })
+    : readFileSync(path)
+  const out = []
+  for (const line of raw.toString('utf8').split('\n')) {
+    if (!line.trim()) continue
+    try {
+      out.push(JSON.parse(line))
+    } catch {
+      // a torn tail line is not fatal: the replay is a census, not a forensic read
+    }
+  }
+  return out
+}
+
+const reminderText = notice => {
+  const text = notice?.content?.[0]?.text ?? ''
+  return text.includes(NEEDLE)
+}
+
+async function main() {
+  const argv = process.argv.slice(2)
+  const path = argv.find(arg => !arg.startsWith('--'))
+  if (!path) throw new Error('usage: node bench/replay.mjs <transcript.jsonl.zstd> [--min-steps N] [--json]')
+  const minSteps = Number(argv[argv.indexOf('--min-steps') + 1]) || 4
+  const asJson = argv.includes('--json')
+  const reentry = argv.includes('--simulate-reentry')
+
+  const ctx = fakeContext()
+  apply(ctx, CONFIG)
+  const session = { id: 'replay' }
+  const turns = []
+  let cur = { steps: 0, reminders: [], compactions: 0, turnNo: 0 }
+  for (const event of events(path)) {
+    const type = event.type
+    const data = event.data ?? {}
+    if (type === 'turn/start') {
+      if (cur.steps || cur.reminders.length) turns.push(cur)
+      cur = { steps: 0, reminders: [], compactions: 0, turnNo: data.turn ?? turns.length + 1 }
+      await ctx.fire('session/event', session, event)
+    } else if (type === 'step/start') {
+      cur.steps += 1
+      await ctx.fire('session/event', session, event)
+      const decision = await preStep(ctx, session.id, cur.steps)
+      const added = (decision?.messages ?? []).filter(message => message?.source?.form === 'notice' && reminderText(message))
+      if (added.length) cur.reminders.push(cur.steps)
+    } else if (type === 'compaction/start') {
+      cur.compactions += 1
+      await ctx.fire('session/event', session, event)
+      if (reentry) {
+        // Same turn number, delivered the way the harness delivers it: the event
+        // exists in the hook stream but never reaches the transcript.
+        await ctx.fire('session/event', session, { type: 'turn/start', turn: cur.turnNo, time: event.time })
+      }
+    } else if (type === 'compaction/end') {
+      await ctx.fire('session/event', session, event)
+    } else if (type === 'assistant/message' || type === 'tool/call' || type === 'tool/result') {
+      await ctx.fire('session/event', session, event)
+    }
+  }
+  if (cur.steps || cur.reminders.length) turns.push(cur)
+
+  const rows = turns.map((turn, index) => ({
+    turn: index + 1,
+    steps: turn.steps,
+    compactions: turn.compactions,
+    reminders: turn.reminders.length,
+    at: turn.reminders,
+  }))
+  if (asJson) {
+    console.log(JSON.stringify({ path, minSteps, simulateReentry: reentry, turns: rows }, null, 2))
+    return
+  }
+  const qualifying = rows.filter(row => row.steps >= minSteps)
+  const doubled = qualifying.filter(row => row.reminders > 1)
+  console.log(`${path}`)
+  for (const row of qualifying) {
+    console.log(`  turn ${row.turn}: steps ${row.steps} compactions ${row.compactions} `
+      + `reminders ${row.reminders}${row.at.length ? ` at ${row.at.join(',')}` : ''}`)
+  }
+  console.log(`qualifying turns (>= ${minSteps} steps) ${qualifying.length}  doubled ${doubled.length}`)
+}
+
+main().catch(error => {
+  console.error(error.message)
+  process.exit(1)
+})

package/bench/turns.py

@@ -0,0 +1,210 @@
+#!/usr/bin/env python3
+"""Does the late-discipline reminder re-arm per turn, and only once per turn?
+
+The reminder is a once-per-turn lever: `disciplineSent` latches it so a long turn
+is told to stop narrating exactly once, and `noteTurn` clears the latch so the
+next request in the same session is told again. Both halves are invisible in the
+wall-clock numbers — a turn that never got the reminder looks like a turn that
+ignored it, and a turn that got it twice looks like a model that keeps forgetting.
+
+This reads retained transcripts, segments them on `turn/start`, and reports:
+
+- turns per session, steps per turn;
+- turns long enough to qualify (>= 4 steps) that got **no** reminder;
+- turns that got **two or more** reminders (the latch leaking);
+- re-arm rate: in multi-turn sessions, the share of later qualifying turns that
+  were reminded again.
+
+Usage:
+  python3 bench/turns.py                  # suite sessions
+  python3 bench/turns.py --limit 100      # first 100 transcripts
+  python3 bench/turns.py --min-steps 4 --json
+"""
+
+from __future__ import annotations
+
+import argparse
+import collections
+import glob
+import json
+import os
+import subprocess
+import time
+
+SESS = os.path.expanduser('~/.dsh/sessions')
+NEEDLE = 'this turn is long enough that prose'
+# A delivered directive is a message part; the model quoting it back inside its
+# own prose is an echo, so assistant-authored text never counts as delivery.
+PLUGIN_ROLES = {'user/message', 'agent/inbox/spliced', 'system/message'}
+
+
+def transcripts(pattern, limit, exclude_arm=('off',)):
+    """Session transcripts whose run directory (the slug) is not one of `exclude_arm`.
+
+    The slug mirrors the run directory, so `...-suite-D2G2-ledger-p1-def15--`
+    identifies the arm without re-reading the YAML.
+    """
+    out = []
+    for d in sorted(glob.glob(os.path.join(SESS, pattern))):
+        name = os.path.basename(d)
+        if any(name.endswith(f'-{arm}--') for arm in exclude_arm):
+            continue
+        out.extend(glob.glob(os.path.join(d, 'session-*', 'session.v4.jsonl.zstd')))
+    return out[:limit] if limit else out
+
+
+def text_of(data):
+    msg = data.get('message') or data
+    parts = msg.get('content') if isinstance(msg, dict) else None
+    if not isinstance(parts, list):
+        return ''
+    return ' '.join(p.get('text') or '' for p in parts
+                    if isinstance(p, dict) and p.get('type') == 'text')
+
+
+def scan(path):
+    """Return (started_ms, turns) for one transcript, or None if unreadable."""
+    p = subprocess.run(['zstd', '-dc', path], capture_output=True)
+    if p.returncode != 0:
+        return None
+    started = 0
+    turns = []
+    cur = {'steps': 0, 'reminders': 0, 'reminderSeqs': [], 'compactions': [],
+           'reminderSteps': []}
+    for line in p.stdout.decode('utf-8', 'replace').splitlines():
+        if ('turn/start' not in line and 'step/start' not in line
+                and 'Auto schedule' not in line and 'compaction/start' not in line):
+            continue
+        try:
+            o = json.loads(line)
+        except Exception:  # noqa: BLE001 - a torn line is not fatal
+            continue
+        kind = o.get('type')
+        data = o.get('data') or {}
+        if not started and isinstance(o.get('time'), int):
+            started = o['time']
+        if kind == 'turn/start':
+            if cur['steps'] or cur['reminders']:
+                turns.append(cur)
+            cur = {'steps': 0, 'reminders': 0, 'reminderSeqs': [], 'compactions': [],
+           'reminderSteps': []}
+        elif kind == 'step/start':
+            cur['steps'] += 1
+        elif kind == 'compaction/start':
+            cur['compactions'].append(o.get('seq'))
+        elif kind in PLUGIN_ROLES and NEEDLE in text_of(data):
+            cur['reminders'] += 1
+            cur['reminderSeqs'].append(o.get('seq'))
+            cur['reminderSteps'].append(cur['steps'])
+    if cur['steps'] or cur['reminders']:
+        turns.append(cur)
+    return started, turns
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument('--limit', type=int, default=0, help='transcripts; 0 = all')
+    ap.add_argument('--dir', default='--*dsh-runs-suite-*')
+    ap.add_argument('--min-steps', type=int, default=4)
+    ap.add_argument('--json', action='store_true')
+    ap.add_argument('--exclude-arm', action='append', default=[],
+                    help='skip sessions whose run directory ends in this arm (default: off)')
+    ap.add_argument('--explain-doubles', action='store_true',
+                    help='list every turn that got two reminders, with seqs and compactions')
+    ap.add_argument('--since-hours', type=float, default=0,
+                    help='only sessions started within this many hours')
+    a = ap.parse_args()
+
+    sessions = turns_total = multi = 0
+    qualifying = reminded = 0
+    misses = doubled = doubled_compacted = 0
+    later_qualifying = later_reminded = 0
+    per_session_turns = collections.Counter()
+    entry_steps = collections.Counter()
+    offenders = []
+    cutoff = (time.time() - a.since_hours * 3600) * 1000 if a.since_hours else 0
+    skipped_old = 0
+    for f in transcripts(a.dir, a.limit, tuple(a.exclude_arm or ['off'])):
+        r = scan(f)
+        if not r:
+            continue
+        started, turns = r
+        if cutoff and started and started < cutoff:
+            skipped_old += 1
+            continue
+        if not turns:
+            continue
+        sessions += 1
+        turns_total += len(turns)
+        per_session_turns[len(turns)] += 1
+        if len(turns) > 1:
+            multi += 1
+        for i, t in enumerate(turns):
+            if t['steps'] >= a.min_steps:
+                qualifying += 1
+                if t['reminders']:
+                    reminded += 1
+                else:
+                    misses += 1
+                if i > 0:
+                    later_qualifying += 1
+                    later_reminded += 1 if t['reminders'] else 0
+            if t['reminders']:
+                entry_steps[t['reminderSteps'][0]] += 1
+            if t['reminders'] > 1:
+                doubled += 1
+                seqs = t['reminderSeqs']
+                if any(c is not None and seqs and seqs[0] < c < seqs[-1] for c in t['compactions']):
+                    doubled_compacted += 1
+                if a.explain_doubles:
+                    offenders.append({
+                        'session': os.path.basename(os.path.dirname(os.path.dirname(f))),
+                        'turn': i + 1,
+                        'steps': t['steps'],
+                        'reminderSeqs': seqs,
+                        'reminderSteps': t['reminderSteps'],
+                        'compactions': t['compactions'],
+                    })
+
+    out = {
+        'sessions': sessions,
+        'turns': turns_total,
+        'multi_turn_sessions': multi,
+        'turns_per_session': dict(sorted(per_session_turns.items())),
+        'min_steps': a.min_steps,
+        'qualifying_turns': qualifying,
+        'reminded': reminded,
+        'missed': misses,
+        'doubled': doubled,
+        'doubled_with_compaction_between': doubled_compacted,
+        'entry_steps': dict(sorted(entry_steps.items())),
+        'later_qualifying_turns': later_qualifying,
+        'later_reminded': later_reminded,
+        'skipped_old': skipped_old,
+        'since_hours': a.since_hours,
+    }
+    if a.json:
+        print(json.dumps(out, ensure_ascii=False, indent=2))
+        return
+    print(f"sessions {sessions}  turns {turns_total}  multi-turn {multi} "
+          f"({100 * multi / max(sessions, 1):.0f} %)")
+    print(f"turns per session: {dict(sorted(per_session_turns.items()))}")
+    print(f"qualifying turns (>= {a.min_steps} steps) {qualifying}  "
+          f"reminded {reminded} ({100 * reminded / max(qualifying, 1):.0f} %)  "
+          f"missed {misses}  doubled {doubled} "
+          f"(with a compaction between: {doubled_compacted})")
+    if entry_steps:
+        print(f"first reminder landed at step: {dict(sorted(entry_steps.items()))}")
+    if a.explain_doubles and offenders:
+        print('doubled turns:')
+        for o in offenders[:20]:
+            print(f"  {o['session']} turn {o['turn']} steps {o['steps']} "
+                  f"reminders at steps {o['reminderSteps']} seqs {o['reminderSeqs']} "
+                  f"compactions {o['compactions']}")
+    print(f"later turns in multi-turn sessions: qualifying {later_qualifying}  "
+          f"reminded {later_reminded} "
+          f"({100 * later_reminded / max(later_qualifying, 1):.0f} %)")
+
+
+if __name__ == '__main__':
+    main()

package/index.js

@@ -97,7 +97,7 @@ const TAG = '[fast-mode]'
  * Shipped identity of this build. Traced into the debug log so a stale install is
  * visible at a glance, and pinned by a test against `package.json`.
  */
-export const VERSION = '1.5.28'
+export const VERSION = '1.5.30'
 const GLOBAL_AGENT = '__global__'
 
 /**

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dsh-fast-mode",
-  "version": "1.5.28",
+  "version": "1.5.30",
   "description": "DeepSeek Harness fast mode: same thinking effort, fewer round trips. An adaptive parallel tool fan-out, output discipline that never trims the answer, and a cancellable stall guard. Never changes reasoning effort, the model, or the request body. Provider-agnostic, one dependency (@deepseek-ai/schemastery).",
   "repository": {
     "type": "git",
@@ -44,6 +44,8 @@
     "bench/reminders.py",
     "bench/patchshare.py",
     "bench/ablate.py",
+    "bench/turns.py",
+    "bench/replay.mjs",
     "docs/",
     "LICENSE"
   ],