@catheadowl/dsh-eval 0.2.1 → 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -8,6 +8,77 @@ All notable changes to `@catheadowl/dsh-eval` are documented here. Versions
8
8
  follow [Semantic Versioning](https://semver.org/); entries follow
9
9
  [Keep a Changelog](https://keepachangelog.com/) conventions.
10
10
 
11
+ ## [0.3.0] — 2026-09-13
12
+
13
+ ### Added
14
+
15
+ - **Projection census** on every case that produced a session trace:
16
+ `census.eventTypeCounts` (the main session log's events per type),
17
+ `census.projectionLengths` (the five projection lengths) with
18
+ `census.projectionSkipped` naming where a count exceeds its projection length,
19
+ `census.projectionFieldGaps` naming the events that projected while a field
20
+ they carry went missing (the `tool/call` / `tool/result` / `request/header`
21
+ projections are 1:1, so a moved field never shows up as a length difference),
22
+ and `census.subagent` (the child logs behind `subagentChildren`: their
23
+ `subagent/descriptor` event counts, how many carry the supported descriptor
24
+ version, and the folded identity). The census reports numbers only and never
25
+ decides whether a difference is a defect — it makes "the host log carried no
26
+ such event" and "the projection dropped it" separable in `--format json` and
27
+ in `.runs/<id>/trace.json`. Cases with no trace carry no census.
28
+ - Session-seam boundary assertions: `collectSessionTrace` reports which
29
+ candidate artifact files were actually present when no session trace
30
+ materializes (behavior failures name the host artifact naming instead of a
31
+ bare "no session trace materialized"), and `parseSessionLog` refuses a
32
+ `header.version` outside `KNOWN_SESSION_FORMAT_VERSIONS` with the version
33
+ number instead of projecting empty fields. `EvalRunResult.traceGap` carries
34
+ the diagnosis to the CLI failure text.
35
+
36
+ ### Removed
37
+
38
+ - `loadTraceDir` (experimental): replaced by `collectSessionTrace`, which
39
+ returns the trace together with the reason none was built. Migrate
40
+ `loadTraceDir(root)` to `collectSessionTrace(root).trace`.
41
+ - **`options.dshRepoDir`** on `runEvalCase` (and the same legacy option on
42
+ the review executor and `executeBehaviorExperiment`): the CLI location is
43
+ now `options.cliPath` only. **Migrating a case that trips this**: resolve
44
+ once with `resolveDshCliChain` (the `/experimental` export) and pass its
45
+ `cli` result as `cliPath` — the `dsh-eval` / `dsh-review` CLIs already do
46
+ this via `--repo`, the `node_modules` resolution layer, or the config
47
+ `repo` key.
48
+
49
+ ### Fixed
50
+
51
+ - **Mock mode against host 0.1.5-rc.2: the scripted adapter now carries its
52
+ own `prepareCall`.** The host's LLM service dispatches every model call
53
+ through `registration.adapter.prepareCall(...)`, a wire-contract step the
54
+ adapter's inherited base class did not have — the base resolves from this
55
+ package's `@deepseek-ai/dsh-llm` peer instance, which lagged the host
56
+ runtime (0.0.1-rc.1 vs 0.1.5-rc.2), so every mock run died at startup with
57
+ `registration.adapter.prepareCall is not a function`. The override mirrors
58
+ the host base-class default (`{ model, stream }` bound to one adapter
59
+ generation), making the wire contract independent of the peer instance's
60
+ generation; the seam is now documented in `docs/host-wiring.md` with a
61
+ maintenance trigger.
62
+ - `census.eventTypeCounts` counts prototype-named event types correctly: a
63
+ plug-in event type such as `constructor` or `__proto__` used to produce a
64
+ string-concatenated value or vanish from the map entirely, so a field named
65
+ like a count could hold a non-number.
66
+ - Session-trace discovery follows the host's **format-generation artifact
67
+ names**: `session.jsonl` for v0 and `session.vN.jsonl` for later
68
+ generations (`session.v3.jsonl` on the current host). Matching only the v0
69
+ name made every behavior case fail with "no session trace materialized"
70
+ after the host bumped the session format.
71
+ - The multi-turn driver reads the durable log through
72
+ `Session#snapshotEvents()`; the `session.events` getter it used was removed
73
+ upstream, so any case declaring `followups` aborted the headless run with
74
+ `agent.session.events is not iterable`.
75
+ - Review runs no longer fail open when no session artifact materializes:
76
+ `validateToolBoundary` reports `status: 'not-executed'` (not a pass), the
77
+ executor result carries the gap, the report states
78
+ `tool boundary: NOT EXECUTED on run(s) N`, and both `run-N.txt` and
79
+ `run.json` record it — a review whose tool face was never verified no longer
80
+ reads as a normal one.
81
+
11
82
  ## [0.2.1] — 2026-09-09
12
83
 
13
84
  ### Added
@@ -52,6 +123,7 @@ follow [Semantic Versioning](https://semver.org/); entries follow
52
123
  `@deepseek-ai/dsh-llm` is a peerDependency provided by the host
53
124
  ecosystem.
54
125
 
126
+ [0.3.0]: https://github.com/CatheadOwl/dsh-eval/releases/tag/v0.3.0
55
127
  [0.2.1]: https://github.com/CatheadOwl/dsh-eval/releases/tag/v0.2.1
56
128
  [0.2.0]: https://github.com/CatheadOwl/dsh-eval/releases/tag/v0.2.0
57
129
  [0.1.0]: https://github.com/CatheadOwl/dsh-eval/releases/tag/v0.1.0
package/bin/dsh-eval.mjs CHANGED
@@ -11,9 +11,10 @@
11
11
  * A case path is a `*.eval.mjs` file or a directory scanned recursively for
12
12
  * them. Each file default-exports one case object (or an array of them):
13
13
  * `{ id, task, mode?: 'real'|'mock', expect: Matcher[], script?, persona?,
14
- * prepare?, timeoutMs? }`. Real cases skip when DEEPSEEK_API_KEY is absent;
15
- * the exit code is 1 when any run fails. Failures keep their artifacts under
16
- * `<case file dir>/.runs/<case id>/`.
14
+ * prepare?, timeoutMs? }`. Real cases skip when NO credential is visible —
15
+ * the env var DEEPSEEK_API_KEY OR the staged home's .credentials.yaml,
16
+ * either one counts. The exit code is 1 when any run fails. Failures keep
17
+ * their artifacts under `<case file dir>/.runs/<case id>/`.
17
18
  *
18
19
  * Output formats:
19
20
  * - `--format text` (default): unchanged human output on stdout/stderr.
@@ -134,7 +135,8 @@ function writeArtifacts(evalCase, result, mode) {
134
135
  writeFileSync(join(artifactsDir, 'stderr.txt'), result.stderr)
135
136
  writeFileSync(join(artifactsDir, 'trace.json'), JSON.stringify({
136
137
  caseId: evalCase.id, mode, task: evalCase.task,
137
- exitCode: result.exitCode, timedOut: result.timedOut, trace: result.trace,
138
+ exitCode: result.exitCode, timedOut: result.timedOut,
139
+ traceGap: result.traceGap, trace: result.trace,
138
140
  }, undefined, 2))
139
141
  result.sessionLogs.forEach((text, index) => {
140
142
  writeFileSync(join(artifactsDir, `session-${index}.jsonl`), text)
@@ -233,7 +235,7 @@ for (const file of files.sort()) {
233
235
  const runStartedAt = Date.now()
234
236
  let result
235
237
  try {
236
- result = await runEvalCase(evalCase, { profile, cliPath, dshRepoDir: repoDir, mode })
238
+ result = await runEvalCase(evalCase, { profile, cliPath, mode })
237
239
  } catch (error) {
238
240
  records.push(createCaseRecord({
239
241
  id: evalCase.id, file, mode, status: 'fail',
@@ -247,14 +249,19 @@ for (const file of files.sort()) {
247
249
 
248
250
  if (result.trace === undefined) {
249
251
  const artifactsDir = writeArtifacts(evalCase, result, mode)
252
+ // The runner's seam diagnosis (which candidate file names were actually
253
+ // collected, or which artifact was refused) IS the failure text: a bare
254
+ // "no session trace materialized" pointed readers at the parser while
255
+ // the real drift was the host's artifact naming (EVAL-019/EVAL-020).
256
+ const reason = `${result.traceGap ?? 'no session trace materialized'} (exit ${result.exitCode}${result.timedOut ? ', timed out' : ''})`
250
257
  records.push(createCaseRecord({
251
258
  id: evalCase.id, file, mode, status: 'fail',
252
- failures: [`no session trace materialized (exit ${result.exitCode}${result.timedOut ? ', timed out' : ''})`],
259
+ failures: [reason],
253
260
  exitCode: result.exitCode, timedOut: result.timedOut,
254
261
  durationMs, artifactsDir,
255
262
  }))
256
263
  process.stderr.write(
257
- `FAIL ${evalCase.id}: no session trace materialized (exit ${result.exitCode}${result.timedOut ? ', timed out' : ''})\n`
264
+ `FAIL ${evalCase.id}: ${reason}\n`
258
265
  + ` artifacts: ${artifactsDir}\n--- stderr ---\n${result.stderr}\n`,
259
266
  )
260
267
  continue
@@ -278,7 +285,8 @@ for (const file of files.sort()) {
278
285
  records.push(createCaseRecord({
279
286
  id: evalCase.id, file, mode, status: 'pass',
280
287
  exitCode: result.exitCode, timedOut: result.timedOut,
281
- durationMs, ...(artifactsDir !== undefined ? { artifactsDir } : {}),
288
+ durationMs, ...(result.trace?.census !== undefined ? { census: result.trace.census } : {}),
289
+ ...(artifactsDir !== undefined ? { artifactsDir } : {}),
282
290
  }))
283
291
  say(`PASS ${evalCase.id}`)
284
292
  } else {
@@ -293,6 +301,7 @@ for (const file of files.sort()) {
293
301
  id: evalCase.id, file, mode, status: 'fail', failures: hint ? [...failures, hint] : failures,
294
302
  exitCode: result.exitCode, timedOut: result.timedOut,
295
303
  durationMs, artifactsDir,
304
+ ...(result.trace?.census !== undefined ? { census: result.trace.census } : {}),
296
305
  }))
297
306
  process.stderr.write(`FAIL ${evalCase.id} (exit ${result.exitCode}):\n${failures.map(f => ` - ${f}`).join('\n')}\n`)
298
307
  if (hint !== undefined) process.stderr.write(` ! ${hint}\n`)
@@ -95,11 +95,11 @@ const { config } = await loadEvalConfig(process.cwd())
95
95
  const profile = options.profile ?? config.profile ?? 'headless'
96
96
  // CLI resolution (C6): `--repo` flag > resolution layer (node_modules) >
97
97
  // config repo key (legacy). Dry-run never boots the CLI, so resolve lazily.
98
- let cli = { cliPath: undefined, repoDir: undefined }
98
+ let cli = { cliPath: undefined }
99
99
  if (!options.dryRun) {
100
100
  try {
101
101
  const resolved = resolveDshCliChain({ repoFlag: options.repo, configRepo: config.repo })
102
- cli = { cliPath: resolved.cli, repoDir: resolved.repo }
102
+ cli = { cliPath: resolved.cli }
103
103
  } catch (error) {
104
104
  usage(`error: ${error.message}`)
105
105
  }
@@ -127,7 +127,6 @@ for (const file of files) {
127
127
  const result = await runDshReviewExperiment(experiment, {
128
128
  profile,
129
129
  cliPath: cli.cliPath,
130
- dshRepoDir: cli.repoDir,
131
130
  runs: options.runs,
132
131
  timeoutMs: options.timeoutMs,
133
132
  keepPluginRows: options.keepPluginRows,
@@ -136,15 +135,29 @@ for (const file of files) {
136
135
  adapter: 'dsh-headless',
137
136
  profile,
138
137
  runs: result.runs,
138
+ // Machine-readable boundary accounting per run: `checked` (the request
139
+ // headers were inspected) vs `not-executed` (no session artifact, so the
140
+ // reviewer's tool face was never verified) — the report and run-N.txt
141
+ // carry the same fact in prose (EVAL-021).
142
+ toolBoundaries: result.attempts.map(attempt => ({
143
+ run: attempt.index,
144
+ status: attempt.result?.toolValidation?.status ?? 'not-reported',
145
+ ...(attempt.result?.traceGap === undefined ? {} : { reason: attempt.result.traceGap }),
146
+ })),
139
147
  }, result)
140
148
  for (const attempt of result.attempts) {
141
149
  const payload = attempt.result ?? {}
142
150
  // run-N.txt is the reviewer's ANSWER (trace-derived, splice-proof),
143
151
  // falling back to stdout for executors/trace-less runs; the raw final
144
152
  // message stays in run-N.stdout.txt when it differs from the answer,
145
- // the full session transcript in run-N.stderr.txt.
146
- if (payload.answer !== undefined) writeFileSync(join(output, `run-${attempt.index}.txt`), payload.answer, 'utf8')
147
- else if (payload.stdout !== undefined) writeFileSync(join(output, `run-${attempt.index}.txt`), payload.stdout, 'utf8')
153
+ // the full session transcript in run-N.stderr.txt. A run whose tool
154
+ // boundary was never verified says so at the TOP of this file: it is
155
+ // what a grader opens first, and its answer is the fallback one.
156
+ const boundaryNotice = payload.toolValidation?.status === 'not-executed'
157
+ ? `[tool-boundary: NOT EXECUTED — ${payload.traceGap ?? 'no session trace materialized'}]\n\n`
158
+ : ''
159
+ if (payload.answer !== undefined) writeFileSync(join(output, `run-${attempt.index}.txt`), boundaryNotice + payload.answer, 'utf8')
160
+ else if (payload.stdout !== undefined) writeFileSync(join(output, `run-${attempt.index}.txt`), boundaryNotice + payload.stdout, 'utf8')
148
161
  if (payload.stdout !== undefined && payload.answer !== undefined && payload.stdout !== payload.answer) {
149
162
  writeFileSync(join(output, `run-${attempt.index}.stdout.txt`), payload.stdout, 'utf8')
150
163
  }
@@ -20,16 +20,17 @@ import { resolveDshCliChain } from '@catheadowl/dsh-eval/experimental'
20
20
  | `stageProfileStore` | 把真实 profile store junction 感知地暂存进沙箱 home(沙箱机制) |
21
21
  | `buildOverlayYaml` | 由片段拼装 dsh overlay YAML(整段发射器) |
22
22
  | `overlayDisableRows` | 生成 `disabled: true` 的行禁用 overlay 片段 |
23
- | `parseSessionLog` | 解析一条未压缩 JSONL session artifact 为 `{ header, events }` |
23
+ | `parseSessionLog` | 解析一条未压缩 JSONL session artifact 为 `{ header, events }`;header 的 `version` 戳不在已知代际集合内即抛错 |
24
24
  | `buildTrace` | 把 session 事件投影为 matcher 使用的 trace 对象 |
25
- | `loadTraceDir` | run 目录装载并解析 trace(无日志时返回 `undefined`) |
25
+ | `collectSessionTrace` | 收集一个 run trace 与「为什么没有 trace」的 seam 诊断:返回 `{ trace, gap }`,`gap` 文案含实际候选文件名与代际嫌疑(唯一的收集入口) |
26
+ | `KNOWN_SESSION_FORMAT_VERSIONS` | 本包接受的 session 格式代际集合(`parseSessionLog` 的准入面;宿主新增代际时与本包重验同步) |
26
27
  | `executeReviewExperiment` | 用给定 executor 执行抽象 review 实验 |
27
28
  | `materializeReviewExperiment` | 把实验定义物化为产物目录 |
28
29
  | `renderObservationSections` | 标准 observation renderer(自定义 executor 用) |
29
30
  | `OBSERVATIONS_PLACEHOLDER` | prompt 中的观测占位符常量(必须恰好出现一次) |
30
31
  | `createDshHeadlessReviewExecutor` | 构造 dsh headless review executor |
31
32
  | `runDshReviewExperiment` | 端到端跑一个 dsh review 实验 |
32
- | `validateToolBoundary` | 校验 trace 满足 turn-close 工具边界契约 |
33
+ | `validateToolBoundary` | 校验 trace 满足 turn-close 工具边界契约;结果带 `status`(`checked` / `not-executed`),无 trace 时**不是**通过 |
33
34
  | `renderToolBoundaryEvidence` | 渲染边界校验的机器可读证据 |
34
35
 
35
36
  新公开能力先进本入口;稳定后经明确决策才升入包根入口(升入即接受 semver 义务)。
@@ -1,5 +1,5 @@
1
1
  ---
2
- description: 安装与宿主接线——dsh-llm peer 的三形解析结局与 junction 步骤、构建 CLI 与 profile/凭证/spawn 三类运行前置
2
+ description: 安装与宿主接线——dsh-llm peer 的三形解析结局与 junction 步骤、构建 CLI 与 profile/凭证/spawn 三类运行前置、本包对宿主 session seam 的四处硬断言及其执法面(artifact 代际命名 / header 代际戳 / 拼接帧容器 / snapshotEvents 读取面),以及 mock 模式依赖的宿主 LLM adapter 线上契约(prepareCall,自带覆写对 peer 实例代差免疫)。
3
3
  ---
4
4
 
5
5
  # 安装与宿主接线
@@ -56,6 +56,31 @@ node -e "console.log(require('fs').existsSync('node_modules/@deepseek-ai/dsh/lib
56
56
 
57
57
  `false` = 解析层缺 CLI:先把上述 junction 重建为指向宿主检出;仍 `false` 则宿主检出未构建(先构建宿主)。这类 junction 维护是机器相关的开发环境事务,不入库,由各开发环境自行承接(同上文 peer 接线的 gitignore 纪律)。behavior 与 review 的真实运行都从定位到的 CLI spawn dsh 本体。
58
58
 
59
+ ## 宿主 session seam:本包硬断言的四处,坏了多是具名失败
60
+
61
+ behavior 与 review 的证据都取自**真实 dsh 会话的产物与进程内日志**,因此本包直接断言宿主的几处 session seam。它们随宿主演进时不会有编译期提示,所以每一处都配一行**执法面**(表里的符号就是);宿主检出更新后、动本包引用它们的文档前,先按本节对源码重新验证:
62
+
63
+ | 断言 | 宿主依据 | 本包执法面 | 坏了长什么样 |
64
+ |---|---|---|---|
65
+ | 会话 artifact 按**格式代**命名:v0 是 `session.jsonl`,之后每代带小写数字(当前 `session.v3.jsonl`);`compression: none` 时无 `.zstd` 后缀 | `session-persistence-jsonl/src/format.ts` 的 `generationLogFilename`,配 `core/session/src/types.ts` 的 `SESSION_FORMAT_VERSION` | `isSessionLogFilename`(命名判定)+ `collectSessionTrace`(收集)⇒ behavior 失败文案 / review 记账里的 `traceGap` | 只按 v0 名收集 ⇒ 一条日志都收不到;失败文案列出**实际扫到的候选文件名**并写明「宿主 artifact 命名可能已换代」,不再只报 `no session trace materialized` |
66
+ | 会话 header 的 `version` 戳是宿主对逻辑代际的声明(当前 v3) | 同上;已发布的代际链见 `session-format-catalog/src/generated.ts`(codecs v0–v3、`currentVersion: 3`) | `KNOWN_SESSION_FORMAT_VERSIONS`(`parseSessionLog` 入口准入) | 未知代际 ⇒ 解析当场拒绝并报出版本号(`session header version vN is not a known generation`),不再把各投影字段静默降级成空数组 |
67
+ | 会话日志是**拼接帧容器**(宿主默认 zstd),须逐帧扫描 | `session-persistence-jsonl/src/zstd.ts` 的帧扫描 | eval overlay 固定 `compression: none` + `packChunks: false`(`src/overlay.mjs`) | 整文件一次解压 ⇒ `ZSTD_error_prefix_unknown`(第二帧魔数被当输入) |
68
+ | 进程内读 durable 事件的 API 是 `Session#snapshotEvents()`(不可变冻结快照);早期的 `session.events` getter 已被删除 | `core/session/src/index.ts` 的 `snapshotEvents` | driver 行(`src/driver/multi-turn-driver.mjs`)直接调用,没有回退路径 | 属性访问得到 `undefined` ⇒ 进程内消费者抛 `agent.session.events is not iterable`,整个 headless run 直接死 |
69
+
70
+ 命名行与代际行的读取面由本包的 eval overlay 固定(`compression: none` + `packChunks: false`),所以每轮 run 的 artifact 是**明文逐事件**布局;命名判定、代际准入与收集入口都在 `src/trace.mjs`(`isSessionLogFilename` / `KNOWN_SESSION_FORMAT_VERSIONS` / `collectSessionTrace`),behavior runner 与 review adapter 共用同一个收集入口,缺 artifact 时各自把 `traceGap` 带进失败文案与产物记账。帧容器行走 overlay 固定;最后一行是 driver 行读日志时直接依赖的方法。
71
+
72
+ > **维护触发器**:宿主 session 格式、持久化命名或 `Session` 读取面变更 ⇒ 先按上表对 vendored 检出重新验证断言,再更新本篇与引用它们的源码/认知。前两行现在是**机械的**——命名或代际戳变了,跑一条 case 就红在具名文案上(候选文件名 / 版本号);后两行仍只有真跑一条 case 才会暴露。
73
+
74
+ ### 宿主 LLM adapter 线上契约(mock 模式专用)
75
+
76
+ mock 模式经 `eval-mock-llm` 插件(`src/mock/mock-adapter.mjs`)注册 `EvalMockAdapter`,它 `extends` 的 `LlmAdapter` 基类解析自**本包的 peer 实例**——该实例可以落后于驱动它的宿主运行时(实测:宿主 0.1.5-rc.2 的 LLM 服务对已注册 adapter 新增 `registration.adapter.prepareCall(...)` 调用面时,本地 peer 还是 0.0.1-rc.1,继承面缺失,全部 mock run 死在启动期)。因此适配器**自带** `prepareCall` 覆写、不依赖继承面在不在:形状镜像宿主基类默认契约(`PreparedAdapterCall`——`{ model: resolveModel(...), stream: options => this.stream(options) }`,model 元数据与派发入口绑定同一代适配器)。
77
+
78
+ | 断言 | 宿主依据 | 本包执法面 | 坏了长什么样 |
79
+ |---|---|---|---|
80
+ | 宿主 LLM 服务经 `prepareCall` 派发每次模型调用:adapter 级 `prepareCall(provider, model, signal)` 返回 `{ model, stream }`(一次性句柄,防 HMR 混代) | `packages/llm/llm/src/index.ts` 的 `registration.adapter.prepareCall(...)` 调用与 `LlmAdapter` 基类(`PreparedAdapterCall`) | `EvalMockAdapter` 自带 `prepareCall` 覆写(`src/mock/mock-adapter.mjs`)——真跑一条 mock case 即红在具名错误串上 | mock run 空转:无 session 事件、workspace 未落、final text 空,stderr 带 `registration.adapter.<method> is not a function` |
81
+
82
+ > **维护触发器**:宿主 adapter 线上契约演进(新增/改签名线上方法)⇒ 先对宿主 `packages/llm/llm/src/index.ts` 重验 `EvalMockAdapter` 的自带面(`prepareCall` / `resolveModel` / `stream` 的形状与语义),再同 commit 改本节与 `src/mock/mock-adapter.mjs`。
83
+
59
84
  ## 环境面:profile 与插件安装
60
85
 
61
86
  - 被测插件须已装进所选 profile:`dsh plugin --profile <profile> add <插件目录>`;
package/docs/matchers.md CHANGED
@@ -1,5 +1,5 @@
1
1
  ---
2
- description: trace matcher 与 mock helper 全集——工具面/文本面/输入面/派发面断言语义(toolCalled 到 subagentCompletedCount)与 toolCallStep/textStep 脚本构件
2
+ description: trace matcher 与 mock helper 全集——工具面/文本面/输入面/派发面断言语义(toolCalled 到 subagentCompletedCount)、投影普查(trace.census)与 toolCallStep/textStep 脚本构件
3
3
  ---
4
4
 
5
5
  # Trace matchers 与 mock helpers
@@ -19,6 +19,7 @@ description: trace matcher 与 mock helper 全集——工具面/文本面/输
19
19
  | `userMessages` | `{ seq, source, text }[]`(`source` 原样透传:任务 prompt `{ kind: 'user' }`,插件 steer `{ kind: 'plugin', plugin }`) |
20
20
  | `requestHeaders` | `{ seq, reason, system, toolNames }[]`(组装后 system prompt + 挂载工具名) |
21
21
  | `subagentChildren` | `{ sessionId, parentSession, delegationDepth, label, mode, provider, assistantTexts, finalText }[]`——每个子 agent 独立 session 日志一条;身份(label/mode/provider)取子日志首条 version-3 的 `subagent/descriptor` 事件(镜像宿主 `foldSubagentDescriptor` 的首条权威语义),`finalText` 是子会话自己的最后一条非空 assistant 文本(无则 `''` = 派发了但没答) |
22
+ | `census` | 投影普查(只报数,不判定):`{ eventTypeCounts, projectionLengths, projectionSkipped: { main, children }, projectionFieldGaps, subagent: { mainLogDescriptorEvents, supportedDescriptors, children } }`。语义见下「投影普查」节;手搓 trace(不经 `buildTrace`)时可为 `undefined` |
22
23
  | `sessions` / `sessionId` | 原始解析结果 `{ header, events }[]` 与主 session id |
23
24
 
24
25
  `runEvalCase` 返回的 `result.trace` 即此形状(无 session 日志时为 `undefined`;字段语义见 [runner-api.md](runner-api.md))。
@@ -55,6 +56,26 @@ description: trace matcher 与 mock helper 全集——工具面/文本面/输
55
56
 
56
57
  边界:子会话产物(独立 JSONL)经 `subagentChildren` 记录进入断言面(身份 + 子自身文本);子会话内部的工具调用**不**并入主投影的 `toolCalls`/`toolResults`(那属于主会话行为面),需要时经 `sessions` 原始日志自行投影。
57
58
 
59
+ ## 投影普查(`trace.census`)
60
+
61
+ 宽松投影(tolerant reader)的补救面:宿主事件 payload 演进时 `buildTrace` 不抛错,只把字段填成空值或丢掉整条记录。空投影会让负向断言真空通过——`toolNotCalled`、`userMessageTextExcludes`、以及 `subagentDispatchCount` / `subagentCompletedCount` 的 `expected === 0` 档都判 ok。**普查只报数,不判定**:它让「宿主日志里本来就没有这类事件」与「有事件但投影丢掉了」在报告里可分,是否降级由人判读。
62
+
63
+ 三个信号,对应三种坏法:
64
+
65
+ | 信号 | 看什么 |
66
+ |---|---|
67
+ | 主 session 事件(`eventTypeCounts` / `projectionLengths` / `projectionSkipped.main`) | 主日志(`buildTrace` 的投影输入)逐事件类型计数(任何类型,含插件扩展类型);五个投影的长度;以及**每个投影上「计数 − 长度 > 0」的差额**(`projectionSkipped.main`,按投影字段名)——记录被丢了的档 |
68
+ | **字段级缺口**(`projectionFieldGaps`) | 记录**留住了但字段读不到**的事件,按缺什么计数:`toolCallWithoutName` / `toolCallWithoutCallId` / `toolResultWithoutCallId` / `headerWithoutSystem` / `headerWithoutToolNames`。`tool/call`、`tool/result`、`request/header` 是 1:1 投影(计数 − 长度恒为 0),宿主搬字段时只在这里可见。**不计数**:`request/header` 的 `tools` 数组整个缺失(与真空列表投影一致) |
69
+ | 子会话(`census.subagent`) | `subagentChildren` 的输入面:`children[]` 逐条给该子日志的 `subagent/descriptor` 事件数、其中 `version === 3` 的条数(`supportedDescriptors`,**数事件不是数子会话**)**以及折叠出的身份**(`label` / `mode` / `provider`);`projectionSkipped.children` 两个身份计数——`withoutIdentity`(三项全缺)与 `withoutLabel`(`label` 缺,哪怕 mode/provider 有)。`mainLogDescriptorEvents` 是**主日志自己**的 `subagent/descriptor` 事件数(现宿主把 descriptor 写进子日志,这个数通常为 0)。**子日志不是主日志**,`eventTypeCounts` 不统计它们 |
70
+
71
+ 判读要点:
72
+
73
+ - **差额 ≠ 缺陷**。`assistant/message`、`user/message` 的**空文本消息是设计上整条丢弃**(保护「组装文本」投影语义),这类差额属合法,普查不替你做白名单;
74
+ - **两类信号别混**:`projectionSkipped.main` 看「记录被丢了」,`projectionFieldGaps` 看「记录在、字段没了」。后者正是 `toolNotCalled` 最危险的形态——调用记录还在、`name` 为 `undefined`,`nameMatches` 对任何 matcher 都不命中,负向断言照绿;
75
+ - **身份缺失型降级**:某子日志 `descriptorEvents > 0` 而 `supportedDescriptors === 0`,即它进了 `subagentChildren` 但身份全空;但**只要 `label` 缺**(`withoutLabel`),按 label 匹配的 `*Count(label, 0)` 就会真空通过——哪怕 `supportedDescriptors` 看起来健康、mode/provider 都在。两个计数分开报就是为了这个档;
76
+ - **子会话集合是启发式**:`subagentChildren` 收「header 带 `parentSession`」的日志,而宿主对 fork/resume/seed 日志也写这个字段——它们会以「无身份子记录」出现在普查里。这是集合的性质,不是本次降级(日志层无法复现宿主的 agent 链所有权判定);
77
+ - 只出现在**运行面**:`--format json` 的每条 case 记录(`census` 字段,pass 与 fail 都带;**无 trace 的记录没有**)与 `.runs/<id>/trace.json` 的 `trace.census`;**文本输出零新增**(逐字节输出契约不动),失败文案也不带计数。
78
+
58
79
  ## Mock script helpers
59
80
 
60
81
  - `toolCallStep(name, args)`:一步「模型调工具」,结束于 tool-calls;
package/docs/review.md CHANGED
@@ -55,6 +55,8 @@ dsh-review \
55
55
 
56
56
  真实运行的 reviewer 会话默认在**空白环境**启动:适配器先照常暂存所选 profile,再枚举它组合出的**全部树外插件行**(`package.json` 的 `dsh.profile.bundles` 中非 `@deepseek-ai/*` 的 bundle 各自 patch 文件里的行,加上 profile 自有 `cordis.patch.yml` 的行),在 `--patch` overlay 里逐行禁用——宿主 profile 装了什么 gates/插件都与 reviewer 无关,可复现性不再依赖「本机 profile 恰好干净」。白名单保留 reviewer 起不来就无测可言的接线行(`agent-default-model`、`session-title-llm`、`system-prompt`、`session-persistence-jsonl`);宿主模板工具行(`tool-fs`、shell、web、subagent 等)由静态清单继续禁用,cwd 指向空临时目录——reviewer 只能从物化的观测文本推理。运行后解析 session trace 的 `request/header` 事件做**工具边界校验**:发现任何非预期工具即视为 adapter failure(证据写入 `.runs/<id>/run-N.tool-boundary-evidence.json`)。
57
57
 
58
+ **校验没跑成也要记账**:收不到 session artifact(宿主 artifact 命名或会话格式换代是最常见的成因)时 `validateToolBoundary` 返回 `status: 'not-executed'` 而**不是**通过,适配器把该状态与 seam 诊断挂上执行结果;报告头写明 `tool boundary: NOT EXECUTED on run(s) N`,对应轮次的 `run-N.txt` 顶部与报告条目也各写一行原因,`run.json` 的 `toolBoundaries` 给机器读。这是有意的取向:**「没验证」必须看起来像没验证**——静默 fail-open 会让一份「工具边界从未检查过」的 review 产物读起来完全正常。
59
+
58
60
  **刻意复用宿主插件面**(例如要评审某插件自己的 gate 行为):加 `--keep-plugin-rows`——跳过树外行枚举,仅保留静态工具禁用,宿主 gates 恢复运行。
59
61
 
60
62
  > 注意:无 `id` 的组合条目对 id 定位的禁用天然不可见(宿主 loader 语义),本包的树外 bundle 生态均为带 id 行形态;发现无 id 树外行时以工具边界校验 fail-loud 兜底。白名单是**按行名**无条件保留——若某树外 bundle 刻意以白名单名(如 `system-prompt`)insert 自己的行,该行不会被禁(威胁模型是本机自己的 profile,非对抗面);此类泄漏同样由工具边界校验兜底。
@@ -65,9 +67,9 @@ dsh-review \
65
67
 
66
68
  - `observations.md`:本次实时物化的可见证据;
67
69
  - `task.txt`:实际发给每位 reviewer 的完整任务;
68
- - `run-N.txt`:该轮 reviewer 的**答案**——trace 推导(首条插件注入消息之前的最后一条 assistant 文本,抗回合尾部劫持),无 trace 时回落到 stdout 的最终消息;劫持发生时原始最终消息另存 `run-N.stdout.txt`;
70
+ - `run-N.txt`:该轮 reviewer 的**答案**——trace 推导(首条插件注入消息之前的最后一条 assistant 文本,抗回合尾部劫持),无 trace 时回落到 stdout 的最终消息;劫持发生时原始最终消息另存 `run-N.stdout.txt`;工具边界校验未执行时,本条文件顶部先写一行 `[tool-boundary: NOT EXECUTED — <原因>]`(答案本身退回 stdout 的最终消息);
69
71
  - `run-N.stderr.txt` / error:完整会话转录(判读回溯的 transcript 指针,报告每轮引用)/ 失败原因;
70
- - `run.json`:experiment、rubric、adapter、profile run 数;
72
+ - `run.json`:experiment、rubric、adapter、profile、run 数与每轮 `toolBoundaries` 记账(`checked` / `not-executed` + 原因);
71
73
  - `review-report.md`:判读报告骨架——机器字段自动填(experiment/adapter/ profile/runs、observations 指纹、rubric 位置、每轮 reviewer **答案**与 transcript 指针),三个 **人工判读栏目**留白待填:intentional design 命中项、新 red flag、下一步(改输出 / 改 rubric / 改 behavior case / 不处理)。刻意不做自动评分——review 层的价值在人工判断,报告只把判断物化成可归档、可对比的工程证据(dry-run 也会生成,runs 记 0)。
72
74
 
73
75
  ## 六条评审规则
@@ -17,10 +17,9 @@ const result = await runEvalCase(evalCase, { profile: 'headless', cliPath, mode:
17
17
  | 键 | 类型 | 语义 |
18
18
  |---|---|---|
19
19
  | `profile` | `string` | 必填。承载被测插件的 dsh profile(沙箱会暂存其 store,不污染真实 home)。 |
20
- | `cliPath` | `string` | 编译好的 dsh CLI 入口(`apps/cli/lib/bin.js`)绝对路径;优先于 `dshRepoDir`。程序化取值见下节。 |
20
+ | `cliPath` | `string` | 必填。编译好的 dsh CLI 入口(`apps/cli/lib/bin.js`)绝对路径(`resolveDshCliChain` 结果)。程序化取值见下节。 |
21
21
  | `mode` | `'real' \| 'mock'` | 覆写 case 自带的 mode;mock 需 `script.steps`。 |
22
22
  | `artifactsDir` | `string` | 提供则把 stdout/stderr/trace/session 日志拷贝到该目录(自动创建)。 |
23
- | `dshRepoDir` | `string` | **已弃用**:宿主 checkout 目录(从中拼出 CLI 路径)。下个 minor 删除——迁移到 `cliPath`。 |
24
23
 
25
24
  ## `cliPath` 从哪来:跨档关系(读我)
26
25
 
package/package.json CHANGED
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "name": "@catheadowl/dsh-eval",
3
3
  "description": "Agent eval framework over dsh headless runs: case runner, session-trace assertions, and a scripted mock-LLM layer for plugin intent tests.",
4
- "version": "0.2.1",
4
+ "version": "0.3.0",
5
5
  "license": "MIT",
6
6
  "repository": {
7
7
  "type": "git",
@@ -42,6 +42,6 @@
42
42
  "verify:experimental": "node scripts/verify-experimental-face.mjs",
43
43
  "verify:readme-i18n": "node scripts/verify-readme-i18n.mjs",
44
44
  "prepack": "node scripts/verify-manifest-face.mjs && node scripts/verify-publish-readiness.mjs && node scripts/verify-readme-i18n.mjs && node scripts/verify-experimental-face.mjs",
45
- "test": "node --test --test-isolation=none tests/trace.test.mjs tests/assertions.test.mjs tests/overlay.test.mjs tests/staging.test.mjs tests/runner.test.mjs tests/cli.test.mjs tests/discovery.test.mjs tests/review.test.mjs tests/report.test.mjs tests/config.test.mjs tests/review-report.test.mjs && node scripts/verify-publish-readiness.mjs && node scripts/verify-manifest-face.mjs && node scripts/verify-readme-i18n.mjs && node scripts/verify-experimental-face.mjs"
45
+ "test": "node --test --test-isolation=none tests/trace.test.mjs tests/assertions.test.mjs tests/overlay.test.mjs tests/staging.test.mjs tests/runner.test.mjs tests/cli.test.mjs tests/cli-report.test.mjs tests/discovery.test.mjs tests/review.test.mjs tests/tool-validation.test.mjs tests/report.test.mjs tests/config.test.mjs tests/review-report.test.mjs && node scripts/verify-publish-readiness.mjs && node scripts/verify-manifest-face.mjs && node scripts/verify-readme-i18n.mjs && node scripts/verify-experimental-face.mjs"
46
46
  }
47
47
  }
@@ -1,15 +1,14 @@
1
1
  /** dsh-headless execution adapter for model-independent review experiments. */
2
2
 
3
- import { existsSync, mkdtempSync, writeFileSync } from 'node:fs'
3
+ import { mkdtempSync, writeFileSync } from 'node:fs'
4
4
  import { tmpdir } from 'node:os'
5
5
  import { join, resolve } from 'node:path'
6
6
  import { executeReviewExperiment } from '../../experiment/review.mjs'
7
- import { CLI_RELATIVE_PATH } from '../../cli.mjs'
8
7
  import { overlayDisableRows } from '../../overlay.mjs'
9
8
  import {
10
9
  resolveRealDshHome, stageSandboxHome, stagedPluginRows, teardownSandbox, spawnHeadlessDsh,
11
10
  } from '../../sandbox.mjs'
12
- import { loadTraceDir } from '../../trace.mjs'
11
+ import { collectSessionTrace } from '../../trace.mjs'
13
12
  import { validateToolBoundary, renderToolBoundaryEvidence } from '../../tool-validation.mjs'
14
13
 
15
14
  /**
@@ -70,23 +69,14 @@ function buildReviewOverlayYaml(pluginRows, { keepPluginRows }) {
70
69
  return overlayDisableRows([...disabled])
71
70
  }
72
71
 
73
- /** Resolve and validate the compiled dsh CLI entry point. */
74
- export function resolveDshCli(dshRepoDir) {
75
- const repoDir = resolve(dshRepoDir)
76
- const cli = join(repoDir, ...CLI_RELATIVE_PATH.split(/[\\/]/))
77
- if (!existsSync(cli)) {
78
- throw new Error(`no compiled dsh CLI at '${cli}' (build deepseek-harness first)`)
79
- }
80
- return cli
81
- }
82
-
83
- /** The CLI entry for an executor: explicit cliPath (C6 chain result) wins;
84
- * otherwise fall back to the legacy repo form. Neither being set is a caller
85
- * bug the CLI bins already catch — this guard serves direct API consumers. */
72
+ /** The CLI entry for an executor: the explicit cliPath (a `resolveDshCliChain`
73
+ * result). Its absence is a caller bug the CLI bins already catch — this
74
+ * guard serves direct API consumers. */
86
75
  function executorCli(options) {
87
- if (options.cliPath !== undefined) return resolve(options.cliPath)
88
- if (options.dshRepoDir !== undefined) return resolveDshCli(options.dshRepoDir)
89
- throw new Error('review adapter needs a CLI location: pass cliPath (C6 chain result) or dshRepoDir')
76
+ if (options.cliPath === undefined) {
77
+ throw new Error('review adapter needs options.cliPath (a resolveDshCliChain result)')
78
+ }
79
+ return resolve(options.cliPath)
90
80
  }
91
81
 
92
82
  /**
@@ -162,39 +152,43 @@ export function createDshHeadlessReviewExecutor(options) {
162
152
 
163
153
  // Post-run tool boundary check: parse the session
164
154
  // trace, verify no unexpected tools were mounted in the reviewer's
165
- // session, fail the run on violation. An absent session log skips
166
- // the check gracefully (accepted fail-open).
155
+ // session, fail the run on violation. A missing session log does NOT
156
+ // skip the check silently: `validateToolBoundary` reports
157
+ // `status: 'not-executed'`, and the adapter carries that fact (plus the
158
+ // seam diagnosis) on the result so the report and run artifacts state
159
+ // that the boundary guarantee was not verified (EVAL-021).
167
160
  // Validation inspects the main session only (buildTrace selects
168
161
  // non-subagent logs); plugin tools leaking in a subagent session
169
162
  // would not be caught — irrelevant in review where the overlay
170
163
  // disables every subagent tool row.
171
- const trace = loadTraceDir(join(dshHome, 'sessions'))
172
- if (trace) {
173
- const validation = validateToolBoundary(trace, { allowedTools })
174
- result.toolValidation = validation
175
- if (!validation.ok) {
176
- // Attach evidence for the caller to persist (the adapter's
177
- // runDir is ephemeral — removed by the finally block). The
178
- // bin writes this to `.runs/<id>/tool-boundary-evidence.json`.
179
- result.toolBoundaryEvidence = renderToolBoundaryEvidence(validation, { runDir, profile })
180
- const boundaryError = new Error(
181
- `tool boundary violation: unexpected tools [${validation.unexpected.join(', ')}]`,
182
- )
183
- boundaryError.result = result
184
- throw boundaryError
185
- }
186
- // The ANSWER to the task, not the last message: stdout carries the
187
- // headless CLI's final assistant message — whatever the reviewer
188
- // said LAST. If any tail interaction intervened (a turn-close gate
189
- // splice that slipped past the blank environment, an infra
190
- // complaint), stdout holds that instead of the analysis. The
191
- // trace's answerText (last assistant text before the first
192
- // plugin-sourced injection) IS the analysis; stdout remains the
193
- // fallback for trace-less runs.
194
- result.answer = trace.answerText !== '' ? trace.answerText : stdout
195
- } else {
196
- result.answer = stdout
164
+ const { trace, gap } = collectSessionTrace(join(dshHome, 'sessions'))
165
+ const validation = validateToolBoundary(trace, { allowedTools })
166
+ result.toolValidation = validation
167
+ if (validation.status === 'not-executed') {
168
+ // `traceGap` mirrors the behavior runner's field name for the same
169
+ // seam diagnosis; the bin writes both into run-N.txt and the report.
170
+ result.traceGap = gap
171
+ }
172
+ if (validation.status === 'checked' && !validation.ok) {
173
+ // Attach evidence for the caller to persist (the adapter's
174
+ // runDir is ephemeral removed by the finally block). The
175
+ // bin writes this to `.runs/<id>/tool-boundary-evidence.json`.
176
+ result.toolBoundaryEvidence = renderToolBoundaryEvidence(validation, { runDir, profile })
177
+ const boundaryError = new Error(
178
+ `tool boundary violation: unexpected tools [${validation.unexpected.join(', ')}]`,
179
+ )
180
+ boundaryError.result = result
181
+ throw boundaryError
197
182
  }
183
+ // The ANSWER to the task, not the last message: stdout carries the
184
+ // headless CLI's final assistant message — whatever the reviewer
185
+ // said LAST. If any tail interaction intervened (a turn-close gate
186
+ // splice that slipped past the blank environment, an infra
187
+ // complaint), stdout holds that instead of the analysis. The
188
+ // trace's answerText (last assistant text before the first
189
+ // plugin-sourced injection) IS the analysis; stdout remains the
190
+ // fallback for trace-less runs (recorded by `traceGap` above).
191
+ result.answer = trace !== undefined && trace.answerText !== '' ? trace.answerText : stdout
198
192
 
199
193
  return result
200
194
  } finally {
@@ -60,7 +60,10 @@ function isSubagentSession(session) {
60
60
  function summarize(agent) {
61
61
  let text = ''
62
62
  let reason
63
- for (const event of agent.session.events) {
63
+ // `snapshotEvents()` replaced the removed `session.events` getter upstream
64
+ // (5660f44d29); reading the dead property throws "agent.session.events is not
65
+ // iterable" and kills the whole headless run.
66
+ for (const event of agent.session.snapshotEvents()) {
64
67
  if (event.type === 'assistant/message') {
65
68
  const joined = event.data.message.content
66
69
  .filter(block => block.type === 'text')
@@ -16,7 +16,12 @@ export { stageProfileStore } from './sandbox.mjs'
16
16
  export { buildOverlayYaml, overlayDisableRows } from './overlay.mjs'
17
17
 
18
18
  // --- session-trace primitives ---
19
- export { parseSessionLog, buildTrace, loadTraceDir } from './trace.mjs'
19
+ export {
20
+ parseSessionLog,
21
+ buildTrace,
22
+ collectSessionTrace,
23
+ KNOWN_SESSION_FORMAT_VERSIONS,
24
+ } from './trace.mjs'
20
25
 
21
26
  // --- review experiment execution layer ---
22
27
  export {
@@ -40,6 +40,20 @@ class EvalMockAdapter extends LlmAdapter {
40
40
  }
41
41
  }
42
42
 
43
+ // Carried explicitly instead of inherited: this plugin's `LlmAdapter` base
44
+ // resolves from THIS package's peer instance, which can lag the host
45
+ // runtime that drives it (host 0.1.5-rc.2 grew `prepareCall` while the
46
+ // local peer was 0.0.1-rc.1 — the inherited face was missing and every mock
47
+ // run died at `registration.adapter.prepareCall is not a function`). The
48
+ // shape is the host base-class default: model metadata plus a dispatch
49
+ // entry bound to this same adapter generation.
50
+ async prepareCall(provider, model, signal) {
51
+ return {
52
+ model: await this.resolveModel(provider, model, signal),
53
+ stream: options => this.stream(options),
54
+ }
55
+ }
56
+
43
57
  async * stream(_options) {
44
58
  const step = this.steps[this.cursor]
45
59
  this.cursor += 1
package/src/report.mjs CHANGED
@@ -30,6 +30,10 @@
30
30
  * @param {boolean} [parts.timedOut]
31
31
  * @param {number} [parts.durationMs] - wall time of the run, when it ran.
32
32
  * @param {string} [parts.artifactsDir] - where post-mortem artifacts landed, when written.
33
+ * @param {object} [parts.census] - the run trace's projection census
34
+ * (`trace.census`), carried on both pass and fail records so a green case
35
+ * whose evidence surface degraded is still inspectable after the fact.
36
+ * Absent when no trace materialized.
33
37
  */
34
38
  export function createCaseRecord(parts) {
35
39
  const record = { id: parts.id, file: parts.file }
@@ -41,6 +45,7 @@ export function createCaseRecord(parts) {
41
45
  if (parts.timedOut !== undefined) record.timedOut = parts.timedOut
42
46
  if (parts.durationMs !== undefined) record.durationMs = parts.durationMs
43
47
  if (parts.artifactsDir !== undefined) record.artifactsDir = parts.artifactsDir
48
+ if (parts.census !== undefined) record.census = parts.census
44
49
  return record
45
50
  }
46
51
 
@@ -23,6 +23,29 @@ export function observationsFingerprint(observations) {
23
23
  return createHash('sha256').update(observations, 'utf8').digest('hex').slice(0, 16)
24
24
  }
25
25
 
26
+ /**
27
+ * One-line statement of whether the tool boundary check RAN on the review
28
+ * runs. A run without a session artifact reports `status: 'not-executed'`;
29
+ * that must show up in the report header, because a review whose tool face was
30
+ * never verified is weaker evidence than one whose was (EVAL-021). Executors
31
+ * that do not validate at all (custom executors, dry runs) say so instead of
32
+ * implying a pass.
33
+ */
34
+ function toolBoundarySummary(attempts) {
35
+ const skipped = attempts.filter(attempt => attempt.result?.toolValidation?.status === 'not-executed')
36
+ if (skipped.length > 0) {
37
+ return `NOT EXECUTED on run(s) ${skipped.map(attempt => attempt.index).join(', ')}`
38
+ + ' — no session artifact; the reviewer tool face was not verified (see the per-run notes)'
39
+ }
40
+ const checked = attempts.filter(attempt => attempt.result?.toolValidation?.status === 'checked')
41
+ if (attempts.length > 0 && checked.length === attempts.length) return 'checked on every run'
42
+ if (checked.length > 0) {
43
+ return `checked on ${checked.length} of ${attempts.length} run(s); the rest reported none`
44
+ + ' (failed before validation, or a non-verifying executor)'
45
+ }
46
+ return 'not reported by this executor'
47
+ }
48
+
26
49
  /**
27
50
  * Render the review report markdown.
28
51
  *
@@ -54,6 +77,7 @@ export function renderReviewReport(parts) {
54
77
  lines.push(`- adapter: ${parts.adapter ?? 'none (dry run)'}`)
55
78
  if (parts.profile !== undefined) lines.push(`- profile: \`${parts.profile}\``)
56
79
  lines.push(`- runs: ${dry ? '0 (dry run — observations materialized only)' : result.runs}`)
80
+ if (!dry) lines.push(`- tool boundary: ${toolBoundarySummary(result.attempts)}`)
57
81
  lines.push(`- observations: \`observations.md\` (sha256:${observationsFingerprint(observations)})`)
58
82
  lines.push(`- rubric: ${rubric.includes('\n') ? '(inline string — see experiment definition)' : `\`${rubric}\``}`)
59
83
  lines.push('')
@@ -82,6 +106,13 @@ export function renderReviewReport(parts) {
82
106
  && attempt.result.answer !== attempt.result.stdout
83
107
  lines.push('')
84
108
  lines.push(`- transcript: \`run-${attempt.index}.stderr.txt\`${diverged ? ` (answer differs from the final message — see \`run-${attempt.index}.stdout.txt\`)` : ''}`)
109
+ if (attempt.result?.toolValidation?.status === 'not-executed') {
110
+ // The unverified-boundary fact rides the run it applies to, next to
111
+ // the answer it qualifies: this answer is stdout's final message
112
+ // (there was no trace to derive it from) and no request header was
113
+ // ever inspected for tool leakage.
114
+ lines.push(`- **tool-boundary check NOT EXECUTED**: ${attempt.result.traceGap ?? 'no session trace materialized'} — the answer above is stdout's final message, not a trace-derived answer.`)
115
+ }
85
116
  } else {
86
117
  lines.push(`### run ${attempt.index} — FAIL`)
87
118
  lines.push('')
package/src/runner.mjs CHANGED
@@ -29,11 +29,11 @@
29
29
  * module is the orchestration only.
30
30
  */
31
31
 
32
- import { existsSync, mkdirSync, mkdtempSync, writeFileSync, cpSync, readdirSync, readFileSync } from 'node:fs'
32
+ import { existsSync, mkdirSync, mkdtempSync, readFileSync, writeFileSync, cpSync } from 'node:fs'
33
33
  import { tmpdir } from 'node:os'
34
34
  import { isAbsolute, join, resolve } from 'node:path'
35
35
  import { fileURLToPath } from 'node:url'
36
- import { loadTraceDir } from './trace.mjs'
36
+ import { collectSessionTrace, listSessionLogFiles } from './trace.mjs'
37
37
  import { validateRowConfig, validateDisableRows, validateFollowups } from './discovery.mjs'
38
38
  import { CLI_RELATIVE_PATH } from './cli.mjs'
39
39
  import { buildOverlayYaml } from './overlay.mjs'
@@ -62,21 +62,18 @@ const FRAMEWORK_ROOT = fileURLToPath(new URL('..', import.meta.url))
62
62
  * @param {object} evalCase - the case under test.
63
63
  * @param {object} options
64
64
  * @param {string} options.profile - the dsh profile booting the run (plugin installed there).
65
- * @param {string} [options.dshRepoDir] - the deepseek-harness checkout (legacy CLI
66
- * location; ignored when cliPath is given).
67
- * @deprecated options.dshRepoDir — pass the C6 chain result via cliPath
68
- * instead; this legacy option is removed in the next minor release.
69
- * @param {string} [options.cliPath] - explicit compiled CLI entry (C6 chain result;
70
- * takes precedence over dshRepoDir).
65
+ * @param {string} options.cliPath - the compiled dsh CLI entry (a
66
+ * `resolveDshCliChain` result); required.
71
67
  * @param {'real' | 'mock'} [options.mode] - force a mode over the case's own.
72
68
  * @param {string} [options.artifactsDir] - copy stdout/stderr/trace/session logs here (created).
73
69
  * @returns {Promise<EvalRunResult>}
74
70
  */
75
71
  export async function runEvalCase(evalCase, options) {
76
72
  const mode = options.mode ?? evalCase.mode ?? 'real'
77
- const binPath = options.cliPath !== undefined
78
- ? resolve(options.cliPath)
79
- : join(resolve(options.dshRepoDir), ...CLI_RELATIVE_PATH.split(/[\\/]/))
73
+ if (options.cliPath === undefined) {
74
+ throw new TypeError('runEvalCase needs options.cliPath (a resolveDshCliChain result)')
75
+ }
76
+ const binPath = resolve(options.cliPath)
80
77
  const timeoutMs = evalCase.timeoutMs ?? 180_000
81
78
 
82
79
  const runDir = mkdtempSync(join(tmpdir(), 'dsh-eval-'))
@@ -150,7 +147,7 @@ export async function runEvalCase(evalCase, options) {
150
147
  timeoutMs,
151
148
  })
152
149
 
153
- const trace = loadTraceDir(sessionsRoot)
150
+ const { trace, gap: traceGap } = collectSessionTrace(sessionsRoot)
154
151
  const sessionLogs = collectSessionLogTexts(sessionsRoot)
155
152
 
156
153
  // Workspace assertions live HERE, before the run dir cleanup: a case's
@@ -184,7 +181,7 @@ export async function runEvalCase(evalCase, options) {
184
181
 
185
182
  return {
186
183
  caseId: evalCase.id, mode, task: evalCase.task, exitCode, timedOut,
187
- stdout, stderr, trace, sessionLogs, inspectError, runDir,
184
+ stdout, stderr, trace, traceGap, sessionLogs, inspectError, runDir,
188
185
  }
189
186
  } finally {
190
187
  teardownSandbox(runDir, { keep: process.env.DSH_EVAL_KEEP_TMP === '1' })
@@ -193,22 +190,7 @@ export async function runEvalCase(evalCase, options) {
193
190
 
194
191
  /** Read every session artifact under the root as text (best-effort, pre-cleanup). */
195
192
  function collectSessionLogTexts(sessionsRoot) {
196
- const texts = []
197
- const walk = (dir) => {
198
- let entries
199
- try {
200
- entries = readdirSync(dir, { withFileTypes: true })
201
- } catch {
202
- return
203
- }
204
- for (const entry of entries) {
205
- const path = join(dir, entry.name)
206
- if (entry.isDirectory()) walk(path)
207
- else if (entry.name === 'session.jsonl') texts.push(readFileSync(path, 'utf8'))
208
- }
209
- }
210
- walk(sessionsRoot)
211
- return texts
193
+ return listSessionLogFiles(sessionsRoot).map(path => readFileSync(path, 'utf8'))
212
194
  }
213
195
 
214
196
  /**
@@ -221,6 +203,9 @@ function collectSessionLogTexts(sessionsRoot) {
221
203
  * @property {string} stdout - printed final assistant text (plus any startup chatter).
222
204
  * @property {string} stderr
223
205
  * @property {import('./trace.mjs').EvalTrace | undefined} trace
206
+ * @property {string | undefined} traceGap - why no trace was built (the host
207
+ * session seam diagnosis from `collectSessionTrace`); `undefined` whenever
208
+ * `trace` is defined. The CLI prints it as the failure text.
224
209
  * @property {string[]} sessionLogs - raw session artifact texts, pre-cleanup.
225
210
  * @property {string | undefined} inspectError - the case's `inspect` failure text, when it threw.
226
211
  * @property {string} runDir - removed unless DSH_EVAL_KEEP_TMP=1.
@@ -37,19 +37,27 @@ function collectMountedToolNames(trace) {
37
37
  * against the allowed set. An empty allowed set (the review default)
38
38
  * means the reviewer must see no tools at all.
39
39
  *
40
- * @param {import('./trace.mjs').EvalTrace | undefined} trace - the parsed trace; `undefined` skips validation.
40
+ * The result is explicit about whether the check RAN: `status` is
41
+ * `'checked'` only when a trace was available, and `'not-executed'` when
42
+ * there was none. `ok` is true only for a checked boundary with no
43
+ * unexpected tool — a missing trace is NOT a pass, so callers must branch
44
+ * on `status` and account for the skip (see `createDshHeadlessReviewExecutor`
45
+ * and docs/review.md) instead of reading `ok` alone.
46
+ *
47
+ * @param {import('./trace.mjs').EvalTrace | undefined} trace - the parsed trace; `undefined` means the check could not run.
41
48
  * @param {{ allowedTools?: Set<string> }} [options]
42
- * @returns {{ ok: boolean, unexpected: string[], actual: string[], allowed: string[] }}
49
+ * @returns {{ status: 'checked' | 'not-executed', ok: boolean, unexpected: string[], actual: string[], allowed: string[] }}
43
50
  */
44
51
  export function validateToolBoundary(trace, options = {}) {
45
52
  const allowed = options.allowedTools ?? new Set()
46
53
  const allowedNames = [...allowed].sort()
47
54
  if (trace === undefined || trace === null) {
48
- return { ok: true, unexpected: [], actual: [], allowed: allowedNames }
55
+ return { status: 'not-executed', ok: false, unexpected: [], actual: [], allowed: allowedNames }
49
56
  }
50
57
  const actual = collectMountedToolNames(trace)
51
58
  const unexpected = actual.filter(name => !allowed.has(name))
52
59
  return {
60
+ status: 'checked',
53
61
  ok: unexpected.length === 0,
54
62
  unexpected,
55
63
  actual,
@@ -61,7 +69,7 @@ export function validateToolBoundary(trace, options = {}) {
61
69
  * Render a diagnostic evidence document for a tool boundary failure.
62
70
  * Suitable for writing to `.runs/<id>/tool-boundary-evidence.json`.
63
71
  *
64
- * @param {{ ok: boolean, unexpected: string[], actual: string[], allowed: string[] }} validation
72
+ * @param {{ status: 'checked' | 'not-executed', ok: boolean, unexpected: string[], actual: string[], allowed: string[] }} validation
65
73
  * @param {{ runDir: string, profile: string }} context
66
74
  * @returns {string}
67
75
  */
package/src/trace.mjs CHANGED
@@ -6,14 +6,53 @@
6
6
  * Event shapes follow `deepseek-harness/packages/core/session/src/types.ts`
7
7
  * (`SessionEventMap`); packed `*-chunks` storage rows are tolerated and
8
8
  * skipped — they only carry `assistant/chunk` deltas eval never asserts on.
9
+ *
10
+ * Both seam directions carry an explicit boundary: an artifact whose header
11
+ * stamp is not a known generation is refused here (`parseSessionLog`), and a
12
+ * collection that finds no artifact yields a named diagnosis rather than an
13
+ * unexplained `undefined` (`collectSessionTrace`). See docs/host-wiring.md.
9
14
  */
10
15
 
11
16
  import { readdirSync, readFileSync } from 'node:fs'
12
- import { join } from 'node:path'
17
+ import { basename, join } from 'node:path'
13
18
 
14
19
  /** Storage row types that pack `assistant/chunk` delta runs (see chunk-rows.ts). */
15
20
  const CHUNK_ROW_TYPES = new Set(['text-chunks', 'reasoning-chunks', 'tool-call-chunks'])
16
21
 
22
+ /** Projection field name per projected event type. */
23
+ const PROJECTED_FIELD_BY_EVENT_TYPE = new Map([
24
+ ['tool/call', 'toolCalls'],
25
+ ['tool/result', 'toolResults'],
26
+ ['assistant/message', 'assistantTexts'],
27
+ ['user/message', 'userMessages'],
28
+ ['request/header', 'requestHeaders'],
29
+ ])
30
+
31
+ /**
32
+ * Session format generations this parser accepts. Mirror of the generation
33
+ * chain the vendored host ships codecs for
34
+ * (`session-format-catalog/src/generated.ts`: codecs v0–v3,
35
+ * `currentVersion: 3` = `SESSION_FORMAT_VERSION` in
36
+ * `core/session/src/types.ts`). The projection is written and verified
37
+ * against the current generation; older ones parse tolerantly. A stamp
38
+ * outside this set means the host moved to a generation whose payload this
39
+ * projection was never verified against — fail at the seam instead of
40
+ * projecting empty fields, and bump this set only together with the
41
+ * re-verification the docs' maintenance trigger describes.
42
+ */
43
+ export const KNOWN_SESSION_FORMAT_VERSIONS = new Set([0, 1, 2, 3])
44
+
45
+ /** Known generations rendered for an error message: `v0, v1, v2, v3`. */
46
+ function knownGenerationsLabel() {
47
+ return [...KNOWN_SESSION_FORMAT_VERSIONS].sort((a, b) => a - b).map(version => `v${version}`).join(', ')
48
+ }
49
+
50
+ /** One header version stamp, rendered compactly for a diagnostic. */
51
+ function headerVersionLabel(version) {
52
+ if (typeof version === 'number') return `v${version}`
53
+ return `(${JSON.stringify(version ?? null)})`
54
+ }
55
+
17
56
  /**
18
57
  * Parse one uncompressed JSONL session artifact.
19
58
  * @param {string} text - the artifact's full text (header line first).
@@ -24,6 +63,15 @@ export function parseSessionLog(text) {
24
63
  if (lines.length === 0) throw new Error('empty session log')
25
64
  const header = JSON.parse(lines[0])
26
65
  if (header.type !== 'session') throw new Error('first line is not a session header')
66
+ // Generation gate: the header stamp is the host's own declaration of the
67
+ // artifact's logical layout. An unknown one is a seam drift, not a parse
68
+ // detail — say so here rather than degrade every projection to empty.
69
+ if (!KNOWN_SESSION_FORMAT_VERSIONS.has(header.version)) {
70
+ throw new Error(
71
+ `session header version ${headerVersionLabel(header.version)} is not a known generation`
72
+ + ` (known: ${knownGenerationsLabel()}); the host session format may have changed generation`,
73
+ )
74
+ }
27
75
  const events = []
28
76
  for (const line of lines.slice(1)) {
29
77
  let record
@@ -90,30 +138,49 @@ function parseArguments(raw) {
90
138
  }
91
139
 
92
140
  /**
93
- * Project one subagent child log into an assertable record. The durable
94
- * identity (`label` / `mode` / `provider`) comes from the FIRST
95
- * `subagent/descriptor` event whose payload carries the descriptor version
96
- * this projection supports (3) mirroring `foldSubagentDescriptor` in
97
- * `deepseek-harness/packages/subagent/subagent/src/descriptor.ts`, where the
98
- * establishing provider appends exactly one authoritative descriptor and
99
- * later events cannot rewrite it. Completion is the child's own last
100
- * assistant text a child that produced none may have been dispatched but
101
- * never ran to an answer (turn/end reasons are not consulted).
141
+ * Fold one child log's descriptor events exactly once: the identity
142
+ * `projectChild` asserts on, plus the counts the census reports. Single source
143
+ * on purpose — `projectChild` and `censusForChild` must agree on which
144
+ * descriptor established the identity, and the "supported" predicate must match
145
+ * the fold (a log whose only descriptors are unsupported yields both an empty
146
+ * identity and `supportedDescriptors: 0`, which is the census signal).
147
+ * @param {{ events: object[] }} log - one parsed child log.
148
+ * @returns {{ label: string | undefined, mode: string | undefined, provider: string | undefined, descriptorEvents: number, supportedDescriptors: number }}
102
149
  */
103
- function projectChild(log) {
150
+ function foldChildDescriptor(log) {
104
151
  let label
105
152
  let mode
106
153
  let provider
154
+ let descriptorEvents = 0
155
+ let supportedDescriptors = 0
107
156
  for (const event of log.events) {
108
157
  if (event.type !== 'subagent/descriptor') continue
158
+ descriptorEvents += 1
109
159
  const data = event.data
110
160
  if (data === null || typeof data !== 'object') continue
111
- if (label !== undefined || mode !== undefined || provider !== undefined) break
161
+ if (data.version === 3) supportedDescriptors += 1
162
+ if (label !== undefined || mode !== undefined || provider !== undefined) continue
112
163
  if (data.version !== 3) continue
113
164
  if (typeof data.label === 'string') label = data.label
114
165
  if (typeof data.mode === 'string') mode = data.mode
115
166
  if (typeof data.provider === 'string') provider = data.provider
116
167
  }
168
+ return { label, mode, provider, descriptorEvents, supportedDescriptors }
169
+ }
170
+
171
+ /**
172
+ * Project one subagent child log into an assertable record. The durable
173
+ * identity (`label` / `mode` / `provider`) comes from the FIRST
174
+ * `subagent/descriptor` event whose payload carries the descriptor version
175
+ * this projection supports (3) — mirroring `foldSubagentDescriptor` in
176
+ * `deepseek-harness/packages/subagent/subagent/src/descriptor.ts`, where the
177
+ * establishing provider appends exactly one authoritative descriptor and
178
+ * later events cannot rewrite it. Completion is the child's own last
179
+ * assistant text — a child that produced none may have been dispatched but
180
+ * never ran to an answer (turn/end reasons are not consulted).
181
+ */
182
+ function projectChild(log) {
183
+ const { label, mode, provider } = foldChildDescriptor(log)
117
184
  const assistantTexts = log.events
118
185
  .filter(event => event.type === 'assistant/message')
119
186
  .map(event => messageText(event.data.message))
@@ -130,6 +197,109 @@ function projectChild(log) {
130
197
  }
131
198
  }
132
199
 
200
+ /**
201
+ * Count events by type, plug-in event types included (the only live registry is
202
+ * `SessionEventMap`, so no closed list exists).
203
+ * @param {object[]} events - parsed event records.
204
+ * @returns {Record<string, number>} count per event type, insertion-ordered.
205
+ */
206
+ function countEventTypes(events) {
207
+ // Null-prototype accumulator: a plug-in event type may name an
208
+ // `Object.prototype` member (`constructor`, `toString`, `__proto__`), and
209
+ // `counts['constructor'] ?? 0` would otherwise read the inherited function
210
+ // and string-concatenate, while `__proto__` would be swallowed by its setter.
211
+ const counts = Object.create(null)
212
+ for (const event of events) {
213
+ if (typeof event?.type !== 'string') continue
214
+ counts[event.type] = (counts[event.type] ?? 0) + 1
215
+ }
216
+ return counts
217
+ }
218
+
219
+ /**
220
+ * Census for one candidate child log: whether its `subagent/descriptor`
221
+ * events exist, how many, and how many carry the supported descriptor
222
+ * version, plus the folded identity itself. `label` is the field the
223
+ * subagent-count matchers key on, so `label: undefined` with a non-zero
224
+ * `descriptorEvents` is exactly the shape whose `*Count(label, 0)` assertion
225
+ * is green only because there was nothing to match — the identity-loss
226
+ * degradation the census exists to make visible.
227
+ * @param {{ header: object, events: object[] }} log - one parsed child candidate.
228
+ * @returns {{ sessionId: string | undefined, parentSession: string | undefined, delegationDepth: number | undefined, descriptorEvents: number, supportedDescriptors: number, label: string | undefined, mode: string | undefined, provider: string | undefined }}
229
+ */
230
+ function censusForChild(log) {
231
+ return {
232
+ sessionId: log.header.id,
233
+ parentSession: log.header.parentSession,
234
+ delegationDepth: log.header.delegationDepth,
235
+ ...foldChildDescriptor(log),
236
+ }
237
+ }
238
+
239
+ /**
240
+ * Projection census for one built trace (see {@link EvalTrace.census}).
241
+ * Numbers only: this reports what the raw logs contained against what the
242
+ * projections kept, and never decides whether the difference is a defect.
243
+ *
244
+ * Three signals, because one number cannot cover three shapes: length
245
+ * differences (`projectionSkipped.main`) see records the projection dropped,
246
+ * `projectionFieldGaps` sees records it kept while a field went missing
247
+ * (`tool/call` and friends project 1:1, so their count − length is structurally
248
+ * zero), and the subagent block sees the child-log data source, which no main-log
249
+ * count can reach.
250
+ *
251
+ * @param {object[]} events - the MAIN log's events (the projection input).
252
+ * @param {EvalTrace} trace - the built trace, read for projection lengths.
253
+ * @param {object[]} childLogs - candidate child logs entering `subagentChildren`.
254
+ * @param {Record<string, number>} projectionFieldGaps - gaps the projection loop
255
+ * recorded while reading fields, keyed by what was missing.
256
+ * @returns {object} the census record.
257
+ */
258
+ function buildCensus(events, trace, childLogs, projectionFieldGaps) {
259
+ const eventTypeCounts = countEventTypes(events)
260
+ const projectionLengths = {
261
+ toolCalls: trace.toolCalls.length,
262
+ toolResults: trace.toolResults.length,
263
+ assistantTexts: trace.assistantTexts.length,
264
+ userMessages: trace.userMessages.length,
265
+ requestHeaders: trace.requestHeaders.length,
266
+ }
267
+ const projectionSkipped = { main: {}, children: {} }
268
+ for (const [type, field] of PROJECTED_FIELD_BY_EVENT_TYPE) {
269
+ const missing = (eventTypeCounts[type] ?? 0) - projectionLengths[field]
270
+ if (missing > 0) projectionSkipped.main[field] = missing
271
+ }
272
+ const children = childLogs.map(censusForChild)
273
+ const supportedDescriptors = children.reduce((total, child) => total + child.supportedDescriptors, 0)
274
+ // Two degradation shapes, different signals: `withoutIdentity` is a child
275
+ // that folded no identity at all; `withoutLabel` is one that folded some
276
+ // identity but no label — the only field the `subagent*Count` matchers can
277
+ // match on, so its zero-count assertions are the vacuous ones.
278
+ //
279
+ // The child set is the parentSession heuristic (any log whose header carries
280
+ // `parentSession`, which the host also writes for fork/resume/seed logs), so a
281
+ // non-subagent fork log shows up here as an identity-less child. The census
282
+ // reports the set it was given; it cannot re-derive the host's agent-chain
283
+ // ownership check from a log alone.
284
+ const withoutIdentity = children.filter(
285
+ child => child.label === undefined && child.mode === undefined && child.provider === undefined,
286
+ ).length
287
+ const withoutLabel = children.filter(child => child.label === undefined).length
288
+ if (withoutIdentity > 0) projectionSkipped.children.withoutIdentity = withoutIdentity
289
+ if (withoutLabel > 0) projectionSkipped.children.withoutLabel = withoutLabel
290
+ return {
291
+ eventTypeCounts,
292
+ projectionLengths,
293
+ projectionSkipped,
294
+ projectionFieldGaps,
295
+ subagent: {
296
+ mainLogDescriptorEvents: eventTypeCounts['subagent/descriptor'] ?? 0,
297
+ supportedDescriptors,
298
+ children,
299
+ },
300
+ }
301
+ }
302
+
133
303
  /**
134
304
  * Build one assertable trace from parsed session logs. Child sessions surface
135
305
  * only through the parent's tool events, so the MAIN log (no `origin:
@@ -146,20 +316,27 @@ export function buildTrace(logs) {
146
316
  const mains = logs.filter(log => log.header.origin !== 'subagent')
147
317
  const main = [...mains].sort((a, b) => b.events.length - a.events.length)[0]
148
318
  const events = main?.events ?? []
149
- const subagentChildren = logs
319
+ const childLogs = logs
150
320
  .filter(log => log.header.origin === 'subagent' || log.header.parentSession !== undefined)
151
- .map(projectChild)
321
+ const subagentChildren = childLogs.map(projectChild)
152
322
 
153
323
  const toolCalls = []
154
324
  const toolResults = []
155
325
  const assistantEntries = []
156
326
  const userMessages = []
157
327
  const requestHeaders = []
328
+ const gaps = {}
329
+ const recordGap = key => { gaps[key] = (gaps[key] ?? 0) + 1 }
158
330
  for (const event of events) {
159
331
  if (event.type === 'request/header') {
160
332
  // The assembled model request header: system prompt + mounted tool
161
333
  // schemas. What the model is told it can do and how — the "did my
162
334
  // plugin's section inject?" projection.
335
+ if (typeof event.data?.header?.system !== 'string') recordGap('headerWithoutSystem')
336
+ if (Array.isArray(event.data?.header?.tools)
337
+ && !event.data.header.tools.some(tool => typeof tool?.name === 'string')) {
338
+ recordGap('headerWithoutToolNames')
339
+ }
163
340
  requestHeaders.push({
164
341
  seq: event.seq,
165
342
  reason: event.data?.reason,
@@ -169,6 +346,8 @@ export function buildTrace(logs) {
169
346
  : [],
170
347
  })
171
348
  } else if (event.type === 'tool/call') {
349
+ if (typeof event.data?.name !== 'string') recordGap('toolCallWithoutName')
350
+ if (typeof event.data?.callId !== 'string') recordGap('toolCallWithoutCallId')
172
351
  toolCalls.push({
173
352
  seq: event.seq,
174
353
  turn: event.data.turn,
@@ -179,6 +358,7 @@ export function buildTrace(logs) {
179
358
  parsedArguments: parseArguments(event.data.arguments),
180
359
  })
181
360
  } else if (event.type === 'tool/result') {
361
+ if (typeof event.data?.message?.source?.callId !== 'string') recordGap('toolResultWithoutCallId')
182
362
  toolResults.push({
183
363
  seq: event.seq,
184
364
  turn: event.data.turn,
@@ -222,7 +402,7 @@ export function buildTrace(logs) {
222
402
  : assistantEntries.filter(entry => entry.seq < firstInjectionSeq)
223
403
  const answerText = answerEntries.at(-1)?.text ?? ''
224
404
 
225
- return {
405
+ const result = {
226
406
  sessions: logs,
227
407
  sessionId: main?.header.id,
228
408
  toolCalls,
@@ -233,36 +413,118 @@ export function buildTrace(logs) {
233
413
  requestHeaders,
234
414
  subagentChildren,
235
415
  finalText: assistantTexts.at(-1) ?? '',
416
+ census: undefined,
236
417
  }
418
+ result.census = buildCensus(events, result, childLogs, gaps)
419
+ return result
420
+ }
421
+
422
+ /**
423
+ * Session artifact basenames: format v0 keeps `session.jsonl`, every later
424
+ * generation carries a `vN` component (`session.v3.jsonl` — the host's
425
+ * `generationLogFilename`). Matching the v0 name alone finds no trace at all
426
+ * once the host bumps the format, which surfaces as "no session trace
427
+ * materialized" rather than as a parse error.
428
+ */
429
+ const SESSION_LOG_FILENAME = /^session(?:\.v\d+)?\.jsonl$/u
430
+
431
+ /**
432
+ * Whether one file basename is a session JSONL artifact of any format generation.
433
+ * @param {string} name - the file basename to test.
434
+ * @returns {boolean} true for `session.jsonl` and `session.vN.jsonl`.
435
+ */
436
+ export function isSessionLogFilename(name) {
437
+ return SESSION_LOG_FILENAME.test(name)
237
438
  }
238
439
 
239
- /** Recursively collect files named `name` under `dir`. */
240
- function collectFiles(dir, name, out = []) {
241
- for (const entry of readdirSync(dir, { withFileTypes: true })) {
440
+ /** Recursively list files under `dir`; an unreadable directory contributes nothing. */
441
+ function listFiles(dir, out = []) {
442
+ let entries
443
+ try {
444
+ entries = readdirSync(dir, { withFileTypes: true })
445
+ } catch {
446
+ return out
447
+ }
448
+ for (const entry of entries) {
242
449
  const path = join(dir, entry.name)
243
- if (entry.isDirectory()) collectFiles(path, name, out)
244
- else if (entry.name === name) out.push(path)
450
+ if (entry.isDirectory()) listFiles(path, out)
451
+ else out.push(path)
245
452
  }
246
453
  return out
247
454
  }
248
455
 
249
456
  /**
250
- * Load every session log under a persistence root and build one trace.
457
+ * Every session artifact under `sessionsRoot` (any generation, see
458
+ * `isSessionLogFilename`) as absolute paths — the collection half of the
459
+ * seam, shared by the trace builder and by the raw-log capture the behavior
460
+ * runner does before cleanup.
251
461
  * @param {string} sessionsRoot - the run's `session-persistence-jsonl` root.
252
- * @returns {EvalTrace | undefined} the trace, or `undefined` when no log materialized.
462
+ * @returns {string[]} artifact paths, in directory order.
253
463
  */
254
- export function loadTraceDir(sessionsRoot) {
255
- let files
256
- try {
257
- files = collectFiles(sessionsRoot, 'session.jsonl')
258
- } catch {
259
- return undefined
464
+ export function listSessionLogFiles(sessionsRoot) {
465
+ return listFiles(sessionsRoot).filter(path => isSessionLogFilename(basename(path)))
466
+ }
467
+
468
+ /** Most candidate names one gap diagnostic lists before it truncates. */
469
+ const GAP_NAME_LIMIT = 10
470
+
471
+ /**
472
+ * Why a collection produced no artifact, phrased for a failure message: the
473
+ * candidate names actually seen (a renamed artifact is the likeliest host
474
+ * drift) plus the generation suspicion. Never returns an empty string — an
475
+ * empty root is itself the fact to report.
476
+ */
477
+ function traceGapMessage(sessionsRoot, files) {
478
+ const names = [...new Set(files.map(file => basename(file)))]
479
+ const lookalikes = names.filter(name => name.toLowerCase().startsWith('session'))
480
+ const pool = lookalikes.length > 0 ? lookalikes : names
481
+ const shown = pool.slice(0, GAP_NAME_LIMIT)
482
+ const rest = pool.length - shown.length
483
+ const scan = shown.length === 0
484
+ ? 'the root holds no files (missing or empty)'
485
+ : `${lookalikes.length > 0 ? 'session-like file(s)' : 'file(s)'} under it: `
486
+ + `${shown.join(', ')}${rest > 0 ? ` (+${rest} more)` : ''}`
487
+ return 'no session trace materialized: no session artifact'
488
+ + ` (session.jsonl / session.vN.jsonl) under '${sessionsRoot}' — ${scan}`
489
+ + '; the host artifact naming may have changed generation'
490
+ }
491
+
492
+ /**
493
+ * Collect one run's session trace and, when there is none, the seam
494
+ * diagnosis for it.
495
+ *
496
+ * The `gap` string exists so that "the host's artifact/session layout moved"
497
+ * surfaces as that sentence in the behavior runner's failure text and in the
498
+ * review adapter's accounting, instead of as a bare `undefined` the reader
499
+ * has to trace back through the parser (see docs/host-wiring.md).
500
+ *
501
+ * @param {string} sessionsRoot - the run's `session-persistence-jsonl` root.
502
+ * @returns {{ trace: EvalTrace | undefined, gap: string | undefined }} the
503
+ * trace, or `undefined` plus the reason no trace could be built.
504
+ */
505
+ export function collectSessionTrace(sessionsRoot) {
506
+ const files = listFiles(sessionsRoot)
507
+ const artifacts = files.filter(file => isSessionLogFilename(basename(file)))
508
+ if (artifacts.length === 0) {
509
+ return { trace: undefined, gap: traceGapMessage(sessionsRoot, files) }
510
+ }
511
+ const logs = []
512
+ const broken = []
513
+ for (const artifact of artifacts) {
514
+ try {
515
+ logs.push(parseSessionLog(readFileSync(artifact, 'utf8')))
516
+ } catch (error) {
517
+ broken.push(`${basename(artifact)}: ${error instanceof Error ? error.message : String(error)}`)
518
+ }
519
+ }
520
+ if (broken.length > 0) {
521
+ return {
522
+ trace: undefined,
523
+ gap: `session artifact(s) failed to parse — ${broken.join('; ')}`
524
+ + '; the host session format may have changed generation',
525
+ }
260
526
  }
261
- if (files.length === 0) return undefined
262
- const logs = files
263
- .map(file => readFileSync(file, 'utf8'))
264
- .map(parseSessionLog)
265
- return buildTrace(logs)
527
+ return { trace: buildTrace(logs), gap: undefined }
266
528
  }
267
529
 
268
530
  /**
@@ -290,4 +552,39 @@ export function loadTraceDir(sessionsRoot) {
290
552
  * last assistant text ('' when it produced none — dispatched but not
291
553
  * answered).
292
554
  * @property {string} finalText - the last assembled assistant text ('' when none).
555
+ * @property {object | undefined} census - what the raw logs contained against
556
+ * what the projections kept (numbers only, never a verdict). Two data
557
+ * sources: `eventTypeCounts` counts the MAIN log's events by type (any type,
558
+ * plug-in ones included), and `subagent` censuses the child logs that enter
559
+ * `subagentChildren` (their `subagent/descriptor` event counts and how many
560
+ * carry the supported `version === 3`). `projectionLengths` are the five
561
+ * main-log projections' lengths after empty-text drops; `projectionSkipped`
562
+ * records where count minus length is positive, per projection, plus the two
563
+ * child-identity counters. Undefined only on a hand-built trace; a nested log
564
+ * under `sessions` carries none because the parsers never add one.
565
+ * @property {Record<string, number>} census.projectionFieldGaps
566
+ * - events the projection kept while a field it reads went missing, keyed by
567
+ * what was missing (`toolCallWithoutName`, `toolCallWithoutCallId`,
568
+ * `toolResultWithoutCallId`, `headerWithoutSystem`, `headerWithoutToolNames`).
569
+ * This is the 1:1-projection signal: for `tool/call`, `tool/result` and
570
+ * `request/header`, count − length is structurally zero, so a moved field
571
+ * shows up only here. An absent `request/header.tools` array is NOT counted
572
+ * (it projects to the same empty list as an empty one).
573
+ * @property {{ mainLogDescriptorEvents: number, supportedDescriptors: number, children: object[] }} census.subagent
574
+ * - the child-log data source the main-log counts cannot reach.
575
+ * `mainLogDescriptorEvents` counts `subagent/descriptor` events in the MAIN
576
+ * log itself (the current host writes them into the child log, so this is
577
+ * usually 0); `supportedDescriptors` sums the per-child counts below, i.e.
578
+ * it counts DESCRIPTOR EVENTS, not child sessions — one child may fold its
579
+ * identity from a single descriptor while logging several.
580
+ * @property {{ sessionId: string | undefined, parentSession: string | undefined, delegationDepth: number | undefined, descriptorEvents: number, supportedDescriptors: number, label: string | undefined, mode: string | undefined, provider: string | undefined }[]} census.subagent.children
581
+ * - the accepted child logs, each with the identity `projectChild` folded
582
+ * from them; `descriptorEvents === 0` means the log states no identity at
583
+ * all, and a non-zero count with `supportedDescriptors === 0` means every
584
+ * descriptor was outside the supported version.
585
+ * @property {Record<string, number>} census.projectionSkipped.children
586
+ * - `withoutIdentity` counts children that folded no identity field at all;
587
+ * `withoutLabel` counts children whose `label` is absent — the field the
588
+ * `subagent*Count` matchers match on, so those are the records whose
589
+ * zero-count assertions pass only because there was nothing to match.
293
590
  */