@catheadowl/dsh-eval 0.2.1 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +72 -0
- package/bin/dsh-eval.mjs +17 -8
- package/bin/dsh-review.mjs +19 -6
- package/docs/experimental.md +4 -3
- package/docs/host-wiring.md +26 -1
- package/docs/matchers.md +22 -1
- package/docs/review.md +4 -2
- package/docs/runner-api.md +1 -2
- package/package.json +2 -2
- package/src/adapters/dsh/review.mjs +41 -47
- package/src/driver/multi-turn-driver.mjs +4 -1
- package/src/experimental.mjs +6 -1
- package/src/mock/mock-adapter.mjs +14 -0
- package/src/report.mjs +5 -0
- package/src/review-report.mjs +31 -0
- package/src/runner.mjs +14 -29
- package/src/tool-validation.mjs +12 -4
- package/src/trace.mjs +330 -33
package/CHANGELOG.md
CHANGED
|
@@ -8,6 +8,77 @@ All notable changes to `@catheadowl/dsh-eval` are documented here. Versions
|
|
|
8
8
|
follow [Semantic Versioning](https://semver.org/); entries follow
|
|
9
9
|
[Keep a Changelog](https://keepachangelog.com/) conventions.
|
|
10
10
|
|
|
11
|
+
## [0.3.0] — 2026-09-13
|
|
12
|
+
|
|
13
|
+
### Added
|
|
14
|
+
|
|
15
|
+
- **Projection census** on every case that produced a session trace:
|
|
16
|
+
`census.eventTypeCounts` (the main session log's events per type),
|
|
17
|
+
`census.projectionLengths` (the five projection lengths) with
|
|
18
|
+
`census.projectionSkipped` naming where a count exceeds its projection length,
|
|
19
|
+
`census.projectionFieldGaps` naming the events that projected while a field
|
|
20
|
+
they carry went missing (the `tool/call` / `tool/result` / `request/header`
|
|
21
|
+
projections are 1:1, so a moved field never shows up as a length difference),
|
|
22
|
+
and `census.subagent` (the child logs behind `subagentChildren`: their
|
|
23
|
+
`subagent/descriptor` event counts, how many carry the supported descriptor
|
|
24
|
+
version, and the folded identity). The census reports numbers only and never
|
|
25
|
+
decides whether a difference is a defect — it makes "the host log carried no
|
|
26
|
+
such event" and "the projection dropped it" separable in `--format json` and
|
|
27
|
+
in `.runs/<id>/trace.json`. Cases with no trace carry no census.
|
|
28
|
+
- Session-seam boundary assertions: `collectSessionTrace` reports which
|
|
29
|
+
candidate artifact files were actually present when no session trace
|
|
30
|
+
materializes (behavior failures name the host artifact naming instead of a
|
|
31
|
+
bare "no session trace materialized"), and `parseSessionLog` refuses a
|
|
32
|
+
`header.version` outside `KNOWN_SESSION_FORMAT_VERSIONS` with the version
|
|
33
|
+
number instead of projecting empty fields. `EvalRunResult.traceGap` carries
|
|
34
|
+
the diagnosis to the CLI failure text.
|
|
35
|
+
|
|
36
|
+
### Removed
|
|
37
|
+
|
|
38
|
+
- `loadTraceDir` (experimental): replaced by `collectSessionTrace`, which
|
|
39
|
+
returns the trace together with the reason none was built. Migrate
|
|
40
|
+
`loadTraceDir(root)` to `collectSessionTrace(root).trace`.
|
|
41
|
+
- **`options.dshRepoDir`** on `runEvalCase` (and the same legacy option on
|
|
42
|
+
the review executor and `executeBehaviorExperiment`): the CLI location is
|
|
43
|
+
now `options.cliPath` only. **Migrating a case that trips this**: resolve
|
|
44
|
+
once with `resolveDshCliChain` (the `/experimental` export) and pass its
|
|
45
|
+
`cli` result as `cliPath` — the `dsh-eval` / `dsh-review` CLIs already do
|
|
46
|
+
this via `--repo`, the `node_modules` resolution layer, or the config
|
|
47
|
+
`repo` key.
|
|
48
|
+
|
|
49
|
+
### Fixed
|
|
50
|
+
|
|
51
|
+
- **Mock mode against host 0.1.5-rc.2: the scripted adapter now carries its
|
|
52
|
+
own `prepareCall`.** The host's LLM service dispatches every model call
|
|
53
|
+
through `registration.adapter.prepareCall(...)`, a wire-contract step the
|
|
54
|
+
adapter's inherited base class did not have — the base resolves from this
|
|
55
|
+
package's `@deepseek-ai/dsh-llm` peer instance, which lagged the host
|
|
56
|
+
runtime (0.0.1-rc.1 vs 0.1.5-rc.2), so every mock run died at startup with
|
|
57
|
+
`registration.adapter.prepareCall is not a function`. The override mirrors
|
|
58
|
+
the host base-class default (`{ model, stream }` bound to one adapter
|
|
59
|
+
generation), making the wire contract independent of the peer instance's
|
|
60
|
+
generation; the seam is now documented in `docs/host-wiring.md` with a
|
|
61
|
+
maintenance trigger.
|
|
62
|
+
- `census.eventTypeCounts` counts prototype-named event types correctly: a
|
|
63
|
+
plug-in event type such as `constructor` or `__proto__` used to produce a
|
|
64
|
+
string-concatenated value or vanish from the map entirely, so a field named
|
|
65
|
+
like a count could hold a non-number.
|
|
66
|
+
- Session-trace discovery follows the host's **format-generation artifact
|
|
67
|
+
names**: `session.jsonl` for v0 and `session.vN.jsonl` for later
|
|
68
|
+
generations (`session.v3.jsonl` on the current host). Matching only the v0
|
|
69
|
+
name made every behavior case fail with "no session trace materialized"
|
|
70
|
+
after the host bumped the session format.
|
|
71
|
+
- The multi-turn driver reads the durable log through
|
|
72
|
+
`Session#snapshotEvents()`; the `session.events` getter it used was removed
|
|
73
|
+
upstream, so any case declaring `followups` aborted the headless run with
|
|
74
|
+
`agent.session.events is not iterable`.
|
|
75
|
+
- Review runs no longer fail open when no session artifact materializes:
|
|
76
|
+
`validateToolBoundary` reports `status: 'not-executed'` (not a pass), the
|
|
77
|
+
executor result carries the gap, the report states
|
|
78
|
+
`tool boundary: NOT EXECUTED on run(s) N`, and both `run-N.txt` and
|
|
79
|
+
`run.json` record it — a review whose tool face was never verified no longer
|
|
80
|
+
reads as a normal one.
|
|
81
|
+
|
|
11
82
|
## [0.2.1] — 2026-09-09
|
|
12
83
|
|
|
13
84
|
### Added
|
|
@@ -52,6 +123,7 @@ follow [Semantic Versioning](https://semver.org/); entries follow
|
|
|
52
123
|
`@deepseek-ai/dsh-llm` is a peerDependency provided by the host
|
|
53
124
|
ecosystem.
|
|
54
125
|
|
|
126
|
+
[0.3.0]: https://github.com/CatheadOwl/dsh-eval/releases/tag/v0.3.0
|
|
55
127
|
[0.2.1]: https://github.com/CatheadOwl/dsh-eval/releases/tag/v0.2.1
|
|
56
128
|
[0.2.0]: https://github.com/CatheadOwl/dsh-eval/releases/tag/v0.2.0
|
|
57
129
|
[0.1.0]: https://github.com/CatheadOwl/dsh-eval/releases/tag/v0.1.0
|
package/bin/dsh-eval.mjs
CHANGED
|
@@ -11,9 +11,10 @@
|
|
|
11
11
|
* A case path is a `*.eval.mjs` file or a directory scanned recursively for
|
|
12
12
|
* them. Each file default-exports one case object (or an array of them):
|
|
13
13
|
* `{ id, task, mode?: 'real'|'mock', expect: Matcher[], script?, persona?,
|
|
14
|
-
* prepare?, timeoutMs? }`. Real cases skip when
|
|
15
|
-
* the
|
|
16
|
-
*
|
|
14
|
+
* prepare?, timeoutMs? }`. Real cases skip when NO credential is visible —
|
|
15
|
+
* the env var DEEPSEEK_API_KEY OR the staged home's .credentials.yaml,
|
|
16
|
+
* either one counts. The exit code is 1 when any run fails. Failures keep
|
|
17
|
+
* their artifacts under `<case file dir>/.runs/<case id>/`.
|
|
17
18
|
*
|
|
18
19
|
* Output formats:
|
|
19
20
|
* - `--format text` (default): unchanged human output on stdout/stderr.
|
|
@@ -134,7 +135,8 @@ function writeArtifacts(evalCase, result, mode) {
|
|
|
134
135
|
writeFileSync(join(artifactsDir, 'stderr.txt'), result.stderr)
|
|
135
136
|
writeFileSync(join(artifactsDir, 'trace.json'), JSON.stringify({
|
|
136
137
|
caseId: evalCase.id, mode, task: evalCase.task,
|
|
137
|
-
exitCode: result.exitCode, timedOut: result.timedOut,
|
|
138
|
+
exitCode: result.exitCode, timedOut: result.timedOut,
|
|
139
|
+
traceGap: result.traceGap, trace: result.trace,
|
|
138
140
|
}, undefined, 2))
|
|
139
141
|
result.sessionLogs.forEach((text, index) => {
|
|
140
142
|
writeFileSync(join(artifactsDir, `session-${index}.jsonl`), text)
|
|
@@ -233,7 +235,7 @@ for (const file of files.sort()) {
|
|
|
233
235
|
const runStartedAt = Date.now()
|
|
234
236
|
let result
|
|
235
237
|
try {
|
|
236
|
-
result = await runEvalCase(evalCase, { profile, cliPath,
|
|
238
|
+
result = await runEvalCase(evalCase, { profile, cliPath, mode })
|
|
237
239
|
} catch (error) {
|
|
238
240
|
records.push(createCaseRecord({
|
|
239
241
|
id: evalCase.id, file, mode, status: 'fail',
|
|
@@ -247,14 +249,19 @@ for (const file of files.sort()) {
|
|
|
247
249
|
|
|
248
250
|
if (result.trace === undefined) {
|
|
249
251
|
const artifactsDir = writeArtifacts(evalCase, result, mode)
|
|
252
|
+
// The runner's seam diagnosis (which candidate file names were actually
|
|
253
|
+
// collected, or which artifact was refused) IS the failure text: a bare
|
|
254
|
+
// "no session trace materialized" pointed readers at the parser while
|
|
255
|
+
// the real drift was the host's artifact naming (EVAL-019/EVAL-020).
|
|
256
|
+
const reason = `${result.traceGap ?? 'no session trace materialized'} (exit ${result.exitCode}${result.timedOut ? ', timed out' : ''})`
|
|
250
257
|
records.push(createCaseRecord({
|
|
251
258
|
id: evalCase.id, file, mode, status: 'fail',
|
|
252
|
-
failures: [
|
|
259
|
+
failures: [reason],
|
|
253
260
|
exitCode: result.exitCode, timedOut: result.timedOut,
|
|
254
261
|
durationMs, artifactsDir,
|
|
255
262
|
}))
|
|
256
263
|
process.stderr.write(
|
|
257
|
-
`FAIL ${evalCase.id}:
|
|
264
|
+
`FAIL ${evalCase.id}: ${reason}\n`
|
|
258
265
|
+ ` artifacts: ${artifactsDir}\n--- stderr ---\n${result.stderr}\n`,
|
|
259
266
|
)
|
|
260
267
|
continue
|
|
@@ -278,7 +285,8 @@ for (const file of files.sort()) {
|
|
|
278
285
|
records.push(createCaseRecord({
|
|
279
286
|
id: evalCase.id, file, mode, status: 'pass',
|
|
280
287
|
exitCode: result.exitCode, timedOut: result.timedOut,
|
|
281
|
-
durationMs, ...(
|
|
288
|
+
durationMs, ...(result.trace?.census !== undefined ? { census: result.trace.census } : {}),
|
|
289
|
+
...(artifactsDir !== undefined ? { artifactsDir } : {}),
|
|
282
290
|
}))
|
|
283
291
|
say(`PASS ${evalCase.id}`)
|
|
284
292
|
} else {
|
|
@@ -293,6 +301,7 @@ for (const file of files.sort()) {
|
|
|
293
301
|
id: evalCase.id, file, mode, status: 'fail', failures: hint ? [...failures, hint] : failures,
|
|
294
302
|
exitCode: result.exitCode, timedOut: result.timedOut,
|
|
295
303
|
durationMs, artifactsDir,
|
|
304
|
+
...(result.trace?.census !== undefined ? { census: result.trace.census } : {}),
|
|
296
305
|
}))
|
|
297
306
|
process.stderr.write(`FAIL ${evalCase.id} (exit ${result.exitCode}):\n${failures.map(f => ` - ${f}`).join('\n')}\n`)
|
|
298
307
|
if (hint !== undefined) process.stderr.write(` ! ${hint}\n`)
|
package/bin/dsh-review.mjs
CHANGED
|
@@ -95,11 +95,11 @@ const { config } = await loadEvalConfig(process.cwd())
|
|
|
95
95
|
const profile = options.profile ?? config.profile ?? 'headless'
|
|
96
96
|
// CLI resolution (C6): `--repo` flag > resolution layer (node_modules) >
|
|
97
97
|
// config repo key (legacy). Dry-run never boots the CLI, so resolve lazily.
|
|
98
|
-
let cli = { cliPath: undefined
|
|
98
|
+
let cli = { cliPath: undefined }
|
|
99
99
|
if (!options.dryRun) {
|
|
100
100
|
try {
|
|
101
101
|
const resolved = resolveDshCliChain({ repoFlag: options.repo, configRepo: config.repo })
|
|
102
|
-
cli = { cliPath: resolved.cli
|
|
102
|
+
cli = { cliPath: resolved.cli }
|
|
103
103
|
} catch (error) {
|
|
104
104
|
usage(`error: ${error.message}`)
|
|
105
105
|
}
|
|
@@ -127,7 +127,6 @@ for (const file of files) {
|
|
|
127
127
|
const result = await runDshReviewExperiment(experiment, {
|
|
128
128
|
profile,
|
|
129
129
|
cliPath: cli.cliPath,
|
|
130
|
-
dshRepoDir: cli.repoDir,
|
|
131
130
|
runs: options.runs,
|
|
132
131
|
timeoutMs: options.timeoutMs,
|
|
133
132
|
keepPluginRows: options.keepPluginRows,
|
|
@@ -136,15 +135,29 @@ for (const file of files) {
|
|
|
136
135
|
adapter: 'dsh-headless',
|
|
137
136
|
profile,
|
|
138
137
|
runs: result.runs,
|
|
138
|
+
// Machine-readable boundary accounting per run: `checked` (the request
|
|
139
|
+
// headers were inspected) vs `not-executed` (no session artifact, so the
|
|
140
|
+
// reviewer's tool face was never verified) — the report and run-N.txt
|
|
141
|
+
// carry the same fact in prose (EVAL-021).
|
|
142
|
+
toolBoundaries: result.attempts.map(attempt => ({
|
|
143
|
+
run: attempt.index,
|
|
144
|
+
status: attempt.result?.toolValidation?.status ?? 'not-reported',
|
|
145
|
+
...(attempt.result?.traceGap === undefined ? {} : { reason: attempt.result.traceGap }),
|
|
146
|
+
})),
|
|
139
147
|
}, result)
|
|
140
148
|
for (const attempt of result.attempts) {
|
|
141
149
|
const payload = attempt.result ?? {}
|
|
142
150
|
// run-N.txt is the reviewer's ANSWER (trace-derived, splice-proof),
|
|
143
151
|
// falling back to stdout for executors/trace-less runs; the raw final
|
|
144
152
|
// message stays in run-N.stdout.txt when it differs from the answer,
|
|
145
|
-
// the full session transcript in run-N.stderr.txt.
|
|
146
|
-
|
|
147
|
-
|
|
153
|
+
// the full session transcript in run-N.stderr.txt. A run whose tool
|
|
154
|
+
// boundary was never verified says so at the TOP of this file: it is
|
|
155
|
+
// what a grader opens first, and its answer is the fallback one.
|
|
156
|
+
const boundaryNotice = payload.toolValidation?.status === 'not-executed'
|
|
157
|
+
? `[tool-boundary: NOT EXECUTED — ${payload.traceGap ?? 'no session trace materialized'}]\n\n`
|
|
158
|
+
: ''
|
|
159
|
+
if (payload.answer !== undefined) writeFileSync(join(output, `run-${attempt.index}.txt`), boundaryNotice + payload.answer, 'utf8')
|
|
160
|
+
else if (payload.stdout !== undefined) writeFileSync(join(output, `run-${attempt.index}.txt`), boundaryNotice + payload.stdout, 'utf8')
|
|
148
161
|
if (payload.stdout !== undefined && payload.answer !== undefined && payload.stdout !== payload.answer) {
|
|
149
162
|
writeFileSync(join(output, `run-${attempt.index}.stdout.txt`), payload.stdout, 'utf8')
|
|
150
163
|
}
|
package/docs/experimental.md
CHANGED
|
@@ -20,16 +20,17 @@ import { resolveDshCliChain } from '@catheadowl/dsh-eval/experimental'
|
|
|
20
20
|
| `stageProfileStore` | 把真实 profile store junction 感知地暂存进沙箱 home(沙箱机制) |
|
|
21
21
|
| `buildOverlayYaml` | 由片段拼装 dsh overlay YAML(整段发射器) |
|
|
22
22
|
| `overlayDisableRows` | 生成 `disabled: true` 的行禁用 overlay 片段 |
|
|
23
|
-
| `parseSessionLog` | 解析一条未压缩 JSONL session artifact 为 `{ header, events }` |
|
|
23
|
+
| `parseSessionLog` | 解析一条未压缩 JSONL session artifact 为 `{ header, events }`;header 的 `version` 戳不在已知代际集合内即抛错 |
|
|
24
24
|
| `buildTrace` | 把 session 事件投影为 matcher 使用的 trace 对象 |
|
|
25
|
-
| `
|
|
25
|
+
| `collectSessionTrace` | 收集一个 run 的 trace 与「为什么没有 trace」的 seam 诊断:返回 `{ trace, gap }`,`gap` 文案含实际候选文件名与代际嫌疑(唯一的收集入口) |
|
|
26
|
+
| `KNOWN_SESSION_FORMAT_VERSIONS` | 本包接受的 session 格式代际集合(`parseSessionLog` 的准入面;宿主新增代际时与本包重验同步) |
|
|
26
27
|
| `executeReviewExperiment` | 用给定 executor 执行抽象 review 实验 |
|
|
27
28
|
| `materializeReviewExperiment` | 把实验定义物化为产物目录 |
|
|
28
29
|
| `renderObservationSections` | 标准 observation renderer(自定义 executor 用) |
|
|
29
30
|
| `OBSERVATIONS_PLACEHOLDER` | prompt 中的观测占位符常量(必须恰好出现一次) |
|
|
30
31
|
| `createDshHeadlessReviewExecutor` | 构造 dsh headless review executor |
|
|
31
32
|
| `runDshReviewExperiment` | 端到端跑一个 dsh review 实验 |
|
|
32
|
-
| `validateToolBoundary` | 校验 trace 满足 turn-close
|
|
33
|
+
| `validateToolBoundary` | 校验 trace 满足 turn-close 工具边界契约;结果带 `status`(`checked` / `not-executed`),无 trace 时**不是**通过 |
|
|
33
34
|
| `renderToolBoundaryEvidence` | 渲染边界校验的机器可读证据 |
|
|
34
35
|
|
|
35
36
|
新公开能力先进本入口;稳定后经明确决策才升入包根入口(升入即接受 semver 义务)。
|
package/docs/host-wiring.md
CHANGED
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
---
|
|
2
|
-
description: 安装与宿主接线——dsh-llm peer 的三形解析结局与 junction 步骤、构建 CLI 与 profile/凭证/spawn
|
|
2
|
+
description: 安装与宿主接线——dsh-llm peer 的三形解析结局与 junction 步骤、构建 CLI 与 profile/凭证/spawn 三类运行前置、本包对宿主 session seam 的四处硬断言及其执法面(artifact 代际命名 / header 代际戳 / 拼接帧容器 / snapshotEvents 读取面),以及 mock 模式依赖的宿主 LLM adapter 线上契约(prepareCall,自带覆写对 peer 实例代差免疫)。
|
|
3
3
|
---
|
|
4
4
|
|
|
5
5
|
# 安装与宿主接线
|
|
@@ -56,6 +56,31 @@ node -e "console.log(require('fs').existsSync('node_modules/@deepseek-ai/dsh/lib
|
|
|
56
56
|
|
|
57
57
|
`false` = 解析层缺 CLI:先把上述 junction 重建为指向宿主检出;仍 `false` 则宿主检出未构建(先构建宿主)。这类 junction 维护是机器相关的开发环境事务,不入库,由各开发环境自行承接(同上文 peer 接线的 gitignore 纪律)。behavior 与 review 的真实运行都从定位到的 CLI spawn dsh 本体。
|
|
58
58
|
|
|
59
|
+
## 宿主 session seam:本包硬断言的四处,坏了多是具名失败
|
|
60
|
+
|
|
61
|
+
behavior 与 review 的证据都取自**真实 dsh 会话的产物与进程内日志**,因此本包直接断言宿主的几处 session seam。它们随宿主演进时不会有编译期提示,所以每一处都配一行**执法面**(表里的符号就是);宿主检出更新后、动本包引用它们的文档前,先按本节对源码重新验证:
|
|
62
|
+
|
|
63
|
+
| 断言 | 宿主依据 | 本包执法面 | 坏了长什么样 |
|
|
64
|
+
|---|---|---|---|
|
|
65
|
+
| 会话 artifact 按**格式代**命名:v0 是 `session.jsonl`,之后每代带小写数字(当前 `session.v3.jsonl`);`compression: none` 时无 `.zstd` 后缀 | `session-persistence-jsonl/src/format.ts` 的 `generationLogFilename`,配 `core/session/src/types.ts` 的 `SESSION_FORMAT_VERSION` | `isSessionLogFilename`(命名判定)+ `collectSessionTrace`(收集)⇒ behavior 失败文案 / review 记账里的 `traceGap` | 只按 v0 名收集 ⇒ 一条日志都收不到;失败文案列出**实际扫到的候选文件名**并写明「宿主 artifact 命名可能已换代」,不再只报 `no session trace materialized` |
|
|
66
|
+
| 会话 header 的 `version` 戳是宿主对逻辑代际的声明(当前 v3) | 同上;已发布的代际链见 `session-format-catalog/src/generated.ts`(codecs v0–v3、`currentVersion: 3`) | `KNOWN_SESSION_FORMAT_VERSIONS`(`parseSessionLog` 入口准入) | 未知代际 ⇒ 解析当场拒绝并报出版本号(`session header version vN is not a known generation`),不再把各投影字段静默降级成空数组 |
|
|
67
|
+
| 会话日志是**拼接帧容器**(宿主默认 zstd),须逐帧扫描 | `session-persistence-jsonl/src/zstd.ts` 的帧扫描 | eval overlay 固定 `compression: none` + `packChunks: false`(`src/overlay.mjs`) | 整文件一次解压 ⇒ `ZSTD_error_prefix_unknown`(第二帧魔数被当输入) |
|
|
68
|
+
| 进程内读 durable 事件的 API 是 `Session#snapshotEvents()`(不可变冻结快照);早期的 `session.events` getter 已被删除 | `core/session/src/index.ts` 的 `snapshotEvents` | driver 行(`src/driver/multi-turn-driver.mjs`)直接调用,没有回退路径 | 属性访问得到 `undefined` ⇒ 进程内消费者抛 `agent.session.events is not iterable`,整个 headless run 直接死 |
|
|
69
|
+
|
|
70
|
+
命名行与代际行的读取面由本包的 eval overlay 固定(`compression: none` + `packChunks: false`),所以每轮 run 的 artifact 是**明文逐事件**布局;命名判定、代际准入与收集入口都在 `src/trace.mjs`(`isSessionLogFilename` / `KNOWN_SESSION_FORMAT_VERSIONS` / `collectSessionTrace`),behavior runner 与 review adapter 共用同一个收集入口,缺 artifact 时各自把 `traceGap` 带进失败文案与产物记账。帧容器行走 overlay 固定;最后一行是 driver 行读日志时直接依赖的方法。
|
|
71
|
+
|
|
72
|
+
> **维护触发器**:宿主 session 格式、持久化命名或 `Session` 读取面变更 ⇒ 先按上表对 vendored 检出重新验证断言,再更新本篇与引用它们的源码/认知。前两行现在是**机械的**——命名或代际戳变了,跑一条 case 就红在具名文案上(候选文件名 / 版本号);后两行仍只有真跑一条 case 才会暴露。
|
|
73
|
+
|
|
74
|
+
### 宿主 LLM adapter 线上契约(mock 模式专用)
|
|
75
|
+
|
|
76
|
+
mock 模式经 `eval-mock-llm` 插件(`src/mock/mock-adapter.mjs`)注册 `EvalMockAdapter`,它 `extends` 的 `LlmAdapter` 基类解析自**本包的 peer 实例**——该实例可以落后于驱动它的宿主运行时(实测:宿主 0.1.5-rc.2 的 LLM 服务对已注册 adapter 新增 `registration.adapter.prepareCall(...)` 调用面时,本地 peer 还是 0.0.1-rc.1,继承面缺失,全部 mock run 死在启动期)。因此适配器**自带** `prepareCall` 覆写、不依赖继承面在不在:形状镜像宿主基类默认契约(`PreparedAdapterCall`——`{ model: resolveModel(...), stream: options => this.stream(options) }`,model 元数据与派发入口绑定同一代适配器)。
|
|
77
|
+
|
|
78
|
+
| 断言 | 宿主依据 | 本包执法面 | 坏了长什么样 |
|
|
79
|
+
|---|---|---|---|
|
|
80
|
+
| 宿主 LLM 服务经 `prepareCall` 派发每次模型调用:adapter 级 `prepareCall(provider, model, signal)` 返回 `{ model, stream }`(一次性句柄,防 HMR 混代) | `packages/llm/llm/src/index.ts` 的 `registration.adapter.prepareCall(...)` 调用与 `LlmAdapter` 基类(`PreparedAdapterCall`) | `EvalMockAdapter` 自带 `prepareCall` 覆写(`src/mock/mock-adapter.mjs`)——真跑一条 mock case 即红在具名错误串上 | mock run 空转:无 session 事件、workspace 未落、final text 空,stderr 带 `registration.adapter.<method> is not a function` |
|
|
81
|
+
|
|
82
|
+
> **维护触发器**:宿主 adapter 线上契约演进(新增/改签名线上方法)⇒ 先对宿主 `packages/llm/llm/src/index.ts` 重验 `EvalMockAdapter` 的自带面(`prepareCall` / `resolveModel` / `stream` 的形状与语义),再同 commit 改本节与 `src/mock/mock-adapter.mjs`。
|
|
83
|
+
|
|
59
84
|
## 环境面:profile 与插件安装
|
|
60
85
|
|
|
61
86
|
- 被测插件须已装进所选 profile:`dsh plugin --profile <profile> add <插件目录>`;
|
package/docs/matchers.md
CHANGED
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
---
|
|
2
|
-
description: trace matcher 与 mock helper 全集——工具面/文本面/输入面/派发面断言语义(toolCalled 到 subagentCompletedCount)与 toolCallStep/textStep 脚本构件
|
|
2
|
+
description: trace matcher 与 mock helper 全集——工具面/文本面/输入面/派发面断言语义(toolCalled 到 subagentCompletedCount)、投影普查(trace.census)与 toolCallStep/textStep 脚本构件
|
|
3
3
|
---
|
|
4
4
|
|
|
5
5
|
# Trace matchers 与 mock helpers
|
|
@@ -19,6 +19,7 @@ description: trace matcher 与 mock helper 全集——工具面/文本面/输
|
|
|
19
19
|
| `userMessages` | `{ seq, source, text }[]`(`source` 原样透传:任务 prompt `{ kind: 'user' }`,插件 steer `{ kind: 'plugin', plugin }`) |
|
|
20
20
|
| `requestHeaders` | `{ seq, reason, system, toolNames }[]`(组装后 system prompt + 挂载工具名) |
|
|
21
21
|
| `subagentChildren` | `{ sessionId, parentSession, delegationDepth, label, mode, provider, assistantTexts, finalText }[]`——每个子 agent 独立 session 日志一条;身份(label/mode/provider)取子日志首条 version-3 的 `subagent/descriptor` 事件(镜像宿主 `foldSubagentDescriptor` 的首条权威语义),`finalText` 是子会话自己的最后一条非空 assistant 文本(无则 `''` = 派发了但没答) |
|
|
22
|
+
| `census` | 投影普查(只报数,不判定):`{ eventTypeCounts, projectionLengths, projectionSkipped: { main, children }, projectionFieldGaps, subagent: { mainLogDescriptorEvents, supportedDescriptors, children } }`。语义见下「投影普查」节;手搓 trace(不经 `buildTrace`)时可为 `undefined` |
|
|
22
23
|
| `sessions` / `sessionId` | 原始解析结果 `{ header, events }[]` 与主 session id |
|
|
23
24
|
|
|
24
25
|
`runEvalCase` 返回的 `result.trace` 即此形状(无 session 日志时为 `undefined`;字段语义见 [runner-api.md](runner-api.md))。
|
|
@@ -55,6 +56,26 @@ description: trace matcher 与 mock helper 全集——工具面/文本面/输
|
|
|
55
56
|
|
|
56
57
|
边界:子会话产物(独立 JSONL)经 `subagentChildren` 记录进入断言面(身份 + 子自身文本);子会话内部的工具调用**不**并入主投影的 `toolCalls`/`toolResults`(那属于主会话行为面),需要时经 `sessions` 原始日志自行投影。
|
|
57
58
|
|
|
59
|
+
## 投影普查(`trace.census`)
|
|
60
|
+
|
|
61
|
+
宽松投影(tolerant reader)的补救面:宿主事件 payload 演进时 `buildTrace` 不抛错,只把字段填成空值或丢掉整条记录。空投影会让负向断言真空通过——`toolNotCalled`、`userMessageTextExcludes`、以及 `subagentDispatchCount` / `subagentCompletedCount` 的 `expected === 0` 档都判 ok。**普查只报数,不判定**:它让「宿主日志里本来就没有这类事件」与「有事件但投影丢掉了」在报告里可分,是否降级由人判读。
|
|
62
|
+
|
|
63
|
+
三个信号,对应三种坏法:
|
|
64
|
+
|
|
65
|
+
| 信号 | 看什么 |
|
|
66
|
+
|---|---|
|
|
67
|
+
| 主 session 事件(`eventTypeCounts` / `projectionLengths` / `projectionSkipped.main`) | 主日志(`buildTrace` 的投影输入)逐事件类型计数(任何类型,含插件扩展类型);五个投影的长度;以及**每个投影上「计数 − 长度 > 0」的差额**(`projectionSkipped.main`,按投影字段名)——记录被丢了的档 |
|
|
68
|
+
| **字段级缺口**(`projectionFieldGaps`) | 记录**留住了但字段读不到**的事件,按缺什么计数:`toolCallWithoutName` / `toolCallWithoutCallId` / `toolResultWithoutCallId` / `headerWithoutSystem` / `headerWithoutToolNames`。`tool/call`、`tool/result`、`request/header` 是 1:1 投影(计数 − 长度恒为 0),宿主搬字段时只在这里可见。**不计数**:`request/header` 的 `tools` 数组整个缺失(与真空列表投影一致) |
|
|
69
|
+
| 子会话(`census.subagent`) | `subagentChildren` 的输入面:`children[]` 逐条给该子日志的 `subagent/descriptor` 事件数、其中 `version === 3` 的条数(`supportedDescriptors`,**数事件不是数子会话**)**以及折叠出的身份**(`label` / `mode` / `provider`);`projectionSkipped.children` 两个身份计数——`withoutIdentity`(三项全缺)与 `withoutLabel`(`label` 缺,哪怕 mode/provider 有)。`mainLogDescriptorEvents` 是**主日志自己**的 `subagent/descriptor` 事件数(现宿主把 descriptor 写进子日志,这个数通常为 0)。**子日志不是主日志**,`eventTypeCounts` 不统计它们 |
|
|
70
|
+
|
|
71
|
+
判读要点:
|
|
72
|
+
|
|
73
|
+
- **差额 ≠ 缺陷**。`assistant/message`、`user/message` 的**空文本消息是设计上整条丢弃**(保护「组装文本」投影语义),这类差额属合法,普查不替你做白名单;
|
|
74
|
+
- **两类信号别混**:`projectionSkipped.main` 看「记录被丢了」,`projectionFieldGaps` 看「记录在、字段没了」。后者正是 `toolNotCalled` 最危险的形态——调用记录还在、`name` 为 `undefined`,`nameMatches` 对任何 matcher 都不命中,负向断言照绿;
|
|
75
|
+
- **身份缺失型降级**:某子日志 `descriptorEvents > 0` 而 `supportedDescriptors === 0`,即它进了 `subagentChildren` 但身份全空;但**只要 `label` 缺**(`withoutLabel`),按 label 匹配的 `*Count(label, 0)` 就会真空通过——哪怕 `supportedDescriptors` 看起来健康、mode/provider 都在。两个计数分开报就是为了这个档;
|
|
76
|
+
- **子会话集合是启发式**:`subagentChildren` 收「header 带 `parentSession`」的日志,而宿主对 fork/resume/seed 日志也写这个字段——它们会以「无身份子记录」出现在普查里。这是集合的性质,不是本次降级(日志层无法复现宿主的 agent 链所有权判定);
|
|
77
|
+
- 只出现在**运行面**:`--format json` 的每条 case 记录(`census` 字段,pass 与 fail 都带;**无 trace 的记录没有**)与 `.runs/<id>/trace.json` 的 `trace.census`;**文本输出零新增**(逐字节输出契约不动),失败文案也不带计数。
|
|
78
|
+
|
|
58
79
|
## Mock script helpers
|
|
59
80
|
|
|
60
81
|
- `toolCallStep(name, args)`:一步「模型调工具」,结束于 tool-calls;
|
package/docs/review.md
CHANGED
|
@@ -55,6 +55,8 @@ dsh-review \
|
|
|
55
55
|
|
|
56
56
|
真实运行的 reviewer 会话默认在**空白环境**启动:适配器先照常暂存所选 profile,再枚举它组合出的**全部树外插件行**(`package.json` 的 `dsh.profile.bundles` 中非 `@deepseek-ai/*` 的 bundle 各自 patch 文件里的行,加上 profile 自有 `cordis.patch.yml` 的行),在 `--patch` overlay 里逐行禁用——宿主 profile 装了什么 gates/插件都与 reviewer 无关,可复现性不再依赖「本机 profile 恰好干净」。白名单保留 reviewer 起不来就无测可言的接线行(`agent-default-model`、`session-title-llm`、`system-prompt`、`session-persistence-jsonl`);宿主模板工具行(`tool-fs`、shell、web、subagent 等)由静态清单继续禁用,cwd 指向空临时目录——reviewer 只能从物化的观测文本推理。运行后解析 session trace 的 `request/header` 事件做**工具边界校验**:发现任何非预期工具即视为 adapter failure(证据写入 `.runs/<id>/run-N.tool-boundary-evidence.json`)。
|
|
57
57
|
|
|
58
|
+
**校验没跑成也要记账**:收不到 session artifact(宿主 artifact 命名或会话格式换代是最常见的成因)时 `validateToolBoundary` 返回 `status: 'not-executed'` 而**不是**通过,适配器把该状态与 seam 诊断挂上执行结果;报告头写明 `tool boundary: NOT EXECUTED on run(s) N`,对应轮次的 `run-N.txt` 顶部与报告条目也各写一行原因,`run.json` 的 `toolBoundaries` 给机器读。这是有意的取向:**「没验证」必须看起来像没验证**——静默 fail-open 会让一份「工具边界从未检查过」的 review 产物读起来完全正常。
|
|
59
|
+
|
|
58
60
|
**刻意复用宿主插件面**(例如要评审某插件自己的 gate 行为):加 `--keep-plugin-rows`——跳过树外行枚举,仅保留静态工具禁用,宿主 gates 恢复运行。
|
|
59
61
|
|
|
60
62
|
> 注意:无 `id` 的组合条目对 id 定位的禁用天然不可见(宿主 loader 语义),本包的树外 bundle 生态均为带 id 行形态;发现无 id 树外行时以工具边界校验 fail-loud 兜底。白名单是**按行名**无条件保留——若某树外 bundle 刻意以白名单名(如 `system-prompt`)insert 自己的行,该行不会被禁(威胁模型是本机自己的 profile,非对抗面);此类泄漏同样由工具边界校验兜底。
|
|
@@ -65,9 +67,9 @@ dsh-review \
|
|
|
65
67
|
|
|
66
68
|
- `observations.md`:本次实时物化的可见证据;
|
|
67
69
|
- `task.txt`:实际发给每位 reviewer 的完整任务;
|
|
68
|
-
- `run-N.txt`:该轮 reviewer 的**答案**——trace 推导(首条插件注入消息之前的最后一条 assistant 文本,抗回合尾部劫持),无 trace 时回落到 stdout 的最终消息;劫持发生时原始最终消息另存 `run-N.stdout.txt
|
|
70
|
+
- `run-N.txt`:该轮 reviewer 的**答案**——trace 推导(首条插件注入消息之前的最后一条 assistant 文本,抗回合尾部劫持),无 trace 时回落到 stdout 的最终消息;劫持发生时原始最终消息另存 `run-N.stdout.txt`;工具边界校验未执行时,本条文件顶部先写一行 `[tool-boundary: NOT EXECUTED — <原因>]`(答案本身退回 stdout 的最终消息);
|
|
69
71
|
- `run-N.stderr.txt` / error:完整会话转录(判读回溯的 transcript 指针,报告每轮引用)/ 失败原因;
|
|
70
|
-
- `run.json`:experiment、rubric、adapter、profile
|
|
72
|
+
- `run.json`:experiment、rubric、adapter、profile、run 数与每轮 `toolBoundaries` 记账(`checked` / `not-executed` + 原因);
|
|
71
73
|
- `review-report.md`:判读报告骨架——机器字段自动填(experiment/adapter/ profile/runs、observations 指纹、rubric 位置、每轮 reviewer **答案**与 transcript 指针),三个 **人工判读栏目**留白待填:intentional design 命中项、新 red flag、下一步(改输出 / 改 rubric / 改 behavior case / 不处理)。刻意不做自动评分——review 层的价值在人工判断,报告只把判断物化成可归档、可对比的工程证据(dry-run 也会生成,runs 记 0)。
|
|
72
74
|
|
|
73
75
|
## 六条评审规则
|
package/docs/runner-api.md
CHANGED
|
@@ -17,10 +17,9 @@ const result = await runEvalCase(evalCase, { profile: 'headless', cliPath, mode:
|
|
|
17
17
|
| 键 | 类型 | 语义 |
|
|
18
18
|
|---|---|---|
|
|
19
19
|
| `profile` | `string` | 必填。承载被测插件的 dsh profile(沙箱会暂存其 store,不污染真实 home)。 |
|
|
20
|
-
| `cliPath` | `string` |
|
|
20
|
+
| `cliPath` | `string` | 必填。编译好的 dsh CLI 入口(`apps/cli/lib/bin.js`)绝对路径(`resolveDshCliChain` 结果)。程序化取值见下节。 |
|
|
21
21
|
| `mode` | `'real' \| 'mock'` | 覆写 case 自带的 mode;mock 需 `script.steps`。 |
|
|
22
22
|
| `artifactsDir` | `string` | 提供则把 stdout/stderr/trace/session 日志拷贝到该目录(自动创建)。 |
|
|
23
|
-
| `dshRepoDir` | `string` | **已弃用**:宿主 checkout 目录(从中拼出 CLI 路径)。下个 minor 删除——迁移到 `cliPath`。 |
|
|
24
23
|
|
|
25
24
|
## `cliPath` 从哪来:跨档关系(读我)
|
|
26
25
|
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@catheadowl/dsh-eval",
|
|
3
3
|
"description": "Agent eval framework over dsh headless runs: case runner, session-trace assertions, and a scripted mock-LLM layer for plugin intent tests.",
|
|
4
|
-
"version": "0.
|
|
4
|
+
"version": "0.3.0",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"repository": {
|
|
7
7
|
"type": "git",
|
|
@@ -42,6 +42,6 @@
|
|
|
42
42
|
"verify:experimental": "node scripts/verify-experimental-face.mjs",
|
|
43
43
|
"verify:readme-i18n": "node scripts/verify-readme-i18n.mjs",
|
|
44
44
|
"prepack": "node scripts/verify-manifest-face.mjs && node scripts/verify-publish-readiness.mjs && node scripts/verify-readme-i18n.mjs && node scripts/verify-experimental-face.mjs",
|
|
45
|
-
"test": "node --test --test-isolation=none tests/trace.test.mjs tests/assertions.test.mjs tests/overlay.test.mjs tests/staging.test.mjs tests/runner.test.mjs tests/cli.test.mjs tests/discovery.test.mjs tests/review.test.mjs tests/report.test.mjs tests/config.test.mjs tests/review-report.test.mjs && node scripts/verify-publish-readiness.mjs && node scripts/verify-manifest-face.mjs && node scripts/verify-readme-i18n.mjs && node scripts/verify-experimental-face.mjs"
|
|
45
|
+
"test": "node --test --test-isolation=none tests/trace.test.mjs tests/assertions.test.mjs tests/overlay.test.mjs tests/staging.test.mjs tests/runner.test.mjs tests/cli.test.mjs tests/cli-report.test.mjs tests/discovery.test.mjs tests/review.test.mjs tests/tool-validation.test.mjs tests/report.test.mjs tests/config.test.mjs tests/review-report.test.mjs && node scripts/verify-publish-readiness.mjs && node scripts/verify-manifest-face.mjs && node scripts/verify-readme-i18n.mjs && node scripts/verify-experimental-face.mjs"
|
|
46
46
|
}
|
|
47
47
|
}
|
|
@@ -1,15 +1,14 @@
|
|
|
1
1
|
/** dsh-headless execution adapter for model-independent review experiments. */
|
|
2
2
|
|
|
3
|
-
import {
|
|
3
|
+
import { mkdtempSync, writeFileSync } from 'node:fs'
|
|
4
4
|
import { tmpdir } from 'node:os'
|
|
5
5
|
import { join, resolve } from 'node:path'
|
|
6
6
|
import { executeReviewExperiment } from '../../experiment/review.mjs'
|
|
7
|
-
import { CLI_RELATIVE_PATH } from '../../cli.mjs'
|
|
8
7
|
import { overlayDisableRows } from '../../overlay.mjs'
|
|
9
8
|
import {
|
|
10
9
|
resolveRealDshHome, stageSandboxHome, stagedPluginRows, teardownSandbox, spawnHeadlessDsh,
|
|
11
10
|
} from '../../sandbox.mjs'
|
|
12
|
-
import {
|
|
11
|
+
import { collectSessionTrace } from '../../trace.mjs'
|
|
13
12
|
import { validateToolBoundary, renderToolBoundaryEvidence } from '../../tool-validation.mjs'
|
|
14
13
|
|
|
15
14
|
/**
|
|
@@ -70,23 +69,14 @@ function buildReviewOverlayYaml(pluginRows, { keepPluginRows }) {
|
|
|
70
69
|
return overlayDisableRows([...disabled])
|
|
71
70
|
}
|
|
72
71
|
|
|
73
|
-
/**
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
const cli = join(repoDir, ...CLI_RELATIVE_PATH.split(/[\\/]/))
|
|
77
|
-
if (!existsSync(cli)) {
|
|
78
|
-
throw new Error(`no compiled dsh CLI at '${cli}' (build deepseek-harness first)`)
|
|
79
|
-
}
|
|
80
|
-
return cli
|
|
81
|
-
}
|
|
82
|
-
|
|
83
|
-
/** The CLI entry for an executor: explicit cliPath (C6 chain result) wins;
|
|
84
|
-
* otherwise fall back to the legacy repo form. Neither being set is a caller
|
|
85
|
-
* bug the CLI bins already catch — this guard serves direct API consumers. */
|
|
72
|
+
/** The CLI entry for an executor: the explicit cliPath (a `resolveDshCliChain`
|
|
73
|
+
* result). Its absence is a caller bug the CLI bins already catch — this
|
|
74
|
+
* guard serves direct API consumers. */
|
|
86
75
|
function executorCli(options) {
|
|
87
|
-
if (options.cliPath
|
|
88
|
-
|
|
89
|
-
|
|
76
|
+
if (options.cliPath === undefined) {
|
|
77
|
+
throw new Error('review adapter needs options.cliPath (a resolveDshCliChain result)')
|
|
78
|
+
}
|
|
79
|
+
return resolve(options.cliPath)
|
|
90
80
|
}
|
|
91
81
|
|
|
92
82
|
/**
|
|
@@ -162,39 +152,43 @@ export function createDshHeadlessReviewExecutor(options) {
|
|
|
162
152
|
|
|
163
153
|
// Post-run tool boundary check: parse the session
|
|
164
154
|
// trace, verify no unexpected tools were mounted in the reviewer's
|
|
165
|
-
// session, fail the run on violation.
|
|
166
|
-
// the check
|
|
155
|
+
// session, fail the run on violation. A missing session log does NOT
|
|
156
|
+
// skip the check silently: `validateToolBoundary` reports
|
|
157
|
+
// `status: 'not-executed'`, and the adapter carries that fact (plus the
|
|
158
|
+
// seam diagnosis) on the result so the report and run artifacts state
|
|
159
|
+
// that the boundary guarantee was not verified (EVAL-021).
|
|
167
160
|
// Validation inspects the main session only (buildTrace selects
|
|
168
161
|
// non-subagent logs); plugin tools leaking in a subagent session
|
|
169
162
|
// would not be caught — irrelevant in review where the overlay
|
|
170
163
|
// disables every subagent tool row.
|
|
171
|
-
const trace =
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
|
|
185
|
-
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
|
|
189
|
-
// splice that slipped past the blank environment, an infra
|
|
190
|
-
// complaint), stdout holds that instead of the analysis. The
|
|
191
|
-
// trace's answerText (last assistant text before the first
|
|
192
|
-
// plugin-sourced injection) IS the analysis; stdout remains the
|
|
193
|
-
// fallback for trace-less runs.
|
|
194
|
-
result.answer = trace.answerText !== '' ? trace.answerText : stdout
|
|
195
|
-
} else {
|
|
196
|
-
result.answer = stdout
|
|
164
|
+
const { trace, gap } = collectSessionTrace(join(dshHome, 'sessions'))
|
|
165
|
+
const validation = validateToolBoundary(trace, { allowedTools })
|
|
166
|
+
result.toolValidation = validation
|
|
167
|
+
if (validation.status === 'not-executed') {
|
|
168
|
+
// `traceGap` mirrors the behavior runner's field name for the same
|
|
169
|
+
// seam diagnosis; the bin writes both into run-N.txt and the report.
|
|
170
|
+
result.traceGap = gap
|
|
171
|
+
}
|
|
172
|
+
if (validation.status === 'checked' && !validation.ok) {
|
|
173
|
+
// Attach evidence for the caller to persist (the adapter's
|
|
174
|
+
// runDir is ephemeral — removed by the finally block). The
|
|
175
|
+
// bin writes this to `.runs/<id>/tool-boundary-evidence.json`.
|
|
176
|
+
result.toolBoundaryEvidence = renderToolBoundaryEvidence(validation, { runDir, profile })
|
|
177
|
+
const boundaryError = new Error(
|
|
178
|
+
`tool boundary violation: unexpected tools [${validation.unexpected.join(', ')}]`,
|
|
179
|
+
)
|
|
180
|
+
boundaryError.result = result
|
|
181
|
+
throw boundaryError
|
|
197
182
|
}
|
|
183
|
+
// The ANSWER to the task, not the last message: stdout carries the
|
|
184
|
+
// headless CLI's final assistant message — whatever the reviewer
|
|
185
|
+
// said LAST. If any tail interaction intervened (a turn-close gate
|
|
186
|
+
// splice that slipped past the blank environment, an infra
|
|
187
|
+
// complaint), stdout holds that instead of the analysis. The
|
|
188
|
+
// trace's answerText (last assistant text before the first
|
|
189
|
+
// plugin-sourced injection) IS the analysis; stdout remains the
|
|
190
|
+
// fallback for trace-less runs (recorded by `traceGap` above).
|
|
191
|
+
result.answer = trace !== undefined && trace.answerText !== '' ? trace.answerText : stdout
|
|
198
192
|
|
|
199
193
|
return result
|
|
200
194
|
} finally {
|
|
@@ -60,7 +60,10 @@ function isSubagentSession(session) {
|
|
|
60
60
|
function summarize(agent) {
|
|
61
61
|
let text = ''
|
|
62
62
|
let reason
|
|
63
|
-
|
|
63
|
+
// `snapshotEvents()` replaced the removed `session.events` getter upstream
|
|
64
|
+
// (5660f44d29); reading the dead property throws "agent.session.events is not
|
|
65
|
+
// iterable" and kills the whole headless run.
|
|
66
|
+
for (const event of agent.session.snapshotEvents()) {
|
|
64
67
|
if (event.type === 'assistant/message') {
|
|
65
68
|
const joined = event.data.message.content
|
|
66
69
|
.filter(block => block.type === 'text')
|
package/src/experimental.mjs
CHANGED
|
@@ -16,7 +16,12 @@ export { stageProfileStore } from './sandbox.mjs'
|
|
|
16
16
|
export { buildOverlayYaml, overlayDisableRows } from './overlay.mjs'
|
|
17
17
|
|
|
18
18
|
// --- session-trace primitives ---
|
|
19
|
-
export {
|
|
19
|
+
export {
|
|
20
|
+
parseSessionLog,
|
|
21
|
+
buildTrace,
|
|
22
|
+
collectSessionTrace,
|
|
23
|
+
KNOWN_SESSION_FORMAT_VERSIONS,
|
|
24
|
+
} from './trace.mjs'
|
|
20
25
|
|
|
21
26
|
// --- review experiment execution layer ---
|
|
22
27
|
export {
|
|
@@ -40,6 +40,20 @@ class EvalMockAdapter extends LlmAdapter {
|
|
|
40
40
|
}
|
|
41
41
|
}
|
|
42
42
|
|
|
43
|
+
// Carried explicitly instead of inherited: this plugin's `LlmAdapter` base
|
|
44
|
+
// resolves from THIS package's peer instance, which can lag the host
|
|
45
|
+
// runtime that drives it (host 0.1.5-rc.2 grew `prepareCall` while the
|
|
46
|
+
// local peer was 0.0.1-rc.1 — the inherited face was missing and every mock
|
|
47
|
+
// run died at `registration.adapter.prepareCall is not a function`). The
|
|
48
|
+
// shape is the host base-class default: model metadata plus a dispatch
|
|
49
|
+
// entry bound to this same adapter generation.
|
|
50
|
+
async prepareCall(provider, model, signal) {
|
|
51
|
+
return {
|
|
52
|
+
model: await this.resolveModel(provider, model, signal),
|
|
53
|
+
stream: options => this.stream(options),
|
|
54
|
+
}
|
|
55
|
+
}
|
|
56
|
+
|
|
43
57
|
async * stream(_options) {
|
|
44
58
|
const step = this.steps[this.cursor]
|
|
45
59
|
this.cursor += 1
|
package/src/report.mjs
CHANGED
|
@@ -30,6 +30,10 @@
|
|
|
30
30
|
* @param {boolean} [parts.timedOut]
|
|
31
31
|
* @param {number} [parts.durationMs] - wall time of the run, when it ran.
|
|
32
32
|
* @param {string} [parts.artifactsDir] - where post-mortem artifacts landed, when written.
|
|
33
|
+
* @param {object} [parts.census] - the run trace's projection census
|
|
34
|
+
* (`trace.census`), carried on both pass and fail records so a green case
|
|
35
|
+
* whose evidence surface degraded is still inspectable after the fact.
|
|
36
|
+
* Absent when no trace materialized.
|
|
33
37
|
*/
|
|
34
38
|
export function createCaseRecord(parts) {
|
|
35
39
|
const record = { id: parts.id, file: parts.file }
|
|
@@ -41,6 +45,7 @@ export function createCaseRecord(parts) {
|
|
|
41
45
|
if (parts.timedOut !== undefined) record.timedOut = parts.timedOut
|
|
42
46
|
if (parts.durationMs !== undefined) record.durationMs = parts.durationMs
|
|
43
47
|
if (parts.artifactsDir !== undefined) record.artifactsDir = parts.artifactsDir
|
|
48
|
+
if (parts.census !== undefined) record.census = parts.census
|
|
44
49
|
return record
|
|
45
50
|
}
|
|
46
51
|
|
package/src/review-report.mjs
CHANGED
|
@@ -23,6 +23,29 @@ export function observationsFingerprint(observations) {
|
|
|
23
23
|
return createHash('sha256').update(observations, 'utf8').digest('hex').slice(0, 16)
|
|
24
24
|
}
|
|
25
25
|
|
|
26
|
+
/**
|
|
27
|
+
* One-line statement of whether the tool boundary check RAN on the review
|
|
28
|
+
* runs. A run without a session artifact reports `status: 'not-executed'`;
|
|
29
|
+
* that must show up in the report header, because a review whose tool face was
|
|
30
|
+
* never verified is weaker evidence than one whose was (EVAL-021). Executors
|
|
31
|
+
* that do not validate at all (custom executors, dry runs) say so instead of
|
|
32
|
+
* implying a pass.
|
|
33
|
+
*/
|
|
34
|
+
function toolBoundarySummary(attempts) {
|
|
35
|
+
const skipped = attempts.filter(attempt => attempt.result?.toolValidation?.status === 'not-executed')
|
|
36
|
+
if (skipped.length > 0) {
|
|
37
|
+
return `NOT EXECUTED on run(s) ${skipped.map(attempt => attempt.index).join(', ')}`
|
|
38
|
+
+ ' — no session artifact; the reviewer tool face was not verified (see the per-run notes)'
|
|
39
|
+
}
|
|
40
|
+
const checked = attempts.filter(attempt => attempt.result?.toolValidation?.status === 'checked')
|
|
41
|
+
if (attempts.length > 0 && checked.length === attempts.length) return 'checked on every run'
|
|
42
|
+
if (checked.length > 0) {
|
|
43
|
+
return `checked on ${checked.length} of ${attempts.length} run(s); the rest reported none`
|
|
44
|
+
+ ' (failed before validation, or a non-verifying executor)'
|
|
45
|
+
}
|
|
46
|
+
return 'not reported by this executor'
|
|
47
|
+
}
|
|
48
|
+
|
|
26
49
|
/**
|
|
27
50
|
* Render the review report markdown.
|
|
28
51
|
*
|
|
@@ -54,6 +77,7 @@ export function renderReviewReport(parts) {
|
|
|
54
77
|
lines.push(`- adapter: ${parts.adapter ?? 'none (dry run)'}`)
|
|
55
78
|
if (parts.profile !== undefined) lines.push(`- profile: \`${parts.profile}\``)
|
|
56
79
|
lines.push(`- runs: ${dry ? '0 (dry run — observations materialized only)' : result.runs}`)
|
|
80
|
+
if (!dry) lines.push(`- tool boundary: ${toolBoundarySummary(result.attempts)}`)
|
|
57
81
|
lines.push(`- observations: \`observations.md\` (sha256:${observationsFingerprint(observations)})`)
|
|
58
82
|
lines.push(`- rubric: ${rubric.includes('\n') ? '(inline string — see experiment definition)' : `\`${rubric}\``}`)
|
|
59
83
|
lines.push('')
|
|
@@ -82,6 +106,13 @@ export function renderReviewReport(parts) {
|
|
|
82
106
|
&& attempt.result.answer !== attempt.result.stdout
|
|
83
107
|
lines.push('')
|
|
84
108
|
lines.push(`- transcript: \`run-${attempt.index}.stderr.txt\`${diverged ? ` (answer differs from the final message — see \`run-${attempt.index}.stdout.txt\`)` : ''}`)
|
|
109
|
+
if (attempt.result?.toolValidation?.status === 'not-executed') {
|
|
110
|
+
// The unverified-boundary fact rides the run it applies to, next to
|
|
111
|
+
// the answer it qualifies: this answer is stdout's final message
|
|
112
|
+
// (there was no trace to derive it from) and no request header was
|
|
113
|
+
// ever inspected for tool leakage.
|
|
114
|
+
lines.push(`- **tool-boundary check NOT EXECUTED**: ${attempt.result.traceGap ?? 'no session trace materialized'} — the answer above is stdout's final message, not a trace-derived answer.`)
|
|
115
|
+
}
|
|
85
116
|
} else {
|
|
86
117
|
lines.push(`### run ${attempt.index} — FAIL`)
|
|
87
118
|
lines.push('')
|
package/src/runner.mjs
CHANGED
|
@@ -29,11 +29,11 @@
|
|
|
29
29
|
* module is the orchestration only.
|
|
30
30
|
*/
|
|
31
31
|
|
|
32
|
-
import { existsSync, mkdirSync, mkdtempSync, writeFileSync, cpSync
|
|
32
|
+
import { existsSync, mkdirSync, mkdtempSync, readFileSync, writeFileSync, cpSync } from 'node:fs'
|
|
33
33
|
import { tmpdir } from 'node:os'
|
|
34
34
|
import { isAbsolute, join, resolve } from 'node:path'
|
|
35
35
|
import { fileURLToPath } from 'node:url'
|
|
36
|
-
import {
|
|
36
|
+
import { collectSessionTrace, listSessionLogFiles } from './trace.mjs'
|
|
37
37
|
import { validateRowConfig, validateDisableRows, validateFollowups } from './discovery.mjs'
|
|
38
38
|
import { CLI_RELATIVE_PATH } from './cli.mjs'
|
|
39
39
|
import { buildOverlayYaml } from './overlay.mjs'
|
|
@@ -62,21 +62,18 @@ const FRAMEWORK_ROOT = fileURLToPath(new URL('..', import.meta.url))
|
|
|
62
62
|
* @param {object} evalCase - the case under test.
|
|
63
63
|
* @param {object} options
|
|
64
64
|
* @param {string} options.profile - the dsh profile booting the run (plugin installed there).
|
|
65
|
-
* @param {string}
|
|
66
|
-
*
|
|
67
|
-
* @deprecated options.dshRepoDir — pass the C6 chain result via cliPath
|
|
68
|
-
* instead; this legacy option is removed in the next minor release.
|
|
69
|
-
* @param {string} [options.cliPath] - explicit compiled CLI entry (C6 chain result;
|
|
70
|
-
* takes precedence over dshRepoDir).
|
|
65
|
+
* @param {string} options.cliPath - the compiled dsh CLI entry (a
|
|
66
|
+
* `resolveDshCliChain` result); required.
|
|
71
67
|
* @param {'real' | 'mock'} [options.mode] - force a mode over the case's own.
|
|
72
68
|
* @param {string} [options.artifactsDir] - copy stdout/stderr/trace/session logs here (created).
|
|
73
69
|
* @returns {Promise<EvalRunResult>}
|
|
74
70
|
*/
|
|
75
71
|
export async function runEvalCase(evalCase, options) {
|
|
76
72
|
const mode = options.mode ?? evalCase.mode ?? 'real'
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
73
|
+
if (options.cliPath === undefined) {
|
|
74
|
+
throw new TypeError('runEvalCase needs options.cliPath (a resolveDshCliChain result)')
|
|
75
|
+
}
|
|
76
|
+
const binPath = resolve(options.cliPath)
|
|
80
77
|
const timeoutMs = evalCase.timeoutMs ?? 180_000
|
|
81
78
|
|
|
82
79
|
const runDir = mkdtempSync(join(tmpdir(), 'dsh-eval-'))
|
|
@@ -150,7 +147,7 @@ export async function runEvalCase(evalCase, options) {
|
|
|
150
147
|
timeoutMs,
|
|
151
148
|
})
|
|
152
149
|
|
|
153
|
-
const trace =
|
|
150
|
+
const { trace, gap: traceGap } = collectSessionTrace(sessionsRoot)
|
|
154
151
|
const sessionLogs = collectSessionLogTexts(sessionsRoot)
|
|
155
152
|
|
|
156
153
|
// Workspace assertions live HERE, before the run dir cleanup: a case's
|
|
@@ -184,7 +181,7 @@ export async function runEvalCase(evalCase, options) {
|
|
|
184
181
|
|
|
185
182
|
return {
|
|
186
183
|
caseId: evalCase.id, mode, task: evalCase.task, exitCode, timedOut,
|
|
187
|
-
stdout, stderr, trace, sessionLogs, inspectError, runDir,
|
|
184
|
+
stdout, stderr, trace, traceGap, sessionLogs, inspectError, runDir,
|
|
188
185
|
}
|
|
189
186
|
} finally {
|
|
190
187
|
teardownSandbox(runDir, { keep: process.env.DSH_EVAL_KEEP_TMP === '1' })
|
|
@@ -193,22 +190,7 @@ export async function runEvalCase(evalCase, options) {
|
|
|
193
190
|
|
|
194
191
|
/** Read every session artifact under the root as text (best-effort, pre-cleanup). */
|
|
195
192
|
function collectSessionLogTexts(sessionsRoot) {
|
|
196
|
-
|
|
197
|
-
const walk = (dir) => {
|
|
198
|
-
let entries
|
|
199
|
-
try {
|
|
200
|
-
entries = readdirSync(dir, { withFileTypes: true })
|
|
201
|
-
} catch {
|
|
202
|
-
return
|
|
203
|
-
}
|
|
204
|
-
for (const entry of entries) {
|
|
205
|
-
const path = join(dir, entry.name)
|
|
206
|
-
if (entry.isDirectory()) walk(path)
|
|
207
|
-
else if (entry.name === 'session.jsonl') texts.push(readFileSync(path, 'utf8'))
|
|
208
|
-
}
|
|
209
|
-
}
|
|
210
|
-
walk(sessionsRoot)
|
|
211
|
-
return texts
|
|
193
|
+
return listSessionLogFiles(sessionsRoot).map(path => readFileSync(path, 'utf8'))
|
|
212
194
|
}
|
|
213
195
|
|
|
214
196
|
/**
|
|
@@ -221,6 +203,9 @@ function collectSessionLogTexts(sessionsRoot) {
|
|
|
221
203
|
* @property {string} stdout - printed final assistant text (plus any startup chatter).
|
|
222
204
|
* @property {string} stderr
|
|
223
205
|
* @property {import('./trace.mjs').EvalTrace | undefined} trace
|
|
206
|
+
* @property {string | undefined} traceGap - why no trace was built (the host
|
|
207
|
+
* session seam diagnosis from `collectSessionTrace`); `undefined` whenever
|
|
208
|
+
* `trace` is defined. The CLI prints it as the failure text.
|
|
224
209
|
* @property {string[]} sessionLogs - raw session artifact texts, pre-cleanup.
|
|
225
210
|
* @property {string | undefined} inspectError - the case's `inspect` failure text, when it threw.
|
|
226
211
|
* @property {string} runDir - removed unless DSH_EVAL_KEEP_TMP=1.
|
package/src/tool-validation.mjs
CHANGED
|
@@ -37,19 +37,27 @@ function collectMountedToolNames(trace) {
|
|
|
37
37
|
* against the allowed set. An empty allowed set (the review default)
|
|
38
38
|
* means the reviewer must see no tools at all.
|
|
39
39
|
*
|
|
40
|
-
*
|
|
40
|
+
* The result is explicit about whether the check RAN: `status` is
|
|
41
|
+
* `'checked'` only when a trace was available, and `'not-executed'` when
|
|
42
|
+
* there was none. `ok` is true only for a checked boundary with no
|
|
43
|
+
* unexpected tool — a missing trace is NOT a pass, so callers must branch
|
|
44
|
+
* on `status` and account for the skip (see `createDshHeadlessReviewExecutor`
|
|
45
|
+
* and docs/review.md) instead of reading `ok` alone.
|
|
46
|
+
*
|
|
47
|
+
* @param {import('./trace.mjs').EvalTrace | undefined} trace - the parsed trace; `undefined` means the check could not run.
|
|
41
48
|
* @param {{ allowedTools?: Set<string> }} [options]
|
|
42
|
-
* @returns {{ ok: boolean, unexpected: string[], actual: string[], allowed: string[] }}
|
|
49
|
+
* @returns {{ status: 'checked' | 'not-executed', ok: boolean, unexpected: string[], actual: string[], allowed: string[] }}
|
|
43
50
|
*/
|
|
44
51
|
export function validateToolBoundary(trace, options = {}) {
|
|
45
52
|
const allowed = options.allowedTools ?? new Set()
|
|
46
53
|
const allowedNames = [...allowed].sort()
|
|
47
54
|
if (trace === undefined || trace === null) {
|
|
48
|
-
return { ok:
|
|
55
|
+
return { status: 'not-executed', ok: false, unexpected: [], actual: [], allowed: allowedNames }
|
|
49
56
|
}
|
|
50
57
|
const actual = collectMountedToolNames(trace)
|
|
51
58
|
const unexpected = actual.filter(name => !allowed.has(name))
|
|
52
59
|
return {
|
|
60
|
+
status: 'checked',
|
|
53
61
|
ok: unexpected.length === 0,
|
|
54
62
|
unexpected,
|
|
55
63
|
actual,
|
|
@@ -61,7 +69,7 @@ export function validateToolBoundary(trace, options = {}) {
|
|
|
61
69
|
* Render a diagnostic evidence document for a tool boundary failure.
|
|
62
70
|
* Suitable for writing to `.runs/<id>/tool-boundary-evidence.json`.
|
|
63
71
|
*
|
|
64
|
-
* @param {{ ok: boolean, unexpected: string[], actual: string[], allowed: string[] }} validation
|
|
72
|
+
* @param {{ status: 'checked' | 'not-executed', ok: boolean, unexpected: string[], actual: string[], allowed: string[] }} validation
|
|
65
73
|
* @param {{ runDir: string, profile: string }} context
|
|
66
74
|
* @returns {string}
|
|
67
75
|
*/
|
package/src/trace.mjs
CHANGED
|
@@ -6,14 +6,53 @@
|
|
|
6
6
|
* Event shapes follow `deepseek-harness/packages/core/session/src/types.ts`
|
|
7
7
|
* (`SessionEventMap`); packed `*-chunks` storage rows are tolerated and
|
|
8
8
|
* skipped — they only carry `assistant/chunk` deltas eval never asserts on.
|
|
9
|
+
*
|
|
10
|
+
* Both seam directions carry an explicit boundary: an artifact whose header
|
|
11
|
+
* stamp is not a known generation is refused here (`parseSessionLog`), and a
|
|
12
|
+
* collection that finds no artifact yields a named diagnosis rather than an
|
|
13
|
+
* unexplained `undefined` (`collectSessionTrace`). See docs/host-wiring.md.
|
|
9
14
|
*/
|
|
10
15
|
|
|
11
16
|
import { readdirSync, readFileSync } from 'node:fs'
|
|
12
|
-
import { join } from 'node:path'
|
|
17
|
+
import { basename, join } from 'node:path'
|
|
13
18
|
|
|
14
19
|
/** Storage row types that pack `assistant/chunk` delta runs (see chunk-rows.ts). */
|
|
15
20
|
const CHUNK_ROW_TYPES = new Set(['text-chunks', 'reasoning-chunks', 'tool-call-chunks'])
|
|
16
21
|
|
|
22
|
+
/** Projection field name per projected event type. */
|
|
23
|
+
const PROJECTED_FIELD_BY_EVENT_TYPE = new Map([
|
|
24
|
+
['tool/call', 'toolCalls'],
|
|
25
|
+
['tool/result', 'toolResults'],
|
|
26
|
+
['assistant/message', 'assistantTexts'],
|
|
27
|
+
['user/message', 'userMessages'],
|
|
28
|
+
['request/header', 'requestHeaders'],
|
|
29
|
+
])
|
|
30
|
+
|
|
31
|
+
/**
|
|
32
|
+
* Session format generations this parser accepts. Mirror of the generation
|
|
33
|
+
* chain the vendored host ships codecs for
|
|
34
|
+
* (`session-format-catalog/src/generated.ts`: codecs v0–v3,
|
|
35
|
+
* `currentVersion: 3` = `SESSION_FORMAT_VERSION` in
|
|
36
|
+
* `core/session/src/types.ts`). The projection is written and verified
|
|
37
|
+
* against the current generation; older ones parse tolerantly. A stamp
|
|
38
|
+
* outside this set means the host moved to a generation whose payload this
|
|
39
|
+
* projection was never verified against — fail at the seam instead of
|
|
40
|
+
* projecting empty fields, and bump this set only together with the
|
|
41
|
+
* re-verification the docs' maintenance trigger describes.
|
|
42
|
+
*/
|
|
43
|
+
export const KNOWN_SESSION_FORMAT_VERSIONS = new Set([0, 1, 2, 3])
|
|
44
|
+
|
|
45
|
+
/** Known generations rendered for an error message: `v0, v1, v2, v3`. */
|
|
46
|
+
function knownGenerationsLabel() {
|
|
47
|
+
return [...KNOWN_SESSION_FORMAT_VERSIONS].sort((a, b) => a - b).map(version => `v${version}`).join(', ')
|
|
48
|
+
}
|
|
49
|
+
|
|
50
|
+
/** One header version stamp, rendered compactly for a diagnostic. */
|
|
51
|
+
function headerVersionLabel(version) {
|
|
52
|
+
if (typeof version === 'number') return `v${version}`
|
|
53
|
+
return `(${JSON.stringify(version ?? null)})`
|
|
54
|
+
}
|
|
55
|
+
|
|
17
56
|
/**
|
|
18
57
|
* Parse one uncompressed JSONL session artifact.
|
|
19
58
|
* @param {string} text - the artifact's full text (header line first).
|
|
@@ -24,6 +63,15 @@ export function parseSessionLog(text) {
|
|
|
24
63
|
if (lines.length === 0) throw new Error('empty session log')
|
|
25
64
|
const header = JSON.parse(lines[0])
|
|
26
65
|
if (header.type !== 'session') throw new Error('first line is not a session header')
|
|
66
|
+
// Generation gate: the header stamp is the host's own declaration of the
|
|
67
|
+
// artifact's logical layout. An unknown one is a seam drift, not a parse
|
|
68
|
+
// detail — say so here rather than degrade every projection to empty.
|
|
69
|
+
if (!KNOWN_SESSION_FORMAT_VERSIONS.has(header.version)) {
|
|
70
|
+
throw new Error(
|
|
71
|
+
`session header version ${headerVersionLabel(header.version)} is not a known generation`
|
|
72
|
+
+ ` (known: ${knownGenerationsLabel()}); the host session format may have changed generation`,
|
|
73
|
+
)
|
|
74
|
+
}
|
|
27
75
|
const events = []
|
|
28
76
|
for (const line of lines.slice(1)) {
|
|
29
77
|
let record
|
|
@@ -90,30 +138,49 @@ function parseArguments(raw) {
|
|
|
90
138
|
}
|
|
91
139
|
|
|
92
140
|
/**
|
|
93
|
-
*
|
|
94
|
-
*
|
|
95
|
-
* `
|
|
96
|
-
*
|
|
97
|
-
*
|
|
98
|
-
*
|
|
99
|
-
*
|
|
100
|
-
*
|
|
101
|
-
* never ran to an answer (turn/end reasons are not consulted).
|
|
141
|
+
* Fold one child log's descriptor events exactly once: the identity
|
|
142
|
+
* `projectChild` asserts on, plus the counts the census reports. Single source
|
|
143
|
+
* on purpose — `projectChild` and `censusForChild` must agree on which
|
|
144
|
+
* descriptor established the identity, and the "supported" predicate must match
|
|
145
|
+
* the fold (a log whose only descriptors are unsupported yields both an empty
|
|
146
|
+
* identity and `supportedDescriptors: 0`, which is the census signal).
|
|
147
|
+
* @param {{ events: object[] }} log - one parsed child log.
|
|
148
|
+
* @returns {{ label: string | undefined, mode: string | undefined, provider: string | undefined, descriptorEvents: number, supportedDescriptors: number }}
|
|
102
149
|
*/
|
|
103
|
-
function
|
|
150
|
+
function foldChildDescriptor(log) {
|
|
104
151
|
let label
|
|
105
152
|
let mode
|
|
106
153
|
let provider
|
|
154
|
+
let descriptorEvents = 0
|
|
155
|
+
let supportedDescriptors = 0
|
|
107
156
|
for (const event of log.events) {
|
|
108
157
|
if (event.type !== 'subagent/descriptor') continue
|
|
158
|
+
descriptorEvents += 1
|
|
109
159
|
const data = event.data
|
|
110
160
|
if (data === null || typeof data !== 'object') continue
|
|
111
|
-
if (
|
|
161
|
+
if (data.version === 3) supportedDescriptors += 1
|
|
162
|
+
if (label !== undefined || mode !== undefined || provider !== undefined) continue
|
|
112
163
|
if (data.version !== 3) continue
|
|
113
164
|
if (typeof data.label === 'string') label = data.label
|
|
114
165
|
if (typeof data.mode === 'string') mode = data.mode
|
|
115
166
|
if (typeof data.provider === 'string') provider = data.provider
|
|
116
167
|
}
|
|
168
|
+
return { label, mode, provider, descriptorEvents, supportedDescriptors }
|
|
169
|
+
}
|
|
170
|
+
|
|
171
|
+
/**
|
|
172
|
+
* Project one subagent child log into an assertable record. The durable
|
|
173
|
+
* identity (`label` / `mode` / `provider`) comes from the FIRST
|
|
174
|
+
* `subagent/descriptor` event whose payload carries the descriptor version
|
|
175
|
+
* this projection supports (3) — mirroring `foldSubagentDescriptor` in
|
|
176
|
+
* `deepseek-harness/packages/subagent/subagent/src/descriptor.ts`, where the
|
|
177
|
+
* establishing provider appends exactly one authoritative descriptor and
|
|
178
|
+
* later events cannot rewrite it. Completion is the child's own last
|
|
179
|
+
* assistant text — a child that produced none may have been dispatched but
|
|
180
|
+
* never ran to an answer (turn/end reasons are not consulted).
|
|
181
|
+
*/
|
|
182
|
+
function projectChild(log) {
|
|
183
|
+
const { label, mode, provider } = foldChildDescriptor(log)
|
|
117
184
|
const assistantTexts = log.events
|
|
118
185
|
.filter(event => event.type === 'assistant/message')
|
|
119
186
|
.map(event => messageText(event.data.message))
|
|
@@ -130,6 +197,109 @@ function projectChild(log) {
|
|
|
130
197
|
}
|
|
131
198
|
}
|
|
132
199
|
|
|
200
|
+
/**
|
|
201
|
+
* Count events by type, plug-in event types included (the only live registry is
|
|
202
|
+
* `SessionEventMap`, so no closed list exists).
|
|
203
|
+
* @param {object[]} events - parsed event records.
|
|
204
|
+
* @returns {Record<string, number>} count per event type, insertion-ordered.
|
|
205
|
+
*/
|
|
206
|
+
function countEventTypes(events) {
|
|
207
|
+
// Null-prototype accumulator: a plug-in event type may name an
|
|
208
|
+
// `Object.prototype` member (`constructor`, `toString`, `__proto__`), and
|
|
209
|
+
// `counts['constructor'] ?? 0` would otherwise read the inherited function
|
|
210
|
+
// and string-concatenate, while `__proto__` would be swallowed by its setter.
|
|
211
|
+
const counts = Object.create(null)
|
|
212
|
+
for (const event of events) {
|
|
213
|
+
if (typeof event?.type !== 'string') continue
|
|
214
|
+
counts[event.type] = (counts[event.type] ?? 0) + 1
|
|
215
|
+
}
|
|
216
|
+
return counts
|
|
217
|
+
}
|
|
218
|
+
|
|
219
|
+
/**
|
|
220
|
+
* Census for one candidate child log: whether its `subagent/descriptor`
|
|
221
|
+
* events exist, how many, and how many carry the supported descriptor
|
|
222
|
+
* version, plus the folded identity itself. `label` is the field the
|
|
223
|
+
* subagent-count matchers key on, so `label: undefined` with a non-zero
|
|
224
|
+
* `descriptorEvents` is exactly the shape whose `*Count(label, 0)` assertion
|
|
225
|
+
* is green only because there was nothing to match — the identity-loss
|
|
226
|
+
* degradation the census exists to make visible.
|
|
227
|
+
* @param {{ header: object, events: object[] }} log - one parsed child candidate.
|
|
228
|
+
* @returns {{ sessionId: string | undefined, parentSession: string | undefined, delegationDepth: number | undefined, descriptorEvents: number, supportedDescriptors: number, label: string | undefined, mode: string | undefined, provider: string | undefined }}
|
|
229
|
+
*/
|
|
230
|
+
function censusForChild(log) {
|
|
231
|
+
return {
|
|
232
|
+
sessionId: log.header.id,
|
|
233
|
+
parentSession: log.header.parentSession,
|
|
234
|
+
delegationDepth: log.header.delegationDepth,
|
|
235
|
+
...foldChildDescriptor(log),
|
|
236
|
+
}
|
|
237
|
+
}
|
|
238
|
+
|
|
239
|
+
/**
|
|
240
|
+
* Projection census for one built trace (see {@link EvalTrace.census}).
|
|
241
|
+
* Numbers only: this reports what the raw logs contained against what the
|
|
242
|
+
* projections kept, and never decides whether the difference is a defect.
|
|
243
|
+
*
|
|
244
|
+
* Three signals, because one number cannot cover three shapes: length
|
|
245
|
+
* differences (`projectionSkipped.main`) see records the projection dropped,
|
|
246
|
+
* `projectionFieldGaps` sees records it kept while a field went missing
|
|
247
|
+
* (`tool/call` and friends project 1:1, so their count − length is structurally
|
|
248
|
+
* zero), and the subagent block sees the child-log data source, which no main-log
|
|
249
|
+
* count can reach.
|
|
250
|
+
*
|
|
251
|
+
* @param {object[]} events - the MAIN log's events (the projection input).
|
|
252
|
+
* @param {EvalTrace} trace - the built trace, read for projection lengths.
|
|
253
|
+
* @param {object[]} childLogs - candidate child logs entering `subagentChildren`.
|
|
254
|
+
* @param {Record<string, number>} projectionFieldGaps - gaps the projection loop
|
|
255
|
+
* recorded while reading fields, keyed by what was missing.
|
|
256
|
+
* @returns {object} the census record.
|
|
257
|
+
*/
|
|
258
|
+
function buildCensus(events, trace, childLogs, projectionFieldGaps) {
|
|
259
|
+
const eventTypeCounts = countEventTypes(events)
|
|
260
|
+
const projectionLengths = {
|
|
261
|
+
toolCalls: trace.toolCalls.length,
|
|
262
|
+
toolResults: trace.toolResults.length,
|
|
263
|
+
assistantTexts: trace.assistantTexts.length,
|
|
264
|
+
userMessages: trace.userMessages.length,
|
|
265
|
+
requestHeaders: trace.requestHeaders.length,
|
|
266
|
+
}
|
|
267
|
+
const projectionSkipped = { main: {}, children: {} }
|
|
268
|
+
for (const [type, field] of PROJECTED_FIELD_BY_EVENT_TYPE) {
|
|
269
|
+
const missing = (eventTypeCounts[type] ?? 0) - projectionLengths[field]
|
|
270
|
+
if (missing > 0) projectionSkipped.main[field] = missing
|
|
271
|
+
}
|
|
272
|
+
const children = childLogs.map(censusForChild)
|
|
273
|
+
const supportedDescriptors = children.reduce((total, child) => total + child.supportedDescriptors, 0)
|
|
274
|
+
// Two degradation shapes, different signals: `withoutIdentity` is a child
|
|
275
|
+
// that folded no identity at all; `withoutLabel` is one that folded some
|
|
276
|
+
// identity but no label — the only field the `subagent*Count` matchers can
|
|
277
|
+
// match on, so its zero-count assertions are the vacuous ones.
|
|
278
|
+
//
|
|
279
|
+
// The child set is the parentSession heuristic (any log whose header carries
|
|
280
|
+
// `parentSession`, which the host also writes for fork/resume/seed logs), so a
|
|
281
|
+
// non-subagent fork log shows up here as an identity-less child. The census
|
|
282
|
+
// reports the set it was given; it cannot re-derive the host's agent-chain
|
|
283
|
+
// ownership check from a log alone.
|
|
284
|
+
const withoutIdentity = children.filter(
|
|
285
|
+
child => child.label === undefined && child.mode === undefined && child.provider === undefined,
|
|
286
|
+
).length
|
|
287
|
+
const withoutLabel = children.filter(child => child.label === undefined).length
|
|
288
|
+
if (withoutIdentity > 0) projectionSkipped.children.withoutIdentity = withoutIdentity
|
|
289
|
+
if (withoutLabel > 0) projectionSkipped.children.withoutLabel = withoutLabel
|
|
290
|
+
return {
|
|
291
|
+
eventTypeCounts,
|
|
292
|
+
projectionLengths,
|
|
293
|
+
projectionSkipped,
|
|
294
|
+
projectionFieldGaps,
|
|
295
|
+
subagent: {
|
|
296
|
+
mainLogDescriptorEvents: eventTypeCounts['subagent/descriptor'] ?? 0,
|
|
297
|
+
supportedDescriptors,
|
|
298
|
+
children,
|
|
299
|
+
},
|
|
300
|
+
}
|
|
301
|
+
}
|
|
302
|
+
|
|
133
303
|
/**
|
|
134
304
|
* Build one assertable trace from parsed session logs. Child sessions surface
|
|
135
305
|
* only through the parent's tool events, so the MAIN log (no `origin:
|
|
@@ -146,20 +316,27 @@ export function buildTrace(logs) {
|
|
|
146
316
|
const mains = logs.filter(log => log.header.origin !== 'subagent')
|
|
147
317
|
const main = [...mains].sort((a, b) => b.events.length - a.events.length)[0]
|
|
148
318
|
const events = main?.events ?? []
|
|
149
|
-
const
|
|
319
|
+
const childLogs = logs
|
|
150
320
|
.filter(log => log.header.origin === 'subagent' || log.header.parentSession !== undefined)
|
|
151
|
-
|
|
321
|
+
const subagentChildren = childLogs.map(projectChild)
|
|
152
322
|
|
|
153
323
|
const toolCalls = []
|
|
154
324
|
const toolResults = []
|
|
155
325
|
const assistantEntries = []
|
|
156
326
|
const userMessages = []
|
|
157
327
|
const requestHeaders = []
|
|
328
|
+
const gaps = {}
|
|
329
|
+
const recordGap = key => { gaps[key] = (gaps[key] ?? 0) + 1 }
|
|
158
330
|
for (const event of events) {
|
|
159
331
|
if (event.type === 'request/header') {
|
|
160
332
|
// The assembled model request header: system prompt + mounted tool
|
|
161
333
|
// schemas. What the model is told it can do and how — the "did my
|
|
162
334
|
// plugin's section inject?" projection.
|
|
335
|
+
if (typeof event.data?.header?.system !== 'string') recordGap('headerWithoutSystem')
|
|
336
|
+
if (Array.isArray(event.data?.header?.tools)
|
|
337
|
+
&& !event.data.header.tools.some(tool => typeof tool?.name === 'string')) {
|
|
338
|
+
recordGap('headerWithoutToolNames')
|
|
339
|
+
}
|
|
163
340
|
requestHeaders.push({
|
|
164
341
|
seq: event.seq,
|
|
165
342
|
reason: event.data?.reason,
|
|
@@ -169,6 +346,8 @@ export function buildTrace(logs) {
|
|
|
169
346
|
: [],
|
|
170
347
|
})
|
|
171
348
|
} else if (event.type === 'tool/call') {
|
|
349
|
+
if (typeof event.data?.name !== 'string') recordGap('toolCallWithoutName')
|
|
350
|
+
if (typeof event.data?.callId !== 'string') recordGap('toolCallWithoutCallId')
|
|
172
351
|
toolCalls.push({
|
|
173
352
|
seq: event.seq,
|
|
174
353
|
turn: event.data.turn,
|
|
@@ -179,6 +358,7 @@ export function buildTrace(logs) {
|
|
|
179
358
|
parsedArguments: parseArguments(event.data.arguments),
|
|
180
359
|
})
|
|
181
360
|
} else if (event.type === 'tool/result') {
|
|
361
|
+
if (typeof event.data?.message?.source?.callId !== 'string') recordGap('toolResultWithoutCallId')
|
|
182
362
|
toolResults.push({
|
|
183
363
|
seq: event.seq,
|
|
184
364
|
turn: event.data.turn,
|
|
@@ -222,7 +402,7 @@ export function buildTrace(logs) {
|
|
|
222
402
|
: assistantEntries.filter(entry => entry.seq < firstInjectionSeq)
|
|
223
403
|
const answerText = answerEntries.at(-1)?.text ?? ''
|
|
224
404
|
|
|
225
|
-
|
|
405
|
+
const result = {
|
|
226
406
|
sessions: logs,
|
|
227
407
|
sessionId: main?.header.id,
|
|
228
408
|
toolCalls,
|
|
@@ -233,36 +413,118 @@ export function buildTrace(logs) {
|
|
|
233
413
|
requestHeaders,
|
|
234
414
|
subagentChildren,
|
|
235
415
|
finalText: assistantTexts.at(-1) ?? '',
|
|
416
|
+
census: undefined,
|
|
236
417
|
}
|
|
418
|
+
result.census = buildCensus(events, result, childLogs, gaps)
|
|
419
|
+
return result
|
|
420
|
+
}
|
|
421
|
+
|
|
422
|
+
/**
|
|
423
|
+
* Session artifact basenames: format v0 keeps `session.jsonl`, every later
|
|
424
|
+
* generation carries a `vN` component (`session.v3.jsonl` — the host's
|
|
425
|
+
* `generationLogFilename`). Matching the v0 name alone finds no trace at all
|
|
426
|
+
* once the host bumps the format, which surfaces as "no session trace
|
|
427
|
+
* materialized" rather than as a parse error.
|
|
428
|
+
*/
|
|
429
|
+
const SESSION_LOG_FILENAME = /^session(?:\.v\d+)?\.jsonl$/u
|
|
430
|
+
|
|
431
|
+
/**
|
|
432
|
+
* Whether one file basename is a session JSONL artifact of any format generation.
|
|
433
|
+
* @param {string} name - the file basename to test.
|
|
434
|
+
* @returns {boolean} true for `session.jsonl` and `session.vN.jsonl`.
|
|
435
|
+
*/
|
|
436
|
+
export function isSessionLogFilename(name) {
|
|
437
|
+
return SESSION_LOG_FILENAME.test(name)
|
|
237
438
|
}
|
|
238
439
|
|
|
239
|
-
/** Recursively
|
|
240
|
-
function
|
|
241
|
-
|
|
440
|
+
/** Recursively list files under `dir`; an unreadable directory contributes nothing. */
|
|
441
|
+
function listFiles(dir, out = []) {
|
|
442
|
+
let entries
|
|
443
|
+
try {
|
|
444
|
+
entries = readdirSync(dir, { withFileTypes: true })
|
|
445
|
+
} catch {
|
|
446
|
+
return out
|
|
447
|
+
}
|
|
448
|
+
for (const entry of entries) {
|
|
242
449
|
const path = join(dir, entry.name)
|
|
243
|
-
if (entry.isDirectory())
|
|
244
|
-
else
|
|
450
|
+
if (entry.isDirectory()) listFiles(path, out)
|
|
451
|
+
else out.push(path)
|
|
245
452
|
}
|
|
246
453
|
return out
|
|
247
454
|
}
|
|
248
455
|
|
|
249
456
|
/**
|
|
250
|
-
*
|
|
457
|
+
* Every session artifact under `sessionsRoot` (any generation, see
|
|
458
|
+
* `isSessionLogFilename`) as absolute paths — the collection half of the
|
|
459
|
+
* seam, shared by the trace builder and by the raw-log capture the behavior
|
|
460
|
+
* runner does before cleanup.
|
|
251
461
|
* @param {string} sessionsRoot - the run's `session-persistence-jsonl` root.
|
|
252
|
-
* @returns {
|
|
462
|
+
* @returns {string[]} artifact paths, in directory order.
|
|
253
463
|
*/
|
|
254
|
-
export function
|
|
255
|
-
|
|
256
|
-
|
|
257
|
-
|
|
258
|
-
|
|
259
|
-
|
|
464
|
+
export function listSessionLogFiles(sessionsRoot) {
|
|
465
|
+
return listFiles(sessionsRoot).filter(path => isSessionLogFilename(basename(path)))
|
|
466
|
+
}
|
|
467
|
+
|
|
468
|
+
/** Most candidate names one gap diagnostic lists before it truncates. */
|
|
469
|
+
const GAP_NAME_LIMIT = 10
|
|
470
|
+
|
|
471
|
+
/**
|
|
472
|
+
* Why a collection produced no artifact, phrased for a failure message: the
|
|
473
|
+
* candidate names actually seen (a renamed artifact is the likeliest host
|
|
474
|
+
* drift) plus the generation suspicion. Never returns an empty string — an
|
|
475
|
+
* empty root is itself the fact to report.
|
|
476
|
+
*/
|
|
477
|
+
function traceGapMessage(sessionsRoot, files) {
|
|
478
|
+
const names = [...new Set(files.map(file => basename(file)))]
|
|
479
|
+
const lookalikes = names.filter(name => name.toLowerCase().startsWith('session'))
|
|
480
|
+
const pool = lookalikes.length > 0 ? lookalikes : names
|
|
481
|
+
const shown = pool.slice(0, GAP_NAME_LIMIT)
|
|
482
|
+
const rest = pool.length - shown.length
|
|
483
|
+
const scan = shown.length === 0
|
|
484
|
+
? 'the root holds no files (missing or empty)'
|
|
485
|
+
: `${lookalikes.length > 0 ? 'session-like file(s)' : 'file(s)'} under it: `
|
|
486
|
+
+ `${shown.join(', ')}${rest > 0 ? ` (+${rest} more)` : ''}`
|
|
487
|
+
return 'no session trace materialized: no session artifact'
|
|
488
|
+
+ ` (session.jsonl / session.vN.jsonl) under '${sessionsRoot}' — ${scan}`
|
|
489
|
+
+ '; the host artifact naming may have changed generation'
|
|
490
|
+
}
|
|
491
|
+
|
|
492
|
+
/**
|
|
493
|
+
* Collect one run's session trace and, when there is none, the seam
|
|
494
|
+
* diagnosis for it.
|
|
495
|
+
*
|
|
496
|
+
* The `gap` string exists so that "the host's artifact/session layout moved"
|
|
497
|
+
* surfaces as that sentence in the behavior runner's failure text and in the
|
|
498
|
+
* review adapter's accounting, instead of as a bare `undefined` the reader
|
|
499
|
+
* has to trace back through the parser (see docs/host-wiring.md).
|
|
500
|
+
*
|
|
501
|
+
* @param {string} sessionsRoot - the run's `session-persistence-jsonl` root.
|
|
502
|
+
* @returns {{ trace: EvalTrace | undefined, gap: string | undefined }} the
|
|
503
|
+
* trace, or `undefined` plus the reason no trace could be built.
|
|
504
|
+
*/
|
|
505
|
+
export function collectSessionTrace(sessionsRoot) {
|
|
506
|
+
const files = listFiles(sessionsRoot)
|
|
507
|
+
const artifacts = files.filter(file => isSessionLogFilename(basename(file)))
|
|
508
|
+
if (artifacts.length === 0) {
|
|
509
|
+
return { trace: undefined, gap: traceGapMessage(sessionsRoot, files) }
|
|
510
|
+
}
|
|
511
|
+
const logs = []
|
|
512
|
+
const broken = []
|
|
513
|
+
for (const artifact of artifacts) {
|
|
514
|
+
try {
|
|
515
|
+
logs.push(parseSessionLog(readFileSync(artifact, 'utf8')))
|
|
516
|
+
} catch (error) {
|
|
517
|
+
broken.push(`${basename(artifact)}: ${error instanceof Error ? error.message : String(error)}`)
|
|
518
|
+
}
|
|
519
|
+
}
|
|
520
|
+
if (broken.length > 0) {
|
|
521
|
+
return {
|
|
522
|
+
trace: undefined,
|
|
523
|
+
gap: `session artifact(s) failed to parse — ${broken.join('; ')}`
|
|
524
|
+
+ '; the host session format may have changed generation',
|
|
525
|
+
}
|
|
260
526
|
}
|
|
261
|
-
|
|
262
|
-
const logs = files
|
|
263
|
-
.map(file => readFileSync(file, 'utf8'))
|
|
264
|
-
.map(parseSessionLog)
|
|
265
|
-
return buildTrace(logs)
|
|
527
|
+
return { trace: buildTrace(logs), gap: undefined }
|
|
266
528
|
}
|
|
267
529
|
|
|
268
530
|
/**
|
|
@@ -290,4 +552,39 @@ export function loadTraceDir(sessionsRoot) {
|
|
|
290
552
|
* last assistant text ('' when it produced none — dispatched but not
|
|
291
553
|
* answered).
|
|
292
554
|
* @property {string} finalText - the last assembled assistant text ('' when none).
|
|
555
|
+
* @property {object | undefined} census - what the raw logs contained against
|
|
556
|
+
* what the projections kept (numbers only, never a verdict). Two data
|
|
557
|
+
* sources: `eventTypeCounts` counts the MAIN log's events by type (any type,
|
|
558
|
+
* plug-in ones included), and `subagent` censuses the child logs that enter
|
|
559
|
+
* `subagentChildren` (their `subagent/descriptor` event counts and how many
|
|
560
|
+
* carry the supported `version === 3`). `projectionLengths` are the five
|
|
561
|
+
* main-log projections' lengths after empty-text drops; `projectionSkipped`
|
|
562
|
+
* records where count minus length is positive, per projection, plus the two
|
|
563
|
+
* child-identity counters. Undefined only on a hand-built trace; a nested log
|
|
564
|
+
* under `sessions` carries none because the parsers never add one.
|
|
565
|
+
* @property {Record<string, number>} census.projectionFieldGaps
|
|
566
|
+
* - events the projection kept while a field it reads went missing, keyed by
|
|
567
|
+
* what was missing (`toolCallWithoutName`, `toolCallWithoutCallId`,
|
|
568
|
+
* `toolResultWithoutCallId`, `headerWithoutSystem`, `headerWithoutToolNames`).
|
|
569
|
+
* This is the 1:1-projection signal: for `tool/call`, `tool/result` and
|
|
570
|
+
* `request/header`, count − length is structurally zero, so a moved field
|
|
571
|
+
* shows up only here. An absent `request/header.tools` array is NOT counted
|
|
572
|
+
* (it projects to the same empty list as an empty one).
|
|
573
|
+
* @property {{ mainLogDescriptorEvents: number, supportedDescriptors: number, children: object[] }} census.subagent
|
|
574
|
+
* - the child-log data source the main-log counts cannot reach.
|
|
575
|
+
* `mainLogDescriptorEvents` counts `subagent/descriptor` events in the MAIN
|
|
576
|
+
* log itself (the current host writes them into the child log, so this is
|
|
577
|
+
* usually 0); `supportedDescriptors` sums the per-child counts below, i.e.
|
|
578
|
+
* it counts DESCRIPTOR EVENTS, not child sessions — one child may fold its
|
|
579
|
+
* identity from a single descriptor while logging several.
|
|
580
|
+
* @property {{ sessionId: string | undefined, parentSession: string | undefined, delegationDepth: number | undefined, descriptorEvents: number, supportedDescriptors: number, label: string | undefined, mode: string | undefined, provider: string | undefined }[]} census.subagent.children
|
|
581
|
+
* - the accepted child logs, each with the identity `projectChild` folded
|
|
582
|
+
* from them; `descriptorEvents === 0` means the log states no identity at
|
|
583
|
+
* all, and a non-zero count with `supportedDescriptors === 0` means every
|
|
584
|
+
* descriptor was outside the supported version.
|
|
585
|
+
* @property {Record<string, number>} census.projectionSkipped.children
|
|
586
|
+
* - `withoutIdentity` counts children that folded no identity field at all;
|
|
587
|
+
* `withoutLabel` counts children whose `label` is absent — the field the
|
|
588
|
+
* `subagent*Count` matchers match on, so those are the records whose
|
|
589
|
+
* zero-count assertions pass only because there was nothing to match.
|
|
293
590
|
*/
|