@catheadowl/dsh-eval 0.2.0 → 0.3.0

package/docs/review.md

@@ -1,5 +1,5 @@
 ---
-description: comprehension review 指南——defineReviewExperiment 实验定义、sterile profile 盲评运行、.runs 产物与 review-report 判读模板、六条评审规则
+description: comprehension review 指南——defineReviewExperiment 实验定义、空白环境（默认禁树外插件行）盲评运行、.runs 产物与 review-report 判读模板、六条评审规则
 ---
 
 # Comprehension review
@@ -34,7 +34,7 @@ export default defineReviewExperiment({
 })
 ```
 
-抽象层公开 `defineReviewExperiment`（稳定根入口）。实验执行与渲染原语——`materializeReviewExperiment`、`executeReviewExperiment`、标准 observation renderer（`renderObservationSections`；prompt 里的观测占位符常量是 `OBSERVATIONS_PLACEHOLDER`，必须恰好出现一次）——在 `@catheadowl/dsh-eval/experimental` 逃生面（见 [experimental.md](experimental.md)，无兼容承诺）。自定义 executor 只需实现 `(task, context) => result`；接入其他 agent runtime 不需要改试验定义。
+抽象层公开 `defineReviewExperiment`（稳定根入口）。实验执行与渲染原语——`materializeReviewExperiment`、`executeReviewExperiment`、标准 observation renderer（`renderObservationSections`；prompt 里的观测占位符常量是 `OBSERVATIONS_PLACEHOLDER`，必须恰好出现一次）——在 `@catheadowl/dsh-eval/experimental` 逃生面（见 [experimental.md](experimental.md)，无兼容承诺）。自定义 executor 只需实现 `(task, context) => result`；接入其他 agent runtime 不需要改试验定义。结果契约上，报告消费 `result.answer`（对任务的回答；未提供时回落 `result.stdout`），`result.stderr` 落盘为每轮 transcript——自定义 executor 提供这两者即可获得同等的产物与报告行为。
 
 ## CLI
 
@@ -43,19 +43,23 @@ export default defineReviewExperiment({
 dsh-review --dry-run <experiment file or directory>
 
 dsh-review \
-  --profile <sterile-profile> \
+  --profile <profile> \
   --repo <deepseek-harness checkout> \
-  [--runs 5] [--timeout 300000] \
+  [--runs 5] [--timeout 300000] [--keep-plugin-rows] \
   <experiment file or directory>
 ```
 
 `--profile`/`--repo` 可来自 `dsh-eval.config.mjs`（见 README），flags 覆盖 config。
 
-## sterile profile 与工具边界
+## 空白环境（默认）与工具边界
 
-真实运行使用**专用 sterile profile**（默认 `headless`，即宿主模板 `dsh-base` + `dsh-headless`，无树外插件）：适配器生成一份 `--patch` 覆盖层禁用所有宿主模型可见工具（`tool-fs`、`tool-fs-search`、shell、web、subagent 等），并把 cwd 指向空临时目录——reviewer 只能从物化的观测文本推理。运行后解析 session trace 的 `request/header` 事件做**工具边界校验**：发现任何非预期工具即视为 adapter failure（证据写入 `.runs/<id>/run-N.tool-boundary-evidence.json`）。
+真实运行的 reviewer 会话默认在**空白环境**启动：适配器先照常暂存所选 profile，再枚举它组合出的**全部树外插件行**（`package.json` 的 `dsh.profile.bundles` 中非 `@deepseek-ai/*` 的 bundle 各自 patch 文件里的行，加上 profile 自有 `cordis.patch.yml` 的行），在 `--patch` overlay 里逐行禁用——宿主 profile 装了什么 gates/插件都与 reviewer 无关，可复现性不再依赖「本机 profile 恰好干净」。白名单保留 reviewer 起不来就无测可言的接线行（`agent-default-model`、`session-title-llm`、`system-prompt`、`session-persistence-jsonl`）；宿主模板工具行（`tool-fs`、shell、web、subagent 等）由静态清单继续禁用，cwd 指向空临时目录——reviewer 只能从物化的观测文本推理。运行后解析 session trace 的 `request/header` 事件做**工具边界校验**：发现任何非预期工具即视为 adapter failure（证据写入 `.runs/<id>/run-N.tool-boundary-evidence.json`）。
 
-> **运维前提**：profile staging 在真实 home 已有同名 profile 时**原样复制**（含已安装插件与 patch 层）。若本机 `headless` profile 装过树外插件，暂存后的 profile **不是无菌的**——工具边界校验会当场 fail-loud（这是设计的正确行为）。保持无菌：删掉本机 `headless` profile 让 boot 重建出厂模板，或指定一个确认无插件的 profile。
+**校验没跑成也要记账**：收不到 session artifact（宿主 artifact 命名或会话格式换代是最常见的成因）时 `validateToolBoundary` 返回 `status: 'not-executed'` 而**不是**通过，适配器把该状态与 seam 诊断挂上执行结果；报告头写明 `tool boundary: NOT EXECUTED on run(s) N`，对应轮次的 `run-N.txt` 顶部与报告条目也各写一行原因，`run.json` 的 `toolBoundaries` 给机器读。这是有意的取向：**「没验证」必须看起来像没验证**——静默 fail-open 会让一份「工具边界从未检查过」的 review 产物读起来完全正常。
+
+**刻意复用宿主插件面**（例如要评审某插件自己的 gate 行为）：加 `--keep-plugin-rows`——跳过树外行枚举，仅保留静态工具禁用，宿主 gates 恢复运行。
+
+> 注意：无 `id` 的组合条目对 id 定位的禁用天然不可见（宿主 loader 语义），本包的树外 bundle 生态均为带 id 行形态；发现无 id 树外行时以工具边界校验 fail-loud 兜底。白名单是**按行名**无条件保留——若某树外 bundle 刻意以白名单名（如 `system-prompt`）insert 自己的行，该行不会被禁（威胁模型是本机自己的 profile，非对抗面）；此类泄漏同样由工具边界校验兜底。
 
 ## 产物
 
@@ -63,9 +67,10 @@ dsh-review \
 
 - `observations.md`：本次实时物化的可见证据；
 - `task.txt`：实际发给每位 reviewer 的完整任务；
-- `run-N.txt` / stderr / error：各次独立运行；
-- `run.json`：experiment、rubric、adapter、profile 与 run 数；
-- `review-report.md`：判读报告骨架——机器字段自动填（experiment/adapter/ profile/runs、observations 指纹、rubric 位置、每轮 reviewer 原文），三个 **人工判读栏目**留白待填：intentional design 命中项、新 red flag、下一步（改输出 / 改 rubric / 改 behavior case / 不处理）。刻意不做自动评分——review 层的价值在人工判断，报告只把判断物化成可归档、可对比的工程证据（dry-run 也会生成，runs 记 0）。
+- `run-N.txt`：该轮 reviewer 的**答案**——trace 推导（首条插件注入消息之前的最后一条 assistant 文本，抗回合尾部劫持），无 trace 时回落到 stdout 的最终消息；劫持发生时原始最终消息另存 `run-N.stdout.txt`；工具边界校验未执行时，本条文件顶部先写一行 `[tool-boundary: NOT EXECUTED — <原因>]`（答案本身退回 stdout 的最终消息）；
+- `run-N.stderr.txt` / error：完整会话转录（判读回溯的 transcript 指针，报告每轮引用）/ 失败原因；
+- `run.json`：experiment、rubric、adapter、profile、run 数与每轮 `toolBoundaries` 记账（`checked` / `not-executed` + 原因）；
+- `review-report.md`：判读报告骨架——机器字段自动填（experiment/adapter/ profile/runs、observations 指纹、rubric 位置、每轮 reviewer **答案**与 transcript 指针），三个 **人工判读栏目**留白待填：intentional design 命中项、新 red flag、下一步（改输出 / 改 rubric / 改 behavior case / 不处理）。刻意不做自动评分——review 层的价值在人工判断，报告只把判断物化成可归档、可对比的工程证据（dry-run 也会生成，runs 记 0）。
 
 ## 六条评审规则
 

package/docs/runner-api.md

@@ -17,10 +17,9 @@ const result = await runEvalCase(evalCase, { profile: 'headless', cliPath, mode:
 | 键 | 类型 | 语义 |
 |---|---|---|
 | `profile` | `string` | 必填。承载被测插件的 dsh profile（沙箱会暂存其 store，不污染真实 home）。 |
-| `cliPath` | `string` | 编译好的 dsh CLI 入口（`apps/cli/lib/bin.js`）绝对路径；优先于 `dshRepoDir`。程序化取值见下节。 |
+| `cliPath` | `string` | 必填。编译好的 dsh CLI 入口（`apps/cli/lib/bin.js`）绝对路径（`resolveDshCliChain` 结果）。程序化取值见下节。 |
 | `mode` | `'real' \| 'mock'` | 覆写 case 自带的 mode；mock 需 `script.steps`。 |
 | `artifactsDir` | `string` | 提供则把 stdout/stderr/trace/session 日志拷贝到该目录（自动创建）。 |
-| `dshRepoDir` | `string` | **已弃用**：宿主 checkout 目录（从中拼出 CLI 路径）。下个 minor 删除——迁移到 `cliPath`。 |
 
 ## `cliPath` 从哪来：跨档关系（读我）
 

package/package.json

@@ -1,9 +1,14 @@
 {
   "name": "@catheadowl/dsh-eval",
   "description": "Agent eval framework over dsh headless runs: case runner, session-trace assertions, and a scripted mock-LLM layer for plugin intent tests.",
-  "version": "0.2.0",
+  "version": "0.3.0",
   "license": "MIT",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/CatheadOwl/dsh-eval.git"
+  },
   "type": "module",
+  "packageManager": "pnpm@11.7.0",
   "main": "src/index.mjs",
   "exports": {
     ".": "./src/index.mjs",
@@ -28,6 +33,7 @@
     "docs/",
     "README.md",
     "README.zh.md",
+    "CHANGELOG.md",
     "LICENSE"
   ],
   "scripts": {
@@ -36,6 +42,6 @@
     "verify:experimental": "node scripts/verify-experimental-face.mjs",
     "verify:readme-i18n": "node scripts/verify-readme-i18n.mjs",
     "prepack": "node scripts/verify-manifest-face.mjs && node scripts/verify-publish-readiness.mjs && node scripts/verify-readme-i18n.mjs && node scripts/verify-experimental-face.mjs",
-    "test": "node --test --test-isolation=none tests/trace.test.mjs tests/assertions.test.mjs tests/overlay.test.mjs tests/staging.test.mjs tests/runner.test.mjs tests/cli.test.mjs tests/discovery.test.mjs tests/review.test.mjs tests/report.test.mjs tests/config.test.mjs tests/review-report.test.mjs && node scripts/verify-publish-readiness.mjs && node scripts/verify-manifest-face.mjs && node scripts/verify-readme-i18n.mjs && node scripts/verify-experimental-face.mjs"
+    "test": "node --test --test-isolation=none tests/trace.test.mjs tests/assertions.test.mjs tests/overlay.test.mjs tests/staging.test.mjs tests/runner.test.mjs tests/cli.test.mjs tests/cli-report.test.mjs tests/discovery.test.mjs tests/review.test.mjs tests/tool-validation.test.mjs tests/report.test.mjs tests/config.test.mjs tests/review-report.test.mjs && node scripts/verify-publish-readiness.mjs && node scripts/verify-manifest-face.mjs && node scripts/verify-readme-i18n.mjs && node scripts/verify-experimental-face.mjs"
   }
 }

package/src/adapters/dsh/review.mjs

@@ -1,23 +1,23 @@
 /** dsh-headless execution adapter for model-independent review experiments. */
 
-import { existsSync, mkdtempSync, writeFileSync } from 'node:fs'
+import { mkdtempSync, writeFileSync } from 'node:fs'
 import { tmpdir } from 'node:os'
 import { join, resolve } from 'node:path'
 import { executeReviewExperiment } from '../../experiment/review.mjs'
-import { CLI_RELATIVE_PATH } from '../../cli.mjs'
 import { overlayDisableRows } from '../../overlay.mjs'
 import {
-  resolveRealDshHome, stageSandboxHome, teardownSandbox, spawnHeadlessDsh,
+  resolveRealDshHome, stageSandboxHome, stagedPluginRows, teardownSandbox, spawnHeadlessDsh,
 } from '../../sandbox.mjs'
-import { loadTraceDir } from '../../trace.mjs'
+import { collectSessionTrace } from '../../trace.mjs'
 import { validateToolBoundary, renderToolBoundaryEvidence } from '../../tool-validation.mjs'
 
 /**
  * Model-facing tool rows every shipped dsh profile mounts from `dsh-base`.
- * The overlay disables all host tool rows as a supplementary guard; the
- * primary isolation comes from the sterile review profile (dsh-base +
- * dsh-headless only, no out-of-tree plugins).  Post-run tool boundary
- * validation (see `validateToolBoundary`) detects any residual drift.
+ * A static supplementary guard on top of the blank-environment enumeration
+ * (host template tool rows never appear in `stagedPluginRows` — they are the
+ * sterile baseline, so their ids are enumerated here instead).  Post-run
+ * tool boundary validation (see `validateToolBoundary`) detects any residual
+ * drift.
  */
 const REVIEW_DISABLED_TOOL_ROWS = [
   'tool-bash',
@@ -39,43 +39,65 @@ const REVIEW_DISABLED_TOOL_ROWS = [
   'tool-workflow',
 ]
 
-/** Serialize the tool-less overlay: disable every host model-facing tool row. */
-function buildReviewOverlayYaml() {
-  return overlayDisableRows(REVIEW_DISABLED_TOOL_ROWS)
-}
+/**
+ * Rows the reviewer session must KEEP even when an out-of-tree bundle or the
+ * profile patch touches them: model wiring and the session log — without
+ * these the reviewer cannot answer at all and the trace (tool boundary
+ * check, run artifacts) never materializes.  These ids are host-template
+ * rows; an out-of-tree patch row targeting one of them only overrides its
+ * config, so keeping it enabled stays safe.
+ */
+const REVIEW_REQUIRED_ROWS = new Set([
+  'agent-default-model',
+  'session-title-llm',
+  'system-prompt',
+  'session-persistence-jsonl',
+])
 
-/** Resolve and validate the compiled dsh CLI entry point. */
-export function resolveDshCli(dshRepoDir) {
-  const repoDir = resolve(dshRepoDir)
-  const cli = join(repoDir, ...CLI_RELATIVE_PATH.split(/[\\/]/))
-  if (!existsSync(cli)) {
-    throw new Error(`no compiled dsh CLI at '${cli}' (build deepseek-harness first)`)
-  }
-  return cli
+/**
+ * Serialize the blank-environment overlay: disable every plugin row the
+ * staged profile composes beyond the host templates (`stagedPluginRows` —
+ * out-of-tree bundle rows + profile patch rows), UNION the static host tool
+ * rows, MINUS the model/session wiring the reviewer needs.  With
+ * `keepPluginRows` (explicit opt-in to study the host plugin face itself)
+ * the enumeration is skipped and only the static tool lockdown remains.
+ */
+function buildReviewOverlayYaml(pluginRows, { keepPluginRows }) {
+  const disabled = new Set(keepPluginRows ? [] : pluginRows)
+  for (const row of REVIEW_DISABLED_TOOL_ROWS) disabled.add(row)
+  for (const row of REVIEW_REQUIRED_ROWS) disabled.delete(row)
+  return overlayDisableRows([...disabled])
 }
 
-/** The CLI entry for an executor: explicit cliPath (C6 chain result) wins;
- * otherwise fall back to the legacy repo form. Neither being set is a caller
- * bug the CLI bins already catch — this guard serves direct API consumers. */
+/** The CLI entry for an executor: the explicit cliPath (a `resolveDshCliChain`
+ * result). Its absence is a caller bug the CLI bins already catch — this
+ * guard serves direct API consumers. */
 function executorCli(options) {
-  if (options.cliPath !== undefined) return resolve(options.cliPath)
-  if (options.dshRepoDir !== undefined) return resolveDshCli(options.dshRepoDir)
-  throw new Error('review adapter needs a CLI location: pass cliPath (C6 chain result) or dshRepoDir')
+  if (options.cliPath === undefined) {
+    throw new Error('review adapter needs options.cliPath (a resolveDshCliChain result)')
+  }
+  return resolve(options.cliPath)
 }
 
 /**
  * Create an executor compatible with executeReviewExperiment.
  *
- * The executor boots a sterile review profile (default: the host's
- * `headless` template, bundles = dsh-base + dsh-headless, no out-of-tree
- * plugins) in an isolated DSH_HOME (staging/teardown mechanics shared with
- * the behavior runner via sandbox.mjs), disables every host tool row via
- * overlay, and validates the tool boundary after the run.  `options.profile`
- * must name a profile whose installed plugin set is empty or review-safe.
+ * The executor boots a blank review environment in an isolated DSH_HOME
+ * (staging/teardown mechanics shared with the behavior runner via
+ * sandbox.mjs): the staged profile's every out-of-tree plugin row and every
+ * host model-facing tool row is disabled via overlay (blank =
+ * dsh-base/dsh-headless templates + model wiring, nothing else — regardless
+ * of what the host profile carries), and the tool boundary is validated
+ * after the run.
  *
  * @param {object} options
- * @param {string} [options.profile='headless'] - the sterile review profile.
+ * @param {string} [options.profile='headless'] - the review profile (host
+ *   templates + whatever the host machine carries; plugin rows are disabled
+ *   by the overlay anyway).
  * @param {Set<string>} [options.allowedTools] - tool names permitted in the reviewer's session (default: empty).
+ * @param {boolean} [options.keepPluginRows=false] - opt back into the host
+ *   profile's plugin face (e.g. to review a plugin's own gate behavior);
+ *   only the static tool lockdown remains.
  */
 export function createDshHeadlessReviewExecutor(options) {
   const cli = executorCli(options)
@@ -85,6 +107,7 @@ export function createDshHeadlessReviewExecutor(options) {
   }
   const timeoutMs = options.timeoutMs ?? 300_000
   const allowedTools = options.allowedTools ?? new Set()
+  const keepPluginRows = options.keepPluginRows ?? false
 
   return async function executeWithDsh(task) {
     // A fresh process alone is not enough: dsh also stores settings, titles,
@@ -93,9 +116,18 @@ export function createDshHeadlessReviewExecutor(options) {
     // while retaining the selected profile's model config and plugin links.
     const runDir = mkdtempSync(join(tmpdir(), 'dsh-review-'))
     const dshHome = join(runDir, 'dsh-home')
-    const overlayPath = join(runDir, 'review-overlay.yml')
-    writeFileSync(overlayPath, buildReviewOverlayYaml(), 'utf8')
     stageSandboxHome(options.dshHome ?? resolveRealDshHome(), dshHome, profile)
+    // Blank environment by DEFAULT (review-blank-environment TODO): disable
+    // every row the staged profile composes beyond the host templates, so a
+    // host profile carrying out-of-tree plugins (gates included) can no
+    // longer steer or crash the reviewer. `keepPluginRows` opts back into
+    // the host plugin face deliberately. Enumeration happens on the STAGED
+    // copy, after staging — the staged home is what actually boots.
+    const overlayPath = join(runDir, 'review-overlay.yml')
+    writeFileSync(overlayPath, buildReviewOverlayYaml(
+      stagedPluginRows(dshHome, profile),
+      { keepPluginRows },
+    ), 'utf8')
 
     try {
       const { stdout, stderr, exitCode, timedOut } = await spawnHeadlessDsh({
@@ -120,28 +152,43 @@ export function createDshHeadlessReviewExecutor(options) {
 
       // Post-run tool boundary check: parse the session
       // trace, verify no unexpected tools were mounted in the reviewer's
-      // session, fail the run on violation.  An absent session log skips
-      // the check gracefully (accepted fail-open).
+      // session, fail the run on violation.  A missing session log does NOT
+      // skip the check silently: `validateToolBoundary` reports
+      // `status: 'not-executed'`, and the adapter carries that fact (plus the
+      // seam diagnosis) on the result so the report and run artifacts state
+      // that the boundary guarantee was not verified (EVAL-021).
       // Validation inspects the main session only (buildTrace selects
       // non-subagent logs); plugin tools leaking in a subagent session
       // would not be caught — irrelevant in review where the overlay
       // disables every subagent tool row.
-      const trace = loadTraceDir(join(dshHome, 'sessions'))
-      if (trace) {
-        const validation = validateToolBoundary(trace, { allowedTools })
-        result.toolValidation = validation
-        if (!validation.ok) {
-          // Attach evidence for the caller to persist (the adapter's
-          // runDir is ephemeral — removed by the finally block).  The
-          // bin writes this to `.runs/<id>/tool-boundary-evidence.json`.
-          result.toolBoundaryEvidence = renderToolBoundaryEvidence(validation, { runDir, profile })
-          const boundaryError = new Error(
-            `tool boundary violation: unexpected tools [${validation.unexpected.join(', ')}]`,
-          )
-          boundaryError.result = result
-          throw boundaryError
-        }
+      const { trace, gap } = collectSessionTrace(join(dshHome, 'sessions'))
+      const validation = validateToolBoundary(trace, { allowedTools })
+      result.toolValidation = validation
+      if (validation.status === 'not-executed') {
+        // `traceGap` mirrors the behavior runner's field name for the same
+        // seam diagnosis; the bin writes both into run-N.txt and the report.
+        result.traceGap = gap
+      }
+      if (validation.status === 'checked' && !validation.ok) {
+        // Attach evidence for the caller to persist (the adapter's
+        // runDir is ephemeral — removed by the finally block).  The
+        // bin writes this to `.runs/<id>/tool-boundary-evidence.json`.
+        result.toolBoundaryEvidence = renderToolBoundaryEvidence(validation, { runDir, profile })
+        const boundaryError = new Error(
+          `tool boundary violation: unexpected tools [${validation.unexpected.join(', ')}]`,
+        )
+        boundaryError.result = result
+        throw boundaryError
       }
+      // The ANSWER to the task, not the last message: stdout carries the
+      // headless CLI's final assistant message — whatever the reviewer
+      // said LAST. If any tail interaction intervened (a turn-close gate
+      // splice that slipped past the blank environment, an infra
+      // complaint), stdout holds that instead of the analysis. The
+      // trace's answerText (last assistant text before the first
+      // plugin-sourced injection) IS the analysis; stdout remains the
+      // fallback for trace-less runs (recorded by `traceGap` above).
+      result.answer = trace !== undefined && trace.answerText !== '' ? trace.answerText : stdout
 
       return result
     } finally {