@catheadowl/dsh-eval 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 CatheadOwl
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,125 @@
+---
+description: '@catheadowl/dsh-eval — dsh-native agent evaluation layer for plugin authors：behavior case 跑真实 headless dsh trace，review experiment 测 fresh model 能否理解插件输出'
+---
+
+# @catheadowl/dsh-eval
+
+**A dsh-native agent evaluation layer for plugin authors**: behavior cases run against real headless dsh traces, while review experiments test whether fresh models understand plugin outputs.
+
+它评测的是**装配后的 agent harness**（插件 + profile + patch + 工具注册表在真实 dsh headless 里接成的那张图），不是孤立函数；判定走 dsh 原生的 session trace 投影与 matcher（契约断言），不是 metric 分数。它不是通用 agent eval 平台（无 dashboard / dataset hosting / metric catalog，也不做 benchmark 排名），也不是 DeepEval / OpenAI Evals 的替代品——那些项目证明了这个问题空间成立，本包选择 dsh-native 的垂直解法。
+
+> 文档以中文为主；深度契约在 [docs/](docs/README.md)（matchers / 边界契约 / review / 报告结构 / 宿主接线 / 已知问题）。
+
+## 为什么需要它
+
+| 类型 | 问题 | 判定 | 执行 |
+|---|---|---|---|
+| 单元/shape test | 确定性字段和值是否正确 | 自动 | plugin 自己的 `node:test` |
+| behavior real | 自然语言意图是否选到正确工具 | trace matcher | dsh + 真实模型 |
+| behavior mock | 工具管线与写入 round-trip 是否稳定 | trace matcher + workspace inspect | dsh + 脚本化 mock LLM |
+| comprehension review | 一个 fresh model 能否从输出理解含义和下一步 | 人工对照 rubric，多次收敛 | 抽象 review experiment + 可替换 executor |
+
+dsh 插件的正确性来自「装配出的图是否真的把工具、steer、prompt、gate 接到一起」——这类问题插件自己的单测只能覆盖一部分；而「输出能否被理解」根本不是字符串回归。本包把这两层从手动试跑变成可复跑证据。
+
+```text
+plugin-owned experiment             shared framework
+fixtures + prompt + rubric + observe ──► experiment/review.mjs
+                                              │ task
+                                              ▼
+                                       adapters/dsh/review.mjs ──► dsh headless
+
+behavior *.eval.mjs ───────────────────► dsh behavior runner (trace + mock)
+```
+
+- `src/experiment/` 是模型与 runtime 无关的试验设计层：blind review、实时观测、多次 reviewer 字节一致证据。它不 import dsh。
+- `src/adapters/dsh/` 是落地层：把抽象任务交给隔离的 dsh headless。
+- 你的 `eval/` 只保留领域 fixture、projection/observe、prompt、rubric 与 case，不复制 runner。
+
+## Install
+
+```bash
+npm i -D @catheadowl/dsh-eval
+```
+
+**Requirements**（接线细节与失败自诊断见 [docs/host-wiring.md](docs/host-wiring.md)）：
+
+- 一个已构建的 deepseek-harness 检出（`apps/cli/lib/bin.js`）；
+- 被测插件已装进某个 dsh profile；
+- peer 依赖 `@deepseek-ai/dsh-llm` 需手工接线（npm 会自动装到不兼容的古董版，须替换为指向宿主检出的链接）。
+
+## Quickstart
+
+`<plugin>/eval/behavior/mock/smoke.eval.mjs`：
+
+```js
+import { firstTool, toolCalled, toolCallStep, textStep } from '@catheadowl/dsh-eval'
+
+export default {
+  id: 'my-first-case',
+  mode: 'mock',
+  task: '把 guide.md 重命名为 intro.md',
+  async prepare(workspace) { /* 播种 fixture 文件 */ },
+  script: { steps: [toolCallStep('md_rename', { oldPath: 'guide.md', newPath: 'intro.md' }), textStep('done')] },
+  expect: [toolCalled('md_rename')],
+}
+```
+
+```bash
+dsh-eval run --mode mock eval/behavior/mock
+dsh-review --dry-run eval/comprehension     # review 层的免模型预演
+```
+
+真实运行用 `dsh-eval run --profile <p> --repo <harness 检出> <case 路径>`；全部 flags（`--mode/--keep-artifacts/--fail-on-skip/--format/--report`）见 [docs/report.md](docs/report.md)。real case 无凭证时 auto-skip（dsh 自己解析凭证），mock 与 dry-run 不需要任何凭证。
+
+## 规范目录
+
+```text
+<plugin>/eval/
+  .gitignore                 # .runs/（无路径前缀）
+  README.md
+  behavior/                  # 可选
+    real/*.eval.mjs
+    mock/*.eval.mjs
+    _fixtures/
+  comprehension/             # 可选
+    <name>.review.mjs
+    fixtures.json
+    prompt.md
+    rubric.md
+```
+
+## 统一配置 dsh-eval.config.mjs
+
+消费者包根放一份，两个 CLI 从工作目录向上查找，flags 永远覆盖 config：
+
+```js
+export default {
+  profile: 'headless',              // dsh profile
+  repo: '../../deepseek-harness',   // 相对路径锚定 config 文件所在目录
+  mode: 'mock',                     // behavior CLI 的 --mode 默认（review 无此项）
+  failOnSkip: false,                // behavior CI 门禁默认
+  report: 'eval-report.json',       // --report 默认（锚定 config 目录）
+  disableRows: ['gates'],           // case 默认禁用的插件行；case 级声明覆盖
+                                     // （显式 [] = 全启用，gate 交互 case 用）
+}
+```
+
+未知 key 直接报错（拼写错误不静默退化）。`disableRows` 的语义与 turn-close 门禁边界契约见 [docs/disablerows.md](docs/disablerows.md)。
+
+## Docs
+
+| 文档 | 主题 |
+|---|---|
+| [host-wiring](docs/host-wiring.md) | peer 接线（含 npm 古董 peer 坑）、构建 CLI、profile、凭证、spawn 要求 |
+| [review](docs/review.md) | comprehension review：实验定义、sterile profile、产物、六条固化规则 |
+| [matchers](docs/matchers.md) | trace matcher 与 mock helper 全集（工具面 / 文本面 / 模型可见面） |
+| [disablerows](docs/disablerows.md) | `disableRows` 与 turn-close 门禁边界契约 |
+| [intent-cases](docs/intent-cases.md) | real 意图 case 规约：何时写、断言面、守卫、CI 语义 |
+| [report](docs/report.md) | 机器可读报告（`--format json` / `--report`）结构 |
+| [known-issues](docs/known-issues.md) | 已知问题与规避（如 staged home 的 REQUEST_EXTENSION） |
+
+## 运行保障
+
+runner 用 `try/finally` 保证临时目录与链接在任何路径（`prepare` 抛错、mock 校验失败、spawn 错误）都被清理，不污染真实 profile store。behavior 与 review CLI 共享目录扫描（跳过 `.runs` 与 `node_modules`）；behavior CLI 在加载期做 case shape 校验与跨文件重复 id 检测，尽早失败。
+
+License: MIT。框架自身的测试与发布自检由仓库 CI 承接，不随包发布。

package/bin/dsh-eval.mjs

@@ -0,0 +1,335 @@
+#!/usr/bin/env node
+/**
+ * dsh-eval — the plugin agent-eval case executor.
+ *
+ * Usage:
+ *   dsh-eval run --profile <name> --repo <deepseek-harness dir>
+ *                [--mode real|mock|all] [--keep-artifacts] [--fail-on-skip]
+ *                [--format text|json] [--report <file>]
+ *                <case paths...>
+ *
+ * A case path is a `*.eval.mjs` file or a directory scanned recursively for
+ * them. Each file default-exports one case object (or an array of them):
+ * `{ id, task, mode?: 'real'|'mock', expect: Matcher[], script?, persona?,
+ * prepare?, timeoutMs? }`. Real cases skip when DEEPSEEK_API_KEY is absent;
+ * the exit code is 1 when any run fails. Failures keep their artifacts under
+ * `<case file dir>/.runs/<case id>/`.
+ *
+ * Output formats (EVAL-007):
+ * - `--format text` (default): unchanged human output on stdout/stderr.
+ * - `--format json`: all progress and failure chatter moves to stderr;
+ *   stdout receives exactly one JSON report object (see src/report.mjs).
+ * - `--report <file>`: additionally write that report object to a file,
+ *   in either format — the aggregation/CI consumption path.
+ */
+
+import { existsSync, mkdirSync, writeFileSync } from 'node:fs'
+import { homedir } from 'node:os'
+import { dirname, isAbsolute, join, resolve } from 'node:path'
+import { pathToFileURL } from 'node:url'
+import { runEvalCase } from '../src/runner.mjs'
+import { discoverFiles, validateEvalCase, detectDuplicateIds } from '../src/discovery.mjs'
+import { createCaseRecord, buildRunReport, reportExitCode, mockDeterminismHint } from '../src/report.mjs'
+import { loadEvalConfig } from '../src/config.mjs'
+import { resolveDshCliChain } from '../src/cli.mjs'
+
+function usage(error) {
+  const text = [
+    'usage: dsh-eval run [--profile <name>] [--repo <deepseek-harness>] [--mode real|mock|all] [--keep-artifacts] [--fail-on-skip] [--format text|json] [--report <file>] <case paths...>',
+    '       --profile/--repo/--mode/--fail-on-skip/--report may come from a dsh-eval.config.mjs found upward from cwd; flags override it.',
+  ].join('\n')
+  if (error === undefined) {
+    process.stdout.write(`${text}\n`)
+    process.exit(0)
+  }
+  process.stderr.write(`${error}\n${text}\n`)
+  process.exit(2)
+}
+
+/** Parse argv: known flags, then case paths. Profile/repo/mode/failOnSkip
+ * may come from a `dsh-eval.config.mjs` instead of flags (flags win);
+ * required-ness is checked after config merging, not here. */
+function parseArgs(argv) {
+  const options = {
+    profile: undefined, repo: undefined, mode: undefined,
+    keepArtifacts: false, failOnSkip: undefined, format: 'text', report: undefined,
+  }
+  const paths = []
+  for (let i = 0; i < argv.length; i += 1) {
+    const arg = argv[i]
+    if (arg === 'run') continue
+    if (arg === '--profile') { options.profile = argv[++i]; continue }
+    if (arg === '--repo') { options.repo = argv[++i]; continue }
+    if (arg === '--mode') { options.mode = argv[++i]; continue }
+    if (arg === '--keep-artifacts') { options.keepArtifacts = true; continue }
+    if (arg === '--fail-on-skip') { options.failOnSkip = true; continue }
+    if (arg === '--format') { options.format = argv[++i]; continue }
+    if (arg === '--report') { options.report = argv[++i]; continue }
+    if (arg === '-h' || arg === '--help') usage()
+    paths.push(arg)
+  }
+  if (!['real', 'mock', 'all'].includes(options.mode ?? 'all')) usage(`error: --mode must be real, mock, or all (got '${options.mode}')`)
+  if (!['text', 'json'].includes(options.format)) usage(`error: --format must be text or json (got '${options.format}')`)
+  if (paths.length === 0) usage('error: at least one case file or directory is required')
+  return { options, paths }
+}
+
+/**
+ * Line output that respects the format: in `json` mode stdout is reserved
+ * for the single report object, so progress lines go to stderr instead.
+ */
+function say(line) {
+  if (jsonFormat) process.stderr.write(`${line}\n`)
+  else process.stdout.write(`${line}\n`)
+}
+
+/** Recursively collect `*.eval.mjs` files from one file or directory path. */
+function discoverCaseFiles(path) {
+  return discoverFiles(path, '.eval.mjs')
+}
+
+/** Import one case file, validate shape, and normalize to a case array. */
+async function loadCases(file) {
+  const module = await import(pathToFileURL(file).href)
+  const exported = module.default
+  const list = Array.isArray(exported) ? exported : [exported]
+  for (const evalCase of list) {
+    validateEvalCase(evalCase, file)
+  }
+  return list.map(evalCase => ({ ...evalCase, __file: file }))
+}
+
+/**
+ * Whether a model credential is available to a real run: the process
+ * environment, or the managed `$DSH_HOME/.credentials.yaml` document that
+ * `dsh-credentials-local` resolves per request. The key's VALUE is never
+ * read here — presence is the gate.
+ */
+function credentialAvailable() {
+  if (process.env.DEEPSEEK_API_KEY !== undefined) return true
+  const home = (process.env.DSH_HOME ?? '').trim() !== '' ? process.env.DSH_HOME : join(homedir(), '.dsh')
+  return existsSync(join(home, '.credentials.yaml'))
+}
+
+/** Why a case is skipped, or undefined when it should run. */
+function skipReason(evalCase, modeFilter) {
+  const mode = evalCase.mode ?? 'real'
+  if (modeFilter !== 'all' && mode !== modeFilter) return `--mode ${modeFilter}`
+  if (mode === 'real' && !credentialAvailable()) {
+    return 'no credential (DEEPSEEK_API_KEY unset and no $DSH_HOME/.credentials.yaml)'
+  }
+  return undefined
+}
+
+/**
+ * Persist one run's post-mortem artifacts under `.runs/<case id>/` next to
+ * the case file: the in-memory streams/trace plus the raw session logs
+ * captured before the run dir cleanup.
+ */
+function writeArtifacts(evalCase, result, mode) {
+  const artifactsDir = join(dirname(evalCase.__file), '.runs', evalCase.id)
+  try {
+    mkdirSync(artifactsDir, { recursive: true })
+    writeFileSync(join(artifactsDir, 'stdout.txt'), result.stdout)
+    writeFileSync(join(artifactsDir, 'stderr.txt'), result.stderr)
+    writeFileSync(join(artifactsDir, 'trace.json'), JSON.stringify({
+      caseId: evalCase.id, mode, task: evalCase.task,
+      exitCode: result.exitCode, timedOut: result.timedOut, trace: result.trace,
+    }, undefined, 2))
+    result.sessionLogs.forEach((text, index) => {
+      writeFileSync(join(artifactsDir, `session-${index}.jsonl`), text)
+    })
+  } catch { /* artifact persistence is best-effort */ }
+  return artifactsDir
+}
+
+const startedAt = new Date().toISOString()
+const { options, paths } = parseArgs(process.argv.slice(2))
+const jsonFormat = options.format === 'json'
+
+// Config merge (EVAL-008): a `dsh-eval.config.mjs` reachable from cwd
+// supplies defaults; explicit flags always win. Required-ness is only
+// decided after the merge, so config-only invocations work.
+const { config } = await loadEvalConfig(process.cwd())
+const profile = options.profile ?? config.profile
+const modeFilter = options.mode ?? config.mode ?? 'all'
+const failOnSkip = options.failOnSkip ?? config.failOnSkip ?? false
+if (profile === undefined) usage('error: --profile <name> is required (or set profile in dsh-eval.config.mjs)')
+// CLI resolution (C6, spec host-checkout-resolution): `--repo` flag >
+// resolution layer (node_modules/@deepseek-ai/dsh) > config repo key (legacy).
+// Committed files carry no real host-checkout path.
+const { cli: cliPath, repo: repoDir, source: cliSource } = resolveDshCliChain({
+  repoFlag: options.repo,
+  configRepo: config.repo,
+})
+const reportRepo = repoDir ?? cliPath
+
+const files = paths.flatMap(path => {
+  const absolute = resolve(path)
+  if (!existsSync(absolute)) usage(`error: no such case path: ${path}`)
+  return discoverCaseFiles(absolute)
+})
+if (files.length === 0) usage('error: no *.eval.mjs case files found')
+
+const records = []
+const seenIds = new Map()
+
+for (const file of files.sort()) {
+  let cases
+  try {
+    cases = await loadCases(file)
+  } catch (error) {
+    records.push(createCaseRecord({
+      id: file, file, status: 'fail',
+      failures: [`failed to load cases: ${error.message}`],
+    }))
+    process.stderr.write(`FAIL ${file}: failed to load cases: ${error.message}\n`)
+    continue
+  }
+  // Intra-file duplicate check
+  try {
+    detectDuplicateIds(cases)
+  } catch (error) {
+    records.push(createCaseRecord({
+      id: file, file, status: 'fail',
+      failures: [error.message],
+    }))
+    process.stderr.write(`FAIL ${file}: ${error.message}\n`)
+    continue
+  }
+  // Cross-file duplicate check (only add to seenIds after all pass)
+  let hasDuplicate = false
+  for (const c of cases) {
+    if (seenIds.has(c.id)) {
+      const message = `duplicate case id '${c.id}' (also in ${seenIds.get(c.id)})`
+      records.push(createCaseRecord({
+        id: c.id, file, mode: c.mode ?? 'real', status: 'fail',
+        failures: [message],
+      }))
+      process.stderr.write(`FAIL ${file}: ${message}\n`)
+      hasDuplicate = true
+      break
+    }
+  }
+  if (hasDuplicate) continue
+  for (const c of cases) seenIds.set(c.id, file)
+  for (const rawCase of cases) {
+    // Row-disable precedence (EVAL-014): a case's own `disableRows` —
+    // including an explicit `[]` ("disable nothing") — overrides the
+    // config-level default; only an undeclared field inherits it.
+    const evalCase = rawCase.disableRows === undefined && config.disableRows !== undefined
+      ? { ...rawCase, disableRows: config.disableRows }
+      : rawCase
+    const mode = evalCase.mode ?? 'real'
+    const skip = skipReason(evalCase, modeFilter)
+    if (skip !== undefined) {
+      records.push(createCaseRecord({
+        id: evalCase.id, file, mode, status: 'skip', skipReason: skip,
+      }))
+      say(`SKIP ${evalCase.id}: ${skip}`)
+      continue
+    }
+    say(`RUN  ${evalCase.id} (${mode})...`)
+    const runStartedAt = Date.now()
+    let result
+    try {
+      result = await runEvalCase(evalCase, { profile, cliPath, dshRepoDir: repoDir, mode })
+    } catch (error) {
+      records.push(createCaseRecord({
+        id: evalCase.id, file, mode, status: 'fail',
+        failures: [`runner error: ${error.message}`],
+        durationMs: Date.now() - runStartedAt,
+      }))
+      process.stderr.write(`FAIL ${evalCase.id}: runner error: ${error.message}\n`)
+      continue
+    }
+    const durationMs = Date.now() - runStartedAt
+
+    if (result.trace === undefined) {
+      const artifactsDir = writeArtifacts(evalCase, result, mode)
+      records.push(createCaseRecord({
+        id: evalCase.id, file, mode, status: 'fail',
+        failures: [`no session trace materialized (exit ${result.exitCode}${result.timedOut ? ', timed out' : ''})`],
+        exitCode: result.exitCode, timedOut: result.timedOut,
+        durationMs, artifactsDir,
+      }))
+      process.stderr.write(
+        `FAIL ${evalCase.id}: no session trace materialized (exit ${result.exitCode}${result.timedOut ? ', timed out' : ''})\n`
+        + `     artifacts: ${artifactsDir}\n--- stderr ---\n${result.stderr}\n`,
+      )
+      continue
+    }
+
+    const failures = []
+    if (result.exitCode !== 0) {
+      // Headless SSOT: exit 0 iff the turn completed. A run that errored out
+      // must not pass on coincidentally satisfied matchers.
+      failures.push(`dsh CLI exited with code ${result.exitCode} (the turn did not complete)`)
+    }
+    for (const matcher of evalCase.expect) {
+      const outcome = matcher.check(result.trace)
+      if (!outcome.ok) failures.push(`${matcher.describe}: ${outcome.message}`)
+    }
+    if (result.timedOut) failures.push('run timed out')
+    if (result.inspectError !== undefined) failures.push(`workspace inspect failed: ${result.inspectError}`)
+
+    if (failures.length === 0) {
+      const artifactsDir = options.keepArtifacts ? writeArtifacts(evalCase, result, mode) : undefined
+      records.push(createCaseRecord({
+        id: evalCase.id, file, mode, status: 'pass',
+        exitCode: result.exitCode, timedOut: result.timedOut,
+        durationMs, ...(artifactsDir !== undefined ? { artifactsDir } : {}),
+      }))
+      say(`PASS ${evalCase.id}`)
+    } else {
+      const artifactsDir = writeArtifacts(evalCase, result, mode)
+      // Self-explaining failure for broken mock determinism (EVAL-014
+      // alternative): when non-host plugin injections are visible in the
+      // trace, the failure names them and the two framework-native exits —
+      // consumers stop rediscovering the mechanism from raw traces.
+      let hint
+      if (mode === 'mock') hint = mockDeterminismHint({ trace: result.trace, failures })
+      records.push(createCaseRecord({
+        id: evalCase.id, file, mode, status: 'fail', failures: hint ? [...failures, hint] : failures,
+        exitCode: result.exitCode, timedOut: result.timedOut,
+        durationMs, artifactsDir,
+      }))
+      process.stderr.write(`FAIL ${evalCase.id} (exit ${result.exitCode}):\n${failures.map(f => `  - ${f}`).join('\n')}\n`)
+      if (hint !== undefined) process.stderr.write(`  ! ${hint}\n`)
+      process.stderr.write(`     artifacts: ${artifactsDir}\n`)
+    }
+  }
+}
+
+const finishedAt = new Date().toISOString()
+const report = buildRunReport({
+  profile,
+  repo: reportRepo,
+  cliSource,
+  modeFilter,
+  failOnSkip,
+  startedAt,
+  finishedAt,
+  records,
+})
+
+const reportTarget = options.report ?? config.report
+if (reportTarget !== undefined) {
+  const reportPath = isAbsolute(reportTarget) ? reportTarget : resolve(process.cwd(), reportTarget)
+  try {
+    mkdirSync(dirname(reportPath), { recursive: true })
+    writeFileSync(reportPath, JSON.stringify(report, undefined, 2))
+    process.stderr.write(`report: ${reportPath}\n`)
+  } catch (error) {
+    process.stderr.write(`error: failed to write report '${reportPath}': ${error.message}\n`)
+    process.exit(2)
+  }
+}
+
+if (options.format === 'json') {
+  process.stdout.write(`${JSON.stringify(report, undefined, 2)}\n`)
+} else {
+  const { summary } = report
+  process.stdout.write(`\n${summary.selected} selected, ${summary.passed} passed, ${summary.failed} failed, ${summary.skipped} skipped\n`)
+}
+process.exit(reportExitCode(records, failOnSkip))

package/bin/dsh-review.mjs

@@ -0,0 +1,154 @@
+#!/usr/bin/env node
+/**
+ * Execute model-independent `*.review.mjs` experiments through dsh headless.
+ *
+ * Dry-run materializes live observations without touching dsh. Real runs write
+ * the shared task plus each independent review answer beside the experiment:
+ * `.runs/<experiment id>/`.
+ */
+
+import { existsSync, mkdirSync, writeFileSync } from 'node:fs'
+import { dirname, join, resolve } from 'node:path'
+import { pathToFileURL } from 'node:url'
+import { materializeReviewExperiment } from '../src/experiment/review.mjs'
+import { runDshReviewExperiment } from '../src/adapters/dsh/review.mjs'
+import { discoverFiles } from '../src/discovery.mjs'
+import { loadEvalConfig } from '../src/config.mjs'
+import { resolveDshCliChain } from '../src/cli.mjs'
+import { renderReviewReport } from '../src/review-report.mjs'
+
+function usage(error) {
+  const message = [
+    'usage: dsh-review [--dry-run] [--runs N] [--profile NAME (default: headless) --repo DIR] [--timeout MS] <*.review.mjs or directories...>',
+    '       --profile/--repo may come from a dsh-eval.config.mjs found upward from cwd; flags override it.',
+  ].join('\n')
+  if (error) process.stderr.write(`${error}\n${message}\n`)
+  else process.stdout.write(`${message}\n`)
+  process.exit(error ? 2 : 0)
+}
+
+function parseArgs(argv) {
+  const options = { dryRun: false, runs: undefined, timeoutMs: undefined, profile: undefined, repo: undefined }
+  const paths = []
+  for (let index = 0; index < argv.length; index += 1) {
+    const arg = argv[index]
+    if (arg === '--dry-run') { options.dryRun = true; continue }
+    if (arg === '--runs') { options.runs = Number(argv[++index]); continue }
+    if (arg === '--profile') { options.profile = argv[++index]; continue }
+    if (arg === '--repo') { options.repo = argv[++index]; continue }
+    if (arg === '--timeout') { options.timeoutMs = Number(argv[++index]); continue }
+    if (arg === '-h' || arg === '--help') usage()
+    paths.push(arg)
+  }
+  if (paths.length === 0) usage('error: at least one review experiment path is required')
+  if (options.runs !== undefined && (!Number.isInteger(options.runs) || options.runs < 1)) usage('error: --runs must be a positive integer')
+  if (options.timeoutMs !== undefined && (!Number.isInteger(options.timeoutMs) || options.timeoutMs < 1)) usage('error: --timeout must be a positive integer')
+  return { options, paths }
+}
+
+function discover(path) {
+  return discoverFiles(path, '.review.mjs')
+}
+
+async function loadExperiment(file) {
+  const module = await import(pathToFileURL(file).href)
+  const experiment = module.default
+  if (experiment?.kind !== 'review') {
+    throw new Error(`${file}: default export must come from defineReviewExperiment(...)`)
+  }
+  return { ...experiment, __file: file }
+}
+
+function artifactDir(experiment) {
+  return join(dirname(experiment.__file), '.runs', experiment.id)
+}
+
+function writeMaterialized(experiment, materialized, extra = {}, reviewResult = undefined) {
+  const output = artifactDir(experiment)
+  mkdirSync(output, { recursive: true })
+  writeFileSync(join(output, 'task.txt'), materialized.task, 'utf8')
+  writeFileSync(join(output, 'observations.md'), materialized.observations, 'utf8')
+  writeFileSync(join(output, 'run.json'), JSON.stringify({
+    experimentId: experiment.id,
+    summary: experiment.summary,
+    rubric: String(experiment.rubric),
+    ...extra,
+  }, null, 2), 'utf8')
+  writeFileSync(join(output, 'review-report.md'), renderReviewReport({
+    experiment,
+    result: reviewResult,
+    observations: materialized.observations,
+    adapter: extra.adapter,
+    profile: extra.profile,
+  }), 'utf8')
+  return output
+}
+
+const { options, paths } = parseArgs(process.argv.slice(2))
+
+// Config merge (EVAL-008): flags win over a `dsh-eval.config.mjs` found
+// upward from cwd; profile falls back to the sterile default `headless`.
+const { config } = await loadEvalConfig(process.cwd())
+const profile = options.profile ?? config.profile ?? 'headless'
+// CLI resolution (C6): `--repo` flag > resolution layer (node_modules) >
+// config repo key (legacy). Dry-run never boots the CLI, so resolve lazily.
+let cli = { cliPath: undefined, repoDir: undefined }
+if (!options.dryRun) {
+  try {
+    const resolved = resolveDshCliChain({ repoFlag: options.repo, configRepo: config.repo })
+    cli = { cliPath: resolved.cli, repoDir: resolved.repo }
+  } catch (error) {
+    usage(`error: ${error.message}`)
+  }
+}
+
+for (const path of paths) {
+  if (!existsSync(resolve(path))) usage(`error: no such experiment path: ${path}`)
+}
+const files = paths.flatMap(path => discover(path)).sort()
+if (files.length === 0) usage('error: no *.review.mjs experiment files found')
+
+let failures = 0
+for (const file of files) {
+  let experiment
+  try {
+    experiment = await loadExperiment(file)
+    if (options.dryRun) {
+      const materialized = await materializeReviewExperiment(experiment)
+      const output = writeMaterialized(experiment, materialized, { adapter: null, dryRun: true })
+      process.stdout.write(`DRY  ${experiment.id}: ${output}\n`)
+      continue
+    }
+
+    process.stdout.write(`RUN  ${experiment.id} (${options.runs ?? experiment.defaultRuns} reviews)...\n`)
+    const result = await runDshReviewExperiment(experiment, {
+      profile,
+      cliPath: cli.cliPath,
+      dshRepoDir: cli.repoDir,
+      runs: options.runs,
+      timeoutMs: options.timeoutMs,
+    })
+    const output = writeMaterialized(experiment, result, {
+      adapter: 'dsh-headless',
+      profile,
+      runs: result.runs,
+    }, result)
+    for (const attempt of result.attempts) {
+      const payload = attempt.result ?? {}
+      if (payload.stdout !== undefined) writeFileSync(join(output, `run-${attempt.index}.txt`), payload.stdout, 'utf8')
+      if (payload.stderr) writeFileSync(join(output, `run-${attempt.index}.stderr.txt`), payload.stderr, 'utf8')
+      if (payload.toolBoundaryEvidence) writeFileSync(join(output, `run-${attempt.index}.tool-boundary-evidence.json`), payload.toolBoundaryEvidence, 'utf8')
+      if (!attempt.ok) {
+        failures += 1
+        writeFileSync(join(output, `run-${attempt.index}.error.txt`), attempt.error, 'utf8')
+        process.stderr.write(`FAIL ${experiment.id} run ${attempt.index}: ${attempt.error}\n`)
+      }
+    }
+    if (result.attempts.every(attempt => attempt.ok)) process.stdout.write(`DONE ${experiment.id}: ${output}\n`)
+  } catch (error) {
+    failures += 1
+    process.stderr.write(`FAIL ${file}: ${error.message}\n`)
+  }
+}
+
+process.exit(failures === 0 ? 0 : 1)

package/docs/README.md

@@ -0,0 +1,15 @@
+---
+description: dsh-eval 文档索引——安装与宿主接线、review、matcher 全集、disableRows 契约、intent case 规约、报告结构与已知问题七篇的路由表
+---
+
+# dsh-eval · docs index
+
+| 文档 | 主题 |
+|---|---|
+| [host-wiring.md](host-wiring.md) | 安装与宿主接线：peer 依赖（dsh-llm）、构建 CLI、profile、凭证、spawn 要求 |
+| [review.md](review.md) | comprehension review：实验定义、sterile profile、产物清单、六条固化规则 |
+| [matchers.md](matchers.md) | trace matcher 与 mock helper 全集 |
+| [disablerows.md](disablerows.md) | `disableRows` 与 turn-close 门禁边界契约 |
+| [intent-cases.md](intent-cases.md) | real 意图 case 规约：何时写、断言面、守卫、CI 语义 |
+| [report.md](report.md) | 机器可读报告（`--format json` / `--report`）结构 |
+| [known-issues.md](known-issues.md) | 已知问题与规避 |

package/docs/disablerows.md

@@ -0,0 +1,25 @@
+---
+description: disableRows 边界契约——按 loader 行 id 禁用插件行的通用机制、turn-close 门禁 splice 与 finalText 失效的交互根因、case/config 取值优先级
+---
+
+# `disableRows` 与 turn-close 门禁边界契约
+
+`disableRows: string[]` 是通用机制：按 loader 行 id 在本次 run 的 overlay 里禁用任意插件行（`- id: <row> / disabled: true`，与 `session-title-llm` 同一跨层禁用机制）。框架对行 id 无任何内置知识，任何插件都可以成为禁用对象。取值优先级：**case 声明 > config 默认 > 不禁用**——case 级 `disableRows: []` 是合法的显式「全启用」，专门用来在默认禁 gate 行的包里恢复 gate 交互 case 的装载。
+
+## 为什么需要它
+
+首要使用场景是 turn-close blocking gate：gate 会在 turn 收尾自动运行并向 inbox splice 反馈。当 case 的**终态本身**就是 gate 判违规的状态（skip 语义的断链现场、非 git 工作区的 doc-link 报错现场等），splice 会驱动模型产生脚本之外的额外 step，`finalText*` 断言随之失效。
+
+eval 的临时工作区通常**不是 git 仓库**——doc-link 类 gate 在其中只会以 git 报错成 blocking 并 splice 反馈耗尽脚本，因此测插件工具面的包普遍在 `dsh-eval.config.mjs` 里默认 `disableRows: ['gates']`。
+
+## 契约
+
+- 默认**不声明** = 所选 profile 装载的插件照常运行（gate 交互 case——如断言 gate steer 的 `userMessageTextIncludes`——依赖此默认）。
+- 声明 `disableRows: ['gates']`（case 级或 config 级）= 本次 run 禁用 gates 插件行（行 id 权威：`@catheadowl/dsh-extras` 包的 `cordis.patch.yml` `- id: gates`——兄弟 dsh 插件包），终态违规不再触发 splice，`finalText` 保持「脚本终步文本」的确定性语义。禁用其他插件行同理，行 id 以该插件包的 patch 声明为准。
+- gate 交互 case 在默认禁用的包里声明 `disableRows: []` 显式恢复装载。
+- 不依赖插件开关的断言出口：`assistantTextIncludes`（断言脚本台词出现过，不要求是最终文本）。终态干净时仍应优先 `finalText*`。
+- per-gate 白名单（如只关某个 gate）暂不支持：per-gate disable 需要 gate 框架侧先提供 config 面。
+
+## 失败自解释
+
+mock case 的 `finalText*` 失败若伴随 trace 里可见的**非宿主**插件注入 user 消息（gate 反馈、steer 等任何形态），失败输出会点名注入插件并提示上述出口（`disableRows`，或把交互纳入脚本预期 / `assistantTextIncludes`）——确定性被打破时框架当场解释机制，无需读 raw trace 排查。