@catheadowl/dsh-eval 0.1.0 → 0.2.1

package/CHANGELOG.md

@@ -0,0 +1,57 @@
+---
+description: Release history of @catheadowl/dsh-eval — one entry per published version, following Keep a Changelog conventions
+---
+
+# Changelog
+
+All notable changes to `@catheadowl/dsh-eval` are documented here. Versions
+follow [Semantic Versioning](https://semver.org/); entries follow
+[Keep a Changelog](https://keepachangelog.com/) conventions.
+
+## [0.2.1] — 2026-09-09
+
+### Added
+
+- Subagent dispatch observability: `subagentChildren` projection plus
+  `subagentDispatched` / `subagentCompleted` matchers.
+- Multi-turn followups: cross-turn asynchronous driving for review
+  experiments, with dispatch/completion count assertions.
+
+### Changed
+
+- `dsh-review` boots a **blank environment by default**: staged out-of-tree
+  plugin rows are disabled via overlay, so host-profile gates/plugins can no
+  longer steer or crash a reviewer. Pass `--keep-plugin-rows` (or executor
+  option `keepPluginRows`) to opt back in deliberately.
+
+### Fixed
+
+- Review artifacts capture the reviewer's **answer**, not the last message:
+  conclusions use the last assistant text before any plugin-sourced injection
+  (trace-derived); the raw final message is kept as `run-N.stdout.txt` when it
+  diverges, and each report run cites its transcript.
+
+## [0.2.0] — 2026-09-06
+
+### Changed
+
+- **BREAKING (0.x minor)**: public API split into two tiers. The root entry
+  is now the SDK tier only (assertion DSL, step builders, `runEvalCase`,
+  `defineReviewExperiment`); low-level sandbox/overlay/trace primitives and
+  review execution moved to the `./experimental` escape-hatch subpath.
+- `exports` map added — deep path imports are now mechanically blocked.
+
+## [0.1.0] — 2026-09-05
+
+### Added
+
+- Initial public release: case runner over dsh headless runs,
+  session-trace assertions, and a scripted mock-LLM layer for plugin intent
+  tests.
+- CLI binaries `dsh-eval` (run cases) and `dsh-review` (LLM output review);
+  `@deepseek-ai/dsh-llm` is a peerDependency provided by the host
+  ecosystem.
+
+[0.2.1]: https://github.com/CatheadOwl/dsh-eval/releases/tag/v0.2.1
+[0.2.0]: https://github.com/CatheadOwl/dsh-eval/releases/tag/v0.2.0
+[0.1.0]: https://github.com/CatheadOwl/dsh-eval/releases/tag/v0.1.0

package/README.i18n.yaml

@@ -0,0 +1,6 @@
+# Bilingual README pairing record (adapted from deepseek-harness docs/i18n): the sha256 of each
+# side as of the last confirmed-consistent state. Both languages carry equal authority; after
+# editing either side, bring the other along and re-record with:
+#   node scripts/verify-readme-i18n.mjs --write
+README.md: 94e4e6c1172544c94c65329abf8dc6db81c0ba8030c354e1c1e8e64f8bc8276b
+README.zh.md: a1f45726e695fe46931a2988c7293d53c61b3c275dc92e039e707b315f72fc60

package/README.md

@@ -1,39 +1,37 @@
----
-description: '@catheadowl/dsh-eval — dsh-native agent evaluation layer for plugin authors：behavior case 跑真实 headless dsh trace，review experiment 测 fresh model 能否理解插件输出'
----
-
 # @catheadowl/dsh-eval
 
+English | [中文](README.zh.md)
+
 **A dsh-native agent evaluation layer for plugin authors**: behavior cases run against real headless dsh traces, while review experiments test whether fresh models understand plugin outputs.
 
-它评测的是**装配后的 agent harness**（插件 + profile + patch + 工具注册表在真实 dsh headless 里接成的那张图），不是孤立函数；判定走 dsh 原生的 session trace 投影与 matcher（契约断言），不是 metric 分数。它不是通用 agent eval 平台（无 dashboard / dataset hosting / metric catalog，也不做 benchmark 排名），也不是 DeepEval / OpenAI Evals 的替代品——那些项目证明了这个问题空间成立，本包选择 dsh-native 的垂直解法。
+It evaluates the **assembled agent harness** (the graph a plugin + profile + patch + tool registry form inside a real dsh headless run), not isolated functions; verdicts come from dsh-native session-trace projections and matchers (contract assertions), not metric scores. It is not a general agent-eval platform (no dashboard / dataset hosting / metric catalog, no benchmark ranking) and not a DeepEval / OpenAI Evals replacement — those projects proved the problem space; this package picks the dsh-native vertical solution.
 
-> 文档以中文为主；深度契约在 [docs/](docs/README.md)（matchers / 边界契约 / review / 报告结构 / 宿主接线 / 已知问题）。
+> Documentation is Chinese-first; deep contracts live in [docs/](docs/README.md) (matchers / boundary contracts / review / report structure / host wiring / known issues).
 
-## 为什么需要它
+## Why it exists
 
-| 类型 | 问题 | 判定 | 执行 |
+| Layer | Question | Verdict | Execution |
 |---|---|---|---|
-| 单元/shape test | 确定性字段和值是否正确 | 自动 | plugin 自己的 `node:test` |
-| behavior real | 自然语言意图是否选到正确工具 | trace matcher | dsh + 真实模型 |
-| behavior mock | 工具管线与写入 round-trip 是否稳定 | trace matcher + workspace inspect | dsh + 脚本化 mock LLM |
-| comprehension review | 一个 fresh model 能否从输出理解含义和下一步 | 人工对照 rubric，多次收敛 | 抽象 review experiment + 可替换 executor |
+| unit / shape test | are deterministic fields and values correct | automatic | the plugin's own `node:test` |
+| behavior real | does natural-language intent pick the right tool | trace matcher | dsh + real model |
+| behavior mock | is the tool pipeline and write round-trip stable | trace matcher + workspace inspect | dsh + scripted mock LLM |
+| comprehension review | can a fresh model understand the output and the next step | manual rubric, converged over runs | abstract review experiment + replaceable executor |
 
-dsh 插件的正确性来自「装配出的图是否真的把工具、steer、prompt、gate 接到一起」——这类问题插件自己的单测只能覆盖一部分；而「输出能否被理解」根本不是字符串回归。本包把这两层从手动试跑变成可复跑证据。
+A dsh plugin is correct when the assembled graph really wires tools, steers, prompts, and gates together — plugin unit tests cover only part of that, and "is the output understandable" is not a string regression at all. This package turns both layers into replayable evidence instead of manual trial runs.
 
 ```text
 plugin-owned experiment             shared framework
 fixtures + prompt + rubric + observe ──► experiment/review.mjs
-                                              │ task
-                                              ▼
-                                       adapters/dsh/review.mjs ──► dsh headless
+                                               │ task
+                                               ▼
+                                        adapters/dsh/review.mjs ──► dsh headless
 
 behavior *.eval.mjs ───────────────────► dsh behavior runner (trace + mock)
 ```
 
-- `src/experiment/` 是模型与 runtime 无关的试验设计层：blind review、实时观测、多次 reviewer 字节一致证据。它不 import dsh。
-- `src/adapters/dsh/` 是落地层：把抽象任务交给隔离的 dsh headless。
-- 你的 `eval/` 只保留领域 fixture、projection/observe、prompt、rubric 与 case，不复制 runner。
+- `src/experiment/` is the model- and runtime-agnostic experiment layer: blind review, live observation, byte-identical evidence across reviewers. It does not import dsh.
+- `src/adapters/dsh/` is the landing layer: hands the abstract task to an isolated dsh headless run.
+- Your `eval/` keeps only domain fixtures, projections/observe, prompts, rubrics, and cases — no runner duplication.
 
 ## Install
 
@@ -41,15 +39,15 @@ behavior *.eval.mjs ───────────────────►
 npm i -D @catheadowl/dsh-eval
 ```
 
-**Requirements**（接线细节与失败自诊断见 [docs/host-wiring.md](docs/host-wiring.md)）：
+**Requirements** (wiring details and failure self-diagnostics in [docs/host-wiring.md](docs/host-wiring.md)):
 
-- 一个已构建的 deepseek-harness 检出（`apps/cli/lib/bin.js`）；
-- 被测插件已装进某个 dsh profile；
-- peer 依赖 `@deepseek-ai/dsh-llm` 需手工接线（npm 会自动装到不兼容的古董版，须替换为指向宿主检出的链接）。
+- a built deepseek-harness checkout (`apps/cli/lib/bin.js`);
+- the plugin under test installed into a dsh profile;
+- the peer dependency `@deepseek-ai/dsh-llm` must be wired manually (npm auto-installs an incompatible antique version; replace it with a link pointing at the host checkout).
 
 ## Quickstart
 
-`<plugin>/eval/behavior/mock/smoke.eval.mjs`：
+`<plugin>/eval/behavior/mock/smoke.eval.mjs`:
 
 ```js
 import { firstTool, toolCalled, toolCallStep, textStep } from '@catheadowl/dsh-eval'
@@ -57,8 +55,8 @@ import { firstTool, toolCalled, toolCallStep, textStep } from '@catheadowl/dsh-e
 export default {
   id: 'my-first-case',
   mode: 'mock',
-  task: '把 guide.md 重命名为 intro.md',
-  async prepare(workspace) { /* 播种 fixture 文件 */ },
+  task: 'rename guide.md to intro.md',
+  async prepare(workspace) { /* seed fixture files */ },
   script: { steps: [toolCallStep('md_rename', { oldPath: 'guide.md', newPath: 'intro.md' }), textStep('done')] },
   expect: [toolCalled('md_rename')],
 }
@@ -66,60 +64,66 @@ export default {
 
 ```bash
 dsh-eval run --mode mock eval/behavior/mock
-dsh-review --dry-run eval/comprehension     # review 层的免模型预演
+dsh-review --dry-run eval/comprehension     # model-free dry run of the review layer
 ```
 
-真实运行用 `dsh-eval run --profile <p> --repo <harness 检出> <case 路径>`；全部 flags（`--mode/--keep-artifacts/--fail-on-skip/--format/--report`）见 [docs/report.md](docs/report.md)。real case 无凭证时 auto-skip（dsh 自己解析凭证），mock 与 dry-run 不需要任何凭证。
+The command needs to know which dsh profile to use: pass `--profile <name>` explicitly, or drop a `dsh-eval.config.mjs` at the package root (see "Unified config" below).
+
+Real runs use `dsh-eval run --profile <p> --repo <harness checkout> <case path>`; all flags (`--mode/--keep-artifacts/--fail-on-skip/--format/--report`) are documented in [docs/report.md](docs/report.md). Real cases auto-skip without credentials (dsh resolves credentials itself); mock and dry runs need no credentials.
 
-## 规范目录
+## Canonical layout
 
 ```text
 <plugin>/eval/
-  .gitignore                 # .runs/（无路径前缀）
+  .gitignore                 # .runs/ (no path prefix)
   README.md
-  behavior/                  # 可选
+  behavior/                  # optional
     real/*.eval.mjs
     mock/*.eval.mjs
     _fixtures/
-  comprehension/             # 可选
+  comprehension/             # optional
     <name>.review.mjs
     fixtures.json
     prompt.md
     rubric.md
 ```
 
-## 统一配置 dsh-eval.config.mjs
+## Unified config dsh-eval.config.mjs
 
-消费者包根放一份，两个 CLI 从工作目录向上查找，flags 永远覆盖 config：
+Drop one at the consumer package root; both CLIs walk upward from the working directory, and flags always override config:
 
 ```js
 export default {
   profile: 'headless',              // dsh profile
-  repo: '../../deepseek-harness',   // 相对路径锚定 config 文件所在目录
-  mode: 'mock',                     // behavior CLI 的 --mode 默认（review 无此项）
-  failOnSkip: false,                // behavior CI 门禁默认
-  report: 'eval-report.json',       // --report 默认（锚定 config 目录）
-  disableRows: ['gates'],           // case 默认禁用的插件行；case 级声明覆盖
-                                     // （显式 [] = 全启用，gate 交互 case 用）
+  repo: '../../deepseek-harness',   // relative, anchored at the config file's directory
+  mode: 'mock',                     // behavior CLI's --mode default (review has none)
+  failOnSkip: false,                // behavior CI gate default
+  report: 'eval-report.json',       // --report default (anchored at the config dir)
+  disableRows: ['gates'],           // plugin rows disabled by default; case-level declarations win
+                                     // (explicit [] = all enabled, for gate-interaction cases)
 }
 ```
 
-未知 key 直接报错（拼写错误不静默退化）。`disableRows` 的语义与 turn-close 门禁边界契约见 [docs/disablerows.md](docs/disablerows.md)。
+Unknown keys fail loudly (typos never degrade silently). The `disableRows` semantics and the turn-close gate boundary contract are in [docs/disablerows.md](docs/disablerows.md).
 
 ## Docs
 
-| 文档 | 主题 |
+| Doc | Topic |
 |---|---|
-| [host-wiring](docs/host-wiring.md) | peer 接线（含 npm 古董 peer 坑）、构建 CLI、profile、凭证、spawn 要求 |
-| [review](docs/review.md) | comprehension review：实验定义、sterile profile、产物、六条固化规则 |
-| [matchers](docs/matchers.md) | trace matcher 与 mock helper 全集（工具面 / 文本面 / 模型可见面） |
-| [disablerows](docs/disablerows.md) | `disableRows` 与 turn-close 门禁边界契约 |
-| [intent-cases](docs/intent-cases.md) | real 意图 case 规约：何时写、断言面、守卫、CI 语义 |
-| [report](docs/report.md) | 机器可读报告（`--format json` / `--report`）结构 |
-| [known-issues](docs/known-issues.md) | 已知问题与规避（如 staged home 的 REQUEST_EXTENSION） |
-
-## 运行保障
-
-runner 用 `try/finally` 保证临时目录与链接在任何路径（`prepare` 抛错、mock 校验失败、spawn 错误）都被清理，不污染真实 profile store。behavior 与 review CLI 共享目录扫描（跳过 `.runs` 与 `node_modules`）；behavior CLI 在加载期做 case shape 校验与跨文件重复 id 检测，尽早失败。
-
-License: MIT。框架自身的测试与发布自检由仓库 CI 承接，不随包发布。
+| [host-wiring](docs/host-wiring.md) | peer wiring (incl. the npm antique-peer trap), building the CLI, profiles, credentials, spawn requirements |
+| [review](docs/review.md) | comprehension review: experiment definition, blank-environment reviewers, artifacts, the six review rules |
+| [matchers](docs/matchers.md) | the full trace-matcher and mock-helper set (tool face / text face / model-visible face) |
+| [disablerows](docs/disablerows.md) | `disableRows` and the turn-close gate boundary contract |
+| [rowconfig](docs/rowconfig.md) | the `rowConfig` per-row config override contract (whole-segment replacement, restate needed keys) |
+| [cross-turn](docs/cross-turn.md) | the `followups` cross-turn async driving contract (driver-row swap, subagent settle wait, single-cursor mock orchestration, bounded-redispatch assertions) |
+| [intent-cases](docs/intent-cases.md) | real intent-case spec: when to write one, assertion face, guards, CI semantics |
+| [report](docs/report.md) | machine-readable report structure (`--format json` / `--report`) |
+| [known-issues](docs/known-issues.md) | known issues and workarounds (e.g. REQUEST_EXTENSION in staged homes) |
+| [runner-api](docs/runner-api.md) | programmatic runner API: `runEvalCase` options contract, EvalRunResult fields, crossing tiers for `cliPath` |
+| [experimental](docs/experimental.md) | `experimental` subpath symbol list (escape hatch, no compatibility promise) |
+
+## Runtime guarantees
+
+The runner uses `try/finally` so temp directories and links are cleaned up on every path (`prepare` throwing, mock validation failure, spawn errors) — the real profile store is never polluted. The behavior and review CLIs share directory scanning (skipping `.runs` and `node_modules`); the behavior CLI validates case shapes and detects cross-file duplicate ids at load time, failing as early as possible.
+
+License: MIT. The framework's own tests and release self-checks are carried by the repository CI and do not ship with the package.

package/README.zh.md

@@ -0,0 +1,129 @@
+# @catheadowl/dsh-eval
+
+[English](README.md) | 中文
+
+**面向插件作者的 dsh-native agent 评测层**：behavior case 跑真实 headless dsh trace，review experiment 测 fresh model 能否理解插件输出。
+
+它评测的是**装配后的 agent harness**（插件 + profile + patch + 工具注册表在真实 dsh headless 里接成的那张图），不是孤立函数；判定走 dsh 原生的 session trace 投影与 matcher（契约断言），不是 metric 分数。它不是通用 agent eval 平台（无 dashboard / dataset hosting / metric catalog，也不做 benchmark 排名），也不是 DeepEval / OpenAI Evals 的替代品——那些项目证明了这个问题空间成立，本包选择 dsh-native 的垂直解法。
+
+> 文档以中文为主；深度契约在 [docs/](docs/README.md)（matchers / 边界契约 / review / 报告结构 / 宿主接线 / 已知问题）。
+
+## 为什么需要它
+
+| 类型 | 问题 | 判定 | 执行 |
+|---|---|---|---|
+| 单元/shape test | 确定性字段和值是否正确 | 自动 | plugin 自己的 `node:test` |
+| behavior real | 自然语言意图是否选到正确工具 | trace matcher | dsh + 真实模型 |
+| behavior mock | 工具管线与写入 round-trip 是否稳定 | trace matcher + workspace inspect | dsh + 脚本化 mock LLM |
+| comprehension review | 一个 fresh model 能否从输出理解含义和下一步 | 人工对照 rubric，多次收敛 | 抽象 review experiment + 可替换 executor |
+
+dsh 插件的正确性来自「装配出的图是否真的把工具、steer、prompt、gate 接到一起」——这类问题插件自己的单测只能覆盖一部分；而「输出能否被理解」根本不是字符串回归。本包把这两层从手动试跑变成可复跑证据。
+
+```text
+plugin-owned experiment             shared framework
+fixtures + prompt + rubric + observe ──► experiment/review.mjs
+                                               │ task
+                                               ▼
+                                        adapters/dsh/review.mjs ──► dsh headless
+
+behavior *.eval.mjs ───────────────────► dsh behavior runner (trace + mock)
+```
+
+- `src/experiment/` 是模型与 runtime 无关的试验设计层：blind review、实时观测、多次 reviewer 字节一致证据。它不 import dsh。
+- `src/adapters/dsh/` 是落地层：把抽象任务交给隔离的 dsh headless。
+- 你的 `eval/` 只保留领域 fixture、projection/observe、prompt、rubric 与 case，不复制 runner。
+
+## Install
+
+```bash
+npm i -D @catheadowl/dsh-eval
+```
+
+**Requirements**（接线细节与失败自诊断见 [docs/host-wiring.md](docs/host-wiring.md)）：
+
+- 一个已构建的 deepseek-harness 检出（`apps/cli/lib/bin.js`）；
+- 被测插件已装进某个 dsh profile；
+- peer 依赖 `@deepseek-ai/dsh-llm` 需手工接线（npm 会自动装到不兼容的古董版，须替换为指向宿主检出的链接）。
+
+## Quickstart
+
+`<plugin>/eval/behavior/mock/smoke.eval.mjs`：
+
+```js
+import { firstTool, toolCalled, toolCallStep, textStep } from '@catheadowl/dsh-eval'
+
+export default {
+  id: 'my-first-case',
+  mode: 'mock',
+  task: '把 guide.md 重命名为 intro.md',
+  async prepare(workspace) { /* 播种 fixture 文件 */ },
+  script: { steps: [toolCallStep('md_rename', { oldPath: 'guide.md', newPath: 'intro.md' }), textStep('done')] },
+  expect: [toolCalled('md_rename')],
+}
+```
+
+```bash
+dsh-eval run --mode mock eval/behavior/mock
+dsh-review --dry-run eval/comprehension     # review 层的免模型预演
+```
+
+命令需要知道用哪个 dsh profile：显式传 `--profile <name>`，或放一份 `dsh-eval.config.mjs` 到包根（见下节「统一配置」）。
+
+真实运行用 `dsh-eval run --profile <p> --repo <harness 检出> <case 路径>`；全部 flags（`--mode/--keep-artifacts/--fail-on-skip/--format/--report`）见 [docs/report.md](docs/report.md)。real case 无凭证时 auto-skip（dsh 自己解析凭证），mock 与 dry-run 不需要任何凭证。
+
+## 规范目录
+
+```text
+<plugin>/eval/
+  .gitignore                 # .runs/（无路径前缀）
+  README.md
+  behavior/                  # 可选
+    real/*.eval.mjs
+    mock/*.eval.mjs
+    _fixtures/
+  comprehension/             # 可选
+    <name>.review.mjs
+    fixtures.json
+    prompt.md
+    rubric.md
+```
+
+## 统一配置 dsh-eval.config.mjs
+
+消费者包根放一份，两个 CLI 从工作目录向上查找，flags 永远覆盖 config：
+
+```js
+export default {
+  profile: 'headless',              // dsh profile
+  repo: '../../deepseek-harness',   // 相对路径锚定 config 文件所在目录
+  mode: 'mock',                     // behavior CLI 的 --mode 默认（review 无此项）
+  failOnSkip: false,                // behavior CI 门禁默认
+  report: 'eval-report.json',       // --report 默认（锚定 config 目录）
+  disableRows: ['gates'],           // case 默认禁用的插件行；case 级声明覆盖
+                                     // （显式 [] = 全启用，gate 交互 case 用）
+}
+```
+
+未知 key 直接报错（拼写错误不静默退化）。`disableRows` 的语义与 turn-close 门禁边界契约见 [docs/disablerows.md](docs/disablerows.md)。
+
+## Docs
+
+| 文档 | 主题 |
+|---|---|
+| [host-wiring](docs/host-wiring.md) | peer 接线（含 npm 古董 peer 坑）、构建 CLI、profile、凭证、spawn 要求 |
+| [review](docs/review.md) | comprehension review：实验定义、空白环境 reviewer、产物、六条评审规则 |
+| [matchers](docs/matchers.md) | trace matcher 与 mock helper 全集（工具面 / 文本面 / 模型可见面） |
+| [disablerows](docs/disablerows.md) | `disableRows` 与 turn-close 门禁边界契约 |
+| [rowconfig](docs/rowconfig.md) | `rowConfig` 行 config 覆写契约（整段替换、重述所需键） |
+| [cross-turn](docs/cross-turn.md) | `followups` 跨轮异步驱动契约（driver 行换装、子 agent settle 等待、mock 单 cursor 编排、有界重派断言） |
+| [intent-cases](docs/intent-cases.md) | real 意图 case 规约：何时写、断言面、守卫、CI 语义 |
+| [report](docs/report.md) | 机器可读报告（`--format json` / `--report`）结构 |
+| [known-issues](docs/known-issues.md) | 已知问题与规避（如 staged home 的 REQUEST_EXTENSION） |
+| [runner-api](docs/runner-api.md) | 程序化 runner API：`runEvalCase` options 契约、EvalRunResult 字段、跨档取 `cliPath` |
+| [experimental](docs/experimental.md) | `experimental` 子路径符号清单（逃生面，无兼容承诺） |
+
+## 运行保障
+
+runner 用 `try/finally` 保证临时目录与链接在任何路径（`prepare` 抛错、mock 校验失败、spawn 错误）都被清理，不污染真实 profile store。behavior 与 review CLI 共享目录扫描（跳过 `.runs` 与 `node_modules`）；behavior CLI 在加载期做 case shape 校验与跨文件重复 id 检测，尽早失败。
+
+License: MIT。框架自身的测试与发布自检由仓库 CI 承接，不随包发布。