@catheadowl/dsh-eval 0.2.0 → 0.2.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +57 -0
- package/README.i18n.yaml +2 -2
- package/README.md +2 -5
- package/README.zh.md +2 -5
- package/bin/dsh-eval.mjs +335 -335
- package/bin/dsh-review.mjs +166 -154
- package/docs/README.md +3 -2
- package/docs/cross-turn.md +66 -0
- package/docs/host-wiring.md +1 -1
- package/docs/known-issues.md +9 -1
- package/docs/matchers.md +13 -1
- package/docs/review.md +12 -9
- package/package.json +7 -1
- package/src/adapters/dsh/review.mjs +70 -17
- package/src/assertions.mjs +499 -389
- package/src/discovery.mjs +190 -166
- package/src/driver/multi-turn-driver.mjs +163 -0
- package/src/experiment/review.mjs +118 -118
- package/src/index.mjs +4 -0
- package/src/mock/mock-adapter.mjs +73 -73
- package/src/mock/script.mjs +49 -49
- package/src/overlay.mjs +13 -0
- package/src/review-report.mjs +14 -1
- package/src/runner.mjs +21 -1
- package/src/sandbox.mjs +62 -1
- package/src/tool-validation.mjs +77 -77
- package/src/trace.mjs +293 -218
package/CHANGELOG.md
ADDED
|
@@ -0,0 +1,57 @@
|
|
|
1
|
+
---
|
|
2
|
+
description: Release history of @catheadowl/dsh-eval — one entry per published version, following Keep a Changelog conventions
|
|
3
|
+
---
|
|
4
|
+
|
|
5
|
+
# Changelog
|
|
6
|
+
|
|
7
|
+
All notable changes to `@catheadowl/dsh-eval` are documented here. Versions
|
|
8
|
+
follow [Semantic Versioning](https://semver.org/); entries follow
|
|
9
|
+
[Keep a Changelog](https://keepachangelog.com/) conventions.
|
|
10
|
+
|
|
11
|
+
## [0.2.1] — 2026-09-09
|
|
12
|
+
|
|
13
|
+
### Added
|
|
14
|
+
|
|
15
|
+
- Subagent dispatch observability: `subagentChildren` projection plus
|
|
16
|
+
`subagentDispatched` / `subagentCompleted` matchers.
|
|
17
|
+
- Multi-turn followups: cross-turn asynchronous driving for review
|
|
18
|
+
experiments, with dispatch/completion count assertions.
|
|
19
|
+
|
|
20
|
+
### Changed
|
|
21
|
+
|
|
22
|
+
- `dsh-review` boots a **blank environment by default**: staged out-of-tree
|
|
23
|
+
plugin rows are disabled via overlay, so host-profile gates/plugins can no
|
|
24
|
+
longer steer or crash a reviewer. Pass `--keep-plugin-rows` (or executor
|
|
25
|
+
option `keepPluginRows`) to opt back in deliberately.
|
|
26
|
+
|
|
27
|
+
### Fixed
|
|
28
|
+
|
|
29
|
+
- Review artifacts capture the reviewer's **answer**, not the last message:
|
|
30
|
+
conclusions use the last assistant text before any plugin-sourced injection
|
|
31
|
+
(trace-derived); the raw final message is kept as `run-N.stdout.txt` when it
|
|
32
|
+
diverges, and each report run cites its transcript.
|
|
33
|
+
|
|
34
|
+
## [0.2.0] — 2026-09-06
|
|
35
|
+
|
|
36
|
+
### Changed
|
|
37
|
+
|
|
38
|
+
- **BREAKING (0.x minor)**: public API split into two tiers. The root entry
|
|
39
|
+
is now the SDK tier only (assertion DSL, step builders, `runEvalCase`,
|
|
40
|
+
`defineReviewExperiment`); low-level sandbox/overlay/trace primitives and
|
|
41
|
+
review execution moved to the `./experimental` escape-hatch subpath.
|
|
42
|
+
- `exports` map added — deep path imports are now mechanically blocked.
|
|
43
|
+
|
|
44
|
+
## [0.1.0] — 2026-09-05
|
|
45
|
+
|
|
46
|
+
### Added
|
|
47
|
+
|
|
48
|
+
- Initial public release: case runner over dsh headless runs,
|
|
49
|
+
session-trace assertions, and a scripted mock-LLM layer for plugin intent
|
|
50
|
+
tests.
|
|
51
|
+
- CLI binaries `dsh-eval` (run cases) and `dsh-review` (LLM output review);
|
|
52
|
+
`@deepseek-ai/dsh-llm` is a peerDependency provided by the host
|
|
53
|
+
ecosystem.
|
|
54
|
+
|
|
55
|
+
[0.2.1]: https://github.com/CatheadOwl/dsh-eval/releases/tag/v0.2.1
|
|
56
|
+
[0.2.0]: https://github.com/CatheadOwl/dsh-eval/releases/tag/v0.2.0
|
|
57
|
+
[0.1.0]: https://github.com/CatheadOwl/dsh-eval/releases/tag/v0.1.0
|
package/README.i18n.yaml
CHANGED
|
@@ -2,5 +2,5 @@
|
|
|
2
2
|
# side as of the last confirmed-consistent state. Both languages carry equal authority; after
|
|
3
3
|
# editing either side, bring the other along and re-record with:
|
|
4
4
|
# node scripts/verify-readme-i18n.mjs --write
|
|
5
|
-
README.md:
|
|
6
|
-
README.zh.md:
|
|
5
|
+
README.md: 94e4e6c1172544c94c65329abf8dc6db81c0ba8030c354e1c1e8e64f8bc8276b
|
|
6
|
+
README.zh.md: a1f45726e695fe46931a2988c7293d53c61b3c275dc92e039e707b315f72fc60
|
package/README.md
CHANGED
|
@@ -1,7 +1,3 @@
|
|
|
1
|
-
---
|
|
2
|
-
description: '@catheadowl/dsh-eval — a dsh-native agent evaluation layer for plugin authors: behavior cases run against real headless dsh traces, review experiments test whether fresh models understand plugin outputs'
|
|
3
|
-
---
|
|
4
|
-
|
|
5
1
|
# @catheadowl/dsh-eval
|
|
6
2
|
|
|
7
3
|
English | [中文](README.zh.md)
|
|
@@ -115,10 +111,11 @@ Unknown keys fail loudly (typos never degrade silently). The `disableRows` seman
|
|
|
115
111
|
| Doc | Topic |
|
|
116
112
|
|---|---|
|
|
117
113
|
| [host-wiring](docs/host-wiring.md) | peer wiring (incl. the npm antique-peer trap), building the CLI, profiles, credentials, spawn requirements |
|
|
118
|
-
| [review](docs/review.md) | comprehension review: experiment definition,
|
|
114
|
+
| [review](docs/review.md) | comprehension review: experiment definition, blank-environment reviewers, artifacts, the six review rules |
|
|
119
115
|
| [matchers](docs/matchers.md) | the full trace-matcher and mock-helper set (tool face / text face / model-visible face) |
|
|
120
116
|
| [disablerows](docs/disablerows.md) | `disableRows` and the turn-close gate boundary contract |
|
|
121
117
|
| [rowconfig](docs/rowconfig.md) | the `rowConfig` per-row config override contract (whole-segment replacement, restate needed keys) |
|
|
118
|
+
| [cross-turn](docs/cross-turn.md) | the `followups` cross-turn async driving contract (driver-row swap, subagent settle wait, single-cursor mock orchestration, bounded-redispatch assertions) |
|
|
122
119
|
| [intent-cases](docs/intent-cases.md) | real intent-case spec: when to write one, assertion face, guards, CI semantics |
|
|
123
120
|
| [report](docs/report.md) | machine-readable report structure (`--format json` / `--report`) |
|
|
124
121
|
| [known-issues](docs/known-issues.md) | known issues and workarounds (e.g. REQUEST_EXTENSION in staged homes) |
|
package/README.zh.md
CHANGED
|
@@ -1,7 +1,3 @@
|
|
|
1
|
-
---
|
|
2
|
-
description: '@catheadowl/dsh-eval 中文主页——dsh-native agent 评测层:behavior case 跑真实 headless dsh trace,review experiment 测 fresh model 能否理解插件输出'
|
|
3
|
-
---
|
|
4
|
-
|
|
5
1
|
# @catheadowl/dsh-eval
|
|
6
2
|
|
|
7
3
|
[English](README.md) | 中文
|
|
@@ -115,10 +111,11 @@ export default {
|
|
|
115
111
|
| 文档 | 主题 |
|
|
116
112
|
|---|---|
|
|
117
113
|
| [host-wiring](docs/host-wiring.md) | peer 接线(含 npm 古董 peer 坑)、构建 CLI、profile、凭证、spawn 要求 |
|
|
118
|
-
| [review](docs/review.md) | comprehension review
|
|
114
|
+
| [review](docs/review.md) | comprehension review:实验定义、空白环境 reviewer、产物、六条评审规则 |
|
|
119
115
|
| [matchers](docs/matchers.md) | trace matcher 与 mock helper 全集(工具面 / 文本面 / 模型可见面) |
|
|
120
116
|
| [disablerows](docs/disablerows.md) | `disableRows` 与 turn-close 门禁边界契约 |
|
|
121
117
|
| [rowconfig](docs/rowconfig.md) | `rowConfig` 行 config 覆写契约(整段替换、重述所需键) |
|
|
118
|
+
| [cross-turn](docs/cross-turn.md) | `followups` 跨轮异步驱动契约(driver 行换装、子 agent settle 等待、mock 单 cursor 编排、有界重派断言) |
|
|
122
119
|
| [intent-cases](docs/intent-cases.md) | real 意图 case 规约:何时写、断言面、守卫、CI 语义 |
|
|
123
120
|
| [report](docs/report.md) | 机器可读报告(`--format json` / `--report`)结构 |
|
|
124
121
|
| [known-issues](docs/known-issues.md) | 已知问题与规避(如 staged home 的 REQUEST_EXTENSION) |
|