dsh-skill-eval 1.20.2 → 1.20.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +43 -0
- package/README.md +13 -6
- package/README.zh.md +11 -4
- package/client.js +230 -76
- package/docs/ACCEPTANCE.md +1 -1
- package/docs/AUDIT.md +18 -0
- package/docs/DETAIL.md +11 -6
- package/docs/ENTRYPOINTS.md +1 -1
- package/docs/RELEASE.md +2 -2
- package/docs/UI-REVIEW.md +8 -8
- package/docs/WORKFLOW.md +1 -1
- package/index.js +98 -36
- package/lib/accept.js +1 -1
- package/lib/advice.js +12 -2
- package/lib/dev.js +1 -1
- package/lib/history.js +3 -2
- package/lib/html.js +9 -0
- package/lib/recorder.js +16 -6
- package/lib/render.js +5 -7
- package/lib/report.js +49 -6
- package/lib/score.js +40 -0
- package/lib/settings.js +26 -7
- package/lib/snapshot.js +111 -27
- package/lib/store.js +55 -1
- package/lib/validity.js +12 -2
- package/package.json +1 -1
- package/scripts/accept.mjs +7 -6
- package/scripts/doctor.mjs +46 -11
- package/scripts/lib.mjs +35 -0
- package/scripts/verify-live.mjs +16 -6
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,48 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 1.20.3 — review hardening
|
|
4
|
+
|
|
5
|
+
A full independent review (host, client, tests/packaging, plus a second-model
|
|
6
|
+
pass) reproduced the issues below. Every fix has a regression test.
|
|
7
|
+
|
|
8
|
+
- **The per-turn strip counts real errors.** It read `data.isError`, but DSH's
|
|
9
|
+
tool-call payload is `ToolChatData { root }`; errors now come from
|
|
10
|
+
`root.isError`, and the strip retries the report when the 10 s cache predates
|
|
11
|
+
the turn (rapid consecutive turns used to render nothing).
|
|
12
|
+
- **Settings writes cannot lose a field.** Both HTTP paths now merge inside the
|
|
13
|
+
serialized write queue (`updateSettings`); two overlapping partial saves keep
|
|
14
|
+
both keys. `windowTtlMs`, `reportOnTurnEnd`, `turnReports` and `promptHint`
|
|
15
|
+
are read live, so "saved values apply immediately" is true.
|
|
16
|
+
- **A failed `skill` load opens no evidence window** and no longer supersedes the
|
|
17
|
+
previous skill's window; the test now asserts after turn-stop/dispose.
|
|
18
|
+
- **Corrupt window records cannot crash reporting.** Nested `mcp` buckets and
|
|
19
|
+
tool entries are normalised on read (plus defence-in-depth guards in the
|
|
20
|
+
score/report merge), and finite-but-invalid timestamps render as `—`.
|
|
21
|
+
- **The release checks can fail.** `canonical-route` fails on restart skew,
|
|
22
|
+
`turn-boards` fails with zero boards, manual rows are INFO and excluded from
|
|
23
|
+
the PASS count, and `weightsPercent` uses largest-remainder so it always sums
|
|
24
|
+
to 100 (2/2/2/1 used to show 101 and fail verify-live).
|
|
25
|
+
- **The accept CLI is honest.** It reads `dataDir` from the served report
|
|
26
|
+
(previous `JSON.parse` of an object was silently swallowed), and flag values
|
|
27
|
+
no longer become the base URL.
|
|
28
|
+
- **The packaging doctor passes in an installed tarball** (no `client/` sources
|
|
29
|
+
to compare), requires locale/scripts/README/CHANGELOG/LICENSE in `files`, and
|
|
30
|
+
`--release` opens the tarball instead of only checking that it exists.
|
|
31
|
+
- **Client degradation works.** Versioned data/settings routes that answer
|
|
32
|
+
200 HTML (DSH's prefix route) now fall back to the canonical JSON routes
|
|
33
|
+
instead of showing a parse error; the stale banner is reachable again.
|
|
34
|
+
- **Per-turn boards carry the settled turn** (session+turn passed from
|
|
35
|
+
`agent/turn-stopping`), coalesced writes emit one board per turn, snapshots
|
|
36
|
+
are written atomically, and history/impact use the shared timestamp guard.
|
|
37
|
+
- **Security/hygiene:** the settings fence also rejects rebinding Hosts (not
|
|
38
|
+
only a mismatched Origin), 403/413 paths and DNS rebinding have tests,
|
|
39
|
+
analyzer/listener disposers are released on teardown, the scheduler guards
|
|
40
|
+
against writes after dispose, duplicate `/skill-name` invocations collapse on
|
|
41
|
+
the durable read path, the dev tab renders the settings file/warning count,
|
|
42
|
+
the settings form keeps in-flight edits and labels its controls, and the
|
|
43
|
+
score colours follow the configured grade floors.
|
|
44
|
+
- Tests: **258 passing** (`npm test`, jsdom installed).
|
|
45
|
+
|
|
3
46
|
## 1.20.2 — MCP board and settings readability
|
|
4
47
|
|
|
5
48
|
- **MCP tab rebuilt.** The old tab was one SVG chart with two thin bars plus a
|
package/README.md
CHANGED
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
# dsh-skill-eval
|
|
2
2
|
|
|
3
|
+
[](https://www.npmjs.com/package/dsh-skill-eval)
|
|
3
4
|
[](LICENSE)
|
|
4
5
|
[](https://github.com/carlslin/dsh-skill-eval/releases)
|
|
5
6
|
|
|
@@ -32,8 +33,8 @@ is a DSH plugin rather than a CLI.
|
|
|
32
33
|
|
|
33
34
|
## Screenshots
|
|
34
35
|
|
|
35
|
-
| Skills board
|
|
36
|
-
|
|
|
36
|
+
| Skills board | MCP tab | Scoring settings |
|
|
37
|
+
| ----------------------------------------- | ---------------------------------------- | -------------------------------------------------- |
|
|
37
38
|
|  |  |  |
|
|
38
39
|
|
|
39
40
|
More real-window screenshots and the per-screen notes:
|
|
@@ -80,7 +81,7 @@ switches: [docs/ENTRYPOINTS.md](docs/ENTRYPOINTS.md).
|
|
|
80
81
|
```bash
|
|
81
82
|
dsh-skill-eval-doctor # packaging pre-flight (release gate with --release)
|
|
82
83
|
dsh-skill-eval-accept # generation & data contract (17 checks)
|
|
83
|
-
dsh-skill-eval-verify # product surfaces (
|
|
84
|
+
dsh-skill-eval-verify # product surfaces (7 checks + 2 manual, after a reload)
|
|
84
85
|
```
|
|
85
86
|
|
|
86
87
|
## Install
|
|
@@ -88,6 +89,12 @@ dsh-skill-eval-verify # product surfaces (9 checks, after a reload)
|
|
|
88
89
|
Requirements: DeepSeek Harness with plugin bundles (the package declares
|
|
89
90
|
`dsh.engines.dsh >= 0.2.0`) and Node.js ≥ 20 for the CLI helpers.
|
|
90
91
|
|
|
92
|
+
From npm:
|
|
93
|
+
|
|
94
|
+
```bash
|
|
95
|
+
dsh plugin --profile desktop add dsh-skill-eval
|
|
96
|
+
```
|
|
97
|
+
|
|
91
98
|
From GitHub source:
|
|
92
99
|
|
|
93
100
|
```bash
|
|
@@ -159,7 +166,7 @@ the matching `dist/` artifact.
|
|
|
159
166
|
Editing `client/*.js` requires rebuilding the single-file browser bundle:
|
|
160
167
|
|
|
161
168
|
```bash
|
|
162
|
-
npm run build # format check → lint → client build → doctor →
|
|
169
|
+
npm run build # format check → lint → client build → doctor → 258 tests
|
|
163
170
|
npm run build:fast # client build + doctor only
|
|
164
171
|
npm run build:pack # full build + dist/dsh-skill-eval-<version>.tgz
|
|
165
172
|
npm run build:install # pack + install into the desktop profile
|
|
@@ -174,7 +181,7 @@ npm run build:install # pack + install into the desktop profile
|
|
|
174
181
|
node --test test/
|
|
175
182
|
```
|
|
176
183
|
|
|
177
|
-
|
|
184
|
+
258 tests (`npm test` rebuilds the browser bundle first): scoring bands, static checks, correction detection,
|
|
178
185
|
window lifecycle (turn-end snapshot → correction → close), store round-trip with
|
|
179
186
|
a torn line, analyzer join, markdown/HTML rendering incl. escaping and bilingual
|
|
180
187
|
advice copy, host route registration under duplicate-route pressure, and the
|
|
@@ -198,4 +205,4 @@ client surfaces mounted in jsdom.
|
|
|
198
205
|
- Loads are recorded from `skill` tool calls and user-typed `/skill-name` invocations; a skill the harness activates without either is invisible.
|
|
199
206
|
- Effectiveness is inferred from the next user message, not from a task verdict.
|
|
200
207
|
- Advice evidence keeps measured ids/numbers in English even in the Chinese UI.
|
|
201
|
-
- Telemetry grows without pruning
|
|
208
|
+
- Telemetry grows without pruning: each load writes a skill/window pair, each resolved skill also a `static` profile, and every agent step one `step` row (the dominant growth in long sessions).
|
package/README.zh.md
CHANGED
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
# dsh-skill-eval(技能评测)
|
|
2
2
|
|
|
3
|
+
[](https://www.npmjs.com/package/dsh-skill-eval)
|
|
3
4
|
[](LICENSE)
|
|
4
5
|
[](https://github.com/carlslin/dsh-skill-eval/releases)
|
|
5
6
|
|
|
@@ -30,8 +31,8 @@
|
|
|
30
31
|
|
|
31
32
|
## 截图
|
|
32
33
|
|
|
33
|
-
| 技能看板
|
|
34
|
-
|
|
|
34
|
+
| 技能看板 | MCP 标签 | 评分配置 |
|
|
35
|
+
| ------------------------------------- | ----------------------------------------- | --------------------------------------------- |
|
|
35
36
|
|  |  |  |
|
|
36
37
|
|
|
37
38
|
更多真实窗口截图与逐屏记录:[docs/UI-REVIEW.md](docs/UI-REVIEW.md)。
|
|
@@ -68,7 +69,7 @@
|
|
|
68
69
|
```bash
|
|
69
70
|
dsh-skill-eval-doctor # 打包预检(--release 为发布门禁)
|
|
70
71
|
dsh-skill-eval-accept # 数据契约与生成检查(17 项)
|
|
71
|
-
dsh-skill-eval-verify # 产品面可达(
|
|
72
|
+
dsh-skill-eval-verify # 产品面可达(7 项 + 2 项手动,重载后)
|
|
72
73
|
```
|
|
73
74
|
|
|
74
75
|
## 安装
|
|
@@ -76,6 +77,12 @@ dsh-skill-eval-verify # 产品面可达(9 项,重载后)
|
|
|
76
77
|
要求:带插件 bundle 能力的 DeepSeek Harness(包内声明 `dsh.engines.dsh >= 0.2.0`),
|
|
77
78
|
CLI 辅助脚本需要 Node.js ≥ 20。
|
|
78
79
|
|
|
80
|
+
从 npm 安装:
|
|
81
|
+
|
|
82
|
+
```bash
|
|
83
|
+
dsh plugin --profile desktop add dsh-skill-eval
|
|
84
|
+
```
|
|
85
|
+
|
|
79
86
|
从 GitHub 源码安装:
|
|
80
87
|
|
|
81
88
|
```bash
|
|
@@ -132,7 +139,7 @@ registerReviewTool: true # skill_eval_review:泄漏审查 + 审计账本
|
|
|
132
139
|
node --test test/
|
|
133
140
|
```
|
|
134
141
|
|
|
135
|
-
|
|
142
|
+
258 项(`npm test` 会先重建客户端 bundle):评分分档、静态检查、纠正识别、窗口生命周期(回合结束快照 →
|
|
136
143
|
纠正 → 关闭)、带残缺行的存储往返、分析器 join、Markdown/HTML 渲染(含转义与中文
|
|
137
144
|
建议文案)、重复路由压力下的宿主路由注册,以及客户端的 jsdom 挂载测试。
|
|
138
145
|
|