dsh-skill-eval 0.0.0-stage → 1.20.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +718 -0
- package/LICENSE +21 -0
- package/README.md +207 -2
- package/README.zh.md +164 -0
- package/client.js +2791 -0
- package/cordis.patch.yml +9 -0
- package/docs/ACCEPTANCE.md +71 -0
- package/docs/AUDIT.md +82 -0
- package/docs/CODE-REVIEW.md +49 -0
- package/docs/COMPARISON.md +33 -0
- package/docs/DETAIL.md +468 -0
- package/docs/DEV-RELOAD.md +114 -0
- package/docs/ENTRYPOINTS.md +65 -0
- package/docs/RELEASE.md +44 -0
- package/docs/RRSI-NOTES.md +129 -0
- package/docs/SCENARIOS.md +91 -0
- package/docs/UI-REVIEW.md +115 -0
- package/docs/WORKFLOW.md +108 -0
- package/icon.svg +9 -0
- package/index.js +1173 -0
- package/lib/accept.js +350 -0
- package/lib/advice.js +557 -0
- package/lib/critic.js +101 -0
- package/lib/dev.js +148 -0
- package/lib/history.js +141 -0
- package/lib/html.js +17 -0
- package/lib/inspect.js +142 -0
- package/lib/integrity.js +46 -0
- package/lib/loop.js +195 -0
- package/lib/random.js +29 -0
- package/lib/recorder.js +644 -0
- package/lib/render.js +580 -0
- package/lib/report.js +689 -0
- package/lib/score.js +565 -0
- package/lib/settings.js +415 -0
- package/lib/snapshot.js +250 -0
- package/lib/steps.js +141 -0
- package/lib/store.js +240 -0
- package/lib/validity.js +125 -0
- package/locale/en.json +6 -0
- package/locale/zh.json +6 -0
- package/package.json +96 -4
- package/scripts/accept.mjs +227 -0
- package/scripts/build-client.mjs +59 -0
- package/scripts/build.mjs +81 -0
- package/scripts/doctor.mjs +234 -0
- package/scripts/lib.mjs +87 -0
- package/scripts/verify-live.mjs +178 -0
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 dsh-skill-eval contributors
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
CHANGED
|
@@ -1,3 +1,208 @@
|
|
|
1
|
-
#
|
|
1
|
+
# dsh-skill-eval
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
[](https://www.npmjs.com/package/dsh-skill-eval)
|
|
4
|
+
[](LICENSE)
|
|
5
|
+
[](https://github.com/carlslin/dsh-skill-eval/releases)
|
|
6
|
+
|
|
7
|
+
Records what happens **after** a skill is loaded, scores every skill 0–100, and
|
|
8
|
+
turns the evidence into fixes the agent can act on. It is a DeepSeek Harness
|
|
9
|
+
(DSH) plugin: telemetry, the scoring board and the agent-facing
|
|
10
|
+
`skill_eval_report` tool ship together.
|
|
11
|
+
|
|
12
|
+
Most skill-quality tooling checks the skill _file_. This plugin checks the
|
|
13
|
+
_execution_: did the load succeed, did the tools that followed fail, did the user
|
|
14
|
+
push back, was the skill ignored, did it collide with a sibling skill, what did
|
|
15
|
+
it cost. Those signals only exist inside a running harness — which is why this
|
|
16
|
+
is a DSH plugin rather than a CLI.
|
|
17
|
+
|
|
18
|
+
## 中文速览 · TL;DR for Chinese readers
|
|
19
|
+
|
|
20
|
+
- 在每次 `skill` 加载后收集执行遥测:加载成败、后续工具报错、用户是否推翻、是否被忽略、碰撞与成本。
|
|
21
|
+
- 每个技能 0–100 分(A≥90 / B≥80 / C≥70 / D≥60 / F<60),带 95% bootstrap 区间;区间重叠时排名标为不显著。
|
|
22
|
+
- 入口只有三个:回合下方结果条、侧边栏「技能评测」面板、设置 → 插件 → `dsh-skill-eval` 的配置页。
|
|
23
|
+
- 建议靠规则生成,每条都带触发数字和可执行的修复;中文界面下标题与修复为中文。
|
|
24
|
+
- 完整中文文档:[README.zh.md](README.zh.md);入口清单:[docs/ENTRYPOINTS.md](docs/ENTRYPOINTS.md)。
|
|
25
|
+
|
|
26
|
+
## What you get
|
|
27
|
+
|
|
28
|
+
- **a result strip in the conversation** — under every completed turn that loaded a skill, a one-line card shows the skill scores, this turn's call/error counts and an expandable list of fixes (zero tokens, no popup). The old popup is opt-in (`window.__SEV_ENABLE_POPUP__ = true`).
|
|
29
|
+
- `skill_eval_report` tool — the agent reads its own telemetry instead of guessing
|
|
30
|
+
- `/skill-eval [skill]` command — same report in the composer
|
|
31
|
+
- `/skill-eval/board/<version>` — HTML board with grades, dimension bars and top fixes
|
|
32
|
+
- append-only JSONL telemetry in `$DSH_HOME/skill-eval/$DSH_PROFILE/events.jsonl`
|
|
33
|
+
|
|
34
|
+
## Screenshots
|
|
35
|
+
|
|
36
|
+
| Skills board | MCP tab | Scoring settings |
|
|
37
|
+
| ----------------------------------------- | ---------------------------------------- | -------------------------------------------------- |
|
|
38
|
+
|  |  |  |
|
|
39
|
+
|
|
40
|
+
More real-window screenshots and the per-screen notes:
|
|
41
|
+
[docs/UI-REVIEW.md](docs/UI-REVIEW.md).
|
|
42
|
+
|
|
43
|
+
## Scoring in one look
|
|
44
|
+
|
|
45
|
+
| dimension | weight | signal |
|
|
46
|
+
| ------------- | ------ | ----------------------------------------------------------------- |
|
|
47
|
+
| static | 0.30 | description, trigger cues, progressive disclosure, size, examples |
|
|
48
|
+
| reliability | 0.25 | load success × post-load tool error rate |
|
|
49
|
+
| effectiveness | 0.30 | user corrections, ignored loads, sibling collisions |
|
|
50
|
+
| efficiency | 0.15 | tool calls per run, load latency |
|
|
51
|
+
|
|
52
|
+
Grades: `A ≥ 90`, `B ≥ 80`, `C ≥ 70`, `D ≥ 60`, `F < 60`. Every score carries a
|
|
53
|
+
95% bootstrap confidence interval; overlapping intervals are reported as a
|
|
54
|
+
non-significant rank. Dimensions without evidence are dropped and their weight
|
|
55
|
+
redistributed. Full rubric, record schema and formulas: [docs/DETAIL.md](docs/DETAIL.md).
|
|
56
|
+
|
|
57
|
+
Advice is never generic. Each item carries the number that triggered it and a
|
|
58
|
+
concrete edit:
|
|
59
|
+
|
|
60
|
+
```
|
|
61
|
+
🛑 [critical] The user rejects the result right after the skill runs (effectiveness)
|
|
62
|
+
evidence: 67% of runs were followed by a correcting message (e.g. "不对,页眉没删掉")
|
|
63
|
+
fix: The body likely ignores stated constraints. Tighten the output contract and
|
|
64
|
+
the "do not" list; add a short self-check the model must run before answering.
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
## Entry points
|
|
68
|
+
|
|
69
|
+
Everyday use is three places only:
|
|
70
|
+
|
|
71
|
+
| Goal | Where |
|
|
72
|
+
| ---------------------- | ------------------------------------------------------------------------------------------ |
|
|
73
|
+
| See this turn's result | the result strip under a completed turn (score, calls/errors, next fix, edit history) |
|
|
74
|
+
| Open the board | the **Skill eval** panel icon in the sidebar (main panel, same seat as the context plugin) |
|
|
75
|
+
| Change scoring | **Settings → Plugins → the `dsh-skill-eval` bundle page → Configure** |
|
|
76
|
+
|
|
77
|
+
Web board: `/skill-eval/board/<version>` · web settings: `…/config` · JSON:
|
|
78
|
+
`…/data/<version>`. Slash commands, agent tools, terminal checks and debug
|
|
79
|
+
switches: [docs/ENTRYPOINTS.md](docs/ENTRYPOINTS.md).
|
|
80
|
+
|
|
81
|
+
```bash
|
|
82
|
+
dsh-skill-eval-doctor # packaging pre-flight (release gate with --release)
|
|
83
|
+
dsh-skill-eval-accept # generation & data contract (17 checks)
|
|
84
|
+
dsh-skill-eval-verify # product surfaces (7 checks + 2 manual, after a reload)
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
## Install
|
|
88
|
+
|
|
89
|
+
Requirements: DeepSeek Harness with plugin bundles (the package declares
|
|
90
|
+
`dsh.engines.dsh >= 0.2.0`) and Node.js ≥ 20 for the CLI helpers.
|
|
91
|
+
|
|
92
|
+
From npm:
|
|
93
|
+
|
|
94
|
+
```bash
|
|
95
|
+
dsh plugin --profile desktop add dsh-skill-eval
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
From GitHub source:
|
|
99
|
+
|
|
100
|
+
```bash
|
|
101
|
+
git clone https://github.com/carlslin/dsh-skill-eval.git
|
|
102
|
+
dsh plugin --profile desktop add "$PWD/dsh-skill-eval"
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
From a packed release tarball (what a published build ships):
|
|
106
|
+
|
|
107
|
+
```
|
|
108
|
+
dsh plugin --profile desktop add "$PWD/dist/dsh-skill-eval-<version>.tgz"
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
Or straight from the source directory, or through the agent's
|
|
112
|
+
`plugin_manager` → `install_bundle`. The package ships the prebuilt browser
|
|
113
|
+
bundle, so installation needs no build step; editing `client/*.js` requires
|
|
114
|
+
`npm run build:client` (a zero-dependency concatenation script). After install,
|
|
115
|
+
refresh the GUI: the board is the **Skill eval** panel icon in the sidebar,
|
|
116
|
+
settings live under **Settings → Plugins → `dsh-skill-eval`**.
|
|
117
|
+
|
|
118
|
+
## Use
|
|
119
|
+
|
|
120
|
+
- **Result strip** under every completed turn: score chip, calls/errors, next
|
|
121
|
+
fix and edit history (zero tokens).
|
|
122
|
+
- **Board**: skills / steps / runs / MCP / dev tabs with next steps and evidence validity, plus SVG charts (score distribution, skill trend sparkline, evidence coverage, MCP p50/p95, session activity). The **steps** tab scores every loop step and compares recent turns.
|
|
123
|
+
- **Agent**: ask "how did `pdf-tools` perform?" — the agent calls
|
|
124
|
+
`skill_eval_report`; before editing a skill it is told to call
|
|
125
|
+
`skill_eval_review` first.
|
|
126
|
+
- **Slash commands**: `/skill-eval` (summary), `/skill-eval <skill>`,
|
|
127
|
+
`loop [all]`, `validity`, `history`, `impact`, `mcp`, `runs`, `turns`, `dev`,
|
|
128
|
+
`config`, `accept`.
|
|
129
|
+
- **Read-only Inspect**: provider `skill-eval`, methods
|
|
130
|
+
`summary|skill|mcp|runs|edits|standard`.
|
|
131
|
+
- To improve a skill: act only on `critical`/`warning` items, apply the `fix`
|
|
132
|
+
lines, then re-read after a few more loads.
|
|
133
|
+
|
|
134
|
+
## Config
|
|
135
|
+
|
|
136
|
+
The form lives in the bundle page; the same values can be seeded in config or
|
|
137
|
+
`settings.json`. The essentials:
|
|
138
|
+
|
|
139
|
+
```yaml
|
|
140
|
+
enabled: true
|
|
141
|
+
windowToolCalls: 12 # tool calls that count as evidence for one load
|
|
142
|
+
minSamples: 3 # loads before a score stops being provisional
|
|
143
|
+
weights: { static: 30, reliability: 25, effectiveness: 30, efficiency: 15 }
|
|
144
|
+
thresholds: { excellent: 90, good: 80, pass: 70 }
|
|
145
|
+
tokenBudgetPerLoad: 20000 # billable tokens per load before efficiency/advice react
|
|
146
|
+
adviceNoiseFloor: true # advice must clear its 95% interval to stay critical
|
|
147
|
+
turnReports: true # one board per settled turn under reports/turns/
|
|
148
|
+
promptHint: true # the one-line system-prompt hint; false = zero fixed cost
|
|
149
|
+
registerReviewTool: true # skill_eval_review: leakage critic + audit ledger
|
|
150
|
+
```
|
|
151
|
+
|
|
152
|
+
Full key list (25 settings incl. pruning, edit budget, cache weight, dashboard
|
|
153
|
+
path, inspect provider and MCP thresholds): [docs/DETAIL.md](docs/DETAIL.md#configuration-keys).
|
|
154
|
+
`DSH_SKILL_EVAL_DIR` overrides the data directory; `$DSH_HOME/skill-eval/$DSH_PROFILE`
|
|
155
|
+
is the default.
|
|
156
|
+
|
|
157
|
+
## Release
|
|
158
|
+
|
|
159
|
+
See [docs/RELEASE.md](docs/RELEASE.md) for the checklist, packing commands and
|
|
160
|
+
the clean-profile smoke test. `npm run release:check` runs the doctor gate, the
|
|
161
|
+
test suite and the live acceptance; `node scripts/doctor.mjs --release` verifies
|
|
162
|
+
the matching `dist/` artifact.
|
|
163
|
+
|
|
164
|
+
## Build
|
|
165
|
+
|
|
166
|
+
Editing `client/*.js` requires rebuilding the single-file browser bundle:
|
|
167
|
+
|
|
168
|
+
```bash
|
|
169
|
+
npm run build # format check → lint → client build → doctor → 258 tests
|
|
170
|
+
npm run build:fast # client build + doctor only
|
|
171
|
+
npm run build:pack # full build + dist/dsh-skill-eval-<version>.tgz
|
|
172
|
+
npm run build:install # pack + install into the desktop profile
|
|
173
|
+
```
|
|
174
|
+
|
|
175
|
+
`scripts/build.mjs` runs the same commands as the release checklist; add
|
|
176
|
+
`--fix` to format before checking, `--install=<profile>` for another profile.
|
|
177
|
+
|
|
178
|
+
## Tests
|
|
179
|
+
|
|
180
|
+
```
|
|
181
|
+
node --test test/
|
|
182
|
+
```
|
|
183
|
+
|
|
184
|
+
258 tests (`npm test` rebuilds the browser bundle first): scoring bands, static checks, correction detection,
|
|
185
|
+
window lifecycle (turn-end snapshot → correction → close), store round-trip with
|
|
186
|
+
a torn line, analyzer join, markdown/HTML rendering incl. escaping and bilingual
|
|
187
|
+
advice copy, host route registration under duplicate-route pressure, and the
|
|
188
|
+
client surfaces mounted in jsdom.
|
|
189
|
+
|
|
190
|
+
## More docs
|
|
191
|
+
|
|
192
|
+
- [docs/AUDIT.md](docs/AUDIT.md) — multi-layer audit: findings, fixes, residuals
|
|
193
|
+
- [docs/CODE-REVIEW.md](docs/CODE-REVIEW.md) — code-hygiene review: dead code, duplication, cleanup
|
|
194
|
+
- [docs/UI-REVIEW.md](docs/UI-REVIEW.md) — real-window walkthrough with screenshots (`docs/ui/`)
|
|
195
|
+
- [docs/ENTRYPOINTS.md](docs/ENTRYPOINTS.md) — every entry point, including slash commands and debug switches
|
|
196
|
+
- [docs/WORKFLOW.md](docs/WORKFLOW.md) — load → measure → score with intervals → advice → screen edits → re-measure
|
|
197
|
+
- [docs/DETAIL.md](docs/DETAIL.md) — rubric, record schema, full configuration keys, why this exists
|
|
198
|
+
- [docs/RRSI-NOTES.md](docs/RRSI-NOTES.md) — research notes and what we borrowed
|
|
199
|
+
- [docs/COMPARISON.md](docs/COMPARISON.md) — prior art and what each misses
|
|
200
|
+
- [docs/DEV-RELOAD.md](docs/DEV-RELOAD.md) — hot-reload recipe and disposable-profile verification
|
|
201
|
+
- [CHANGELOG.md](CHANGELOG.md) — release history
|
|
202
|
+
|
|
203
|
+
## Known limits
|
|
204
|
+
|
|
205
|
+
- Loads are recorded from `skill` tool calls and user-typed `/skill-name` invocations; a skill the harness activates without either is invisible.
|
|
206
|
+
- Effectiveness is inferred from the next user message, not from a task verdict.
|
|
207
|
+
- Advice evidence keeps measured ids/numbers in English even in the Chinese UI.
|
|
208
|
+
- Telemetry grows without pruning: each load writes a skill/window pair, each resolved skill also a `static` profile, and every agent step one `step` row (the dominant growth in long sessions).
|
package/README.zh.md
ADDED
|
@@ -0,0 +1,164 @@
|
|
|
1
|
+
# dsh-skill-eval(技能评测)
|
|
2
|
+
|
|
3
|
+
[](https://www.npmjs.com/package/dsh-skill-eval)
|
|
4
|
+
[](LICENSE)
|
|
5
|
+
[](https://github.com/carlslin/dsh-skill-eval/releases)
|
|
6
|
+
|
|
7
|
+
记录 **skill 加载之后**真实发生了什么:加载成败、后续工具报错、用户有没有推翻、
|
|
8
|
+
技能是否被忽略、是否与兄弟技能碰撞、花了多少 token。每个技能打 0–100 分,并把
|
|
9
|
+
证据转成 agent 可以直接执行的修复建议。
|
|
10
|
+
|
|
11
|
+
大多数技能质量工具检查的是技能*文件*;这个插件检查的是*执行*。这些信号只存在于
|
|
12
|
+
运行中的 harness 里,所以它是一个 DSH 插件,而不是一个 CLI。
|
|
13
|
+
|
|
14
|
+
## English at a glance
|
|
15
|
+
|
|
16
|
+
- Telemetry after every skill load; 0–100 scores with 95% bootstrap intervals.
|
|
17
|
+
- Three human entry points: the per-turn result strip, the sidebar **Skill eval**
|
|
18
|
+
board panel, and **Settings → Plugins → `dsh-skill-eval`**.
|
|
19
|
+
- Rule-based advice with measured evidence, localized into Chinese in the
|
|
20
|
+
Chinese UI. Full English docs: [README.md](README.md).
|
|
21
|
+
|
|
22
|
+
## 产出
|
|
23
|
+
|
|
24
|
+
- **对话内结果条**:每个加载过技能的完成回合下方一行,显示分数、本轮调用/报错、
|
|
25
|
+
可展开的修复清单(零 token,不弹窗)。旧弹窗保留但需手动开启
|
|
26
|
+
(`window.__SEV_ENABLE_POPUP__ = true`)。
|
|
27
|
+
- `skill_eval_report` 工具 —— agent 直接读自己的遥测,不再靠猜
|
|
28
|
+
- `/skill-eval [技能]` 命令 —— 在输入框里看同一份报告
|
|
29
|
+
- `/skill-eval/board/<版本>` —— HTML 看板:等级、维度条、优先修复
|
|
30
|
+
- 追加式 JSONL 遥测:`$DSH_HOME/skill-eval/$DSH_PROFILE/events.jsonl`
|
|
31
|
+
|
|
32
|
+
## 截图
|
|
33
|
+
|
|
34
|
+
| 技能看板 | MCP 标签 | 评分配置 |
|
|
35
|
+
| ------------------------------------- | ----------------------------------------- | --------------------------------------------- |
|
|
36
|
+
|  |  |  |
|
|
37
|
+
|
|
38
|
+
更多真实窗口截图与逐屏记录:[docs/UI-REVIEW.md](docs/UI-REVIEW.md)。
|
|
39
|
+
|
|
40
|
+
## 评分一眼看懂
|
|
41
|
+
|
|
42
|
+
| 维度 | 权重 | 信号 |
|
|
43
|
+
| -------- | ---- | ------------------------------------ |
|
|
44
|
+
| 静态质量 | 0.30 | 描述、触发线索、渐进披露、体量、示例 |
|
|
45
|
+
| 可靠性 | 0.25 | 加载成功率 × 加载后工具错误率 |
|
|
46
|
+
| 有效性 | 0.30 | 用户纠正、加载后被忽略、兄弟技能碰撞 |
|
|
47
|
+
| 效率 | 0.15 | 每次运行的调用数、加载延迟 |
|
|
48
|
+
|
|
49
|
+
等级:`A ≥ 90`、`B ≥ 80`、`C ≥ 70`、`D ≥ 60`、`F < 60`。每个分数带 95% bootstrap
|
|
50
|
+
区间;区间重叠时排名标为不显著。没有证据的维度会被丢弃并重新分配权重,所以新技能
|
|
51
|
+
不会因为没历史而被惩罚。完整评分规则、记录 schema 与公式:[docs/DETAIL.md](docs/DETAIL.md)。
|
|
52
|
+
|
|
53
|
+
建议不是套话:每条都带触发它的数字和具体改法,中文界面下标题与修复为中文。
|
|
54
|
+
|
|
55
|
+
## 入口
|
|
56
|
+
|
|
57
|
+
日常只用三个地方:
|
|
58
|
+
|
|
59
|
+
| 目标 | 位置 |
|
|
60
|
+
| ------------ | -------------------------------------------------------------- |
|
|
61
|
+
| 看本回合结果 | 完成回合下方的结果条(分数、调用/报错、下一处修复、改动记录) |
|
|
62
|
+
| 打开看板 | 侧边栏 **「技能评测」** 面板图标(主面板,与上下文插件同位置) |
|
|
63
|
+
| 改评分标准 | **设置 → 插件 → `dsh-skill-eval` 包页面 → 配置** |
|
|
64
|
+
|
|
65
|
+
网页版看板:`/skill-eval/board/<版本>` · 网页配置:`…/config` · JSON:
|
|
66
|
+
`…/data/<版本>`。对话命令、agent 工具、终端检查与调试开关见
|
|
67
|
+
[docs/ENTRYPOINTS.md](docs/ENTRYPOINTS.md)。
|
|
68
|
+
|
|
69
|
+
```bash
|
|
70
|
+
dsh-skill-eval-doctor # 打包预检(--release 为发布门禁)
|
|
71
|
+
dsh-skill-eval-accept # 数据契约与生成检查(17 项)
|
|
72
|
+
dsh-skill-eval-verify # 产品面可达(7 项 + 2 项手动,重载后)
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
## 安装
|
|
76
|
+
|
|
77
|
+
要求:带插件 bundle 能力的 DeepSeek Harness(包内声明 `dsh.engines.dsh >= 0.2.0`),
|
|
78
|
+
CLI 辅助脚本需要 Node.js ≥ 20。
|
|
79
|
+
|
|
80
|
+
从 npm 安装:
|
|
81
|
+
|
|
82
|
+
```bash
|
|
83
|
+
dsh plugin --profile desktop add dsh-skill-eval
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
从 GitHub 源码安装:
|
|
87
|
+
|
|
88
|
+
```bash
|
|
89
|
+
git clone https://github.com/carlslin/dsh-skill-eval.git
|
|
90
|
+
dsh plugin --profile desktop add "$PWD/dsh-skill-eval"
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
从发布 tarball 安装(发布产物):
|
|
94
|
+
|
|
95
|
+
```
|
|
96
|
+
dsh plugin --profile desktop add "$PWD/dist/dsh-skill-eval-<版本>.tgz"
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
也可以直接从源码目录安装,或让 agent 调用 `plugin_manager` → `install_bundle`。
|
|
100
|
+
发布包内含预构建的客户端 bundle,安装无需构建;修改 `client/*.js` 后要跑
|
|
101
|
+
`npm run build:client`(零依赖的拼接脚本)。装完刷新 GUI:看板是侧边栏的
|
|
102
|
+
「技能评测」面板,配置在 **设置 → 插件 → `dsh-skill-eval`**。
|
|
103
|
+
|
|
104
|
+
## 用法
|
|
105
|
+
|
|
106
|
+
- **结果条**:完成回合下方,分数、调用/报错、下一处修复、改动记录(零 token)。
|
|
107
|
+
- **看板**:技能 / 轨迹 / 运行 / MCP / 开发者标签,含下一步、效度摘要与 SVG 图表;「轨迹」逐步打分并对比最近几个回合。
|
|
108
|
+
- **让 agent 看**:直接问「`pdf-tools` 表现怎么样」—— agent 会调 `skill_eval_report`;
|
|
109
|
+
改技能前它被告知先调 `skill_eval_review` 做泄漏审查。
|
|
110
|
+
- **对话命令**:`/skill-eval`、`/skill-eval <技能>`、`loop [all]`、`validity`、
|
|
111
|
+
`history`、`impact`、`mcp`、`runs`、`turns`、`dev`、`config`、`accept`。
|
|
112
|
+
- **只读 Inspect**:provider `skill-eval`,方法 `summary|skill|mcp|runs|edits|standard`。
|
|
113
|
+
- **怎么改技能**:只处理 `critical`/`warning`,照 `fix` 改,再用几轮真实加载重测。
|
|
114
|
+
|
|
115
|
+
## 配置
|
|
116
|
+
|
|
117
|
+
配置在包页面的原生表单里;也可以用 config 或 `settings.json` 预置。常用项:
|
|
118
|
+
|
|
119
|
+
```yaml
|
|
120
|
+
enabled: true
|
|
121
|
+
windowToolCalls: 12 # 一次加载的证据窗口内工具调用数
|
|
122
|
+
minSamples: 3 # 样本达到多少后分数不再算临时值
|
|
123
|
+
weights: { static: 30, reliability: 25, effectiveness: 30, efficiency: 15 }
|
|
124
|
+
thresholds: { excellent: 90, good: 80, pass: 70 }
|
|
125
|
+
tokenBudgetPerLoad: 20000 # 单次加载可计费 token 预算
|
|
126
|
+
adviceNoiseFloor: true # 建议必须越过 95% 区间才保持 critical
|
|
127
|
+
turnReports: true # 每个结算回合写一份看板到 reports/turns/
|
|
128
|
+
promptHint: true # 系统提示里一行提示;false = 零固定成本
|
|
129
|
+
registerReviewTool: true # skill_eval_review:泄漏审查 + 审计账本
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
全部 25 项(含剪枝、编辑预算、缓存权重、面板路径、Inspect provider、MCP 阈值):
|
|
133
|
+
[docs/DETAIL.md](docs/DETAIL.md#configuration-keys)。`DSH_SKILL_EVAL_DIR` 可覆盖数据目录,
|
|
134
|
+
默认 `$DSH_HOME/skill-eval/$DSH_PROFILE`。
|
|
135
|
+
|
|
136
|
+
## 测试
|
|
137
|
+
|
|
138
|
+
```
|
|
139
|
+
node --test test/
|
|
140
|
+
```
|
|
141
|
+
|
|
142
|
+
258 项(`npm test` 会先重建客户端 bundle):评分分档、静态检查、纠正识别、窗口生命周期(回合结束快照 →
|
|
143
|
+
纠正 → 关闭)、带残缺行的存储往返、分析器 join、Markdown/HTML 渲染(含转义与中文
|
|
144
|
+
建议文案)、重复路由压力下的宿主路由注册,以及客户端的 jsdom 挂载测试。
|
|
145
|
+
|
|
146
|
+
## 更多文档
|
|
147
|
+
|
|
148
|
+
- [docs/AUDIT.md](docs/AUDIT.md) —— 多层审计:问题、修复与遗留
|
|
149
|
+
- [docs/CODE-REVIEW.md](docs/CODE-REVIEW.md) —— 代码卫生:死代码、重复实现与清理
|
|
150
|
+
- [docs/UI-REVIEW.md](docs/UI-REVIEW.md) —— 真实窗口逐屏记录 + 截图(`docs/ui/`)
|
|
151
|
+
- [docs/ENTRYPOINTS.md](docs/ENTRYPOINTS.md) —— 全部入口、命令与调试开关
|
|
152
|
+
- [docs/WORKFLOW.md](docs/WORKFLOW.md) —— 加载 → 测量 → 区间评分 → 建议 → 审查改动 → 重测
|
|
153
|
+
- [docs/DETAIL.md](docs/DETAIL.md) —— 评分规则、记录 schema、完整配置项
|
|
154
|
+
- [docs/RELEASE.md](docs/RELEASE.md) —— 发布检查表与干净 profile 冒烟
|
|
155
|
+
- [docs/COMPARISON.md](docs/COMPARISON.md) —— 公开实现对比
|
|
156
|
+
- [docs/DEV-RELOAD.md](docs/DEV-RELOAD.md) —— 热重载与一次性 profile 验证
|
|
157
|
+
- [CHANGELOG.md](CHANGELOG.md) —— 版本历史
|
|
158
|
+
|
|
159
|
+
## 已知盲区
|
|
160
|
+
|
|
161
|
+
- 记录 `skill` 工具调用与用户输入的 `/skill-name`;harness 自动激活、既不调工具也不输命令的技能看不到。
|
|
162
|
+
- 有效性推断自下一条用户消息,不是任务结论。
|
|
163
|
+
- 中文界面下建议的 evidence 仍保留测量用的英文 id/数字。
|
|
164
|
+
- 遥测只追加、不自动清理(每次加载一对小记录)。
|