dsh-skill-eval 0.0.0-stage → 1.20.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +675 -0
- package/LICENSE +21 -0
- package/README.md +200 -2
- package/README.zh.md +157 -0
- package/client.js +2637 -0
- package/cordis.patch.yml +9 -0
- package/docs/ACCEPTANCE.md +71 -0
- package/docs/AUDIT.md +64 -0
- package/docs/CODE-REVIEW.md +49 -0
- package/docs/COMPARISON.md +33 -0
- package/docs/DETAIL.md +463 -0
- package/docs/DEV-RELOAD.md +114 -0
- package/docs/ENTRYPOINTS.md +65 -0
- package/docs/RELEASE.md +44 -0
- package/docs/RRSI-NOTES.md +129 -0
- package/docs/SCENARIOS.md +91 -0
- package/docs/UI-REVIEW.md +115 -0
- package/docs/WORKFLOW.md +108 -0
- package/icon.svg +9 -0
- package/index.js +1111 -0
- package/lib/accept.js +350 -0
- package/lib/advice.js +547 -0
- package/lib/critic.js +101 -0
- package/lib/dev.js +148 -0
- package/lib/history.js +140 -0
- package/lib/html.js +8 -0
- package/lib/inspect.js +142 -0
- package/lib/integrity.js +46 -0
- package/lib/loop.js +195 -0
- package/lib/random.js +29 -0
- package/lib/recorder.js +634 -0
- package/lib/render.js +582 -0
- package/lib/report.js +646 -0
- package/lib/score.js +525 -0
- package/lib/settings.js +396 -0
- package/lib/snapshot.js +166 -0
- package/lib/steps.js +141 -0
- package/lib/store.js +186 -0
- package/lib/validity.js +115 -0
- package/locale/en.json +6 -0
- package/locale/zh.json +6 -0
- package/package.json +96 -4
- package/scripts/accept.mjs +226 -0
- package/scripts/build-client.mjs +59 -0
- package/scripts/build.mjs +81 -0
- package/scripts/doctor.mjs +199 -0
- package/scripts/lib.mjs +52 -0
- package/scripts/verify-live.mjs +168 -0
package/CHANGELOG.md
ADDED
|
@@ -0,0 +1,675 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## 1.20.2 — MCP board and settings readability
|
|
4
|
+
|
|
5
|
+
- **MCP tab rebuilt.** The old tab was one SVG chart with two thin bars plus a
|
|
6
|
+
bare grid: the value label collided with the threshold line, the last table
|
|
7
|
+
column was clipped ("2606/260…"), and the table had no container. It is now a
|
|
8
|
+
latency card with CSS meters (p50 solid, p95 tinted, dashed threshold marker
|
|
9
|
+
on the track), a carded server table with a grade chip, and carded advice /
|
|
10
|
+
failing-tool sections. Latencies render as `2.6s` / `180ms` so nothing is
|
|
11
|
+
truncated. With no recorded calls the tab is a single empty card instead of a
|
|
12
|
+
chart card plus an empty table that read as two broken panels.
|
|
13
|
+
- **Settings readability.** Weight fields show a `%` unit, grade-floor help no
|
|
14
|
+
longer repeats the label ("A 级下限 / A 级下限。"), units are localized in the
|
|
15
|
+
Chinese form and web page (`loads → 次加载`, `edits → 次修改`), the config
|
|
16
|
+
dialog is titled 「配置」 with the field summary as its subtitle, and the web
|
|
17
|
+
page keeps readable headings in dark mode.
|
|
18
|
+
- **CSS collision fixed.** The advice container reused `.sev-sub`, which also
|
|
19
|
+
styles the panel subtitle, so the later rule silently restyled the header.
|
|
20
|
+
Advice blocks now use `.sev-advice-list` / `.sev-card`.
|
|
21
|
+
|
|
22
|
+
## 1.20.1 — board honesty pass
|
|
23
|
+
|
|
24
|
+
- **The 「配置」 button no longer relies on `window.open`.** The desktop shell
|
|
25
|
+
denies every popup (`setWindowOpenHandler` returns `deny`; only `https:` URLs
|
|
26
|
+
are handed to the browser), so on an `http://127.0.0.1` board the button did
|
|
27
|
+
nothing at all. It now opens the same settings form in an in-panel dialog,
|
|
28
|
+
with Escape/backdrop close and the web-config link hidden where the shell
|
|
29
|
+
would deny it. The web page and the native bundle seat still exist.
|
|
30
|
+
- **The dev tab reports real diagnostics.** The data route now carries a `dev`
|
|
31
|
+
block (loaded module vs package on disk with the restart-skew flag, claimed
|
|
32
|
+
route/tool counts, settings file, warning count) and the tab renders it next
|
|
33
|
+
to the effective scoring standard; the data endpoint shown is the versioned
|
|
34
|
+
URL the client actually calls, not the bare prefix.
|
|
35
|
+
- **The MCP latency chart follows the configured threshold.** `mcpP95Ms` now
|
|
36
|
+
ships with the report, so the alert line can no longer disagree with the MCP
|
|
37
|
+
advice that uses the same number (3000 ms remains the older-payload fallback).
|
|
38
|
+
|
|
39
|
+
## 1.20.0 — per-step trajectory
|
|
40
|
+
|
|
41
|
+
- **Per-step loop telemetry:** the recorder persists one row per agent step
|
|
42
|
+
(turn, step, tool calls/errors, MCP calls, loop errors, tokens, duration,
|
|
43
|
+
loaded skills). `lib/steps.js` scores each step 0–100 and groups turns.
|
|
44
|
+
- **New board tab 「轨迹 / Steps」:** step KPIs, recent-turn comparison with
|
|
45
|
+
deltas, per-turn step list with score chips, and a skill filter. Each skill
|
|
46
|
+
card gets a「查看轨迹」button that opens the tab filtered to that skill.
|
|
47
|
+
- Step records are additive event-log rows; old telemetry stays readable.
|
|
48
|
+
- **Multi-skill turn strip:** each skill loaded in a turn keeps its own chip
|
|
49
|
+
and now its own "next fix" (up to three); previously only the first global
|
|
50
|
+
suggestion was shown. Covered by a two-skill regression test.
|
|
51
|
+
- **npm bin targets normalized** (`scripts/*.mjs` without `./`): npm 11 was
|
|
52
|
+
stripping all three bins at publish time with an "invalid and removed"
|
|
53
|
+
warning, so a published install would have lost the CLI helpers.
|
|
54
|
+
- **One-command build:** `npm run build` (format/lint/client/doctor/tests),
|
|
55
|
+
`npm run build:pack` (adds tarball + release doctor), `npm run build:install`
|
|
56
|
+
(also installs into the desktop profile), `npm run build:fast` (client +
|
|
57
|
+
doctor only). Implemented in `scripts/build.mjs`.
|
|
58
|
+
- Tests: **239 passing** (new step scoring/recorder/report/client-filter suites).
|
|
59
|
+
|
|
60
|
+
## 1.19.1 — test hygiene
|
|
61
|
+
|
|
62
|
+
- Client tests share one `createHost(window, style)` helper and top-level
|
|
63
|
+
`createRoot`/`act` imports instead of 18 copies of the same setup.
|
|
64
|
+
- No runtime change; 232 tests still passing.
|
|
65
|
+
|
|
66
|
+
## 1.19.0 — board charts + client parts + tooling
|
|
67
|
+
|
|
68
|
+
- **Client split with a zero-dependency builder.** `client.js` is now generated
|
|
69
|
+
from `client/*.js` by `scripts/build-client.mjs`; `prepack`/`pretest` rebuild
|
|
70
|
+
it, doctor fails when the shipped file is stale, and `client/README.md`
|
|
71
|
+
documents the workflow. The package still ships one self-contained loader
|
|
72
|
+
factory, so installation needs no build.
|
|
73
|
+
- **Terminal scripts share plumbing.** `scripts/lib.mjs` now owns bridge/base
|
|
74
|
+
discovery, dashboard base, expected version and timeout HTTP GET; accept and
|
|
75
|
+
verify use it (tests cover precedence and error messages).
|
|
76
|
+
- **Formatting/linting.** Prettier (config + `format`/`format:check`) and ESLint
|
|
77
|
+
10 flat config (`lint`) added as dev dependencies; all source files formatted.
|
|
78
|
+
- **Board charts (pure SVG, no dependency):** score distribution histogram with
|
|
79
|
+
average marker, per-skill trend sparkline, evidence-coverage donut, MCP
|
|
80
|
+
p50/p95 latency chart with the 3000 ms threshold, and session-activity bars.
|
|
81
|
+
Charts also appear on the Runs and MCP tabs; every chart carries an
|
|
82
|
+
`aria-label` with its data.
|
|
83
|
+
- **Tests:** 232 passing; lint/format/doctor/package gates re-run in 1.19.0.
|
|
84
|
+
|
|
85
|
+
## 1.18.5 — hygiene follow-up
|
|
86
|
+
|
|
87
|
+
- **Locale metadata restored.** DSH reads `locale/<lang>.json` for the plugin's
|
|
88
|
+
localized market title/description; 1.18.4 wrongly deleted it. The files are
|
|
89
|
+
back, trimmed to `meta.title` + `meta.description` only (the old 16 unused UI
|
|
90
|
+
keys were the actual drift), and the doctor now checks both locale metas.
|
|
91
|
+
- **Settings group metadata now comes from the host.** Every field ships
|
|
92
|
+
`group`/`groupZh`; the client groups by it and keeps its local classifier only
|
|
93
|
+
as an older-payload fallback, removing a second copy of the grouping rules.
|
|
94
|
+
- Tests: **228 passing**.
|
|
95
|
+
|
|
96
|
+
## 1.18.4 — code hygiene
|
|
97
|
+
|
|
98
|
+
Full review: [docs/CODE-REVIEW.md](docs/CODE-REVIEW.md).
|
|
99
|
+
|
|
100
|
+
- Removed dead `locale/*.json` (unused at runtime, already drifted from the
|
|
101
|
+
client dictionary), with exports/files/doctor/docs updated.
|
|
102
|
+
- Consolidated duplicated helpers: `lib/random.js` (FNV hash + mulberry32) and
|
|
103
|
+
`lib/html.js` (HTML escape) used by score/report/index/render/settings.
|
|
104
|
+
- Deleted dead TEXTS keys and CSS left by the removed More-menu/detail surfaces;
|
|
105
|
+
documented the real client seats in the file header; added the missing
|
|
106
|
+
`.sev-btn--ghost` rule.
|
|
107
|
+
- Restored serialized atomic settings writes (a regression from an earlier
|
|
108
|
+
refactor) and added a regression test; added a settings-group drift test.
|
|
109
|
+
- Added `.editorconfig` and `.gitignore`.
|
|
110
|
+
- Tests: **227 passing**.
|
|
111
|
+
|
|
112
|
+
## 1.18.3 — settings aligned with the general settings layout
|
|
113
|
+
|
|
114
|
+
- Per-setting help lines (zh/en), units (loads / ms / tokens / edits), and
|
|
115
|
+
switch-style booleans for the 25 settings; section rows now use the same
|
|
116
|
+
label + description + control pattern as DSH's 通用设置.
|
|
117
|
+
- The web config page renders the same help text and units.
|
|
118
|
+
- 225 tests.
|
|
119
|
+
|
|
120
|
+
## 1.18.2 — panel and settings polish
|
|
121
|
+
|
|
122
|
+
Reference: the context-insight panel (`dsh-context`) and the generic settings
|
|
123
|
+
layout.
|
|
124
|
+
|
|
125
|
+
- **Board panel**: KPI row (skills / loads / actionable / average score), a
|
|
126
|
+
search box, an "needs a fix" filter chip, a “Configure” shortcut next to
|
|
127
|
+
Refresh/Copy, and a last-updated stamp.
|
|
128
|
+
- **Native settings form**: a bilingual heading + subtitle, and the missing
|
|
129
|
+
`windowTtlMs` field (25 settings); labels/group titles follow the app locale.
|
|
130
|
+
- English TEXTS parity restored for every new string; 225 tests.
|
|
131
|
+
|
|
132
|
+
## 1.18.1 — four-layer audit fixes
|
|
133
|
+
|
|
134
|
+
Full report: [docs/AUDIT.md](docs/AUDIT.md). Findings reproduced and fixed
|
|
135
|
+
across host data, scoring, HTTP, client and packaging.
|
|
136
|
+
|
|
137
|
+
- **Data integrity:** `__proto__` telemetry keys can no longer pollute
|
|
138
|
+
`Object.prototype` or crash reports; a torn final JSONL line is repaired
|
|
139
|
+
before the next append; cumulative window snapshots no longer double-count
|
|
140
|
+
run counters.
|
|
141
|
+
- **Correctness:** the bootstrap CI uses the same scoring options as the point
|
|
142
|
+
estimate; recorded static profiles are normalised; evicted windows are
|
|
143
|
+
censored from abandoned-run scoring; grade floors are sorted, not clamped.
|
|
144
|
+
- **Privacy/size:** skill bodies are no longer persisted or shipped in reports.
|
|
145
|
+
- **Performance:** report cache keyed by store identity + 2 s TTL; stable locale
|
|
146
|
+
subscriptions; memoized cards.
|
|
147
|
+
- **HTTP:** same-origin write gate + `x-skill-eval-config` header, atomic
|
|
148
|
+
serialized settings writes, merged-weight validation, byte-accurate 413,
|
|
149
|
+
versioned board/config pages render their own base.
|
|
150
|
+
- **Client:** request generation guard, focus trap/restore, clipboard cleanup,
|
|
151
|
+
settings-page styles, duplicate i18n key fixed, aria roles/live status,
|
|
152
|
+
English TEXTS parity.
|
|
153
|
+
- Settings: `windowTtlMs` (25 fields), bounds aligned with the Config schema.
|
|
154
|
+
- Tests: **225 passing**.
|
|
155
|
+
|
|
156
|
+
## 1.17.5 — last mixed-language strings
|
|
157
|
+
|
|
158
|
+
- Ranking note and validity statements are bilingual (`noteZh`, `canSayZh`,
|
|
159
|
+
`cannotSayZh`); HTML board and client panel pick the Chinese copy.
|
|
160
|
+
- 212 tests at the time.
|
|
161
|
+
|
|
162
|
+
## 1.17.4 — advice localization and README tightening
|
|
163
|
+
|
|
164
|
+
- Every advice rule carries `titleZh`/`fixZh` (MCP names filled in); localize
|
|
165
|
+
the whole report once for the HTML board; client cards/MCP list/turn strip
|
|
166
|
+
select the Chinese copy. README opens with a Chinese TL;DR; config key list
|
|
167
|
+
moved to `docs/DETAIL.md`; full config documented.
|
|
168
|
+
|
|
169
|
+
## 1.17.3 — release candidate
|
|
170
|
+
|
|
171
|
+
- Copy button falls back to `execCommand('copy')` with failure feedback.
|
|
172
|
+
- Packaging: LICENSE, explicit `files`, three bins, release scripts and doctor
|
|
173
|
+
release gate; clean-profile smoke documented in `docs/RELEASE.md`.
|
|
174
|
+
|
|
175
|
+
## 1.17.0
|
|
176
|
+
|
|
177
|
+
Board and settings moved to the same seats the context plugin (`dsh-context`)
|
|
178
|
+
uses, per the request to follow that pattern.
|
|
179
|
+
|
|
180
|
+
- **Board = sidebar panel + main panel.** Registers `sidebar.panellist`
|
|
181
|
+
(id `skill-eval`, order 40, gauge icon, localized label) and `main` keyed
|
|
182
|
+
`skill-eval`. Clicking the sidebar icon opens a full central board panel:
|
|
183
|
+
header with skill/actionable counts, tabs (skills / runs / MCP / dev),
|
|
184
|
+
refresh and copy, reusing the existing SkillCard/RunsView/McpView/DevView.
|
|
185
|
+
- **Settings = bundle config page.** Registers `plugins.bundle.config` keyed
|
|
186
|
+
`dsh-skill-eval` (the bundle's own page under Plugins), replacing the row
|
|
187
|
+
config seat; the native settings form is unchanged.
|
|
188
|
+
- Removed: the session-header More-menu board row and the row-config seat.
|
|
189
|
+
Kept: the in-conversation result strip, the versioned web board, the
|
|
190
|
+
debug-only popup (opt-in).
|
|
191
|
+
- Client tests rewritten for the new seat set (208 passing).
|
|
192
|
+
|
|
193
|
+
## 1.16.1
|
|
194
|
+
|
|
195
|
+
- **Versioned page aliases.** `/skill-eval/board/<version>` and
|
|
196
|
+
`/skill-eval/board/<version>/config` are registered as exact routes each
|
|
197
|
+
release. Hot-reloaded generations cannot reclaim the canonical page route an
|
|
198
|
+
older generation still owns, which is why the bilingual board initially kept
|
|
199
|
+
serving the old page; the versioned alias is always the current one. The
|
|
200
|
+
client's board/config links and the verify/accept scripts use it (with a
|
|
201
|
+
canonical fallback for a clean start).
|
|
202
|
+
- Verified live: `?lang=zh` / `Accept-Language: zh-CN` return Chinese chrome on
|
|
203
|
+
`/skill-eval/board/1.16.1`, the settings page renders Chinese labels, and the
|
|
204
|
+
English default is unchanged.
|
|
205
|
+
|
|
206
|
+
## 1.16.0
|
|
207
|
+
|
|
208
|
+
The board and the settings page are bilingual (中 / EN).
|
|
209
|
+
|
|
210
|
+
- Language resolution: explicit `?lang=zh|en` → `Accept-Language` header →
|
|
211
|
+
English fallback (pure `resolveLang`, unit-tested).
|
|
212
|
+
- The board translates its chrome: title, scoring line, totals, dimension names,
|
|
213
|
+
next steps, evidence (can / cannot say), screened edits, MCP and runs tables,
|
|
214
|
+
plus a `中文 / English` switch link.
|
|
215
|
+
- The settings page translates group titles, all 24 field labels, banners and
|
|
216
|
+
buttons; the switch keeps you on the page.
|
|
217
|
+
- Advice titles/evidence/fixes come from the host rules and stay in English for
|
|
218
|
+
now; the surrounding chrome is translated. Documented in `docs/ENTRYPOINTS.md`.
|
|
219
|
+
- 208 tests (new i18n suite + route-level language checks: header wins over
|
|
220
|
+
default, explicit query wins over header).
|
|
221
|
+
|
|
222
|
+
## 1.15.5
|
|
223
|
+
|
|
224
|
+
Entry points consolidated to three human surfaces.
|
|
225
|
+
|
|
226
|
+
- Kept: the in-conversation result strip (`conversation.chat.turnTail`), one
|
|
227
|
+
**Skill eval board** row in the session header's **More** menu
|
|
228
|
+
(`conversation.session.header.views.actions`), and the plugin's own row config
|
|
229
|
+
page (`plugins.row.config`) for scoring settings.
|
|
230
|
+
- Removed as duplicates: the plugin detail-page section
|
|
231
|
+
(`plugins.detail.section`) and the always-on sidebar button; the sidebar
|
|
232
|
+
button and the popup now come back together only with
|
|
233
|
+
`window.__SEV_ENABLE_POPUP__ = true` (debug surface).
|
|
234
|
+
- `docs/ENTRYPOINTS.md` (bilingual) is the single reference for where things are;
|
|
235
|
+
the READMEs point at it instead of repeating the tables.
|
|
236
|
+
|
|
237
|
+
## 1.15.4
|
|
238
|
+
|
|
239
|
+
Hot reload now actually works on this machine — documented and verified.
|
|
240
|
+
|
|
241
|
+
- The desktop profile patch gained an `hmr` entry with
|
|
242
|
+
`root: ["/Users/adams/Documents/dsh-plugins/dsh-skill-eval"]`, so Cordis HMR
|
|
243
|
+
watches the plugin source directory. A source edit (or `touch`) replaces the
|
|
244
|
+
running plugin generation **without a restart**; the versioned route changes
|
|
245
|
+
from `text/html` to `application/json` immediately.
|
|
246
|
+
- Verified end to end on the live host: 1.15.2 → 1.15.3 → 1.15.4 all rolled
|
|
247
|
+
over through HMR, `dsh-skill-eval-verify` reached 10/10 PASS (including the
|
|
248
|
+
first per-turn board), native client seats became active
|
|
249
|
+
(`plugins.row.config` key `dsh-skill-eval#skill-eval`,
|
|
250
|
+
`conversation.session.header.views.actions` id `skill-eval-board`,
|
|
251
|
+
`sidebar.footer.action`, `plugins.detail.section`, `conversation.chat.turnTail`),
|
|
252
|
+
and a live settings save (35/25/25/15) moved the report's `weightsPercent`
|
|
253
|
+
immediately.
|
|
254
|
+
- Two caveats recorded in docs: the **canonical** `/skill-eval/data` and
|
|
255
|
+
`/skill-eval/settings` routes can still be served by an older leaked instance
|
|
256
|
+
after a reload — use the versioned routes (what the client and verify do); and
|
|
257
|
+
a page refresh is only needed if the client HMR receiver did not swap the
|
|
258
|
+
bundle.
|
|
259
|
+
- `docs/DEV-RELOAD.md` rewritten around this recipe.
|
|
260
|
+
|
|
261
|
+
## 1.15.3
|
|
262
|
+
|
|
263
|
+
Hot reload works when the plugin is installed as a **tarball** into the profile,
|
|
264
|
+
not as a `link:` dependency.
|
|
265
|
+
|
|
266
|
+
- Root cause of "I edited the code and the live host kept running the old
|
|
267
|
+
module": with a linked package the loaded files live outside the profile
|
|
268
|
+
directory, and Cordis HMR only replaces plugins whose loaded files below the
|
|
269
|
+
profile changed. Installing `npm pack` output (`file:…/dist/<name>-<v>.tgz`)
|
|
270
|
+
puts a real copy under `profile/node_modules`, so a Plugin Manager
|
|
271
|
+
recomposition swaps it in place — verified by bumping the version and watching
|
|
272
|
+
`/skill-eval/data/<new version>` turn from `text/html` into JSON with no
|
|
273
|
+
restart.
|
|
274
|
+
- `dist/` holds the installable tarball; `docs/DEV-RELOAD.md` documents the
|
|
275
|
+
pack → `dsh plugin add` → recompose loop, and the link-install caveat.
|
|
276
|
+
|
|
277
|
+
## 1.15.2
|
|
278
|
+
|
|
279
|
+
Packaging doctor: `dsh-skill-eval-doctor` (or `npm run doctor`).
|
|
280
|
+
|
|
281
|
+
- Twelve pre-flight checks for the failures that only show up as "the plugin
|
|
282
|
+
silently does not appear after a restart": manifest name/version/exports,
|
|
283
|
+
`dsh.engines`, bundle patch presence, client platform, the patch row id, the
|
|
284
|
+
native settings key the row implies (`<package>#<row>` must literally appear in
|
|
285
|
+
the client), every relative import resolving across host/client/lib, locale
|
|
286
|
+
dictionary parity, and the client's `DATA_VERSION` matching package.json.
|
|
287
|
+
- Runs in milliseconds and exits non-zero on any drift, so it can guard a release
|
|
288
|
+
or precede a restart.
|
|
289
|
+
|
|
290
|
+
## 1.15.1
|
|
291
|
+
|
|
292
|
+
One-command post-restart verification: `dsh-skill-eval-verify` (or
|
|
293
|
+
`npm run verify`).
|
|
294
|
+
|
|
295
|
+
- Checks, in order: the versioned data route returns JSON with all report keys;
|
|
296
|
+
the settings API returns 24 fields and effective values; the board HTML carries
|
|
297
|
+
the next-steps / evidence / ledger sections; the settings page renders its
|
|
298
|
+
fields; the latest snapshot has skills and runs; per-turn boards appear once a
|
|
299
|
+
turn has settled; canonical and versioned routes agree and weights sum to 100;
|
|
300
|
+
plus two manual rows for the native seats the CLI cannot inspect.
|
|
301
|
+
- The "no restart yet" case is reported the way it actually looks: versioned
|
|
302
|
+
route answers `text/html`, so `versioned-json` fails and the summary says to
|
|
303
|
+
restart — no false green.
|
|
304
|
+
- Verified against a disposable profile running the same package: 10/10 PASS,
|
|
305
|
+
including `report-contract` (canonical=versioned) and the settings API.
|
|
306
|
+
|
|
307
|
+
## 1.15.0
|
|
308
|
+
|
|
309
|
+
One board per execution.
|
|
310
|
+
|
|
311
|
+
- **Per-turn boards.** Every settled turn that loaded a skill now writes its own
|
|
312
|
+
pair under `reports/turns/<session>-t<turn>.{md,json}` (in addition to
|
|
313
|
+
`latest.*`), with retention `turnReportsKeep` (default 50) and a switch
|
|
314
|
+
`turnReports` (default on). This is the literal "每次执行后结果在单独的看板":
|
|
315
|
+
each run keeps its own snapshot instead of overwriting the previous one.
|
|
316
|
+
- **`skill_eval_report {view:"turns"}` / `/skill-eval turns`** lists those
|
|
317
|
+
boards newest first (time, session, turn, skills, size, file) in markdown or
|
|
318
|
+
JSON, so an agent can open the exact board for one run.
|
|
319
|
+
- Both surfaces cost zero model tokens; the writer is still debounced and
|
|
320
|
+
failure-tolerant (a broken turn board never blocks the latest snapshot).
|
|
321
|
+
|
|
322
|
+
## 1.14.0
|
|
323
|
+
|
|
324
|
+
MCP debugging got per-tool latency, not just per-server.
|
|
325
|
+
|
|
326
|
+
- **Per-tool samples.** Every `mcp__<server>__<tool>` call now records duration into
|
|
327
|
+
the window (bounded at 20 samples per tool, merged across windows), so a tool
|
|
328
|
+
that is fine on average but terrible at the tail shows up.
|
|
329
|
+
- **Report rows** carry `avgMs / p50Ms / p95Ms` per tool next to calls/errors.
|
|
330
|
+
- **`mcp-tool-slow` warning** (`mcpToolP95Ms`, default 5000, needs ≥2 calls):
|
|
331
|
+
`MCP tool "context7/query-docs" has a slow tail — p50 4000ms · p95 9000ms`.
|
|
332
|
+
- **`/skill-eval mcp` markdown** gained a _Slowest tools (p95)_ section and shows
|
|
333
|
+
p95 next to error counts for failing tools.
|
|
334
|
+
- Aggregation was carrying tools only as calls/errors; it now carries samples and
|
|
335
|
+
ms through `aggregate`, so the numbers survive every merge step.
|
|
336
|
+
|
|
337
|
+
## 1.13.0
|
|
338
|
+
|
|
339
|
+
Developer diagnostics — the "is this thing actually working" view.
|
|
340
|
+
|
|
341
|
+
- **`skill_eval_report {view:"dev"}` / `/skill-eval dev`** reports the module
|
|
342
|
+
generation loaded versus the version on disk (the ESM-cache restart trap:
|
|
343
|
+
`module 1.10.1 vs disk 1.13.0 — restart DeepSeek Harness`), the data directory
|
|
344
|
+
and settings override file, store health (records, type counts, last-event
|
|
345
|
+
age, failed writes, queue), the routes this generation actually registered,
|
|
346
|
+
registered tools, inspect-provider state, recent host warnings, and findings
|
|
347
|
+
with severities.
|
|
348
|
+
- Route/tool/inspect registration and warnings are now recorded in a diagnostics
|
|
349
|
+
registry at apply time, so the view reflects what really mounted instead of
|
|
350
|
+
what the code intended.
|
|
351
|
+
- This is the "developer mode" half of the objective: one command answers why an
|
|
352
|
+
edit is not visible and which generation is live.
|
|
353
|
+
|
|
354
|
+
## 1.12.2
|
|
355
|
+
|
|
356
|
+
- **Graceful degradation on an older host.** The native settings page detects a
|
|
357
|
+
non-JSON answer from `/settings` (the stale prefix route of a previous
|
|
358
|
+
generation) and explains that a restart is needed, with a link to the web
|
|
359
|
+
settings page instead of showing an empty form.
|
|
360
|
+
- **Real-payload compatibility fixture.** `test/fixtures/live-1.10.1.json` is an
|
|
361
|
+
actual payload captured from the running 1.10.x host (turnLoads present,
|
|
362
|
+
`edits`/`validity` absent); the new client is tested against it so a newer UI
|
|
363
|
+
never breaks on an old host.
|
|
364
|
+
- **docs/DEV-RELOAD.md** — why edits do not appear until a restart (ESM module
|
|
365
|
+
cache + boot client scan), how to probe the live generation, the debug
|
|
366
|
+
switches, and the disposable-profile verification recipe that produced the
|
|
367
|
+
17-check PASS evidence.
|
|
368
|
+
|
|
369
|
+
## 1.12.1
|
|
370
|
+
|
|
371
|
+
- Settings weight fields use the same 0-100 scale the profile config uses
|
|
372
|
+
(step 5) instead of 0-1, so the native page can save 30/35/... directly.
|
|
373
|
+
Both scales still normalise at scoring time.
|
|
374
|
+
- Acceptance CLI now passes the settings-API and config-page probes to the
|
|
375
|
+
runner and resolves them under the dashboard base path (configurable with
|
|
376
|
+
`DSH_SKILL_EVAL_BASE`), so `dsh-skill-eval-accept` covers 17 checks.
|
|
377
|
+
- Verified end-to-end on a disposable profile: 1.12.0 host routes
|
|
378
|
+
(`/data`, `/settings`, `/config`) all answer, settings POST saves +
|
|
379
|
+
normalises (35/25/25/15 → 0.35/0.25/0.25/0.15), invalid values return 400,
|
|
380
|
+
and acceptance is fully green there.
|
|
381
|
+
|
|
382
|
+
## 1.12.0
|
|
383
|
+
|
|
384
|
+
Settings moved into DSH's own settings, and the board into the More menu.
|
|
385
|
+
|
|
386
|
+
- **Native plugin settings page.** The bundle patch already declares row
|
|
387
|
+
`skill-eval`, so the plugin now registers `plugins.row.config` keyed
|
|
388
|
+
`dsh-skill-eval#skill-eval`: Settings → Plugins → this plugin's row gains the
|
|
389
|
+
standard configure control, and the page renders the same tunables natively
|
|
390
|
+
(grouped weights/thresholds, cost, advice lifecycle, MCP, output) with a
|
|
391
|
+
Save that applies immediately and persists to `settings.json`.
|
|
392
|
+
- **Settings JSON API** (`<base>/settings` plus a versioned alias): GET returns
|
|
393
|
+
field definitions + effective values; POST accepts JSON or form bodies, then
|
|
394
|
+
validates, saves and applies. Backs the native page; the HTML page keeps
|
|
395
|
+
working.
|
|
396
|
+
- **Board in the More menu.** Registers
|
|
397
|
+
`conversation.session.header.views.actions`, so the session header View row's
|
|
398
|
+
**More** menu gains a `技能评测看板` row that opens the board and dismisses the
|
|
399
|
+
menu.
|
|
400
|
+
- Acceptance gained a `settings-api` check (16 in-process checks).
|
|
401
|
+
|
|
402
|
+
## 1.11.0
|
|
403
|
+
|
|
404
|
+
Native DSH entry points instead of only a URL.
|
|
405
|
+
|
|
406
|
+
- **Sidebar entry (always on).** A small gauge button beside Settings now opens
|
|
407
|
+
the reporting board in a new tab (with `window.__SEV_ENABLE_POPUP__ = true` it
|
|
408
|
+
still opens the modal instead). Previously this required the popup opt-in.
|
|
409
|
+
- **Plugin detail page section.** Under Settings → Plugins → `dsh-skill-eval`
|
|
410
|
+
the detail page now carries a section: how many skills have telemetry, how
|
|
411
|
+
many have an actionable fix, and buttons for _Open board_ / _Scoring
|
|
412
|
+
settings_. It renders null for every other plugin's subject.
|
|
413
|
+
- Registered seats now: `conversation.chat.turnTail` (turn strip),
|
|
414
|
+
`sidebar.footer.action`, `plugins.detail.section`, and `shell.overlay` when the
|
|
415
|
+
popup is enabled.
|
|
416
|
+
|
|
417
|
+
## 1.10.1
|
|
418
|
+
|
|
419
|
+
- Hot-reload fix: the dashboard page and the settings page are registered as
|
|
420
|
+
**exact** routes (exact wins over any prefix), so a previous generation still
|
|
421
|
+
owning the `/skill-eval` prefix can no longer shadow them. The prefix route
|
|
422
|
+
stays as a best-effort fallback for other subpaths.
|
|
423
|
+
|
|
424
|
+
## 1.10.0
|
|
425
|
+
|
|
426
|
+
The reporting board became a real board, and it got a settings page.
|
|
427
|
+
|
|
428
|
+
- **Dashboard** (`<dashboardPath>`, e.g. `/skill-eval`) now leads with the
|
|
429
|
+
reporting material instead of only skill cards: **Next steps** ranked by
|
|
430
|
+
expected gain / effort (same ranking as `loop all`), **What the evidence
|
|
431
|
+
supports** (validity: coverage, ranking significance, can/cannot-say lists),
|
|
432
|
+
**Screened edits** (the ledger), then skill cards, MCP, MCP fixes and runs.
|
|
433
|
+
- **Configuration page** (`<dashboardPath>/config`): every tunable in one form
|
|
434
|
+
(weights, grade thresholds, samples, advice, token budget, cache weight,
|
|
435
|
+
prune, edit budget, exploration window, MCP thresholds, output toggles).
|
|
436
|
+
Saving validates bounds, writes `settings.json` into the data directory,
|
|
437
|
+
applies immediately in memory and wins over the profile config on reload.
|
|
438
|
+
Invalid submissions return the page with per-field errors and change nothing.
|
|
439
|
+
- `/skill-eval config` prints the effective values and the override file path.
|
|
440
|
+
- Acceptance gained a `config-page` check (now 15 in-process checks).
|
|
441
|
+
- Dashboard header links to the settings page; settings page links back.
|
|
442
|
+
|
|
443
|
+
## 1.9.0
|
|
444
|
+
|
|
445
|
+
Two time-aware answers, plus a written scenario matrix.
|
|
446
|
+
|
|
447
|
+
- **`view:"history"` / `/skill-eval history <skill>`** — chronological load
|
|
448
|
+
table (ok, calls, errors, corrections, latency, turn) and a half-over-half
|
|
449
|
+
trend (`improving` / `degrading` / `flat` / `not-enough-data`) with the
|
|
450
|
+
problem-rate change spelled out.
|
|
451
|
+
- **`view:"impact"` / `/skill-eval impact <skill>`** — pairs every screened edit
|
|
452
|
+
with the loads recorded before and after it and judges the error-rate delta:
|
|
453
|
+
only ≥2 closed loads per side and >15pp counts as improved/degraded, anything
|
|
454
|
+
else is `inconclusive` with the reason. This is the "did my edit help?" loop
|
|
455
|
+
closing, and it starts working automatically as samples accumulate.
|
|
456
|
+
- **docs/SCENARIOS.md** — 25+ scenarios (single skill, cross-skill, edit
|
|
457
|
+
governance, MCP, runs, config) each mapped to its entry point, expected answer,
|
|
458
|
+
status, plus the six things the tool deliberately does not do.
|
|
459
|
+
|
|
460
|
+
## 1.8.0
|
|
461
|
+
|
|
462
|
+
- **Validity dashboard.** `skill_eval_report {view:"validity"}` (and
|
|
463
|
+
`/skill-eval validity`) reports only what the stored evidence supports:
|
|
464
|
+
coverage (skills, intervals, provisional, actionable), ranking significance,
|
|
465
|
+
advice severity counts, ledger composition and — the point — an explicit
|
|
466
|
+
**Cannot say yet** list (no interval at n < minSamples, before/after effect of
|
|
467
|
+
an edit, token cost of a skill the model never loaded). The same payload is in
|
|
468
|
+
the report JSON as `validity`.
|
|
469
|
+
- **Edit history in the turn strip.** Expanding a turn now shows the last three
|
|
470
|
+
ledger entries for that skill (verdict icon, dimension, hypothesis) next to
|
|
471
|
+
the fixes, so the conversation carries the evolution, not just the score.
|
|
472
|
+
- **Prompt hint** now teaches the safe order: report → `skill_eval_review` →
|
|
473
|
+
edit → re-measure, and points at `view:"loop"` / `view:"validity"`.
|
|
474
|
+
- `docs/WORKFLOW.md` gained the validity entry point.
|
|
475
|
+
|
|
476
|
+
## 1.7.0
|
|
477
|
+
|
|
478
|
+
- **Batch plans are ranked by expected gain per unit of effort** instead of raw
|
|
479
|
+
score: `impactOf()` weighs headroom (100 − score), severity (critical vs
|
|
480
|
+
warning), evidence strength (provisional samples count less) and the usual
|
|
481
|
+
cost of the fix dimension (static edit ≈ 1x, behaviour change ≈ 2.5x). The
|
|
482
|
+
batch view prints `priority N` with the reasons, so "fix this first" is
|
|
483
|
+
defensible rather than alphabetical.
|
|
484
|
+
- **docs/WORKFLOW.md**: the one-page operating manual — flow diagram, the three
|
|
485
|
+
roles (human / agent / plugin), when to use which entry point, the six
|
|
486
|
+
self-deception guards, common states and the honest-limits list. Linked from
|
|
487
|
+
both READMEs.
|
|
488
|
+
|
|
489
|
+
## 1.6.0
|
|
490
|
+
|
|
491
|
+
- **Batch loop.** `skill_eval_report {view:"loop", all:true}` and
|
|
492
|
+
`/skill-eval loop all` return one compact plan per skill that currently has an
|
|
493
|
+
actionable fix, worst score first.
|
|
494
|
+
- **Next step in the conversation strip.** The turn card now shows
|
|
495
|
+
`下一步 · 可改 <skill> · <dimension> · <fix>` when the turn's skill has a
|
|
496
|
+
critical/warning fix, so the strip points forward instead of only reporting
|
|
497
|
+
history. Locale strings added for both languages.
|
|
498
|
+
|
|
499
|
+
## 1.5.0
|
|
500
|
+
|
|
501
|
+
- **Guided optimization loop.** `/skill-eval loop [skill]` (and
|
|
502
|
+
`skill_eval_report {view:"loop"}`) turns the current report into the next
|
|
503
|
+
concrete step, RRSI style: pick the worst skill with actionable advice, cap it
|
|
504
|
+
at that skill's annealed `adviceBudget`, emit the chosen fixes, require a
|
|
505
|
+
`skill_eval_review` screening (leakage critic) before applying, honour the
|
|
506
|
+
exploration directive, then ask for a re-measurement round and warn that
|
|
507
|
+
point-estimate moves are not enough — the interval has to move.
|
|
508
|
+
- A quiet skill or an empty dataset explains itself instead of failing.
|
|
509
|
+
- Pairs with the ledger: accepted edits anneal the budget, same-dimension edits
|
|
510
|
+
trigger the exploration directive, so repeated `/skill-eval loop` calls walk
|
|
511
|
+
the RRSI loop.
|
|
512
|
+
|
|
513
|
+
## 1.4.0
|
|
514
|
+
|
|
515
|
+
RRSI P2/P3 plus the two static-check gaps recorded in `docs/COMPARISON.md`.
|
|
516
|
+
|
|
517
|
+
- **Annealed edit budget (P2).** Each skill carries `adviceBudget`: it starts at
|
|
518
|
+
`editBudgetMax` (4), anneals toward `editBudgetMin` (1) over
|
|
519
|
+
`editBudgetHorizon` (6) _accepted_ ledger edits using the RRSI cosine schedule,
|
|
520
|
+
and caps how many fixes the report presents as actionable this round.
|
|
521
|
+
- **Exploration directive (P3).** When the last `noveltyWindow` (2) reviewed
|
|
522
|
+
edits for a skill all targeted one dimension, the report emits
|
|
523
|
+
`exploration: {stuck, suggestion, reason}` pointing at the least-used
|
|
524
|
+
untouched dimension; `/skill-eval review` now accepts a `dimension`.
|
|
525
|
+
- **Reference integrity.** The static profile gains `reference-integrity`
|
|
526
|
+
(weight 8): every `references/…`, `scripts/…`, `assets/…` path in the body is
|
|
527
|
+
checked against the skill directory at load time (best-effort; the check is
|
|
528
|
+
skipped rather than faked when no filesystem view exists). Missing files raise
|
|
529
|
+
a critical `broken-references` rule.
|
|
530
|
+
- **Anti-trigger check.** `anti-trigger` (weight 5) scores whether the skill
|
|
531
|
+
says when _not_ to use it; absent → info-level advice.
|
|
532
|
+
- **Skill-name check.** `skill-name` (weight 4) enforces kebab-case ids per the
|
|
533
|
+
agentskills.io contract; a violation is a warning.
|
|
534
|
+
- New config: `editBudgetMax`, `editBudgetMin`, `editBudgetHorizon`,
|
|
535
|
+
`noveltyWindow`.
|
|
536
|
+
|
|
537
|
+
## 1.3.0
|
|
538
|
+
|
|
539
|
+
RRSI P1 mechanisms (see `docs/RRSI-NOTES.md` §6.1).
|
|
540
|
+
|
|
541
|
+
- **Leakage critic.** New `skill_eval_review` tool screens a proposed skill edit
|
|
542
|
+
before it is applied: hardcoded answers/expected outputs, named benchmarks,
|
|
543
|
+
task ids, machine-specific paths, date windows and input-shape special-casing
|
|
544
|
+
are classified `reject`/`warn`/`accept`, with a required falsifiable
|
|
545
|
+
hypothesis. `lib/critic.js` is pure and tested.
|
|
546
|
+
- **Audit ledger.** Every review appends an edit record (skill, hypothesis,
|
|
547
|
+
verdict, reason codes, diff hash) to the telemetry store. The report exposes
|
|
548
|
+
`edits`, `skill_eval_report` returns it, and the Cordis Inspect provider grew
|
|
549
|
+
an `edits` method — so a fix is traceable to the hypothesis it was meant to
|
|
550
|
+
test.
|
|
551
|
+
- **Pruning.** A well-sampled skill (`pruneMinLoads`, default 6) whose score is
|
|
552
|
+
below `pruneMaxScore` (60) and whose 95% interval also sits low raises a
|
|
553
|
+
`prune-candidate` critical rule: delete it or merge its one useful step into a
|
|
554
|
+
skill that gets used.
|
|
555
|
+
- New config: `registerReviewTool`, `pruneMinLoads`, `pruneMaxScore`.
|
|
556
|
+
|
|
557
|
+
## 1.2.0
|
|
558
|
+
|
|
559
|
+
Two RRSI-derived P0 mechanisms (see `docs/RRSI-NOTES.md` §6.1).
|
|
560
|
+
|
|
561
|
+
- **Noise floor on advice.** The five rate-based rules (post-load errors, user
|
|
562
|
+
corrections, ignored loads, trigger collisions, loop errors — plus MCP rules
|
|
563
|
+
already thresholded) now bootstrap their window-level rate and only keep
|
|
564
|
+
`critical`/`warning` when the 95% interval's low end clears the threshold.
|
|
565
|
+
Signals that are above the mean but still inside noise are reported as `info`
|
|
566
|
+
with the interval in the evidence ("not outside noise yet"). `adviceNoiseFloor:
|
|
567
|
+
false` restores the old behaviour.
|
|
568
|
+
- **Measured token cost.** Real provider usage from every committed
|
|
569
|
+
`assistant/message` is attributed to the open skill windows
|
|
570
|
+
(`inputTokens/outputTokens/cacheReadTokens`), aggregated into
|
|
571
|
+
`stats.tokensPerLoad` (cached input weighted by `cacheReadWeight`), and used
|
|
572
|
+
for: the efficiency dimension (over `tokenBudgetPerLoad` costs up to 35 points),
|
|
573
|
+
a new `token-cost` advice rule, the Runs board totals and the in-conversation
|
|
574
|
+
card ("≈N tok/load"). `countTokens: false` disables capture. Per-load cost divides only by windows that carried
|
|
575
|
+
real usage; cached input defaults to weight 0.
|
|
576
|
+
- New config: `tokenBudgetPerLoad` (20 000), `cacheReadWeight` (0.1),
|
|
577
|
+
`countTokens` (true), `adviceNoiseFloor` (true).
|
|
578
|
+
- Token attribution is tested per window, including cache reads and ignoring
|
|
579
|
+
other sessions.
|
|
580
|
+
|
|
581
|
+
## 1.1.0
|
|
582
|
+
|
|
583
|
+
- **Results now land in the conversation.** A completed turn that loaded a skill
|
|
584
|
+
gets a result strip above its action row: skill scores, this turn's call and
|
|
585
|
+
error counts, and an expandable fix list. It reads durable per-turn loads from
|
|
586
|
+
`turnLoads`, costs zero model tokens and needs no popup.
|
|
587
|
+
- The popup (sidebar button + modal) is now **opt-in**: set
|
|
588
|
+
`window.__SEV_ENABLE_POPUP__ = true` to bring it back. The JSON/API/html
|
|
589
|
+
surfaces are unchanged and still back the in-conversation card.
|
|
590
|
+
- Skill loads record the loop `turn` that produced them; the report exposes
|
|
591
|
+
`turnLoads` grouped per session+turn.
|
|
592
|
+
- New Claude-Code-pattern test corpus (`test/corpus/`): document-style,
|
|
593
|
+
MCP-dependent, anti-trigger, overlong-monolith and weak-trigger skills, each
|
|
594
|
+
with an asserted verdict; `docs/COMPARISON.md` records the remaining gaps
|
|
595
|
+
(anti-trigger text, frontmatter integrity, reference link integrity, script
|
|
596
|
+
correctness, optional metadata).
|
|
597
|
+
- Repository moved to `~/Documents/dsh-plugins/dsh-skill-eval`; the profile
|
|
598
|
+
dependency and symlink were re-pointed.
|
|
599
|
+
- Acceptance tooling no longer hardcodes a GUI port: `scripts/accept.mjs`
|
|
600
|
+
resolves the base URL from the argument, then `DSH_WEB_URL`, then
|
|
601
|
+
`~/.dsh/remote-web-ui-relay-bridge.json` (`upstream`), and reports
|
|
602
|
+
**UNREACHABLE** separately from **stale generation** so a wrong port is not
|
|
603
|
+
mistaken for an old module.
|
|
604
|
+
- The manual acceptance row now points at the in-conversation strip instead of
|
|
605
|
+
the opt-in popup.
|
|
606
|
+
- Live MCP capture was verified against the installed servers (context7 call
|
|
607
|
+
bucketed with latency, health and per-tool breakdown).
|
|
608
|
+
|
|
609
|
+
## 1.0.0
|
|
610
|
+
|
|
611
|
+
First stable release. Everything below was built and verified inside one session
|
|
612
|
+
against DeepSeek Harness 0.2.2-rc.3; the acceptance report lives in
|
|
613
|
+
[docs/ACCEPTANCE.md](docs/ACCEPTANCE.md).
|
|
614
|
+
|
|
615
|
+
### Recording
|
|
616
|
+
|
|
617
|
+
- Records every `skill` tool load with duration, success/failure and session id.
|
|
618
|
+
- Records user-typed `/skill-name` loads by reading `source.kind ===
|
|
619
|
+
'skill-invocation'` messages on `agent/pre-step`, deduped by message id.
|
|
620
|
+
- Observation window per load: tool calls, tool errors, MCP calls, loop errors,
|
|
621
|
+
retried model requests, user corrections, sibling skill loads.
|
|
622
|
+
- Windows survive the turn boundary (the correction that proves a skill failed
|
|
623
|
+
usually arrives in the next user message) with a TTL sweep.
|
|
624
|
+
- Parallel loads inside one loop step stay open together instead of being scored
|
|
625
|
+
as a collision; a load in a later step is a real switch.
|
|
626
|
+
- Root vs subagent origin is recorded so short subagent runs do not count as
|
|
627
|
+
"ignored skill".
|
|
628
|
+
- MCP tools are bucketed per `mcp__<server>__<tool>` with latency samples.
|
|
629
|
+
|
|
630
|
+
### Scoring
|
|
631
|
+
|
|
632
|
+
- Skill score: static 30% / reliability 25% / effectiveness 30% / efficiency 15%,
|
|
633
|
+
A–F bands, provisional + confidence labels.
|
|
634
|
+
- Static authoring checks (9, weight-summing 100): description, trigger
|
|
635
|
+
language, whenToUse, size, sections, progressive disclosure, placeholders,
|
|
636
|
+
examples.
|
|
637
|
+
- Runtime dimensions from load success, post-load tool error rate, loop errors,
|
|
638
|
+
user corrections, ignored loads, sibling collisions, cost.
|
|
639
|
+
- 95% bootstrap confidence interval per skill (seeded, reproducible) and per run.
|
|
640
|
+
- Ranking caveat when two skills' intervals overlap closely.
|
|
641
|
+
- Run score with its own dimensions (quality 0.40 / stability 0.30 / outcome
|
|
642
|
+
0.20 / efficiency 0.10), interval, confidence, provisional.
|
|
643
|
+
- MCP health per server (`100 − 70·errorRate − slow penalty`) with p50/p95,
|
|
644
|
+
failing tools and MCP-specific fixes.
|
|
645
|
+
- Standard is user configuration: `weights`, `thresholds`, `minSamples`,
|
|
646
|
+
`ciResamples`, `mcpErrorRatePercent`, `mcpToolErrorRatePercent`, `mcpP95Ms`.
|
|
647
|
+
Weights normalise, a zero weight drops the dimension, unordered thresholds are
|
|
648
|
+
repaired, all-zero falls back to defaults.
|
|
649
|
+
|
|
650
|
+
### Surfaces
|
|
651
|
+
|
|
652
|
+
- `skill_eval_report` tool (`skill`, `view: summary|mcp|runs|standard`, `format`).
|
|
653
|
+
- `/skill-eval` command (`<skill>`, `mcp`, `runs`, `standard`, `accept`).
|
|
654
|
+
- Sidebar footer action + modal overlay with 4 tabs (Skills / Runs / MCP / Dev),
|
|
655
|
+
5 s auto-refresh, copy-as-markdown, Escape/backdrop close.
|
|
656
|
+
- HTML dashboard `GET /skill-eval` and JSON API `GET /skill-eval/data`
|
|
657
|
+
(plus a versioned alias).
|
|
658
|
+
- Per-turn snapshot `reports/latest.md` + `latest.json` (no model tokens).
|
|
659
|
+
- Cordis Inspect provider `skill-eval` (`summary|skill|mcp|runs|standard`).
|
|
660
|
+
|
|
661
|
+
### Reliability and diagnostics
|
|
662
|
+
|
|
663
|
+
- Every listener is failure-contained; the two waterfalls always call `next()`.
|
|
664
|
+
- Telemetry never touches the durable session log; it lives in
|
|
665
|
+
`$DSH_HOME/skill-eval/<profile>/events.jsonl`.
|
|
666
|
+
- `dispose()` flushes queued writes (a real data-loss bug found by tests).
|
|
667
|
+
- `node scripts/accept.mjs` and `/skill-eval accept` detect a stale host
|
|
668
|
+
generation; the popup shows a red banner when its payload predates the boards.
|
|
669
|
+
- `DSH_SKILL_EVAL_DEBUG=1` mirrors diagnostics to stderr.
|
|
670
|
+
|
|
671
|
+
### Tests
|
|
672
|
+
|
|
673
|
+
118 tests, no build step: unit, integration through a fake host, jsdom rendering
|
|
674
|
+
of the real popup components, snapshot/acceptance, and seeded property tests over
|
|
675
|
+
the scoring pipeline.
|