dsh-skill-eval 0.0.0-stage → 1.20.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (48) hide show
  1. package/CHANGELOG.md +675 -0
  2. package/LICENSE +21 -0
  3. package/README.md +200 -2
  4. package/README.zh.md +157 -0
  5. package/client.js +2637 -0
  6. package/cordis.patch.yml +9 -0
  7. package/docs/ACCEPTANCE.md +71 -0
  8. package/docs/AUDIT.md +64 -0
  9. package/docs/CODE-REVIEW.md +49 -0
  10. package/docs/COMPARISON.md +33 -0
  11. package/docs/DETAIL.md +463 -0
  12. package/docs/DEV-RELOAD.md +114 -0
  13. package/docs/ENTRYPOINTS.md +65 -0
  14. package/docs/RELEASE.md +44 -0
  15. package/docs/RRSI-NOTES.md +129 -0
  16. package/docs/SCENARIOS.md +91 -0
  17. package/docs/UI-REVIEW.md +115 -0
  18. package/docs/WORKFLOW.md +108 -0
  19. package/icon.svg +9 -0
  20. package/index.js +1111 -0
  21. package/lib/accept.js +350 -0
  22. package/lib/advice.js +547 -0
  23. package/lib/critic.js +101 -0
  24. package/lib/dev.js +148 -0
  25. package/lib/history.js +140 -0
  26. package/lib/html.js +8 -0
  27. package/lib/inspect.js +142 -0
  28. package/lib/integrity.js +46 -0
  29. package/lib/loop.js +195 -0
  30. package/lib/random.js +29 -0
  31. package/lib/recorder.js +634 -0
  32. package/lib/render.js +582 -0
  33. package/lib/report.js +646 -0
  34. package/lib/score.js +525 -0
  35. package/lib/settings.js +396 -0
  36. package/lib/snapshot.js +166 -0
  37. package/lib/steps.js +141 -0
  38. package/lib/store.js +186 -0
  39. package/lib/validity.js +115 -0
  40. package/locale/en.json +6 -0
  41. package/locale/zh.json +6 -0
  42. package/package.json +96 -4
  43. package/scripts/accept.mjs +226 -0
  44. package/scripts/build-client.mjs +59 -0
  45. package/scripts/build.mjs +81 -0
  46. package/scripts/doctor.mjs +199 -0
  47. package/scripts/lib.mjs +52 -0
  48. package/scripts/verify-live.mjs +168 -0
package/CHANGELOG.md ADDED
@@ -0,0 +1,675 @@
1
+ # Changelog
2
+
3
+ ## 1.20.2 — MCP board and settings readability
4
+
5
+ - **MCP tab rebuilt.** The old tab was one SVG chart with two thin bars plus a
6
+ bare grid: the value label collided with the threshold line, the last table
7
+ column was clipped ("2606/260…"), and the table had no container. It is now a
8
+ latency card with CSS meters (p50 solid, p95 tinted, dashed threshold marker
9
+ on the track), a carded server table with a grade chip, and carded advice /
10
+ failing-tool sections. Latencies render as `2.6s` / `180ms` so nothing is
11
+ truncated. With no recorded calls the tab is a single empty card instead of a
12
+ chart card plus an empty table that read as two broken panels.
13
+ - **Settings readability.** Weight fields show a `%` unit, grade-floor help no
14
+ longer repeats the label ("A 级下限 / A 级下限。"), units are localized in the
15
+ Chinese form and web page (`loads → 次加载`, `edits → 次修改`), the config
16
+ dialog is titled 「配置」 with the field summary as its subtitle, and the web
17
+ page keeps readable headings in dark mode.
18
+ - **CSS collision fixed.** The advice container reused `.sev-sub`, which also
19
+ styles the panel subtitle, so the later rule silently restyled the header.
20
+ Advice blocks now use `.sev-advice-list` / `.sev-card`.
21
+
22
+ ## 1.20.1 — board honesty pass
23
+
24
+ - **The 「配置」 button no longer relies on `window.open`.** The desktop shell
25
+ denies every popup (`setWindowOpenHandler` returns `deny`; only `https:` URLs
26
+ are handed to the browser), so on an `http://127.0.0.1` board the button did
27
+ nothing at all. It now opens the same settings form in an in-panel dialog,
28
+ with Escape/backdrop close and the web-config link hidden where the shell
29
+ would deny it. The web page and the native bundle seat still exist.
30
+ - **The dev tab reports real diagnostics.** The data route now carries a `dev`
31
+ block (loaded module vs package on disk with the restart-skew flag, claimed
32
+ route/tool counts, settings file, warning count) and the tab renders it next
33
+ to the effective scoring standard; the data endpoint shown is the versioned
34
+ URL the client actually calls, not the bare prefix.
35
+ - **The MCP latency chart follows the configured threshold.** `mcpP95Ms` now
36
+ ships with the report, so the alert line can no longer disagree with the MCP
37
+ advice that uses the same number (3000 ms remains the older-payload fallback).
38
+
39
+ ## 1.20.0 — per-step trajectory
40
+
41
+ - **Per-step loop telemetry:** the recorder persists one row per agent step
42
+ (turn, step, tool calls/errors, MCP calls, loop errors, tokens, duration,
43
+ loaded skills). `lib/steps.js` scores each step 0–100 and groups turns.
44
+ - **New board tab 「轨迹 / Steps」:** step KPIs, recent-turn comparison with
45
+ deltas, per-turn step list with score chips, and a skill filter. Each skill
46
+ card gets a「查看轨迹」button that opens the tab filtered to that skill.
47
+ - Step records are additive event-log rows; old telemetry stays readable.
48
+ - **Multi-skill turn strip:** each skill loaded in a turn keeps its own chip
49
+ and now its own "next fix" (up to three); previously only the first global
50
+ suggestion was shown. Covered by a two-skill regression test.
51
+ - **npm bin targets normalized** (`scripts/*.mjs` without `./`): npm 11 was
52
+ stripping all three bins at publish time with an "invalid and removed"
53
+ warning, so a published install would have lost the CLI helpers.
54
+ - **One-command build:** `npm run build` (format/lint/client/doctor/tests),
55
+ `npm run build:pack` (adds tarball + release doctor), `npm run build:install`
56
+ (also installs into the desktop profile), `npm run build:fast` (client +
57
+ doctor only). Implemented in `scripts/build.mjs`.
58
+ - Tests: **239 passing** (new step scoring/recorder/report/client-filter suites).
59
+
60
+ ## 1.19.1 — test hygiene
61
+
62
+ - Client tests share one `createHost(window, style)` helper and top-level
63
+ `createRoot`/`act` imports instead of 18 copies of the same setup.
64
+ - No runtime change; 232 tests still passing.
65
+
66
+ ## 1.19.0 — board charts + client parts + tooling
67
+
68
+ - **Client split with a zero-dependency builder.** `client.js` is now generated
69
+ from `client/*.js` by `scripts/build-client.mjs`; `prepack`/`pretest` rebuild
70
+ it, doctor fails when the shipped file is stale, and `client/README.md`
71
+ documents the workflow. The package still ships one self-contained loader
72
+ factory, so installation needs no build.
73
+ - **Terminal scripts share plumbing.** `scripts/lib.mjs` now owns bridge/base
74
+ discovery, dashboard base, expected version and timeout HTTP GET; accept and
75
+ verify use it (tests cover precedence and error messages).
76
+ - **Formatting/linting.** Prettier (config + `format`/`format:check`) and ESLint
77
+ 10 flat config (`lint`) added as dev dependencies; all source files formatted.
78
+ - **Board charts (pure SVG, no dependency):** score distribution histogram with
79
+ average marker, per-skill trend sparkline, evidence-coverage donut, MCP
80
+ p50/p95 latency chart with the 3000 ms threshold, and session-activity bars.
81
+ Charts also appear on the Runs and MCP tabs; every chart carries an
82
+ `aria-label` with its data.
83
+ - **Tests:** 232 passing; lint/format/doctor/package gates re-run in 1.19.0.
84
+
85
+ ## 1.18.5 — hygiene follow-up
86
+
87
+ - **Locale metadata restored.** DSH reads `locale/<lang>.json` for the plugin's
88
+ localized market title/description; 1.18.4 wrongly deleted it. The files are
89
+ back, trimmed to `meta.title` + `meta.description` only (the old 16 unused UI
90
+ keys were the actual drift), and the doctor now checks both locale metas.
91
+ - **Settings group metadata now comes from the host.** Every field ships
92
+ `group`/`groupZh`; the client groups by it and keeps its local classifier only
93
+ as an older-payload fallback, removing a second copy of the grouping rules.
94
+ - Tests: **228 passing**.
95
+
96
+ ## 1.18.4 — code hygiene
97
+
98
+ Full review: [docs/CODE-REVIEW.md](docs/CODE-REVIEW.md).
99
+
100
+ - Removed dead `locale/*.json` (unused at runtime, already drifted from the
101
+ client dictionary), with exports/files/doctor/docs updated.
102
+ - Consolidated duplicated helpers: `lib/random.js` (FNV hash + mulberry32) and
103
+ `lib/html.js` (HTML escape) used by score/report/index/render/settings.
104
+ - Deleted dead TEXTS keys and CSS left by the removed More-menu/detail surfaces;
105
+ documented the real client seats in the file header; added the missing
106
+ `.sev-btn--ghost` rule.
107
+ - Restored serialized atomic settings writes (a regression from an earlier
108
+ refactor) and added a regression test; added a settings-group drift test.
109
+ - Added `.editorconfig` and `.gitignore`.
110
+ - Tests: **227 passing**.
111
+
112
+ ## 1.18.3 — settings aligned with the general settings layout
113
+
114
+ - Per-setting help lines (zh/en), units (loads / ms / tokens / edits), and
115
+ switch-style booleans for the 25 settings; section rows now use the same
116
+ label + description + control pattern as DSH's 通用设置.
117
+ - The web config page renders the same help text and units.
118
+ - 225 tests.
119
+
120
+ ## 1.18.2 — panel and settings polish
121
+
122
+ Reference: the context-insight panel (`dsh-context`) and the generic settings
123
+ layout.
124
+
125
+ - **Board panel**: KPI row (skills / loads / actionable / average score), a
126
+ search box, an "needs a fix" filter chip, a “Configure” shortcut next to
127
+ Refresh/Copy, and a last-updated stamp.
128
+ - **Native settings form**: a bilingual heading + subtitle, and the missing
129
+ `windowTtlMs` field (25 settings); labels/group titles follow the app locale.
130
+ - English TEXTS parity restored for every new string; 225 tests.
131
+
132
+ ## 1.18.1 — four-layer audit fixes
133
+
134
+ Full report: [docs/AUDIT.md](docs/AUDIT.md). Findings reproduced and fixed
135
+ across host data, scoring, HTTP, client and packaging.
136
+
137
+ - **Data integrity:** `__proto__` telemetry keys can no longer pollute
138
+ `Object.prototype` or crash reports; a torn final JSONL line is repaired
139
+ before the next append; cumulative window snapshots no longer double-count
140
+ run counters.
141
+ - **Correctness:** the bootstrap CI uses the same scoring options as the point
142
+ estimate; recorded static profiles are normalised; evicted windows are
143
+ censored from abandoned-run scoring; grade floors are sorted, not clamped.
144
+ - **Privacy/size:** skill bodies are no longer persisted or shipped in reports.
145
+ - **Performance:** report cache keyed by store identity + 2 s TTL; stable locale
146
+ subscriptions; memoized cards.
147
+ - **HTTP:** same-origin write gate + `x-skill-eval-config` header, atomic
148
+ serialized settings writes, merged-weight validation, byte-accurate 413,
149
+ versioned board/config pages render their own base.
150
+ - **Client:** request generation guard, focus trap/restore, clipboard cleanup,
151
+ settings-page styles, duplicate i18n key fixed, aria roles/live status,
152
+ English TEXTS parity.
153
+ - Settings: `windowTtlMs` (25 fields), bounds aligned with the Config schema.
154
+ - Tests: **225 passing**.
155
+
156
+ ## 1.17.5 — last mixed-language strings
157
+
158
+ - Ranking note and validity statements are bilingual (`noteZh`, `canSayZh`,
159
+ `cannotSayZh`); HTML board and client panel pick the Chinese copy.
160
+ - 212 tests at the time.
161
+
162
+ ## 1.17.4 — advice localization and README tightening
163
+
164
+ - Every advice rule carries `titleZh`/`fixZh` (MCP names filled in); localize
165
+ the whole report once for the HTML board; client cards/MCP list/turn strip
166
+ select the Chinese copy. README opens with a Chinese TL;DR; config key list
167
+ moved to `docs/DETAIL.md`; full config documented.
168
+
169
+ ## 1.17.3 — release candidate
170
+
171
+ - Copy button falls back to `execCommand('copy')` with failure feedback.
172
+ - Packaging: LICENSE, explicit `files`, three bins, release scripts and doctor
173
+ release gate; clean-profile smoke documented in `docs/RELEASE.md`.
174
+
175
+ ## 1.17.0
176
+
177
+ Board and settings moved to the same seats the context plugin (`dsh-context`)
178
+ uses, per the request to follow that pattern.
179
+
180
+ - **Board = sidebar panel + main panel.** Registers `sidebar.panellist`
181
+ (id `skill-eval`, order 40, gauge icon, localized label) and `main` keyed
182
+ `skill-eval`. Clicking the sidebar icon opens a full central board panel:
183
+ header with skill/actionable counts, tabs (skills / runs / MCP / dev),
184
+ refresh and copy, reusing the existing SkillCard/RunsView/McpView/DevView.
185
+ - **Settings = bundle config page.** Registers `plugins.bundle.config` keyed
186
+ `dsh-skill-eval` (the bundle's own page under Plugins), replacing the row
187
+ config seat; the native settings form is unchanged.
188
+ - Removed: the session-header More-menu board row and the row-config seat.
189
+ Kept: the in-conversation result strip, the versioned web board, the
190
+ debug-only popup (opt-in).
191
+ - Client tests rewritten for the new seat set (208 passing).
192
+
193
+ ## 1.16.1
194
+
195
+ - **Versioned page aliases.** `/skill-eval/board/<version>` and
196
+ `/skill-eval/board/<version>/config` are registered as exact routes each
197
+ release. Hot-reloaded generations cannot reclaim the canonical page route an
198
+ older generation still owns, which is why the bilingual board initially kept
199
+ serving the old page; the versioned alias is always the current one. The
200
+ client's board/config links and the verify/accept scripts use it (with a
201
+ canonical fallback for a clean start).
202
+ - Verified live: `?lang=zh` / `Accept-Language: zh-CN` return Chinese chrome on
203
+ `/skill-eval/board/1.16.1`, the settings page renders Chinese labels, and the
204
+ English default is unchanged.
205
+
206
+ ## 1.16.0
207
+
208
+ The board and the settings page are bilingual (中 / EN).
209
+
210
+ - Language resolution: explicit `?lang=zh|en` → `Accept-Language` header →
211
+ English fallback (pure `resolveLang`, unit-tested).
212
+ - The board translates its chrome: title, scoring line, totals, dimension names,
213
+ next steps, evidence (can / cannot say), screened edits, MCP and runs tables,
214
+ plus a `中文 / English` switch link.
215
+ - The settings page translates group titles, all 24 field labels, banners and
216
+ buttons; the switch keeps you on the page.
217
+ - Advice titles/evidence/fixes come from the host rules and stay in English for
218
+ now; the surrounding chrome is translated. Documented in `docs/ENTRYPOINTS.md`.
219
+ - 208 tests (new i18n suite + route-level language checks: header wins over
220
+ default, explicit query wins over header).
221
+
222
+ ## 1.15.5
223
+
224
+ Entry points consolidated to three human surfaces.
225
+
226
+ - Kept: the in-conversation result strip (`conversation.chat.turnTail`), one
227
+ **Skill eval board** row in the session header's **More** menu
228
+ (`conversation.session.header.views.actions`), and the plugin's own row config
229
+ page (`plugins.row.config`) for scoring settings.
230
+ - Removed as duplicates: the plugin detail-page section
231
+ (`plugins.detail.section`) and the always-on sidebar button; the sidebar
232
+ button and the popup now come back together only with
233
+ `window.__SEV_ENABLE_POPUP__ = true` (debug surface).
234
+ - `docs/ENTRYPOINTS.md` (bilingual) is the single reference for where things are;
235
+ the READMEs point at it instead of repeating the tables.
236
+
237
+ ## 1.15.4
238
+
239
+ Hot reload now actually works on this machine — documented and verified.
240
+
241
+ - The desktop profile patch gained an `hmr` entry with
242
+ `root: ["/Users/adams/Documents/dsh-plugins/dsh-skill-eval"]`, so Cordis HMR
243
+ watches the plugin source directory. A source edit (or `touch`) replaces the
244
+ running plugin generation **without a restart**; the versioned route changes
245
+ from `text/html` to `application/json` immediately.
246
+ - Verified end to end on the live host: 1.15.2 → 1.15.3 → 1.15.4 all rolled
247
+ over through HMR, `dsh-skill-eval-verify` reached 10/10 PASS (including the
248
+ first per-turn board), native client seats became active
249
+ (`plugins.row.config` key `dsh-skill-eval#skill-eval`,
250
+ `conversation.session.header.views.actions` id `skill-eval-board`,
251
+ `sidebar.footer.action`, `plugins.detail.section`, `conversation.chat.turnTail`),
252
+ and a live settings save (35/25/25/15) moved the report's `weightsPercent`
253
+ immediately.
254
+ - Two caveats recorded in docs: the **canonical** `/skill-eval/data` and
255
+ `/skill-eval/settings` routes can still be served by an older leaked instance
256
+ after a reload — use the versioned routes (what the client and verify do); and
257
+ a page refresh is only needed if the client HMR receiver did not swap the
258
+ bundle.
259
+ - `docs/DEV-RELOAD.md` rewritten around this recipe.
260
+
261
+ ## 1.15.3
262
+
263
+ Hot reload works when the plugin is installed as a **tarball** into the profile,
264
+ not as a `link:` dependency.
265
+
266
+ - Root cause of "I edited the code and the live host kept running the old
267
+ module": with a linked package the loaded files live outside the profile
268
+ directory, and Cordis HMR only replaces plugins whose loaded files below the
269
+ profile changed. Installing `npm pack` output (`file:…/dist/<name>-<v>.tgz`)
270
+ puts a real copy under `profile/node_modules`, so a Plugin Manager
271
+ recomposition swaps it in place — verified by bumping the version and watching
272
+ `/skill-eval/data/<new version>` turn from `text/html` into JSON with no
273
+ restart.
274
+ - `dist/` holds the installable tarball; `docs/DEV-RELOAD.md` documents the
275
+ pack → `dsh plugin add` → recompose loop, and the link-install caveat.
276
+
277
+ ## 1.15.2
278
+
279
+ Packaging doctor: `dsh-skill-eval-doctor` (or `npm run doctor`).
280
+
281
+ - Twelve pre-flight checks for the failures that only show up as "the plugin
282
+ silently does not appear after a restart": manifest name/version/exports,
283
+ `dsh.engines`, bundle patch presence, client platform, the patch row id, the
284
+ native settings key the row implies (`<package>#<row>` must literally appear in
285
+ the client), every relative import resolving across host/client/lib, locale
286
+ dictionary parity, and the client's `DATA_VERSION` matching package.json.
287
+ - Runs in milliseconds and exits non-zero on any drift, so it can guard a release
288
+ or precede a restart.
289
+
290
+ ## 1.15.1
291
+
292
+ One-command post-restart verification: `dsh-skill-eval-verify` (or
293
+ `npm run verify`).
294
+
295
+ - Checks, in order: the versioned data route returns JSON with all report keys;
296
+ the settings API returns 24 fields and effective values; the board HTML carries
297
+ the next-steps / evidence / ledger sections; the settings page renders its
298
+ fields; the latest snapshot has skills and runs; per-turn boards appear once a
299
+ turn has settled; canonical and versioned routes agree and weights sum to 100;
300
+ plus two manual rows for the native seats the CLI cannot inspect.
301
+ - The "no restart yet" case is reported the way it actually looks: versioned
302
+ route answers `text/html`, so `versioned-json` fails and the summary says to
303
+ restart — no false green.
304
+ - Verified against a disposable profile running the same package: 10/10 PASS,
305
+ including `report-contract` (canonical=versioned) and the settings API.
306
+
307
+ ## 1.15.0
308
+
309
+ One board per execution.
310
+
311
+ - **Per-turn boards.** Every settled turn that loaded a skill now writes its own
312
+ pair under `reports/turns/<session>-t<turn>.{md,json}` (in addition to
313
+ `latest.*`), with retention `turnReportsKeep` (default 50) and a switch
314
+ `turnReports` (default on). This is the literal "每次执行后结果在单独的看板":
315
+ each run keeps its own snapshot instead of overwriting the previous one.
316
+ - **`skill_eval_report {view:"turns"}` / `/skill-eval turns`** lists those
317
+ boards newest first (time, session, turn, skills, size, file) in markdown or
318
+ JSON, so an agent can open the exact board for one run.
319
+ - Both surfaces cost zero model tokens; the writer is still debounced and
320
+ failure-tolerant (a broken turn board never blocks the latest snapshot).
321
+
322
+ ## 1.14.0
323
+
324
+ MCP debugging got per-tool latency, not just per-server.
325
+
326
+ - **Per-tool samples.** Every `mcp__<server>__<tool>` call now records duration into
327
+ the window (bounded at 20 samples per tool, merged across windows), so a tool
328
+ that is fine on average but terrible at the tail shows up.
329
+ - **Report rows** carry `avgMs / p50Ms / p95Ms` per tool next to calls/errors.
330
+ - **`mcp-tool-slow` warning** (`mcpToolP95Ms`, default 5000, needs ≥2 calls):
331
+ `MCP tool "context7/query-docs" has a slow tail — p50 4000ms · p95 9000ms`.
332
+ - **`/skill-eval mcp` markdown** gained a _Slowest tools (p95)_ section and shows
333
+ p95 next to error counts for failing tools.
334
+ - Aggregation was carrying tools only as calls/errors; it now carries samples and
335
+ ms through `aggregate`, so the numbers survive every merge step.
336
+
337
+ ## 1.13.0
338
+
339
+ Developer diagnostics — the "is this thing actually working" view.
340
+
341
+ - **`skill_eval_report {view:"dev"}` / `/skill-eval dev`** reports the module
342
+ generation loaded versus the version on disk (the ESM-cache restart trap:
343
+ `module 1.10.1 vs disk 1.13.0 — restart DeepSeek Harness`), the data directory
344
+ and settings override file, store health (records, type counts, last-event
345
+ age, failed writes, queue), the routes this generation actually registered,
346
+ registered tools, inspect-provider state, recent host warnings, and findings
347
+ with severities.
348
+ - Route/tool/inspect registration and warnings are now recorded in a diagnostics
349
+ registry at apply time, so the view reflects what really mounted instead of
350
+ what the code intended.
351
+ - This is the "developer mode" half of the objective: one command answers why an
352
+ edit is not visible and which generation is live.
353
+
354
+ ## 1.12.2
355
+
356
+ - **Graceful degradation on an older host.** The native settings page detects a
357
+ non-JSON answer from `/settings` (the stale prefix route of a previous
358
+ generation) and explains that a restart is needed, with a link to the web
359
+ settings page instead of showing an empty form.
360
+ - **Real-payload compatibility fixture.** `test/fixtures/live-1.10.1.json` is an
361
+ actual payload captured from the running 1.10.x host (turnLoads present,
362
+ `edits`/`validity` absent); the new client is tested against it so a newer UI
363
+ never breaks on an old host.
364
+ - **docs/DEV-RELOAD.md** — why edits do not appear until a restart (ESM module
365
+ cache + boot client scan), how to probe the live generation, the debug
366
+ switches, and the disposable-profile verification recipe that produced the
367
+ 17-check PASS evidence.
368
+
369
+ ## 1.12.1
370
+
371
+ - Settings weight fields use the same 0-100 scale the profile config uses
372
+ (step 5) instead of 0-1, so the native page can save 30/35/... directly.
373
+ Both scales still normalise at scoring time.
374
+ - Acceptance CLI now passes the settings-API and config-page probes to the
375
+ runner and resolves them under the dashboard base path (configurable with
376
+ `DSH_SKILL_EVAL_BASE`), so `dsh-skill-eval-accept` covers 17 checks.
377
+ - Verified end-to-end on a disposable profile: 1.12.0 host routes
378
+ (`/data`, `/settings`, `/config`) all answer, settings POST saves +
379
+ normalises (35/25/25/15 → 0.35/0.25/0.25/0.15), invalid values return 400,
380
+ and acceptance is fully green there.
381
+
382
+ ## 1.12.0
383
+
384
+ Settings moved into DSH's own settings, and the board into the More menu.
385
+
386
+ - **Native plugin settings page.** The bundle patch already declares row
387
+ `skill-eval`, so the plugin now registers `plugins.row.config` keyed
388
+ `dsh-skill-eval#skill-eval`: Settings → Plugins → this plugin's row gains the
389
+ standard configure control, and the page renders the same tunables natively
390
+ (grouped weights/thresholds, cost, advice lifecycle, MCP, output) with a
391
+ Save that applies immediately and persists to `settings.json`.
392
+ - **Settings JSON API** (`<base>/settings` plus a versioned alias): GET returns
393
+ field definitions + effective values; POST accepts JSON or form bodies, then
394
+ validates, saves and applies. Backs the native page; the HTML page keeps
395
+ working.
396
+ - **Board in the More menu.** Registers
397
+ `conversation.session.header.views.actions`, so the session header View row's
398
+ **More** menu gains a `技能评测看板` row that opens the board and dismisses the
399
+ menu.
400
+ - Acceptance gained a `settings-api` check (16 in-process checks).
401
+
402
+ ## 1.11.0
403
+
404
+ Native DSH entry points instead of only a URL.
405
+
406
+ - **Sidebar entry (always on).** A small gauge button beside Settings now opens
407
+ the reporting board in a new tab (with `window.__SEV_ENABLE_POPUP__ = true` it
408
+ still opens the modal instead). Previously this required the popup opt-in.
409
+ - **Plugin detail page section.** Under Settings → Plugins → `dsh-skill-eval`
410
+ the detail page now carries a section: how many skills have telemetry, how
411
+ many have an actionable fix, and buttons for _Open board_ / _Scoring
412
+ settings_. It renders null for every other plugin's subject.
413
+ - Registered seats now: `conversation.chat.turnTail` (turn strip),
414
+ `sidebar.footer.action`, `plugins.detail.section`, and `shell.overlay` when the
415
+ popup is enabled.
416
+
417
+ ## 1.10.1
418
+
419
+ - Hot-reload fix: the dashboard page and the settings page are registered as
420
+ **exact** routes (exact wins over any prefix), so a previous generation still
421
+ owning the `/skill-eval` prefix can no longer shadow them. The prefix route
422
+ stays as a best-effort fallback for other subpaths.
423
+
424
+ ## 1.10.0
425
+
426
+ The reporting board became a real board, and it got a settings page.
427
+
428
+ - **Dashboard** (`<dashboardPath>`, e.g. `/skill-eval`) now leads with the
429
+ reporting material instead of only skill cards: **Next steps** ranked by
430
+ expected gain / effort (same ranking as `loop all`), **What the evidence
431
+ supports** (validity: coverage, ranking significance, can/cannot-say lists),
432
+ **Screened edits** (the ledger), then skill cards, MCP, MCP fixes and runs.
433
+ - **Configuration page** (`<dashboardPath>/config`): every tunable in one form
434
+ (weights, grade thresholds, samples, advice, token budget, cache weight,
435
+ prune, edit budget, exploration window, MCP thresholds, output toggles).
436
+ Saving validates bounds, writes `settings.json` into the data directory,
437
+ applies immediately in memory and wins over the profile config on reload.
438
+ Invalid submissions return the page with per-field errors and change nothing.
439
+ - `/skill-eval config` prints the effective values and the override file path.
440
+ - Acceptance gained a `config-page` check (now 15 in-process checks).
441
+ - Dashboard header links to the settings page; settings page links back.
442
+
443
+ ## 1.9.0
444
+
445
+ Two time-aware answers, plus a written scenario matrix.
446
+
447
+ - **`view:"history"` / `/skill-eval history <skill>`** — chronological load
448
+ table (ok, calls, errors, corrections, latency, turn) and a half-over-half
449
+ trend (`improving` / `degrading` / `flat` / `not-enough-data`) with the
450
+ problem-rate change spelled out.
451
+ - **`view:"impact"` / `/skill-eval impact <skill>`** — pairs every screened edit
452
+ with the loads recorded before and after it and judges the error-rate delta:
453
+ only ≥2 closed loads per side and >15pp counts as improved/degraded, anything
454
+ else is `inconclusive` with the reason. This is the "did my edit help?" loop
455
+ closing, and it starts working automatically as samples accumulate.
456
+ - **docs/SCENARIOS.md** — 25+ scenarios (single skill, cross-skill, edit
457
+ governance, MCP, runs, config) each mapped to its entry point, expected answer,
458
+ status, plus the six things the tool deliberately does not do.
459
+
460
+ ## 1.8.0
461
+
462
+ - **Validity dashboard.** `skill_eval_report {view:"validity"}` (and
463
+ `/skill-eval validity`) reports only what the stored evidence supports:
464
+ coverage (skills, intervals, provisional, actionable), ranking significance,
465
+ advice severity counts, ledger composition and — the point — an explicit
466
+ **Cannot say yet** list (no interval at n < minSamples, before/after effect of
467
+ an edit, token cost of a skill the model never loaded). The same payload is in
468
+ the report JSON as `validity`.
469
+ - **Edit history in the turn strip.** Expanding a turn now shows the last three
470
+ ledger entries for that skill (verdict icon, dimension, hypothesis) next to
471
+ the fixes, so the conversation carries the evolution, not just the score.
472
+ - **Prompt hint** now teaches the safe order: report → `skill_eval_review` →
473
+ edit → re-measure, and points at `view:"loop"` / `view:"validity"`.
474
+ - `docs/WORKFLOW.md` gained the validity entry point.
475
+
476
+ ## 1.7.0
477
+
478
+ - **Batch plans are ranked by expected gain per unit of effort** instead of raw
479
+ score: `impactOf()` weighs headroom (100 − score), severity (critical vs
480
+ warning), evidence strength (provisional samples count less) and the usual
481
+ cost of the fix dimension (static edit ≈ 1x, behaviour change ≈ 2.5x). The
482
+ batch view prints `priority N` with the reasons, so "fix this first" is
483
+ defensible rather than alphabetical.
484
+ - **docs/WORKFLOW.md**: the one-page operating manual — flow diagram, the three
485
+ roles (human / agent / plugin), when to use which entry point, the six
486
+ self-deception guards, common states and the honest-limits list. Linked from
487
+ both READMEs.
488
+
489
+ ## 1.6.0
490
+
491
+ - **Batch loop.** `skill_eval_report {view:"loop", all:true}` and
492
+ `/skill-eval loop all` return one compact plan per skill that currently has an
493
+ actionable fix, worst score first.
494
+ - **Next step in the conversation strip.** The turn card now shows
495
+ `下一步 · 可改 <skill> · <dimension> · <fix>` when the turn's skill has a
496
+ critical/warning fix, so the strip points forward instead of only reporting
497
+ history. Locale strings added for both languages.
498
+
499
+ ## 1.5.0
500
+
501
+ - **Guided optimization loop.** `/skill-eval loop [skill]` (and
502
+ `skill_eval_report {view:"loop"}`) turns the current report into the next
503
+ concrete step, RRSI style: pick the worst skill with actionable advice, cap it
504
+ at that skill's annealed `adviceBudget`, emit the chosen fixes, require a
505
+ `skill_eval_review` screening (leakage critic) before applying, honour the
506
+ exploration directive, then ask for a re-measurement round and warn that
507
+ point-estimate moves are not enough — the interval has to move.
508
+ - A quiet skill or an empty dataset explains itself instead of failing.
509
+ - Pairs with the ledger: accepted edits anneal the budget, same-dimension edits
510
+ trigger the exploration directive, so repeated `/skill-eval loop` calls walk
511
+ the RRSI loop.
512
+
513
+ ## 1.4.0
514
+
515
+ RRSI P2/P3 plus the two static-check gaps recorded in `docs/COMPARISON.md`.
516
+
517
+ - **Annealed edit budget (P2).** Each skill carries `adviceBudget`: it starts at
518
+ `editBudgetMax` (4), anneals toward `editBudgetMin` (1) over
519
+ `editBudgetHorizon` (6) _accepted_ ledger edits using the RRSI cosine schedule,
520
+ and caps how many fixes the report presents as actionable this round.
521
+ - **Exploration directive (P3).** When the last `noveltyWindow` (2) reviewed
522
+ edits for a skill all targeted one dimension, the report emits
523
+ `exploration: {stuck, suggestion, reason}` pointing at the least-used
524
+ untouched dimension; `/skill-eval review` now accepts a `dimension`.
525
+ - **Reference integrity.** The static profile gains `reference-integrity`
526
+ (weight 8): every `references/…`, `scripts/…`, `assets/…` path in the body is
527
+ checked against the skill directory at load time (best-effort; the check is
528
+ skipped rather than faked when no filesystem view exists). Missing files raise
529
+ a critical `broken-references` rule.
530
+ - **Anti-trigger check.** `anti-trigger` (weight 5) scores whether the skill
531
+ says when _not_ to use it; absent → info-level advice.
532
+ - **Skill-name check.** `skill-name` (weight 4) enforces kebab-case ids per the
533
+ agentskills.io contract; a violation is a warning.
534
+ - New config: `editBudgetMax`, `editBudgetMin`, `editBudgetHorizon`,
535
+ `noveltyWindow`.
536
+
537
+ ## 1.3.0
538
+
539
+ RRSI P1 mechanisms (see `docs/RRSI-NOTES.md` §6.1).
540
+
541
+ - **Leakage critic.** New `skill_eval_review` tool screens a proposed skill edit
542
+ before it is applied: hardcoded answers/expected outputs, named benchmarks,
543
+ task ids, machine-specific paths, date windows and input-shape special-casing
544
+ are classified `reject`/`warn`/`accept`, with a required falsifiable
545
+ hypothesis. `lib/critic.js` is pure and tested.
546
+ - **Audit ledger.** Every review appends an edit record (skill, hypothesis,
547
+ verdict, reason codes, diff hash) to the telemetry store. The report exposes
548
+ `edits`, `skill_eval_report` returns it, and the Cordis Inspect provider grew
549
+ an `edits` method — so a fix is traceable to the hypothesis it was meant to
550
+ test.
551
+ - **Pruning.** A well-sampled skill (`pruneMinLoads`, default 6) whose score is
552
+ below `pruneMaxScore` (60) and whose 95% interval also sits low raises a
553
+ `prune-candidate` critical rule: delete it or merge its one useful step into a
554
+ skill that gets used.
555
+ - New config: `registerReviewTool`, `pruneMinLoads`, `pruneMaxScore`.
556
+
557
+ ## 1.2.0
558
+
559
+ Two RRSI-derived P0 mechanisms (see `docs/RRSI-NOTES.md` §6.1).
560
+
561
+ - **Noise floor on advice.** The five rate-based rules (post-load errors, user
562
+ corrections, ignored loads, trigger collisions, loop errors — plus MCP rules
563
+ already thresholded) now bootstrap their window-level rate and only keep
564
+ `critical`/`warning` when the 95% interval's low end clears the threshold.
565
+ Signals that are above the mean but still inside noise are reported as `info`
566
+ with the interval in the evidence ("not outside noise yet"). `adviceNoiseFloor:
567
+ false` restores the old behaviour.
568
+ - **Measured token cost.** Real provider usage from every committed
569
+ `assistant/message` is attributed to the open skill windows
570
+ (`inputTokens/outputTokens/cacheReadTokens`), aggregated into
571
+ `stats.tokensPerLoad` (cached input weighted by `cacheReadWeight`), and used
572
+ for: the efficiency dimension (over `tokenBudgetPerLoad` costs up to 35 points),
573
+ a new `token-cost` advice rule, the Runs board totals and the in-conversation
574
+ card ("≈N tok/load"). `countTokens: false` disables capture. Per-load cost divides only by windows that carried
575
+ real usage; cached input defaults to weight 0.
576
+ - New config: `tokenBudgetPerLoad` (20 000), `cacheReadWeight` (0.1),
577
+ `countTokens` (true), `adviceNoiseFloor` (true).
578
+ - Token attribution is tested per window, including cache reads and ignoring
579
+ other sessions.
580
+
581
+ ## 1.1.0
582
+
583
+ - **Results now land in the conversation.** A completed turn that loaded a skill
584
+ gets a result strip above its action row: skill scores, this turn's call and
585
+ error counts, and an expandable fix list. It reads durable per-turn loads from
586
+ `turnLoads`, costs zero model tokens and needs no popup.
587
+ - The popup (sidebar button + modal) is now **opt-in**: set
588
+ `window.__SEV_ENABLE_POPUP__ = true` to bring it back. The JSON/API/html
589
+ surfaces are unchanged and still back the in-conversation card.
590
+ - Skill loads record the loop `turn` that produced them; the report exposes
591
+ `turnLoads` grouped per session+turn.
592
+ - New Claude-Code-pattern test corpus (`test/corpus/`): document-style,
593
+ MCP-dependent, anti-trigger, overlong-monolith and weak-trigger skills, each
594
+ with an asserted verdict; `docs/COMPARISON.md` records the remaining gaps
595
+ (anti-trigger text, frontmatter integrity, reference link integrity, script
596
+ correctness, optional metadata).
597
+ - Repository moved to `~/Documents/dsh-plugins/dsh-skill-eval`; the profile
598
+ dependency and symlink were re-pointed.
599
+ - Acceptance tooling no longer hardcodes a GUI port: `scripts/accept.mjs`
600
+ resolves the base URL from the argument, then `DSH_WEB_URL`, then
601
+ `~/.dsh/remote-web-ui-relay-bridge.json` (`upstream`), and reports
602
+ **UNREACHABLE** separately from **stale generation** so a wrong port is not
603
+ mistaken for an old module.
604
+ - The manual acceptance row now points at the in-conversation strip instead of
605
+ the opt-in popup.
606
+ - Live MCP capture was verified against the installed servers (context7 call
607
+ bucketed with latency, health and per-tool breakdown).
608
+
609
+ ## 1.0.0
610
+
611
+ First stable release. Everything below was built and verified inside one session
612
+ against DeepSeek Harness 0.2.2-rc.3; the acceptance report lives in
613
+ [docs/ACCEPTANCE.md](docs/ACCEPTANCE.md).
614
+
615
+ ### Recording
616
+
617
+ - Records every `skill` tool load with duration, success/failure and session id.
618
+ - Records user-typed `/skill-name` loads by reading `source.kind ===
619
+ 'skill-invocation'` messages on `agent/pre-step`, deduped by message id.
620
+ - Observation window per load: tool calls, tool errors, MCP calls, loop errors,
621
+ retried model requests, user corrections, sibling skill loads.
622
+ - Windows survive the turn boundary (the correction that proves a skill failed
623
+ usually arrives in the next user message) with a TTL sweep.
624
+ - Parallel loads inside one loop step stay open together instead of being scored
625
+ as a collision; a load in a later step is a real switch.
626
+ - Root vs subagent origin is recorded so short subagent runs do not count as
627
+ "ignored skill".
628
+ - MCP tools are bucketed per `mcp__<server>__<tool>` with latency samples.
629
+
630
+ ### Scoring
631
+
632
+ - Skill score: static 30% / reliability 25% / effectiveness 30% / efficiency 15%,
633
+ A–F bands, provisional + confidence labels.
634
+ - Static authoring checks (9, weight-summing 100): description, trigger
635
+ language, whenToUse, size, sections, progressive disclosure, placeholders,
636
+ examples.
637
+ - Runtime dimensions from load success, post-load tool error rate, loop errors,
638
+ user corrections, ignored loads, sibling collisions, cost.
639
+ - 95% bootstrap confidence interval per skill (seeded, reproducible) and per run.
640
+ - Ranking caveat when two skills' intervals overlap closely.
641
+ - Run score with its own dimensions (quality 0.40 / stability 0.30 / outcome
642
+ 0.20 / efficiency 0.10), interval, confidence, provisional.
643
+ - MCP health per server (`100 − 70·errorRate − slow penalty`) with p50/p95,
644
+ failing tools and MCP-specific fixes.
645
+ - Standard is user configuration: `weights`, `thresholds`, `minSamples`,
646
+ `ciResamples`, `mcpErrorRatePercent`, `mcpToolErrorRatePercent`, `mcpP95Ms`.
647
+ Weights normalise, a zero weight drops the dimension, unordered thresholds are
648
+ repaired, all-zero falls back to defaults.
649
+
650
+ ### Surfaces
651
+
652
+ - `skill_eval_report` tool (`skill`, `view: summary|mcp|runs|standard`, `format`).
653
+ - `/skill-eval` command (`<skill>`, `mcp`, `runs`, `standard`, `accept`).
654
+ - Sidebar footer action + modal overlay with 4 tabs (Skills / Runs / MCP / Dev),
655
+ 5 s auto-refresh, copy-as-markdown, Escape/backdrop close.
656
+ - HTML dashboard `GET /skill-eval` and JSON API `GET /skill-eval/data`
657
+ (plus a versioned alias).
658
+ - Per-turn snapshot `reports/latest.md` + `latest.json` (no model tokens).
659
+ - Cordis Inspect provider `skill-eval` (`summary|skill|mcp|runs|standard`).
660
+
661
+ ### Reliability and diagnostics
662
+
663
+ - Every listener is failure-contained; the two waterfalls always call `next()`.
664
+ - Telemetry never touches the durable session log; it lives in
665
+ `$DSH_HOME/skill-eval/<profile>/events.jsonl`.
666
+ - `dispose()` flushes queued writes (a real data-loss bug found by tests).
667
+ - `node scripts/accept.mjs` and `/skill-eval accept` detect a stale host
668
+ generation; the popup shows a red banner when its payload predates the boards.
669
+ - `DSH_SKILL_EVAL_DEBUG=1` mirrors diagnostics to stderr.
670
+
671
+ ### Tests
672
+
673
+ 118 tests, no build step: unit, integration through a fake host, jsdom rendering
674
+ of the real popup components, snapshot/acceptance, and seeded property tests over
675
+ the scoring pipeline.