dsh-skill-eval 0.0.0-stage → 1.20.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (48) hide show
  1. package/CHANGELOG.md +718 -0
  2. package/LICENSE +21 -0
  3. package/README.md +207 -2
  4. package/README.zh.md +164 -0
  5. package/client.js +2791 -0
  6. package/cordis.patch.yml +9 -0
  7. package/docs/ACCEPTANCE.md +71 -0
  8. package/docs/AUDIT.md +82 -0
  9. package/docs/CODE-REVIEW.md +49 -0
  10. package/docs/COMPARISON.md +33 -0
  11. package/docs/DETAIL.md +468 -0
  12. package/docs/DEV-RELOAD.md +114 -0
  13. package/docs/ENTRYPOINTS.md +65 -0
  14. package/docs/RELEASE.md +44 -0
  15. package/docs/RRSI-NOTES.md +129 -0
  16. package/docs/SCENARIOS.md +91 -0
  17. package/docs/UI-REVIEW.md +115 -0
  18. package/docs/WORKFLOW.md +108 -0
  19. package/icon.svg +9 -0
  20. package/index.js +1173 -0
  21. package/lib/accept.js +350 -0
  22. package/lib/advice.js +557 -0
  23. package/lib/critic.js +101 -0
  24. package/lib/dev.js +148 -0
  25. package/lib/history.js +141 -0
  26. package/lib/html.js +17 -0
  27. package/lib/inspect.js +142 -0
  28. package/lib/integrity.js +46 -0
  29. package/lib/loop.js +195 -0
  30. package/lib/random.js +29 -0
  31. package/lib/recorder.js +644 -0
  32. package/lib/render.js +580 -0
  33. package/lib/report.js +689 -0
  34. package/lib/score.js +565 -0
  35. package/lib/settings.js +415 -0
  36. package/lib/snapshot.js +250 -0
  37. package/lib/steps.js +141 -0
  38. package/lib/store.js +240 -0
  39. package/lib/validity.js +125 -0
  40. package/locale/en.json +6 -0
  41. package/locale/zh.json +6 -0
  42. package/package.json +96 -4
  43. package/scripts/accept.mjs +227 -0
  44. package/scripts/build-client.mjs +59 -0
  45. package/scripts/build.mjs +81 -0
  46. package/scripts/doctor.mjs +234 -0
  47. package/scripts/lib.mjs +87 -0
  48. package/scripts/verify-live.mjs +178 -0
package/CHANGELOG.md ADDED
@@ -0,0 +1,718 @@
1
+ # Changelog
2
+
3
+ ## 1.20.3 — review hardening
4
+
5
+ A full independent review (host, client, tests/packaging, plus a second-model
6
+ pass) reproduced the issues below. Every fix has a regression test.
7
+
8
+ - **The per-turn strip counts real errors.** It read `data.isError`, but DSH's
9
+ tool-call payload is `ToolChatData { root }`; errors now come from
10
+ `root.isError`, and the strip retries the report when the 10 s cache predates
11
+ the turn (rapid consecutive turns used to render nothing).
12
+ - **Settings writes cannot lose a field.** Both HTTP paths now merge inside the
13
+ serialized write queue (`updateSettings`); two overlapping partial saves keep
14
+ both keys. `windowTtlMs`, `reportOnTurnEnd`, `turnReports` and `promptHint`
15
+ are read live, so "saved values apply immediately" is true.
16
+ - **A failed `skill` load opens no evidence window** and no longer supersedes the
17
+ previous skill's window; the test now asserts after turn-stop/dispose.
18
+ - **Corrupt window records cannot crash reporting.** Nested `mcp` buckets and
19
+ tool entries are normalised on read (plus defence-in-depth guards in the
20
+ score/report merge), and finite-but-invalid timestamps render as `—`.
21
+ - **The release checks can fail.** `canonical-route` fails on restart skew,
22
+ `turn-boards` fails with zero boards, manual rows are INFO and excluded from
23
+ the PASS count, and `weightsPercent` uses largest-remainder so it always sums
24
+ to 100 (2/2/2/1 used to show 101 and fail verify-live).
25
+ - **The accept CLI is honest.** It reads `dataDir` from the served report
26
+ (previous `JSON.parse` of an object was silently swallowed), and flag values
27
+ no longer become the base URL.
28
+ - **The packaging doctor passes in an installed tarball** (no `client/` sources
29
+ to compare), requires locale/scripts/README/CHANGELOG/LICENSE in `files`, and
30
+ `--release` opens the tarball instead of only checking that it exists.
31
+ - **Client degradation works.** Versioned data/settings routes that answer
32
+ 200 HTML (DSH's prefix route) now fall back to the canonical JSON routes
33
+ instead of showing a parse error; the stale banner is reachable again.
34
+ - **Per-turn boards carry the settled turn** (session+turn passed from
35
+ `agent/turn-stopping`), coalesced writes emit one board per turn, snapshots
36
+ are written atomically, and history/impact use the shared timestamp guard.
37
+ - **Security/hygiene:** the settings fence also rejects rebinding Hosts (not
38
+ only a mismatched Origin), 403/413 paths and DNS rebinding have tests,
39
+ analyzer/listener disposers are released on teardown, the scheduler guards
40
+ against writes after dispose, duplicate `/skill-name` invocations collapse on
41
+ the durable read path, the dev tab renders the settings file/warning count,
42
+ the settings form keeps in-flight edits and labels its controls, and the
43
+ score colours follow the configured grade floors.
44
+ - Tests: **258 passing** (`npm test`, jsdom installed).
45
+
46
+ ## 1.20.2 — MCP board and settings readability
47
+
48
+ - **MCP tab rebuilt.** The old tab was one SVG chart with two thin bars plus a
49
+ bare grid: the value label collided with the threshold line, the last table
50
+ column was clipped ("2606/260…"), and the table had no container. It is now a
51
+ latency card with CSS meters (p50 solid, p95 tinted, dashed threshold marker
52
+ on the track), a carded server table with a grade chip, and carded advice /
53
+ failing-tool sections. Latencies render as `2.6s` / `180ms` so nothing is
54
+ truncated. With no recorded calls the tab is a single empty card instead of a
55
+ chart card plus an empty table that read as two broken panels.
56
+ - **Settings readability.** Weight fields show a `%` unit, grade-floor help no
57
+ longer repeats the label ("A 级下限 / A 级下限。"), units are localized in the
58
+ Chinese form and web page (`loads → 次加载`, `edits → 次修改`), the config
59
+ dialog is titled 「配置」 with the field summary as its subtitle, and the web
60
+ page keeps readable headings in dark mode.
61
+ - **CSS collision fixed.** The advice container reused `.sev-sub`, which also
62
+ styles the panel subtitle, so the later rule silently restyled the header.
63
+ Advice blocks now use `.sev-advice-list` / `.sev-card`.
64
+
65
+ ## 1.20.1 — board honesty pass
66
+
67
+ - **The 「配置」 button no longer relies on `window.open`.** The desktop shell
68
+ denies every popup (`setWindowOpenHandler` returns `deny`; only `https:` URLs
69
+ are handed to the browser), so on an `http://127.0.0.1` board the button did
70
+ nothing at all. It now opens the same settings form in an in-panel dialog,
71
+ with Escape/backdrop close and the web-config link hidden where the shell
72
+ would deny it. The web page and the native bundle seat still exist.
73
+ - **The dev tab reports real diagnostics.** The data route now carries a `dev`
74
+ block (loaded module vs package on disk with the restart-skew flag, claimed
75
+ route/tool counts, settings file, warning count) and the tab renders it next
76
+ to the effective scoring standard; the data endpoint shown is the versioned
77
+ URL the client actually calls, not the bare prefix.
78
+ - **The MCP latency chart follows the configured threshold.** `mcpP95Ms` now
79
+ ships with the report, so the alert line can no longer disagree with the MCP
80
+ advice that uses the same number (3000 ms remains the older-payload fallback).
81
+
82
+ ## 1.20.0 — per-step trajectory
83
+
84
+ - **Per-step loop telemetry:** the recorder persists one row per agent step
85
+ (turn, step, tool calls/errors, MCP calls, loop errors, tokens, duration,
86
+ loaded skills). `lib/steps.js` scores each step 0–100 and groups turns.
87
+ - **New board tab 「轨迹 / Steps」:** step KPIs, recent-turn comparison with
88
+ deltas, per-turn step list with score chips, and a skill filter. Each skill
89
+ card gets a「查看轨迹」button that opens the tab filtered to that skill.
90
+ - Step records are additive event-log rows; old telemetry stays readable.
91
+ - **Multi-skill turn strip:** each skill loaded in a turn keeps its own chip
92
+ and now its own "next fix" (up to three); previously only the first global
93
+ suggestion was shown. Covered by a two-skill regression test.
94
+ - **npm bin targets normalized** (`scripts/*.mjs` without `./`): npm 11 was
95
+ stripping all three bins at publish time with an "invalid and removed"
96
+ warning, so a published install would have lost the CLI helpers.
97
+ - **One-command build:** `npm run build` (format/lint/client/doctor/tests),
98
+ `npm run build:pack` (adds tarball + release doctor), `npm run build:install`
99
+ (also installs into the desktop profile), `npm run build:fast` (client +
100
+ doctor only). Implemented in `scripts/build.mjs`.
101
+ - Tests: **239 passing** (new step scoring/recorder/report/client-filter suites).
102
+
103
+ ## 1.19.1 — test hygiene
104
+
105
+ - Client tests share one `createHost(window, style)` helper and top-level
106
+ `createRoot`/`act` imports instead of 18 copies of the same setup.
107
+ - No runtime change; 232 tests still passing.
108
+
109
+ ## 1.19.0 — board charts + client parts + tooling
110
+
111
+ - **Client split with a zero-dependency builder.** `client.js` is now generated
112
+ from `client/*.js` by `scripts/build-client.mjs`; `prepack`/`pretest` rebuild
113
+ it, doctor fails when the shipped file is stale, and `client/README.md`
114
+ documents the workflow. The package still ships one self-contained loader
115
+ factory, so installation needs no build.
116
+ - **Terminal scripts share plumbing.** `scripts/lib.mjs` now owns bridge/base
117
+ discovery, dashboard base, expected version and timeout HTTP GET; accept and
118
+ verify use it (tests cover precedence and error messages).
119
+ - **Formatting/linting.** Prettier (config + `format`/`format:check`) and ESLint
120
+ 10 flat config (`lint`) added as dev dependencies; all source files formatted.
121
+ - **Board charts (pure SVG, no dependency):** score distribution histogram with
122
+ average marker, per-skill trend sparkline, evidence-coverage donut, MCP
123
+ p50/p95 latency chart with the 3000 ms threshold, and session-activity bars.
124
+ Charts also appear on the Runs and MCP tabs; every chart carries an
125
+ `aria-label` with its data.
126
+ - **Tests:** 232 passing; lint/format/doctor/package gates re-run in 1.19.0.
127
+
128
+ ## 1.18.5 — hygiene follow-up
129
+
130
+ - **Locale metadata restored.** DSH reads `locale/<lang>.json` for the plugin's
131
+ localized market title/description; 1.18.4 wrongly deleted it. The files are
132
+ back, trimmed to `meta.title` + `meta.description` only (the old 16 unused UI
133
+ keys were the actual drift), and the doctor now checks both locale metas.
134
+ - **Settings group metadata now comes from the host.** Every field ships
135
+ `group`/`groupZh`; the client groups by it and keeps its local classifier only
136
+ as an older-payload fallback, removing a second copy of the grouping rules.
137
+ - Tests: **228 passing**.
138
+
139
+ ## 1.18.4 — code hygiene
140
+
141
+ Full review: [docs/CODE-REVIEW.md](docs/CODE-REVIEW.md).
142
+
143
+ - Removed dead `locale/*.json` (unused at runtime, already drifted from the
144
+ client dictionary), with exports/files/doctor/docs updated.
145
+ - Consolidated duplicated helpers: `lib/random.js` (FNV hash + mulberry32) and
146
+ `lib/html.js` (HTML escape) used by score/report/index/render/settings.
147
+ - Deleted dead TEXTS keys and CSS left by the removed More-menu/detail surfaces;
148
+ documented the real client seats in the file header; added the missing
149
+ `.sev-btn--ghost` rule.
150
+ - Restored serialized atomic settings writes (a regression from an earlier
151
+ refactor) and added a regression test; added a settings-group drift test.
152
+ - Added `.editorconfig` and `.gitignore`.
153
+ - Tests: **227 passing**.
154
+
155
+ ## 1.18.3 — settings aligned with the general settings layout
156
+
157
+ - Per-setting help lines (zh/en), units (loads / ms / tokens / edits), and
158
+ switch-style booleans for the 25 settings; section rows now use the same
159
+ label + description + control pattern as DSH's 通用设置.
160
+ - The web config page renders the same help text and units.
161
+ - 225 tests.
162
+
163
+ ## 1.18.2 — panel and settings polish
164
+
165
+ Reference: the context-insight panel (`dsh-context`) and the generic settings
166
+ layout.
167
+
168
+ - **Board panel**: KPI row (skills / loads / actionable / average score), a
169
+ search box, an "needs a fix" filter chip, a “Configure” shortcut next to
170
+ Refresh/Copy, and a last-updated stamp.
171
+ - **Native settings form**: a bilingual heading + subtitle, and the missing
172
+ `windowTtlMs` field (25 settings); labels/group titles follow the app locale.
173
+ - English TEXTS parity restored for every new string; 225 tests.
174
+
175
+ ## 1.18.1 — four-layer audit fixes
176
+
177
+ Full report: [docs/AUDIT.md](docs/AUDIT.md). Findings reproduced and fixed
178
+ across host data, scoring, HTTP, client and packaging.
179
+
180
+ - **Data integrity:** `__proto__` telemetry keys can no longer pollute
181
+ `Object.prototype` or crash reports; a torn final JSONL line is repaired
182
+ before the next append; cumulative window snapshots no longer double-count
183
+ run counters.
184
+ - **Correctness:** the bootstrap CI uses the same scoring options as the point
185
+ estimate; recorded static profiles are normalised; evicted windows are
186
+ censored from abandoned-run scoring; grade floors are sorted, not clamped.
187
+ - **Privacy/size:** skill bodies are no longer persisted or shipped in reports.
188
+ - **Performance:** report cache keyed by store identity + 2 s TTL; stable locale
189
+ subscriptions; memoized cards.
190
+ - **HTTP:** same-origin write gate + `x-skill-eval-config` header, atomic
191
+ serialized settings writes, merged-weight validation, byte-accurate 413,
192
+ versioned board/config pages render their own base.
193
+ - **Client:** request generation guard, focus trap/restore, clipboard cleanup,
194
+ settings-page styles, duplicate i18n key fixed, aria roles/live status,
195
+ English TEXTS parity.
196
+ - Settings: `windowTtlMs` (25 fields), bounds aligned with the Config schema.
197
+ - Tests: **225 passing**.
198
+
199
+ ## 1.17.5 — last mixed-language strings
200
+
201
+ - Ranking note and validity statements are bilingual (`noteZh`, `canSayZh`,
202
+ `cannotSayZh`); HTML board and client panel pick the Chinese copy.
203
+ - 212 tests at the time.
204
+
205
+ ## 1.17.4 — advice localization and README tightening
206
+
207
+ - Every advice rule carries `titleZh`/`fixZh` (MCP names filled in); localize
208
+ the whole report once for the HTML board; client cards/MCP list/turn strip
209
+ select the Chinese copy. README opens with a Chinese TL;DR; config key list
210
+ moved to `docs/DETAIL.md`; full config documented.
211
+
212
+ ## 1.17.3 — release candidate
213
+
214
+ - Copy button falls back to `execCommand('copy')` with failure feedback.
215
+ - Packaging: LICENSE, explicit `files`, three bins, release scripts and doctor
216
+ release gate; clean-profile smoke documented in `docs/RELEASE.md`.
217
+
218
+ ## 1.17.0
219
+
220
+ Board and settings moved to the same seats the context plugin (`dsh-context`)
221
+ uses, per the request to follow that pattern.
222
+
223
+ - **Board = sidebar panel + main panel.** Registers `sidebar.panellist`
224
+ (id `skill-eval`, order 40, gauge icon, localized label) and `main` keyed
225
+ `skill-eval`. Clicking the sidebar icon opens a full central board panel:
226
+ header with skill/actionable counts, tabs (skills / runs / MCP / dev),
227
+ refresh and copy, reusing the existing SkillCard/RunsView/McpView/DevView.
228
+ - **Settings = bundle config page.** Registers `plugins.bundle.config` keyed
229
+ `dsh-skill-eval` (the bundle's own page under Plugins), replacing the row
230
+ config seat; the native settings form is unchanged.
231
+ - Removed: the session-header More-menu board row and the row-config seat.
232
+ Kept: the in-conversation result strip, the versioned web board, the
233
+ debug-only popup (opt-in).
234
+ - Client tests rewritten for the new seat set (208 passing).
235
+
236
+ ## 1.16.1
237
+
238
+ - **Versioned page aliases.** `/skill-eval/board/<version>` and
239
+ `/skill-eval/board/<version>/config` are registered as exact routes each
240
+ release. Hot-reloaded generations cannot reclaim the canonical page route an
241
+ older generation still owns, which is why the bilingual board initially kept
242
+ serving the old page; the versioned alias is always the current one. The
243
+ client's board/config links and the verify/accept scripts use it (with a
244
+ canonical fallback for a clean start).
245
+ - Verified live: `?lang=zh` / `Accept-Language: zh-CN` return Chinese chrome on
246
+ `/skill-eval/board/1.16.1`, the settings page renders Chinese labels, and the
247
+ English default is unchanged.
248
+
249
+ ## 1.16.0
250
+
251
+ The board and the settings page are bilingual (中 / EN).
252
+
253
+ - Language resolution: explicit `?lang=zh|en` → `Accept-Language` header →
254
+ English fallback (pure `resolveLang`, unit-tested).
255
+ - The board translates its chrome: title, scoring line, totals, dimension names,
256
+ next steps, evidence (can / cannot say), screened edits, MCP and runs tables,
257
+ plus a `中文 / English` switch link.
258
+ - The settings page translates group titles, all 24 field labels, banners and
259
+ buttons; the switch keeps you on the page.
260
+ - Advice titles/evidence/fixes come from the host rules and stay in English for
261
+ now; the surrounding chrome is translated. Documented in `docs/ENTRYPOINTS.md`.
262
+ - 208 tests (new i18n suite + route-level language checks: header wins over
263
+ default, explicit query wins over header).
264
+
265
+ ## 1.15.5
266
+
267
+ Entry points consolidated to three human surfaces.
268
+
269
+ - Kept: the in-conversation result strip (`conversation.chat.turnTail`), one
270
+ **Skill eval board** row in the session header's **More** menu
271
+ (`conversation.session.header.views.actions`), and the plugin's own row config
272
+ page (`plugins.row.config`) for scoring settings.
273
+ - Removed as duplicates: the plugin detail-page section
274
+ (`plugins.detail.section`) and the always-on sidebar button; the sidebar
275
+ button and the popup now come back together only with
276
+ `window.__SEV_ENABLE_POPUP__ = true` (debug surface).
277
+ - `docs/ENTRYPOINTS.md` (bilingual) is the single reference for where things are;
278
+ the READMEs point at it instead of repeating the tables.
279
+
280
+ ## 1.15.4
281
+
282
+ Hot reload now actually works on this machine — documented and verified.
283
+
284
+ - The desktop profile patch gained an `hmr` entry with
285
+ `root: ["/Users/adams/Documents/dsh-plugins/dsh-skill-eval"]`, so Cordis HMR
286
+ watches the plugin source directory. A source edit (or `touch`) replaces the
287
+ running plugin generation **without a restart**; the versioned route changes
288
+ from `text/html` to `application/json` immediately.
289
+ - Verified end to end on the live host: 1.15.2 → 1.15.3 → 1.15.4 all rolled
290
+ over through HMR, `dsh-skill-eval-verify` reached 10/10 PASS (including the
291
+ first per-turn board), native client seats became active
292
+ (`plugins.row.config` key `dsh-skill-eval#skill-eval`,
293
+ `conversation.session.header.views.actions` id `skill-eval-board`,
294
+ `sidebar.footer.action`, `plugins.detail.section`, `conversation.chat.turnTail`),
295
+ and a live settings save (35/25/25/15) moved the report's `weightsPercent`
296
+ immediately.
297
+ - Two caveats recorded in docs: the **canonical** `/skill-eval/data` and
298
+ `/skill-eval/settings` routes can still be served by an older leaked instance
299
+ after a reload — use the versioned routes (what the client and verify do); and
300
+ a page refresh is only needed if the client HMR receiver did not swap the
301
+ bundle.
302
+ - `docs/DEV-RELOAD.md` rewritten around this recipe.
303
+
304
+ ## 1.15.3
305
+
306
+ Hot reload works when the plugin is installed as a **tarball** into the profile,
307
+ not as a `link:` dependency.
308
+
309
+ - Root cause of "I edited the code and the live host kept running the old
310
+ module": with a linked package the loaded files live outside the profile
311
+ directory, and Cordis HMR only replaces plugins whose loaded files below the
312
+ profile changed. Installing `npm pack` output (`file:…/dist/<name>-<v>.tgz`)
313
+ puts a real copy under `profile/node_modules`, so a Plugin Manager
314
+ recomposition swaps it in place — verified by bumping the version and watching
315
+ `/skill-eval/data/<new version>` turn from `text/html` into JSON with no
316
+ restart.
317
+ - `dist/` holds the installable tarball; `docs/DEV-RELOAD.md` documents the
318
+ pack → `dsh plugin add` → recompose loop, and the link-install caveat.
319
+
320
+ ## 1.15.2
321
+
322
+ Packaging doctor: `dsh-skill-eval-doctor` (or `npm run doctor`).
323
+
324
+ - Twelve pre-flight checks for the failures that only show up as "the plugin
325
+ silently does not appear after a restart": manifest name/version/exports,
326
+ `dsh.engines`, bundle patch presence, client platform, the patch row id, the
327
+ native settings key the row implies (`<package>#<row>` must literally appear in
328
+ the client), every relative import resolving across host/client/lib, locale
329
+ dictionary parity, and the client's `DATA_VERSION` matching package.json.
330
+ - Runs in milliseconds and exits non-zero on any drift, so it can guard a release
331
+ or precede a restart.
332
+
333
+ ## 1.15.1
334
+
335
+ One-command post-restart verification: `dsh-skill-eval-verify` (or
336
+ `npm run verify`).
337
+
338
+ - Checks, in order: the versioned data route returns JSON with all report keys;
339
+ the settings API returns 24 fields and effective values; the board HTML carries
340
+ the next-steps / evidence / ledger sections; the settings page renders its
341
+ fields; the latest snapshot has skills and runs; per-turn boards appear once a
342
+ turn has settled; canonical and versioned routes agree and weights sum to 100;
343
+ plus two manual rows for the native seats the CLI cannot inspect.
344
+ - The "no restart yet" case is reported the way it actually looks: versioned
345
+ route answers `text/html`, so `versioned-json` fails and the summary says to
346
+ restart — no false green.
347
+ - Verified against a disposable profile running the same package: 10/10 PASS,
348
+ including `report-contract` (canonical=versioned) and the settings API.
349
+
350
+ ## 1.15.0
351
+
352
+ One board per execution.
353
+
354
+ - **Per-turn boards.** Every settled turn that loaded a skill now writes its own
355
+ pair under `reports/turns/<session>-t<turn>.{md,json}` (in addition to
356
+ `latest.*`), with retention `turnReportsKeep` (default 50) and a switch
357
+ `turnReports` (default on). This is the literal "每次执行后结果在单独的看板":
358
+ each run keeps its own snapshot instead of overwriting the previous one.
359
+ - **`skill_eval_report {view:"turns"}` / `/skill-eval turns`** lists those
360
+ boards newest first (time, session, turn, skills, size, file) in markdown or
361
+ JSON, so an agent can open the exact board for one run.
362
+ - Both surfaces cost zero model tokens; the writer is still debounced and
363
+ failure-tolerant (a broken turn board never blocks the latest snapshot).
364
+
365
+ ## 1.14.0
366
+
367
+ MCP debugging got per-tool latency, not just per-server.
368
+
369
+ - **Per-tool samples.** Every `mcp__<server>__<tool>` call now records duration into
370
+ the window (bounded at 20 samples per tool, merged across windows), so a tool
371
+ that is fine on average but terrible at the tail shows up.
372
+ - **Report rows** carry `avgMs / p50Ms / p95Ms` per tool next to calls/errors.
373
+ - **`mcp-tool-slow` warning** (`mcpToolP95Ms`, default 5000, needs ≥2 calls):
374
+ `MCP tool "context7/query-docs" has a slow tail — p50 4000ms · p95 9000ms`.
375
+ - **`/skill-eval mcp` markdown** gained a _Slowest tools (p95)_ section and shows
376
+ p95 next to error counts for failing tools.
377
+ - Aggregation was carrying tools only as calls/errors; it now carries samples and
378
+ ms through `aggregate`, so the numbers survive every merge step.
379
+
380
+ ## 1.13.0
381
+
382
+ Developer diagnostics — the "is this thing actually working" view.
383
+
384
+ - **`skill_eval_report {view:"dev"}` / `/skill-eval dev`** reports the module
385
+ generation loaded versus the version on disk (the ESM-cache restart trap:
386
+ `module 1.10.1 vs disk 1.13.0 — restart DeepSeek Harness`), the data directory
387
+ and settings override file, store health (records, type counts, last-event
388
+ age, failed writes, queue), the routes this generation actually registered,
389
+ registered tools, inspect-provider state, recent host warnings, and findings
390
+ with severities.
391
+ - Route/tool/inspect registration and warnings are now recorded in a diagnostics
392
+ registry at apply time, so the view reflects what really mounted instead of
393
+ what the code intended.
394
+ - This is the "developer mode" half of the objective: one command answers why an
395
+ edit is not visible and which generation is live.
396
+
397
+ ## 1.12.2
398
+
399
+ - **Graceful degradation on an older host.** The native settings page detects a
400
+ non-JSON answer from `/settings` (the stale prefix route of a previous
401
+ generation) and explains that a restart is needed, with a link to the web
402
+ settings page instead of showing an empty form.
403
+ - **Real-payload compatibility fixture.** `test/fixtures/live-1.10.1.json` is an
404
+ actual payload captured from the running 1.10.x host (turnLoads present,
405
+ `edits`/`validity` absent); the new client is tested against it so a newer UI
406
+ never breaks on an old host.
407
+ - **docs/DEV-RELOAD.md** — why edits do not appear until a restart (ESM module
408
+ cache + boot client scan), how to probe the live generation, the debug
409
+ switches, and the disposable-profile verification recipe that produced the
410
+ 17-check PASS evidence.
411
+
412
+ ## 1.12.1
413
+
414
+ - Settings weight fields use the same 0-100 scale the profile config uses
415
+ (step 5) instead of 0-1, so the native page can save 30/35/... directly.
416
+ Both scales still normalise at scoring time.
417
+ - Acceptance CLI now passes the settings-API and config-page probes to the
418
+ runner and resolves them under the dashboard base path (configurable with
419
+ `DSH_SKILL_EVAL_BASE`), so `dsh-skill-eval-accept` covers 17 checks.
420
+ - Verified end-to-end on a disposable profile: 1.12.0 host routes
421
+ (`/data`, `/settings`, `/config`) all answer, settings POST saves +
422
+ normalises (35/25/25/15 → 0.35/0.25/0.25/0.15), invalid values return 400,
423
+ and acceptance is fully green there.
424
+
425
+ ## 1.12.0
426
+
427
+ Settings moved into DSH's own settings, and the board into the More menu.
428
+
429
+ - **Native plugin settings page.** The bundle patch already declares row
430
+ `skill-eval`, so the plugin now registers `plugins.row.config` keyed
431
+ `dsh-skill-eval#skill-eval`: Settings → Plugins → this plugin's row gains the
432
+ standard configure control, and the page renders the same tunables natively
433
+ (grouped weights/thresholds, cost, advice lifecycle, MCP, output) with a
434
+ Save that applies immediately and persists to `settings.json`.
435
+ - **Settings JSON API** (`<base>/settings` plus a versioned alias): GET returns
436
+ field definitions + effective values; POST accepts JSON or form bodies, then
437
+ validates, saves and applies. Backs the native page; the HTML page keeps
438
+ working.
439
+ - **Board in the More menu.** Registers
440
+ `conversation.session.header.views.actions`, so the session header View row's
441
+ **More** menu gains a `技能评测看板` row that opens the board and dismisses the
442
+ menu.
443
+ - Acceptance gained a `settings-api` check (16 in-process checks).
444
+
445
+ ## 1.11.0
446
+
447
+ Native DSH entry points instead of only a URL.
448
+
449
+ - **Sidebar entry (always on).** A small gauge button beside Settings now opens
450
+ the reporting board in a new tab (with `window.__SEV_ENABLE_POPUP__ = true` it
451
+ still opens the modal instead). Previously this required the popup opt-in.
452
+ - **Plugin detail page section.** Under Settings → Plugins → `dsh-skill-eval`
453
+ the detail page now carries a section: how many skills have telemetry, how
454
+ many have an actionable fix, and buttons for _Open board_ / _Scoring
455
+ settings_. It renders null for every other plugin's subject.
456
+ - Registered seats now: `conversation.chat.turnTail` (turn strip),
457
+ `sidebar.footer.action`, `plugins.detail.section`, and `shell.overlay` when the
458
+ popup is enabled.
459
+
460
+ ## 1.10.1
461
+
462
+ - Hot-reload fix: the dashboard page and the settings page are registered as
463
+ **exact** routes (exact wins over any prefix), so a previous generation still
464
+ owning the `/skill-eval` prefix can no longer shadow them. The prefix route
465
+ stays as a best-effort fallback for other subpaths.
466
+
467
+ ## 1.10.0
468
+
469
+ The reporting board became a real board, and it got a settings page.
470
+
471
+ - **Dashboard** (`<dashboardPath>`, e.g. `/skill-eval`) now leads with the
472
+ reporting material instead of only skill cards: **Next steps** ranked by
473
+ expected gain / effort (same ranking as `loop all`), **What the evidence
474
+ supports** (validity: coverage, ranking significance, can/cannot-say lists),
475
+ **Screened edits** (the ledger), then skill cards, MCP, MCP fixes and runs.
476
+ - **Configuration page** (`<dashboardPath>/config`): every tunable in one form
477
+ (weights, grade thresholds, samples, advice, token budget, cache weight,
478
+ prune, edit budget, exploration window, MCP thresholds, output toggles).
479
+ Saving validates bounds, writes `settings.json` into the data directory,
480
+ applies immediately in memory and wins over the profile config on reload.
481
+ Invalid submissions return the page with per-field errors and change nothing.
482
+ - `/skill-eval config` prints the effective values and the override file path.
483
+ - Acceptance gained a `config-page` check (now 15 in-process checks).
484
+ - Dashboard header links to the settings page; settings page links back.
485
+
486
+ ## 1.9.0
487
+
488
+ Two time-aware answers, plus a written scenario matrix.
489
+
490
+ - **`view:"history"` / `/skill-eval history <skill>`** — chronological load
491
+ table (ok, calls, errors, corrections, latency, turn) and a half-over-half
492
+ trend (`improving` / `degrading` / `flat` / `not-enough-data`) with the
493
+ problem-rate change spelled out.
494
+ - **`view:"impact"` / `/skill-eval impact <skill>`** — pairs every screened edit
495
+ with the loads recorded before and after it and judges the error-rate delta:
496
+ only ≥2 closed loads per side and >15pp counts as improved/degraded, anything
497
+ else is `inconclusive` with the reason. This is the "did my edit help?" loop
498
+ closing, and it starts working automatically as samples accumulate.
499
+ - **docs/SCENARIOS.md** — 25+ scenarios (single skill, cross-skill, edit
500
+ governance, MCP, runs, config) each mapped to its entry point, expected answer,
501
+ status, plus the six things the tool deliberately does not do.
502
+
503
+ ## 1.8.0
504
+
505
+ - **Validity dashboard.** `skill_eval_report {view:"validity"}` (and
506
+ `/skill-eval validity`) reports only what the stored evidence supports:
507
+ coverage (skills, intervals, provisional, actionable), ranking significance,
508
+ advice severity counts, ledger composition and — the point — an explicit
509
+ **Cannot say yet** list (no interval at n < minSamples, before/after effect of
510
+ an edit, token cost of a skill the model never loaded). The same payload is in
511
+ the report JSON as `validity`.
512
+ - **Edit history in the turn strip.** Expanding a turn now shows the last three
513
+ ledger entries for that skill (verdict icon, dimension, hypothesis) next to
514
+ the fixes, so the conversation carries the evolution, not just the score.
515
+ - **Prompt hint** now teaches the safe order: report → `skill_eval_review` →
516
+ edit → re-measure, and points at `view:"loop"` / `view:"validity"`.
517
+ - `docs/WORKFLOW.md` gained the validity entry point.
518
+
519
+ ## 1.7.0
520
+
521
+ - **Batch plans are ranked by expected gain per unit of effort** instead of raw
522
+ score: `impactOf()` weighs headroom (100 − score), severity (critical vs
523
+ warning), evidence strength (provisional samples count less) and the usual
524
+ cost of the fix dimension (static edit ≈ 1x, behaviour change ≈ 2.5x). The
525
+ batch view prints `priority N` with the reasons, so "fix this first" is
526
+ defensible rather than alphabetical.
527
+ - **docs/WORKFLOW.md**: the one-page operating manual — flow diagram, the three
528
+ roles (human / agent / plugin), when to use which entry point, the six
529
+ self-deception guards, common states and the honest-limits list. Linked from
530
+ both READMEs.
531
+
532
+ ## 1.6.0
533
+
534
+ - **Batch loop.** `skill_eval_report {view:"loop", all:true}` and
535
+ `/skill-eval loop all` return one compact plan per skill that currently has an
536
+ actionable fix, worst score first.
537
+ - **Next step in the conversation strip.** The turn card now shows
538
+ `下一步 · 可改 <skill> · <dimension> · <fix>` when the turn's skill has a
539
+ critical/warning fix, so the strip points forward instead of only reporting
540
+ history. Locale strings added for both languages.
541
+
542
+ ## 1.5.0
543
+
544
+ - **Guided optimization loop.** `/skill-eval loop [skill]` (and
545
+ `skill_eval_report {view:"loop"}`) turns the current report into the next
546
+ concrete step, RRSI style: pick the worst skill with actionable advice, cap it
547
+ at that skill's annealed `adviceBudget`, emit the chosen fixes, require a
548
+ `skill_eval_review` screening (leakage critic) before applying, honour the
549
+ exploration directive, then ask for a re-measurement round and warn that
550
+ point-estimate moves are not enough — the interval has to move.
551
+ - A quiet skill or an empty dataset explains itself instead of failing.
552
+ - Pairs with the ledger: accepted edits anneal the budget, same-dimension edits
553
+ trigger the exploration directive, so repeated `/skill-eval loop` calls walk
554
+ the RRSI loop.
555
+
556
+ ## 1.4.0
557
+
558
+ RRSI P2/P3 plus the two static-check gaps recorded in `docs/COMPARISON.md`.
559
+
560
+ - **Annealed edit budget (P2).** Each skill carries `adviceBudget`: it starts at
561
+ `editBudgetMax` (4), anneals toward `editBudgetMin` (1) over
562
+ `editBudgetHorizon` (6) _accepted_ ledger edits using the RRSI cosine schedule,
563
+ and caps how many fixes the report presents as actionable this round.
564
+ - **Exploration directive (P3).** When the last `noveltyWindow` (2) reviewed
565
+ edits for a skill all targeted one dimension, the report emits
566
+ `exploration: {stuck, suggestion, reason}` pointing at the least-used
567
+ untouched dimension; `/skill-eval review` now accepts a `dimension`.
568
+ - **Reference integrity.** The static profile gains `reference-integrity`
569
+ (weight 8): every `references/…`, `scripts/…`, `assets/…` path in the body is
570
+ checked against the skill directory at load time (best-effort; the check is
571
+ skipped rather than faked when no filesystem view exists). Missing files raise
572
+ a critical `broken-references` rule.
573
+ - **Anti-trigger check.** `anti-trigger` (weight 5) scores whether the skill
574
+ says when _not_ to use it; absent → info-level advice.
575
+ - **Skill-name check.** `skill-name` (weight 4) enforces kebab-case ids per the
576
+ agentskills.io contract; a violation is a warning.
577
+ - New config: `editBudgetMax`, `editBudgetMin`, `editBudgetHorizon`,
578
+ `noveltyWindow`.
579
+
580
+ ## 1.3.0
581
+
582
+ RRSI P1 mechanisms (see `docs/RRSI-NOTES.md` §6.1).
583
+
584
+ - **Leakage critic.** New `skill_eval_review` tool screens a proposed skill edit
585
+ before it is applied: hardcoded answers/expected outputs, named benchmarks,
586
+ task ids, machine-specific paths, date windows and input-shape special-casing
587
+ are classified `reject`/`warn`/`accept`, with a required falsifiable
588
+ hypothesis. `lib/critic.js` is pure and tested.
589
+ - **Audit ledger.** Every review appends an edit record (skill, hypothesis,
590
+ verdict, reason codes, diff hash) to the telemetry store. The report exposes
591
+ `edits`, `skill_eval_report` returns it, and the Cordis Inspect provider grew
592
+ an `edits` method — so a fix is traceable to the hypothesis it was meant to
593
+ test.
594
+ - **Pruning.** A well-sampled skill (`pruneMinLoads`, default 6) whose score is
595
+ below `pruneMaxScore` (60) and whose 95% interval also sits low raises a
596
+ `prune-candidate` critical rule: delete it or merge its one useful step into a
597
+ skill that gets used.
598
+ - New config: `registerReviewTool`, `pruneMinLoads`, `pruneMaxScore`.
599
+
600
+ ## 1.2.0
601
+
602
+ Two RRSI-derived P0 mechanisms (see `docs/RRSI-NOTES.md` §6.1).
603
+
604
+ - **Noise floor on advice.** The five rate-based rules (post-load errors, user
605
+ corrections, ignored loads, trigger collisions, loop errors — plus MCP rules
606
+ already thresholded) now bootstrap their window-level rate and only keep
607
+ `critical`/`warning` when the 95% interval's low end clears the threshold.
608
+ Signals that are above the mean but still inside noise are reported as `info`
609
+ with the interval in the evidence ("not outside noise yet"). `adviceNoiseFloor:
610
+ false` restores the old behaviour.
611
+ - **Measured token cost.** Real provider usage from every committed
612
+ `assistant/message` is attributed to the open skill windows
613
+ (`inputTokens/outputTokens/cacheReadTokens`), aggregated into
614
+ `stats.tokensPerLoad` (cached input weighted by `cacheReadWeight`), and used
615
+ for: the efficiency dimension (over `tokenBudgetPerLoad` costs up to 35 points),
616
+ a new `token-cost` advice rule, the Runs board totals and the in-conversation
617
+ card ("≈N tok/load"). `countTokens: false` disables capture. Per-load cost divides only by windows that carried
618
+ real usage; cached input defaults to weight 0.
619
+ - New config: `tokenBudgetPerLoad` (20 000), `cacheReadWeight` (0.1),
620
+ `countTokens` (true), `adviceNoiseFloor` (true).
621
+ - Token attribution is tested per window, including cache reads and ignoring
622
+ other sessions.
623
+
624
+ ## 1.1.0
625
+
626
+ - **Results now land in the conversation.** A completed turn that loaded a skill
627
+ gets a result strip above its action row: skill scores, this turn's call and
628
+ error counts, and an expandable fix list. It reads durable per-turn loads from
629
+ `turnLoads`, costs zero model tokens and needs no popup.
630
+ - The popup (sidebar button + modal) is now **opt-in**: set
631
+ `window.__SEV_ENABLE_POPUP__ = true` to bring it back. The JSON/API/html
632
+ surfaces are unchanged and still back the in-conversation card.
633
+ - Skill loads record the loop `turn` that produced them; the report exposes
634
+ `turnLoads` grouped per session+turn.
635
+ - New Claude-Code-pattern test corpus (`test/corpus/`): document-style,
636
+ MCP-dependent, anti-trigger, overlong-monolith and weak-trigger skills, each
637
+ with an asserted verdict; `docs/COMPARISON.md` records the remaining gaps
638
+ (anti-trigger text, frontmatter integrity, reference link integrity, script
639
+ correctness, optional metadata).
640
+ - Repository moved to `~/Documents/dsh-plugins/dsh-skill-eval`; the profile
641
+ dependency and symlink were re-pointed.
642
+ - Acceptance tooling no longer hardcodes a GUI port: `scripts/accept.mjs`
643
+ resolves the base URL from the argument, then `DSH_WEB_URL`, then
644
+ `~/.dsh/remote-web-ui-relay-bridge.json` (`upstream`), and reports
645
+ **UNREACHABLE** separately from **stale generation** so a wrong port is not
646
+ mistaken for an old module.
647
+ - The manual acceptance row now points at the in-conversation strip instead of
648
+ the opt-in popup.
649
+ - Live MCP capture was verified against the installed servers (context7 call
650
+ bucketed with latency, health and per-tool breakdown).
651
+
652
+ ## 1.0.0
653
+
654
+ First stable release. Everything below was built and verified inside one session
655
+ against DeepSeek Harness 0.2.2-rc.3; the acceptance report lives in
656
+ [docs/ACCEPTANCE.md](docs/ACCEPTANCE.md).
657
+
658
+ ### Recording
659
+
660
+ - Records every `skill` tool load with duration, success/failure and session id.
661
+ - Records user-typed `/skill-name` loads by reading `source.kind ===
662
+ 'skill-invocation'` messages on `agent/pre-step`, deduped by message id.
663
+ - Observation window per load: tool calls, tool errors, MCP calls, loop errors,
664
+ retried model requests, user corrections, sibling skill loads.
665
+ - Windows survive the turn boundary (the correction that proves a skill failed
666
+ usually arrives in the next user message) with a TTL sweep.
667
+ - Parallel loads inside one loop step stay open together instead of being scored
668
+ as a collision; a load in a later step is a real switch.
669
+ - Root vs subagent origin is recorded so short subagent runs do not count as
670
+ "ignored skill".
671
+ - MCP tools are bucketed per `mcp__<server>__<tool>` with latency samples.
672
+
673
+ ### Scoring
674
+
675
+ - Skill score: static 30% / reliability 25% / effectiveness 30% / efficiency 15%,
676
+ A–F bands, provisional + confidence labels.
677
+ - Static authoring checks (9, weight-summing 100): description, trigger
678
+ language, whenToUse, size, sections, progressive disclosure, placeholders,
679
+ examples.
680
+ - Runtime dimensions from load success, post-load tool error rate, loop errors,
681
+ user corrections, ignored loads, sibling collisions, cost.
682
+ - 95% bootstrap confidence interval per skill (seeded, reproducible) and per run.
683
+ - Ranking caveat when two skills' intervals overlap closely.
684
+ - Run score with its own dimensions (quality 0.40 / stability 0.30 / outcome
685
+ 0.20 / efficiency 0.10), interval, confidence, provisional.
686
+ - MCP health per server (`100 − 70·errorRate − slow penalty`) with p50/p95,
687
+ failing tools and MCP-specific fixes.
688
+ - Standard is user configuration: `weights`, `thresholds`, `minSamples`,
689
+ `ciResamples`, `mcpErrorRatePercent`, `mcpToolErrorRatePercent`, `mcpP95Ms`.
690
+ Weights normalise, a zero weight drops the dimension, unordered thresholds are
691
+ repaired, all-zero falls back to defaults.
692
+
693
+ ### Surfaces
694
+
695
+ - `skill_eval_report` tool (`skill`, `view: summary|mcp|runs|standard`, `format`).
696
+ - `/skill-eval` command (`<skill>`, `mcp`, `runs`, `standard`, `accept`).
697
+ - Sidebar footer action + modal overlay with 4 tabs (Skills / Runs / MCP / Dev),
698
+ 5 s auto-refresh, copy-as-markdown, Escape/backdrop close.
699
+ - HTML dashboard `GET /skill-eval` and JSON API `GET /skill-eval/data`
700
+ (plus a versioned alias).
701
+ - Per-turn snapshot `reports/latest.md` + `latest.json` (no model tokens).
702
+ - Cordis Inspect provider `skill-eval` (`summary|skill|mcp|runs|standard`).
703
+
704
+ ### Reliability and diagnostics
705
+
706
+ - Every listener is failure-contained; the two waterfalls always call `next()`.
707
+ - Telemetry never touches the durable session log; it lives in
708
+ `$DSH_HOME/skill-eval/<profile>/events.jsonl`.
709
+ - `dispose()` flushes queued writes (a real data-loss bug found by tests).
710
+ - `node scripts/accept.mjs` and `/skill-eval accept` detect a stale host
711
+ generation; the popup shows a red banner when its payload predates the boards.
712
+ - `DSH_SKILL_EVAL_DEBUG=1` mirrors diagnostics to stderr.
713
+
714
+ ### Tests
715
+
716
+ 118 tests, no build step: unit, integration through a fake host, jsdom rendering
717
+ of the real popup components, snapshot/acceptance, and seeded property tests over
718
+ the scoring pipeline.