dsh-skill-eval 0.0.0-stage → 1.20.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (48) hide show
  1. package/CHANGELOG.md +718 -0
  2. package/LICENSE +21 -0
  3. package/README.md +207 -2
  4. package/README.zh.md +164 -0
  5. package/client.js +2791 -0
  6. package/cordis.patch.yml +9 -0
  7. package/docs/ACCEPTANCE.md +71 -0
  8. package/docs/AUDIT.md +82 -0
  9. package/docs/CODE-REVIEW.md +49 -0
  10. package/docs/COMPARISON.md +33 -0
  11. package/docs/DETAIL.md +468 -0
  12. package/docs/DEV-RELOAD.md +114 -0
  13. package/docs/ENTRYPOINTS.md +65 -0
  14. package/docs/RELEASE.md +44 -0
  15. package/docs/RRSI-NOTES.md +129 -0
  16. package/docs/SCENARIOS.md +91 -0
  17. package/docs/UI-REVIEW.md +115 -0
  18. package/docs/WORKFLOW.md +108 -0
  19. package/icon.svg +9 -0
  20. package/index.js +1173 -0
  21. package/lib/accept.js +350 -0
  22. package/lib/advice.js +557 -0
  23. package/lib/critic.js +101 -0
  24. package/lib/dev.js +148 -0
  25. package/lib/history.js +141 -0
  26. package/lib/html.js +17 -0
  27. package/lib/inspect.js +142 -0
  28. package/lib/integrity.js +46 -0
  29. package/lib/loop.js +195 -0
  30. package/lib/random.js +29 -0
  31. package/lib/recorder.js +644 -0
  32. package/lib/render.js +580 -0
  33. package/lib/report.js +689 -0
  34. package/lib/score.js +565 -0
  35. package/lib/settings.js +415 -0
  36. package/lib/snapshot.js +250 -0
  37. package/lib/steps.js +141 -0
  38. package/lib/store.js +240 -0
  39. package/lib/validity.js +125 -0
  40. package/locale/en.json +6 -0
  41. package/locale/zh.json +6 -0
  42. package/package.json +96 -4
  43. package/scripts/accept.mjs +227 -0
  44. package/scripts/build-client.mjs +59 -0
  45. package/scripts/build.mjs +81 -0
  46. package/scripts/doctor.mjs +234 -0
  47. package/scripts/lib.mjs +87 -0
  48. package/scripts/verify-live.mjs +178 -0
package/docs/DETAIL.md ADDED
@@ -0,0 +1,468 @@
1
+ # dsh-skill-eval — design detail
2
+
3
+ ## Why this exists
4
+
5
+ Every published skill-evaluation tool we looked at is a **static, offline** checker:
6
+
7
+ | Project | What it does | What it cannot see |
8
+ | --------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- |
9
+ | `aws-samples/sample-agent-skill-eval` | audit (100 − deductions), functional/trigger runs through Claude CLI, weighted report 40/40/20, `.skilleval.yaml` | nothing about the skill as it is actually used in this harness |
10
+ | `NVIDIA/SkillEvaluator` | T1 static validation (SkillSpector, Gitleaks, Semgrep), T2 embedding dedup, T3 Docker-sandboxed live eval with an LLM rubric | needs a sandboxed re-run, not the real session |
11
+ | `darkrishabh/agent-skills-eval`, `alibaba/skill-up`, `adewale/skill-eval-harness` | test runners / evolution loops over fixtures | no runtime attachment to a live agent |
12
+
13
+ They all answer "is this skill file well-formed?". None of them can answer
14
+ "when this agent loaded this skill in this session, did it actually help?".
15
+ That second question is what this plugin measures, because inside DSH it can sit
16
+ on the real event stream.
17
+
18
+ The weighting and banding ideas (multi-dimensional rubric, A–F bands, a ~70
19
+ pass line) are borrowed from those projects. No code is copied.
20
+
21
+ ## What is recorded
22
+
23
+ Five append-only JSONL record types live in `$DSH_HOME/skill-eval/$DSH_PROFILE/events.jsonl`
24
+ (override with `dataDir` or `DSH_SKILL_EVAL_DIR`): `skill`, `window`, `static`
25
+ (one authoring profile per resolved skill), `step` (one row per agent step) and
26
+ `edit` (the review ledger). The session log is **never** written to — plugins
27
+ must not append foreign event types to it.
28
+
29
+ ```jsonc
30
+ // written when a `skill` tool call completes
31
+ { "t": "skill", "v": 1, "at": 1763, "sessionId": "…", "callId": "…",
32
+ "skill": "pdf-tools", "ok": true, "loadMs": 37, "windowId": "…" }
33
+
34
+ // written at turn end and again when the window closes
35
+ { "t": "window", "v": 1, "at": 1763, "windowId": "…", "sessionId": "…",
36
+ "skill": "pdf-tools", "toolCalls": 4, "errors": 1, "errorNames": ["ToolError"],
37
+ "corrections": ["不对,页眉没删掉"], "otherSkills": ["docx"], "turnEnded": true,
38
+ "truncated": false, "closeReason": "user-turn" }
39
+ ```
40
+
41
+ A **window** is the stretch of work that follows one skill load. It is the unit
42
+ of evidence. Readers keep the _last_ record per `windowId`, so the turn-end write
43
+ is an update and the close-time write is a second update — durability without
44
+ duplicates.
45
+
46
+ Window lifetime, deliberately unusual:
47
+
48
+ - opens on a successful `skill` load
49
+ - **stays open across the turn boundary** — the evidence that a skill failed is
50
+ usually the next user message ("no, that's wrong"), so closing at turn end
51
+ would throw away the strongest signal. The turn-end write only persists a
52
+ snapshot.
53
+ - closes on: the next user message (correction recorded first), a new skill load
54
+ in the same session (the previous skill records the new one as a sibling), TTL
55
+ expiry (`windowTtlMs`, default 30 min), session eviction, or plugin dispose.
56
+
57
+ ## Scoring
58
+
59
+ ```
60
+ total = Σ(weightᵢ × dimensionᵢ) / Σ(weightᵢ over dimensions with evidence)
61
+ ```
62
+
63
+ | dimension | weight | source |
64
+ | ------------- | ------ | --------------------------------------------------------------- |
65
+ | static | 0.30 | the live skill catalog (`skills.list` + `skills.get`) |
66
+ | reliability | 0.25 | load success × post-load tool error rate |
67
+ | effectiveness | 0.30 | user corrections, ignored loads, sibling collisions, error rate |
68
+ | efficiency | 0.15 | average tool calls per run, median load latency |
69
+
70
+ A dimension with no evidence is dropped and its weight redistributed, so a
71
+ brand-new skill is not punished for having no history; `provisional` marks
72
+ scores below `minSamples` loads.
73
+
74
+ ### static checks (117 points, normalised)
75
+
76
+ Twelve weighted checks; checks that cannot be evaluated (no filesystem view,
77
+ unresolved references) are marked skipped and dropped from both earned and
78
+ total weight, so the static score is always a percentage of what was checkable.
79
+
80
+ | check | weight | rule |
81
+ | ---------------------- | ------ | ------------------------------------------------------- |
82
+ | desc-length | 12 | description length within the useful band |
83
+ | desc-trigger | 12 | description says _when_ to use the skill |
84
+ | when-to-use | 8 | `whenToUse` present |
85
+ | body-present | 10 | SKILL.md body resolves and is non-trivial |
86
+ | body-compact | 18 | ≤ 500 body lines (`1 − (lines − 500)/1000` beyond that) |
87
+ | sections | 10 | ≥ 2 `##` headings |
88
+ | progressive-disclosure | 15 | delegates to `references/`, `scripts/` or `assets/` |
89
+ | no-placeholder | 10 | no TODO/TBD/FIXME/XXX |
90
+ | has-example | 5 | at least one fenced code block |
91
+ | anti-trigger | 5 | states when _not_ to use it |
92
+ | skill-name | 4 | kebab-case skill id |
93
+ | reference-integrity | 8 | every referenced file exists |
94
+
95
+ ### runtime dimensions
96
+
97
+ ```
98
+ reliability = 100 · loadOkRate · (0.6 + 0.4 · (1 − min(1, 2·errRate)))
99
+ effectiveness = 100 − 45·correctionRate − 25·abandonRate − 20·siblingRate − 10·errRate
100
+ efficiency = 100 − (avgCalls > 8 ? min(40, (avgCalls−8)·6) : 0) − (avgCalls < 1 ? 15 : 0)
101
+ − (p50LoadMs > 2000 ? min(30, (p50LoadMs−2000)/100) : 0)
102
+ − (tokensPerLoad > budget ? min(35, (tokensPerLoad−budget)/budget·30) : 0)
103
+ ```
104
+
105
+ A failed load multiplies reliability instead of only halving it: a skill that
106
+ cannot be loaded has zero reliability, not 40/100.
107
+
108
+ Bands: `A ≥ 90`, `B ≥ 80`, `C ≥ 70`, `D ≥ 60`, `F < 60` (configurable via
109
+ `thresholds`).
110
+
111
+ ## Advice
112
+
113
+ `lib/advice.js` is a rule engine over the same numbers. Every rule emits
114
+ `{id, severity, dimension, title, evidence, fix}` where `evidence` is the measured
115
+ value (or the user's own words) and `fix` is an instruction the agent can execute
116
+ against the skill file. Rules:
117
+
118
+ | id | severity | trigger |
119
+ | ------------------------------------------------------------------------- | -------- | ---------------------------- |
120
+ | load-failed | critical | ≥ 1 failed load |
121
+ | post-load-errors | critical | errRate > 25 % |
122
+ | user-corrections | critical | correctionRate > 30 % |
123
+ | dead-load | warning | abandonRate > 30 % |
124
+ | trigger-collision | warning | siblingRate > 30 % |
125
+ | verbose-execution | info | avgCalls > 8 |
126
+ | slow-load | info | p50LoadMs > 2000 |
127
+ | body-too-long / weak-description / no-trigger-language / placeholder-text | warning | matching static check |
128
+ | no-progressive-disclosure / no-example | info | matching static check |
129
+ | unresolved-skill / low-sample | info | catalog miss / too few loads |
130
+
131
+ ## Surfaces
132
+
133
+ - **Client seats** (`client.js`): the result strip under a completed turn
134
+ (`conversation.chat.turnTail`), the sidebar panel icon plus its main board
135
+ panel (`sidebar.panellist` + `main`), and the native settings form on the
136
+ bundle page (`plugins.bundle.config`). An opt-in debug popup
137
+ (`window.__SEV_ENABLE_POPUP__ = true`) additionally registers
138
+ `sidebar.footer.action` + `shell.overlay`. All surfaces read the same host
139
+ route, so the tool, the board and the popup show the same numbers.
140
+ - `skill_eval_report` tool — `{skill?, format?: 'markdown'|'json'}`. Registered
141
+ so the agent can read its own numbers and cite them; a system-prompt section
142
+ tells it to.
143
+ - `/skill-eval [skill]` command — the same report in the composer.
144
+ - **Turn snapshot** — after a turn settles, a debounced write puts the whole
145
+ aggregated report at `<dataDir>/reports/latest.md` and `latest.json`. Zero
146
+ tokens: it never enters the model context, it is just a file you (or the
147
+ agent, on request) can read.
148
+ - `GET /skill-eval` — HTML dashboard served through `webServer` (no build step,
149
+ self-contained, light/dark). `GET /skill-eval/data` returns the raw report
150
+ JSON, and is registered as its own **exact** route so it survives the page
151
+ route failing to re-register after a reload.
152
+
153
+ A Client half loads only when the host scans the package at boot, so installing
154
+ or changing `client.js` needs one DSH restart before the button appears.
155
+
156
+ ## Upgrading without breaking the dashboard
157
+
158
+ The host keeps `webServer` routes registered by a disposed plugin generation, so
159
+ after a hot reload the canonical `/skill-eval/data` can still be answered by the
160
+ old handler while the new module is running. Two measures keep that from
161
+ silently serving stale numbers:
162
+
163
+ 1. the data API is registered both canonically and under the package version
164
+ (`/skill-eval/data/<version>`), and the versioned path is new on every
165
+ release, so a reload can always serve current data;
166
+ 2. the popup asks the versioned path first and falls back to the canonical one
167
+ on 404, and a test pins the client constant to `package.json.version`.
168
+
169
+ `/skill-eval accept` runs the in-process half of the same checks from the
170
+ composer (route shape, boards, intervals, snapshot generation, telemetry), and
171
+ `node scripts/accept.mjs` checks both routes and reports a generation mismatch
172
+ instead of leaving you to guess. The popup does the same from inside: when the
173
+ payload it receives has no `scoring`/`ranking` fields it shows a red banner
174
+ ("the running host serves an older plugin generation … restart DeepSeek
175
+ Harness"), so a stale process is visible where the user actually looks.
176
+
177
+ ## Agent-loop wiring
178
+
179
+ Which loop stages feed the evidence, and which do not:
180
+
181
+ | loop stage | event | what the plugin does |
182
+ | --------------------- | --------------------------------- | -------------------------------------------------------------------------------------------------------------- |
183
+ | loop step | `agent/pre-step` (waterfall) | records the step number (parallel vs switched loads), then `next()` untouched |
184
+ | tool dispatch start | `tools/pre-execute` (waterfall) | timestamps `skill` loads only, then `next()` untouched |
185
+ | tool dispatch end | `tools/result` (emit) | call/error counting per open window, skill load records, static capture |
186
+ | user turn opens | `agent/inbox/claimed` (emit) | reads the message for correction language, then closes the windows |
187
+ | turn about to close | `agent/turn-stopping` (serial) | persists a snapshot, keeps the window open for the follow-up message |
188
+ | step/turn error | `agent/error` (emit) | `loopErrors` + `lastLoopError` on every open window; costs up to 40% of reliability and raises a critical rule |
189
+ | model request failure | `agent/request-error` (waterfall) | `llmRetries` per window, then `next()` untouched (the retry policy stays the host's) |
190
+
191
+ Token accounting **is** wired (`countTokens`): provider usage is attributed to
192
+ the open windows and `tokensPerLoad` feeds the efficiency dimension and the
193
+ token-cost advice. Deliberately **not** wired yet:
194
+
195
+ - **compaction** — a mid-window compaction does not break attribution, but the
196
+ window does not record that the model's context was rewritten.
197
+ - **turn truncation from max tokens** — a turn that dies on the output limit is
198
+ currently only visible through the following correction or error path.
199
+
200
+ Every listener above is read-only except the two waterfalls, which always call
201
+ `next()`, and every body is wrapped so telemetry can never fail a turn.
202
+
203
+ ## Concurrency and multi-agent work
204
+
205
+ Measured on the live host with the `workflow` tool (two `agent()` children in one
206
+ parallel batch, each loading the same skill):
207
+
208
+ - every child session produced its own `skill` record (distinct `sessionId`,
209
+ its own `loadMs`) plus its own `window` and `static` record — a root-scope
210
+ `tools/result` listener does receive child-agent dispatches;
211
+ - the two loads happened ~8 s apart and did not interfere, because recorder
212
+ state is keyed by session id;
213
+ - static authoring quality resolved inside the child agent's scope
214
+ (`skills.list({ scope: agent })`), so subagent loads are scored on the same
215
+ four dimensions as root loads.
216
+
217
+ Model rules that follow from that:
218
+
219
+ - **One window per load, several per session.** A load in a _later_ loop step
220
+ closes the previous window as `superseded` and records the newcomer in
221
+ `otherSkills` (that is the trigger-collision signal). Two loads inside the
222
+ _same_ loop step are parallel use: both windows stay open, and the follow-up
223
+ tool calls count for each of them, because one batch of work is not one
224
+ winner. The step comes from `agent/pre-step`.
225
+ - **Aggregation is per skill, across every session.** A skill loaded by a
226
+ subagent contributes to the same skill entry as a root load. The raw JSONL
227
+ keeps `sessionId`/`step` on every record, so a per-agent breakdown is a report
228
+ projection away, not a recording gap.
229
+ - **Correction evidence is weaker in child sessions.** The next user message in
230
+ a child is usually the parent agent's instruction, not human pushback, so the
231
+ correction rate mostly stays at zero there; effectiveness for subagent runs
232
+ leans on tool-error and abandoned-load rates instead.
233
+ - An interrupted child can leave its window open until the TTL sweep, which
234
+ shows up as an open window rather than a closed one. It is never attributed to
235
+ unrelated later work.
236
+
237
+ ### MCP thresholds are configuration
238
+
239
+ The three MCP rules use configurable limits so a deployment can be stricter than
240
+ the defaults (`20%` server error rate, `3000 ms` p95, `30%` single-tool error
241
+ rate):
242
+
243
+ ```yaml
244
+ config:
245
+ mcpErrorRatePercent: 20
246
+ mcpToolErrorRatePercent: 30
247
+ mcpP95Ms: 3000
248
+ ```
249
+
250
+ ### Validating with real MCP traffic
251
+
252
+ 1. Restart DSH once so the current plugin generation loads (profile plugins are
253
+ imported at boot), then run `node scripts/accept.mjs [baseUrl]`: it checks the
254
+ JSON route shape, the scoring block, the MCP/runs boards, the snapshot files
255
+ and the recorded telemetry, and prints the restart hint when the host is still
256
+ serving an older generation.
257
+ 2. Run a session that uses an MCP tool, e.g. "use context7 to look up the Rails
258
+ routing API".
259
+ 3. Open the popup → **MCP** tab: the server should appear with calls, error rate,
260
+ p50/p95, health and any per-tool failures.
261
+ 4. `cordis_inspect_query { provider: "skill-eval", method: "mcp" }` returns the
262
+ same board for scripting/agents.
263
+
264
+ A headless run (`dsh --profile <p> "task"`) can produce the same evidence, but it
265
+ needs model credentials in that profile (`DEEPSEEK_API_KEY`), which is why the
266
+ in-session validation above is done on the desktop host.
267
+
268
+ ## Run scores (every execution gets a grade)
269
+
270
+ The skill score grades an artifact; a **run score** grades one execution (one
271
+ session) so "how did this run go?" has an answer too. It is built from what the
272
+ session actually did, not from a re-run:
273
+
274
+ | dimension | weight | formula |
275
+ | ---------- | ------ | ------------------------------------------------------------------------ |
276
+ | quality | 0.40 | mean score of the skills this run loaded |
277
+ | stability | 0.30 | `100 − 60·toolErrorRate − 40·loopErrorRate` |
278
+ | outcome | 0.20 | `100 − 45·correctionRate` |
279
+ | efficiency | 0.10 | `100 − max(0, callsPerLoad − 8)·6`, capped at 45 when `callsPerLoad < 1` |
280
+
281
+ Weights differ from the skill score on purpose: a run is a sample of work, so
282
+ outcome and stability carry more than authoring quality. Each run score carries
283
+ `confidence` and `provisional` (fewer loads than `minSamples`), and the
284
+ `efficiency` cap stops "loaded a skill and did nothing" from looking perfect.
285
+ Run scores appear in the **Runs** board (popup), the markdown/HTML reports, the
286
+ `runs` inspect method and `/skill-eval runs`.
287
+
288
+ ## Developer mode: Cordis Inspect provider
289
+
290
+ `cordis_inspect_list` / `cordis_inspect_query` are the harness's read-only
291
+ developer capabilities. The plugin registers its own Host provider (default on,
292
+ `inspectProvider: false` to disable), so eval data is queryable next to the
293
+ harness's own `Service`/`Event`/`Config` providers:
294
+
295
+ ```jsonc
296
+ // cordis_inspect_query { platform: "host", provider: "skill-eval", method: "summary" }
297
+ { "totals": {...}, "scoring": {...},
298
+ "skills": [{ "skill": "pdf-tools", "score": 88, "grade": "B", "confidence": "medium",
299
+ "dimensions": {...}, "advice": [{ "id": "user-corrections", "severity": "critical" }] }] }
300
+ ```
301
+
302
+ | method | returns |
303
+ | ---------- | ---------------------------------------------------------------------------------------------- |
304
+ | `summary` | every skill: score, grade, confidence, four dimensions, advice identities (no prose — bounded) |
305
+ | `skill` | one skill in full, including advice text; errors when nothing is recorded |
306
+ | `mcp` | per-server calls, errors, error rate, p50/p95 latency, health, per-tool breakdown, MCP fixes |
307
+ | `edits` | screened skill edits from the review ledger (hypothesis, dimension, verdict, diff hash) |
308
+ | `runs` | recent per-session runs |
309
+ | `standard` | active weights, grade floors, minimum samples |
310
+
311
+ The registration is a real `cordisInspect.register({ manifest, query })` call:
312
+ the manifest carries `id`, `description` and methods with descriptions and JSON
313
+ schemas, and the query handler is read-only (it reuses the same analyzer as the
314
+ tool, the popup and the HTTP route, so every surface agrees by construction).
315
+
316
+ Text surfaces got the same views without leaving the conversation:
317
+ `skill_eval_report { view: "mcp" | "runs" | "standard" }` and
318
+ `/skill-eval mcp|runs|standard`.
319
+
320
+ ## Changing the grading standard
321
+
322
+ The standard is configuration, not code — a user can re-weight the four
323
+ dimensions and move the grade bands without touching the plugin:
324
+
325
+ ```yaml
326
+ config:
327
+ weights: { static: 30, reliability: 25, effectiveness: 30, efficiency: 15 }
328
+ thresholds: { excellent: 90, good: 80, pass: 70 }
329
+ minSamples: 3
330
+ ```
331
+
332
+ Rules the analyzer enforces so a hand-typed number cannot corrupt a score:
333
+
334
+ - weights are clamped to positive numbers and **normalised to 100%**; the
335
+ report shows the normalised values under `scoring.weightsPercent`;
336
+ - a weight of `0` drops that dimension entirely (weight redistribution already
337
+ handled it) — e.g. `static: 0` grades pure runtime behaviour;
338
+ - all-zero weights fall back to the defaults instead of dividing by zero;
339
+ - thresholds are re-ordered (`excellent ≥ good ≥ pass`), D stays 60, and any
340
+ value outside `1..100` falls back to the default.
341
+
342
+ Optional external basis: `normalizeWeights(input)` accepts either fractions
343
+ (`0.3`) or percentages (`30`), so a workflow can compute weights elsewhere.
344
+
345
+ ## Is the score valid?
346
+
347
+ The score is a _diagnostic index_, not a benchmark. Each dimension is anchored to
348
+ one observable signal, and the suite pins the direction of every dependency:
349
+
350
+ | dimension | anchored to | deliberately not |
351
+ | ----------------- | ------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- |
352
+ | static 30% | the skill definition itself (description, trigger language, sections, size, examples, progressive disclosure) | whether the content is _correct_ |
353
+ | reliability 25% | load success × post-load tool error rate × loop-error rate | error severity (a refused write and a flaky network count the same) |
354
+ | effectiveness 30% | user corrections, ignored loads, sibling collisions | task outcome (there is no ground truth per run) |
355
+ | efficiency 15% | tool calls per run, load latency, billable tokens per load | exact provider billing reconciliation |
356
+
357
+ Every skill score also carries a **95% bootstrap interval** over its recorded
358
+ loads (`score.ci = {low, high, resamples}`): load-level samples are resampled
359
+ with replacement 200 times, each draw is re-aggregated and re-scored, and the
360
+ 2.5/97.5 percentiles become the interval. That is why the report can say
361
+ "88.0 (79-94)" instead of pretending a 5-load point estimate is exact. The
362
+ resample count is configurable (`ciResamples`, 0 disables it), the RNG is seeded
363
+ per skill so the same data always yields the same interval, and fewer than two
364
+ loads yields no interval at all.
365
+
366
+ Intervals also drive a **ranking caveat**: when two adjacent skills' intervals
367
+ overlap and their totals are within 10 points, the report says so
368
+ (`ranking.note`) instead of letting a "worst skill" list imply an order the data
369
+ cannot support. Run scores get the same treatment: each run bootstraps its own
370
+ windows, so `62.5 (41.2–88.3)` shows how little a two-window run can prove.
371
+
372
+ Beyond the sensitivity tests, a property suite generates random windows with a
373
+ seeded RNG and asserts invariants over the real scoring pipeline: scores stay in
374
+ 0-100, the interval always contains the point estimate, one more tool error never
375
+ raises reliability, one more correction never raises effectiveness, custom
376
+ weights always normalise, the ranking caveat appears exactly when intervals
377
+ overlap closely, MCP advice appears exactly past its threshold, and the same
378
+ evidence always yields the same score and advice (reproducibility).
379
+
380
+ `node --test` contains sensitivity tests that inject exactly one bad signal at a
381
+ time and assert that _only_ the matching dimension moves:
382
+
383
+ - four tool errors out of eight calls → reliability drops, static unchanged
384
+ - three correcting messages → effectiveness drops, reliability unchanged
385
+ - twenty calls and a 9 s load → efficiency drops, effectiveness roughly stable
386
+ - three failed loads with a perfect body → reliability is 0 and the total falls
387
+ below 60, so good writing cannot rescue an unusable skill
388
+
389
+ Known validity limits, stated as limits rather than fixed:
390
+
391
+ 1. **Short sessions are now judged separately.** A subagent session that loads a
392
+ skill and immediately returns used to count as "ignored after load" — literally
393
+ true, but a 2-step child is not the same failure mode as a root session
394
+ dropping a skill. Windows now record `origin` (`subagent` or null) and the
395
+ abandonment rate is computed over root runs only; the denominator is exposed
396
+ as `abandonBase` so the report shows how much evidence it used.
397
+ 2. **Corrections are language-shaped.** The detector covers common Chinese and
398
+ English pushback; sarcasm, silent dissatisfaction, or a correction phrased as
399
+ a new request are missed.
400
+ 3. **No outcome ground truth.** Effectiveness is a proxy built from what the user
401
+ did next, not from a verified task verdict.
402
+ 4. **Provisional scores.** Below `minSamples` loads the number is marked
403
+ provisional on purpose; treat grades on 1–2 loads as anecdote.
404
+
405
+ ## Known blind spots
406
+
407
+ - Two load paths are recorded: the `skill` tool call (`via` absent, timed) and a
408
+ user-typed `/skill-name`, which the harness resolves without a tool call and
409
+ injects as a user message carrying `source.kind === 'skill-invocation'`
410
+ (`via: 'invocation'`, no `loadMs`). The message repeats on later steps, so it
411
+ is deduped by message id. Still invisible: content injected by another plugin
412
+ (`agent.inject()`), instructions baked into a preset's own prompt (not a skill
413
+ load at all), and any other profile or host, whose telemetry lives in its own
414
+ data directory.
415
+ - "Effectiveness" is inferred from the follow-up message, not from a task
416
+ outcome. A correct result followed by an unrelated complaint counts as a
417
+ correction; the TTL keeps that window narrow but cannot make it exact.
418
+ - Static checks read the _resolved_ catalog entry for the global layer. Skills
419
+ that only exist inside an agent-preset layer may resolve to `null` static
420
+ evidence (the runtime dimensions still work).
421
+ - The store grows without pruning. One record pair per skill load; a very busy
422
+ profile should eventually get a compaction pass.
423
+ - The dashboard route is served by `webServer` and cannot be reclaimed during a
424
+ hot reload in the same host process: after a disable/enable cycle the earlier
425
+ instance keeps answering `dashboardPath` (old handler, old state), and the new
426
+ registration is refused as a duplicate. Registration is therefore best-effort
427
+ and logged instead of thrown, and `dashboardPath` is a config knob so a
428
+ source edit can be previewed on a fresh path. A host restart re-registers the
429
+ canonical path cleanly.
430
+
431
+ ## Configuration keys
432
+
433
+ The config keys below are the ones worth documenting; the native form renders
434
+ the 25 tunable settings (the authoritative list is `SETTINGS_FIELDS`). Several
435
+ keys here are config-only (`dataDir`, `dashboardPath`, `registerTool`,
436
+ `inspectProvider`). The form persists to `settings.json`;
437
+ `dataDir`/`DSH_SKILL_EVAL_DIR` still wins for the data root.
438
+
439
+ ```yaml
440
+ enabled: true
441
+ windowToolCalls: 12 # tool calls that count as evidence for one load
442
+ minSamples: 3 # loads before a score stops being provisional
443
+ maxAdvice: 3 # advice items per skill in the summary
444
+ weights: { static: 30, reliability: 25, effectiveness: 30, efficiency: 15 } # normalised to 100%
445
+ thresholds: { excellent: 90, good: 80, pass: 70 } # A/B/C floors
446
+ windowTtlMs: 1800000 # a run older than this is not blamed on a later message
447
+ dataDir: ~/.dsh/skill-eval # optional override
448
+ turnReports: true # one aggregate snapshot per settled turn under reports/turns/
449
+ turnReportsKeep: 50 # how many per-turn boards to retain
450
+ reportOnTurnEnd: true # write reports/latest.md + latest.json after each turn (no tokens)
451
+ tokenBudgetPerLoad: 20000 # billable tokens per load before efficiency/advice react
452
+ cacheReadWeight: 0 # weight of cached input tokens (near-free by default)
453
+ countTokens: true # attribute real provider usage to skill windows
454
+ adviceNoiseFloor: true # advice must clear its 95% interval to stay critical
455
+ registerReviewTool: true # skill_eval_review: leakage critic + audit ledger
456
+ editBudgetMax: 4 # concurrent edits a first-round skill may carry
457
+ editBudgetMin: 1 # annealed minimum
458
+ editBudgetHorizon: 6 # accepted edits over which the budget anneals
459
+ noveltyWindow: 2 # same-dimension edits that trigger exploration
460
+ pruneMinLoads: 6 # loads before a prune recommendation is allowed
461
+ pruneMaxScore: 60 # low score + low interval -> prune candidate
462
+ promptHint: true # the one-line system-prompt hint (~60 tokens/turn); false = zero fixed cost
463
+ inspectProvider: true # register the eval data as a Cordis Inspect provider
464
+ registerTool: true
465
+ registerCommand: true
466
+ webDashboard: true # HTML page + the JSON route the client reads (versioned alias included)
467
+ dashboardPath: /skill-eval # change it if a reload could not reclaim the old route
468
+ ```
@@ -0,0 +1,114 @@
1
+ # 开发模式与热重载(给改插件的人看)
2
+
3
+ 这份文档解决一个反复踩的坑:**改了插件代码,界面上却看不到**。
4
+
5
+ ## 为什么改了没生效
6
+
7
+ DSH 的插件在启动时作为 ES module 加载。之后:
8
+
9
+ - **disable / enable 只会重跑 `apply`,不会重新 import 文件**——模块实例仍在 ESM 缓存里
10
+ - 因此 `PLUGIN_VERSION`、路由注册表、客户端模块清单都停留在启动那一刻
11
+ - 客户端(`client.js`)更严格:模块清单来自 boot 扫描,**刷新页面也不重新扫描**
12
+
13
+ 所以判断"是不是新代次",不要看文件 mtime,要探测**只有新一代才有的路由**:
14
+
15
+ ```bash
16
+ curl -s -o /dev/null -w '%{http_code} %{content_type}\n' \
17
+ http://127.0.0.1:<port>/skill-eval/data/<当前版本>
18
+ # application/json → 宿主就是这一代
19
+ # text/html → 命中旧前缀兜底路由,宿主还是旧代次
20
+ ```
21
+
22
+ `dsh-skill-eval-accept` 的 `versioned-route` 与 `settings-api` 两项就是在做这件事。
23
+ 更快的办法是在对话里跑 `/skill-eval dev`:它会直接对比**已加载模块版本**与
24
+ **磁盘 package.json 版本**,不一致就给出 `module-skew` 与重启提示。
25
+
26
+ ## 让热重载真正生效(已在本机验证)
27
+
28
+ 根因:插件模块是 ES module,禁用/启用只会重跑 `apply`,不会重新 `import` 文件;
29
+ 只有当 **Cordis HMR 监视到源文件变化**时,它才会让 loader 重新导入该模块。
30
+
31
+ 启用监视(一次性,写入 profile patch):
32
+
33
+ ```yaml
34
+ # ~/.dsh/profiles/desktop/cordis.patch.yml
35
+ - id: hmr
36
+ name: '@deepseek-ai/dsh-hmr'
37
+ config:
38
+ root:
39
+ - /Users/adams/Documents/dsh-plugins/dsh-skill-eval
40
+ ignored:
41
+ - '**/node_modules/**'
42
+ - '**/.git/**'
43
+ - '**/dist/**'
44
+ - 'cache'
45
+ - 'data'
46
+ ```
47
+
48
+ 之后每次改完源码:
49
+
50
+ ```bash
51
+ touch index.js lib/*.js client.js # 或任何真实改动
52
+ sleep 3 # HMR 默认 debounce + awaitWriteFinish
53
+ curl -s -o /dev/null -w '%{content_type}\n' \
54
+ http://127.0.0.1:<port>/skill-eval/data/<新版本>
55
+ # application/json → 新代次已生效(改版本号时用新版本号探;不改版本用旧版本号)
56
+ ```
57
+
58
+ 实测:1.15.2 → 1.15.3 → 1.15.4 都靠 HMR 完成,没有重启;客户端半个由
59
+ client HMR receiver 替换已构建的 bundle(改的是 `client/*.js`,先
60
+ `node scripts/build-client.mjs` 再 `touch client.js`;未替换时刷新一次页面)。
61
+
62
+ 注意两点:
63
+
64
+ - **canonical 路由可能仍由旧实例应答**(`/skill-eval/data`、`/skill-eval/settings`)。
65
+ 热重载后请用**版本化路由**(`/data/<version>`、`/settings/<version>`)做判断,
66
+ 客户端和 verify 脚本用的都是版本化路由。
67
+ - profile patch 改动本身也会被 HMR 监视并重排,所以启用监视不需要重启;
68
+ 但把 patch 写坏会导致下次启动失败,改前先备份(本仓库操作时已生成
69
+ `cordis.patch.yml.bak-skill-eval-<时间>`)。
70
+
71
+ ## 什么时候必须重启
72
+
73
+ | 改了什么 | 生效方式 |
74
+ | --------------------------------------- | -------------------------------------------------------------------------- |
75
+ | `index.js` / `lib/*`(host 半) | 正常情况:重启;极少数情况下 disable/enable 会重新 import,不要依赖 |
76
+ | `client/*.js`(client 源) | `node scripts/build-client.mjs` → `touch client.js`,HMR 替换已构建 bundle |
77
+ | `client.js`(生成产物) | 不要手改;由 `client/*.js` 构建,doctor 检查两者一致 |
78
+ | `package.json` 版本号 | 重启(版本化路由前缀由模块加载时的版本决定) |
79
+ | 数据文件(events.jsonl、settings.json) | 立即(每次请求重新读) |
80
+ | 由 `settings.json` 覆盖的配置 | 立即(保存即改内存中的 cfg) |
81
+
82
+ ## 给调试用的开关
83
+
84
+ | 开关 | 作用 |
85
+ | ------------------------------------ | --------------------------------------------------- |
86
+ | `window.__SEV_ENABLE_POPUP__ = true` | 恢复侧边栏弹窗模态框(默认只有常驻按钮打开看板) |
87
+ | `DSH_SKILL_EVAL_DEBUG=1` | host 侧把警告同时写 stderr |
88
+ | `DSH_SKILL_EVAL_DIR=<path>` | 覆盖数据目录(隔离实验数据) |
89
+ | `DSH_SKILL_EVAL_BASE=<path>` | 验收脚本使用的看板路径前缀(默认 `/skill-eval`) |
90
+ | `DSH_SKILL_EVAL_VERSION=<v>` | 验收脚本探测的版本化路由版本(默认取 package.json) |
91
+
92
+ ## 客户端与旧宿主兼容
93
+
94
+ 新客户端必须能跑在旧宿主返回的数据上(1.10.x 没有 `edits` / `validity`)。测试用
95
+ `test/fixtures/live-1.10.1.json`(真实宿主抓下来的载荷)锁定这一点;原生设置页在
96
+ 旧宿主上会显示"重启 DeepSeek Harness 后可用"并给出网页配置页链接,而不是白屏。
97
+
98
+ ## 一个可复制的验证流程
99
+
100
+ ```bash
101
+ dsh-skill-eval-verify # 重启后一键验证产品面(10 项)
102
+ dsh-skill-eval-accept # 探当前代次(端口自动发现)
103
+ curl -s http://127.0.0.1:<port>/skill-eval/data | head -c 200
104
+ # host 需重启时会出现:/data/<new-version> 返回 text/html
105
+
106
+ # 想在不动 GUI 进程的前提下验证 host 代码:起一个一次性 profile
107
+ dsh sevcheck --from-default-profile web --help
108
+ dsh plugin --profile sevcheck add "$PWD"
109
+ dsh --profile sevcheck --no-open --port 59231 &
110
+ node scripts/accept.mjs http://127.0.0.1:59231
111
+ kill %1 && rm -rf ~/.dsh/profiles/sevcheck
112
+ ```
113
+
114
+ 这条流程在 1.12.0/1.12.1 上实际用过:17 项 PASS,设置 API 的保存/归一化/400 都验证过。
@@ -0,0 +1,65 @@
1
+ # 入口 · Entry points
2
+
3
+ 三个常用入口,其余都是给 agent / 终端 / 调试用的。日常只需要记住前三行。
4
+ Three everyday entry points; everything else is for agents, terminals or debugging.
5
+
6
+ ## 日常 · Everyday
7
+
8
+ | 要做什么 · Goal | 中文入口 · Chinese | English entry | 说明 · Notes |
9
+ | --------------- | -------------------------------------------------- | --------------------------------------------------------------------- | ------------------------------------------------------------------- |
10
+ | 看这一轮的结果 | 每个完成回合下方的**结果条** | the **result strip** under a completed turn | 分数、调用/报错、下一步修复、改动记录;点「本轮详情」展开 |
11
+ | 打开看板 | 侧边栏底部的**面板图标「技能评测」** → 主面板 | the **panel icon "Skill eval"** in the sidebar → main panel | 与「上下文」插件同位置;面板内含 技能/轨迹/运行/MCP/开发者 五个标签 |
12
+ | 改评分标准 | **设置 → 插件 → `dsh-skill-eval` 的包页面 → 配置** | **Settings → Plugins → the `dsh-skill-eval` bundle page → Configure** | 与上下文插件同一位置(bundle config);权重、阈值、预算等保存即生效 |
13
+
14
+ ## 网页 · Web
15
+
16
+ | 用途 · Purpose | 地址 · URL |
17
+ | ----------------------------------------- | ----------------------------------------------------------------------------------- |
18
+ | 汇报看板 · Board | `http://127.0.0.1:<当前端口>/skill-eval`(热重载期间用 `/skill-eval/board/<版本>`) |
19
+ | 网页配置页(与设置页同源) · Web settings | `…/skill-eval/config` |
20
+ | JSON 数据 · JSON report | `…/skill-eval/data/<version>` |
21
+
22
+ 端口每次启动会变;`dsh-skill-eval-verify` 会自动发现。Port changes per launch; the verify script discovers it.
23
+
24
+ 看板与设置页支持中英双语:按浏览器 `Accept-Language` 自动选择,也可用 `?lang=zh|en` 强转;页面右上角有切换链接。建议正文(advice)目前仍是引擎输出的英文。
25
+
26
+ The board and settings page are bilingual: `Accept-Language` picks the language, `?lang=zh|en` overrides it, and a toggle link is on the page. Rule-generated advice text stays English for now.
27
+
28
+ ## 对话内 · In chat
29
+
30
+ | 命令 · Command | 作用 · What |
31
+ | ------------------------------------------------------- | ------------------------------------------- |
32
+ | `/skill-eval` | 总览 · cross-skill summary |
33
+ | `/skill-eval <skill>` | 单个技能 · one skill |
34
+ | `/skill-eval loop [all]` | 下一步改什么 · next optimization step |
35
+ | `/skill-eval validity` | 证据能支持什么 · what the evidence supports |
36
+ | `/skill-eval history <skill>` / `impact <skill>` | 趋势 / 改动前后 · trend / edit impact |
37
+ | `/skill-eval mcp` / `runs` / `turns` / `dev` / `config` | MCP / 运行 / 每次执行的板 / 诊断 / 生效配置 |
38
+
39
+ ## 给 agent · For agents
40
+
41
+ | 工具 · Tool | 作用 · What |
42
+ | ------------------- | --------------------------------------------------------------------------------- |
43
+ | `skill_eval_report` | 同一份报告(`view` 可选 summary/mcp/runs/loop/validity/history/impact/turns/dev) |
44
+ | `skill_eval_review` | 改技能前做泄漏审查并写 ledger · leak screen before editing a skill |
45
+
46
+ ## 终端 · Terminal
47
+
48
+ ```bash
49
+ dsh-skill-eval-doctor # 打包预检 19 项(--release 加产物检查)· packaging pre-flight
50
+ dsh-skill-eval-accept # 代次/数据契约 17 项 · generation & data contract
51
+ dsh-skill-eval-verify # 产品面可达 7 项 + 2 项手动 · product surfaces (after a reload)
52
+ ```
53
+
54
+ ## 调试(可选) · Debug (opt-in)
55
+
56
+ | 开关 · Switch | 作用 · What |
57
+ | --------------------------------------------- | ---------------------------------------------------------------------- |
58
+ | 页面里设 `window.__SEV_ENABLE_POPUP__ = true` | 弹窗 + 侧边栏按钮回来 · popup + sidebar button return |
59
+ | `/skill-eval dev` | 代次漂移、存储健康、已注册路由 · generation skew, store health, routes |
60
+
61
+ ## 刻意不做的入口 · Deliberately removed
62
+
63
+ - 插件详情页里再放一个区块(与看板重复)· a second section on the plugin detail page
64
+ - 侧边栏常驻按钮(默认不注册,避免和面板图标重复)· an always-on sidebar button
65
+ - 「更多」菜单里的看板行(看板已升为侧边栏主面板)· the More-menu board row (the board is now a main panel)