dsh-skill-eval 0.0.0-stage → 1.20.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (48) hide show
  1. package/CHANGELOG.md +675 -0
  2. package/LICENSE +21 -0
  3. package/README.md +200 -2
  4. package/README.zh.md +157 -0
  5. package/client.js +2637 -0
  6. package/cordis.patch.yml +9 -0
  7. package/docs/ACCEPTANCE.md +71 -0
  8. package/docs/AUDIT.md +64 -0
  9. package/docs/CODE-REVIEW.md +49 -0
  10. package/docs/COMPARISON.md +33 -0
  11. package/docs/DETAIL.md +463 -0
  12. package/docs/DEV-RELOAD.md +114 -0
  13. package/docs/ENTRYPOINTS.md +65 -0
  14. package/docs/RELEASE.md +44 -0
  15. package/docs/RRSI-NOTES.md +129 -0
  16. package/docs/SCENARIOS.md +91 -0
  17. package/docs/UI-REVIEW.md +115 -0
  18. package/docs/WORKFLOW.md +108 -0
  19. package/icon.svg +9 -0
  20. package/index.js +1111 -0
  21. package/lib/accept.js +350 -0
  22. package/lib/advice.js +547 -0
  23. package/lib/critic.js +101 -0
  24. package/lib/dev.js +148 -0
  25. package/lib/history.js +140 -0
  26. package/lib/html.js +8 -0
  27. package/lib/inspect.js +142 -0
  28. package/lib/integrity.js +46 -0
  29. package/lib/loop.js +195 -0
  30. package/lib/random.js +29 -0
  31. package/lib/recorder.js +634 -0
  32. package/lib/render.js +582 -0
  33. package/lib/report.js +646 -0
  34. package/lib/score.js +525 -0
  35. package/lib/settings.js +396 -0
  36. package/lib/snapshot.js +166 -0
  37. package/lib/steps.js +141 -0
  38. package/lib/store.js +186 -0
  39. package/lib/validity.js +115 -0
  40. package/locale/en.json +6 -0
  41. package/locale/zh.json +6 -0
  42. package/package.json +96 -4
  43. package/scripts/accept.mjs +226 -0
  44. package/scripts/build-client.mjs +59 -0
  45. package/scripts/build.mjs +81 -0
  46. package/scripts/doctor.mjs +199 -0
  47. package/scripts/lib.mjs +52 -0
  48. package/scripts/verify-live.mjs +168 -0
package/docs/DETAIL.md ADDED
@@ -0,0 +1,463 @@
1
+ # dsh-skill-eval — design detail
2
+
3
+ ## Why this exists
4
+
5
+ Every published skill-evaluation tool we looked at is a **static, offline** checker:
6
+
7
+ | Project | What it does | What it cannot see |
8
+ | --------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- |
9
+ | `aws-samples/sample-agent-skill-eval` | audit (100 − deductions), functional/trigger runs through Claude CLI, weighted report 40/40/20, `.skilleval.yaml` | nothing about the skill as it is actually used in this harness |
10
+ | `NVIDIA/SkillEvaluator` | T1 static validation (SkillSpector, Gitleaks, Semgrep), T2 embedding dedup, T3 Docker-sandboxed live eval with an LLM rubric | needs a sandboxed re-run, not the real session |
11
+ | `darkrishabh/agent-skills-eval`, `alibaba/skill-up`, `adewale/skill-eval-harness` | test runners / evolution loops over fixtures | no runtime attachment to a live agent |
12
+
13
+ They all answer "is this skill file well-formed?". None of them can answer
14
+ "when this agent loaded this skill in this session, did it actually help?".
15
+ That second question is what this plugin measures, because inside DSH it can sit
16
+ on the real event stream.
17
+
18
+ The weighting and banding ideas (multi-dimensional rubric, A–F bands, a ~70
19
+ pass line) are borrowed from those projects. No code is copied.
20
+
21
+ ## What is recorded
22
+
23
+ Two append-only JSONL record types live in `$DSH_HOME/skill-eval/$DSH_PROFILE/events.jsonl`
24
+ (override with `dataDir` or `DSH_SKILL_EVAL_DIR`). The session log is **never**
25
+ written to — plugins must not append foreign event types to it.
26
+
27
+ ```jsonc
28
+ // written when a `skill` tool call completes
29
+ { "t": "skill", "v": 1, "at": 1763, "sessionId": "…", "callId": "…",
30
+ "skill": "pdf-tools", "ok": true, "loadMs": 37, "windowId": "…" }
31
+
32
+ // written at turn end and again when the window closes
33
+ { "t": "window", "v": 1, "at": 1763, "windowId": "…", "sessionId": "…",
34
+ "skill": "pdf-tools", "toolCalls": 4, "errors": 1, "errorNames": ["ToolError"],
35
+ "corrections": ["不对,页眉没删掉"], "otherSkills": ["docx"], "turnEnded": true,
36
+ "truncated": false, "closeReason": "user-turn" }
37
+ ```
38
+
39
+ A **window** is the stretch of work that follows one skill load. It is the unit
40
+ of evidence. Readers keep the _last_ record per `windowId`, so the turn-end write
41
+ is an update and the close-time write is a second update — durability without
42
+ duplicates.
43
+
44
+ Window lifetime, deliberately unusual:
45
+
46
+ - opens on a successful `skill` load
47
+ - **stays open across the turn boundary** — the evidence that a skill failed is
48
+ usually the next user message ("no, that's wrong"), so closing at turn end
49
+ would throw away the strongest signal. The turn-end write only persists a
50
+ snapshot.
51
+ - closes on: the next user message (correction recorded first), a new skill load
52
+ in the same session (the previous skill records the new one as a sibling), TTL
53
+ expiry (`windowTtlMs`, default 30 min), session eviction, or plugin dispose.
54
+
55
+ ## Scoring
56
+
57
+ ```
58
+ total = Σ(weightᵢ × dimensionᵢ) / Σ(weightᵢ over dimensions with evidence)
59
+ ```
60
+
61
+ | dimension | weight | source |
62
+ | ------------- | ------ | --------------------------------------------------------------- |
63
+ | static | 0.30 | the live skill catalog (`skills.list` + `skills.get`) |
64
+ | reliability | 0.25 | load success × post-load tool error rate |
65
+ | effectiveness | 0.30 | user corrections, ignored loads, sibling collisions, error rate |
66
+ | efficiency | 0.15 | average tool calls per run, median load latency |
67
+
68
+ A dimension with no evidence is dropped and its weight redistributed, so a
69
+ brand-new skill is not punished for having no history; `provisional` marks
70
+ scores below `minSamples` loads.
71
+
72
+ ### static checks (117 points, normalised)
73
+
74
+ Twelve weighted checks; checks that cannot be evaluated (no filesystem view,
75
+ unresolved references) are marked skipped and dropped from both earned and
76
+ total weight, so the static score is always a percentage of what was checkable.
77
+
78
+ | check | weight | rule |
79
+ | ---------------------- | ------ | ------------------------------------------------------- |
80
+ | desc-length | 12 | description length within the useful band |
81
+ | desc-trigger | 12 | description says _when_ to use the skill |
82
+ | when-to-use | 8 | `whenToUse` present |
83
+ | body-present | 10 | SKILL.md body resolves and is non-trivial |
84
+ | body-compact | 18 | ≤ 500 body lines (`1 − (lines − 500)/1000` beyond that) |
85
+ | sections | 10 | ≥ 2 `##` headings |
86
+ | progressive-disclosure | 15 | delegates to `references/`, `scripts/` or `assets/` |
87
+ | no-placeholder | 10 | no TODO/TBD/FIXME/XXX |
88
+ | has-example | 5 | at least one fenced code block |
89
+ | anti-trigger | 5 | states when _not_ to use it |
90
+ | skill-name | 4 | kebab-case skill id |
91
+ | reference-integrity | 8 | every referenced file exists |
92
+
93
+ ### runtime dimensions
94
+
95
+ ```
96
+ reliability = 100 · loadOkRate · (0.6 + 0.4 · (1 − min(1, 2·errRate)))
97
+ effectiveness = 100 − 45·correctionRate − 25·abandonRate − 20·siblingRate − 10·errRate
98
+ efficiency = 100 − (avgCalls > 8 ? min(40, (avgCalls−8)·6) : 0) − (avgCalls < 1 ? 15 : 0)
99
+ − (p50LoadMs > 2000 ? min(30, (p50LoadMs−2000)/100) : 0)
100
+ − (tokensPerLoad > budget ? min(35, (tokensPerLoad−budget)/budget·30) : 0)
101
+ ```
102
+
103
+ A failed load multiplies reliability instead of only halving it: a skill that
104
+ cannot be loaded has zero reliability, not 40/100.
105
+
106
+ Bands: `A ≥ 90`, `B ≥ 80`, `C ≥ 70`, `D ≥ 60`, `F < 60` (configurable via
107
+ `thresholds`).
108
+
109
+ ## Advice
110
+
111
+ `lib/advice.js` is a rule engine over the same numbers. Every rule emits
112
+ `{id, severity, dimension, title, evidence, fix}` where `evidence` is the measured
113
+ value (or the user's own words) and `fix` is an instruction the agent can execute
114
+ against the skill file. Rules:
115
+
116
+ | id | severity | trigger |
117
+ | ------------------------------------------------------------------------- | -------- | ---------------------------- |
118
+ | load-failed | critical | ≥ 1 failed load |
119
+ | post-load-errors | critical | errRate > 25 % |
120
+ | user-corrections | critical | correctionRate > 30 % |
121
+ | dead-load | warning | abandonRate > 30 % |
122
+ | trigger-collision | warning | siblingRate > 30 % |
123
+ | verbose-execution | info | avgCalls > 8 |
124
+ | slow-load | info | p50LoadMs > 2000 |
125
+ | body-too-long / weak-description / no-trigger-language / placeholder-text | warning | matching static check |
126
+ | no-progressive-disclosure / no-example | info | matching static check |
127
+ | unresolved-skill / low-sample | info | catalog miss / too few loads |
128
+
129
+ ## Surfaces
130
+
131
+ - **Client seats** (`client.js`): the result strip under a completed turn
132
+ (`conversation.chat.turnTail`), the sidebar panel icon plus its main board
133
+ panel (`sidebar.panellist` + `main`), and the native settings form on the
134
+ bundle page (`plugins.bundle.config`). An opt-in debug popup
135
+ (`window.__SEV_ENABLE_POPUP__ = true`) additionally registers
136
+ `sidebar.footer.action` + `shell.overlay`. All surfaces read the same host
137
+ route, so the tool, the board and the popup show the same numbers.
138
+ - `skill_eval_report` tool — `{skill?, format?: 'markdown'|'json'}`. Registered
139
+ so the agent can read its own numbers and cite them; a system-prompt section
140
+ tells it to.
141
+ - `/skill-eval [skill]` command — the same report in the composer.
142
+ - **Turn snapshot** — after a turn settles, a debounced write puts the whole
143
+ aggregated report at `<dataDir>/reports/latest.md` and `latest.json`. Zero
144
+ tokens: it never enters the model context, it is just a file you (or the
145
+ agent, on request) can read.
146
+ - `GET /skill-eval` — HTML dashboard served through `webServer` (no build step,
147
+ self-contained, light/dark). `GET /skill-eval/data` returns the raw report
148
+ JSON, and is registered as its own **exact** route so it survives the page
149
+ route failing to re-register after a reload.
150
+
151
+ A Client half loads only when the host scans the package at boot, so installing
152
+ or changing `client.js` needs one DSH restart before the button appears.
153
+
154
+ ## Upgrading without breaking the dashboard
155
+
156
+ The host keeps `webServer` routes registered by a disposed plugin generation, so
157
+ after a hot reload the canonical `/skill-eval/data` can still be answered by the
158
+ old handler while the new module is running. Two measures keep that from
159
+ silently serving stale numbers:
160
+
161
+ 1. the data API is registered both canonically and under the package version
162
+ (`/skill-eval/data/<version>`), and the versioned path is new on every
163
+ release, so a reload can always serve current data;
164
+ 2. the popup asks the versioned path first and falls back to the canonical one
165
+ on 404, and a test pins the client constant to `package.json.version`.
166
+
167
+ `/skill-eval accept` runs the in-process half of the same checks from the
168
+ composer (route shape, boards, intervals, snapshot generation, telemetry), and
169
+ `node scripts/accept.mjs` checks both routes and reports a generation mismatch
170
+ instead of leaving you to guess. The popup does the same from inside: when the
171
+ payload it receives has no `scoring`/`ranking` fields it shows a red banner
172
+ ("the running host serves an older plugin generation … restart DeepSeek
173
+ Harness"), so a stale process is visible where the user actually looks.
174
+
175
+ ## Agent-loop wiring
176
+
177
+ Which loop stages feed the evidence, and which do not:
178
+
179
+ | loop stage | event | what the plugin does |
180
+ | --------------------- | --------------------------------- | -------------------------------------------------------------------------------------------------------------- |
181
+ | loop step | `agent/pre-step` (waterfall) | records the step number (parallel vs switched loads), then `next()` untouched |
182
+ | tool dispatch start | `tools/pre-execute` (waterfall) | timestamps `skill` loads only, then `next()` untouched |
183
+ | tool dispatch end | `tools/result` (emit) | call/error counting per open window, skill load records, static capture |
184
+ | user turn opens | `agent/inbox/claimed` (emit) | reads the message for correction language, then closes the windows |
185
+ | turn about to close | `agent/turn-stopping` (serial) | persists a snapshot, keeps the window open for the follow-up message |
186
+ | step/turn error | `agent/error` (emit) | `loopErrors` + `lastLoopError` on every open window; costs up to 40% of reliability and raises a critical rule |
187
+ | model request failure | `agent/request-error` (waterfall) | `llmRetries` per window, then `next()` untouched (the retry policy stays the host's) |
188
+
189
+ Token accounting **is** wired (`countTokens`): provider usage is attributed to
190
+ the open windows and `tokensPerLoad` feeds the efficiency dimension and the
191
+ token-cost advice. Deliberately **not** wired yet:
192
+
193
+ - **compaction** — a mid-window compaction does not break attribution, but the
194
+ window does not record that the model's context was rewritten.
195
+ - **turn truncation from max tokens** — a turn that dies on the output limit is
196
+ currently only visible through the following correction or error path.
197
+
198
+ Every listener above is read-only except the two waterfalls, which always call
199
+ `next()`, and every body is wrapped so telemetry can never fail a turn.
200
+
201
+ ## Concurrency and multi-agent work
202
+
203
+ Measured on the live host with the `workflow` tool (two `agent()` children in one
204
+ parallel batch, each loading the same skill):
205
+
206
+ - every child session produced its own `skill` record (distinct `sessionId`,
207
+ its own `loadMs`) plus its own `window` and `static` record — a root-scope
208
+ `tools/result` listener does receive child-agent dispatches;
209
+ - the two loads happened ~8 s apart and did not interfere, because recorder
210
+ state is keyed by session id;
211
+ - static authoring quality resolved inside the child agent's scope
212
+ (`skills.list({ scope: agent })`), so subagent loads are scored on the same
213
+ four dimensions as root loads.
214
+
215
+ Model rules that follow from that:
216
+
217
+ - **One window per load, several per session.** A load in a _later_ loop step
218
+ closes the previous window as `superseded` and records the newcomer in
219
+ `otherSkills` (that is the trigger-collision signal). Two loads inside the
220
+ _same_ loop step are parallel use: both windows stay open, and the follow-up
221
+ tool calls count for each of them, because one batch of work is not one
222
+ winner. The step comes from `agent/pre-step`.
223
+ - **Aggregation is per skill, across every session.** A skill loaded by a
224
+ subagent contributes to the same skill entry as a root load. The raw JSONL
225
+ keeps `sessionId`/`step` on every record, so a per-agent breakdown is a report
226
+ projection away, not a recording gap.
227
+ - **Correction evidence is weaker in child sessions.** The next user message in
228
+ a child is usually the parent agent's instruction, not human pushback, so the
229
+ correction rate mostly stays at zero there; effectiveness for subagent runs
230
+ leans on tool-error and abandoned-load rates instead.
231
+ - An interrupted child can leave its window open until the TTL sweep, which
232
+ shows up as an open window rather than a closed one. It is never attributed to
233
+ unrelated later work.
234
+
235
+ ### MCP thresholds are configuration
236
+
237
+ The three MCP rules use configurable limits so a deployment can be stricter than
238
+ the defaults (`20%` server error rate, `3000 ms` p95, `30%` single-tool error
239
+ rate):
240
+
241
+ ```yaml
242
+ config:
243
+ mcpErrorRatePercent: 20
244
+ mcpToolErrorRatePercent: 30
245
+ mcpP95Ms: 3000
246
+ ```
247
+
248
+ ### Validating with real MCP traffic
249
+
250
+ 1. Restart DSH once so the current plugin generation loads (profile plugins are
251
+ imported at boot), then run `node scripts/accept.mjs [baseUrl]`: it checks the
252
+ JSON route shape, the scoring block, the MCP/runs boards, the snapshot files
253
+ and the recorded telemetry, and prints the restart hint when the host is still
254
+ serving an older generation.
255
+ 2. Run a session that uses an MCP tool, e.g. "use context7 to look up the Rails
256
+ routing API".
257
+ 3. Open the popup → **MCP** tab: the server should appear with calls, error rate,
258
+ p50/p95, health and any per-tool failures.
259
+ 4. `cordis_inspect_query { provider: "skill-eval", method: "mcp" }` returns the
260
+ same board for scripting/agents.
261
+
262
+ A headless run (`dsh --profile <p> "task"`) can produce the same evidence, but it
263
+ needs model credentials in that profile (`DEEPSEEK_API_KEY`), which is why the
264
+ in-session validation above is done on the desktop host.
265
+
266
+ ## Run scores (every execution gets a grade)
267
+
268
+ The skill score grades an artifact; a **run score** grades one execution (one
269
+ session) so "how did this run go?" has an answer too. It is built from what the
270
+ session actually did, not from a re-run:
271
+
272
+ | dimension | weight | formula |
273
+ | ---------- | ------ | ------------------------------------------------------------------------ |
274
+ | quality | 0.40 | mean score of the skills this run loaded |
275
+ | stability | 0.30 | `100 − 60·toolErrorRate − 40·loopErrorRate` |
276
+ | outcome | 0.20 | `100 − 45·correctionRate` |
277
+ | efficiency | 0.10 | `100 − max(0, callsPerLoad − 8)·6`, capped at 45 when `callsPerLoad < 1` |
278
+
279
+ Weights differ from the skill score on purpose: a run is a sample of work, so
280
+ outcome and stability carry more than authoring quality. Each run score carries
281
+ `confidence` and `provisional` (fewer loads than `minSamples`), and the
282
+ `efficiency` cap stops "loaded a skill and did nothing" from looking perfect.
283
+ Run scores appear in the **Runs** board (popup), the markdown/HTML reports, the
284
+ `runs` inspect method and `/skill-eval runs`.
285
+
286
+ ## Developer mode: Cordis Inspect provider
287
+
288
+ `cordis_inspect_list` / `cordis_inspect_query` are the harness's read-only
289
+ developer capabilities. The plugin registers its own Host provider (default on,
290
+ `inspectProvider: false` to disable), so eval data is queryable next to the
291
+ harness's own `Service`/`Event`/`Config` providers:
292
+
293
+ ```jsonc
294
+ // cordis_inspect_query { platform: "host", provider: "skill-eval", method: "summary" }
295
+ { "totals": {...}, "scoring": {...},
296
+ "skills": [{ "skill": "pdf-tools", "score": 88, "grade": "B", "confidence": "medium",
297
+ "dimensions": {...}, "advice": [{ "id": "user-corrections", "severity": "critical" }] }] }
298
+ ```
299
+
300
+ | method | returns |
301
+ | ---------- | ---------------------------------------------------------------------------------------------- |
302
+ | `summary` | every skill: score, grade, confidence, four dimensions, advice identities (no prose — bounded) |
303
+ | `skill` | one skill in full, including advice text; errors when nothing is recorded |
304
+ | `mcp` | per-server calls, errors, error rate, p50/p95 latency, health, per-tool breakdown, MCP fixes |
305
+ | `edits` | screened skill edits from the review ledger (hypothesis, dimension, verdict, diff hash) |
306
+ | `runs` | recent per-session runs |
307
+ | `standard` | active weights, grade floors, minimum samples |
308
+
309
+ The registration is a real `cordisInspect.register({ manifest, query })` call:
310
+ the manifest carries `id`, `description` and methods with descriptions and JSON
311
+ schemas, and the query handler is read-only (it reuses the same analyzer as the
312
+ tool, the popup and the HTTP route, so every surface agrees by construction).
313
+
314
+ Text surfaces got the same views without leaving the conversation:
315
+ `skill_eval_report { view: "mcp" | "runs" | "standard" }` and
316
+ `/skill-eval mcp|runs|standard`.
317
+
318
+ ## Changing the grading standard
319
+
320
+ The standard is configuration, not code — a user can re-weight the four
321
+ dimensions and move the grade bands without touching the plugin:
322
+
323
+ ```yaml
324
+ config:
325
+ weights: { static: 30, reliability: 25, effectiveness: 30, efficiency: 15 }
326
+ thresholds: { excellent: 90, good: 80, pass: 70 }
327
+ minSamples: 3
328
+ ```
329
+
330
+ Rules the analyzer enforces so a hand-typed number cannot corrupt a score:
331
+
332
+ - weights are clamped to positive numbers and **normalised to 100%**; the
333
+ report shows the normalised values under `scoring.weightsPercent`;
334
+ - a weight of `0` drops that dimension entirely (weight redistribution already
335
+ handled it) — e.g. `static: 0` grades pure runtime behaviour;
336
+ - all-zero weights fall back to the defaults instead of dividing by zero;
337
+ - thresholds are re-ordered (`excellent ≥ good ≥ pass`), D stays 60, and any
338
+ value outside `1..100` falls back to the default.
339
+
340
+ Optional external basis: `normalizeWeights(input)` accepts either fractions
341
+ (`0.3`) or percentages (`30`), so a workflow can compute weights elsewhere.
342
+
343
+ ## Is the score valid?
344
+
345
+ The score is a _diagnostic index_, not a benchmark. Each dimension is anchored to
346
+ one observable signal, and the suite pins the direction of every dependency:
347
+
348
+ | dimension | anchored to | deliberately not |
349
+ | ----------------- | ------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- |
350
+ | static 30% | the skill definition itself (description, trigger language, sections, size, examples, progressive disclosure) | whether the content is _correct_ |
351
+ | reliability 25% | load success × post-load tool error rate × loop-error rate | error severity (a refused write and a flaky network count the same) |
352
+ | effectiveness 30% | user corrections, ignored loads, sibling collisions | task outcome (there is no ground truth per run) |
353
+ | efficiency 15% | tool calls per run, load latency, billable tokens per load | exact provider billing reconciliation |
354
+
355
+ Every skill score also carries a **95% bootstrap interval** over its recorded
356
+ loads (`score.ci = {low, high, resamples}`): load-level samples are resampled
357
+ with replacement 200 times, each draw is re-aggregated and re-scored, and the
358
+ 2.5/97.5 percentiles become the interval. That is why the report can say
359
+ "88.0 (79-94)" instead of pretending a 5-load point estimate is exact. The
360
+ resample count is configurable (`ciResamples`, 0 disables it), the RNG is seeded
361
+ per skill so the same data always yields the same interval, and fewer than two
362
+ loads yields no interval at all.
363
+
364
+ Intervals also drive a **ranking caveat**: when two adjacent skills' intervals
365
+ overlap and their totals are within 10 points, the report says so
366
+ (`ranking.note`) instead of letting a "worst skill" list imply an order the data
367
+ cannot support. Run scores get the same treatment: each run bootstraps its own
368
+ windows, so `62.5 (41.2–88.3)` shows how little a two-window run can prove.
369
+
370
+ Beyond the sensitivity tests, a property suite generates random windows with a
371
+ seeded RNG and asserts invariants over the real scoring pipeline: scores stay in
372
+ 0-100, the interval always contains the point estimate, one more tool error never
373
+ raises reliability, one more correction never raises effectiveness, custom
374
+ weights always normalise, the ranking caveat appears exactly when intervals
375
+ overlap closely, MCP advice appears exactly past its threshold, and the same
376
+ evidence always yields the same score and advice (reproducibility).
377
+
378
+ `node --test` contains sensitivity tests that inject exactly one bad signal at a
379
+ time and assert that _only_ the matching dimension moves:
380
+
381
+ - four tool errors out of eight calls → reliability drops, static unchanged
382
+ - three correcting messages → effectiveness drops, reliability unchanged
383
+ - twenty calls and a 9 s load → efficiency drops, effectiveness roughly stable
384
+ - three failed loads with a perfect body → reliability is 0 and the total falls
385
+ below 60, so good writing cannot rescue an unusable skill
386
+
387
+ Known validity limits, stated as limits rather than fixed:
388
+
389
+ 1. **Short sessions are now judged separately.** A subagent session that loads a
390
+ skill and immediately returns used to count as "ignored after load" — literally
391
+ true, but a 2-step child is not the same failure mode as a root session
392
+ dropping a skill. Windows now record `origin` (`subagent` or null) and the
393
+ abandonment rate is computed over root runs only; the denominator is exposed
394
+ as `abandonBase` so the report shows how much evidence it used.
395
+ 2. **Corrections are language-shaped.** The detector covers common Chinese and
396
+ English pushback; sarcasm, silent dissatisfaction, or a correction phrased as
397
+ a new request are missed.
398
+ 3. **No outcome ground truth.** Effectiveness is a proxy built from what the user
399
+ did next, not from a verified task verdict.
400
+ 4. **Provisional scores.** Below `minSamples` loads the number is marked
401
+ provisional on purpose; treat grades on 1–2 loads as anecdote.
402
+
403
+ ## Known blind spots
404
+
405
+ - Two load paths are recorded: the `skill` tool call (`via` absent, timed) and a
406
+ user-typed `/skill-name`, which the harness resolves without a tool call and
407
+ injects as a user message carrying `source.kind === 'skill-invocation'`
408
+ (`via: 'invocation'`, no `loadMs`). The message repeats on later steps, so it
409
+ is deduped by message id. Still invisible: content injected by another plugin
410
+ (`agent.inject()`), instructions baked into a preset's own prompt (not a skill
411
+ load at all), and any other profile or host, whose telemetry lives in its own
412
+ data directory.
413
+ - "Effectiveness" is inferred from the follow-up message, not from a task
414
+ outcome. A correct result followed by an unrelated complaint counts as a
415
+ correction; the TTL keeps that window narrow but cannot make it exact.
416
+ - Static checks read the _resolved_ catalog entry for the global layer. Skills
417
+ that only exist inside an agent-preset layer may resolve to `null` static
418
+ evidence (the runtime dimensions still work).
419
+ - The store grows without pruning. One record pair per skill load; a very busy
420
+ profile should eventually get a compaction pass.
421
+ - The dashboard route is served by `webServer` and cannot be reclaimed during a
422
+ hot reload in the same host process: after a disable/enable cycle the earlier
423
+ instance keeps answering `dashboardPath` (old handler, old state), and the new
424
+ registration is refused as a duplicate. Registration is therefore best-effort
425
+ and logged instead of thrown, and `dashboardPath` is a config knob so a
426
+ source edit can be previewed on a fresh path. A host restart re-registers the
427
+ canonical path cleanly.
428
+
429
+ ## Configuration keys
430
+
431
+ All 25 settings, in the order the native form renders them. The form persists to
432
+ `settings.json`; `dataDir`/`DSH_SKILL_EVAL_DIR` still wins for the data root.
433
+
434
+ ```yaml
435
+ enabled: true
436
+ windowToolCalls: 12 # tool calls that count as evidence for one load
437
+ minSamples: 3 # loads before a score stops being provisional
438
+ maxAdvice: 3 # advice items per skill in the summary
439
+ weights: { static: 30, reliability: 25, effectiveness: 30, efficiency: 15 } # normalised to 100%
440
+ thresholds: { excellent: 90, good: 80, pass: 70 } # A/B/C floors
441
+ windowTtlMs: 1800000 # a run older than this is not blamed on a later message
442
+ dataDir: ~/.dsh/skill-eval # optional override
443
+ turnReports: true # one board per settled turn under reports/turns/
444
+ turnReportsKeep: 50 # how many per-turn boards to retain
445
+ reportOnTurnEnd: true # write reports/latest.md + latest.json after each turn (no tokens)
446
+ tokenBudgetPerLoad: 20000 # billable tokens per load before efficiency/advice react
447
+ cacheReadWeight: 0 # weight of cached input tokens (near-free by default)
448
+ countTokens: true # attribute real provider usage to skill windows
449
+ adviceNoiseFloor: true # advice must clear its 95% interval to stay critical
450
+ registerReviewTool: true # skill_eval_review: leakage critic + audit ledger
451
+ editBudgetMax: 4 # concurrent edits a first-round skill may carry
452
+ editBudgetMin: 1 # annealed minimum
453
+ editBudgetHorizon: 6 # accepted edits over which the budget anneals
454
+ noveltyWindow: 2 # same-dimension edits that trigger exploration
455
+ pruneMinLoads: 6 # loads before a prune recommendation is allowed
456
+ pruneMaxScore: 60 # low score + low interval -> prune candidate
457
+ promptHint: true # the one-line system-prompt hint (~60 tokens/turn); false = zero fixed cost
458
+ inspectProvider: true # register the eval data as a Cordis Inspect provider
459
+ registerTool: true
460
+ registerCommand: true
461
+ webDashboard: true # HTML page + the JSON route the client reads (versioned alias included)
462
+ dashboardPath: /skill-eval # change it if a reload could not reclaim the old route
463
+ ```
@@ -0,0 +1,114 @@
1
+ # 开发模式与热重载(给改插件的人看)
2
+
3
+ 这份文档解决一个反复踩的坑:**改了插件代码,界面上却看不到**。
4
+
5
+ ## 为什么改了没生效
6
+
7
+ DSH 的插件在启动时作为 ES module 加载。之后:
8
+
9
+ - **disable / enable 只会重跑 `apply`,不会重新 import 文件**——模块实例仍在 ESM 缓存里
10
+ - 因此 `PLUGIN_VERSION`、路由注册表、客户端模块清单都停留在启动那一刻
11
+ - 客户端(`client.js`)更严格:模块清单来自 boot 扫描,**刷新页面也不重新扫描**
12
+
13
+ 所以判断"是不是新代次",不要看文件 mtime,要探测**只有新一代才有的路由**:
14
+
15
+ ```bash
16
+ curl -s -o /dev/null -w '%{http_code} %{content_type}\n' \
17
+ http://127.0.0.1:<port>/skill-eval/data/<当前版本>
18
+ # application/json → 宿主就是这一代
19
+ # text/html → 命中旧前缀兜底路由,宿主还是旧代次
20
+ ```
21
+
22
+ `dsh-skill-eval-accept` 的 `versioned-route` 与 `settings-api` 两项就是在做这件事。
23
+ 更快的办法是在对话里跑 `/skill-eval dev`:它会直接对比**已加载模块版本**与
24
+ **磁盘 package.json 版本**,不一致就给出 `module-skew` 与重启提示。
25
+
26
+ ## 让热重载真正生效(已在本机验证)
27
+
28
+ 根因:插件模块是 ES module,禁用/启用只会重跑 `apply`,不会重新 `import` 文件;
29
+ 只有当 **Cordis HMR 监视到源文件变化**时,它才会让 loader 重新导入该模块。
30
+
31
+ 启用监视(一次性,写入 profile patch):
32
+
33
+ ```yaml
34
+ # ~/.dsh/profiles/desktop/cordis.patch.yml
35
+ - id: hmr
36
+ name: '@deepseek-ai/dsh-hmr'
37
+ config:
38
+ root:
39
+ - /Users/adams/Documents/dsh-plugins/dsh-skill-eval
40
+ ignored:
41
+ - '**/node_modules/**'
42
+ - '**/.git/**'
43
+ - '**/dist/**'
44
+ - 'cache'
45
+ - 'data'
46
+ ```
47
+
48
+ 之后每次改完源码:
49
+
50
+ ```bash
51
+ touch index.js lib/*.js client.js # 或任何真实改动
52
+ sleep 3 # HMR 默认 debounce + awaitWriteFinish
53
+ curl -s -o /dev/null -w '%{content_type}\n' \
54
+ http://127.0.0.1:<port>/skill-eval/data/<新版本>
55
+ # application/json → 新代次已生效(改版本号时用新版本号探;不改版本用旧版本号)
56
+ ```
57
+
58
+ 实测:1.15.2 → 1.15.3 → 1.15.4 都靠 HMR 完成,没有重启;客户端半个由
59
+ client HMR receiver 替换已构建的 bundle(改的是 `client/*.js`,先
60
+ `node scripts/build-client.mjs` 再 `touch client.js`;未替换时刷新一次页面)。
61
+
62
+ 注意两点:
63
+
64
+ - **canonical 路由可能仍由旧实例应答**(`/skill-eval/data`、`/skill-eval/settings`)。
65
+ 热重载后请用**版本化路由**(`/data/<version>`、`/settings/<version>`)做判断,
66
+ 客户端和 verify 脚本用的都是版本化路由。
67
+ - profile patch 改动本身也会被 HMR 监视并重排,所以启用监视不需要重启;
68
+ 但把 patch 写坏会导致下次启动失败,改前先备份(本仓库操作时已生成
69
+ `cordis.patch.yml.bak-skill-eval-<时间>`)。
70
+
71
+ ## 什么时候必须重启
72
+
73
+ | 改了什么 | 生效方式 |
74
+ | --------------------------------------- | -------------------------------------------------------------------------- |
75
+ | `index.js` / `lib/*`(host 半) | 正常情况:重启;极少数情况下 disable/enable 会重新 import,不要依赖 |
76
+ | `client/*.js`(client 源) | `node scripts/build-client.mjs` → `touch client.js`,HMR 替换已构建 bundle |
77
+ | `client.js`(生成产物) | 不要手改;由 `client/*.js` 构建,doctor 检查两者一致 |
78
+ | `package.json` 版本号 | 重启(版本化路由前缀由模块加载时的版本决定) |
79
+ | 数据文件(events.jsonl、settings.json) | 立即(每次请求重新读) |
80
+ | 由 `settings.json` 覆盖的配置 | 立即(保存即改内存中的 cfg) |
81
+
82
+ ## 给调试用的开关
83
+
84
+ | 开关 | 作用 |
85
+ | ------------------------------------ | --------------------------------------------------- |
86
+ | `window.__SEV_ENABLE_POPUP__ = true` | 恢复侧边栏弹窗模态框(默认只有常驻按钮打开看板) |
87
+ | `DSH_SKILL_EVAL_DEBUG=1` | host 侧把警告同时写 stderr |
88
+ | `DSH_SKILL_EVAL_DIR=<path>` | 覆盖数据目录(隔离实验数据) |
89
+ | `DSH_SKILL_EVAL_BASE=<path>` | 验收脚本使用的看板路径前缀(默认 `/skill-eval`) |
90
+ | `DSH_SKILL_EVAL_VERSION=<v>` | 验收脚本探测的版本化路由版本(默认取 package.json) |
91
+
92
+ ## 客户端与旧宿主兼容
93
+
94
+ 新客户端必须能跑在旧宿主返回的数据上(1.10.x 没有 `edits` / `validity`)。测试用
95
+ `test/fixtures/live-1.10.1.json`(真实宿主抓下来的载荷)锁定这一点;原生设置页在
96
+ 旧宿主上会显示"重启 DeepSeek Harness 后可用"并给出网页配置页链接,而不是白屏。
97
+
98
+ ## 一个可复制的验证流程
99
+
100
+ ```bash
101
+ dsh-skill-eval-verify # 重启后一键验证产品面(10 项)
102
+ dsh-skill-eval-accept # 探当前代次(端口自动发现)
103
+ curl -s http://127.0.0.1:<port>/skill-eval/data | head -c 200
104
+ # host 需重启时会出现:/data/<new-version> 返回 text/html
105
+
106
+ # 想在不动 GUI 进程的前提下验证 host 代码:起一个一次性 profile
107
+ dsh sevcheck --from-default-profile web --help
108
+ dsh plugin --profile sevcheck add "$PWD"
109
+ dsh --profile sevcheck --no-open --port 59231 &
110
+ node scripts/accept.mjs http://127.0.0.1:59231
111
+ kill %1 && rm -rf ~/.dsh/profiles/sevcheck
112
+ ```
113
+
114
+ 这条流程在 1.12.0/1.12.1 上实际用过:17 项 PASS,设置 API 的保存/归一化/400 都验证过。
@@ -0,0 +1,65 @@
1
+ # 入口 · Entry points
2
+
3
+ 三个常用入口,其余都是给 agent / 终端 / 调试用的。日常只需要记住前三行。
4
+ Three everyday entry points; everything else is for agents, terminals or debugging.
5
+
6
+ ## 日常 · Everyday
7
+
8
+ | 要做什么 · Goal | 中文入口 · Chinese | English entry | 说明 · Notes |
9
+ | --------------- | -------------------------------------------------- | --------------------------------------------------------------------- | ------------------------------------------------------------------- |
10
+ | 看这一轮的结果 | 每个完成回合下方的**结果条** | the **result strip** under a completed turn | 分数、调用/报错、下一步修复、改动记录;点「本轮详情」展开 |
11
+ | 打开看板 | 侧边栏底部的**面板图标「技能评测」** → 主面板 | the **panel icon "Skill eval"** in the sidebar → main panel | 与「上下文」插件同位置;面板内含 技能/轨迹/运行/MCP/开发者 五个标签 |
12
+ | 改评分标准 | **设置 → 插件 → `dsh-skill-eval` 的包页面 → 配置** | **Settings → Plugins → the `dsh-skill-eval` bundle page → Configure** | 与上下文插件同一位置(bundle config);权重、阈值、预算等保存即生效 |
13
+
14
+ ## 网页 · Web
15
+
16
+ | 用途 · Purpose | 地址 · URL |
17
+ | ----------------------------------------- | ----------------------------------------------------------------------------------- |
18
+ | 汇报看板 · Board | `http://127.0.0.1:<当前端口>/skill-eval`(热重载期间用 `/skill-eval/board/<版本>`) |
19
+ | 网页配置页(与设置页同源) · Web settings | `…/skill-eval/config` |
20
+ | JSON 数据 · JSON report | `…/skill-eval/data/<version>` |
21
+
22
+ 端口每次启动会变;`dsh-skill-eval-verify` 会自动发现。Port changes per launch; the verify script discovers it.
23
+
24
+ 看板与设置页支持中英双语:按浏览器 `Accept-Language` 自动选择,也可用 `?lang=zh|en` 强转;页面右上角有切换链接。建议正文(advice)目前仍是引擎输出的英文。
25
+
26
+ The board and settings page are bilingual: `Accept-Language` picks the language, `?lang=zh|en` overrides it, and a toggle link is on the page. Rule-generated advice text stays English for now.
27
+
28
+ ## 对话内 · In chat
29
+
30
+ | 命令 · Command | 作用 · What |
31
+ | ------------------------------------------------------- | ------------------------------------------- |
32
+ | `/skill-eval` | 总览 · cross-skill summary |
33
+ | `/skill-eval <skill>` | 单个技能 · one skill |
34
+ | `/skill-eval loop [all]` | 下一步改什么 · next optimization step |
35
+ | `/skill-eval validity` | 证据能支持什么 · what the evidence supports |
36
+ | `/skill-eval history <skill>` / `impact <skill>` | 趋势 / 改动前后 · trend / edit impact |
37
+ | `/skill-eval mcp` / `runs` / `turns` / `dev` / `config` | MCP / 运行 / 每次执行的板 / 诊断 / 生效配置 |
38
+
39
+ ## 给 agent · For agents
40
+
41
+ | 工具 · Tool | 作用 · What |
42
+ | ------------------- | --------------------------------------------------------------------------------- |
43
+ | `skill_eval_report` | 同一份报告(`view` 可选 summary/mcp/runs/loop/validity/history/impact/turns/dev) |
44
+ | `skill_eval_review` | 改技能前做泄漏审查并写 ledger · leak screen before editing a skill |
45
+
46
+ ## 终端 · Terminal
47
+
48
+ ```bash
49
+ dsh-skill-eval-doctor # 打包预检 19 项(--release 加产物检查)· packaging pre-flight
50
+ dsh-skill-eval-accept # 代次/数据契约 17 项 · generation & data contract
51
+ dsh-skill-eval-verify # 产品面可达 9 项 · product surfaces (after a reload)
52
+ ```
53
+
54
+ ## 调试(可选) · Debug (opt-in)
55
+
56
+ | 开关 · Switch | 作用 · What |
57
+ | --------------------------------------------- | ---------------------------------------------------------------------- |
58
+ | 页面里设 `window.__SEV_ENABLE_POPUP__ = true` | 弹窗 + 侧边栏按钮回来 · popup + sidebar button return |
59
+ | `/skill-eval dev` | 代次漂移、存储健康、已注册路由 · generation skew, store health, routes |
60
+
61
+ ## 刻意不做的入口 · Deliberately removed
62
+
63
+ - 插件详情页里再放一个区块(与看板重复)· a second section on the plugin detail page
64
+ - 侧边栏常驻按钮(默认不注册,避免和面板图标重复)· an always-on sidebar button
65
+ - 「更多」菜单里的看板行(看板已升为侧边栏主面板)· the More-menu board row (the board is now a main panel)