dsh-skill-eval 0.0.0-stage → 1.20.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +675 -0
- package/LICENSE +21 -0
- package/README.md +200 -2
- package/README.zh.md +157 -0
- package/client.js +2637 -0
- package/cordis.patch.yml +9 -0
- package/docs/ACCEPTANCE.md +71 -0
- package/docs/AUDIT.md +64 -0
- package/docs/CODE-REVIEW.md +49 -0
- package/docs/COMPARISON.md +33 -0
- package/docs/DETAIL.md +463 -0
- package/docs/DEV-RELOAD.md +114 -0
- package/docs/ENTRYPOINTS.md +65 -0
- package/docs/RELEASE.md +44 -0
- package/docs/RRSI-NOTES.md +129 -0
- package/docs/SCENARIOS.md +91 -0
- package/docs/UI-REVIEW.md +115 -0
- package/docs/WORKFLOW.md +108 -0
- package/icon.svg +9 -0
- package/index.js +1111 -0
- package/lib/accept.js +350 -0
- package/lib/advice.js +547 -0
- package/lib/critic.js +101 -0
- package/lib/dev.js +148 -0
- package/lib/history.js +140 -0
- package/lib/html.js +8 -0
- package/lib/inspect.js +142 -0
- package/lib/integrity.js +46 -0
- package/lib/loop.js +195 -0
- package/lib/random.js +29 -0
- package/lib/recorder.js +634 -0
- package/lib/render.js +582 -0
- package/lib/report.js +646 -0
- package/lib/score.js +525 -0
- package/lib/settings.js +396 -0
- package/lib/snapshot.js +166 -0
- package/lib/steps.js +141 -0
- package/lib/store.js +186 -0
- package/lib/validity.js +115 -0
- package/locale/en.json +6 -0
- package/locale/zh.json +6 -0
- package/package.json +96 -4
- package/scripts/accept.mjs +226 -0
- package/scripts/build-client.mjs +59 -0
- package/scripts/build.mjs +81 -0
- package/scripts/doctor.mjs +199 -0
- package/scripts/lib.mjs +52 -0
- package/scripts/verify-live.mjs +168 -0
package/cordis.patch.yml
ADDED
|
@@ -0,0 +1,9 @@
|
|
|
1
|
+
# bundle patch for dsh-skill-eval: inserts the plugin row into the profile roster.
|
|
2
|
+
# Applied as a profile bundle layer (the `dsh.bundle.patch` manifest field).
|
|
3
|
+
- insert:
|
|
4
|
+
- id: skill-eval
|
|
5
|
+
name: dsh-skill-eval
|
|
6
|
+
config:
|
|
7
|
+
enabled: true
|
|
8
|
+
windowToolCalls: 12
|
|
9
|
+
minSamples: 3
|
|
@@ -0,0 +1,71 @@
|
|
|
1
|
+
# Acceptance report — dsh-skill-eval 1.20.0
|
|
2
|
+
|
|
3
|
+
Status snapshot for the objective: _skill debugging, MCP debugging, dedicated
|
|
4
|
+
boards, developer-mode integration, per-run boards and scores, a scientific
|
|
5
|
+
scoring model, a user-adjustable standard, and a publishable release._
|
|
6
|
+
|
|
7
|
+
Evidence below was produced in this workspace; commands are listed so each row
|
|
8
|
+
can be re-run. Screenshots for the UI rows live under `docs/ui/` and the full
|
|
9
|
+
walkthrough is in [UI-REVIEW.md](UI-REVIEW.md).
|
|
10
|
+
|
|
11
|
+
## Objective → delivery
|
|
12
|
+
|
|
13
|
+
| Objective item | Delivered | How it is verified |
|
|
14
|
+
| ---------------------- | --------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- |
|
|
15
|
+
| 完善 skill 调试 | `skill_eval_report` tool, `/skill-eval` command, result strip, board, read-only Inspect provider | `node --test` (228 tests) + live `verify` |
|
|
16
|
+
| MCP 调试 | per-server calls/errors/p50/p95/health, per-tool latency and MCP-specific advice | `test/report.test.js`, `test/advice.test.js`; live board MCP tab |
|
|
17
|
+
| 专门的看板 | sidebar **Skill eval** panel + `main` panel (skills/runs/MCP/dev tabs), versioned HTML board | jsdom client tests; real-window screenshots (`board-*.png`) |
|
|
18
|
+
| 开发者模式结合 | Cordis Inspect provider `skill-eval` (`summary\|skill\|mcp\|runs\|edits\|standard`), dev tab, `/skill-eval dev` | `test/inspect.test.js`, `test/dev.test.js`; live dev tab shows version 1.17.5 |
|
|
19
|
+
| 每次执行后结果单独看板 | per-turn `turnLoads` + result strip + `reports/turns/*.{md,json}` | `test/recorder.test.js`, `test/snapshot.test.js`; live turn boards |
|
|
20
|
+
| 对每个结果评分 | 0–100 / A–F with four dimensions, confidence, seeded 95% bootstrap CI, significance caveat | `test/score.test.js`, `test/properties.test.js`, `test/validity.test.js` |
|
|
21
|
+
| 评分有效/科学 | sensitivity + calibration archetypes + interval-contains-point property; noise-floor gating | `test/score.test.js`, `test/properties.test.js`, `test/advice.test.js` |
|
|
22
|
+
| 用户可改评分标准 | 25 settings in the native bundle-config form; weights normalised, zero weight drops, thresholds repaired; JSON settings API | `test/settings.test.js`, `test/accept.test.js`; live save verified (20000 → 25000 → restored) |
|
|
23
|
+
| 中英双语 | board chrome, rules' titles/fixes, ranking note and validity statements localized; README dialect | `test/i18n.test.js`; live `?lang=zh` board + screenshot |
|
|
24
|
+
| 可发布 | tarball, LICENSE, explicit `files`, bins, release gate, clean-profile install | `npm publish --dry-run`; `doctor --release`; clean-profile smoke below |
|
|
25
|
+
|
|
26
|
+
## Verification layers
|
|
27
|
+
|
|
28
|
+
1. **Unit + property suites** — `npm test` (rebuilds the browser bundle) → **239 passing**, incl. per-step scoring/recorder/report and the client trajectory filter.
|
|
29
|
+
2. **Live host acceptance** — `dsh-skill-eval-accept` → **17/17 PASS**.
|
|
30
|
+
3. **Live product surfaces** — `dsh-skill-eval-verify` → **9/9 PASS**
|
|
31
|
+
(`versioned-json`, `settings-api`, `board-sections`, `config-page`,
|
|
32
|
+
`turn-boards`, `report-contract`, `latest-snapshot`, manual seats, turn strip).
|
|
33
|
+
4. **Packaging doctor** — `node scripts/doctor.mjs --release` → **PASS**
|
|
34
|
+
(manifest, patch row, client seats, imports, licenses, `files`, bins,
|
|
35
|
+
client build freshness, artifact).
|
|
36
|
+
5. **Clean-profile smoke** — the tarball flow is still pending for 1.20.0:
|
|
37
|
+
the desktop profile currently links the source directory. Run the
|
|
38
|
+
`--from-default-profile web` smoke from RELEASE.md before publishing.
|
|
39
|
+
6. **Real GUI (Orca computer-use on the running DSH window)** — 1.20.0 verified
|
|
40
|
+
live: sidebar panel opens the board, the 「轨迹」 tab shows step KPIs,
|
|
41
|
+
recent-turn comparison and per-step scores, and skill cards expose
|
|
42
|
+
「查看轨迹」. The multi-skill turn strip is covered by the client test; a
|
|
43
|
+
real multi-skill turn still needs one visual confirmation.
|
|
44
|
+
7. **Publishability** — `npm publish --dry-run` succeeded for 1.20.0:
|
|
45
|
+
**47 files, 164.9 kB**. It also exposed npm 11 stripping the bins because
|
|
46
|
+
their targets had a leading `./`; package.json now uses `scripts/*.mjs`
|
|
47
|
+
directly, so re-run the dry-run and confirm the warning is gone.
|
|
48
|
+
|
|
49
|
+
## Audit
|
|
50
|
+
|
|
51
|
+
The 1.20.0 four-layer audit and its fixes are recorded in [AUDIT.md](AUDIT.md).
|
|
52
|
+
|
|
53
|
+
## Known limits (as of this report)
|
|
54
|
+
|
|
55
|
+
- Loads are recorded from `skill` tool calls and user-typed `/skill-name` invocations; a skill the harness activates without either is invisible.
|
|
56
|
+
- Effectiveness is inferred from the next user message, not a task verdict.
|
|
57
|
+
- Advice evidence keeps measured ids/numbers in English even in the Chinese UI.
|
|
58
|
+
- The plugin detail page renders the English `README.md` (DSH does not switch
|
|
59
|
+
README by locale); a Chinese TL;DR sits at the top of that file.
|
|
60
|
+
- The installed-plugin list has no search (platform behaviour).
|
|
61
|
+
- Telemetry is append-only; there is no automatic pruning.
|
|
62
|
+
|
|
63
|
+
## Reproduce
|
|
64
|
+
|
|
65
|
+
```bash
|
|
66
|
+
node --test test/ # 239 passing (npm test builds the client first)
|
|
67
|
+
node scripts/doctor.mjs --release # release gate incl. dist artifact
|
|
68
|
+
dsh-skill-eval-accept # 17 checks against the running host
|
|
69
|
+
dsh-skill-eval-verify # 9 checks against the running host
|
|
70
|
+
npm publish --dry-run # packaging/publishability preview
|
|
71
|
+
```
|
package/docs/AUDIT.md
ADDED
|
@@ -0,0 +1,64 @@
|
|
|
1
|
+
# 多维度审计 · Multi-angle audit (1.18.0)
|
|
2
|
+
|
|
3
|
+
Four parallel read-only audits plus the root agent's own probes covered the host
|
|
4
|
+
data pipeline, the HTTP/rendering layer, the client UI and packaging/docs. Every
|
|
5
|
+
finding below was reproduced against the real modules before a fix was written;
|
|
6
|
+
the fixes have regression tests (`test/audit-regressions.test.js`, `test/store.test.js`,
|
|
7
|
+
`test/settings.test.js`, `test/client.test.js`).
|
|
8
|
+
|
|
9
|
+
## Findings and fixes
|
|
10
|
+
|
|
11
|
+
| # | Layer | Severity | Finding | Fix |
|
|
12
|
+
| --- | ----------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
13
|
+
| 1 | host / data | high | A telemetry key `__proto__` (a crafted `static`/window record, or an MCP tool literally named that) resolved to `Object.prototype`; aggregation then wrote `calls/errors/ms` onto it and threw, breaking every report until restart | All telemetry-keyed maps (`mcp`, `tools`, `siblings`, `errorNames`, `loopErrorNames`, validity `dimensions`) use null-prototype objects and `Object.hasOwn`; recorder buckets too |
|
|
14
|
+
| 2 | host / store | high | After a crash left a record without its newline, the next append glued both records into one unparseable line and lost the new record | `EventStore.append` closes a torn tail before writing; regression test |
|
|
15
|
+
| 3 | host / report | medium | A window is persisted at turn end and again at close; run counters summed every snapshot (`closed: 2, toolCalls: 6` for one 3-call window) and `scoreRun` used the doubled values | Run counters are summed from the deduplicated latest-per-`windowId` map, matching the bootstrap path |
|
|
16
|
+
| 4 | host / scoring | medium | The 95% CI was computed without `cacheReadWeight`/`tokenBudgetPerLoad` while the point estimate used them, so a score could lie outside its own interval | One `scoreSamples()` closure used for both point and bootstrap draws |
|
|
17
|
+
| 5 | host / storage | medium | Every load persisted the whole SKILL.md body (`static` record ~200 KB per load) and embedded it in every report/snapshot JSON | `staticProfile` no longer returns `content`; old records with `content` are stripped on read |
|
|
18
|
+
| 6 | host / validation | medium | A hand-edited/corrupt `static` record (`description: 123`) or an `edit` with `verdict: "dimensions"` crashed the whole report | Recorded static profiles are normalised field-by-field; validity counts only known verdicts and string dimensions; advice coerces text |
|
|
19
|
+
| 7 | host / memory | low | The 129th tracked session evicted the oldest _inserted_ session and its zero-call window was scored as "abandoned" | Least-recently-active eviction; `closeReason: "evicted"` is carried into aggregation and censored from `abandonRate` |
|
|
20
|
+
| 8 | host / perf | medium | Each `/data` request, turn-end snapshot and 5 s popup poll re-read and re-scored the entire store (3.0 s at 200k records) | Report cache keyed by file `mtime:size` + 2 s TTL, invalidated by appends and settings saves; measure again with `node /tmp/perf.mjs`-style fixtures |
|
|
21
|
+
| 9 | HTTP | medium | Settings writes accepted cross-site requests (plugin routes bypass the host trust fence); a cross-site page could persist settings, DNS-rebinding could read `/data` | Writes require same-origin `Origin`/`Sec-Fetch-Site`; the JSON API additionally requires `x-skill-eval-config: 1` (forces a preflight). Rebinding _reads_ still depend on the DSH host-level fence — documented residual |
|
|
22
|
+
| 10 | HTTP | medium | A partial form POST turned **every** boolean setting off (`parseFormSettings` defaulted absent checkboxes) | Only a submission carrying the hidden `__full_form=1` marker defaults checkboxes; partial posts stay partial |
|
|
23
|
+
| 11 | HTTP | medium | `settings.json` was written in place with a read-modify-write: concurrent saves lost fields; a torn write silently discarded all overrides | Serialized writes + `settings.json.tmp` → `rename` (atomic) |
|
|
24
|
+
| 12 | HTTP | low | The merged effective weights were only validated when all four arrived; four single-key POSTs could reach an all-zero weight set → every score `null` | Merged weights are validated on both write paths |
|
|
25
|
+
| 13 | HTTP | low | Body limit counted UTF-16 units, decoded split multi-byte chunks, and destroyed the socket before the error was sent | Byte-based limit, `content-length` fast path, `setEncoding('utf8')`, clean `413` |
|
|
26
|
+
| 14 | HTTP | low | `/settings` exposed the absolute `settingsFile` path; write failures returned raw `error.message` (HTML content type on the form route) | Path removed from the payload; generic messages with the correct status/content type |
|
|
27
|
+
| 15 | HTTP | low | `accept.mjs` validated against the newest profile's `events.jsonl`, not the directory the server served | It now reads `dataDir` from the served report |
|
|
28
|
+
| 16 | client | medium | Overlapping `load()` requests could overwrite fresh data with a stale response or paint an error over fresh data | Single in-flight request + generation guard; Refresh disabled while loading |
|
|
29
|
+
| 17 | client | medium | `useChat` was called conditionally; a host that stopped passing it would change the hook count and crash the turn strip | Split into `WithChat` / `Plain` wrappers around a hook-free body |
|
|
30
|
+
| 18 | client | medium | Popup declared `aria-modal` but had no focus trap/restore | Tab cycling inside the dialog, focus restored to the opener on close |
|
|
31
|
+
| 19 | client | medium | Each skill card re-subscribed to the locale on every render (≤123 subscribe/unsubscribe pairs per render at 121 cards) | Stable module-level subscribe/snapshot identities; `SkillCard` is memoized and receives `lang`/`t` |
|
|
32
|
+
| 20 | client | low | Clipboard fallback leaked a textarea when `execCommand` threw, and the popup silently kept "copy" on failure | `finally` cleanup, tri-state label on both surfaces, mounted guard for late timers |
|
|
33
|
+
| 21 | client | low | The bundle-config page rendered raw browser Save buttons because only `TURN_STYLES` was injected | The settings form injects the shared stylesheet |
|
|
34
|
+
| 22 | client | low | Duplicate `noAdvice` key made a skill card with no advice say "nothing to fix this turn"; several strings stayed English/raw keys | Keys renamed; dims/turn/footer/dev strings localized; popup uses `noteZh`; panel label follows the app locale |
|
|
35
|
+
| 23 | client | low | Copy/tab status was visual-only; severity was color-only | `role="status" aria-live="polite"` copy feedback, `role=tablist/tab` + `aria-selected`, severity glyphs |
|
|
36
|
+
|
|
37
|
+
## Verification
|
|
38
|
+
|
|
39
|
+
- `node --test test/` → **232 passing** (215 before this audit; regression tests for every fixed finding plus CLI exit codes, EN locale coverage, settings group metadata and client build helpers).
|
|
40
|
+
- Live host after hot reload: `/skill-eval/data/<version>` JSON, `accept` 17/17, `verify` 9/9.
|
|
41
|
+
- Live security probes: cross-site `Origin` POST → **403**; missing `x-skill-eval-config` → **403**; 70 KB form body → **413**; same-origin GET unchanged.
|
|
42
|
+
- Clean profile install of the packed tarball: accept 17/17, verify 9/9.
|
|
43
|
+
|
|
44
|
+
## Follow-up findings from the architecture audit (also fixed)
|
|
45
|
+
|
|
46
|
+
- **Versioned pages rendered canonical links** (high): `/board/<v>` posted its form to `/config` and linked there, so a hot reload saved into the stale generation. Each page route now renders against its own base; regression test checks the versioned `action` and link.
|
|
47
|
+
- **English TEXTS were missing the new localization keys** (medium): the EN turn strip rendered `undefined`; keys restored and a test renders the EN strip asserting no `undefined`.
|
|
48
|
+
- **Threshold repair collapsed bands** (medium): `{90,50,70}` became `{90,50,50}`; floors are now sorted descending. Settings bounds now match the Config schema (`windowTtlMs` added, 25 fields).
|
|
49
|
+
- **Custom `dashboardPath` broke the client**: data/settings routes now stay under the fixed `/skill-eval` client contract while pages follow the configured path.
|
|
50
|
+
- **Silent telemetry write failures**: `EventStore.onError` logs once and the board shows `telemetry write failed`.
|
|
51
|
+
- **Dead code**: removed unused `MoreMenuBoard`/`PluginDetailSection`, `GRADE_BANDS`; rendered loop budget now follows `scoring.editBudgetMax`.
|
|
52
|
+
|
|
53
|
+
## Deferred / residual
|
|
54
|
+
|
|
55
|
+
- Store rotation/compaction is still absent. The report cache removes the repeated
|
|
56
|
+
cost, but a very long-lived profile keeps every record. A future version should
|
|
57
|
+
roll `events.jsonl` monthly or keep N windows per skill.
|
|
58
|
+
- DNS-rebinding **read** protection belongs to the DSH host's trust layer; this
|
|
59
|
+
plugin now fences writes only.
|
|
60
|
+
- Client still loses scroll position when the main panel unmounts, and long
|
|
61
|
+
skill lists are not virtualized (121-card probe: ~89 ms first paint in jsdom).
|
|
62
|
+
- Threshold ordering (`excellent ≥ good ≥ pass`) is repaired at scoring time but
|
|
63
|
+
not rejected at write time; settings `mcpErrorRatePercent` allows 0 while the
|
|
64
|
+
Config schema requires ≥1.
|
|
@@ -0,0 +1,49 @@
|
|
|
1
|
+
# 代码卫生审查 · Code hygiene review (1.20.0 + unreleased follow-up)
|
|
2
|
+
|
|
3
|
+
范围:`index.js`、`client/*.js`、`lib/*.js`、`scripts/*.mjs`、`test/`、`package.json`、
|
|
4
|
+
`docs/`。方法:静态扫描(未使用导出/导入、死 CSS、TEXTS 键使用)、重复实现比对、
|
|
5
|
+
覆盖率地图、打包清单核对,然后运行 239 项测试与线上验收。
|
|
6
|
+
|
|
7
|
+
## 已清理
|
|
8
|
+
|
|
9
|
+
| # | 类别 | 发现 | 处理 |
|
|
10
|
+
| --- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
11
|
+
| 1 | 元数据资源 | `locale/{en,zh}.json` 不是死文件:DSH 从 `locale/<lang>.json` 读 `meta.title`/`meta.description` 做插件市场与详情页本地化(删除后中文描述会回退英文)。旧文件里另有 16 个未被任何代码引用的 UI 键,属于漂移残留 | 恢复目录,**只保留 `meta.title`/`meta.description`**;恢复 `exports`/`files`;doctor 改为检查两份 meta 非空;插件列表重新显示中文描述 |
|
|
12
|
+
| 2 | 重复实现 | `hashSeed`(report)与 `simpleHash`(index)是同一 FNV;`mulberry32` 在 report 与 score 各写一份;`esc`(render)与 `escape`(settings)是同一转义 | 新增 `lib/random.js`(`hashSeed`/`mulberry32`/`hashHex`)与 `lib/html.js`(`escapeHtml`),四处改为引用;index 的 diff 短哈希改为引用 `hashHex`,不再保留本地 `simpleHash`;bootstrap 输出不变(测试锁定) |
|
|
13
|
+
| 3 | 死代码 | 移除组件后仍留 `TEXTS` 键(`advice`/`hint`/`topTool`/`detailTitle`/`detailOpen*`/`moreBoard`)和 CSS(`.sev-detail-*`、`.sev-more-row`) | 删除;`.sev-btn--ghost` 缺样式,补一条透明背景规则 |
|
|
14
|
+
| 4 | 注释漂移 | `client.js` 头注释仍写"只在侧边栏 footer 加按钮+弹窗" | 拆分后 `client.js` 为生成文件,头注释改为 GENERATED FILE 横幅(来源 `client/*.js` + `scripts/build-client.mjs`),不再描述旧 seat 文案 |
|
|
15
|
+
| 5 | 回归 | `saveSettings` 在重构中丢失了串行化/原子写(AUDIT 已记录但被覆盖) | 恢复 `settings.json.tmp` → `rename` + 队列,并加回归测试 |
|
|
16
|
+
| 6 | 测试缺口 | 设置分组可能漏字段;并发写可能截断 | 新增"每个设置恰好属于一个分组"和"并发保存后文件完整、无 tmp 残留"两项测试 |
|
|
17
|
+
| 7 | 工程约定 | 无 .gitignore/.editorconfig | 添加(node_modules/dist/tgz/.DS_Store;LF、UTF-8、2 空格);`dist/*.tgz` 为本地打包产物,不入库(已清理陈旧 `1.19.1.tgz`) |
|
|
18
|
+
| 8 | 表单加固 | `__full_form` 守卫、零权重求和拒绝、`FIELD_HELP`/`unit` 等表单加固逻辑有测试但未列入清理表 | 补记:逻辑保留,`test/settings.test.js` 覆盖缺席 checkbox → false、零权重行为;不删除 |
|
|
19
|
+
|
|
20
|
+
## 复查结论(未发现问题)
|
|
21
|
+
|
|
22
|
+
- 运行时无 TODO/FIXME/console.log(`console.log` 仅存在于 CLI 脚本的输出)。
|
|
23
|
+
- 无注释掉的代码块;`node --check` 全部通过。
|
|
24
|
+
- CSS 类均有非样式引用;TEXTS zh/en 键完全一致(测试覆盖 EN 回合条不出现 `undefined`)。
|
|
25
|
+
- 未使用导出仅 5 个,全部由测试或 CLI 脚本使用(`LEAK_RULES`(`test/critic.test.js`)、`LOOP_HINT_DIMENSIONS`(`test/loop.test.js` 跟踪 `EDIT_DIMENSIONS` 一致性)、`loadSettings`、`pickBaseUrl`、`runAcceptance`)。
|
|
26
|
+
- 包清单与 `files` 一致;`npm pack` 后无死文件。
|
|
27
|
+
|
|
28
|
+
## 后续落地(1.20.0)
|
|
29
|
+
|
|
30
|
+
1. **client.js 拆分完成**:`client/*.js`(13 个 part)+ `scripts/build-client.mjs`
|
|
31
|
+
零依赖拼接;`prepack`/`pretest` 自动构建,doctor 增加 `client-build`
|
|
32
|
+
新鲜度检查,`client/README.md` 记录流程。发布包仍只含构建后的单文件工厂。
|
|
33
|
+
2. ~~设置分组重复~~:已完成(host 下发 `group`/`groupZh`)。
|
|
34
|
+
3. **CLI 去重**:`scripts/lib.mjs` 统一 bridge/base/版本/超时 GET;accept 与
|
|
35
|
+
verify 改为引用,新增 `test/cli.test.js`。
|
|
36
|
+
4. **Prettier + ESLint**:已加配置、devDependencies 与 `format:check`/`lint`
|
|
37
|
+
脚本,源码已按 Prettier 格式化(`prettier --check` 当次通过)。
|
|
38
|
+
5. ~~测试 helper 抽取~~:已抽 `test/helpers.js`(`createHost(window, style)`)+ `test/client.test.js` 顶层 `createRoot`/`act` 导入,去掉 18 份重复;`BoardPanel` 与 `apply` 仍偏长(可接受,未再拆)。
|
|
39
|
+
|
|
40
|
+
## 验证
|
|
41
|
+
|
|
42
|
+
- `npm run release:check` 全绿:client 构建 → Prettier check → ESLint(0 error/0
|
|
43
|
+
warning)→ doctor `--release` → **239 tests passing** → live accept 17/17。
|
|
44
|
+
- `npm publish --dry-run`:1.20.0,**47 files,164.9 kB**;npm 11 曾把带 `./`
|
|
45
|
+
前缀的 bin target 判为 invalid 并移除,package.json 已改成 `scripts/*.mjs`,
|
|
46
|
+
需重跑 dry-run 确认警告消失。
|
|
47
|
+
- 干净 profile 安装 tarball 的冒烟仍待跑(desktop profile 当前 link 到源码目录)。
|
|
48
|
+
- 真实窗口:图表页 `docs/ui/charts-1.19.0.png`;逐步轨迹页
|
|
49
|
+
`docs/ui/steps-1.20.0.png`;设置页 `docs/ui/settings-1.18.3.png`。
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
# Comparison with the Claude Code / Agent Skills corpus
|
|
2
|
+
|
|
3
|
+
Reference: [anthropics/skills](https://github.com/anthropics/skills) README (inspected 2026-11-15) and the
|
|
4
|
+
[agentskills.io](https://agentskills.io) spec it points at. That repository groups skills as
|
|
5
|
+
Creative & Design, Development & Technical (including MCP server generation), Enterprise &
|
|
6
|
+
Communication, and the production document skills (`docx`, `pdf`, `pptx`, `xlsx`), plus a
|
|
7
|
+
`spec/` and a `template/`. It states the frontmatter contract as two required fields:
|
|
8
|
+
`name` (lowercase, hyphenated) and `description` (what it does **and when to use it**).
|
|
9
|
+
|
|
10
|
+
## Where our evaluator already lines up
|
|
11
|
+
|
|
12
|
+
| Claude Code / spec expectation | What dsh-skill-eval measures |
|
|
13
|
+
| ------------------------------------------------------- | ------------------------------------------------------------------------ |
|
|
14
|
+
| `description` says what it does **and when to use it** | `desc-length` + `desc-trigger` + `when-to-use` static checks |
|
|
15
|
+
| Body is instructions, not an essay | `body-present`, `sections`, `body-compact` |
|
|
16
|
+
| Complex skills delegate to `references/` and `scripts/` | `progressive-disclosure` static check + runtime tool-call cost |
|
|
17
|
+
| Skills should actually help when loaded | reliability / effectiveness / efficiency dimensions from live windows |
|
|
18
|
+
| Document skills are heavy but must activate precisely | trigger collisions (`otherSkills`), ignored loads, post-load tool errors |
|
|
19
|
+
| MCP-flavoured skills (e.g. MCP server generation) | per-server MCP board: calls, error rate, p50/p95, health, failing tools |
|
|
20
|
+
|
|
21
|
+
## Gaps we found, and what we did
|
|
22
|
+
|
|
23
|
+
| Gap vs the CC corpus | Status |
|
|
24
|
+
| -------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
|
|
25
|
+
| **Anti-trigger language** ("do not use for …") is part of good trigger design | ✅ 1.4.0: `anti-trigger` static check + `no-anti-trigger` info advice |
|
|
26
|
+
| **Frontmatter integrity**: `name` must equal the skill id and be lowercase/hyphenated | ✅ 1.4.0: `skill-name` check enforces hard kebab-case ids from the parsed summary (raw frontmatter itself still not read) |
|
|
27
|
+
| **Reference/script link integrity**: a skill may promise `references/forms.md` that does not exist | ✅ 1.4.0: checked at load time (`lib/integrity.js`), missing files raise `broken-references` |
|
|
28
|
+
| **Script correctness**: document skills ship real scripts (`python -m scripts …`); we never execute them | out of scope — a runtime skill cannot be executed by an evaluator safely |
|
|
29
|
+
| Optional metadata (`license`, allowed-tools style fields) | not evaluated |
|
|
30
|
+
| Asset bundles beyond markdown (templates, fonts) | only inferred through progressive-disclosure text |
|
|
31
|
+
|
|
32
|
+
The test corpus in `test/corpus/skills.mjs` encodes one skill per pattern so the evaluator's
|
|
33
|
+
behaviour on each is asserted, and the open gaps above stay visible instead of being forgotten.
|