dsh-eval-pack 0.1.0-alpha.10 → 0.1.0-alpha.12

package/CHANGELOG.md

@@ -1,6 +1,6 @@
 # Changelog
 
-## [0.1.0-alpha.10] — 2026-11-01
+## [0.1.0-alpha.12] — 2026-11-01
 
 ### Security / Fixes
 

package/README-zh.md

@@ -37,6 +37,81 @@ dsh plugin --profile <profile> add dsh-eval-pack
 
 *本地真实测试运行；不调用模型或网络。*
 
+## 一句话
+
+把一小包评测样本逐条按 JSON 精确比对，给出通过率、失败项和 pass@k。
+
+## 适合 / 不适合
+
+- **适合**：想复核一个小评测包（3–50 个样本）的通过率，需要确定性的 JSON 相等评分、不接受模糊匹配的场景。
+- **不适合**：需要调用模型、主观打分或跑真实 benchmark 的场景；本包只做评分与汇总，不运行被测对象。
+
+## 输入
+
+- `pack.json`：`name`、`version` 与 `tasks`（任务 id 列表）。
+- 每个样本一个 JSON：`id`、`prompt`、`expected`；评分只看 `expected` 与运行结果里的 `actual`。
+- `run.json`：一次运行的结果 `results[].actual`，以及 pass@k 用的 `sampling[]`（`samples` / `correct`）。
+- 自带样例在 `samples/quickstart/input/`：3 个样本 JSON + 1 个 run 结果，合计远小于 500 KB。
+- 工具不设硬性大小上限；单个 JSON 建议 ≤1 MB，样例只是最小可复现集。
+
+## 输出
+
+- 终端摘要：lint 是否通过、逐条 pass/fail、通过率与失败 id、pass@1 与 pass@3。
+- 不写任何文件；需要留档时把 `--json` 的输出重定向到自己的文件即可。
+- 评分规则：对象键顺序不影响结果，数组顺序会影响结果，数字必须完全相等。
+
+## 示例
+
+直接复制这句话：
+
+> 我用自带样例跑一次评测：3 个样本里过了几个、哪个没过、pass@3 是多少？
+
+也可以自己跑（真实运行，截图即来自这条命令）：
+
+```sh
+node samples/quickstart/run.mjs
+```
+
+![dsh-eval-pack quickstart 真实运行输出](docs/assets/quickstart.png)
+
+*合成评测包 → 金标 `samples/quickstart/expected/result.json`；命令与哈希见 `screenshots.json`。*
+
+## 权限
+
+- 读：你指定的评测包 JSON、样本 JSON 与运行结果 JSON（由会话读取后作为参数传入）；不读其他目录
+- 写：无（四个工具全部只读，不写任何文件）
+- 联网：否（源码中没有 HTTP / DNS / WebSocket 调用）
+- 外部系统：无（不调用模型、不连接评测平台、不上传数据）
+
+四个工具全部标 **R0（只读）**；逐项见下方工具表。
+
+## FAQ
+
+### 装不上怎么办？
+
+确认 DSH `>=0.2.1-rc.3 <0.4.0`、Node `>=24.9 <25`，并用全新的 `DSH_HOME` 重试；完整步骤见 `TROUBLESHOOTING.md`。
+
+### 为什么两份 JSON 看起来一样却没通过？
+
+评分先做稳定序列化：对象的键顺序无所谓，但数组顺序、数字精度和字符串大小写都按原样比较；`12.34` 与 `12.35` 就是不相等。
+
+### pass@k 是怎么算的？
+
+用无偏估计 `1 - C(n-c, k) / C(n, k)`，其中 `n` 是每个任务的采样数、`c` 是其中正确的次数；`k` 不能超过 `n`。
+
+### 能处理我的真实评测集吗？
+
+可以本地运行，但本包是 alpha、只用公开/合成评测包验证，不构成 benchmark 结论；真实评测集请先脱敏。
+
+### 会联网或调用模型吗？
+
+不会。lint、评分与汇总全部在本机完成。
+
+## 版本
+
+- 当前版本：`0.1.0-alpha.12`（与 `package.json` 一致）
+- 测试：`node --test test/*.test.mjs` 全绿（14 个用例，0 失败）
+
 ## 安装
 
 安装并验证：
@@ -76,6 +151,8 @@ dsh plugin --profile <profile> remove dsh-eval-pack
 |---|---|---|
 | `eval.pack_lint` | 校验 eval-pack 名称/版本及任务 id 唯一性 | R0 |
 | `eval.grade` | 确定性的 JSON 相等评分 | R0 |
+| `eval.summary` | 汇总一次运行：通过数、通过率、失败任务 id | R0 |
+| `eval.pass_at_k` | 按采样数计算无偏 pass@k 与均值 | R0 |
 
 ## 快速开始
 
@@ -84,9 +161,9 @@ dsh plugin --profile <profile> add dsh-eval-pack
 dsh --profile <profile> --dump-config | grep dsh-eval-pack
 ```
 
-## 示例
+## 验收
 
-### fixture 支撑的示例
+### fixture 支撑的验收行
 
 输入: [`ACCEPTANCE.csv`](https://github.com/PerryLink/dsh-eval-pack/blob/main/ACCEPTANCE.csv)
 
@@ -109,7 +186,8 @@ dsh --profile <profile> --dump-config | grep dsh-eval-pack
 
 ## 公开样例
 
-公开/合成 fixture 在 `fixtures/`；`ACCEPTANCE.csv` 记录公开检查，全部 `counted=false`。
+- `samples/quickstart/`：随 npm 包一起发布的合成评测包（3 个样本 + run 结果 + 金标），`node samples/quickstart/run.mjs --check` 可直接复现金标。
+- `ACCEPTANCE.csv` 记录公开检查，除 `PS1-QS01` 外全部 `counted=false`。
 
 ## 开发
 

package/TROUBLESHOOTING.md

@@ -1,8 +1,29 @@
 # Troubleshooting — dsh-eval-pack
 
+每条按「症状 / 原因 / 动作」写。报错原文都是 `<一句人话>（E_XXX）｜下一步：<动作>`，
+把 `E_XXX` 和这句话一起发到 issue 即可定位；不要把客户评测数据贴进 issue。
+
+## 1. 评测包本身不合格（E_PACK_TASKS_EMPTY / E_TASK_DUPLICATE / E_PACK_NAME）
+
+- **症状**：`评测包里一个任务都没有（E_PACK_TASKS_EMPTY）｜下一步：往 tasks 里至少放一个带 id 的任务`
+- **原因**：`pack.json` 缺少 `name` / `version`，或 `tasks` 为空、任务 id 重复。
+- **动作**：先跑 `eval.pack_lint`；按返回的错误码补齐字段、把重复 id 改成唯一值，再评分。
+
+## 2. 正确样本数超过采样数（E_EVAL_CORRECT）
+
+- **症状**：`正确的样本数超过了采样数（E_EVAL_CORRECT）｜下一步：确认 correct ≤ samples 再算 pass@k`
+- **原因**：`sampling[]` 里 `correct` 填得比 `samples` 大，通常是两次运行的计数拼错了。
+- **动作**：确认每个任务的 `samples` 和 `correct` 来自同一次运行；`correct` 最多等于 `samples`。
+
+## 3. k 超过采样数（E_EVAL_K）
+
+- **症状**：`k 超过了采样数（E_EVAL_K）｜下一步：把 k 改成 1 到 samples 之间的整数`
+- **原因**：拿 10 次采样的数据去算 pass@20，估计量没有定义。
+- **动作**：把 `k` 改成不超过该任务 `samples` 的整数；不同任务采样数不同时，先取最小值再统一 `k`。
+
 ## Install or activation fails
 
-- DSH must satisfy `>=0.2.1-rc.3 <0.3.0`; Node must be `>=24.9 <25`.
+- DSH must satisfy `>=0.2.1-rc.3 <0.4.0`; Node must be `>=24.9 <25`.
 - Run `dsh --version` and `node --version`, then retry with a fresh `DSH_HOME`:
 
 ```sh
@@ -13,25 +34,23 @@ dsh --profile <profile> --dump-config | grep dsh-eval-pack
 
 ## The tool cannot find an input
 
-- All paths are local. Use an absolute path, or a path relative to the process working directory.
-- The public samples under `fixtures/` are repository-only; they are not shipped in the npm tarball. Download them from the GitHub repository if needed.
-- Supported material: eval-pack JSON.
+- All paths are local. Pass the pack, case and run JSON as arguments; the shipped quickstart lives in `samples/quickstart/`.
+- Supported material: eval-pack JSON, per-case sample JSON and one run-result JSON.
 
 ## Parse or encoding failures
 
-- Text inputs must be UTF-8; CSV headers must match the README's documented column names.
-- Remove a UTF-8 BOM if the first header is reported as unknown.
+- All inputs must be UTF-8 JSON; remove a UTF-8 BOM if the parser rejects the first key.
 - Check the pack name/version and unique task ids; grading is deterministic JSON equality, not a benchmark.
 
 ## Permission or write failures
 
-- Tools marked `R1 local write` write under the configured DSH data directory.
-- For tests, point `DSH_HOME` at a temporary directory; do not write into a real profile.
+- All four tools are read-only (`R0`); they do not write files.
+- For tests, run against the shipped synthetic sample; no `DSH_HOME` writes are involved.
 
 ## The result looks wrong
 
-- Check the tool boundary in the README first. These are public/synthetic alpha tools, not a production or customer-validated determination.
-- Compare the reported row ids and input rows; the output is deliberately explicit about what it could not run.
+- Check the tool boundary in the README first. These are public/synthetic alpha tools, not a production or customer-validated benchmark.
+- Compare the reported task ids with the input tasks; the output is deliberately explicit about which case failed.
 
 ## Run the tests
 

package/docs/assets/quickstart.log

@@ -0,0 +1,6 @@
+$ node samples/quickstart/run.mjs
+dsh-eval-pack quickstart - synthetic eval pack (not a benchmark)
+pack synthetic-invoice-basics@1.0.0: lint ok, 3 tasks
+grades: case-01 pass, case-02 fail, case-03 pass
+summary: 2/3 passed (0.666667), failures: case-02
+pass@1 mean 0.633333; pass@3 mean 0.9

package/lib/core/harness.d.ts

@@ -7,6 +7,8 @@ export interface EvalPack {
     version: string;
     tasks: EvalTask[];
 }
+export type EvalErrorCode = 'E_PACK_NOT_OBJECT' | 'E_PACK_NAME' | 'E_PACK_VERSION' | 'E_PACK_TASKS_EMPTY' | 'E_TASK_ID' | 'E_TASK_DUPLICATE' | 'E_EVAL_SAMPLES' | 'E_EVAL_CORRECT' | 'E_EVAL_K';
+export declare function evalErrorMessage(code: EvalErrorCode): string;
 export declare function lintPack(input: unknown): string[];
 /** Stable serialisation: object keys sorted at every level, arrays kept in order. */
 export declare function canonical(value: unknown): string;

package/lib/core/harness.js

@@ -1,3 +1,22 @@
+/**
+ * Human-facing wording for every code this package can surface.
+ * Renderer: `<一句人话>（E_CODE）｜下一步：<一个动作>` (Package Standard v1 §4).
+ */
+const EVAL_HUMAN = {
+    E_PACK_NOT_OBJECT: { text: '这不是一个评测包对象，读不出名称、版本和任务', next: '传一个包含 name / version / tasks 的 JSON 对象' },
+    E_PACK_NAME: { text: '评测包缺少名称', next: '给评测包加一个非空的 name 再跑 lint' },
+    E_PACK_VERSION: { text: '评测包缺少版本号', next: '给评测包加一个非空的 version（例如 "1.0.0"）' },
+    E_PACK_TASKS_EMPTY: { text: '评测包里一个任务都没有', next: '往 tasks 里至少放一个带 id 的任务' },
+    E_TASK_ID: { text: '有一个任务没有 id', next: '给每个任务补一个非空且唯一的 id' },
+    E_TASK_DUPLICATE: { text: '评测包里有重复的任务 id', next: '把重复的 id 改成唯一值再跑 lint' },
+    E_EVAL_SAMPLES: { text: '采样数必须是正整数', next: '把 samples 改成 ≥1 的整数' },
+    E_EVAL_CORRECT: { text: '正确的样本数超过了采样数', next: '确认 correct ≤ samples 再算 pass@k' },
+    E_EVAL_K: { text: 'k 超过了采样数', next: '把 k 改成 1 到 samples 之间的整数' },
+};
+export function evalErrorMessage(code) {
+    const human = EVAL_HUMAN[code];
+    return `${human.text}（${code}）｜下一步：${human.next}`;
+}
 export function lintPack(input) {
     const errors = [];
     if (!input || typeof input !== 'object')
@@ -68,11 +87,11 @@ function comb(n, k) {
 export function passAtK(task, k) {
     const { samples, correct } = task;
     if (!Number.isInteger(samples) || samples < 1)
-        throw new Error('E_EVAL_SAMPLES: samples must be a positive integer');
+        throw new Error(evalErrorMessage('E_EVAL_SAMPLES'));
     if (!Number.isInteger(correct) || correct < 0 || correct > samples)
-        throw new Error('E_EVAL_CORRECT: correct must be between 0 and samples');
+        throw new Error(evalErrorMessage('E_EVAL_CORRECT'));
     if (!Number.isInteger(k) || k < 1 || k > samples)
-        throw new Error('E_EVAL_K: k must be between 1 and samples');
+        throw new Error(evalErrorMessage('E_EVAL_K'));
     const denominator = comb(samples, k);
     const numerator = comb(samples - correct, k);
     const value = 1 - Number(numerator) / Number(denominator);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dsh-eval-pack",
-  "version": "0.1.0-alpha.10",
+  "version": "0.1.0-alpha.12",
   "description": "Public/synthetic eval-pack lint and deterministic JSON grader; not a validated benchmark.",
   "type": "module",
   "main": "./lib/plugin/index.js",
@@ -29,11 +29,24 @@
     "CHANGELOG.md",
     "TROUBLESHOOTING.md",
     "screenshots.json",
-    "icon.svg"
+    "icon.svg",
+    "samples"
   ],
   "dsh": {
     "bundle": {
       "patch": "./cordis.patch.yml"
+    },
+    "demo": {
+      "sample": "samples/quickstart/input",
+      "prompt": "我用自带样例跑一次评测：3 个样本里过了几个、哪个没过、pass@3 是多少？",
+      "expect": "samples/quickstart/expected/result.json",
+      "screenshot": "docs/assets/quickstart.png",
+      "reads": [
+        "用户指定的评测包 JSON、样本 JSON 与运行结果 JSON（样例：samples/quickstart/input）"
+      ],
+      "writes": [],
+      "network": false,
+      "external": []
     }
   },
   "engines": {
@@ -80,5 +93,10 @@
   "bugs": {
     "url": "https://github.com/PerryLink/dsh-eval-pack/issues"
   },
-  "icon": "icon.svg"
+  "icon": "icon.svg",
+  "peerDependenciesMeta": {
+    "@deepseek-ai/dsh-tools": {
+      "optional": true
+    }
+  }
 }

package/samples/quickstart/expected/result.json

@@ -0,0 +1,70 @@
+{
+  "sample": "samples/quickstart/input",
+  "synthetic": true,
+  "pack": {
+    "name": "synthetic-invoice-basics",
+    "version": "1.0.0",
+    "tasks": 3
+  },
+  "lint": {
+    "ok": true,
+    "errors": []
+  },
+  "grades": [
+    {
+      "id": "case-01",
+      "pass": true
+    },
+    {
+      "id": "case-02",
+      "pass": false
+    },
+    {
+      "id": "case-03",
+      "pass": true
+    }
+  ],
+  "summary": {
+    "total": 3,
+    "passed": 2,
+    "failed": 1,
+    "pass_rate": 0.666667,
+    "failures": [
+      "case-02"
+    ]
+  },
+  "pass_at_1": {
+    "mean": 0.633333,
+    "tasks": [
+      {
+        "task_id": "case-01",
+        "pass_at_k": 0.7
+      },
+      {
+        "task_id": "case-02",
+        "pass_at_k": 0.3
+      },
+      {
+        "task_id": "case-03",
+        "pass_at_k": 0.9
+      }
+    ]
+  },
+  "pass_at_3": {
+    "mean": 0.9,
+    "tasks": [
+      {
+        "task_id": "case-01",
+        "pass_at_k": 0.991667
+      },
+      {
+        "task_id": "case-02",
+        "pass_at_k": 0.708333
+      },
+      {
+        "task_id": "case-03",
+        "pass_at_k": 1
+      }
+    ]
+  }
+}

package/samples/quickstart/input/cases/case-01.json

@@ -0,0 +1,5 @@
+{
+  "id": "case-01",
+  "prompt": "Two invoice lines are 88.50 and 44.00. Return the total as {\"total\": number}.",
+  "expected": { "total": 132.5 }
+}

package/samples/quickstart/input/cases/case-02.json

@@ -0,0 +1,5 @@
+{
+  "id": "case-02",
+  "prompt": "Round 12.345 EUR half-up to two decimals. Return {\"amount\": string, \"currency\": string}.",
+  "expected": { "amount": "12.35", "currency": "EUR" }
+}

package/samples/quickstart/input/cases/case-03.json

@@ -0,0 +1,5 @@
+{
+  "id": "case-03",
+  "prompt": "Net is 100 and tax is 8%. Return {net, tax, gross} as numbers.",
+  "expected": { "net": 100, "tax": 8, "gross": 108 }
+}

package/samples/quickstart/input/pack.json

@@ -0,0 +1,10 @@
+{
+  "name": "synthetic-invoice-basics",
+  "version": "1.0.0",
+  "description": "Three tiny invoice-arithmetic tasks for the quickstart. Synthetic; not a benchmark.",
+  "tasks": [
+    { "id": "case-01" },
+    { "id": "case-02" },
+    { "id": "case-03" }
+  ]
+}

package/samples/quickstart/input/run.json

@@ -0,0 +1,14 @@
+{
+  "run_id": "synthetic-run-2026-11-02",
+  "note": "Actual outputs of a synthetic reference run. No model was called; no customer data.",
+  "results": [
+    { "id": "case-01", "actual": { "total": 132.5 } },
+    { "id": "case-02", "actual": { "amount": "12.34", "currency": "EUR" } },
+    { "id": "case-03", "actual": { "gross": 108, "tax": 8, "net": 100 } }
+  ],
+  "sampling": [
+    { "task_id": "case-01", "samples": 10, "correct": 7 },
+    { "task_id": "case-02", "samples": 10, "correct": 3 },
+    { "task_id": "case-03", "samples": 10, "correct": 9 }
+  ]
+}

package/samples/quickstart/run.mjs

@@ -0,0 +1,82 @@
+#!/usr/bin/env node
+/**
+ * dsh-eval-pack quickstart — lints the bundled synthetic eval pack, grades the shipped
+ * reference run, and reports pass@k. `expected/result.json` is the stdout of
+ * `node samples/quickstart/run.mjs --json`, so it is a real run over the sample.
+ *
+ *   node samples/quickstart/run.mjs            # human-readable summary
+ *   node samples/quickstart/run.mjs --json     # the golden result on stdout
+ *   node samples/quickstart/run.mjs --check    # recompute and diff against the golden file
+ *
+ * Everything is local and synthetic: no model call, no network, no customer data.
+ */
+import assert from 'node:assert/strict'
+import { readFile } from 'node:fs/promises'
+import path from 'node:path'
+import { fileURLToPath } from 'node:url'
+import { createTools } from '../../lib/plugin/tools.js'
+
+const here = path.dirname(fileURLToPath(import.meta.url))
+const inputDir = path.join(here, 'input')
+const expectedPath = path.join(here, 'expected', 'result.json')
+const mode = process.argv.includes('--check') ? 'check' : process.argv.includes('--json') ? 'json' : 'summary'
+const readJson = async (name) => JSON.parse(await readFile(path.join(inputDir, name), 'utf8'))
+
+async function run() {
+  const pack = await readJson('pack.json')
+  const reference = await readJson('run.json')
+  const cases = []
+  for (const task of pack.tasks) cases.push(await readJson(path.join('cases', `${task.id}.json`)))
+
+  const tools = Object.fromEntries(createTools().map((tool) => [tool.name, tool]))
+  const call = (name, args) => tools[name].execute(args, { callId: 'quickstart' })
+  const actualById = new Map(reference.results.map((result) => [result.id, result.actual]))
+
+  const lint = await call('eval.pack_lint', { pack })
+  const results = []
+  const grades = []
+  for (const testCase of cases) {
+    const actual = actualById.get(testCase.id)
+    assert.notEqual(actual, undefined, `run.json has no result for ${testCase.id}`)
+    const graded = await call('eval.grade', { actual, expected: testCase.expected })
+    grades.push({ id: testCase.id, pass: graded.pass })
+    results.push({ id: testCase.id, actual, expected: testCase.expected })
+  }
+  const summary = await call('eval.summary', { results })
+  const pass1 = await call('eval.pass_at_k', { k: 1, tasks: reference.sampling })
+  const pass3 = await call('eval.pass_at_k', { k: 3, tasks: reference.sampling })
+  const compact = (result) => ({ mean: result.mean, tasks: result.tasks.map((task) => ({ task_id: task.task_id, pass_at_k: task.pass_at_k })) })
+
+  return {
+    sample: 'samples/quickstart/input',
+    synthetic: true,
+    pack: { name: pack.name, version: pack.version, tasks: pack.tasks.length },
+    lint: { ok: lint.ok, errors: lint.errors },
+    grades,
+    summary,
+    pass_at_1: compact(pass1),
+    pass_at_3: compact(pass3),
+  }
+}
+
+function printSummary(result) {
+  console.log('dsh-eval-pack quickstart - synthetic eval pack (not a benchmark)')
+  console.log(`pack ${result.pack.name}@${result.pack.version}: lint ${result.lint.ok ? 'ok' : result.lint.errors.join(', ')}, ${result.pack.tasks} tasks`)
+  console.log(`grades: ${result.grades.map((grade) => `${grade.id} ${grade.pass ? 'pass' : 'fail'}`).join(', ')}`)
+  console.log(`summary: ${result.summary.passed}/${result.summary.total} passed (${result.summary.pass_rate})${result.summary.failures.length ? `, failures: ${result.summary.failures.join(', ')}` : ''}`)
+  console.log(`pass@1 mean ${result.pass_at_1.mean}; pass@3 mean ${result.pass_at_3.mean}`)
+}
+
+const result = await run()
+
+if (mode === 'json') {
+  process.stdout.write(`${JSON.stringify(result, null, 2)}\n`)
+} else if (mode === 'check') {
+  const expected = JSON.parse(await readFile(expectedPath, 'utf8'))
+  assert.deepEqual(result, expected)
+  printSummary(result)
+  console.log('golden samples/quickstart/expected/result.json matches')
+  console.log('quickstart ok')
+} else {
+  printSummary(result)
+}

package/screenshots.json

@@ -1,4 +1,17 @@
 [
+  {
+    "path": "docs/assets/quickstart.png",
+    "kind": "quickstart",
+    "alt": "dsh-eval-pack quickstart run on the shipped synthetic eval pack",
+    "caption": "Real run: node samples/quickstart/run.mjs on samples/quickstart/input (3 sample JSONs + one reference run); output matches samples/quickstart/expected/result.json.",
+    "command": "node samples/quickstart/run.mjs",
+    "run_at": "2026-11-02T22:15:29+08:00",
+    "output_sha256": "2ba7ea5016ceded8fdeb186844ddd89d51125920ad9fa4bee9eef888989ea7d1",
+    "output_path": "samples/quickstart/expected/result.json",
+    "log": "docs/assets/quickstart.log",
+    "log_sha256": "bc94be001ab24a17449d124decbc283676a4ee46a61b1a18bd91613e0370594a",
+    "image_sha256": "f2e019f6129925293f80dc5147d77d4a92ff49a629a9becfa46452d08273ae0b"
+  },
   {
     "path": "docs/assets/dsh-eval-pack-hero.png",
     "kind": "hero",