dsh-eval-pack 0.1.0-alpha.6 → 0.1.0-alpha.8

package/CHANGELOG.md

@@ -1,5 +1,22 @@
 # Changelog
 
+## [0.1.0-alpha.8] — 2026-11-01
+
+### Added
+
+- `eval.summary`: aggregate pass count, pass rate and failing task ids for a graded run, using the same grader as `eval.grade`.
+- Canonical-JSON grading: object key order no longer changes the verdict; array order still matters.
+
+Verified by `node --test test/*.test.mjs` (7/7).
+
+## [0.1.0-alpha.7] — 2026-11-01
+
+### Changed
+
+- Documentation: five-language fixture/acceptance links now resolve on npm; added specific related-checker/tool links.
+- Marketplace metadata: taxonomy category, archetype, domains and status; docs page now shows the same.
+- Added a real terminal hero (and GIF for the active docs set) plus `TROUBLESHOOTING.md`.
+
 All notable changes to `dsh-eval-pack` are documented here. The format follows Keep a Changelog; this project uses alpha versions.
 
 ## [0.1.0-alpha.6] — 2026-11-01

package/README-es.md

@@ -33,6 +33,10 @@ dsh plugin --profile <profile> add dsh-eval-pack
 
 *Flujo: entrada local → plugin → salida local. No requiere llamada a un modelo.*
 
+![Real dsh-eval-pack terminal output](docs/assets/dsh-eval-pack-hero.png)
+
+*Ejecución local real; sin modelo ni red.*
+
 ## Instalación
 
 Instalar y verificar:
@@ -84,7 +88,7 @@ dsh --profile <profile> --dump-config | grep dsh-eval-pack
 
 ### Ejemplo respaldado por fixture
 
-Entrada: [`ACCEPTANCE.csv`](ACCEPTANCE.csv)
+Entrada: [`ACCEPTANCE.csv`](https://github.com/PerryLink/dsh-eval-pack/blob/main/ACCEPTANCE.csv)
 
 Comando: `pnpm test`
 
@@ -134,6 +138,9 @@ pnpm build && pnpm lint && pnpm typecheck && pnpm test
 
 Paquetes hermanos e índices de familia: [dsh-policy-ledger](https://github.com/PerryLink/dsh-policy-ledger) · [dsh-pim-lint](https://github.com/PerryLink/dsh-pim-lint)
 
+Herramientas relacionadas (tabla en inglés): [Related tools](README.md#related-tools).
+
+- [Solución de problemas (inglés)](TROUBLESHOOTING.md)
 - [Índice de dominio](https://perrylink.github.io/dsh-plugins/)
 - [Familia de checkers](https://perrylink.github.io/zh/checkers/)
 - [Familia principal](https://github.com/PerryLink/dsh-kit)

package/README-hi.md

@@ -33,6 +33,10 @@ dsh plugin --profile <profile> add dsh-eval-pack
 
 *वर्कफ़्लो: स्थानीय इनपुट → प्लगइन → स्थानीय आउटपुट। मॉडल कॉल आवश्यक नहीं।*
 
+![Real dsh-eval-pack terminal output](docs/assets/dsh-eval-pack-hero.png)
+
+*वास्तविक स्थानीय परीक्षण; कोई मॉडल या नेटवर्क कॉल नहीं।*
+
 ## इंस्टॉल
 
 इंस्टॉल और सत्यापन:
@@ -84,7 +88,7 @@ dsh --profile <profile> --dump-config | grep dsh-eval-pack
 
 ### fixture-आधारित उदाहरण
 
-इनपुट: [`ACCEPTANCE.csv`](ACCEPTANCE.csv)
+इनपुट: [`ACCEPTANCE.csv`](https://github.com/PerryLink/dsh-eval-pack/blob/main/ACCEPTANCE.csv)
 
 कमांड: `pnpm test`
 
@@ -134,6 +138,9 @@ pnpm build && pnpm lint && pnpm typecheck && pnpm test
 
 सहोदर पैकेज और परिवार सूचकांक: [dsh-policy-ledger](https://github.com/PerryLink/dsh-policy-ledger) · [dsh-pim-lint](https://github.com/PerryLink/dsh-pim-lint)
 
+संबंधित टूल (अंग्रेज़ी तालिका): [Related tools](README.md#related-tools).
+
+- [समस्या निवारण (अंग्रेज़ी)](TROUBLESHOOTING.md)
 - [डोमेन सूचकांक](https://perrylink.github.io/dsh-plugins/)
 - [चेकर परिवार](https://perrylink.github.io/zh/checkers/)
 - [मुख्य परिवार](https://github.com/PerryLink/dsh-kit)

package/README-pt.md

@@ -33,6 +33,10 @@ dsh plugin --profile <profile> add dsh-eval-pack
 
 *Fluxo: entrada local → plugin → saída local. Não exige chamada a modelo.*
 
+![Real dsh-eval-pack terminal output](docs/assets/dsh-eval-pack-hero.png)
+
+*Execução local real; sem modelo ou rede.*
+
 ## Instalação
 
 Instalar e verificar:
@@ -84,7 +88,7 @@ dsh --profile <profile> --dump-config | grep dsh-eval-pack
 
 ### Exemplo com fixture
 
-Entrada: [`ACCEPTANCE.csv`](ACCEPTANCE.csv)
+Entrada: [`ACCEPTANCE.csv`](https://github.com/PerryLink/dsh-eval-pack/blob/main/ACCEPTANCE.csv)
 
 Comando: `pnpm test`
 
@@ -134,6 +138,9 @@ pnpm build && pnpm lint && pnpm typecheck && pnpm test
 
 Pacotes irmãos e índices da família: [dsh-policy-ledger](https://github.com/PerryLink/dsh-policy-ledger) · [dsh-pim-lint](https://github.com/PerryLink/dsh-pim-lint)
 
+Ferramentas relacionadas (tabela em inglês): [Related tools](README.md#related-tools).
+
+- [Solução de problemas (inglês)](TROUBLESHOOTING.md)
 - [Índice de domínio](https://perrylink.github.io/dsh-plugins/)
 - [Família de checkers](https://perrylink.github.io/zh/checkers/)
 - [Família principal](https://github.com/PerryLink/dsh-kit)

package/README-zh.md

@@ -33,6 +33,10 @@ dsh plugin --profile <profile> add dsh-eval-pack
 
 *工作流：本地输入 → 插件 → 本地输出；不需要调用模型。*
 
+![Real dsh-eval-pack terminal output](docs/assets/dsh-eval-pack-hero.png)
+
+*本地真实测试运行；不调用模型或网络。*
+
 ## 安装
 
 安装并验证：
@@ -84,7 +88,7 @@ dsh --profile <profile> --dump-config | grep dsh-eval-pack
 
 ### fixture 支撑的示例
 
-输入: [`ACCEPTANCE.csv`](ACCEPTANCE.csv)
+输入: [`ACCEPTANCE.csv`](https://github.com/PerryLink/dsh-eval-pack/blob/main/ACCEPTANCE.csv)
 
 命令: `pnpm test`
 
@@ -128,12 +132,14 @@ pnpm build && pnpm lint && pnpm typecheck && pnpm test
 - 本地运行；不联网、不调用模型、不上传。
 - 只使用公开/合成 fixture；不含客户数据或客户证据。
 - alpha/partial；not counted，不作 GA 或生产可用声明。
-- 冻结包仍可安装；不再加新功能，只修真实故障。
 
 ## 相关项目
 
 兄弟包与家族索引： [dsh-policy-ledger](https://github.com/PerryLink/dsh-policy-ledger) · [dsh-pim-lint](https://github.com/PerryLink/dsh-pim-lint)
 
+相关工具（英文表）: [Related tools](README.md#related-tools).
+
+- [故障排查（英文）](TROUBLESHOOTING.md)
 - [领域索引](https://perrylink.github.io/dsh-plugins/)
 - [Checker 家族](https://perrylink.github.io/zh/checkers/)
 - [主线家族](https://github.com/PerryLink/dsh-kit)

package/README.md

@@ -33,6 +33,10 @@ dsh plugin --profile <profile> add dsh-eval-pack
 
 *Workflow: local input → plugin → local output. No model call is required.*
 
+![Real dsh-eval-pack terminal output](docs/assets/dsh-eval-pack-hero.png)
+
+*Real local test run; no model or network call.*
+
 ## Install
 
 Install and verify:
@@ -84,7 +88,7 @@ dsh --profile <profile> --dump-config | grep dsh-eval-pack
 
 ### Fixture-backed example
 
-Input: [`ACCEPTANCE.csv`](ACCEPTANCE.csv)
+Input: [`ACCEPTANCE.csv`](https://github.com/PerryLink/dsh-eval-pack/blob/main/ACCEPTANCE.csv)
 
 Command: `pnpm test`
 
@@ -128,12 +132,23 @@ pnpm build && pnpm lint && pnpm typecheck && pnpm test
 - Runs locally; no network, no model call, no upload.
 - Public/synthetic fixtures only; no customer data, no customer evidence.
 - Alpha/partial; not counted, no GA or production-readiness claim.
-- Frozen packages stay installable; no new features, only genuine breakage fixes.
 
 ## Related work
 
 Sibling packages and family indexes: [dsh-policy-ledger](https://github.com/PerryLink/dsh-policy-ledger) · [dsh-pim-lint](https://github.com/PerryLink/dsh-pim-lint)
 
+
+### Related tools
+
+Each entry is a separate package with its own tool surface and version; install only the ones you need.
+
+| Package | Why it is relevant |
+|---|---|
+| [`dsh-score`](https://github.com/PerryLink/dsh-score) | Scoring and evaluation utilities. |
+| [`dsh-doublecheck`](https://github.com/PerryLink/dsh-doublecheck) | Requirements/test/adversary quality gate. |
+| [`dsh-test-drive`](https://github.com/PerryLink/dsh-test-drive) | Test-driving workflows and CI checks. |
+
+- [Troubleshooting](TROUBLESHOOTING.md)
 - [Domain index](https://perrylink.github.io/dsh-plugins/)
 - [Checker family](https://perrylink.github.io/zh/checkers/)
 - [Mainline family](https://github.com/PerryLink/dsh-kit)

package/TROUBLESHOOTING.md

@@ -0,0 +1,45 @@
+# Troubleshooting — dsh-eval-pack
+
+## Install or activation fails
+
+- DSH must satisfy `>=0.2.1-rc.3 <0.3.0`; Node must be `>=24.9 <25`.
+- Run `dsh --version` and `node --version`, then retry with a fresh `DSH_HOME`:
+
+```sh
+export DSH_HOME="$(mktemp -d)"
+dsh plugin --profile <profile> add dsh-eval-pack
+dsh --profile <profile> --dump-config | grep dsh-eval-pack
+```
+
+## The tool cannot find an input
+
+- All paths are local. Use an absolute path, or a path relative to the process working directory.
+- The public samples under `fixtures/` are repository-only; they are not shipped in the npm tarball. Download them from the GitHub repository if needed.
+- Supported material: eval-pack JSON.
+
+## Parse or encoding failures
+
+- Text inputs must be UTF-8; CSV headers must match the README's documented column names.
+- Remove a UTF-8 BOM if the first header is reported as unknown.
+- Check the pack name/version and unique task ids; grading is deterministic JSON equality, not a benchmark.
+
+## Permission or write failures
+
+- Tools marked `R1 local write` write under the configured DSH data directory.
+- For tests, point `DSH_HOME` at a temporary directory; do not write into a real profile.
+
+## The result looks wrong
+
+- Check the tool boundary in the README first. These are public/synthetic alpha tools, not a production or customer-validated determination.
+- Compare the reported row ids and input rows; the output is deliberately explicit about what it could not run.
+
+## Run the tests
+
+```sh
+pnpm install --frozen-lockfile
+pnpm test
+```
+
+## Report
+
+Open an issue on <https://github.com/PerryLink/dsh-eval-pack/issues> with the package version, a minimal synthetic input, and the exact tool call. Do not attach customer data.

package/lib/core/harness.d.ts

@@ -8,4 +8,19 @@ export interface EvalPack {
     tasks: EvalTask[];
 }
 export declare function lintPack(input: unknown): string[];
+/** Stable serialisation: object keys sorted at every level, arrays kept in order. */
+export declare function canonical(value: unknown): string;
 export declare function grade(actual: unknown, expected: unknown): boolean;
+export interface EvalResult {
+    id: string;
+    actual: unknown;
+    expected: unknown;
+}
+export interface EvalSummary {
+    total: number;
+    passed: number;
+    failed: number;
+    pass_rate: number;
+    failures: string[];
+}
+export declare function summarize(results: EvalResult[]): EvalSummary;

package/lib/core/harness.js

@@ -23,6 +23,33 @@ export function lintPack(input) {
     }
     return errors;
 }
+/** Stable serialisation: object keys sorted at every level, arrays kept in order. */
+export function canonical(value) {
+    const walk = (input) => {
+        if (Array.isArray(input))
+            return input.map(walk);
+        if (input && typeof input === 'object') {
+            const out = {};
+            for (const key of Object.keys(input).sort())
+                out[key] = walk(input[key]);
+            return out;
+        }
+        return input;
+    };
+    return JSON.stringify(walk(value));
+}
 export function grade(actual, expected) {
-    return JSON.stringify(actual) === JSON.stringify(expected);
+    return canonical(actual) === canonical(expected);
+}
+export function summarize(results) {
+    const failures = [];
+    let passed = 0;
+    for (const r of results) {
+        if (grade(r.actual, r.expected))
+            passed += 1;
+        else
+            failures.push(r.id);
+    }
+    const total = results.length;
+    return { total, passed, failed: total - passed, pass_rate: total ? Math.round((passed / total) * 1e6) / 1e6 : 0, failures };
 }

package/lib/plugin/tools.js

@@ -1,5 +1,5 @@
 import { defineTool } from '@deepseek-ai/dsh-tools';
-import { grade, lintPack } from '../core/harness.js';
+import { grade, lintPack, summarize } from '../core/harness.js';
 const text = (t) => [{ type: 'text', text: t }];
 export function createTools() {
     const lint = defineTool({
@@ -16,5 +16,12 @@ export function createTools() {
         output: { schema: { type: 'object', additionalProperties: false, properties: { pass: { type: 'boolean', required: true } } }, render: (_a, v) => text(`grade ${v.pass ? 'pass' : 'fail'}`) },
         async execute(args) { return { pass: grade(args.actual, args.expected) }; },
     });
-    return [lint, gradeTool];
+    const summary = defineTool({
+        name: 'eval.summary',
+        description: 'Aggregate graded results for a public/synthetic run: pass count, pass rate and the failing task ids, using the same canonical-JSON grader. Read-only; no runner.',
+        parameters: { results: { type: 'array', required: true, items: { type: 'object', additionalProperties: false, properties: { id: { type: 'string', required: true }, actual: { type: 'json', required: true }, expected: { type: 'json', required: true } } } } },
+        output: { schema: { type: 'object', additionalProperties: false, properties: { total: { type: 'number', required: true }, passed: { type: 'number', required: true }, failed: { type: 'number', required: true }, pass_rate: { type: 'number', required: true }, failures: { type: 'array', items: { type: 'string' }, required: true } } }, render: (_a, v) => text(`summary ${v.passed}/${v.total}`) },
+        async execute(args) { return summarize(args.results); },
+    });
+    return [lint, gradeTool, summary];
 }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dsh-eval-pack",
-  "version": "0.1.0-alpha.6",
+  "version": "0.1.0-alpha.8",
   "description": "Public/synthetic eval-pack lint and deterministic JSON grader; not a validated benchmark.",
   "type": "module",
   "main": "./lib/plugin/index.js",
@@ -27,6 +27,8 @@
     "CODE_OF_CONDUCT.md",
     "CONTRIBUTING.md",
     "CHANGELOG.md",
+    "TROUBLESHOOTING.md",
+    "screenshots.json",
     "icon.svg"
   ],
   "dsh": {

package/screenshots.json

@@ -0,0 +1,20 @@
+[
+  {
+    "path": "docs/assets/dsh-eval-pack-hero.png",
+    "kind": "hero",
+    "alt": "dsh-eval-pack terminal output",
+    "caption": "Real dsh-eval-pack local test run; 3 tests passed."
+  },
+  {
+    "path": "docs/assets/dsh-eval-pack-demo.gif",
+    "kind": "demo",
+    "alt": "Animated dsh-eval-pack terminal session",
+    "caption": "The same run, assembled line by line."
+  },
+  {
+    "path": "docs/assets/dsh-eval-pack-capability.png",
+    "kind": "capability",
+    "alt": "dsh-eval-pack real test run",
+    "caption": "Real command: node --test test/summary.test.mjs — 4 tests passed, 0 failed (canonical grading and run summary). Run 2026-11-01; the image is rendered from the captured terminal output."
+  }
+]