dsh-auto-review 0.4.0 → 0.5.0

package/CHANGELOG.md

@@ -2,6 +2,26 @@
 
 All notable changes to `dsh-auto-review` are documented here. The repo is pre-release; versions follow the DeepSeek Harness `0.1.0-rc.x` target runtime and bump on every behavior change.
 
+## [0.5.0] — 2026-08-16
+
+### Added
+
+- **dsh-eval agent evaluation engine** (`dsh-auto-review/eval` + the `dsh-eval` CLI): a YAML case DSL (`input`, structured `expect` block — tool-call sequence with argument matchers, per-tool results, final-output matchers, turn outcome, token budget — optional second-model `review` block, per-case `timeoutMs`/`model`/`tier`, workspace `files`/`seedFrom`, batch suites), one isolated headless session per case (fresh scratch workspace, official Minimal persona as the baseline system prompt, approval `never`, workspace-write sandbox), tool-call-trace collection from the session event log, and Markdown/JSON reports with token stats, per-assertion explanations, replayable session-log artifacts, and a CI gate (exit 0 only when every case of every suite passed).
+- The second-model review is a supplementary assertion layer over the same run, reusing the approval reviewer's subagent seam (the mounted runtime is now published as `ctx.autoReviewRuntime`, so the eval engine reads the exact mounted configuration instead of a driftable copy).
+- Shipped evaluation composition `eval/cordis.yml`, the self-regression suite `eval/cases/demo.yaml` (output assertion, tool-trace assertion, second-model review — three cases over this repository), the `bin/dsh-eval.mjs` launcher, and the `./eval` / `./eval-cli` package exports.
+- CLI flags: `--provider`, `--model`, `--tier`, `--timeout-ms`, `--concurrency`, `--out`, `--workspace-root`, `--keep-workspaces`, `--no-markdown`, `--no-gate`, `--review-provider`, `--review-model`, `--review-timeout-ms`. A case whose model or timeout resolves from nothing fails the suite loudly (no hardcoded defaults); runs are cancellable (`AbortSignal` + SIGINT/SIGTERM) and the worker pool respects the concurrency cap.
+
+### Changed
+
+- README (all five languages) documents the evaluation engine with a case example, a CI integration snippet, and the contrast with codex-research.
+- The unit suite covers the eval engine end to end (DSL, assertion engine, trace collection, runner, reports, CLI).
+
+## [0.4.1] — 2026-08-15
+
+### Added
+
+- `package.json#dshWorkshop` manifest (`omdsh-workshop-package/v1`): transactional `profile-bundle` install declaration, `harness-profile` integration protocol, restart-profile lifecycle, the `/auto-review status` named capability, and author-run install/remove evidence under `docs/omdsh-evidence/` — the omdsh hub intake surface (author declarations only; verification stays with the hub).
+
 ## [0.4.0] — 2026-08-15
 
 ### Added

package/README.es.md

@@ -11,6 +11,8 @@ Cuando una acción del agente cruza el límite del sandbox, un **subagente revis
 
 [![license](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE)
 [![dsh](https://img.shields.io/badge/dsh-0.1.0--rc.6-4c51bf.svg)](https://www.npmjs.com/package/@deepseek-ai/dsh)
+[![npm version](https://img.shields.io/npm/v/dsh-auto-review)](https://www.npmjs.com/package/dsh-auto-review)
+[![npm downloads](https://img.shields.io/npm/dm/dsh-auto-review)](https://www.npmjs.com/package/dsh-auto-review)
 [![tests](https://img.shields.io/github/actions/workflow/status/PerryLink/dsh-auto-review/ci.yml?label=tests&logo=githubactions)](.github/workflows/ci.yml)
 [![typescript](https://img.shields.io/badge/TypeScript-strict-3178c6.svg)](src)
 [![type](https://img.shields.io/badge/type-cordis%20bundle-8a5cf6.svg)](cordis.patch.yml)
@@ -177,6 +179,43 @@ Cómo está cableado:
 
 El panel solo lee valores de proyección completos — nunca recibe el flujo crudo de eventos de sesión.
 
+## 🧪 dsh-eval — motor de evaluación de agentes
+
+Más allá del revisor de aprobación, `dsh-auto-review` incluye `dsh-eval`: una plataforma de evaluación de agentes dirigida por YAML que ejecuta sesiones DSH headless reales (un agente aislado + un workspace de scratch por caso, la persona oficial Minimal como prompt de sistema base), recoge la traza de llamadas a herramientas del registro de eventos de la sesión y evalúa aserciones estructuradas más una revisión opcional de segundo modelo — el mismo seam de revisor que el answerer de aprobación.
+
+```yaml
+# eval/cases/demo.yaml (resumido)
+suite:
+  name: my-suite
+  cases:
+    - id: math-output
+      input: Solve 17 × 24 and reply with only the final number, nothing else.
+      expect:
+        output: { contains: "408" }
+    - id: glob-trace
+      seedFrom: '.'
+      input: Use the glob tool with pattern "src/**" to list the source files…
+      expect:
+        toolCalls: [{ tool: glob, arguments: { contains: { pattern: "src" } } }]
+        results: [{ tool: glob, contains: "index.ts" }]
+    - id: review-write
+      input: Read src/config.ts, write the default reviewerTimeoutMs into scratch/answer.txt…
+      expect:
+        output: { contains: "60000" }
+      review:
+        statement: The agent read the default reviewerTimeoutMs and wrote it to the file.
+```
+
+Ejecútalo (una clave de API de DeepSeek debe estar en el entorno):
+
+```sh
+dsh-eval eval/cases --model deepseek-v4-flash --timeout-ms 240000 --out .eval-reports
+```
+
+Puerta de CI: el proceso sale con 0 solo cuando todos los casos de todas las suites han pasado — intégralo en un paso de GitHub Action y las evaluaciones fallidas harán fallar el build. Cada caso deja un JSONL de sesión reproducible y un JSON de traza junto a `report.md`/`report.json`; los resultados de las aserciones, el uso de tokens y el veredicto de la revisión se escriben todos en los archivos de reporte. El motor nunca sustituye valores por defecto de modelo o timeout hardcodeados, aborta limpiamente con SIGINT/SIGTERM y limita el pool de workers a la concurrencia configurada.
+
+A diferencia de [codex-research](https://github.com/openai/codex/tree/main/codex-rs/research) (investigación de agentes de automatización de navegador), `dsh-eval` apunta a la evaluación de agentes a nivel de harness: aserciones sobre la traza de llamadas a herramientas contra el registro de eventos de la sesión, revisión de segundo modelo como capa de aserción suplementaria y sesiones headless aisladas por caso — sin pila de navegador ni Selenium.
+
 ## 🔒 Seguridad
 
 - El revisor corre con una **cara de herramientas de solo lectura** (lista blanca `toolFilter`). No puede escribir, editar, ejecutar shell, acceder a la red ni delegar (`maxDepth` = su propia profundidad). Su registro de sesión persiste y es auditable.
@@ -213,13 +252,21 @@ Recomendados al publicar: `dsh` · `dsh-plugin` · `deepseek-harness` · `deepse
 ```sh
 pnpm install                # node ^22.19 || >=24
 pnpm run typecheck          # tsc, src + tests
-pnpm test                   # vitest: 135 tests, 8 suites
+pnpm test                   # vitest: 190 tests, 14 files
 pnpm run build              # declaraciones tsc + bundles tsdown (lib/)
 pnpm run verify:self-contained
 pnpm pack                   # artefacto de publicación
 ```
 
-Estructura del repositorio (estructura plugin-template): `src/index.ts` (contrato del plugin) · `src/config.ts` (esquema Schemastery + resolución) · `src/runtime.ts` (answerer, comando, inyección del motivo de denegación) · `src/review.ts` (orquestación del revisor, prompt, redacción) · `src/events.ts` (vocabulario de eventos de sesión + folds) · `src/projection.ts` + `src/projection-types.ts` (la proyección de sesión `autoReview`) · `src/invariant.ts` (companion de invariantes) · `src/client/` (mitad navegador: panel de revisión, locales, estilos) · `test/` · `fixtures/`.
+Estructura del repositorio (estructura plugin-template): `src/index.ts` (contrato del plugin) · `src/config.ts` (esquema Schemastery + resolución) · `src/runtime.ts` (answerer, comando, inyección del motivo de denegación) · `src/review.ts` (orquestación del revisor, prompt, redacción) · `src/events.ts` (vocabulario de eventos de sesión + folds) · `src/projection.ts` + `src/projection-types.ts` (la proyección de sesión `autoReview`) · `src/invariant.ts` (companion de invariantes) · `src/eval/` (el motor dsh-eval: DSL, runner, assertions, trace, review, reports, CLI) · `eval/` (composición de evaluación incluida + demo suite) · `bin/dsh-eval.mjs` (lanzador CLI) · `src/client/` (mitad navegador: panel de revisión, locales, estilos) · `test/` · `fixtures/`.
+
+## 👥 Contribuidores
+
+Gracias a todos los que han contribuido a `dsh-auto-review`:
+
+- [PerryLink](https://github.com/PerryLink) — autor y mantenedor: answerer de aprobación, subagente revisor, política de riesgo y disyuntor, panel de revisión por proyección de sesión, companion de invariantes, documentación, CI/CD y lanzamientos.
+
+¿Quieres ayudar? Revisa las [plantillas de issues](.github/ISSUE_TEMPLATE/), la [política de seguridad](SECURITY.md) y [AGENTS.md](AGENTS.md) con las convenciones del repositorio — los PR son bienvenidos en inglés o chino.
 
 ## 📄 Licencia
 

package/README.hi.md

@@ -11,6 +11,8 @@
 
 [![license](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE)
 [![dsh](https://img.shields.io/badge/dsh-0.1.0--rc.6-4c51bf.svg)](https://www.npmjs.com/package/@deepseek-ai/dsh)
+[![npm version](https://img.shields.io/npm/v/dsh-auto-review)](https://www.npmjs.com/package/dsh-auto-review)
+[![npm downloads](https://img.shields.io/npm/dm/dsh-auto-review)](https://www.npmjs.com/package/dsh-auto-review)
 [![tests](https://img.shields.io/github/actions/workflow/status/PerryLink/dsh-auto-review/ci.yml?label=tests&logo=githubactions)](.github/workflows/ci.yml)
 [![typescript](https://img.shields.io/badge/TypeScript-strict-3178c6.svg)](src)
 [![type](https://img.shields.io/badge/type-cordis%20bundle-8a5cf6.svg)](cordis.patch.yml)
@@ -177,6 +179,43 @@ dsh --profile web --dump-config | grep -A4 'id: auto-review'
 
 पैनल केवल संपूर्ण प्रोजेक्शन मान पढ़ता है — ब्राउज़र प्लगइन को कच्ची सत्र-ईवेंट धारा कभी नहीं मिलती।
 
+## 🧪 dsh-eval — एजेंट मूल्यांकन इंजन
+
+अनुमोदन समीक्षक के अलावा, `dsh-auto-review` `dsh-eval` भी देता है: एक YAML-चालित एजेंट मूल्यांकन प्लेटफ़ॉर्म जो वास्तविक headless DSH सत्र चलाता है (प्रति केस एक अलग-थलग एजेंट + scratch workspace, आधारभूत सिस्टम prompt के रूप में आधिकारिक Minimal persona), सत्र ईवेंट लॉग से टूल-कॉल ट्रेस एकत्र करता है, और संरचित assertions तथा एक वैकल्पिक द्वितीय-मॉडल समीक्षा का मूल्यांकन करता है — वही समीक्षक seam जो अनुमोदन answerer का है।
+
+```yaml
+# eval/cases/demo.yaml (संक्षिप्त)
+suite:
+  name: my-suite
+  cases:
+    - id: math-output
+      input: Solve 17 × 24 and reply with only the final number, nothing else.
+      expect:
+        output: { contains: "408" }
+    - id: glob-trace
+      seedFrom: '.'
+      input: Use the glob tool with pattern "src/**" to list the source files…
+      expect:
+        toolCalls: [{ tool: glob, arguments: { contains: { pattern: "src" } } }]
+        results: [{ tool: glob, contains: "index.ts" }]
+    - id: review-write
+      input: Read src/config.ts, write the default reviewerTimeoutMs into scratch/answer.txt…
+      expect:
+        output: { contains: "60000" }
+      review:
+        statement: The agent read the default reviewerTimeoutMs and wrote it to the file.
+```
+
+इसे चलाएँ (वातावरण में एक DeepSeek API कुंजी मौजूद होनी चाहिए):
+
+```sh
+dsh-eval eval/cases --model deepseek-v4-flash --timeout-ms 240000 --out .eval-reports
+```
+
+CI द्वार: प्रक्रिया 0 से तभी बाहर निकलती है जब हर suite के हर केस पास हुए हों — इसे एक GitHub Action चरण में डालें और विफल मूल्यांकन build को विफल कर देंगे। हर केस `report.md`/`report.json` के पास एक रीप्ले-योग्य सत्र JSONL और एक ट्रेस JSON छोड़ता है; assertion परिणाम, token उपयोग और समीक्षा फ़ैसला सभी रिपोर्ट फ़ाइलों में लिखे जाते हैं। इंजन कभी भी hardcoded मॉडल या टाइमआउट डिफ़ॉल्ट नहीं डालता, SIGINT/SIGTERM पर साफ़-सुथरे ढंग से रुकता है, और worker pool को कॉन्फ़िगर की गई concurrency तक सीमित रखता है।
+
+[codex-research](https://github.com/openai/codex/tree/main/codex-rs/research) (ब्राउज़र-ऑटोमेशन एजेंट अनुसंधान) के विपरीत, `dsh-eval` harness-स्तर एजेंट मूल्यांकन को लक्ष्य करता है: सत्र ईवेंट लॉग के विरुद्ध टूल-कॉल-ट्रेस assertions, पूरक assertion परत के रूप में द्वितीय-मॉडल समीक्षा, और प्रति-केस अलग-थलग headless सत्र — कोई ब्राउज़र या Selenium स्टैक नहीं।
+
 ## 🔒 सुरक्षा
 
 - समीक्षक **केवल-पढ़ने वाले टूल-सेट** (`toolFilter` अनुमति-सूची) में चलता है। वह लिख, बदल, shell चला, नेटवर्क छू या आगे delegation नहीं कर सकता (`maxDepth` = उसकी अपनी गहराई)। उसका सत्र लॉग स्थायी और ऑडिट-योग्य है।
@@ -213,13 +252,21 @@ dsh --profile web --dump-config | grep -A4 'id: auto-review'
 ```sh
 pnpm install                # node ^22.19 || >=24
 pnpm run typecheck          # tsc, src + टेस्ट
-pnpm test                   # vitest: 135 टेस्ट, 8 सुइट
+pnpm test                   # vitest: 190 टेस्ट, 14 फ़ाइलें
 pnpm run build              # tsc डिक्लेरेशन + tsdown बंडल (lib/)
 pnpm run verify:self-contained
 pnpm pack                   # प्रकाशन आर्टिफ़ैक्ट
 ```
 
-रेपो संरचना (plugin-template संरचना): `src/index.ts` (प्लगइन अनुबंध) · `src/config.ts` (Schemastery schema + रिज़ॉल्यूशन) · `src/runtime.ts` (answerer, कमांड, अस्वीकृति-कारण इंजेक्शन) · `src/review.ts` (समीक्षक ऑर्केस्ट्रेशन, prompt, redaction) · `src/events.ts` (सत्र-ईवेंट शब्दावली + folds) · `src/projection.ts` + `src/projection-types.ts` (`autoReview` सत्र प्रोजेक्शन) · `src/invariant.ts` (invariant companion) · `src/client/` (ब्राउज़र आधा: समीक्षा पैनल, locales, स्टाइल) · `test/` · `fixtures/`।
+रेपो संरचना (plugin-template संरचना): `src/index.ts` (प्लगइन अनुबंध) · `src/config.ts` (Schemastery schema + रिज़ॉल्यूशन) · `src/runtime.ts` (answerer, कमांड, अस्वीकृति-कारण इंजेक्शन) · `src/review.ts` (समीक्षक ऑर्केस्ट्रेशन, prompt, redaction) · `src/events.ts` (सत्र-ईवेंट शब्दावली + folds) · `src/projection.ts` + `src/projection-types.ts` (`autoReview` सत्र प्रोजेक्शन) · `src/invariant.ts` (invariant companion) · `src/eval/` (dsh-eval इंजन: DSL, runner, assertions, trace, review, reports, CLI) · `eval/` (शामिल मूल्यांकन संयोजन + demo suite) · `bin/dsh-eval.mjs` (CLI लॉन्चर) · `src/client/` (ब्राउज़र आधा: समीक्षा पैनल, locales, स्टाइल) · `test/` · `fixtures/`।
+
+## 👥 योगदानकर्ता
+
+`dsh-auto-review` में योगदान देने वाले सभी लोगों का धन्यवाद:
+
+- [PerryLink](https://github.com/PerryLink) — लेखक और अनुरक्षक: approval answerer, समीक्षक सबएजेंट, जोखिम नीति व circuit breaker, सत्र-प्रोजेक्शन समीक्षा पैनल, invariant companion, दस्तावेज़, CI/CD और रिलीज़।
+
+मदद करना चाहते हैं? [issue टेम्पलेट](.github/ISSUE_TEMPLATE/), [सुरक्षा नीति](SECURITY.md) और [AGENTS.md](AGENTS.md) की रेपो परंपराएँ देखें — अंग्रेज़ी या चीनी में PR का स्वागत है।
 
 ## 📄 लाइसेंस
 

package/README.md

@@ -11,6 +11,8 @@ When an agent's action crosses the sandbox boundary, a **read-only reviewer suba
 
 [![license](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE)
 [![dsh](https://img.shields.io/badge/dsh-0.1.0--rc.6-4c51bf.svg)](https://www.npmjs.com/package/@deepseek-ai/dsh)
+[![npm version](https://img.shields.io/npm/v/dsh-auto-review)](https://www.npmjs.com/package/dsh-auto-review)
+[![npm downloads](https://img.shields.io/npm/dm/dsh-auto-review)](https://www.npmjs.com/package/dsh-auto-review)
 [![tests](https://img.shields.io/github/actions/workflow/status/PerryLink/dsh-auto-review/ci.yml?label=tests&logo=githubactions)](.github/workflows/ci.yml)
 [![typescript](https://img.shields.io/badge/TypeScript-strict-3178c6.svg)](src)
 [![type](https://img.shields.io/badge/type-cordis%20bundle-8a5cf6.svg)](cordis.patch.yml)
@@ -177,6 +179,43 @@ How it is wired:
 
 The panel reads only whole projection values — it never receives the raw session event stream.
 
+## 🧪 dsh-eval — agent evaluation engine
+
+Beyond the approval reviewer, `dsh-auto-review` ships `dsh-eval`: a YAML-driven agent evaluation platform that runs real headless DSH sessions (one isolated agent + scratch workspace per case, the official Minimal persona as the baseline system prompt), collects the tool-call trace from the session event log, and evaluates structured assertions plus an optional second-model review — the same reviewer seam as the approval answerer.
+
+```yaml
+# eval/cases/demo.yaml (abridged)
+suite:
+  name: my-suite
+  cases:
+    - id: math-output
+      input: Solve 17 × 24 and reply with only the final number, nothing else.
+      expect:
+        output: { contains: "408" }
+    - id: glob-trace
+      seedFrom: '.'
+      input: Use the glob tool with pattern "src/**" to list the source files…
+      expect:
+        toolCalls: [{ tool: glob, arguments: { contains: { pattern: "src" } } }]
+        results: [{ tool: glob, contains: "index.ts" }]
+    - id: review-write
+      input: Read src/config.ts, write the default reviewerTimeoutMs into scratch/answer.txt…
+      expect:
+        output: { contains: "60000" }
+      review:
+        statement: The agent read the default reviewerTimeoutMs and wrote it to the file.
+```
+
+Run it (a DeepSeek API key must be in the environment):
+
+```sh
+dsh-eval eval/cases --model deepseek-v4-flash --timeout-ms 240000 --out .eval-reports
+```
+
+CI gate: the process exits 0 only when every case of every suite passed — drop it into a GitHub Action step and failing evaluations fail the build. Each case leaves a replayable session JSONL and a trace JSON beside `report.md`/`report.json`; assertion results, token usage, and the review verdict are all written into the report files. The engine never substitutes hardcoded model or timeout defaults, aborts cleanly on SIGINT/SIGTERM, and caps the worker pool at the configured concurrency.
+
+Unlike [codex-research](https://github.com/openai/codex/tree/main/codex-rs/research) (browser-automation agent research), `dsh-eval` targets harness-level agent evaluation: tool-call-trace assertions against the session event log, second-model review as a supplementary assertion layer, and per-case isolated headless sessions — no browser or Selenium stack.
+
 ## 🔒 Security
 
 - The reviewer runs in a **read-only tool face** (`toolFilter` allow-list). It cannot write, edit, run bash, fetch the network, or delegate (`maxDepth` = its own depth). Its session log is persisted and auditable.
@@ -213,13 +252,43 @@ Recommended when you publish: `dsh` · `dsh-plugin` · `deepseek-harness` · `de
 ```sh
 pnpm install                # node ^22.19 || >=24
 pnpm run typecheck          # tsc, src + tests
-pnpm test                   # vitest: 135 tests, 8 suites
+pnpm test                   # vitest: 190 tests, 14 files
 pnpm run build              # tsc declarations + tsdown bundles (lib/, incl. the client bundle)
 pnpm run verify:self-contained
 pnpm pack                   # publish artifact
 ```
 
-Repository layout (plugin-template structure): `src/index.ts` (plugin contract) · `src/config.ts` (Schemastery schema + resolution) · `src/runtime.ts` (answerer, command, deny-reason injection) · `src/review.ts` (reviewer orchestration, prompt, sanitization) · `src/events.ts` (session-event vocabulary + folds) · `src/projection.ts` + `src/projection-types.ts` (the `autoReview` session projection) · `src/invariant.ts` (invariant companion) · `src/client/` (browser half: review panel, locales, styles) · `test/` · `fixtures/`.
+Repository layout (plugin-template structure): `src/index.ts` (plugin contract) · `src/config.ts` (Schemastery schema + resolution) · `src/runtime.ts` (answerer, command, deny-reason injection) · `src/review.ts` (reviewer orchestration, prompt, sanitization) · `src/events.ts` (session-event vocabulary + folds) · `src/projection.ts` + `src/projection-types.ts` (the `autoReview` session projection) · `src/invariant.ts` (invariant companion) · `src/eval/` (the dsh-eval engine: DSL, runner, assertions, trace, review, reports, CLI) · `eval/` (shipped evaluation composition + demo suite) · `bin/dsh-eval.mjs` (CLI launcher) · `src/client/` (browser half: review panel, locales, styles) · `test/` · `fixtures/`.
+
+## 👥 Contributors
+
+Thanks to everyone who has contributed to `dsh-auto-review`:
+
+- [PerryLink](https://github.com/PerryLink) — author and maintainer: approval answerer, reviewer subagent, risk policy and circuit breaker, session-projection review panel, invariant companion, docs, CI/CD and releases.
+
+Want to help? Check the [issue templates](.github/ISSUE_TEMPLATE/), the [security policy](SECURITY.md), and [AGENTS.md](AGENTS.md) for repo conventions — PRs are welcome in English or Chinese.
+
+## PerryLink DSH Plugin Family
+
+This project is one of the [15 DeepSeek Harness plugins](https://github.com/PerryLink) maintained by [PerryLink](https://github.com/PerryLink). If this one helps you, the others likely will too:
+
+| Plugin | One-liner |
+|---|---|
+| [dsh-mcp-panel](https://github.com/PerryLink/dsh-mcp-panel) | Read-only MCP runtime panel: /mcp command + Settings tab with status, tools and errors |
+| [dsh-doublecheck](https://github.com/PerryLink/dsh-doublecheck) | Engineering-discipline guard: requirements grill, test gates, adversary review |
+| [dsh-background-agents](https://github.com/PerryLink/dsh-background-agents) | Durable background child agents with a Web UI sidebar, messaging and interrupt |
+| [dsh-lsp-actions](https://github.com/PerryLink/dsh-lsp-actions) | LSP diagnostics, formatting, completion, code actions and rename over language servers |
+| [dsh-output-styles](https://github.com/PerryLink/dsh-output-styles) | Claude Code outputStyles-equivalent runtime style switching |
+| [dsh-checkpoint-rewind](https://github.com/PerryLink/dsh-checkpoint-rewind) | Claude Code /rewind-equivalent: snapshots, session forks, one-shot restore |
+| [dsh-permission-rules](https://github.com/PerryLink/dsh-permission-rules) | Claude Code-style declarative allow/deny/ask permission rules with audit |
+| **[dsh-auto-review](https://github.com/PerryLink/dsh-auto-review)** | Second-model auto-review on the approval chain, fail-closed by default |
+| [dsh-memento](https://github.com/PerryLink/dsh-memento) | Approval-gated cross-session memory: ctx.memory seam + SQLite + memory tool |
+| [dsh-skill-pack-security](https://github.com/PerryLink/dsh-skill-pack-security) | Security-audit skill pack: secret scan, dependency and supply-chain review |
+| [dsh-session-pin](https://github.com/PerryLink/dsh-session-pin) | Pin sessions in the Web sidebar with durable ordering |
+| [dsh-composer-history](https://github.com/PerryLink/dsh-composer-history) | Terminal-style input history for the web composer: arrows, Ctrl+R search |
+| [dsh-github](https://github.com/PerryLink/dsh-github) | GitHub PR/issues integration for DSH, every write gated by approval |
+| [dsh-plugin-guide](https://github.com/PerryLink/dsh-plugin-guide) | Plugin-development knowledge base as an on-demand agent skill |
+| [dsh-claude-move](https://github.com/PerryLink/dsh-claude-move) | Migrate Claude Code sessions, memory, skills and CLAUDE.md into DSH |
 
 ## 📄 License
 

package/README.pt.md

@@ -11,6 +11,8 @@ Quando uma ação do agente cruza o limite do sandbox, um **subagente revisor so
 
 [![license](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE)
 [![dsh](https://img.shields.io/badge/dsh-0.1.0--rc.6-4c51bf.svg)](https://www.npmjs.com/package/@deepseek-ai/dsh)
+[![npm version](https://img.shields.io/npm/v/dsh-auto-review)](https://www.npmjs.com/package/dsh-auto-review)
+[![npm downloads](https://img.shields.io/npm/dm/dsh-auto-review)](https://www.npmjs.com/package/dsh-auto-review)
 [![tests](https://img.shields.io/github/actions/workflow/status/PerryLink/dsh-auto-review/ci.yml?label=tests&logo=githubactions)](.github/workflows/ci.yml)
 [![typescript](https://img.shields.io/badge/TypeScript-strict-3178c6.svg)](src)
 [![type](https://img.shields.io/badge/type-cordis%20bundle-8a5cf6.svg)](cordis.patch.yml)
@@ -177,6 +179,43 @@ Como é ligado:
 
 O painel lê apenas valores de projeção completos — nunca recebe o fluxo bruto de eventos de sessão.
 
+## 🧪 dsh-eval — motor de avaliação de agentes
+
+Além do revisor de aprovação, o `dsh-auto-review` inclui o `dsh-eval`: uma plataforma de avaliação de agentes dirigida por YAML que executa sessões DSH headless reais (um agente isolado + workspace de scratch por caso, a persona oficial Minimal como prompt de sistema base), coleta a trilha de chamadas de ferramentas do registro de eventos da sessão e avalia asserções estruturadas mais uma revisão opcional de segundo modelo — o mesmo seam de revisor do answerer de aprovação.
+
+```yaml
+# eval/cases/demo.yaml (resumido)
+suite:
+  name: my-suite
+  cases:
+    - id: math-output
+      input: Solve 17 × 24 and reply with only the final number, nothing else.
+      expect:
+        output: { contains: "408" }
+    - id: glob-trace
+      seedFrom: '.'
+      input: Use the glob tool with pattern "src/**" to list the source files…
+      expect:
+        toolCalls: [{ tool: glob, arguments: { contains: { pattern: "src" } } }]
+        results: [{ tool: glob, contains: "index.ts" }]
+    - id: review-write
+      input: Read src/config.ts, write the default reviewerTimeoutMs into scratch/answer.txt…
+      expect:
+        output: { contains: "60000" }
+      review:
+        statement: The agent read the default reviewerTimeoutMs and wrote it to the file.
+```
+
+Execute (uma chave de API da DeepSeek deve estar no ambiente):
+
+```sh
+dsh-eval eval/cases --model deepseek-v4-flash --timeout-ms 240000 --out .eval-reports
+```
+
+Portão de CI: o processo sai com 0 somente quando todos os casos de todas as suítes passaram — coloque-o em uma etapa de GitHub Action e avaliações que falharem farão o build falhar. Cada caso deixa um JSONL de sessão reproduzível e um JSON de trilha ao lado de `report.md`/`report.json`; os resultados das asserções, o uso de tokens e o veredicto da revisão são todos gravados nos arquivos de relatório. O motor nunca substitui padrões hardcoded de modelo ou timeout, aborta de forma limpa em SIGINT/SIGTERM e limita o pool de workers à concorrência configurada.
+
+Diferente do [codex-research](https://github.com/openai/codex/tree/main/codex-rs/research) (pesquisa de agentes de automação de navegador), o `dsh-eval` mira a avaliação de agentes em nível de harness: asserções sobre a trilha de chamadas de ferramentas contra o registro de eventos da sessão, revisão de segundo modelo como camada de asserção suplementar e sessões headless isoladas por caso — sem pilha de navegador nem Selenium.
+
 ## 🔒 Segurança
 
 - O revisor roda com uma **face de ferramentas somente leitura** (lista de permissões `toolFilter`). Ele não pode escrever, editar, executar shell, acessar a rede nem delegar (`maxDepth` = sua própria profundidade). Seu log de sessão persiste e é auditável.
@@ -213,13 +252,21 @@ Recomendados ao publicar: `dsh` · `dsh-plugin` · `deepseek-harness` · `deepse
 ```sh
 pnpm install                # node ^22.19 || >=24
 pnpm run typecheck          # tsc, src + testes
-pnpm test                   # vitest: 135 testes, 8 suítes
+pnpm test                   # vitest: 190 testes, 14 arquivos
 pnpm run build              # declarações tsc + bundles tsdown (lib/)
 pnpm run verify:self-contained
 pnpm pack                   # artefato de publicação
 ```
 
-Estrutura do repositório (estrutura plugin-template): `src/index.ts` (contrato do plugin) · `src/config.ts` (schema Schemastery + resolução) · `src/runtime.ts` (answerer, comando, injeção do motivo de negação) · `src/review.ts` (orquestração do revisor, prompt, redação) · `src/events.ts` (vocabulário de eventos de sessão + folds) · `src/projection.ts` + `src/projection-types.ts` (a projeção de sessão `autoReview`) · `src/invariant.ts` (companion de invariantes) · `src/client/` (metade navegador: painel de revisão, locales, estilos) · `test/` · `fixtures/`.
+Estrutura do repositório (estrutura plugin-template): `src/index.ts` (contrato do plugin) · `src/config.ts` (schema Schemastery + resolução) · `src/runtime.ts` (answerer, comando, injeção do motivo de negação) · `src/review.ts` (orquestração do revisor, prompt, redação) · `src/events.ts` (vocabulário de eventos de sessão + folds) · `src/projection.ts` + `src/projection-types.ts` (a projeção de sessão `autoReview`) · `src/invariant.ts` (companion de invariantes) · `src/eval/` (o motor dsh-eval: DSL, runner, assertions, trace, review, reports, CLI) · `eval/` (composição de avaliação incluída + demo suite) · `bin/dsh-eval.mjs` (lançador CLI) · `src/client/` (metade navegador: painel de revisão, locales, estilos) · `test/` · `fixtures/`.
+
+## 👥 Contribuidores
+
+Obrigado a todos que contribuíram com o `dsh-auto-review`:
+
+- [PerryLink](https://github.com/PerryLink) — autor e mantenedor: answerer de aprovação, subagente revisor, política de risco e disjuntor, painel de revisão via projeção de sessão, companion de invariantes, documentação, CI/CD e lançamentos.
+
+Quer ajudar? Veja os [modelos de issues](.github/ISSUE_TEMPLATE/), a [política de segurança](SECURITY.md) e o [AGENTS.md](AGENTS.md) com as convenções do repositório — PRs são bem-vindos em inglês ou chinês.
 
 ## 📄 Licença
 

package/README.zh.md

@@ -11,6 +11,8 @@
 
 [![license](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE)
 [![dsh](https://img.shields.io/badge/dsh-0.1.0--rc.6-4c51bf.svg)](https://www.npmjs.com/package/@deepseek-ai/dsh)
+[![npm version](https://img.shields.io/npm/v/dsh-auto-review)](https://www.npmjs.com/package/dsh-auto-review)
+[![npm downloads](https://img.shields.io/npm/dm/dsh-auto-review)](https://www.npmjs.com/package/dsh-auto-review)
 [![tests](https://img.shields.io/github/actions/workflow/status/PerryLink/dsh-auto-review/ci.yml?label=tests&logo=githubactions)](.github/workflows/ci.yml)
 [![typescript](https://img.shields.io/badge/TypeScript-strict-3178c6.svg)](src)
 [![type](https://img.shields.io/badge/type-cordis%20bundle-8a5cf6.svg)](cordis.patch.yml)
@@ -177,6 +179,43 @@ dsh --profile web --dump-config | grep -A4 'id: auto-review'
 
 面板只读取投影整值——浏览器插件永远不会收到原始会话事件流。
 
+## 🧪 dsh-eval — agent 评估引擎
+
+除审批审查代理之外，`dsh-auto-review` 还附带 `dsh-eval`：一个由 YAML 驱动的 agent 评估平台，运行真实的 headless DSH 会话（每个用例一个隔离 agent + scratch 工作区，以官方 Minimal persona 作为基线系统提示词），从会话事件日志中收集工具调用轨迹，并评估结构化断言以及可选的第二模型审查——与审批 answerer 使用同一个审查接缝。
+
+```yaml
+# eval/cases/demo.yaml（节选）
+suite:
+  name: my-suite
+  cases:
+    - id: math-output
+      input: Solve 17 × 24 and reply with only the final number, nothing else.
+      expect:
+        output: { contains: "408" }
+    - id: glob-trace
+      seedFrom: '.'
+      input: Use the glob tool with pattern "src/**" to list the source files…
+      expect:
+        toolCalls: [{ tool: glob, arguments: { contains: { pattern: "src" } } }]
+        results: [{ tool: glob, contains: "index.ts" }]
+    - id: review-write
+      input: Read src/config.ts, write the default reviewerTimeoutMs into scratch/answer.txt…
+      expect:
+        output: { contains: "60000" }
+      review:
+        statement: The agent read the default reviewerTimeoutMs and wrote it to the file.
+```
+
+运行方式（环境中必须提供 DeepSeek API key）：
+
+```sh
+dsh-eval eval/cases --model deepseek-v4-flash --timeout-ms 240000 --out .eval-reports
+```
+
+CI 门槛：仅当每个 suite 的每个用例都通过时进程才以 0 退出——把它放进一个 GitHub Action 步骤，评估失败即构建失败。每个用例都会在 `report.md`/`report.json` 旁留下一个可重放的会话 JSONL 与一个 trace JSON；断言结果、token 用量与审查裁决全部写入报告文件。引擎从不替换硬编码的模型或超时默认值，在 SIGINT/SIGTERM 时干净中止，并把 worker 池限制在配置的并发数以内。
+
+与 [codex-research](https://github.com/openai/codex/tree/main/codex-rs/research)（浏览器自动化 agent 研究）不同，`dsh-eval` 面向 harness 层面的 agent 评估：针对会话事件日志的工具调用轨迹断言、作为补充断言层的第二模型审查，以及每个用例隔离的 headless 会话——不依赖浏览器或 Selenium 技术栈。
+
 ## 🔒 安全边界
 
 - 审查代理运行在**只读工具面**（`toolFilter` 白名单）内：不能写、改、执行 shell、访问网络、再委派（`maxDepth` = 自身深度）。其会话日志同样落盘可审计。
@@ -213,13 +252,43 @@ dsh --profile web --dump-config | grep -A4 'id: auto-review'
 ```sh
 pnpm install                # node ^22.19 || >=24
 pnpm run typecheck          # tsc，src + 测试
-pnpm test                   # vitest：135 个测试、8 个套件
+pnpm test                   # vitest：190 个测试、14 个文件
 pnpm run build              # tsc 声明 + tsdown 打包（lib/）
 pnpm run verify:self-contained
 pnpm pack                   # 发布产物
 ```
 
-仓库结构（plugin-template 结构）：`src/index.ts`（插件契约）· `src/config.ts`（Schemastery schema + 解析）· `src/runtime.ts`（answerer、命令、拒绝理由注入）· `src/review.ts`（审查代理编排、prompt、脱敏）· `src/events.ts`（会话事件词汇 + fold）· `src/projection.ts` + `src/projection-types.ts`（`autoReview` 会话投影）· `src/invariant.ts`（invariant 伴生）· `src/client/`（浏览器半：审查面板、locales、样式）· `test/` · `fixtures/`。
+仓库结构（plugin-template 结构）：`src/index.ts`（插件契约）· `src/config.ts`（Schemastery schema + 解析）· `src/runtime.ts`（answerer、命令、拒绝理由注入）· `src/review.ts`（审查代理编排、prompt、脱敏）· `src/events.ts`（会话事件词汇 + fold）· `src/projection.ts` + `src/projection-types.ts`（`autoReview` 会话投影）· `src/invariant.ts`（invariant 伴生）· `src/eval/`（dsh-eval 引擎：DSL、runner、assertions、trace、review、reports、CLI）· `eval/`（随附评估组合 + demo 套件）· `bin/dsh-eval.mjs`（CLI 启动器）· `src/client/`（浏览器半：审查面板、locales、样式）· `test/` · `fixtures/`。
+
+## 👥 贡献者
+
+感谢所有为 `dsh-auto-review` 做出贡献的人：
+
+- [PerryLink](https://github.com/PerryLink) — 作者与维护者：审批 answerer、审查子代理、风险策略与熔断器、会话投影审查面板、invariant 伴生、文档、CI/CD 与发布。
+
+想参与？请先看 [issue 模板](.github/ISSUE_TEMPLATE/)、[安全策略](SECURITY.md) 与 [AGENTS.md](AGENTS.md) 的仓库约定——欢迎中英文 PR。
+
+## PerryLink DSH 插件家族
+
+本项目是 [PerryLink](https://github.com/PerryLink) 维护的 [15 个 DeepSeek Harness 插件](https://github.com/PerryLink)之一。如果你觉得这个插件有用，其余的很可能同样有用：
+
+| 插件 | 一句话说明 |
+|---|---|
+| [dsh-mcp-panel](https://github.com/PerryLink/dsh-mcp-panel) | 只读 MCP 运行时面板：/mcp 命令 + 设置页，状态/工具/错误一览 |
+| [dsh-doublecheck](https://github.com/PerryLink/dsh-doublecheck) | 工程纪律守门：需求审讯、测试证据门、对抗评审 |
+| [dsh-background-agents](https://github.com/PerryLink/dsh-background-agents) | 持久化后台子代理：Web 侧边栏进度、随时留言与打断 |
+| [dsh-lsp-actions](https://github.com/PerryLink/dsh-lsp-actions) | 基于语言服务器的诊断/格式化/补全/代码动作/重命名 |
+| [dsh-output-styles](https://github.com/PerryLink/dsh-output-styles) | 对标 Claude Code outputStyles 的运行时风格切换 |
+| [dsh-checkpoint-rewind](https://github.com/PerryLink/dsh-checkpoint-rewind) | 对标 Claude Code /rewind：快照、会话 fork、一键回退 |
+| [dsh-permission-rules](https://github.com/PerryLink/dsh-permission-rules) | Claude Code 风格声明式 allow/deny/ask 权限规则，带审计 |
+| **[dsh-auto-review](https://github.com/PerryLink/dsh-auto-review)** | 审批链上的第二模型自动审查，默认 fail-closed |
+| [dsh-memento](https://github.com/PerryLink/dsh-memento) | 带审批门的跨会话记忆：ctx.memory + SQLite + memory 工具 |
+| [dsh-skill-pack-security](https://github.com/PerryLink/dsh-skill-pack-security) | 安全审计技能包：密钥扫描、依赖与供应链审查 |
+| [dsh-session-pin](https://github.com/PerryLink/dsh-session-pin) | 在 Web 侧边栏置顶会话，持久排序 |
+| [dsh-composer-history](https://github.com/PerryLink/dsh-composer-history) | Web 作曲器终端式输入历史：方向键、Ctrl+R 搜索 |
+| [dsh-github](https://github.com/PerryLink/dsh-github) | DSH 的 GitHub PR/issue 集成，所有写操作经审批门 |
+| [dsh-plugin-guide](https://github.com/PerryLink/dsh-plugin-guide) | 插件开发知识库，随 bundle 安装的按需 agent 技能 |
+| [dsh-claude-move](https://github.com/PerryLink/dsh-claude-move) | 把 Claude Code 会话、记忆、技能和 CLAUDE.md 迁入 DSH |
 
 ## 📄 License
 

package/bin/dsh-eval.mjs

@@ -0,0 +1,6 @@
+#!/usr/bin/env node
+// dsh-eval: the dsh-auto-review evaluation CLI. Thin launcher over the
+// bundled eval-cli face; the build produces lib/eval-cli.js.
+import { main } from '../lib/eval-cli.js'
+
+process.exitCode = await main()

package/eval/cases/demo.yaml

@@ -0,0 +1,78 @@
+# dsh-eval demo suite — three cases that exercise this repository itself.
+#
+# Run from the repo root:
+#   dsh-eval eval/cases --model deepseek-v4-flash --out .eval-reports
+#
+# The cases need no hardcoded model/timeout here: the CLI (or a suite
+# override) supplies them; a case that resolves neither fails loudly.
+
+suite:
+  name: dsh-auto-review-self
+  description: "The engine's own regression suite: output assertion, tool-trace assertion, and second-model review."
+  models:
+    tiers:
+      fast: deepseek-v4-flash
+      pro: deepseek-v4-pro
+  concurrency: 1
+  cases:
+    - id: math-output
+      description: Pure reasoning with a final-output assertion (no tools needed).
+      tier: fast
+      input: >-
+        Solve 17 × 24 and reply with only the final number, nothing else.
+      expect:
+        output:
+          contains: "408"
+
+    - id: glob-trace
+      description: "Tool-call trace: the agent must glob the seeded repository's src directory."
+      tier: fast
+      seedFrom: '../..'
+      input: >-
+        Use the glob tool with pattern "src/**" to list the source files of this
+        repository. Then tell me how many TypeScript files there are, and name the
+        file src/index.ts in your final answer.
+      expect:
+        toolCalls:
+          - tool: glob
+            arguments:
+              contains:
+                pattern: "src"
+        results:
+          - tool: glob
+            contains: "index.ts"
+        output:
+          contains: "index.ts"
+
+    - id: review-write
+      description: >-
+        Read src/config.ts for the default reviewerTimeoutMs, write the value into
+        scratch/answer.txt, and let a second model verify the whole run.
+      tier: fast
+      seedFrom: '../..'
+      input: >-
+        Read src/config.ts and find the default reviewerTimeoutMs (the default the
+        resolveConfig function applies). Write that numeric value in milliseconds
+        into the file scratch/answer.txt. Then report the value and confirm which
+        file you wrote it to.
+      expect:
+        toolCalls:
+          - tool: read
+            arguments:
+              contains:
+                file_path: "config.ts"
+          - tool: write
+            arguments:
+              contains:
+                file_path: "answer.txt"
+        results:
+          - tool: read
+            contains: "60_000"
+        output:
+          contains: "60000"
+      review:
+        statement: The agent read src/config.ts and wrote scratch/answer.txt containing 60000.
+        criteria:
+          - The read call targeted src/config.ts.
+          - The write call created scratch/answer.txt with content 60000.
+          - The final output mentions 60000.