dsh-auto-model-routing 0.4.0 → 0.4.1

package/CHANGELOG.md

@@ -1,5 +1,9 @@
 # Changelog
 
+## 0.4.1
+
+- Add the on-demand `auto-model-routing-report` skill and read-only `lib/usage-report.js`: joins policy-v2 decisions to actual per-call usage by message ID, grouped by version × model × depth, with coverage gaps, sample thresholds and fixed caveats. Observational only; never writes, spawns or uses the network, and never prints message text, paths or session IDs.
+
 ## 0.4.0
 
 - Upgrade AUTO by default: keep the main model fixed, independently judge child capability and reasoning investment, and enforce exact fresh-child bindings with persisted replay. Preserve explicit legacy rollback and MANUAL behavior.

package/README.md

@@ -2,8 +2,37 @@
 
 English · [简体中文](README.zh.md)
 
+![AUTO interaction: toggle AUTO/MANUAL and inspect the decision](docs/auto-routing.gif)
+
+*Interaction recording from the earlier four-tier UI (isolated mock Host); current decisions use independent capability and depth as described below.*
+
+[Repository](https://github.com/prgrmrwy/dsh-auto-model-routing) · [Releases](https://github.com/prgrmrwy/dsh-auto-model-routing/releases) · [npm](https://www.npmjs.com/package/dsh-auto-model-routing)
+
 <!-- problem -->
-Keep the main model you chose; let AUTO select reasoning investment for each turn and suitable resources for fresh sub-agents. Jev judges **capability and effort independently**. DSH applies the route, and the UI distinguishes a decision from a configuration that actually reached a request.
+**Add AUTO mode to DSH: automatic effort for the main agent, automatic model + effort for sub-agents.** Stop adjusting reasoning investment every turn or choosing a model for every delegated task.
+
+- **Main agent: keep your selected model and adjust reasoning effort per user turn.**
+- **Sub-agents: judge capability needs and reasoning investment independently, then select model + effort.** A stronger model does not automatically mean deeper reasoning.
+- **Visible decisions, local feedback:** inspect candidate distributions and actual application status, then rate the allocation below a reply.
+
+### Which DSH scenarios does AUTO cover?
+
+| Scenario | AUTO behavior | Current boundary |
+|---|---|---|
+| Main-agent user turn | Judge effort for each new user turn; keep the main model | Not rejudged at every tool step; model must support the effort |
+| Official fresh `subagent` | Select and bind child model + effort before its first request | Real Magpie Claude/GPT ordinary child calls verified; catalog and allowlist apply |
+| Official fresh `spawn_teammate` | Bind the new teammate's model + effort using the same policy | Mock-host coverage; real-provider teammate acceptance remains unverified |
+| Resume or continue a bound child | Replay its initial allocation without rejudging | Requires the plugin enabled and intact route records |
+| Explicit model / effort | Preserve the specified dimension; judge only the other | Both specified: validate without calling Jev |
+| Fork, third-party or unbound creation paths | No automatic-allocation guarantee | Fork and unbound AUTO child paths are explicitly rejected, not silently replaced |
+
+This describes official integration paths—not acceptance of every DSH entry point, CLI/TUI interface or third-party plugin. The Web UI is the currently verified interface. MANUAL makes no new judgments. `route_subagent` is a preview, not a required step before dispatch.
+
+### Local feedback → future learning and adjustment
+
+**Available now:** rate a decision-bound reply as appropriate, too high, too low or unknown; change or clear the rating. Feedback and decision snapshots stay on your machine. Submitting feedback does not upload it to the decision service or immediately change routing; AUTO task judgments themselves still use your selected external service.
+
+**Planned direction:** add learning, evaluation and policy adjustment based on local feedback and execution outcomes, so AUTO can better fit your tasks. **Automatic learning and feedback-driven tuning are not implemented yet.** Today's records support later evaluation; they are not evidence of measured quality gains or cost savings.
 
 **Install.** Tested with DSH `0.2.0-rc.2`, Web UI, Node.js `>=22.19`.
 
@@ -13,7 +42,7 @@ dsh plugin --profile web add dsh-auto-model-routing
 dsh plugin --profile web add prgrmrwy/dsh-auto-model-routing
 ```
 
-Restart DSH and refresh the Web page. In **Settings → 自动模型路由**, choose a decision service, save its API key, and use **测试连接** with a fixed public sample. New chats default to **MANUAL**; click the chip to use AUTO. The companion delegation skill is included—no separate skill installation is needed.
+Restart DSH and refresh the Web page. In **Settings → Auto model routing**, choose a decision service, save its API key, and use **Test connection** with a fixed public sample. These are English descriptions of the controls; the plugin's interface labels are currently in Chinese. New chats default to **MANUAL**; click the chip to use AUTO. The companion delegation skill is included—no separate skill installation is needed.
 
 ## Use
 
@@ -22,7 +51,7 @@ Restart DSH and refresh the Web page. In **Settings → 自动模型路由**, ch
 | **AUTO / MANUAL chip**, left of the model picker | Click to switch. Hover for the decision, candidate distribution, certainty, model/effort and application status. |
 | **Decision hint** | Shows new decisions briefly; its Settings switch does not hide the hover panel. |
 | **Gauge button** below an eligible reply | Mark the allocation appropriate, too high, too low or unknown; change or clear your feedback later. Only replies bound to an applied main decision qualify. |
-| **Settings → 自动模型路由** | Choose service/model, manage its separate key, test the connection, set defaults/cap/deadline, or expand **查看决策策略** to inspect the actual rubric and model slots. |
+| **Settings → Auto model routing** | Choose service/model, manage its separate key, test the connection, set defaults/cap/deadline, or expand **View decision strategy** to inspect the actual rubric and model slots. |
 
 ### Main model stays fixed; children are selected separately
 
@@ -58,12 +87,9 @@ Preset model menus expose documented versions, not live availability guarantees.
 ## Architecture
 <!-- section: architecture -->
 
-```text
-user turn → pre-step → selected Jev service → depth → main request effort
-fresh child creation → exact child binding → capability + depth → validated route
-request/header → configuration match → applied evidence
-assistant message → immutable decision snapshot → human feedback events
-```
+![AUTO architecture — independent judgments, validated requests and allocation feedback](docs/architecture-v2.en.svg)
+
+Main failures disable AUTO; child failures stop dispatch. Explicit model/effort constraints are preserved. The diagram shows the AUTO paths; MANUAL makes no new judgments and fork is unsupported. Feedback does not automatically alter routing.
 
 Entry: [index.ts](src/index.ts). Enforcement: [policy-host.ts](src/policy-host.ts). Actual rubric: [policy-v2-rubric.json](src/policy-v2-rubric.json). Model slots: [policy-v2.ts](src/policy-v2.ts). UI: [client](src/client/index.ts). Coverage and limitations: [development verification](docs/policy-v2-development.md), [real Magpie acceptance](docs/magpie-live-acceptance.md).
 
@@ -72,7 +98,6 @@ Entry: [index.ts](src/index.ts). Enforcement: [policy-host.ts](src/policy-host.t
 
 These isolated-host recordings use mock Jev/model responses and show the previous UI, not the current default policy.
 
-![Legacy AUTO effort adjustment demo](docs/auto-routing.gif)
 ![Legacy four-tier routing settings](docs/settings.gif)
 
 </details>
@@ -85,7 +110,9 @@ npm run typecheck
 npm run build
 ```
 
-Tests use the official AgentLoop with fake LLM streams and mock/loopback Jev responses, not paid model calls. `scripts/label-report.mjs` reports **legacy tier labels only**; it is not a report for new allocation feedback. Evaluation schema and deduplication rules are in [feedback-evaluation.md](docs/feedback-evaluation.md).
+Tests use the official AgentLoop with fake LLM streams and mock/loopback Jev responses, not paid model calls. `scripts/label-report.mjs` reports **legacy tier labels only**. To observe AUTO decisions and their actual token usage, ask the agent to use the `auto-model-routing-report` skill (on demand only; it runs the read-only `lib/usage-report.js`). It is observational, not a savings measurement. Evaluation schema and deduplication rules are in [feedback-evaluation.md](docs/feedback-evaluation.md).
+
+Development stays in the source monorepo; this repository is its release mirror. The package builds and loads independently using its declared dependencies, bundled skill and official DSH services; no monorepo checkout or monorepo instruction file is required. External design references are optional background, not runtime inputs.
 
 The [current specification](https://github.com/prgrmrwy/ohmydsh/blob/main/openspec/specs/auto-model-routing/spec.md) lives in the source monorepo. The bundled [delegation skill](skills/subagent-model-routing/SKILL.md) supplies guidance; enforcement belongs to the official host hooks, not the prompt. No measured task-quality or token-savings claim is made.
 

package/README.zh.md

@@ -2,8 +2,37 @@
 
 [English](README.md) · 简体中文
 
+![AUTO 交互：切换 AUTO/MANUAL 并查看决策](docs/auto-routing.gif)
+
+*此交互录屏来自早期四档界面（隔离 mock 宿主）；当前策略按下文独立判断能力与投入。*
+
+[插件仓库](https://github.com/prgrmrwy/dsh-auto-model-routing) · [版本发布](https://github.com/prgrmrwy/dsh-auto-model-routing/releases) · [npm](https://www.npmjs.com/package/dsh-auto-model-routing)
+
 <!-- problem -->
-主模型由你选，AUTO 为每轮选择推理投入，并为 fresh 子任务选配资源。Jev **独立判断能力需求和投入**，DSH 验证并应用配置；界面区分“做出了判断”和“实际请求已生效”。
+**为 DSH 增加 AUTO 模式：主 Agent 自动调 effort，子 Agent 自动选 model + effort。** 不必每轮手动选择推理投入，也不必为每个子任务手动挑模型。
+
+- **主 Agent：保留你选择的模型，按回合调整推理投入（effort）。**
+- **子 Agent：按任务独立判断能力需求与推理投入，选择模型与 effort。** 不是把“更强模型”和“更深推理”捆成一个档位。
+- **看得见、能反馈：** 展示决策分布与实际生效状态，在回复下方提交本机分配反馈。
+
+### AUTO 覆盖哪些 DSH 场景？
+
+| 场景 | AUTO 的作用 | 当前边界 |
+|---|---|---|
+| 主 Agent 用户回合 | 每个新用户回合判断 effort，主模型不换 | 不逐工具步骤重判；模型必须支持相应 effort |
+| 官方 fresh `subagent` | 首次请求前选择并绑定子 model + effort | 真实 Magpie Claude/GPT 普通子调用已验证；受目录和允许列表约束 |
+| 官方 fresh `spawn_teammate` | 按同一策略绑定新 teammate 的 model + effort | 已有 mock 宿主覆盖，尚未完成真实模型 teammate 验收 |
+| 已绑定子实例恢复或继续执行 | 重用初始路线，不重新分配 | 依赖插件保持启用及路由记录完整 |
+| 显式指定 model / effort | 保留已指定维度，仅判断未指定维度 | 两者都指定则只校验，不调用 Jev |
+| fork、第三方或未绑定创建路径 | 不承诺自动选配 | fork 与未绑定 AUTO 子路径会明确拒绝，不静默替代 |
+
+这里是官方接入路径的覆盖说明，不代表所有 DSH 入口、CLI/TUI 界面或第三方插件都已验收。Web UI 是当前验证入口；MANUAL 不做新判断。`route_subagent` 仅预览，不是必须先调用的派发步骤。
+
+### 本机反馈 → 未来的学习调整
+
+**现在：** 在已绑定决策的回复下评价“合适 / 投入偏高 / 投入不足 / 无法判断”，可修改、撤销。反馈与决策快照保存在本机，不会因点击反馈而上传到决策服务，也不会立即改变路由；AUTO 任务判断本身仍按所选服务外发。
+
+**未来方向：** 增加基于本机反馈与执行结果的学习、评估和策略调整，让 AUTO 更贴合你的任务。**尚未实现自动学习或反馈驱动调参**，目前记录用于后续评估，不承诺已证明的质量提升或成本节省。
 
 **安装**：已验证 DSH `0.2.0-rc.2`、Web UI，Node.js `>=22.19`。
 
@@ -58,12 +87,9 @@ dsh plugin --profile web add prgrmrwy/dsh-auto-model-routing
 ## 架构
 <!-- section: architecture -->
 
-```text
-用户回合 → pre-step → 所选 Jev 服务 → depth → 主请求 effort
-fresh 子创建 → 精确身份绑定 → capability + depth → 校验路由
-request/header → 配置匹配 → applied 证据
-assistant 消息 → 不可变决策快照 → 人工反馈事件
-```
+![AUTO 架构：独立判断、请求校验与分配反馈](docs/architecture-v2.zh.svg)
+
+主判断失败停用 AUTO，子失败停止派发；保留显式 model/effort。图示为 AUTO 路径，MANUAL 不做新判断，fork 暂不支持。反馈不会自动改变路由。
 
 入口：[index.ts](src/index.ts)。执行接入：[policy-host.ts](src/policy-host.ts)。实际判据：[policy-v2-rubric.json](src/policy-v2-rubric.json)。模型槽：[policy-v2.ts](src/policy-v2.ts)。界面：[client](src/client/index.ts)。覆盖边界：[开发验证](docs/policy-v2-development.md)、[真实 Magpie 验收](docs/magpie-live-acceptance.md)。
 
@@ -72,7 +98,6 @@ assistant 消息 → 不可变决策快照 → 人工反馈事件
 
 下列隔离宿主录屏使用模拟 Jev/模型响应，展示旧界面，不代表当前默认策略。
 
-![旧版 AUTO 推理强度调整](docs/auto-routing.gif)
 ![旧版四档路由设置](docs/settings.gif)
 
 </details>
@@ -85,7 +110,9 @@ npm run typecheck
 npm run build
 ```
 
-测试使用官方 AgentLoop、假 LLM 流和 mock/回环 Jev，不产生真实模型费用。`scripts/label-report.mjs` **仅汇总旧四档标注**，不是新分配反馈报表；新数据结构与去重口径见[feedback-evaluation.md](docs/feedback-evaluation.md)。
+测试使用官方 AgentLoop、假 LLM 流和 mock/回环 Jev，不产生真实模型费用。`scripts/label-report.mjs` **仅汇总旧四档标注**。如需观察 AUTO 决策及实际 token 用量，请明确让 agent 使用 `auto-model-routing-report` skill（仅按需调用，运行只读的 `lib/usage-report.js`）；它是观察统计，不衡量节省；新数据结构与去重口径见[feedback-evaluation.md](docs/feedback-evaluation.md)。
+
+继续在主仓开发，本仓库仅作发布镜像。插件使用自身声明的依赖、包内 skill 和官方 DSH 服务即可独立构建与加载，不要求检出主仓，也不要求安装主仓环境指令。外部设计链接仅供背景阅读，不是运行输入。
 
 [当前规范](https://github.com/prgrmrwy/ohmydsh/blob/main/openspec/specs/auto-model-routing/spec.md)在源码 monorepo 中维护。包内[委派 skill](skills/subagent-model-routing/SKILL.md)提供指引，执行保障来自官方宿主钩子，而不是提示词。暂无完整任务质量或 token 节省的量化结论。
 

package/docs/architecture-v2.en.json

@@ -0,0 +1,26 @@
+{
+ "schema_version":1,"diagram_type":"architecture",
+ "meta":{"title":"AUTO resource allocation","output":".archify/architecture-auto-routing-20261026-014646/auto-routing.html","locale":"en","quality_profile":"showcase","repository":{"url":"git@github.com:prgrmrwy/ohmydsh.git","revision":"79f9eab467047e054a2d66f8bce060834a261a2a","link_mode":"local-only"}},
+ "components":[
+ {"id":"turn","type":"frontend","label":"AUTO user turn","sublabel":"Main model stays fixed","pos":[40,80],"size":[190,68],"sources":[{"path":"packages/auto-model-routing/src/policy-host.ts","line":68,"end_line":88}]},
+ {"id":"judge","type":"cloud","label":"Selected Jev service","sublabel":"Depth / capability independently","pos":[360,80],"size":[220,68],"sources":[{"path":"packages/auto-model-routing/src/decision-service.ts","line":1,"end_line":23},{"path":"packages/auto-model-routing/src/policy-host.ts","line":81,"end_line":88}]},
+ {"id":"route","type":"backend","label":"Validated allocation","sublabel":"Catalog / allowlist / effort cap","pos":[710,80],"size":[220,68],"sources":[{"path":"packages/auto-model-routing/src/policy-host.ts","line":122,"end_line":150}]},
+ {"id":"child","type":"backend","label":"Fresh child binding","sublabel":"Exact child identity; explicit wins","pos":[360,270],"size":[220,68],"sources":[{"path":"packages/auto-model-routing/src/policy-host.ts","line":94,"end_line":132}]},
+ {"id":"request","type":"backend","label":"Official model request","sublabel":"Main effort / child full route","pos":[710,270],"size":[220,68],"sources":[{"path":"packages/auto-model-routing/src/policy-host.ts","line":138,"end_line":160}]},
+ {"id":"persist","type":"database","label":"Persisted child route","sublabel":"Resume keeps initial allocation","pos":[40,270],"size":[220,68],"sources":[{"path":"packages/auto-model-routing/src/policy-host.ts","line":132,"end_line":160}]},
+ {"id":"evidence","type":"backend","label":"Request match evidence","sublabel":"Applied only when config matches","pos":[710,460],"size":[220,68],"sources":[{"path":"packages/auto-model-routing/src/policy-host.ts","line":178,"end_line":201}]},
+ {"id":"feedback","type":"frontend","label":"Decision and reply feedback","sublabel":"Observe allocation; never auto-train","pos":[360,460],"size":[220,68],"sources":[{"path":"packages/auto-model-routing/src/policy-host.ts","line":162,"end_line":174}]},
+ {"id":"snapshot","type":"database","label":"Bound decision snapshot","sublabel":"Metadata / IDs; no task text","pos":[40,460],"size":[220,68],"sources":[{"path":"packages/auto-model-routing/src/policy-host.ts","line":167,"end_line":172}]}
+ ],
+ "connections":[
+ {"id":"main-judge","from":"turn","to":"judge","label":"depth"},
+ {"id":"judge-route","from":"judge","to":"route","label":"decision"},
+ {"id":"child-judge","from":"child","to":"judge","label":"task","fromSide":"top","toSide":"bottom"},
+ {"id":"child-store","from":"child","to":"persist","label":"bind"},
+ {"id":"route-request","from":"route","to":"request","label":"apply"},
+ {"id":"request-match","from":"request","to":"evidence","label":"header"},
+ {"id":"match-feedback","from":"evidence","to":"feedback","label":"matched"},
+ {"id":"feedback-snapshot","from":"feedback","to":"snapshot","label":"bind reply"}
+ ],
+ "cards":[{"dot":"cyan","title":"Decision services","items":["OpenRouter / TypeSafe / Concentrate / custom System One","No automatic service fallback or retry"]},{"dot":"amber","title":"Boundaries","items":["MANUAL: no new judgments; explicit constraints preserved","Fork unsupported; applied is not task success"]}]
+}