dsh-auto-model-routing 0.0.0-stage → 0.4.0

package/CHANGELOG.md

@@ -0,0 +1,19 @@
+# Changelog
+
+## 0.4.0
+
+- Upgrade AUTO by default: keep the main model fixed, independently judge child capability and reasoning investment, and enforce exact fresh-child bindings with persisted replay. Preserve explicit legacy rollback and MANUAL behavior.
+- Add TypeSafe, Concentrate and custom System One decision services with separate credentials, preset model menus and explicit public-sample connection testing.
+- Expose executable strategy details, candidate distributions, normalized Choice certainty and actual application status; no task-quality or savings claims.
+- Restore per-reply allocation feedback with immutable evaluation snapshots, request-event correlation, timestamped change/cancel events and legacy-record compatibility.
+- Refresh bilingual user documentation; label earlier GIFs as legacy mock-host demonstrations. Fork and real-provider teammate acceptance remain unsupported/unverified as documented.
+
+
+- Accept the DSH 0.2 host's React 18.3.1 as well as React 19 in the runtime peer declaration; client builds continue to use the host's shared React instance.
+
+## 0.3.0
+
+- Publish as a standalone DSH bundle with the companion `subagent-model-routing` skill included.
+- Replace the separate ohmydsh skill customization with a package-owned filesystem skill provider. Skill and tools now enable/disable together.
+- Add npm/Git installation instructions, demo GIFs, licensing and OIDC release automation.
+- Keep the existing tier rubric, effort mappings, default MANUAL mode, credentials and stored state unchanged.

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 prgrmrwy
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,99 @@
-# Temporary Holding Version
+# dsh-auto-model-routing
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+English · [简体中文](README.zh.md)
+
+<!-- problem -->
+Keep the main model you chose; let AUTO select reasoning investment for each turn and suitable resources for fresh sub-agents. Jev judges **capability and effort independently**. DSH applies the route, and the UI distinguishes a decision from a configuration that actually reached a request.
+
+**Install.** Tested with DSH `0.2.0-rc.2`, Web UI, Node.js `>=22.19`.
+
+```sh
+dsh plugin --profile web add dsh-auto-model-routing
+# Or install source (approve the local build when prompted):
+dsh plugin --profile web add prgrmrwy/dsh-auto-model-routing
+```
+
+Restart DSH and refresh the Web page. In **Settings → 自动模型路由**, choose a decision service, save its API key, and use **测试连接** with a fixed public sample. New chats default to **MANUAL**; click the chip to use AUTO. The companion delegation skill is included—no separate skill installation is needed.
+
+## Use
+
+| Where | What you do |
+|---|---|
+| **AUTO / MANUAL chip**, left of the model picker | Click to switch. Hover for the decision, candidate distribution, certainty, model/effort and application status. |
+| **Decision hint** | Shows new decisions briefly; its Settings switch does not hide the hover panel. |
+| **Gauge button** below an eligible reply | Mark the allocation appropriate, too high, too low or unknown; change or clear your feedback later. Only replies bound to an applied main decision qualify. |
+| **Settings → 自动模型路由** | Choose service/model, manage its separate key, test the connection, set defaults/cap/deadline, or expand **查看决策策略** to inspect the actual rubric and model slots. |
+
+### Main model stays fixed; children are selected separately
+
+- **Main chat:** `lower / baseline / higher` moves at most one supported effort step below/at/above the model's verified recommended starting point. The cap defaults to `xhigh`; AUTO never selects `max`. It does not change the main provider/model or write global model defaults.
+- **Fresh children:** capability is `bounded / analytical / frontier`; effort is judged separately. Claude slots are Haiku / Sonnet / Opus 5.5; GPT slots are Luna / Sol / Sol. The current catalog, supported efforts and official child allowlist must all permit the route. Magpie uses its exact provider and full family-prefixed model IDs, not a fallback to local transports.
+- **Explicit choices win:** specifying a model or effort leaves only the other dimension to judge. Specifying both skips Jev and validates the route. Missing task scope stops delegation rather than guessing.
+- **Official creation binds the route:** fresh `subagent` and fresh `spawn_teammate` paths bind an exact child identity before its first request. `route_subagent` is an optional preview—not proof of application and not a user constraint to copy into a subsequent call. Fork and unbound creation paths are unsupported; there is no silent substitution.
+- **Resume keeps the initial route.** Switching the root to MANUAL stops new judgments; already bound children retain their initial route. `applied` means the actual request configuration matched, **not** that the task succeeded. Real Magpie Claude/GPT ordinary calls were tested; fresh teammate routing and cold resume have mock-host coverage, not real-provider teammate acceptance.
+
+### Choose the decision service
+
+| Service | Endpoint | Default model | Credential reference |
+|---|---|---|---|
+| OpenRouter (default, preserves existing installations) | `https://openrouter.ai/api/alpha/decisions` | `typesafe/jev-1.13` | `OPENROUTER_API_KEY` |
+| TypeSafe | `https://api.typesafe.ai/v1/systemone` | `jev-1.13.0` | `TYPESAFE_API_KEY` |
+| Concentrate | `https://api.concentrate.ai/v1/systemone` | `jev-1.13.0` | `CONCENTRATE_API_KEY` |
+| Custom System One | Your full endpoint | Editable | `AUTO_DECISION_CUSTOM_API_KEY` |
+
+Preset model menus expose documented versions, not live availability guarantees. Custom services keep an editable model ID and must support `state + questions → answers`; an OpenAI chat-completions endpoint alone is not compatible. Custom endpoints require HTTPS (loopback HTTP is allowed), reject URL credentials/query/fragment and redirects. Only configure endpoints you trust with the dedicated key. TypeSafe and Concentrate have contract/mock coverage; their live authentication and regional availability were not verified here. No service failover occurs automatically.
+
+**Certainty is not the selected probability.** Choice certainty is `(pmax − 1/n) / (1 − 1/n)`: with three candidates, a 70% top probability gives 55% certainty. The panel shows the service's returned certainty and full candidate distribution separately, including returned zeroes; missing values are marked unavailable. Neither number is task success rate.
+
+## Good to know
+
+- **Failures are visible.** Main judgment failure disables AUTO with a reason; child judgment/route failure stops dispatch. No local classifier, automatic retry, upgrade or cross-service fallback.
+- **MANUAL stops adjustments, not history.** It does not restore an earlier effort. The official current selection remains in control; existing bound children keep their route while the plugin is enabled.
+- **Data leaves the machine.** AUTO sends shape-redacted task text (up to 8000 characters) and short previous-turn excerpts to your chosen service. Redaction is best effort, not a guarantee that all secrets are removed. Test Connection sends only a fixed public sample and can incur a small charge.
+- **Keys use official DSH credentials**, never plugin configuration. Launch-environment keys take precedence and are read-only. Service changes clear key drafts; save the service before managing its separate key. Changing a custom endpoint does not create a new credential namespace—review the key before trusting a new host.
+- **Local state:** configuration is under `$DSH_HOME/plugins/dsh-auto-model-routing/`; mode, child routes and feedback are under `$DSH_HOME/state/auto-model-routing/`. Feedback stores resource metadata, IDs, versions, returned distributions and timestamps—not task/reply text. Request event IDs allow later usage correlation; costs are not fabricated when unavailable. See [feedback and evaluation](docs/feedback-evaluation.md).
+- **No automatic learning.** Human allocation feedback is observational. It does not change routing or prove quality/cost benefits. Old messages without decision bindings are not backfilled; old records with missing evaluation fields remain unknown.
+- **Upgrade:** existing AUTO sessions use the new policy without a version switch or rewriting their stored mode. New chats still default to MANUAL unless configured otherwise. Historical four-tier mappings remain only for explicit `policy: "legacy"` rollback in plugin configuration; they are not the current policy or feedback menu.
+
+## Architecture
+<!-- section: architecture -->
+
+```text
+user turn → pre-step → selected Jev service → depth → main request effort
+fresh child creation → exact child binding → capability + depth → validated route
+request/header → configuration match → applied evidence
+assistant message → immutable decision snapshot → human feedback events
+```
+
+Entry: [index.ts](src/index.ts). Enforcement: [policy-host.ts](src/policy-host.ts). Actual rubric: [policy-v2-rubric.json](src/policy-v2-rubric.json). Model slots: [policy-v2.ts](src/policy-v2.ts). UI: [client](src/client/index.ts). Coverage and limitations: [development verification](docs/policy-v2-development.md), [real Magpie acceptance](docs/magpie-live-acceptance.md).
+
+<details>
+<summary>Earlier four-tier UI demos (legacy rollback only)</summary>
+
+These isolated-host recordings use mock Jev/model responses and show the previous UI, not the current default policy.
+
+![Legacy AUTO effort adjustment demo](docs/auto-routing.gif)
+![Legacy four-tier routing settings](docs/settings.gif)
+
+</details>
+
+## Development
+
+```sh
+npm test
+npm run typecheck
+npm run build
+```
+
+Tests use the official AgentLoop with fake LLM streams and mock/loopback Jev responses, not paid model calls. `scripts/label-report.mjs` reports **legacy tier labels only**; it is not a report for new allocation feedback. Evaluation schema and deduplication rules are in [feedback-evaluation.md](docs/feedback-evaluation.md).
+
+The [current specification](https://github.com/prgrmrwy/ohmydsh/blob/main/openspec/specs/auto-model-routing/spec.md) lives in the source monorepo. The bundled [delegation skill](skills/subagent-model-routing/SKILL.md) supplies guidance; enforcement belongs to the official host hooks, not the prompt. No measured task-quality or token-savings claim is made.
+
+## Removal
+<!-- section: removal -->
+
+```sh
+dsh plugin --profile web remove dsh-auto-model-routing
+```
+
+Restart DSH and refresh the page. The plugin's skill/tools and request hooks disappear; settings and logs remain. Without the plugin, child replay is no longer enforced and host restoration uses the original creation configuration. In [ohmydsh](https://github.com/prgrmrwy/ohmydsh), disable the local entry and rebuild instead.

package/README.zh.md

@@ -0,0 +1,99 @@
+# dsh-auto-model-routing
+
+[English](README.md) · 简体中文
+
+<!-- problem -->
+主模型由你选，AUTO 为每轮选择推理投入，并为 fresh 子任务选配资源。Jev **独立判断能力需求和投入**，DSH 验证并应用配置；界面区分“做出了判断”和“实际请求已生效”。
+
+**安装**：已验证 DSH `0.2.0-rc.2`、Web UI，Node.js `>=22.19`。
+
+```sh
+dsh plugin --profile web add dsh-auto-model-routing
+# 或从源码安装（出现提示时批准本地构建）：
+dsh plugin --profile web add prgrmrwy/dsh-auto-model-routing
+```
+
+重启 DSH 并刷新页面，到**设置 → 自动模型路由**选择决策服务、保存对应 API key，点击**测试连接**发送固定公开样例。新会话缺省 **MANUAL**，点击芯片切换 AUTO。配套委派 skill 已包含，无需单独安装。
+
+## 使用
+
+| 位置 | 操作 |
+|---|---|
+| **AUTO / MANUAL 芯片**（模型选择器左侧） | 点击切换；悬停查看判断、候选分布、确定度、模型与投入、生效状态 |
+| **判档提示** | 短暂显示新决策；关闭设置中的提示开关不会隐藏悬浮面板 |
+| 回复下方的**仪表按钮** | 评价分配合适、投入偏高、投入不足或无法判断，支持修改和撤销；仅对绑定已应用主决策的回复显示 |
+| **设置 → 自动模型路由** | 选择服务与模型、管理独立 key、测试连接、设置模式/强度上限/等待时间；展开**查看决策策略**查看实际判据与模型槽 |
+
+### 主模型不换，子任务独立选配
+
+- **主会话**：`lower / baseline / higher` 相对于具体模型已验证的推荐起点，在受支持序列降低一档、保持或提高一档。缺省上限 `xhigh`，自动不选 `max`；不修改主 provider/model，也不改写全局模型默认值。
+- **fresh 子任务**：能力需求为 `bounded / analytical / frontier`，推理投入独立判断。Claude 能力槽为 Haiku / Sonnet / Opus 5.5；GPT 为 Luna / Sol / Sol。实际路由必须同时满足当前目录、支持的 effort 和官方子智能体允许列表。Magpie 保留真实 provider 与家族前缀完整模型 ID，不回退到本地入口。
+- **尊重显式选择**：只指定模型或投入时，仅判断另一维；两者都指定则不请求 Jev，只校验路由。缺少任务目标或范围时停止委派，不猜。
+- **官方创建绑定路由**：fresh `subagent` 和 fresh `spawn_teammate` 在子任务首次请求前绑定精确身份。`route_subagent` 只是可选预览，不代表已应用，也不能把推荐复制成用户约束。fork 和未绑定创建路径暂不支持，不静默替换。
+- **恢复保持初始路线**：根会话改为 MANUAL 后停止新判断，已有绑定子实例维持原路线。`applied` 只证明实际请求配置匹配，**不代表任务成功**。真实 Magpie Claude/GPT ordinary 调用已验证；fresh teammate 路由及冷恢复有 mock 宿主覆盖，不等于真实模型 teammate 已验收。
+
+### 选择决策服务
+
+| 服务 | 接口 | 缺省模型 | 凭据引用 |
+|---|---|---|---|
+| OpenRouter（默认，兼容已有安装） | `https://openrouter.ai/api/alpha/decisions` | `typesafe/jev-1.13` | `OPENROUTER_API_KEY` |
+| TypeSafe 官方 | `https://api.typesafe.ai/v1/systemone` | `jev-1.13.0` | `TYPESAFE_API_KEY` |
+| Concentrate | `https://api.concentrate.ai/v1/systemone` | `jev-1.13.0` | `CONCENTRATE_API_KEY` |
+| 自定义 System One | 完整接口地址 | 可编辑 | `AUTO_DECISION_CUSTOM_API_KEY` |
+
+预置模型菜单列出文档确认的版本，不保证实时可用。自定义服务可填写自己的模型 ID，但必须兼容 `state + questions → answers`；只有 OpenAI chat-completions 接口并不兼容。自定义端点限 HTTPS（回环可 HTTP），禁止 URL 凭据、查询参数、片段及重定向。只配置你信任且适合使用该专用 key 的地址。TypeSafe、Concentrate 目前有协议/mock 覆盖，未在本次验证其真实凭据与地区可用性。失败不自动切换服务。
+
+**判定确定度不等于选中概率**：Choice 公式为 `(pmax − 1/n) / (1 − 1/n)`，三个候选中最高概率 70% 对应确定度 55%。面板分别显示服务返回的确定度和完整分布，保留返回的 0%，缺值标未提供。两者都不是任务成功率。
+
+## 须知
+
+- **失败可见**：主判断失败会停用 AUTO 并显示原因，子判断或路由失败则停止派发。没有本地分类器、自动重试、自动升级或跨服务降级。
+- **MANUAL 停止调整，不复原历史值**：使用官方当前选择；只要插件启用，已绑定子任务仍保留原路线。
+- **任务会外发**：AUTO 将按形态脱敏的任务文本（最多 8000 字符）及上一轮少量片段发送给所选服务。脱敏尽力而为，不保证去除全部秘密。测试连接只发送固定公开样例，可能产生少量费用。
+- **key 经官方 DSH 凭据服务保存**，不写入插件配置。启动环境 key 优先且只读；切换服务清空 key 草稿，先保存服务再管理新 key。修改自定义端点不会产生新凭据空间，换地址前请确认专用 key 的信任边界。
+- **本地状态**：配置在 `$DSH_HOME/plugins/dsh-auto-model-routing/`，模式、子路由及反馈在 `$DSH_HOME/state/auto-model-routing/`。反馈只保存资源元数据、身份、版本、返回分布及时间，不保存任务或回复正文。请求事件序号供后续关联用量，缺失成本不会编造。见[反馈评估说明](docs/feedback-evaluation.md)。
+- **不自动学习**：人工反馈仅作观察，不改变路由，也不能直接证明质量提升或节省。无决策绑定的旧消息不回填，旧记录缺少的评估字段保持未知。
+- **升级兼容**：已有 AUTO 会话直接使用新策略，无需版本开关，也不重写会话模式。新会话缺省仍为 MANUAL，除非你改过设置。旧四档映射仅供配置中显式 `policy: "legacy"` 回滚，不是当前策略或反馈菜单。
+
+## 架构
+<!-- section: architecture -->
+
+```text
+用户回合 → pre-step → 所选 Jev 服务 → depth → 主请求 effort
+fresh 子创建 → 精确身份绑定 → capability + depth → 校验路由
+request/header → 配置匹配 → applied 证据
+assistant 消息 → 不可变决策快照 → 人工反馈事件
+```
+
+入口：[index.ts](src/index.ts)。执行接入：[policy-host.ts](src/policy-host.ts)。实际判据：[policy-v2-rubric.json](src/policy-v2-rubric.json)。模型槽：[policy-v2.ts](src/policy-v2.ts)。界面：[client](src/client/index.ts)。覆盖边界：[开发验证](docs/policy-v2-development.md)、[真实 Magpie 验收](docs/magpie-live-acceptance.md)。
+
+<details>
+<summary>旧四档界面演示（仅供 legacy 回滚参考）</summary>
+
+下列隔离宿主录屏使用模拟 Jev/模型响应，展示旧界面，不代表当前默认策略。
+
+![旧版 AUTO 推理强度调整](docs/auto-routing.gif)
+![旧版四档路由设置](docs/settings.gif)
+
+</details>
+
+## 开发验证
+
+```sh
+npm test
+npm run typecheck
+npm run build
+```
+
+测试使用官方 AgentLoop、假 LLM 流和 mock/回环 Jev，不产生真实模型费用。`scripts/label-report.mjs` **仅汇总旧四档标注**，不是新分配反馈报表；新数据结构与去重口径见[feedback-evaluation.md](docs/feedback-evaluation.md)。
+
+[当前规范](https://github.com/prgrmrwy/ohmydsh/blob/main/openspec/specs/auto-model-routing/spec.md)在源码 monorepo 中维护。包内[委派 skill](skills/subagent-model-routing/SKILL.md)提供指引，执行保障来自官方宿主钩子，而不是提示词。暂无完整任务质量或 token 节省的量化结论。
+
+## 移除
+<!-- section: removal -->
+
+```sh
+dsh plugin --profile web remove dsh-auto-model-routing
+```
+
+重启 DSH 并刷新页面。插件的 skill、工具和请求钩子撤销，配置与日志保留。关闭插件后不再执行子路由恢复，宿主使用原始创建配置。在 [ohmydsh](https://github.com/prgrmrwy/ohmydsh) 中则禁用 local 条目并重新构建。

package/cordis.patch.yml

@@ -0,0 +1,9 @@
+- insert:
+    - id: dsh-auto-model-routing
+      name: dsh-auto-model-routing
+    - id: auto-model-routing-skills
+      name: '@deepseek-ai/dsh-skill-filesystem'
+      config:
+        providerName: auto-model-routing-plugin
+        includeDefaultRoots: false
+        bundledSkillDir: !!js process.getBuiltinModule('node:path').join(process.getBuiltinModule('node:path').dirname(process.getBuiltinModule('node:module').createRequire(baseUrl).resolve('dsh-auto-model-routing/package.json')), 'skills')

package/docs/SCREENSHOTS.md

@@ -0,0 +1,6 @@
+# Screenshots
+
+| file | kind | source | date | reviewer | verdict | note |
+| --- | --- | --- | --- | --- | --- | --- |
+| auto-routing.gif | screenshot | isolated DSH_HOME instance (temp HOME/XDG/DSH_HOME under /tmp, port 3199), synthetic data: local mock model and mock Jev on loopback, demo prompts; recorded with agent-browser, converted with ffmpeg | 2026-10-23 | lead | pass (viewed frames: demo prompts and mock replies only; no hostname, user, path, token or real session) | AUTO chip, decision hint, hover panel, switch to MANUAL |
+| settings.gif | screenshot | same isolated instance and recording run as auto-routing.gif | 2026-10-23 | lead | pass (viewed frames: settings section only; key field empty, source shown as launch environment of the isolated process) | Settings → 自动模型路由, Claude/GPT routing maps |