dsh-tool-vision 0.8.0 → 0.9.0

package/README.md

@@ -78,7 +78,10 @@ Or load it from a local path without npm:
 | `description` | default | Tool description shown to the model. |
 | `bridgeTextOnly` | `true` | Bridge pasted images to text hints on models that cannot see images. |
 | `bridgeExportDir` | temp | Export dir for bridged images (`os.tmpdir()/dsh-vision-bridge`). |
-| `multimodalModels` | `[]` | Model ids that receive image blocks directly (e.g. `mimo-v2.5`). |
+| `multimodalModels` | `[]` | Model list (comma-separated). Each entry is matched case-insensitively against the full id, its bare id after the last `/`, and `provider/id`, with `*` / `?` globs (`*vl*`, `deepseek/*`). What the list *means* is set by the mode below. |
+| `multimodalListMode` | `whitelist` | **List mode (v0.9.0).** `whitelist`: listed models receive image blocks directly (the historical behaviour). `blacklist`: listed models are forced through the bridge — the correction layer for a model that claims image support it does not have. `off`: the list is ignored. An unknown value falls back to `whitelist`. |
+| `autoDetectMultimodal` | `true` | **Auto-detect (v0.9.0).** Decide from the current route's own declared `inputModalities`, then combine with the list (whitelist unions, blacklist subtracts). On by default: a text-only route is bridged, a multimodal one is treated like a whitelist member and gets images directly. The declaration is always read *before* this plugin's admission wrap, so `bridgeAutoImage` can never feed its own claim back in as evidence. Set false for the hand-maintained "list only" behaviour. |
+| `probeResults` | `{}` | **Measured verdicts (v0.9.0):** `"provider/model" → "yes"`/`"no"`, written by the `vision_probe_model` tool — do not edit by hand. A measurement outranks a **declaration** (a real request beats a claim) but not `multimodalModels` (explicit human intent has the last word). |
 | `bridgePreview` | `true` | Inline preview for bridged images: thumbnail above the hint text in the user bubble (click to zoom). |
 | `bridgePreviewScanIntervalMs` | `2000` | Fallback scan interval for the preview scanner (ms); `0` disables the fallback. |
 | `bridgePreviewHideHint` | `true` | Hide the bridged hint text once the preview image has loaded (kept on failure — safe degradation). |
@@ -101,11 +104,12 @@ Or load it from a local path without npm:
              input: [text, image]
    ```
 2. List genuinely multimodal models in the plugin config so they receive
-   image blocks untouched:
+   image blocks untouched (see the next section for the list modes):
    ```yaml
    - id: tool-vision
      name: 'dsh-tool-vision'
      config:
+       multimodalListMode: whitelist    # default: listed models get images directly
        multimodalModels: ['mimo-v2.5', 'grok-4.5']
    ```
 
@@ -222,6 +226,154 @@ degrade lazily with an install hint and never break other tools).
 default** — set `desktopScreenshot: true` in the tool-vision settings to
 enable desktop capture.
 
+## v0.9.0: list modes, and auto-detection
+
+The bridge answers one question: **can the current model see images directly?**
+v0.9.0 splits it into two independent inputs.
+
+```
+base = autoDetectMultimodal ? (route declares image) : {}
+off        → direct = base              the list takes no part
+whitelist  → direct = base ∪ list       the list only adds
+blacklist  → direct = base \ list       the list only subtracts
+```
+
+**A list hit always wins**: the list is explicit user intent, so it outranks the
+model's own declaration — which is what makes it a usable correction layer.
+
+**A blacklist never degrades into "everything unlisted is direct".** Its base set
+is the auto-detected one; with auto-detection off that base set is empty, so an
+unlisted model is still bridged. That is deliberate: the alternative lets one
+typo push images at a text-only endpoint.
+
+**Matching**: `mimo-v2.5`, `xiaomi/mimo-v2.5` and `commandcode/xiaomi/mimo-v2.5`
+all address the same route; `*` / `?` are globs; matching is case-insensitive.
+v0.8.1 compared ids literally, so this README's own `mimo-v2.5` example silently
+did nothing on a route spelled `xiaomi/mimo-v2.5` — fixed here, and the fix only
+ever *adds* models to the direct set (no entry that used to force a model direct
+stops doing so).
+
+**In the panel** (Settings → Vision Model):
+
+- the list field is followed by a **clickable list of the models dsh actually has
+  configured** (`llm.listProviders()` + `llm.listModels()`), grouped by provider
+  and labelled with whether the route *declares* image input. Tick to add,
+  untick to remove; the text field above still takes globs by hand. Both edit the
+  *same* draft, persisted by Save — the picker head flags it as unsaved until then.
+- a **current route** readout shows `provider / model`, whether images go direct
+  or through the bridge, and why (list hit / auto-detect / default).
+
+> Why not just a `<datalist>`: a native datalist stays invisible until the user
+> focuses the field and types, which made v0.9.0's first cut look like a dead
+> panel. The list is now always visible, with the datalist kept as a typing aid.
+>
+> Unticking removes **the entry that actually matched** — if `mimo-v2.5` in the
+> list is what covers `xiaomi/mimo-v2.5`, unticking drops `mimo-v2.5` rather than
+> inventing a full id. Which entry hit is computed server-side with the *same*
+> matcher the bridge uses (`matchedEntries` in the payload), so the panel can
+> never display a state that disagrees with the decision.
+
+**Read path (the easiest thing to get wrong here)**: `autoDetectMultimodal` MUST
+read the value from *before* `resolveModelInfo` was wrapped, or the "image
+support" that `bridgeAutoImage` stamps onto every model becomes evidence for
+itself. `unwrappedResolveModelInfo()` enforces that, with a dedicated regression
+test.
+
+Candidates come from the plugin's own loopback route:
+`GET /plugins/dsh-tool-vision/models` (loopback Host only, read-only, `no-store`).
+It returns provider/model ids and one declared-capability boolean — **no keys and
+no endpoint addresses**. It is registered on the plugin fiber rather than the
+master switch's child fiber, so the panel keeps working while the plugin is off.
+
+> ⚠️ `inputModalities` is a **declaration**, not a guarantee — profiles commonly
+> set `input: [text, image]` on text-only models just to pass the admission gate.
+> Upstream `dsh-llm-pi-ai` makes the same call for undeclared models, and its
+> source says why: the two wrong answers **do not cost the same**. Under-claiming
+> refuses the image before it is attached and names the model; over-claiming
+> admits one the provider rejects mid-turn, after the message is already durable.
+>
+> That is why detection is on by default *with* three backstops: (1) the first
+> time a route is promoted purely by its own declaration, the log says so and
+> names the fix; (2) the panel always shows the current route, the decision and
+> the reason; (3) listing that model under `blacklist` mode forces the bridge
+> back on.
+
+## Tests
+
+```bash
+npm test             # server-side unit tests (no extra dependencies)
+npm run test:render  # panel render test (needs devDependencies)
+```
+
+`npm run test:render` loads the **real client bundle** in jsdom, drives the
+**real registration path** (`apply` → `slots.register` → the component), feeds it
+from the **real server route handler**, and asserts on the real DOM and the real
+settings writes.
+
+It is a separate command and deliberately **not part of `npm test`**: it needs
+`react` / `react-dom` / `jsdom`, and a DOM test that silently skips when a
+dependency is missing is a false comfort. Install with
+`npm i -D react@18 react-dom@18 jsdom`.
+
+Its reason to exist is specific: v0.9.0's first cut rendered the model list only
+into a native `<datalist>` — every server-side unit test passed while the panel
+looked completely dead. Nothing below the DOM can catch that class of bug.
+
+## Capability probe: `vision_probe_model` (v0.9.0)
+
+Every other signal here rests on what a model **says** about itself. This tool
+sends a real image to the route and reports what it **does** — the only
+ground truth in the plugin.
+
+```
+base = autoDetect ? route declares image : {}
+a measured verdict (if any) overrides base    measurement > claim
+a list hit (if any) overrides everything      human intent > measurement
+```
+
+**Why one request is not enough** (each of these was learned the hard way):
+
+- **Models guess.** One may answer "blue" to an orange square, so a probe asks
+  about **two different colors** and only passes a model that reads both.
+- **Reasoning models return empty `content`** when the thinking budget eats
+  `max_tokens`; the default is 2048 and the reader falls back to
+  `reasoning_content`.
+- **A control group is mandatory.** A text-only request runs first to prove the
+  route works at all; without it a 401 or a timeout would be misread as "cannot
+  see images" and push a perfectly good multimodal route back onto the bridge
+  **permanently**.
+- **An accepting endpoint is not an image-reading model.** Measured against a
+  real gateway: `meituan/LongCat-2.0:free` returned HTTP 200 for the image
+  request and answered `"I can't see any image."` Only an endpoint that
+  *actively rejects* the image part is a conclusive negative.
+
+So a probe ends in one of three verdicts: `yes` (control passed, both colors
+correct), `no` (the endpoint rejected the image part, or answered without
+reading it), or `unknown` (network, auth or protocol trouble — **never** turned
+into a capability claim).
+
+**How it gets a route's endpoint and credential** (no new configuration):
+`llm.listConfigurableProviders()` names the provider's settings namespace and
+path → `settings.get(ns)` resolves `baseURL`/`apiKeyEnv`/`api` →
+`credentials.resolve(apiKeyEnv)` yields the secret (`.value`) — the same path
+`dsh-llm-pi-ai` uses for a real call. Read-only, and neither the endpoint nor
+the key ever appears in a probe result.
+
+**Linking**: a verdict is written to `probeResults` and takes effect in the
+bridge decision **immediately** — `yes` sends images directly, `no` forces the
+bridge back on — regardless of the list mode. (That is also why it does not
+silently write into `multimodalModels`: a `blacklist` list means the *opposite*,
+so an automatic entry there would produce exactly the wrong result.) Each row in
+the panel's picker carries a badge:
+
+- `measured: reads images` / `measured: no image reading` (blue / red), shown in
+  preference to the declaration;
+- a never-probed route falls back to the `declares image` badge;
+- the current-route readout reports `measured (a real image was read / NOT read)`.
+
+> A route on an unknown protocol (e.g. `openai-responses`) returns `unknown`
+> with a reason, rather than guessing confidently with `chat/completions`.
+
 ## v0.8.0: master switch, and the save-path fix
 
 **Master switch.** `enabled`, plus a one-click button at the top of the section
@@ -247,11 +399,23 @@ effect" plus a reload of the form. Hosts without `mutate()` fall back to
 **sequential** writes (each waits for its predecessor, keeping the revision chain
 intact) — never parallel.
 
+Also removes three `if (typeof scope.load === "function") scope.load()` guards.
+The `SettingsScope` seam has never had `load()` — it is `getSnapshot` /
+`subscribe` / `mutate` / `set` / `unset`, and reads ride the shared describe
+mirror driven by the Host's `settings/document-updated`. Those guards were dead
+code that read like a refresh which never happened, and they made the missing
+write verification look intentional.
+
 ## Limitations
 
 - A bridged image enters the conversation as a text hint (a transcript, not
   pixels) — pixel-precise in-context reasoning is not available to text-only
   models; the vision model's description comes back through `inspect_image`.
+- The bridge is a **one-way door**: an image pasted on a text-only model is
+  rewritten into the durable log at `agent/pre-step`, so switching to a
+  multimodal model later does not turn it back into an image block. (The other
+  direction — multimodal to text-only — is repaired automatically by
+  `repairLoggedImages`.)
 - Images are base64-transferred; mind privacy and size limits.
 - Independent of the dsh-llm routing/retry system; failures return clear
   errors to the agent.

package/README.zh.md

@@ -53,7 +53,10 @@ DSH 0.1.1 已为 DeepSeek 视觉模型目录加入原生图片输入。本插件
 | `description` | 默认描述 | 工具描述（模型可见） |
 | `bridgeTextOnly` | `true` | 把粘贴图片转成文本指引（发给看不懂图片的模型时） |
 | `bridgeExportDir` | 临时目录 | 桥接图片导出目录（`os.tmpdir()/dsh-vision-bridge`） |
-| `multimodalModels` | `[]` | 直发图片块的模型 id（如 `mimo-v2.5`） |
+| `multimodalModels` | `[]` | 模型名单（逗号分隔）。每项按「完整 id / 末段裸 id / `provider/id`」三种写法匹配，大小写不敏感，支持 `*` `?` 通配（如 `*vl*`、`deepseek/*`）。含义由下面的模式决定 |
+| `multimodalListMode` | `whitelist` | **名单模式（v0.9.0）**：`whitelist` 名单内模型直收图片块（旧行为）；`blacklist` 名单内模型强制走桥接（用来纠正"声明支持图片但实际不支持"的模型）；`off` 忽略名单。未知值一律回退 `whitelist` |
+| `autoDetectMultimodal` | `true` | **自动识别（v0.9.0）**：按当前路由自己声明的 `inputModalities` 判定，再与名单合成（白名单取并集、黑名单取差集）。默认开启＝识别到纯文本就交给桥接、识别到多模态就等同白名单成员直发图片；声明永远读"包装前"的真值，不会被 `bridgeAutoImage` 的假声明污染。设为 `false` 可退回"只认名单"的纯手工行为 |
+| `probeResults` | `{}` | **实测结果（v0.9.0）**：`"provider/model" → "yes"`/`"no"`，由 `vision_probe_model` 工具写入，不要手改。实测优先级**高于声明**（真实请求 > 自称），但**低于** `multimodalModels`（人的明确意图有最终话语权） |
 | `bridgePreview` | `true` | 桥接图片内联预览：用户气泡内显示缩略图，点击放大 |
 | `bridgePreviewScanIntervalMs` | `2000` | 预览兜底扫描间隔（毫秒）；`0` 关闭兜底 |
 | `bridgePreviewHideHint` | `true` | 图片加载成功后隐藏桥接提示文本（失败时保留，安全降级） |
@@ -73,11 +76,12 @@ DSH 0.1.1 已为 DeepSeek 视觉模型目录加入原生图片输入。本插件
            - id: deepseek-v4-flash
              input: [text, image]
    ```
-2. 在插件配置里列出真正多模态的模型，让它们直收图片块：
+2. 在插件配置里列出真正多模态的模型，让它们直收图片块（名单模式见下一节）：
    ```yaml
    - id: tool-vision
      name: 'dsh-tool-vision'
      config:
+       multimodalListMode: whitelist   # 默认，名单内直发图片
        multimodalModels: ['mimo-v2.5', 'grok-4.5']
    ```
 
@@ -157,6 +161,82 @@ DSH 0.1.1 已为 DeepSeek 视觉模型目录加入原生图片输入。本插件
 `vision_screenshot` 属于隐私敏感能力,**默认不注册**——在 tool-vision 设置中
 开启 `desktopScreenshot: true` 后才会注册桌面截屏工具。
 
+## v0.9.0：模型名单模式与自动识别
+
+桥接要回答一个问题：**当前这个模型能不能直接看图片？** v0.9.0 把它拆成两个相互独立的输入。
+
+```
+base = autoDetectMultimodal ? (路由声明含 image) : 空集
+off        → direct = base              名单不参与
+whitelist  → direct = base ∪ 名单        名单只做"加"
+blacklist  → direct = base \ 名单        名单只做"减"
+```
+
+**命中名单时以名单为准**：名单是用户明确的意图，优先级高于模型自己的声明，所以永远能纠正误判。
+
+**黑名单绝不会退化成"未列出的一律直发"**：黑名单的基准是自动识别集；没开自动识别时基准是空集，未列出的模型照样走桥接。这是刻意设计的——否则一次误配就能把图片硬塞给纯文本端点。
+
+**匹配规则**：`mimo-v2.5`、`xiaomi/mimo-v2.5`、`commandcode/xiaomi/mimo-v2.5` 指向同一个路由；`*` / `?` 为通配符；大小写不敏感。v0.8.1 只做字符串全等，README 自己举的 `mimo-v2.5` 例子在 `xiaomi/mimo-v2.5` 这类路由上其实**静默无效**，v0.9.0 修好了（只会多放行，不会收回任何旧配置已经放行的模型）。
+
+**面板**（设置 → 视觉模型）：
+
+- 名单字段下面是一份**可直接勾选的模型清单**：候选来自 dsh 已配置的模型（`llm.listProviders()` + `llm.listModels()`），按 provider 分组，标注哪些**声明**支持图片；勾选加入、取消勾选移出，上方输入框仍可手打通配符（两者共用同一份草稿，改完按「保存」生效，未保存时清单头部会标"未保存"）；
+- 顶部常驻**当前路由判定**：`provider / model`、直发还是桥接、依据是什么（名单命中 / 自动识别 / 默认）。
+
+> 为什么不只用 `<datalist>`：原生 datalist **必须聚焦输入框并敲字才会弹出**，v0.9.0 第一版因此看起来像"面板没反应"。现在清单常驻可见，datalist 只作打字辅助。
+>
+> 取消勾选移除的是**命中的那条记录本身**（名单里写 `mimo-v2.5` 命中了 `xiaomi/mimo-v2.5`，取消勾选就移除 `mimo-v2.5`，不会凭空虚增一个完整 id）。"哪条命中了"由服务端用与桥接**同一套匹配器**算出（payload 的 `matchedEntries`），客户端不复制规则，所以面板显示的状态永远不会和实际判定打架。
+
+**判定依据的读取路径**（这是本版最容易做错的地方）：`autoDetectMultimodal` **必须**读 `resolveModelInfo` 被包装之前的真值，否则 `bridgeAutoImage` 给所有模型贴上的"支持图片"就成了自证。代码里由 `unwrappedResolveModelInfo()` 保证，并有专门的回归测试。
+
+候选项由插件自己的回环路由提供：`GET /plugins/dsh-tool-vision/models`（仅本机 Host、只读、`no-store`），只返回 provider/model id 与声明能力，**不含任何密钥或端点地址**。它挂在插件主 fiber 而非总开关的子 fiber 上，所以插件关闭时面板依然可用。
+
+> ⚠️ `inputModalities` 是**声明**，不是保证——profile 里常见"为了过准入检查而写 `input: [text, image]`"的纯文本模型。上游 `dsh-llm-pi-ai` 自己也把"未声明"一律当作纯文本，理由写在源码注释里：**两种误判的代价不对等**——少声明会在贴图前就拒绝并点名模型，多声明则会放行一张"上游中途拒收、而消息已经落盘"的图片。
+>
+> 所以自动识别是默认开启的，同时带三层兜底：① 某个路由**首次**因"仅凭自己的声明"被放行时，日志会打一条明确提示并点名怎么改；② 面板顶部常驻当前判定与依据；③ 把该模型写进 `multimodalModels` 并切到 `blacklist` 模式即可强制回到桥接。
+
+## 测试
+
+```bash
+npm test           # 服务端单测（不需要额外依赖）
+npm run test:render  # 面板渲染测试（需要 devDependencies）
+```
+
+`npm run test:render` 在 jsdom 里加载**真实客户端 bundle**、走**真实注册路径**（`apply` → `slots.register` → 组件），数据由**真实的服务端路由处理器**产出，最后断言真实 DOM 与真实的设置写入。
+
+它单独成命令、**不并入 `npm test`**：它需要 `react` / `react-dom` / `jsdom`，而一个"依赖缺失就静默跳过"的 DOM 测试只会带来虚假的安全感。需要时先 `npm i -D react@18 react-dom@18 jsdom`。
+
+它存在的理由很具体：v0.9.0 第一版把模型清单只渲染进原生 `<datalist>` —— 服务端单测**全绿**，面板却看起来完全没反应。这类 bug 在 DOM 之下根本抓不到。
+
+## 实测探测：`vision_probe_model`（v0.9.0）
+
+前面所有判定都建立在"模型**说自己**能不能看图"之上。这个工具改为**真的发一张图过去试**，是插件里唯一的地面真相。
+
+```
+base = autoDetect ? 路由声明含 image : ∅
+探测结果（若有）覆盖 base                  实测 > 自称
+名单命中（若有）覆盖一切                    人的意图 > 实测
+```
+
+**为什么一次请求不够**（都是实测踩出来的）：
+
+- **会瞎猜**：模型可能对橙色方块回答"blue"。所以探测跑**两个不同颜色**，两个都答对才算通过；
+- **推理模型会返回空 `content`**：思考预算不够时 `max_tokens` 全被烧光。默认给 2048，并回退读 `reasoning_content`；
+- **必须要有对照组**：先发一条纯文本请求确认"这条路本身是通的"，否则一个 401/超时会被误读成"它不能看图"——那会把一条好端端的多模态路由**永久**推回桥接；
+- **端点不报错 ≠ 能看图**。真实网关实测：`meituan/LongCat-2.0:free` 收到图片请求返回 200，但回答是 `"I can't see any image."`。只有"端点主动拒绝图片部分"才是确定性的负面信号。
+
+结论分三档：`yes`（对照组通过 + 两色全对）、`no`（端点拒绝图片，或答了但没读图）、`unknown`（网络/鉴权/协议问题——**绝不**当作能力结论）。
+
+**它怎么拿到主模型的端点与凭据**（不新增任何配置）：`llm.listConfigurableProviders()` 给出 provider 的 settings 命名空间与路径 → `settings.get(ns)` 读出 `baseURL`/`apiKeyEnv`/`api` → `credentials.resolve(apiKeyEnv)` 取出密钥（`.value`），与 `dsh-llm-pi-ai` 真实调用同一条路。只读，且**端点与密钥绝不出现在探测结果里**。
+
+**联动**：探测结果写入 `probeResults` 并**立即参与桥接判定**——`yes` 直接放行图片、`no` 强制回到桥接，且**与名单模式无关**。（这比"自动往名单里写一笔"更正确：名单在 `blacklist` 模式下含义是反的，自动写入会得到完全相反的效果。）面板清单里每个模型带徽章：
+
+- `已实测可看图` / `已实测不可看图`（蓝 / 红）——实测优先展示；
+- 没探测过的才回退显示声明的 `声明支持图片`；
+- 顶部"依据"栏显示 `实测（真实发图验证通过/未通过）`。
+
+> 未知协议的路由（如 `openai-responses`）会直接返回 `unknown` 并说明原因，而不是用 `chat/completions` 去猜一个自信的错误答案。
+
 ## v0.8.0：总开关，以及保存修复
 
 **总开关。** `enabled` 字段加设置栏顶部的一键按钮（`一键关闭` / `重新启用`）。
@@ -176,12 +256,18 @@ fiber，15 个工具一并从模型侧消失。设置栏留在父 fiber 上，
 重新载入表单。没有 `mutate()` 的旧宿主回退为**串行**写入（每个等前一个完成，revision
 链条依然正确），绝不并行。
 
+另外删掉了 3 处 `if (typeof scope.load === "function") scope.load()`：`SettingsScope` 接口
+从来没有 `load()`（完整 seam 只有 getSnapshot / subscribe / mutate / set / unset，读走的是
+共享 describe 镜像，由宿主 `settings/document-updated` 驱动刷新）。这些守卫是照臆测 API 写的
+死代码，读起来像"已经刷新过了"，反而掩盖了缺失的写入校验。
+
 ## 限制
 
 - 从 `0.6.3` 起最低支持 DSH `0.1.0-rc.7`，已针对 `0.1.0-rc.7`、
   `0.1.0-rc.8` 和 `0.1.1-rc.1` 测试。仍使用 DSH `0.1.0-rc.6` 的用户请锁定
   `dsh-tool-vision@0.6.1`；这是最后一个包含旧 settings 白名单兼容补丁的版本。
 - 被桥接的图片以文本指引进入对话（转录而非像素）——文本模型无法做像素级上下文推理；视觉模型的描述通过 `inspect_image` 回传。
+- 桥接是**单向门**：文本模型下贴的图会在 `agent/pre-step` 被写进持久化日志，之后切到多模态模型也不会还原成图片块（反方向——多模态切到文本模型——会由 `repairLoggedImages` 自动补上桥接）。
 - 图片以 base64 传输；注意隐私与大小限制。
 - 独立于 dsh-llm 的路由/重试体系；失败会向 Agent 返回明确错误。