dsh-plugin-image-tools 0.3.1 → 0.4.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Pasumao
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
package/README.md CHANGED
@@ -1,11 +1,17 @@
1
1
  # dsh-plugin-image-tools
2
2
 
3
- DeepSeek Harness Web GUI 增加**图片能力**的插件,两个工具覆盖两种场景:
3
+ ![npm version](https://img.shields.io/npm/v/dsh-plugin-image-tools)
4
+ ![License](https://img.shields.io/github/license/Pasumao/dsh-plugin-image-tools)
5
+ ![AI Assisted](https://img.shields.io/badge/AI-Assisted-8A2BE2)
6
+
7
+ **dsh 插件市场里唯一支持「图片选择卡」的插件**:给 DeepSeek Harness Web GUI 增加图片能力,
8
+ 三个工具覆盖三种场景——模型让你在选项里挑图、在回复正文里展示图、你把图发给盲模型。全部零 token 本地渲染:
4
9
 
5
10
  | 工具 | 场景 | 效果 |
6
11
  |---|---|---|
7
12
  | `ask_user_choice` | 模型让用户在**选项**里挑图 | Web GUI 渲染图片选择卡,可放大查看,答案协议与原生一致 |
8
13
  | `show_images` | 模型在**回复正文**里展示图片 | 图片与文字混排显示在聊天里,点击可放大 |
14
+ | `save_received_images` | 用户**发图给盲模型**(无视觉输入的适配器) | 图片以文件形式保存到工作区,模型可下载/分析 |
9
15
 
10
16
  图片来源统一支持三种:**本地路径**(相对会话工作区或绝对路径,含 ComfyUI 出图产物)、
11
17
  **http(s) URL**(服务端拉取后转存)、**base64 data URI**。纯插件实现,不改核心包。
@@ -24,6 +30,8 @@
24
30
 
25
31
  模型在回复里调用 `show_images`,把返回的 markdown 片段粘贴进正文,图片就随文字一起显示。
26
32
 
33
+ > 两张效果图均为真实 Web GUI 截图(maid-atelier 皮肤)。
34
+
27
35
  ## 功能
28
36
 
29
37
  - **`ask_user_choice`**(图片 / 图文混合选项):
@@ -35,6 +43,15 @@
35
43
  - 一次展示 1~9 张图,每张可带 `caption` 说明;
36
44
  - 工具返回绝对 URL 的 markdown 图片片段,模型原样粘贴进回复正文即随文显示;
37
45
  - 客户端插件自动增强这类图片:圆角样式、悬停显示说明、点击放大查看、加载失败降级。
46
+ - **`save_received_images`**(盲模型收图 → 文件):
47
+ - 用户往聊天里发图片时,`agent/pre-step` 监听器把消息里的 image 内容块重写为
48
+ 文本占位符(`dshimg:<attachmentId>`)——文本-only 适配器(如 DeepSeek)不再
49
+ 因图片块报 `UNSUPPORTED_CONTENT`,回合照常运行;
50
+ - 用户气泡里由客户端增强器把占位符替换为可放大的图片回显(附件经
51
+ `/dsh-plugin-image-tools/attachment/<id>` 路由出字节);
52
+ - 模型看到占位符后调用 `save_received_images`,把图片按 attachmentId 保存为
53
+ 工作区文件(默认 `received/`),之后可用文件/命令工具分析(尺寸、像素、哈希等);
54
+ - 保存的文件名优先用附件自带的安全文件名,否则按 `image-<n>-<时间戳>.<ext>` 生成。
38
55
  - 纯文字问题不带图片时,客户端自动放行给原生 UI,互不影响。
39
56
 
40
57
  ## 实现要点(为什么是插件而不是改核心)
@@ -44,9 +61,10 @@
44
61
  所以图片**不能**塞进 option / content 字段。本插件改为:
45
62
 
46
63
  1. 服务端把图片字节归一化进内存注册表,通过自定义 web 路由
47
- `/dsh-plugin-image-tools/<pickId>/<index>`(选择卡)与
48
- `/dsh-plugin-image-tools/show/<showId>/<index>`(回复内嵌)直接提供字节
49
- (同源 `<img src>` 加载);
64
+ `/dsh-plugin-image-tools/<pickId>/<index>`(选择卡)、
65
+ `/dsh-plugin-image-tools/show/<showId>/<index>`(回复内嵌)与
66
+ `/dsh-plugin-image-tools/attachment/<attachmentId>`(盲模型收图回显)
67
+ 直接提供字节(同源 `<img src>` 加载);
50
68
  2. 选择卡:在问题的 `detail`(标准字符串字段,原样透传)开头写入不可见的
51
69
  HTML 注释标记 `<!--dsh-pick:v1:<base64url JSON>-->`,携带 pickId 与带图选项下标;
52
70
  客户端插件在 `conversation.composer` slot 链注册条目(priority 更小,优先于原生),
@@ -54,49 +72,38 @@
54
72
  3. 回复内嵌:`show_images` 返回绝对 URL(宿主 origin 由 `ctx.webServer.host/port`
55
73
  推导),模型粘贴进正文,核心 markdown 渲染器原生显示;客户端再对
56
74
  `/dsh-plugin-image-tools/show/` 前缀的图片做渐进增强(MutationObserver 发现 +
57
- 单节点样式/事件注入,纯 DOM,不侵入 React 渲染树)。
75
+ 单节点样式/事件注入,纯 DOM,不侵入 React 渲染树);
76
+ 4. 盲模型收图:注册 `agent/pre-step` waterfall 监听器(与 agent-instructions /
77
+ time-context 同机制),把进入 LLM 步骤的消息批次里的 image 块重写为文本占位符
78
+ (登记附件 ref 到 TTL 注册表),会话日志/UI 因此保持纯文本安全;客户端增强器
79
+ 在用户气泡文本里识别 `dshimg:<id>` 占位符并替换为可放大图片;`save_received_images`
80
+ 经 `ctx.attachments.readImage` 取回附件字节落盘。
58
81
 
59
82
  详见 `设计说明.md`。
60
83
 
61
84
  ## 目录结构
62
85
 
63
86
  ```
64
- lib/index.js 服务端:两个工具注册 + 图片注册表 + web 路由(零运行时依赖)
65
- lib/client.js 客户端:composer 链条目 + 图片选择 UI + 内嵌图片增强(浏览器模块加载器格式,免构建)
87
+ lib/index.js 服务端:三个工具注册 + pre-step 重写 + 图片/附件注册表 + web 路由(零运行时依赖)
88
+ lib/client.js 客户端:composer 链条目 + 图片选择 UI + 内嵌/收图图片增强(浏览器模块加载器格式,免构建)
66
89
  scripts/selfcheck.mjs 纯函数自检(node scripts/selfcheck.mjs)
67
- scripts/smoke-server.mjs 服务端集成冒烟(假 ctx 跑通工具→路由→回答全链路)
90
+ scripts/smoke-server.mjs 服务端集成冒烟(假 ctx 跑通工具→路由→pre-step 重写→落盘全链路)
68
91
  scripts/smoke-client.mjs 客户端冒烟(真实 react 渲染选择卡 + 增强纯函数)
69
92
  cordis.patch.yml bundle 补丁(挂载行)
70
93
  docs/ 效果图(README 展示用)
71
94
  ```
72
95
 
73
- ## 安装(web profile)
96
+ ## 安装
74
97
 
75
98
  ```powershell
76
- # 1. 把插件 link 进 profile(已写入 package.json 时跳过这步的编辑)
77
- cd C:\Users\18303\.dsh\profiles\web
78
- pnpm install
79
-
80
- # 2. 重启 dsh(launcher),然后刷新浏览器页面
81
- ```
82
-
83
- profile 的 `package.json` 需要包含:
84
-
85
- ```jsonc
86
- {
87
- "dependencies": {
88
- "dsh-plugin-image-tools": "link:D:/dsh/plugins/dsh-plugin-image-tools"
89
- },
90
- "dsh": {
91
- "profile": {
92
- "bundles": [ /* ... */, "dsh-plugin-image-tools" ]
93
- }
94
- }
95
- }
99
+ # npm(推荐)
100
+ dsh plugin --profile web add dsh-plugin-image-tools
101
+ # 或 GitHub
102
+ dsh plugin --profile web add github:Pasumao/dsh-plugin-image-tools
96
103
  ```
97
104
 
98
- `dsh.profile.bundles` 里的包会自动应用其自带 `cordis.patch.yml` 的挂载行
99
- (与 dsh-notify 同机制)。
105
+ 装完重启 dsh(launcher),然后刷新浏览器页面。包自带 `cordis.patch.yml` 挂载行,
106
+ `dsh.profile.bundles` 自动应用(与 dsh-notify 同机制),无需手动改配置。
100
107
 
101
108
  ## 模型用法示例
102
109
 
@@ -158,4 +165,4 @@ profile 的 `package.json` 需要包含:
158
165
 
159
166
  ## 许可证
160
167
 
161
- MIT
168
+ [MIT](./LICENSE)
Binary file
Binary file
package/lib/client.js CHANGED
@@ -352,6 +352,7 @@ window.__ModuleLoader__.load({
352
352
  // 纯 DOM 单节点变更:不插入/移除 React 管理的结构,重渲染时安全。
353
353
  // ------------------------------------------------------------------
354
354
  var IMAGE_SHOW_PREFIX = '/dsh-plugin-image-tools/show/';
355
+ var IMAGE_TOKEN_PREFIX = 'dshimg:';
355
356
  var ZOOM_CLOSE_LABEL = (typeof document !== 'undefined' && (document.documentElement.lang || '').toLowerCase().indexOf('zh') === 0)
356
357
  ? '\u5173\u95ed'
357
358
  : 'Close';
@@ -361,6 +362,69 @@ window.__ModuleLoader__.load({
361
362
  return typeof src === 'string' && src.indexOf(IMAGE_SHOW_PREFIX) >= 0;
362
363
  }
363
364
 
365
+ /** 从文本里提取 dshimg: 占位符的 attachmentId(纯函数,供冒烟测试)。 */
366
+ function extractImageTokenIds(text) {
367
+ if (typeof text !== 'string') return [];
368
+ var seen = {};
369
+ var ids = [];
370
+ var re = new RegExp(IMAGE_TOKEN_PREFIX + '([A-Za-z0-9_-]{8,})', 'g');
371
+ var m;
372
+ while ((m = re.exec(text)) !== null) {
373
+ if (seen[m[1]] === void 0) {
374
+ seen[m[1]] = true;
375
+ ids.push(m[1]);
376
+ }
377
+ }
378
+ return ids;
379
+ }
380
+
381
+ /** 文本节点扫描去重表:node → 上次扫描到的 nodeValue(React 原位更新文本时重新扫描)。 */
382
+ var scannedTexts = typeof WeakMap === 'undefined' ? null : new WeakMap();
383
+
384
+ /**
385
+ * 把文本节点里的 dshimg:<id> 占位符替换为可放大的图片元素(盲模型收图回显)。
386
+ * 只做文本节点级替换,不触碰 React 管理的元素结构。
387
+ */
388
+ function upgradeBubbleTexts() {
389
+ if (typeof document === 'undefined' || scannedTexts === null || typeof document.createTreeWalker === 'undefined') return;
390
+ var walker = document.createTreeWalker(document.body, NodeFilter.SHOW_TEXT);
391
+ var node;
392
+ while ((node = walker.nextNode()) !== null) {
393
+ var value = node.nodeValue;
394
+ if (typeof value !== 'string' || value.indexOf(IMAGE_TOKEN_PREFIX) < 0) continue;
395
+ if (scannedTexts.get(node) === value) continue;
396
+ var ids = extractImageTokenIds(value);
397
+ if (ids.length === 0) {
398
+ scannedTexts.set(node, value);
399
+ continue;
400
+ }
401
+ var parent = node.parentNode;
402
+ if (parent === null) continue;
403
+ var re = new RegExp(IMAGE_TOKEN_PREFIX + '([A-Za-z0-9_-]{8,})', 'g');
404
+ var frag = document.createDocumentFragment();
405
+ var cursor = 0;
406
+ var m;
407
+ while ((m = re.exec(value)) !== null) {
408
+ if (m.index > cursor) frag.appendChild(document.createTextNode(value.slice(cursor, m.index)));
409
+ var img = document.createElement('img');
410
+ img.className = 'dshimg-inline dshimg-received';
411
+ img.src = '/dsh-plugin-image-tools/attachment/' + m[1];
412
+ img.alt = '\u7528\u6237\u53d1\u6765\u7684\u56fe\u7247';
413
+ img.loading = 'lazy';
414
+ img.addEventListener('click', function (event) {
415
+ event.preventDefault();
416
+ event.stopPropagation();
417
+ openImageZoom(img.src, '\u7528\u6237\u53d1\u6765\u7684\u56fe\u7247\uff08\u70b9\u51fb\u653e\u5927\uff09');
418
+ });
419
+ img.addEventListener('error', function () { img.dataset.failed = '1'; });
420
+ frag.appendChild(img);
421
+ cursor = m.index + m[0].length;
422
+ }
423
+ if (cursor < value.length) frag.appendChild(document.createTextNode(value.slice(cursor)));
424
+ parent.replaceChild(frag, node);
425
+ }
426
+ }
427
+
364
428
  /** 打开命令式 Lightbox 放大查看(复用选择卡的 .dshpick-lightbox 样式)。 */
365
429
  function openImageZoom(src, caption) {
366
430
  if (typeof document === 'undefined') return;
@@ -417,7 +481,7 @@ window.__ModuleLoader__.load({
417
481
  var caption = (img.getAttribute('alt') || '').trim();
418
482
  if (caption !== '') img.title = caption;
419
483
  var src = img.getAttribute('src') || '';
420
- if (!isShowImageSrc(src)) return;
484
+ if (!(isShowImageSrc(src) || src.indexOf('/dsh-plugin-image-tools/attachment/') >= 0)) return;
421
485
  img.addEventListener('click', function (event) {
422
486
  event.preventDefault();
423
487
  event.stopPropagation();
@@ -436,8 +500,9 @@ window.__ModuleLoader__.load({
436
500
  var pending = false;
437
501
  var scan = function () {
438
502
  pending = false;
439
- var images = document.querySelectorAll('img[src^="' + IMAGE_SHOW_PREFIX + '"]');
503
+ var images = document.querySelectorAll('img[src^="' + IMAGE_SHOW_PREFIX + '"], img[src^="/dsh-plugin-image-tools/attachment/"]');
440
504
  for (var i = 0; i < images.length; i++) upgradeInlineImage(images[i]);
505
+ upgradeBubbleTexts();
441
506
  };
442
507
  var observer = new MutationObserver(function () {
443
508
  if (pending) return;
@@ -856,6 +921,7 @@ window.__ModuleLoader__.load({
856
921
  exports.Lightbox = Lightbox;
857
922
  exports.IMAGE_SHOW_PREFIX = IMAGE_SHOW_PREFIX;
858
923
  exports.isShowImageSrc = isShowImageSrc;
924
+ exports.extractImageTokenIds = extractImageTokenIds;
859
925
  exports.upgradeInlineImage = upgradeInlineImage;
860
926
  exports.startInlineEnhancer = startInlineEnhancer;
861
927
  return module.exports;
package/lib/index.js CHANGED
@@ -29,13 +29,13 @@
29
29
  *
30
30
  * @module dsh-plugin-image-tools
31
31
  */
32
- import { readFile } from 'node:fs/promises'
33
- import { isAbsolute, join } from 'node:path'
32
+ import { mkdir, readFile, writeFile } from 'node:fs/promises'
33
+ import { basename, isAbsolute, join, normalize } from 'node:path'
34
34
  import { randomUUID } from 'node:crypto'
35
35
 
36
36
  export const name = 'dsh-plugin-image-tools'
37
- /** 需要的主机端服务:tools(注册工具)、userQuestions(问询)、webServer(图片路由与 origin)。 */
38
- export const inject = ['tools', 'userQuestions', 'webServer']
37
+ /** 需要的主机端服务:tools(注册工具)、userQuestions(问询)、webServer(图片路由与 origin)、attachments(附件字节存取)、llm(模型能力补丁)。 */
38
+ export const inject = ['tools', 'userQuestions', 'webServer', 'attachments', 'llm']
39
39
 
40
40
  // ---------------------------------------------------------------------------
41
41
  // 常量
@@ -52,6 +52,10 @@ export const FETCH_TIMEOUT_MS = 30 * 1000
52
52
  /** detail 里不可见标记的前缀/后缀。 */
53
53
  export const MARKER_PREFIX = '<!--dsh-pick:v1:'
54
54
  export const MARKER_SUFFIX = '-->'
55
+ /** 盲模型收图:文本占位符里的图片 token 前缀(attachmentId 紧随其后)。 */
56
+ export const IMAGE_TOKEN_PREFIX = 'dshimg:'
57
+ /** save_received_images 的默认保存目录(相对会话工作区)。 */
58
+ export const RECEIVED_DIR = 'received'
55
59
 
56
60
  /** 支持的图片媒体类型。 */
57
61
  const SUPPORTED_MEDIA_TYPES = new Set(['image/png', 'image/jpeg', 'image/webp', 'image/gif'])
@@ -60,12 +64,16 @@ const SUPPORTED_MEDIA_TYPES = new Set(['image/png', 'image/jpeg', 'image/webp',
60
64
  // 内存图片注册表(进程生命周期;每次新增/路由命中时顺带做 TTL 清理)
61
65
  // picks:ask_user_choice 的选择卡图片(随回答/取消立即释放)
62
66
  // shows:show_images 的回复内嵌图片(依赖 TTL 清理,需存活到回复渲染完)
67
+ // attachmentRefs:盲模型收图的附件 ref 登记(pre-step 重写时记录,供
68
+ // /attachment 路由与 save_received_images 解析;TTL 清理)
63
69
  // ---------------------------------------------------------------------------
64
70
 
65
71
  /** @type {Map<string, { createdAt: number, images: { bytes: Buffer, mediaType: string, name?: string }[] }>} */
66
72
  const picks = new Map()
67
73
  /** @type {Map<string, { createdAt: number, images: { bytes: Buffer, mediaType: string, caption?: string }[] }>} */
68
74
  const shows = new Map()
75
+ /** @type {Map<string, { createdAt: number, ref: object }>} */
76
+ const attachmentRefs = new Map()
69
77
 
70
78
  function pruneRegistry(registry, now = Date.now()) {
71
79
  for (const [id, entry] of registry) {
@@ -81,6 +89,10 @@ function pruneShows(now = Date.now()) {
81
89
  pruneRegistry(shows, now)
82
90
  }
83
91
 
92
+ function pruneAttachmentRefs(now = Date.now()) {
93
+ pruneRegistry(attachmentRefs, now)
94
+ }
95
+
84
96
  function getPick(pickId) {
85
97
  prunePicks()
86
98
  return picks.get(pickId)
@@ -91,6 +103,19 @@ function getShow(showId) {
91
103
  return shows.get(showId)
92
104
  }
93
105
 
106
+ /** 登记一次 pre-step 重写见到的附件 ref(供路由与工具解析)。 */
107
+ function recordAttachmentRef(ref) {
108
+ if (ref === null || typeof ref !== 'object' || typeof ref.attachmentId !== 'string' || ref.attachmentId === '') return
109
+ pruneAttachmentRefs()
110
+ attachmentRefs.set(ref.attachmentId, { createdAt: Date.now(), ref })
111
+ }
112
+
113
+ /** 按 attachmentId 取回登记的 ref;未知/过期返回 undefined。 */
114
+ function resolveAttachmentRef(attachmentId) {
115
+ pruneAttachmentRefs()
116
+ return attachmentRefs.get(attachmentId)?.ref
117
+ }
118
+
94
119
  // ---------------------------------------------------------------------------
95
120
  // 纯函数:媒体类型探测 / 标记编解码 / 宿主 origin(导出供 selfcheck 测试)
96
121
  // ---------------------------------------------------------------------------
@@ -186,6 +211,74 @@ export function safeAlt(caption) {
186
211
  return raw.replace(/[\]\n\r]/g, ' ').slice(0, 200)
187
212
  }
188
213
 
214
+ /**
215
+ * 媒体类型 → 文件扩展名。
216
+ * @param {string} mediaType - image/png 等。
217
+ * @returns {string} 带点扩展名(.png 等)。
218
+ */
219
+ export function mediaTypeToExt(mediaType) {
220
+ switch (mediaType) {
221
+ case 'image/png': return '.png'
222
+ case 'image/jpeg': return '.jpg'
223
+ case 'image/webp': return '.webp'
224
+ case 'image/gif': return '.gif'
225
+ default: return ''
226
+ }
227
+ }
228
+
229
+ /**
230
+ * 构造盲模型收图的文本占位符(图片块的模型侧表示)。
231
+ * 含 token `dshimg:<attachmentId>`(客户端据此在用户气泡里回显图片),
232
+ * 以及一句提示语(引导模型用 save_received_images 保存为文件)。
233
+ * @param {object} ref - ImageAttachmentRef { attachmentId, mediaType, bytes, width, height, name? }。
234
+ * @returns {string} 占位符文本。
235
+ */
236
+ export function imagePlaceholderText(ref) {
237
+ const id = typeof ref?.attachmentId === 'string' ? ref.attachmentId : 'unknown'
238
+ const name = typeof ref?.name === 'string' && ref.name !== '' ? `(${ref.name})` : ''
239
+ return `📷 用户发来的图片${name} ${IMAGE_TOKEN_PREFIX}${id}`
240
+ }
241
+
242
+ /**
243
+ * 从文本里提取图片 token 的 attachmentId 列表(客户端回显与冒烟测试用)。
244
+ * @param {string} text - 任意文本。
245
+ * @returns {string[]} attachmentId 列表(去重保序)。
246
+ */
247
+ export function extractImageTokenIds(text) {
248
+ if (typeof text !== 'string') return []
249
+ const seen = new Set()
250
+ const ids = []
251
+ const re = new RegExp(`${IMAGE_TOKEN_PREFIX}([A-Za-z0-9_-]{8,})`, 'g')
252
+ let m
253
+ while ((m = re.exec(text)) !== null) {
254
+ if (!seen.has(m[1])) {
255
+ seen.add(m[1])
256
+ ids.push(m[1])
257
+ }
258
+ }
259
+ return ids
260
+ }
261
+
262
+ /**
263
+ * 把一条消息里的 image 内容块重写为文本占位符(盲模型适配器不接收图片块)。
264
+ * 重写时登记附件 ref;消息与内容块保持冻结语义(构造新对象)。
265
+ * @param {object} message - UserMessage { id, role, content, source }。
266
+ * @returns {object|null} 重写后的新消息;无图片块时返回 null(原样保留)。
267
+ */
268
+ export function rewriteMessageImages(message) {
269
+ if (message === null || typeof message !== 'object' || !Array.isArray(message.content)) return null
270
+ let changed = false
271
+ const content = message.content.map((block) => {
272
+ if (block === null || typeof block !== 'object' || block.type !== 'image' || block.attachment === undefined) return block
273
+ const ref = block.attachment
274
+ recordAttachmentRef(ref)
275
+ changed = true
276
+ return Object.freeze({ type: 'text', text: imagePlaceholderText(ref) })
277
+ })
278
+ if (!changed) return null
279
+ return Object.freeze({ ...message, content: Object.freeze(content) })
280
+ }
281
+
189
282
  // ---------------------------------------------------------------------------
190
283
  // 图片加载
191
284
  // ---------------------------------------------------------------------------
@@ -369,12 +462,35 @@ export function choiceTool(ctx, opts = {}) {
369
462
  const now = Date.now()
370
463
  prunePicks(now)
371
464
 
465
+ // 与浏览器端 askUserQuestionItemSchema(muxFrameSchema)对齐的必填/类型校验:
466
+ // 模型偶发漏传 question 等必填字段时,若原样转发,浏览器 zod 解析会把
467
+ // question/requested 整帧丢弃,选择卡静默不渲染(用户只见"图呢,看不到啊",
468
+ // 最终 ASK_ABORTED)。这里在服务端提前校验并抛面向模型的错误,让模型自我修正。
469
+ const questions = Array.isArray(args.questions) ? args.questions : []
470
+ for (let qi = 0; qi < questions.length; qi++) {
471
+ const q = questions[qi]
472
+ if (q === null || typeof q !== 'object') throw new Error(`questions[${qi}] 必须是对象`)
473
+ if (typeof q.id !== 'string' || q.id === '') throw new Error(`questions[${qi}].id 缺失或不是非空字符串(必填,稳定问题 id,随答案原样回传)`)
474
+ if (typeof q.question !== 'string' || q.question === '') throw new Error(`questions[${qi}].question 缺失或不是非空字符串(必填,用户看到的问题文本;请补上后重新调用 ask_user_choice)`)
475
+ if (q.header !== undefined && typeof q.header !== 'string') throw new Error(`questions[${qi}].header 必须是字符串`)
476
+ if (q.detail !== undefined && typeof q.detail !== 'string') throw new Error(`questions[${qi}].detail 必须是字符串`)
477
+ if (q.multi_select !== undefined && typeof q.multi_select !== 'boolean') throw new Error(`questions[${qi}].multi_select 必须是布尔值`)
478
+ const options = Array.isArray(q.options) ? q.options : []
479
+ for (let oi = 0; oi < options.length; oi++) {
480
+ const o = options[oi]
481
+ if (o === null || typeof o !== 'object') throw new Error(`questions[${qi}].options[${oi}] 必须是对象`)
482
+ if (typeof o.label !== 'string' || o.label === '') throw new Error(`questions[${qi}].options[${oi}].label 缺失或不是非空字符串(必填,选项文字/答案值)`)
483
+ if (o.description !== undefined && typeof o.description !== 'string') throw new Error(`questions[${qi}].options[${oi}].description 必须是字符串`)
484
+ if (o.image !== undefined && o.image !== null && (typeof o.image !== 'object' || o.image === null)) throw new Error(`questions[${qi}].options[${oi}].image 必须是对象(path / url / data 三选一)`)
485
+ }
486
+ }
487
+
372
488
  // 本次 ask 创建的 pickId,无论成功/失败/中止都统一释放(含中途加载报错)。
373
489
  const batches = []
374
490
  try {
375
491
  // 1) 逐题归一化图片,生成带图问题的 pickId 与标记。
376
492
  const askQuestions = []
377
- for (const question of args.questions ?? []) {
493
+ for (const question of questions) {
378
494
  const options = Array.isArray(question.options) ? question.options : []
379
495
  const imageIndexes = []
380
496
  const normalized = []
@@ -392,7 +508,12 @@ export function choiceTool(ctx, opts = {}) {
392
508
  let detail = typeof question.detail === 'string' ? question.detail : ''
393
509
  if (imageIndexes.length > 0) {
394
510
  const pickId = randomUUID()
395
- picks.set(pickId, { createdAt: now, images: normalized.filter((o) => o.image !== undefined).map((o) => o.image) })
511
+ // 按原始选项下标存图片(无图选项留空),保证客户端
512
+ // `/pickId/<原始下标>` 的 URL 与服务端索引一一对应;
513
+ // 紧凑数组会让带图选项下标不连续时错位/404。
514
+ const images = new Array(options.length).fill(undefined)
515
+ for (const index of imageIndexes) images[index] = normalized[index].image
516
+ picks.set(pickId, { createdAt: now, images })
396
517
  batches.push(pickId)
397
518
  detail = buildPickMarker(pickId, imageIndexes) + detail
398
519
  }
@@ -517,14 +638,151 @@ export function showImagesTool(ctx, opts = {}) {
517
638
  // 插件入口
518
639
  // ---------------------------------------------------------------------------
519
640
 
641
+ /**
642
+ * save_received_images 工具定义:把用户发来的图片附件保存为工作区文件。
643
+ * 盲模型(无视觉输入的适配器)经 agent/pre-step 重写把图片块替换为
644
+ * 文本占位符(dshimg:<attachmentId>),模型据此调用本工具把附件字节
645
+ * 落盘,之后可用文件/命令工具分析(尺寸、像素、哈希等)。
646
+ * @param {object} ctx - 插件上下文(execute 闭包使用 ctx.attachments.readImage)。
647
+ * @returns {object} 工具 definition。
648
+ */
649
+ export function saveReceivedImagesTool(ctx) {
650
+ return {
651
+ name: 'save_received_images',
652
+ description:
653
+ '把用户发来的图片附件保存为工作区文件(默认保存到 received/ 目录)。' +
654
+ '当用户消息里出现 "📷 用户发来的图片 dshimg:<attachmentId>" 占位符时,' +
655
+ '说明用户上传了一张图片(当前模型无法直接查看)。如需查看/分析该图片:' +
656
+ '调用本工具,把占位符里的 attachmentId 填进 images 数组,工具会把图片写入' +
657
+ '工作区文件并返回路径;之后可用文件读取/命令工具分析该文件(如尺寸、内容等)。',
658
+ parameters: {
659
+ type: 'object',
660
+ additionalProperties: false,
661
+ required: ['images'],
662
+ properties: {
663
+ images: {
664
+ type: 'array',
665
+ description: '要保存的图片附件列表(attachmentId 来自用户消息里的 dshimg: 占位符)。',
666
+ items: {
667
+ type: 'object',
668
+ additionalProperties: false,
669
+ required: ['attachmentId'],
670
+ properties: {
671
+ attachmentId: {
672
+ type: 'string',
673
+ description: '附件 id(用户消息里 dshimg: 后面的那串)。'
674
+ },
675
+ mediaType: {
676
+ type: 'string',
677
+ description: '可选:image/png|image/jpeg|image/webp|image/gif(占位符未给全时由登记信息补全)。'
678
+ },
679
+ width: { type: 'number', description: '可选:像素宽。' },
680
+ height: { type: 'number', description: '可选:像素高。' },
681
+ bytes: { type: 'number', description: '可选:字节数。' },
682
+ name: { type: 'string', description: '可选:显示名(会被清洗为安全文件名)。' }
683
+ }
684
+ }
685
+ },
686
+ dir: {
687
+ type: 'string',
688
+ description: `可选:保存目录(相对会话工作区,默认 ${RECEIVED_DIR}/)。`
689
+ }
690
+ }
691
+ },
692
+ output: {
693
+ schema: {
694
+ type: 'object',
695
+ additionalProperties: false,
696
+ required: ['saved'],
697
+ properties: {
698
+ saved: {
699
+ type: 'array',
700
+ items: {
701
+ type: 'object',
702
+ additionalProperties: false,
703
+ required: ['attachmentId', 'path'],
704
+ properties: {
705
+ attachmentId: { type: 'string' },
706
+ path: { type: 'string' },
707
+ mediaType: { type: 'string' },
708
+ bytes: { type: 'number' },
709
+ width: { type: 'number' },
710
+ height: { type: 'number' },
711
+ error: { type: 'string' }
712
+ }
713
+ }
714
+ }
715
+ }
716
+ },
717
+ render: (_args, value) => [{ type: 'text', text: JSON.stringify(value) }]
718
+ },
719
+ async execute(args, exec) {
720
+ const cwd = sessionCwd(exec?.agent)
721
+ pruneAttachmentRefs()
722
+ const dirName = typeof args.dir === 'string' && args.dir.trim() !== '' ? args.dir.trim() : RECEIVED_DIR
723
+ const items = Array.isArray(args.images) ? args.images : []
724
+ if (items.length === 0) throw new Error('images 不能为空')
725
+
726
+ const saved = []
727
+ for (let index = 0; index < items.length; index++) {
728
+ const item = items[index]
729
+ const attachmentId = typeof item?.attachmentId === 'string' ? item.attachmentId : ''
730
+ try {
731
+ if (attachmentId === '') throw new Error('attachmentId 缺失')
732
+ // 合并模型传入的字段与登记信息,构造完整 ref(readImage 会校验 mediaType/bytes/尺寸)。
733
+ const known = resolveAttachmentRef(attachmentId)
734
+ const ref = {
735
+ attachmentId,
736
+ mediaType: item.mediaType ?? known?.mediaType,
737
+ bytes: item.bytes ?? known?.bytes,
738
+ width: item.width ?? known?.width,
739
+ height: item.height ?? known?.height,
740
+ ...(item.name !== undefined ? { name: item.name } : known?.name !== undefined ? { name: known.name } : {})
741
+ }
742
+ if (ref.mediaType === undefined || ref.bytes === undefined || ref.width === undefined || ref.height === undefined) {
743
+ throw new Error(`附件 ${attachmentId} 的元数据不完整(可能是过期占位符),请提供 mediaType/width/height/bytes 或重新触发对话`)
744
+ }
745
+ const stored = await ctx.attachments.readImage(ref, exec?.signal)
746
+ const data = Buffer.from(stored.data)
747
+ const targetDir = isAbsolute(dirName) ? dirName : join(cwd, dirName)
748
+ await mkdir(targetDir, { recursive: true })
749
+ const ext = mediaTypeToExt(ref.mediaType)
750
+ const rawName = typeof ref.name === 'string' ? basename(ref.name) : ''
751
+ const safeName = rawName !== '' && !rawName.includes('..') && !rawName.includes('/') && !rawName.includes('\\') && /^[\w.-]+$/.test(rawName)
752
+ ? rawName
753
+ : `image-${index + 1}-${Date.now()}${ext}`
754
+ const filePath = join(targetDir, safeName)
755
+ await writeFile(filePath, data)
756
+ saved.push({
757
+ attachmentId,
758
+ path: filePath,
759
+ mediaType: ref.mediaType,
760
+ bytes: data.byteLength,
761
+ width: ref.width,
762
+ height: ref.height
763
+ })
764
+ } catch (error) {
765
+ saved.push({
766
+ attachmentId,
767
+ path: '',
768
+ error: error instanceof Error ? error.message : String(error)
769
+ })
770
+ }
771
+ }
772
+ return { saved }
773
+ }
774
+ }
775
+ }
776
+
520
777
  export function apply(ctx) {
521
778
  // 图片字节服务路由:同源 <img src> 直接加载。
522
779
  // /dsh-plugin-image-tools/<pickId>/<index> 选择卡图片
523
780
  // /dsh-plugin-image-tools/show/<showId>/<index> 回复内嵌图片
781
+ // /dsh-plugin-image-tools/attachment/<attachmentId> 盲模型收图的附件回显
524
782
  const disposeRoute = ctx.effect(() => ctx.webServer.register({
525
783
  kind: 'prefix',
526
784
  path: ROUTE_PREFIX,
527
- handler: (req, res) => {
785
+ handler: async (req, res) => {
528
786
  try {
529
787
  const method = (req.method ?? 'GET').toUpperCase()
530
788
  if (method !== 'GET') {
@@ -534,6 +792,24 @@ export function apply(ctx) {
534
792
  }
535
793
  const url = new URL(req.url ?? '/', 'http://dsh.internal')
536
794
  const rest = url.pathname.slice(ROUTE_PREFIX.length).replace(/^\/+/, '')
795
+ if (rest.startsWith('attachment/')) {
796
+ const attachmentId = rest.slice('attachment/'.length)
797
+ const ref = attachmentId !== '' ? resolveAttachmentRef(attachmentId) : undefined
798
+ if (ref === undefined) {
799
+ res.writeHead(404, { 'content-type': 'text/plain; charset=utf-8' })
800
+ res.end('not found')
801
+ return
802
+ }
803
+ const stored = await ctx.attachments.readImage(ref, req.signal)
804
+ res.writeHead(200, {
805
+ 'content-type': ref.mediaType,
806
+ 'content-length': stored.data.byteLength,
807
+ 'cache-control': 'private, max-age=300',
808
+ 'x-content-type-options': 'nosniff'
809
+ })
810
+ res.end(Buffer.from(stored.data))
811
+ return
812
+ }
537
813
  let image
538
814
  if (rest.startsWith('show/')) {
539
815
  const [, showId, indexRaw] = rest.split('/')
@@ -567,12 +843,56 @@ export function apply(ctx) {
567
843
 
568
844
  const disposeChoice = ctx.effect(() => ctx.tools.register(choiceTool(ctx)), 'dsh-plugin-image-tools: ask_user_choice tool')
569
845
  const disposeShow = ctx.effect(() => ctx.tools.register(showImagesTool(ctx)), 'dsh-plugin-image-tools: show_images tool')
846
+ const disposeSave = ctx.effect(() => ctx.tools.register(saveReceivedImagesTool(ctx)), 'dsh-plugin-image-tools: save_received_images tool')
847
+
848
+ // 盲模型收图:把进入 LLM 步骤的消息批次里的 image 块重写为文本占位符,
849
+ // 避免文本-only 适配器(如 DeepSeek)对图片内容块直接抛 UNSUPPORTED_CONTENT。
850
+ const disposePreStep = ctx.effect(() => ctx.on('agent/pre-step', async ({ agent, messages, turn, step, signal }, next) => {
851
+ const decision = await next()
852
+ if (decision === null || typeof decision !== 'object' || decision.kind !== 'enter') return decision
853
+ let changed = false
854
+ const rewritten = decision.messages.map((message) => {
855
+ const replaced = rewriteMessageImages(message)
856
+ if (replaced !== null) {
857
+ changed = true
858
+ return replaced
859
+ }
860
+ return message
861
+ })
862
+ return changed ? { kind: 'enter', messages: rewritten } : decision
863
+ }), 'dsh-plugin-image-tools: image-to-text pre-step')
864
+
865
+ // 模型能力补丁:apiproxy 在 prompt 入队前校验 inputModalities,
866
+ // 盲模型(未声明 image 输入)会直接拒绝带图消息(MODEL_DOES_NOT_SUPPORT_IMAGES)。
867
+ // 这里把 resolveModelInfo 的结果补上 image 模态,放行带图消息进入会话;
868
+ // 图片随后由上面的 pre-step 重写为文本占位符,适配器不会真的收到图片块。
869
+ const llmService = ctx.llm
870
+ const originalResolveModelInfo = llmService !== null && llmService !== undefined && typeof llmService.resolveModelInfo === 'function'
871
+ ? llmService.resolveModelInfo
872
+ : null
873
+ if (originalResolveModelInfo !== null) {
874
+ const patchedResolveModelInfo = async (provider, model, signal) => {
875
+ const info = await originalResolveModelInfo.call(llmService, provider, model, signal)
876
+ if (info === null || typeof info !== 'object') return info
877
+ const modalities = info.inputModalities
878
+ if (modalities === undefined || modalities.includes('image')) return info
879
+ return Object.freeze({ ...info, inputModalities: Object.freeze([...modalities, 'image']) })
880
+ }
881
+ llmService.resolveModelInfo = patchedResolveModelInfo
882
+ }
570
883
 
571
884
  ctx.effect(() => () => {
572
885
  disposeRoute()
573
886
  disposeChoice()
574
887
  disposeShow()
888
+ disposeSave()
889
+ disposePreStep()
890
+ // 卸载时还原 llm 能力补丁
891
+ if (originalResolveModelInfo !== null && llmService.resolveModelInfo !== originalResolveModelInfo) {
892
+ llmService.resolveModelInfo = originalResolveModelInfo
893
+ }
575
894
  picks.clear()
576
895
  shows.clear()
896
+ attachmentRefs.clear()
577
897
  })
578
898
  }
package/package.json CHANGED
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "name": "dsh-plugin-image-tools",
3
3
  "description": "DSH 图片插件:ask_user_choice 图片/图文混合选项(Web GUI 渲染图片选择卡,可放大查看)+ show_images 在回复中内嵌图片(图片与文字混排)。图片来源支持本地路径 / http(s) URL / base64 data URI。纯插件实现,不改核心包。",
4
- "version": "0.3.1",
4
+ "version": "0.4.1",
5
5
  "type": "module",
6
6
  "main": "lib/index.js",
7
7
  "exports": {
@@ -15,6 +15,7 @@
15
15
  "docs",
16
16
  "cordis.patch.yml",
17
17
  "README.md",
18
+ "LICENSE",
18
19
  "设计说明.md"
19
20
  ],
20
21
  "scripts": {
@@ -11,6 +11,10 @@ import {
11
11
  loadOptionImage,
12
12
  originOf,
13
13
  safeAlt,
14
+ mediaTypeToExt,
15
+ imagePlaceholderText,
16
+ extractImageTokenIds,
17
+ rewriteMessageImages,
14
18
  MAX_IMAGE_BYTES,
15
19
  } from '../lib/index.js'
16
20
 
@@ -97,6 +101,48 @@ async function main() {
97
101
  assert.equal(safeAlt(undefined), '图片')
98
102
  })
99
103
 
104
+ // --- 盲模型收图:媒体类型/占位符/token 提取/消息重写 ---
105
+ await ok('mediaTypeToExt maps', () => {
106
+ assert.equal(mediaTypeToExt('image/png'), '.png')
107
+ assert.equal(mediaTypeToExt('image/jpeg'), '.jpg')
108
+ assert.equal(mediaTypeToExt('image/webp'), '.webp')
109
+ assert.equal(mediaTypeToExt('image/gif'), '.gif')
110
+ assert.equal(mediaTypeToExt('image/bmp'), '')
111
+ })
112
+ await ok('imagePlaceholderText embeds token', () => {
113
+ const text = imagePlaceholderText({ attachmentId: 'abc123def456', mediaType: 'image/png', width: 10, height: 20 })
114
+ assert.ok(text.includes('dshimg:abc123def456'), '占位符应含 token')
115
+ assert.ok(text.includes('图片'), '占位符应说明是图片')
116
+ })
117
+ await ok('extractImageTokenIds finds ids', () => {
118
+ assert.deepEqual(extractImageTokenIds('📷 图片 dshimg:aaa111bbb222 和 dshimg:ccc333ddd444 和 dshimg:aaa111bbb222'), ['aaa111bbb222', 'ccc333ddd444'])
119
+ assert.deepEqual(extractImageTokenIds('没有 token'), [])
120
+ assert.deepEqual(extractImageTokenIds(undefined), [])
121
+ })
122
+ await ok('rewriteMessageImages replaces image block', () => {
123
+ const ref = { attachmentId: 'abc123def456', mediaType: 'image/png', bytes: 4, width: 10, height: 20, name: 'shot.png' }
124
+ const message = { id: 'm1', role: 'user', source: { kind: 'direct' }, content: [
125
+ { type: 'text', text: '看看这张图' },
126
+ { type: 'image', attachment: ref },
127
+ ] }
128
+ const rewritten = rewriteMessageImages(message)
129
+ assert.ok(rewritten !== null, '应返回重写后的消息')
130
+ assert.equal(rewritten.id, 'm1', 'id 保留')
131
+ assert.equal(rewritten.content.length, 2)
132
+ assert.equal(rewritten.content[0].type, 'text')
133
+ assert.equal(rewritten.content[1].type, 'text', 'image 块应变为 text 块')
134
+ assert.ok(rewritten.content[1].text.includes('dshimg:abc123def456'), '占位符应含 token')
135
+ assert.ok(Object.isFrozen(rewritten), '新消息应冻结')
136
+ // 原消息不被修改
137
+ assert.equal(message.content[1].type, 'image')
138
+ })
139
+ await ok('rewriteMessageImages leaves text-only messages alone', () => {
140
+ const message = { id: 'm2', role: 'user', source: { kind: 'direct' }, content: [{ type: 'text', text: 'hi' }] }
141
+ assert.equal(rewriteMessageImages(message), null)
142
+ assert.equal(rewriteMessageImages(null), null)
143
+ assert.equal(rewriteMessageImages({ id: 'x', content: 'not-array' }), null)
144
+ })
145
+
100
146
  // --- loadOptionImage:data URI / path(url 需要网络,跳过) ---
101
147
  await ok('load data URI', async () => {
102
148
  const pngBytes = Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a])
@@ -50,6 +50,7 @@ assert.equal(typeof mod.selectPickChoice, 'function')
50
50
  assert.equal(typeof mod.ImageChoiceComposer, 'function')
51
51
  assert.equal(typeof mod.Lightbox, 'function')
52
52
  assert.equal(typeof mod.isShowImageSrc, 'function')
53
+ assert.equal(typeof mod.extractImageTokenIds, 'function')
53
54
  assert.equal(typeof mod.startInlineEnhancer, 'function')
54
55
 
55
56
  // 5) 内嵌图片增强纯函数
@@ -59,6 +60,12 @@ assert.ok(!mod.isShowImageSrc('/dsh-plugin-image-tools/abc/0'), '选择卡路由
59
60
  assert.ok(!mod.isShowImageSrc('https://example.com/a.png'), '外部图片不应被识别')
60
61
  assert.equal(mod.startInlineEnhancer(), null, '无 DOM 环境下不应启动观察器')
61
62
 
63
+ // 5b) 盲模型收图占位符 token 提取
64
+ assert.deepEqual(mod.extractImageTokenIds('📷 图片 dshimg:att-aaa111bbb222 和 dshimg:att-ccc333ddd444'), ['att-aaa111bbb222', 'att-ccc333ddd444'])
65
+ assert.deepEqual(mod.extractImageTokenIds('📷 图片 dshimg:att-aaa111bbb222 重复 dshimg:att-aaa111bbb222'), ['att-aaa111bbb222'])
66
+ assert.deepEqual(mod.extractImageTokenIds('没有 token'), [])
67
+ assert.deepEqual(mod.extractImageTokenIds(undefined), [])
68
+
62
69
  // 2) select:认领带标记问题,放过纯文字问题
63
70
  const markerDetail = buildPickMarker('pick-1', [0, 2]) + '\n\n请选择一张图'
64
71
  const withImage = {
@@ -13,6 +13,9 @@
13
13
  * 运行:node scripts/smoke-server.mjs
14
14
  */
15
15
  import { strict as assert } from 'node:assert'
16
+ import { mkdtempSync, readFileSync, existsSync } from 'node:fs'
17
+ import { tmpdir } from 'node:os'
18
+ import { join } from 'node:path'
16
19
  import { apply, ROUTE_PREFIX } from '../lib/index.js'
17
20
 
18
21
  const pngBytes = Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a, 1, 2, 3, 4])
@@ -22,7 +25,9 @@ let toolDefs = []
22
25
  let route = null
23
26
  let capturedAsk = null
24
27
  let resolveAsk = null
28
+ let preStepHandler = null
25
29
  const askPromise = new Promise((resolve) => { resolveAsk = resolve })
30
+ const storedAttachments = new Map([['att-abc123def456', pngBytes]])
26
31
 
27
32
  const ctx = {
28
33
  effect(fn) {
@@ -43,6 +48,22 @@ const ctx = {
43
48
  return askPromise
44
49
  },
45
50
  },
51
+ on(event, handler) {
52
+ if (event === 'agent/pre-step') preStepHandler = handler
53
+ return () => { if (preStepHandler === handler) preStepHandler = null }
54
+ },
55
+ attachments: {
56
+ async readImage(ref) {
57
+ const data = storedAttachments.get(ref.attachmentId)
58
+ if (data === undefined) throw new Error(`attachment not found: ${ref.attachmentId}`)
59
+ return { ref, data }
60
+ },
61
+ },
62
+ llm: {
63
+ async resolveModelInfo(provider, model) {
64
+ return { provider, id: model, name: model, inputModalities: ['text'] }
65
+ },
66
+ },
46
67
  }
47
68
 
48
69
  apply(ctx)
@@ -50,6 +71,7 @@ apply(ctx)
50
71
  const defByName = (name) => toolDefs.find((def) => def.name === name)
51
72
  const choiceDef = defByName('ask_user_choice')
52
73
  const showDef = defByName('show_images')
74
+ const saveDef = defByName('save_received_images')
53
75
 
54
76
  // --- 1) 工具定义 ---
55
77
  assert.ok(choiceDef, 'ask_user_choice 未注册')
@@ -58,6 +80,9 @@ assert.ok(choiceDef.parameters.properties.questions.items.properties.options.ite
58
80
  assert.equal(typeof choiceDef.execute, 'function')
59
81
  assert.ok(showDef, 'show_images 未注册')
60
82
  assert.ok(showDef.parameters.properties.images.items.properties.image, 'show_images schema 缺少 image 字段')
83
+ assert.ok(saveDef, 'save_received_images 未注册')
84
+ assert.ok(saveDef.parameters.properties.images.items.properties.attachmentId, 'save_received_images schema 缺少 attachmentId')
85
+ assert.equal(typeof preStepHandler, 'function', 'agent/pre-step 监听器未注册')
61
86
  assert.equal(typeof showDef.execute, 'function')
62
87
 
63
88
  // --- 2) ask_user_choice 执行:data URI 图片 + 纯文字选项混排 ---
@@ -191,4 +216,71 @@ await assert.rejects(() => showDef.execute({ images: [] }, exec), /不能为空/
191
216
  await assert.rejects(() => showDef.execute({ images: [{ caption: '没有图' }] }, exec), /image 字段/)
192
217
  await assert.rejects(() => showDef.execute({ images: [{ image: {} }] }, exec), /path \/ url \/ data/)
193
218
 
219
+ // --- 6) 盲模型收图:pre-step 重写 → 附件路由 → save_received_images ---
220
+ const imgRef = { attachmentId: 'att-abc123def456', mediaType: 'image/png', bytes: pngBytes.byteLength, width: 12, height: 12, name: 'shot.png' }
221
+ const userMsg = {
222
+ id: 'm-img-1',
223
+ role: 'user',
224
+ source: { kind: 'direct' },
225
+ content: [
226
+ { type: 'text', text: '看看这张截图' },
227
+ { type: 'image', attachment: imgRef },
228
+ ],
229
+ }
230
+ const decision = await preStepHandler(
231
+ { agent: { id: 'a1' }, messages: [userMsg], turn: 1, step: 1, signal: undefined },
232
+ async () => ({ kind: 'enter', messages: [userMsg] }),
233
+ )
234
+ assert.equal(decision.kind, 'enter', 'pre-step 应返回 enter')
235
+ assert.equal(decision.messages.length, 1)
236
+ const rewritten = decision.messages[0]
237
+ assert.equal(rewritten.id, 'm-img-1', '重写后消息 id 保留')
238
+ assert.equal(rewritten.content[1].type, 'text', 'image 块应被重写为 text')
239
+ assert.ok(rewritten.content[1].text.includes('dshimg:att-abc123def456'), '占位符应含 token')
240
+ assert.equal(userMsg.content[1].type, 'image', '原消息不被修改')
241
+
242
+ // 附件路由:重写登记的 ref 可出字节
243
+ let attServed = null
244
+ await route.handler(
245
+ { url: `${ROUTE_PREFIX}/attachment/att-abc123def456` },
246
+ {
247
+ writeHead(status, headers) { attServed = { status, headers } },
248
+ end(body) { attServed.body = body },
249
+ },
250
+ )
251
+ assert.equal(attServed.status, 200)
252
+ assert.equal(attServed.headers['content-type'], 'image/png')
253
+ assert.deepEqual(attServed.body, pngBytes)
254
+ // 未知附件 → 404
255
+ let attNotFound = null
256
+ await route.handler(
257
+ { url: `${ROUTE_PREFIX}/attachment/att-nope000000` },
258
+ { writeHead(status) { attNotFound = { status } }, end() {} },
259
+ )
260
+ assert.equal(attNotFound.status, 404)
261
+
262
+ // save_received_images:保存到临时 cwd
263
+ const tmpCwd = mkdtempSync(join(tmpdir(), 'dsh-img-tools-'))
264
+ const saveExec = { agent: { session: { header: { cwd: tmpCwd } } }, signal: undefined }
265
+ const saveResult = await saveDef.execute({
266
+ images: [{ attachmentId: 'att-abc123def456' }],
267
+ }, saveExec)
268
+ assert.equal(saveResult.saved.length, 1)
269
+ assert.equal(saveResult.saved[0].error, undefined, `保存不应出错:${saveResult.saved[0].error ?? ''}`)
270
+ assert.ok(saveResult.saved[0].path.startsWith(tmpCwd), '应保存到工作区')
271
+ assert.ok(existsSync(saveResult.saved[0].path), '文件应落盘')
272
+ assert.deepEqual(readFileSync(saveResult.saved[0].path), pngBytes, '文件内容应为附件字节')
273
+ assert.equal(saveResult.saved[0].mediaType, 'image/png')
274
+ // 未知附件 → 返回 error 字段而非抛错
275
+ const badSave = await saveDef.execute({ images: [{ attachmentId: 'att-nope000000' }] }, saveExec)
276
+ assert.ok(badSave.saved[0].error, '未知附件应返回 error')
277
+ // 参数校验
278
+ await assert.rejects(() => saveDef.execute({ images: [] }, saveExec), /不能为空/)
279
+
280
+ // --- 7) 模型能力补丁(放行盲模型收图) ---
281
+ const patchedInfo = await ctx.llm.resolveModelInfo('deepseek', 'deepseek-v4-flash')
282
+ assert.ok(Array.isArray(patchedInfo.inputModalities), '补丁后 inputModalities 应为数组')
283
+ assert.ok(patchedInfo.inputModalities.includes('image'), '补丁后应声称支持 image 输入')
284
+ assert.equal(patchedInfo.id, 'deepseek-v4-flash', '其余元数据应保留')
285
+
194
286
  console.log('[dsh-plugin-image-tools] server smoke passed')
@@ -95,15 +95,32 @@
95
95
  - 工具 `show_images`:
96
96
  - schema:`images: [{ image: {path|url|data|mediaType}, caption? }]`(1~9 张);
97
97
  - execute:归一化 → 注册 shows → 返回绝对 URL markdown 片段 + 提示语。
98
+ - 工具 `save_received_images`(盲模型收图 → 文件):
99
+ - schema:`images: [{ attachmentId, mediaType?, width?, height?, bytes?, name? }]` + `dir?`(默认 `received/`);
100
+ - execute:合并模型传入字段与 pre-step 登记的 ref → `ctx.attachments.readImage` 取回
101
+ 附件字节 → 落盘工作区(文件名优先附件安全名,否则 `image-<n>-<时间戳>.<ext>`)→
102
+ 返回路径与元数据;单个失败记入 error 字段不中断。
98
103
  - web 路由 `ROUTE_PREFIX = /dsh-plugin-image-tools`(`ctx.webServer.register`,kind=prefix):
99
104
  - `/dsh-plugin-image-tools/<pickId>/<index>` → 选择卡图片;
100
105
  - `/dsh-plugin-image-tools/show/<showId>/<index>` → 回复内嵌图片;
106
+ - `/dsh-plugin-image-tools/attachment/<attachmentId>` → 盲模型收图的附件回显
107
+ (`ctx.attachments.readImage` 出字节,需要 pre-step 重写时登记过 ref);
101
108
  - bytes + content-type(nosniff + 短缓存),越界/未知 id → 404。
102
109
  - 图片来源归一化(两种工具共用 `loadOptionImage`):
103
110
  - `data`:data URI 解析(声明类型优先,魔数兜底,两者冲突报错);
104
111
  - `url`:`fetch`(30s 超时),content-type 或魔数定类型;
105
112
  - `path`:相对会话 cwd(`agent.session.header.cwd`)解析,`readFile` 读取。
106
113
  - 上限 20 MiB/张;类型限 PNG/JPEG/WebP/GIF(魔数校验)。
114
+ - **盲模型收图链路**(`agent/pre-step` + `llm` 能力补丁):
115
+ - 补丁:保留 `ctx.llm.resolveModelInfo` 原方法,返回结果里把 `inputModalities`
116
+ 补上 `image` —— apiproxy 的 prompt 入队校验(`MODEL_DOES_NOT_SUPPORT_IMAGES`)
117
+ 据此放行带图消息;卸载时还原原方法;
118
+ - `agent/pre-step` waterfall 监听器(与 agent-instructions/time-context 同机制):
119
+ 把进入 LLM 步骤的消息批次里的 `{type:'image', attachment}` 块重写为文本占位符
120
+ `📷 用户发来的图片(name) dshimg:<attachmentId>`,同时把完整 ref 登记进
121
+ `attachmentRefs`(TTL 清理)——会话日志/UI 保持纯文本安全,文本-only 适配器
122
+ 不再抛 `UNSUPPORTED_CONTENT`;
123
+ - 重写是无条件的(插件定位即盲模型收图);使用视觉模型的原生看图需停用本插件。
107
124
 
108
125
  ### 客户端 lib/client.js(浏览器模块加载器格式,免构建)
109
126
 
@@ -125,7 +142,11 @@
125
142
  - `startInlineEnhancer()`:`MutationObserver(document.body, childList+subtree)`
126
143
  + rAF 合帧扫描;无 DOM 环境返回 null(优雅降级);
127
144
  - `openImageZoom(src, caption)`:命令式 Lightbox(复用 `.dshpick-lightbox` 样式),
128
- 带 Esc / 点遮罩 / 关闭按钮。
145
+ 带 Esc / 点遮罩 / 关闭按钮;
146
+ - **盲模型收图回显**:`extractImageTokenIds(text)` 提取 `dshimg:<id>` 占位符;
147
+ `upgradeBubbleTexts()` 用 TreeWalker 扫文本节点,把占位符替换为
148
+ `<img src="/dsh-plugin-image-tools/attachment/<id>">`(可放大、失败降级),
149
+ 用 WeakMap 记录已扫描的 nodeValue(React 原位更新文本时会重新扫描)。
129
150
  - 注册:
130
151
  - `ctx.slots.inject("conversation.composer", ...)`(priority -100,先于原生);
131
152
  - `ctx.effect(() => startInlineEnhancer())`(随插件卸载停止观察器)。
@@ -1,185 +0,0 @@
1
- # dsh-plugin-image-tools 效果图生成脚本(README 展示用)
2
- # 用法:powershell -ExecutionPolicy Bypass -File docs\gen-mockups.ps1
3
- # 输出:docs/mockup-choice.png(图片选择卡)、docs/mockup-inline.png(回复内嵌图片)
4
- Add-Type -AssemblyName System.Drawing
5
-
6
- $docs = Join-Path $PSScriptRoot '.'
7
- $sunset = [System.Drawing.Image]::FromFile((Join-Path $docs '..\..\..\.tmp\pickdemo\sunset.png'))
8
- $stars = [System.Drawing.Image]::FromFile((Join-Path $docs '..\..\..\.tmp\pickdemo\stars.png'))
9
- $whale = [System.Drawing.Image]::FromFile((Join-Path $docs '..\..\..\.tmp\pickdemo\whale.png'))
10
- $font = New-Object System.Drawing.Font('Microsoft YaHei UI', 20, [System.Drawing.FontStyle]::Regular)
11
- $fontBold = New-Object System.Drawing.Font('Microsoft YaHei UI', 22, [System.Drawing.FontStyle]::Bold)
12
- $fontSmall = New-Object System.Drawing.Font('Microsoft YaHei UI', 13, [System.Drawing.FontStyle]::Regular)
13
- $fontTiny = New-Object System.Drawing.Font('Microsoft YaHei UI', 11, [System.Drawing.FontStyle]::Regular)
14
-
15
- $bg = [System.Drawing.Color]::FromArgb(255, 14, 17, 23)
16
- $panel = [System.Drawing.Color]::FromArgb(255, 27, 32, 43)
17
- $border = [System.Drawing.Color]::FromArgb(255, 43, 51, 69)
18
- $thumbBg = [System.Drawing.Color]::FromArgb(255, 35, 41, 54)
19
- $text1 = [System.Drawing.Color]::FromArgb(255, 237, 240, 246)
20
- $text2 = [System.Drawing.Color]::FromArgb(255, 154, 163, 181)
21
- $text3 = [System.Drawing.Color]::FromArgb(255, 107, 116, 136)
22
- $accent = [System.Drawing.Color]::FromArgb(255, 76, 125, 255)
23
- $warnBg = [System.Drawing.Color]::FromArgb(255, 58, 46, 20)
24
- $warnText = [System.Drawing.Color]::FromArgb(255, 232, 163, 61)
25
- $white = [System.Drawing.Color]::FromArgb(255, 255, 255, 255)
26
-
27
- function New-Brush($c) { New-Object System.Drawing.SolidBrush($c) }
28
- function RoundedPath([float]$x, [float]$y, [float]$w, [float]$h, [float]$r) {
29
- $p = New-Object System.Drawing.Drawing2D.GraphicsPath
30
- $d = $r * 2
31
- $p.AddArc($x, $y, $d, $d, 180, 90)
32
- $p.AddArc($x + $w - $d, $y, $d, $d, 270, 90)
33
- $p.AddArc($x + $w - $d, $y + $h - $d, $d, $d, 0, 90)
34
- $p.AddArc($x, $y + $h - $d, $d, $d, 90, 90)
35
- $p.CloseFigure()
36
- return $p
37
- }
38
- function Draw-Rounded([System.Drawing.Graphics]$g, $brush, $rect, [float]$radius) {
39
- $p = RoundedPath $rect.X $rect.Y $rect.Width $rect.Height $radius
40
- $g.FillPath($brush, $p)
41
- $p.Dispose()
42
- }
43
- function Draw-ImageCrop([System.Drawing.Graphics]$g, [System.Drawing.Image]$img, $rect) {
44
- # object-fit: cover 的近似:按目标矩形等比裁剪
45
- $ratio = [Math]::Min($rect.Width / $img.Width, $rect.Height / $img.Height)
46
- $w = $img.Width * $ratio; $h = $img.Height * $ratio
47
- $sx = ($rect.Width - $w) / 2; $sy = ($rect.Height - $h) / 2
48
- $src = New-Object System.Drawing.Rectangle(0, 0, $img.Width, $img.Height)
49
- $dst = New-Object System.Drawing.RectangleF(($rect.X + $sx), ($rect.Y + $sy), $w, $h)
50
- $g.DrawImage($img, $dst, $src, [System.Drawing.GraphicsUnit]::Pixel)
51
- }
52
- function Text-Len($g, $text, $font) { return $g.MeasureString($text, $font).Width }
53
-
54
- # ============================================================
55
- # 效果图 1:图片选择卡(mockup-choice.png)
56
- # 布局对齐修复后的真实卡片:标题/说明/图片网格/选项行统一左缩进 24px
57
- # ============================================================
58
- $W = 1240; $H = 860
59
- $bmp = New-Object System.Drawing.Bitmap($W, $H)
60
- $g = [System.Drawing.Graphics]::FromImage($bmp)
61
- $g.SmoothingMode = [System.Drawing.Drawing2D.SmoothingMode]::AntiAlias
62
- $g.TextRenderingHint = [System.Drawing.Text.TextRenderingHint]::ClearTypeGridFit
63
- $g.Clear($bg)
64
-
65
- # 卡片
66
- $cardX = 120.0; $cardY = 60.0; $cardW = 1000.0; $cardH = 720.0
67
- $inner = $cardX + 24.0 # 统一 24px 左缩进
68
- $right = $cardX + $cardW - 24.0 # 右侧对称
69
- Draw-Rounded $g (New-Brush $panel) (New-Object System.Drawing.RectangleF($cardX, $cardY, $cardW, $cardH)) 24
70
- $g.DrawPath((New-Object System.Drawing.Pen($border, 1.5)), (RoundedPath $cardX $cardY $cardW $cardH 24))
71
-
72
- # 头部:eyebrow + 标题(24px)
73
- $g.DrawString('封面选择', $fontTiny, (New-Brush $text3), $inner, 92.0)
74
- $g.DrawString('选一张封面图', $fontBold, (New-Brush $text1), $inner, 114.0)
75
-
76
- # 说明文字(与标题左对齐,24px)
77
- $g.DrawString('三张候选图,点击卡片或放大镜可查看大图。', $fontTiny, (New-Brush $text2), $inner, 158.0)
78
-
79
- # 图片卡片(3 张,网格左缩进 24px;列宽按可用宽度均分)
80
- $thumbW = 298.0; $thumbH = 298.0; $gap = 29.0
81
- $x0 = $inner; $y0 = 210.0
82
- $imgs = @($sunset, $stars, $whale)
83
- $labels = @('深海鲸鱼', '星空', '手绘风')
84
- for ($i = 0; $i -lt 3; $i++) {
85
- $tx = $x0 + $i * ($thumbW + $gap)
86
- $tRect = New-Object System.Drawing.RectangleF($tx, $y0, $thumbW, $thumbH)
87
- # 选中态:第一张带 accent 边框
88
- if ($i -eq 0) {
89
- $selPen = New-Object System.Drawing.Pen($accent, 3)
90
- $g.DrawPath($selPen, (RoundedPath $tx $y0 $thumbW $thumbH 16))
91
- $selPen.Dispose()
92
- }
93
- Draw-Rounded $g (New-Brush $thumbBg) $tRect 16
94
- # 图(内缩 2px 让边框可见)
95
- $inner2 = New-Object System.Drawing.RectangleF(($tx + 2), ($y0 + 2), ($thumbW - 4), ($thumbH - 4))
96
- $clip = New-Object System.Drawing.Region((RoundedPath ($inner2.X) ($inner2.Y) ($inner2.Width) ($inner2.Height) 14))
97
- $g.SetClip($clip, [System.Drawing.Drawing2D.CombineMode]::Replace)
98
- Draw-ImageCrop $g $imgs[$i] $inner2
99
- $g.ResetClip(); $clip.Dispose()
100
- # 放大镜提示(第一张)
101
- if ($i -eq 0) {
102
- $hintRect = New-Object System.Drawing.RectangleF(($tx + 10), ($y0 + 10), 30, 30)
103
- $g.FillEllipse((New-Brush ([System.Drawing.Color]::FromArgb(140, 145, 160, 180))), $hintRect)
104
- $g.DrawEllipse((New-Object System.Drawing.Pen($white, 2)), ($tx + 17), ($y0 + 17), 10, 10)
105
- $g.DrawLine((New-Object System.Drawing.Pen($white, 2.5)), ($tx + 25), ($y0 + 25), ($tx + 30), ($y0 + 30))
106
- }
107
- # 选中对勾(第一张)
108
- if ($i -eq 0) {
109
- $checkRect = New-Object System.Drawing.RectangleF(($tx + $thumbW - 40), ($y0 + 10), 24, 24)
110
- $g.FillEllipse((New-Brush $accent), $checkRect)
111
- $g.DrawString('✓', (New-Object System.Drawing.Font('Microsoft YaHei UI', 15, [System.Drawing.FontStyle]::Bold)), (New-Brush $white), ($tx + $thumbW - 33.5), ($y0 + 9))
112
- }
113
- # label + 推荐徽标(24px 对齐)
114
- $ly = $y0 + $thumbH + 14
115
- $g.DrawString($labels[$i], $font, (New-Brush $text1), $tx, $ly)
116
- if ($i -eq 0) {
117
- $badgeX = $tx + (Text-Len $g $labels[$i] $font) + 12
118
- $badgeRect = New-Object System.Drawing.RectangleF($badgeX, ($ly + 3), 48, 22)
119
- Draw-Rounded $g (New-Brush $warnBg) $badgeRect 11
120
- $g.DrawString('推荐', $fontTiny, (New-Brush $warnText), ($badgeX + 11), ($ly + 2))
121
- }
122
- $g.DrawString('点击可放大查看', $fontTiny, (New-Brush $text3), $tx, ($ly + 34))
123
- }
124
-
125
- # 页脚:进度 + 按钮(右侧对称 24px)
126
- $g.DrawString('1 / 1', $font, (New-Brush $text2), $inner, 700.0)
127
- $skipRect = New-Object System.Drawing.RectangleF(($right - 86 - 12 - 86), 686.0, 86, 36)
128
- $g.DrawPath((New-Object System.Drawing.Pen($border, 1.5)), (RoundedPath $skipRect.X $skipRect.Y $skipRect.Width $skipRect.Height 18))
129
- $g.DrawString('跳过', $fontSmall, (New-Brush $text1), ($skipRect.X + 23), 692.0)
130
- $submitRect = New-Object System.Drawing.RectangleF(($right - 86), 686.0, 86, 36)
131
- Draw-Rounded $g (New-Brush $accent) $submitRect 18
132
- $g.DrawString('提交', $fontSmall, (New-Brush $white), ($submitRect.X + 28), 692.0)
133
-
134
- $g.Dispose()
135
- $bmp.Save((Join-Path $docs 'mockup-choice.png'), [System.Drawing.Imaging.ImageFormat]::Png)
136
- $bmp.Dispose()
137
- Write-Host 'mockup-choice.png saved'
138
-
139
- # ============================================================
140
- # 效果图 2:回复内嵌图片(mockup-inline.png)
141
- # ============================================================
142
- $W2 = 1240; $H2 = 900
143
- $bmp2 = New-Object System.Drawing.Bitmap($W2, $H2)
144
- $g2 = [System.Drawing.Graphics]::FromImage($bmp2)
145
- $g2.SmoothingMode = [System.Drawing.Drawing2D.SmoothingMode]::AntiAlias
146
- $g2.TextRenderingHint = [System.Drawing.Text.TextRenderingHint]::ClearTypeGridFit
147
- $g2.Clear($bg)
148
-
149
- # 助手身份行
150
- $g2.FillEllipse((New-Brush $accent), 120.0, 84.0, 40, 40)
151
- $g2.DrawString('D', (New-Object System.Drawing.Font('Microsoft YaHei UI', 20, [System.Drawing.FontStyle]::Bold)), (New-Brush $white), 131.0, 88.0)
152
- $g2.DrawString('DeepSeek Harness', $fontSmall, (New-Brush $text1), 172.0, 90.0)
153
- $g2.DrawString('刚刚', $fontTiny, (New-Brush $text3), 340.0, 96.0)
154
-
155
- # 消息正文(markdown 风格)
156
- $g2.DrawString('这是为你生成的两张候选图,点图片可以放大查看:', $font, (New-Brush $text1), 122.0, 152.0)
157
-
158
- # 图片(限宽 560,圆角)
159
- $imgW = 560.0; $imgH = 360.0; $imgX = 122.0; $imgY = 214.0; $imgGap = 34.0
160
- for ($i = 0; $i -lt 2; $i++) {
161
- $iy = $imgY + $i * ($imgH + $imgGap)
162
- $iRect = New-Object System.Drawing.RectangleF($imgX, $iy, $imgW, $imgH)
163
- Draw-Rounded $g2 (New-Brush $thumbBg) $iRect 16
164
- $g2.DrawPath((New-Object System.Drawing.Pen($border, 1.5)), (RoundedPath $imgX $iy $imgW $imgH 16))
165
- $clip2 = New-Object System.Drawing.Region((RoundedPath $imgX $iy $imgW $imgH 16))
166
- $g2.SetClip($clip2, [System.Drawing.Drawing2D.CombineMode]::Replace)
167
- Draw-ImageCrop $g2 $imgs[$i] $iRect
168
- $g2.ResetClip(); $clip2.Dispose()
169
- # caption(alt 说明,悬浮可见的语义)
170
- $capY = $iy + $imgH + 10
171
- $g2.DrawString(('图片说明:' + $labels[$i]), $fontTiny, (New-Brush $text3), $imgX, $capY)
172
- }
173
-
174
- # 下方继续文字
175
- $g2.DrawString('需要调整配色或构图,或者换成横版,随时告诉我。', $font, (New-Brush $text1), 122.0, 664.0)
176
-
177
- # 用户消息气泡
178
- $bubble = New-Object System.Drawing.RectangleF(760.0, 764.0, 360, 52)
179
- Draw-Rounded $g2 (New-Brush ([System.Drawing.Color]::FromArgb(255, 59, 98, 214))) $bubble 26
180
- $g2.DrawString('选第 1 张,谢谢!', $fontSmall, (New-Brush $white), 794.0, 780.0)
181
-
182
- $g2.Dispose()
183
- $bmp2.Save((Join-Path $docs 'mockup-inline.png'), [System.Drawing.Imaging.ImageFormat]::Png)
184
- $bmp2.Dispose()
185
- Write-Host 'mockup-inline.png saved'