opencode-vision-analyze 0.4.0 → 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +43 -16
- package/README.zh.md +42 -16
- package/dist/index.d.ts +31 -0
- package/dist/index.js +112 -19
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -13,9 +13,9 @@ A tool-based vision routing plugin for [opencode](https://opencode.ai): when the
|
|
|
13
13
|
## Features
|
|
14
14
|
|
|
15
15
|
- **Tool-based, not pre-analysis.** The turn starts immediately; the model decides when (and with which question) to look. No blocking on submit, failures are visible and retryable inside the agent loop. Same philosophy as production-proven agent designs.
|
|
16
|
-
- **Question-aware descriptions.** The model passes its own focused question to `vision_analyze` — not a one-shot generic caption computed at submit time.
|
|
16
|
+
- **Question-aware descriptions.** The model passes its own focused question to `vision_analyze` — not a one-shot generic caption computed at submit time. For a general parse of an image the model leaves `question` empty, and the tool falls back to a fixed prompt so all generic descriptions of the same image share one cache entry (see *Content-addressed cache*).
|
|
17
17
|
- **Native fast path.** If the main model is vision-capable, `vision_analyze` skips the vision model entirely and returns the raw image as a tool attachment.
|
|
18
|
-
- **Content-addressed cache.** Images are stored
|
|
18
|
+
- **Content-addressed cache.** Images and descriptions are stored by content hash in user-level shared dirs and reused across sessions, projects and restarts — the same image is never described twice, and all generic full-image parses collapse onto that image's single entry. Storage layout, size limits and the canonical prompt are detailed in *Storage and caches* below.
|
|
19
19
|
- **Unified auth.** The vision call runs through an opencode sub-session, so it reuses the provider credentials opencode already manages. No extra API key plumbing.
|
|
20
20
|
|
|
21
21
|
## Installation
|
|
@@ -69,10 +69,6 @@ Notes for the curl path:
|
|
|
69
69
|
|
|
70
70
|
Supported image extensions: png / jpg / jpeg / gif / webp.
|
|
71
71
|
|
|
72
|
-
Image storage: stored in a **user-level shared directory** `<cache>/opencode-vision-analyze/vision` regardless of git scope — one content-addressed `<sha256>.<ext>` file per unique image, shared across all projects. Pasted images and `http(s)` downloads are written here (atomic temp-file + rename, so concurrent opencode processes can safely share the store); **already-local image paths passed straight to the tool are read in place and never copied**. Defaults per platform: Linux `$XDG_CACHE_HOME || ~/.cache`, macOS `~/Library/Caches` (a `$XDG_CACHE_HOME` override is honored), Windows `%LOCALAPPDATA% || ~/AppData/Local`. An empty cache-root env var is treated as unset (falls back to the default). The store is LRU-capped (2000 entries / 500 MB; oldest by file mtime is evicted when either limit is exceeded), so no `.gitignore` entry is needed anywhere.
|
|
73
|
-
|
|
74
|
-
Description cache: stored in a **user-level shared directory** `<cache>/opencode-vision-analyze/descriptions` regardless of git scope — one JSON entry per `<image-sha>:<question>` key, named by `sha256(key)`. It is capped at 2000 entries / 50 MB with LRU eviction (oldest by file mtime is removed when either limit is exceeded). Writes are atomic (temp file + rename), so concurrent opencode processes can safely share the cache.
|
|
75
|
-
|
|
76
72
|
An ordered-candidates example with auto-fallback and free-first discovery:
|
|
77
73
|
|
|
78
74
|
```jsonc
|
|
@@ -91,6 +87,37 @@ An ordered-candidates example with auto-fallback and free-first discovery:
|
|
|
91
87
|
}
|
|
92
88
|
```
|
|
93
89
|
|
|
90
|
+
### Storage and caches
|
|
91
|
+
|
|
92
|
+
Two **user-level shared caches** live side by side under `<cache>/opencode-vision-analyze/`, shared across sessions, projects and restarts, independent of git scope — no `.gitignore` entry is needed anywhere.
|
|
93
|
+
|
|
94
|
+
| Cache | Directory | Content | Naming | Cap |
|
|
95
|
+
|---|---|---|---|---|
|
|
96
|
+
| Images | `vision/` | image bytes | `<sha256>.<ext>` | 2000 entries / 500 MB |
|
|
97
|
+
| Descriptions | `descriptions/` | description text (JSON) | `sha256(key)` | 2000 entries / 50 MB |
|
|
98
|
+
|
|
99
|
+
**Image storage (`vision/`)**
|
|
100
|
+
|
|
101
|
+
- **Written to cache**: clipboard images (raw pixels, no source path) and `http(s)` downloads.
|
|
102
|
+
- **Read in place (never copied)**: path-pasted attachments (the message part carries a real `source.path`, e.g. a file path copied to the clipboard) and already-local paths passed straight to the tool — the file is re-read at analysis time, so re-pasting a path always analyses the latest content.
|
|
103
|
+
- **Eviction**: LRU by file mtime; when either 2000 entries or 500 MB is exceeded, the oldest entries are removed.
|
|
104
|
+
- **Concurrency**: atomic temp-file + rename, so concurrent opencode processes can safely share the store.
|
|
105
|
+
|
|
106
|
+
**Description cache (`descriptions/`)**
|
|
107
|
+
|
|
108
|
+
- **Key**: `<image-sha>:<effective-question>`, filename `sha256(key)`.
|
|
109
|
+
- **General full-image parse** (empty / omitted `question`): normalised to the fixed prompt `Describe this image in full detail, including all text, UI elements, diagrams, or content visible.`, so every generic parse collapses onto one entry.
|
|
110
|
+
- **Specific follow-ups**: keep their own `<image-sha>:<question>` keys (format unchanged, existing entries keep hitting).
|
|
111
|
+
- **Write threshold**: generic entries are only written when the description is ≥ 100 chars, so a short refuse/fail answer can't poison the shared full-image entry.
|
|
112
|
+
- **Eviction**: LRU by file mtime, capped at 2000 entries / 50 MB; **concurrency** as above.
|
|
113
|
+
|
|
114
|
+
**Default cache root per platform**
|
|
115
|
+
|
|
116
|
+
- Linux: `$XDG_CACHE_HOME || ~/.cache`
|
|
117
|
+
- macOS: `~/Library/Caches` (a `$XDG_CACHE_HOME` override is honored)
|
|
118
|
+
- Windows: `%LOCALAPPDATA% || ~/AppData/Local`
|
|
119
|
+
- An empty cache-root env var is treated as unset (falls back to the default).
|
|
120
|
+
|
|
94
121
|
## How it works
|
|
95
122
|
|
|
96
123
|
```
|
|
@@ -100,9 +127,11 @@ User pastes image + question
|
|
|
100
127
|
├─ main model has image input capability → do nothing (raw image goes to model)
|
|
101
128
|
├─ no image-capable model available → do nothing (no hint, no persist;
|
|
102
129
|
│ core's default image handling applies)
|
|
103
|
-
└─ text-only main model →
|
|
104
|
-
(
|
|
105
|
-
|
|
130
|
+
└─ text-only main model → resolve a stable image path:
|
|
131
|
+
path-pasted attachments use the source path in place (never copied,
|
|
132
|
+
always the latest file content); clipboard images are persisted to
|
|
133
|
+
the user-level vision store (<sha256>.<ext> under
|
|
134
|
+
<cache>/opencode-vision-analyze/vision)
|
|
106
135
|
and inject a synthetic hint (hidden in TUI, visible to model):
|
|
107
136
|
"use the vision_analyze tool with image_path: ..."
|
|
108
137
|
|
|
@@ -114,9 +143,10 @@ vision_analyze tool:
|
|
|
114
143
|
├─ native fast path: session's main model is vision-capable
|
|
115
144
|
│ → return raw image as attachment (no vision model call)
|
|
116
145
|
├─ http(s) image URL → download (20 MB cap) → same disk path
|
|
117
|
-
├─ description cache hit (sha + question) → return cached text
|
|
118
|
-
│ (
|
|
119
|
-
│
|
|
146
|
+
├─ description cache hit (image sha + effective question) → return cached text
|
|
147
|
+
│ (general parses converge on <sha>:<canonical full-detail prompt>;
|
|
148
|
+
│ persisted under <cache>/opencode-vision-analyze/descriptions,
|
|
149
|
+
│ tagged with the model that produced it)
|
|
120
150
|
└─ candidate chain: sub-session under current session per candidate, in order —
|
|
121
151
|
parentID, all tools disabled, dedicated system prompt, image + question
|
|
122
152
|
sent to that vision model → first success returns → sub-session deleted
|
|
@@ -131,10 +161,7 @@ Key behaviors:
|
|
|
131
161
|
- **Loginless free models** — auto-discovery uses `config.providers()`, the same source as the `/models` picker, so image-capable zen free models are found even without login (their provider is `custom` source → default last tier; put them first with `free_first: true`).
|
|
132
162
|
- **The tool never throws** — every failure returns readable text so the agent loop can retry, rephrase, or inform the user.
|
|
133
163
|
- **URL images** — `image_path` accepts `http(s)://...` URLs (must end in a supported image extension: png/jpg/jpeg/gif/webp).
|
|
134
|
-
|
|
135
|
-
## Known limitations
|
|
136
|
-
|
|
137
|
-
- **V1 session flow only** — hooks are attached to the V1 `SessionPrompt` path; if opencode's default interaction moves to the V2 session core, hooks won't fire (silently).
|
|
164
|
+
- **General parses share one cache entry** — an empty or omitted `question` is treated as a full-image parse and reuses that image's cached description; specific follow-ups keep their own entries (details in *Storage and caches*).
|
|
138
165
|
|
|
139
166
|
## Roadmap
|
|
140
167
|
|
package/README.zh.md
CHANGED
|
@@ -13,9 +13,9 @@
|
|
|
13
13
|
## 特性
|
|
14
14
|
|
|
15
15
|
- **工具化,而非提交时预分析。** 轮次即时启动;模型自己决定何时看图、带着什么问题看。提交零阻塞,失败在 agent 循环里可见、可重试。
|
|
16
|
-
- **描述针对问题。** 模型把自己关注的问题传给 `vision_analyze
|
|
16
|
+
- **描述针对问题。** 模型把自己关注的问题传给 `vision_analyze`——而不是提交时预生成的一次性通用描述。若只是让描述整张图,模型留空 `question`,工具用固定文案兜底,使同一张图的泛描述共享一条缓存(见"内容寻址缓存")。
|
|
17
17
|
- **原生快速路径。** 主模型本身有视觉能力时,`vision_analyze` 完全跳过视觉模型,直接把原图作为工具附件返回。
|
|
18
|
-
- **内容寻址缓存。**
|
|
18
|
+
- **内容寻址缓存。** 图片与描述按内容哈希落到用户级共享目录,跨会话 / 项目 / 重启复用——同一张图不会重复付费描述;同一张图的所有泛解析请求都收敛到同一条缓存。存储路径、容量上限与固定文案详见下文「存储与缓存」。
|
|
19
19
|
- **统一鉴权。** 视觉调用走 opencode 子会话,复用 opencode 已管理的 provider 凭据,无需额外配置 API Key。
|
|
20
20
|
|
|
21
21
|
## 安装
|
|
@@ -69,10 +69,6 @@ curl 方式说明:
|
|
|
69
69
|
|
|
70
70
|
受支持的图片扩展名:png / jpg / jpeg / gif / webp。
|
|
71
71
|
|
|
72
|
-
图片存储:**恒定放用户级共享目录** `<cache>/opencode-vision-analyze/vision`(与 git / 非 git 分域无关),同一用户所有项目共享;每个唯一图片一个内容寻址 `<sha256>.<ext>` 文件。贴图与 http(s) 下载写入于此(临时文件 + rename 原子写,多个 opencode 进程可安全共享);**模型直接把本地已存在文件路径传给工具时不复制、原位读用**。各平台默认:Linux `$XDG_CACHE_HOME || ~/.cache`;macOS `~/Library/Caches`(亦接受 `$XDG_CACHE_HOME` 覆盖);Windows `%LOCALAPPDATA% || ~/AppData/Local`。缓存根 env 为空串视为未设置(回退默认)。图片缓存 LRU 上限 2000 条 / 500MB,任一超限即按文件 mtime 淘汰最久未用的条目——无需任何 `.gitignore` 条目。
|
|
73
|
-
|
|
74
|
-
描述缓存:**恒定放用户级共享目录** `<cache>/opencode-vision-analyze/descriptions`(与 git / 非 git 分域无关)——每条目一个 JSON 文件,key 为 `<图片sha256>:<问题>`、文件名取 `sha256(key)`。容量上限 2000 条 / 50MB,任一超限即按文件 mtime 淘汰最久未用的条目(LRU)。写入为原子操作(临时文件 + rename),多个 opencode 进程可安全共享同一描述缓存。
|
|
75
|
-
|
|
76
72
|
有序候选 + 自动续接 + 免费优先的配置示例:
|
|
77
73
|
|
|
78
74
|
```jsonc
|
|
@@ -91,6 +87,37 @@ curl 方式说明:
|
|
|
91
87
|
}
|
|
92
88
|
```
|
|
93
89
|
|
|
90
|
+
### 存储与缓存
|
|
91
|
+
|
|
92
|
+
两套**用户级共享缓存**并列在 `<cache>/opencode-vision-analyze/` 下,跨会话 / 项目 / 重启共享,与 git 分域无关,无需任何 `.gitignore`。
|
|
93
|
+
|
|
94
|
+
| 缓存 | 目录 | 内容 | 命名 | 容量上限 |
|
|
95
|
+
|---|---|---|---|---|
|
|
96
|
+
| 图片 | `vision/` | 图片字节 | `<sha256>.<ext>` | 2000 条 / 500MB |
|
|
97
|
+
| 描述 | `descriptions/` | 描述文本(JSON) | `sha256(key)` | 2000 条 / 50MB |
|
|
98
|
+
|
|
99
|
+
**图片存储(`vision/`)**
|
|
100
|
+
|
|
101
|
+
- **写入缓存**:剪贴板位图(无源路径)与 `http(s)` 下载。
|
|
102
|
+
- **原位读用(不复制)**:路径粘贴(消息 part 带真实 `source.path`,如复制到剪贴板的文件路径)与模型直接传入的本地文件路径——分析时当场重读,故同一路径每次粘贴都取最新内容。
|
|
103
|
+
- **淘汰**:按文件 mtime LRU;超出 2000 条或 500MB 即删最久未用的条目。
|
|
104
|
+
- **并发安全**:临时文件 + rename 原子写,多个 opencode 进程可安全共享。
|
|
105
|
+
|
|
106
|
+
**描述缓存(`descriptions/`)**
|
|
107
|
+
|
|
108
|
+
- **key**:`<图片sha256>:<生效问题>`,文件名取 `sha256(key)`。
|
|
109
|
+
- **泛解析(`question` 空 / 省略)**:归一化到固定文案 `Describe this image in full detail, including all text, UI elements, diagrams, or content visible.`,所有泛解析收敛到同一条。
|
|
110
|
+
- **具体追问**:保留原问句,走各自的 `<图片sha256>:<问题>` key(格式与旧版一致,存量条目照常命中)。
|
|
111
|
+
- **写入门槛**:泛解析条目要求描述 ≥ 100 字符,防止视觉模型敷衍/拒答的短文本毒化共享条目。
|
|
112
|
+
- **淘汰**:按文件 mtime LRU,上限 2000 条 / 50MB;**并发安全**同上。
|
|
113
|
+
|
|
114
|
+
**平台默认缓存根**
|
|
115
|
+
|
|
116
|
+
- Linux:`$XDG_CACHE_HOME || ~/.cache`
|
|
117
|
+
- macOS:`~/Library/Caches`(亦接受 `$XDG_CACHE_HOME` 覆盖)
|
|
118
|
+
- Windows:`%LOCALAPPDATA% || ~/AppData/Local`
|
|
119
|
+
- 空串 env 视为未设置,回退默认。
|
|
120
|
+
|
|
94
121
|
## 工作原理
|
|
95
122
|
|
|
96
123
|
```
|
|
@@ -100,9 +127,10 @@ curl 方式说明:
|
|
|
100
127
|
├─ 主模型支持图片输入 → 不做任何处理(原图直发)
|
|
101
128
|
├─ 无任何 image-capable 模型 → 不做处理(不注入 hint、不落盘;
|
|
102
129
|
│ 交给核心对图片的默认处理)
|
|
103
|
-
└─ 纯文本主模型 →
|
|
104
|
-
|
|
105
|
-
|
|
130
|
+
└─ 纯文本主模型 → 解析出稳定 image_path:
|
|
131
|
+
路径粘贴用源文件路径原位读用(不复制、始终取最新内容);
|
|
132
|
+
剪贴板位图落盘到用户级 vision 存储
|
|
133
|
+
(<sha256>.<ext>;位于 <cache>/opencode-vision-analyze/vision)
|
|
106
134
|
并注入 synthetic 提示(TUI 隐藏、模型可见):
|
|
107
135
|
"用 vision_analyze 工具查看,image_path: ..."
|
|
108
136
|
|
|
@@ -114,9 +142,10 @@ vision_analyze 工具:
|
|
|
114
142
|
├─ 原生快速路径:会话主模型有视觉能力
|
|
115
143
|
│ → 原图作为附件直接返回(不调视觉模型)
|
|
116
144
|
├─ http(s) 图片 URL → 下载(20 MB 上限)→ 统一磁盘路径
|
|
117
|
-
├─ 描述缓存命中(图片哈希 +
|
|
118
|
-
│
|
|
119
|
-
│
|
|
145
|
+
├─ 描述缓存命中(图片哈希 + 生效问题)→ 直接返回缓存文本
|
|
146
|
+
│ (泛解析收敛到 <sha>:<固定整图描述文案>;
|
|
147
|
+
│ 持久化于 <cache>/opencode-vision-analyze/descriptions,
|
|
148
|
+
│ 标签沿用产出该描述的模型)
|
|
120
149
|
└─ 候选链:沿链逐候选建子会话(parentID 挂当前会话、禁用全部工具、
|
|
121
150
|
专用 system prompt,图片 + 问题发给该候选视觉模型)
|
|
122
151
|
→ 首个成功即返回 → 子会话删除
|
|
@@ -131,10 +160,7 @@ vision_analyze 工具:
|
|
|
131
160
|
- **免登录免费模型** —— 自动发现与 `/models` 选择器同源(`config.providers()`),免登录也可发现的 zen free 视觉模型会进入候选链(其 provider 为 `custom` 源 → 默认最末档;`free_first: true` 可提到最前)。
|
|
132
161
|
- **工具永不抛错** —— 所有失败都返回可读文字,agent 循环可以重试、换问题或告知用户。
|
|
133
162
|
- **URL 图片** —— `image_path` 接受 `http(s)://...` 地址(需以受支持的图片扩展名结尾:png/jpg/jpeg/gif/webp)。
|
|
134
|
-
|
|
135
|
-
## 已知限制
|
|
136
|
-
|
|
137
|
-
- **仅 V1 会话流** —— 钩子挂在 V1 `SessionPrompt` 路径上;若 opencode 默认交互切到 V2 会话核心,钩子不会触发(且不报错)。
|
|
163
|
+
- **泛解析共享同一条缓存** —— 空 / 省略的 `question` 视为整图描述,复用该图已缓存的描述;具体追问各自保留条目(细节见「存储与缓存」)。
|
|
138
164
|
|
|
139
165
|
## Roadmap
|
|
140
166
|
|
package/dist/index.d.ts
CHANGED
|
@@ -48,6 +48,37 @@ export declare const visionCacheLimits: {
|
|
|
48
48
|
maxEntries: number;
|
|
49
49
|
maxBytes: number;
|
|
50
50
|
};
|
|
51
|
+
/**
|
|
52
|
+
* 泛解析(描述整张图)用的固定问题文案。
|
|
53
|
+
*
|
|
54
|
+
* 主模型对「解析这张图 / 描述图片」这类请求应省略 question,由工具用这里的文案兜底,
|
|
55
|
+
* 从而让所有泛解析请求的缓存 key 收敛到同一条(key = <图片sha>:GENERIC_QUESTION),
|
|
56
|
+
* 避免主模型每次自编措辞把缓存拆成多份、跨会话永远命中不了。文案点名文字/UI/图表/
|
|
57
|
+
* 可见内容,驱动视觉模型把整图信息尽量带全。
|
|
58
|
+
*
|
|
59
|
+
* 这份文案相当于公开契约:改动它会让旧的泛解析缓存条目失去命中(由 LRU 按 mtime 清理),
|
|
60
|
+
* 属于刻意为之——将来想改泛描述措辞、或按用户/场景定制时,改文案即可自然分开新旧缓存。
|
|
61
|
+
*/
|
|
62
|
+
export declare const GENERIC_QUESTION = "Describe this image in full detail, including all text, UI elements, diagrams, or content visible.";
|
|
63
|
+
/**
|
|
64
|
+
* 泛解析缓存条目的最短文本长度(字符)。
|
|
65
|
+
*
|
|
66
|
+
* canonical 描述要求整图加逐字转录,正常结果不可能太短;太短多半是视觉模型敷衍或拒答。
|
|
67
|
+
* 泛解析条目是全图共享的,存进一条垃圾描述会让之后所有泛解析都命中它,所以写入时长度
|
|
68
|
+
* 低于该值就跳过。具体追问不受限制(“答案是 42”是合法答案)。
|
|
69
|
+
* 做成模块级可变对象,测试注入大值/小值验证两侧行为。
|
|
70
|
+
*/
|
|
71
|
+
export declare const genericWriteMinText: {
|
|
72
|
+
chars: number;
|
|
73
|
+
};
|
|
74
|
+
/**
|
|
75
|
+
* 归一化问题文本,供「是否为泛解析」判定使用:
|
|
76
|
+
* 去掉首尾空白与引号/括号,把连续空白(含全角/换行)折成一个空格,去掉句末标点,统一小写。
|
|
77
|
+
* 只做字符级归一化,不做语义匹配——误判泛解析会把针对性回答套到错误语义上,宁可 miss。
|
|
78
|
+
*/
|
|
79
|
+
export declare function normalizeQuestion(q: string): string;
|
|
80
|
+
/** 是否泛解析请求:没给问题(空/纯空白),或归一化后与 GENERIC_QUESTION 一致。 */
|
|
81
|
+
export declare function isGenericQuestion(q: string): boolean;
|
|
51
82
|
declare const _default: {
|
|
52
83
|
id: string;
|
|
53
84
|
server: Plugin;
|
package/dist/index.js
CHANGED
|
@@ -2,22 +2,28 @@
|
|
|
2
2
|
* opencode-vision-analyze
|
|
3
3
|
*
|
|
4
4
|
* 为「不具备视觉能力的主模型」提供图片解读路由:当用户在消息中附带图片时,
|
|
5
|
-
*
|
|
6
|
-
*
|
|
7
|
-
*
|
|
8
|
-
*
|
|
5
|
+
* 插件解析出一个稳定的图片绝对路径(**路径粘贴**——part 带真实 source.path——
|
|
6
|
+
* 直接原位读用、不复制;**复制图片本身**(剪贴板位图)等无有效源路径时才落盘到
|
|
7
|
+
* 用户级共享目录 <cache>/opencode-vision-analyze/vision,恒用户级、与 git/非 git
|
|
8
|
+
* 无关),并向模型注入一条 synthetic 提示(TUI 界面隐藏、模型可见),引导它通过
|
|
9
|
+
* vision_analyze 工具让指定的视觉模型描述图片。若主模型本身支持图片输入,则不做
|
|
10
|
+
* 任何干预,原图直接发给主模型。
|
|
9
11
|
*
|
|
10
12
|
* 工作方式(vision_analyze 工具路径):主模型调用 vision_analyze 时,插件
|
|
11
13
|
* 创建一个 parentID 挂在当前会话下的临时子会话(不进会话列表、不生成
|
|
12
14
|
* 标题、禁用全部工具),把原图以 data URL 发给视觉模型,取回描述文字后
|
|
13
|
-
*
|
|
14
|
-
*
|
|
15
|
-
*
|
|
16
|
-
*
|
|
15
|
+
* 删除子会话并返回描述。描述按「<图片sha256>:<问题>」键缓存到**用户级共享目录**
|
|
16
|
+
* (<cache>/opencode-vision-analyze/descriptions,每条目一文件、mtime 作 LRU
|
|
17
|
+
* 时钟、2000 条 / 50MB 双上限、单条超限不入缓存)——跨项目/跨进程/插件重启后
|
|
18
|
+
* 同图同问题只描述一次。
|
|
19
|
+
* 泛解析(描述整图)请求的 question 会归一化到固定文案 GENERIC_QUESTION
|
|
20
|
+
* (主模型省略 question 即走该默认值),key 收敛为 <图片sha>:GENERIC_QUESTION,
|
|
21
|
+
* 避免主模型每次自编措辞把缓存拆碎;有具体追问时保留原问句走各自精确 key。
|
|
22
|
+
* 泛解析条目是全图共享的,写入前按最短文本长度把关,防一条垃圾描述毒化整图。
|
|
17
23
|
* image_path 除了绝对路径也接受 http(s) URL:先下载落盘到同一用户级 vision
|
|
18
24
|
* 目录(内容哈希命名,天然与附件落盘去重;LRU + 容量上限,2000 条 / 500MB),
|
|
19
|
-
*
|
|
20
|
-
*
|
|
25
|
+
* 再走统一的磁盘加载路径。**路径粘贴与本地已存在的文件直接原位读取**(不复制、
|
|
26
|
+
* 不 touch,始终读文件最新内容);只有剪贴板贴图/下载才写缓存。
|
|
21
27
|
* 主模型本身支持图片输入时走快速路径:不做子会话描述,直接把原图作为
|
|
22
28
|
* 工具附件回传给模型自行查看。
|
|
23
29
|
*
|
|
@@ -130,6 +136,45 @@ export const visionCacheLimits = {
|
|
|
130
136
|
maxEntries: 2000,
|
|
131
137
|
maxBytes: 500 * 1024 * 1024,
|
|
132
138
|
};
|
|
139
|
+
/**
|
|
140
|
+
* 泛解析(描述整张图)用的固定问题文案。
|
|
141
|
+
*
|
|
142
|
+
* 主模型对「解析这张图 / 描述图片」这类请求应省略 question,由工具用这里的文案兜底,
|
|
143
|
+
* 从而让所有泛解析请求的缓存 key 收敛到同一条(key = <图片sha>:GENERIC_QUESTION),
|
|
144
|
+
* 避免主模型每次自编措辞把缓存拆成多份、跨会话永远命中不了。文案点名文字/UI/图表/
|
|
145
|
+
* 可见内容,驱动视觉模型把整图信息尽量带全。
|
|
146
|
+
*
|
|
147
|
+
* 这份文案相当于公开契约:改动它会让旧的泛解析缓存条目失去命中(由 LRU 按 mtime 清理),
|
|
148
|
+
* 属于刻意为之——将来想改泛描述措辞、或按用户/场景定制时,改文案即可自然分开新旧缓存。
|
|
149
|
+
*/
|
|
150
|
+
export const GENERIC_QUESTION = "Describe this image in full detail, including all text, UI elements, diagrams, or content visible.";
|
|
151
|
+
/**
|
|
152
|
+
* 泛解析缓存条目的最短文本长度(字符)。
|
|
153
|
+
*
|
|
154
|
+
* canonical 描述要求整图加逐字转录,正常结果不可能太短;太短多半是视觉模型敷衍或拒答。
|
|
155
|
+
* 泛解析条目是全图共享的,存进一条垃圾描述会让之后所有泛解析都命中它,所以写入时长度
|
|
156
|
+
* 低于该值就跳过。具体追问不受限制(“答案是 42”是合法答案)。
|
|
157
|
+
* 做成模块级可变对象,测试注入大值/小值验证两侧行为。
|
|
158
|
+
*/
|
|
159
|
+
export const genericWriteMinText = { chars: 100 };
|
|
160
|
+
/**
|
|
161
|
+
* 归一化问题文本,供「是否为泛解析」判定使用:
|
|
162
|
+
* 去掉首尾空白与引号/括号,把连续空白(含全角/换行)折成一个空格,去掉句末标点,统一小写。
|
|
163
|
+
* 只做字符级归一化,不做语义匹配——误判泛解析会把针对性回答套到错误语义上,宁可 miss。
|
|
164
|
+
*/
|
|
165
|
+
export function normalizeQuestion(q) {
|
|
166
|
+
return q
|
|
167
|
+
.trim()
|
|
168
|
+
.replace(/^[\s"'“”「」『』()()\[\]]+|[\s"'“”「」『』()()\[\]]+$/g, "")
|
|
169
|
+
.replace(/[\s]+/g, " ")
|
|
170
|
+
.replace(/[.。!!??]+$/g, "")
|
|
171
|
+
.toLowerCase();
|
|
172
|
+
}
|
|
173
|
+
/** 是否泛解析请求:没给问题(空/纯空白),或归一化后与 GENERIC_QUESTION 一致。 */
|
|
174
|
+
export function isGenericQuestion(q) {
|
|
175
|
+
const s = (q ?? "").trim();
|
|
176
|
+
return s === "" || normalizeQuestion(s) === normalizeQuestion(GENERIC_QUESTION);
|
|
177
|
+
}
|
|
133
178
|
/**
|
|
134
179
|
* http(s) 下载图片的大小上限(20 MB)。提示注入可让模型指向超大图片,
|
|
135
180
|
* 下载不限长是成本/健壮性放大器:先按 content-length 头提前拒绝,
|
|
@@ -533,7 +578,13 @@ const plugin = async (input, optionsArg) => {
|
|
|
533
578
|
const visionAnalyze = async (args, ctx) => {
|
|
534
579
|
const title = "vision_analyze";
|
|
535
580
|
try {
|
|
536
|
-
|
|
581
|
+
// 泛解析判定:没给问题(空/纯空白),或措辞与 GENERIC_QUESTION 归一化一致。
|
|
582
|
+
// 泛解析请求统一改写为 GENERIC_QUESTION——缓存 key 固定成 <图片sha>:GENERIC_QUESTION,
|
|
583
|
+
// 跨会话、跨措辞都能命中;主模型若问的是具体对象/文字/区域/颜色,则保留原问句,
|
|
584
|
+
// 走各自的精确 key(与旧版本格式一致,存量条目不受影响)。
|
|
585
|
+
const trimmed = args.question?.trim() ?? "";
|
|
586
|
+
const generic = trimmed === "" || isGenericQuestion(trimmed);
|
|
587
|
+
const question = generic ? GENERIC_QUESTION : trimmed;
|
|
537
588
|
// http(s) URL:先下载到用户级 vision 缓存目录,再统一走磁盘加载路径。
|
|
538
589
|
// ctx.abort 传入下载:用户中止即刻断下载(含 pre-abort 不再发请求)。
|
|
539
590
|
const download = /^https?:\/\//i.test(args.image_path)
|
|
@@ -563,7 +614,7 @@ const plugin = async (input, optionsArg) => {
|
|
|
563
614
|
],
|
|
564
615
|
};
|
|
565
616
|
}
|
|
566
|
-
// 描述缓存:内容哈希 +
|
|
617
|
+
// 描述缓存:内容哈希 + 生效问题作为 key,命中直接复用(title 标注 cached)。
|
|
567
618
|
// 落盘在用户级共享目录(resolveDescriptionDir)——跨项目/进程/重启命中;磁盘即事实。
|
|
568
619
|
const key = `${createHash("sha256").update(image.bytes).digest("hex")}:${question}`;
|
|
569
620
|
const cached = await descCacheGet(key);
|
|
@@ -575,6 +626,12 @@ const plugin = async (input, optionsArg) => {
|
|
|
575
626
|
const result = await describeWithChain(image, question, ctx);
|
|
576
627
|
if (!result.ok)
|
|
577
628
|
return { title, output: `Image analysis failed: ${result.error}` };
|
|
629
|
+
// 泛解析条目是全图共享的:文本太短多半是视觉模型敷衍/拒答,存进去会毒化这条
|
|
630
|
+
// canonical 缓存,让以后所有泛解析都命中垃圾描述,所以低于门槛就跳过落盘。
|
|
631
|
+
// 具体追问不受限("答案是 42" 是合法短答案)。
|
|
632
|
+
if (generic && result.text.length < genericWriteMinText.chars) {
|
|
633
|
+
return { title, output: format(path.basename(imagePath), result.modelId, result.text) };
|
|
634
|
+
}
|
|
578
635
|
// 入库带上实际产出描述的候选 modelId,供后续缓存命中还原真实标签
|
|
579
636
|
await descCacheSet(key, { modelId: result.modelId, text: result.text });
|
|
580
637
|
// 成功标签直接用实际产出描述的候选引用键(而非链首近似)
|
|
@@ -697,6 +754,34 @@ const plugin = async (input, optionsArg) => {
|
|
|
697
754
|
return undefined;
|
|
698
755
|
}
|
|
699
756
|
};
|
|
757
|
+
/**
|
|
758
|
+
* 解析图片 part 的可用绝对路径:
|
|
759
|
+
* - **路径粘贴**(part 带真实 source.path):原位读用,不复制进 vision 缓存。相对路径
|
|
760
|
+
* 以 input.directory 为基准绝对化,与 opencode 工具一致(tool/read.ts 等用
|
|
761
|
+
* path.resolve(instance.directory, filepath))。提示里给绝对路径,vision_analyze
|
|
762
|
+
* 每次调用当场重读该文件 → 始终分析最新内容;描述缓存按内容 sha,天然不缓存旧内容。
|
|
763
|
+
* - **复制图片本身**(剪贴板位图,source.path="clipboard")、无 source、扩展名不受支持、
|
|
764
|
+
* 文件缺失/为空 → 回退 persistImage 内容寻址落盘(贴图必须给模型稳定 image_path)。
|
|
765
|
+
* 判定失败一律回退,绝不抛错(fail-open)。
|
|
766
|
+
*/
|
|
767
|
+
const resolveImagePath = async (part) => {
|
|
768
|
+
const src = part.source;
|
|
769
|
+
if (src?.type === "file" && src.path) {
|
|
770
|
+
const abs = path.isAbsolute(src.path) ? src.path : path.resolve(input.directory, src.path);
|
|
771
|
+
const ext = path.extname(abs).toLowerCase();
|
|
772
|
+
if (EXT_MIME[ext]) {
|
|
773
|
+
try {
|
|
774
|
+
const st = await fs.stat(abs);
|
|
775
|
+
if (st.isFile() && st.size > 0)
|
|
776
|
+
return abs;
|
|
777
|
+
}
|
|
778
|
+
catch {
|
|
779
|
+
// 缺失/不可读 → 回退落盘
|
|
780
|
+
}
|
|
781
|
+
}
|
|
782
|
+
}
|
|
783
|
+
return (await persistImage(part))?.filepath;
|
|
784
|
+
};
|
|
700
785
|
/**
|
|
701
786
|
* chat.message 钩子:用户消息持久化前触发(parts 数组与持久化同引用,
|
|
702
787
|
* push 进去的 part 会一并入库)。
|
|
@@ -707,7 +792,8 @@ const plugin = async (input, optionsArg) => {
|
|
|
707
792
|
* 3. 递归防护——消息模型 ∈ 候选链全体成员(我们的描述子会话)则放行;
|
|
708
793
|
* 4. 能力门控——主模型本身能看图则不注入提示;
|
|
709
794
|
* 5. 空链降级——没有任何可用视觉模型时不注入 hint;
|
|
710
|
-
* 6.
|
|
795
|
+
* 6. 解析图片路径(路径粘贴原位读用、其余落盘),并注入一条 synthetic text part
|
|
796
|
+
* 引导模型使用 vision_analyze。
|
|
711
797
|
*/
|
|
712
798
|
const onChatMessage = async (hookInput, output) => {
|
|
713
799
|
// 记录会话当前模型,供后续 vision_analyze 快速路径与未显式指定 model 的
|
|
@@ -734,17 +820,21 @@ const plugin = async (input, optionsArg) => {
|
|
|
734
820
|
const chain = await resolveChain();
|
|
735
821
|
if (chain.length === 0)
|
|
736
822
|
return;
|
|
737
|
-
//
|
|
823
|
+
// 每张图解析路径并生成两行提示;任一张解析失败就跳过该图(不影响其余图片)。
|
|
738
824
|
const lines = [];
|
|
739
825
|
for (const part of images) {
|
|
740
|
-
const
|
|
741
|
-
if (!
|
|
826
|
+
const imagePath = await resolveImagePath(part);
|
|
827
|
+
if (!imagePath)
|
|
742
828
|
continue;
|
|
743
829
|
lines.push(`[The user attached an image: ${part.filename ?? "image"}]`);
|
|
744
|
-
lines.push(`[Examine it with the vision_analyze tool using image_path: ${
|
|
830
|
+
lines.push(`[Examine it with the vision_analyze tool using image_path: ${imagePath}]`);
|
|
745
831
|
}
|
|
746
832
|
if (lines.length === 0)
|
|
747
833
|
return;
|
|
834
|
+
// 泛解析指引:主模型对"描述整图/解析图片"这类请求应省略 question(工具内部会用固定
|
|
835
|
+
// canonical 文案兜底,缓存 key 才能跨会话收敛);只有用户问具体对象/文字/区域/颜色时才
|
|
836
|
+
// 填具体问句。措辞分化的泛问句会让缓存 key 拆碎、永远命中不了。
|
|
837
|
+
lines.push("[When the user asks for a general parse of the whole image, call vision_analyze WITHOUT a question (omit the question parameter). Only pass a question when the user asks about a specific object, text, region, or color.]");
|
|
748
838
|
// 注入 synthetic text part:TUI 隐藏(不干扰用户输入展示),但会发给模型。
|
|
749
839
|
// id 需满足 PartID 约定(prt 前缀)。
|
|
750
840
|
const hint = {
|
|
@@ -765,10 +855,13 @@ const plugin = async (input, optionsArg) => {
|
|
|
765
855
|
// 既不引入 zod 运行时依赖(本插件只用 node 内置模块),也不使用 any。
|
|
766
856
|
tool: {
|
|
767
857
|
vision_analyze: {
|
|
768
|
-
description: "Analyze an image with the dedicated vision model. image_path is an absolute file path (as given in the user's attachment hint) or an http(s) image URL. question
|
|
858
|
+
description: "Analyze an image with the dedicated vision model. image_path is an absolute file path (as given in the user's attachment hint) or an http(s) image URL. question is optional: pass it only when you need a specific detail (an object, text, region or color); otherwise omit it to get a full description of the whole image.",
|
|
769
859
|
args: {
|
|
770
860
|
image_path: { type: "string", description: "Absolute path to the image file, or an http(s) image URL." },
|
|
771
|
-
question: {
|
|
861
|
+
question: {
|
|
862
|
+
type: "string",
|
|
863
|
+
description: "Optional. What specific detail to look for (object/text/region/color). Omit for a full description of the whole image.",
|
|
864
|
+
},
|
|
772
865
|
},
|
|
773
866
|
execute: visionAnalyze,
|
|
774
867
|
},
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "opencode-vision-analyze",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.6.0",
|
|
4
4
|
"description": "OpenCode plugin: vision_analyze tool — describe images & image URLs with your own vision model for text-only main models; native-image fast path for multimodal models. Zero runtime dependencies.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "./dist/index.js",
|