opencode-see-image 0.2.1 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +57 -29
- package/index.ts +252 -151
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
Give non-vision opencode models the ability to **see images and screenshots** by routing them to a vision-capable model.
|
|
4
4
|
|
|
5
|
-
When a user attaches a screenshot to a text-only model
|
|
5
|
+
When a user attaches a screenshot to a text-only model, opencode rejects it with an error. This plugin intercepts that flow by registering a `see_image` tool that sends the image to a vision model and returns a textual description the primary model can reason about.
|
|
6
6
|
|
|
7
7
|
## Install
|
|
8
8
|
|
|
@@ -15,28 +15,44 @@ Add the plugin to your opencode config:
|
|
|
15
15
|
"plugin": ["opencode-see-image"]
|
|
16
16
|
}
|
|
17
17
|
```
|
|
18
|
+
Then restart opencode.
|
|
18
19
|
|
|
19
|
-
Then **restart opencode**.
|
|
20
20
|
|
|
21
|
-
|
|
21
|
+
## Install via your agent (copy-paste this)
|
|
22
|
+
|
|
23
|
+
Paste this prompt to your opencode agent, and it'll install the plugin for you.
|
|
24
|
+
|
|
25
|
+
```
|
|
26
|
+
Install the opencode-see-image plugin so I can send you screenshots. Do this:
|
|
27
|
+
|
|
28
|
+
1. Edit ~/.config/opencode/opencode.jsonc (create it if missing). Preserve any existing fields and the $schema. Add "opencode-see-image" to the "plugin" array. If "plugin" doesn't exist, add it as ["opencode-see-image"].
|
|
29
|
+
2. Check that a vision-capable provider is connected by looking for ~/.local/share/opencode/auth.json with either an "opencode-go" entry (paid, fast) OR an "opencode" entry (free). If neither is present, tell me to run /connect and select either opencode-go or opencode (key from opencode.ai/auth).
|
|
30
|
+
3. Tell me to quit and restart opencode for the plugin to load.
|
|
31
|
+
|
|
32
|
+
After I restart and attach a screenshot, you should call the see_image tool to view it.
|
|
33
|
+
```
|
|
22
34
|
|
|
23
35
|
## Prerequisites
|
|
24
36
|
|
|
25
|
-
You need a connected vision-capable provider. The plugin auto-detects whichever you have connected
|
|
37
|
+
You need a connected vision-capable provider. The plugin auto-detects whichever you have connected, **either of these works**:
|
|
26
38
|
|
|
27
|
-
###
|
|
39
|
+
### Free (OpenCode Zen)
|
|
28
40
|
1. Run `/connect` in opencode
|
|
29
41
|
2. Select **opencode** (OpenCode Zen)
|
|
30
42
|
3. Paste your API key from [opencode.ai/auth](https://opencode.ai/auth)
|
|
31
43
|
|
|
32
|
-
The plugin falls back to **big-pickle** (
|
|
44
|
+
The plugin falls back to **big-pickle** (~12000ms). No subscription needed.
|
|
33
45
|
|
|
34
|
-
###
|
|
46
|
+
### Paid, w/ OpenCode Go
|
|
35
47
|
1. Run `/connect` in opencode
|
|
36
48
|
2. Select **opencode-go**
|
|
37
49
|
3. Paste your API key from [opencode.ai/auth](https://opencode.ai/auth)
|
|
38
50
|
|
|
39
|
-
The plugin prefers **minimax-m3** via opencode-go (~
|
|
51
|
+
The plugin prefers **minimax-m3** via opencode-go (~3000ms) when available.
|
|
52
|
+
|
|
53
|
+
### Paid, w/ another provider
|
|
54
|
+
|
|
55
|
+
Set the `SEE_IMAGE_*` env vars to point at any Anthropic-Messages-compatible endpoint. See [Configuration](#configuration) below.
|
|
40
56
|
|
|
41
57
|
**Resolution order:** explicit `SEE_IMAGE_API_KEY` env → configured `SEE_IMAGE_PROVIDER` → `opencode-go` (MiniMax M3) → `opencode` (big-pickle, free).
|
|
42
58
|
|
|
@@ -62,6 +78,17 @@ see_image tool:
|
|
|
62
78
|
primary model answers using the description
|
|
63
79
|
```
|
|
64
80
|
|
|
81
|
+
## The `see_image` tool
|
|
82
|
+
|
|
83
|
+
The plugin registers a `see_image` tool with two arguments:
|
|
84
|
+
|
|
85
|
+
| Arg | Type | Required | Description |
|
|
86
|
+
|---|---|---|---|
|
|
87
|
+
| `filePath` | string | yes | Path to the image. Absolute path, or a bare filename like `"Screenshot 2026-06-18 at 17.32.24.png"` to auto-locate. |
|
|
88
|
+
| `question` | string | no | A specific question about the image. Defaults to a general detailed description. Use this to focus on a particular detail (e.g. `"What error is shown in the terminal?"`). |
|
|
89
|
+
|
|
90
|
+
Your model calls this tool automatically when you attach a screenshot, you don't need to do anything special. The `question` arg is optional; the model uses it when you ask something specific about the image.
|
|
91
|
+
|
|
65
92
|
## Configuration
|
|
66
93
|
|
|
67
94
|
All settings are env-var overrides. Defaults work out-of-the-box for opencode-go + MiniMax M3.
|
|
@@ -97,43 +124,44 @@ export SEE_IMAGE_MODEL="kimi-k2.7-code"
|
|
|
97
124
|
|
|
98
125
|
| Model | Speed | Notes |
|
|
99
126
|
|---|---|---|
|
|
100
|
-
| `big-pickle` | ~
|
|
127
|
+
| `big-pickle` | ~12000ms | Free. Accurate. Default fallback when only Zen is connected. |
|
|
101
128
|
|
|
102
129
|
**Paid (OpenCode Go):**
|
|
103
130
|
|
|
104
131
|
| Model | Speed | Notes |
|
|
105
132
|
|---|---|---|
|
|
106
|
-
| `minimax-m3` | ~
|
|
107
|
-
| `kimi-k2.7-code` | ~
|
|
108
|
-
| `kimi-k2.6` | ~
|
|
109
|
-
| `qwen3.7-plus` | ~
|
|
133
|
+
| `minimax-m3` | ~3000ms | Default. Fast, clean text output. |
|
|
134
|
+
| `kimi-k2.7-code` | ~7000ms | Clean output, accurate. |
|
|
135
|
+
| `kimi-k2.6` | ~20000ms | Accurate but slow. |
|
|
136
|
+
| `qwen3.7-plus` | ~20000ms | Emits thinking blocks (handled). |
|
|
137
|
+
|
|
138
|
+
## Updating
|
|
139
|
+
|
|
140
|
+
**Auto-update (built in):** the plugin checks npm for a newer version on every opencode startup. If one exists, it runs `bun update` automatically and shows a toast: *"opencode-see-image updated to X.Y.Z, restart opencode to apply"*. You just need to restart opencode to load the new version. Nothing to configure.
|
|
141
|
+
|
|
142
|
+
**Manual update** (if you want to force it now):
|
|
143
|
+
```bash
|
|
144
|
+
cd ~/.cache/opencode && bun update opencode-see-image
|
|
145
|
+
```
|
|
146
|
+
Then restart opencode.
|
|
147
|
+
|
|
148
|
+
**Pin a version** in your config to opt out of auto-updates:
|
|
149
|
+
```jsonc
|
|
150
|
+
"plugin": ["opencode-see-image@0.3.0"]
|
|
151
|
+
```
|
|
110
152
|
|
|
111
153
|
## File search locations
|
|
112
154
|
|
|
113
155
|
When opencode rejects an image attachment, the model only receives a bare filename. `see_image` searches these locations in order:
|
|
114
156
|
|
|
115
|
-
1. `$TMPDIR/TemporaryItems/NSIRD_screencaptureui_*/`
|
|
157
|
+
1. `$TMPDIR/TemporaryItems/NSIRD_screencaptureui_*/` (where macOS stashes dragged screenshots)
|
|
116
158
|
2. `$TMPDIR/TemporaryItems/`
|
|
117
|
-
3. `~/Desktop`
|
|
159
|
+
3. `~/Desktop` (default screenshot save location)
|
|
118
160
|
4. `~/Downloads`
|
|
119
161
|
5. Current working directory
|
|
120
162
|
|
|
121
163
|
Pass an absolute `filePath` to skip the search.
|
|
122
164
|
|
|
123
|
-
## Install via your agent (copy-paste this)
|
|
124
|
-
|
|
125
|
-
Paste this prompt to your opencode agent — it'll install the plugin for you:
|
|
126
|
-
|
|
127
|
-
```
|
|
128
|
-
Install the opencode-see-image plugin so I can send you screenshots. Do this:
|
|
129
|
-
|
|
130
|
-
1. Edit ~/.config/opencode/opencode.jsonc (create it if missing). Preserve any existing fields and the $schema. Add "opencode-see-image" to the "plugin" array. If "plugin" doesn't exist, add it as ["opencode-see-image"].
|
|
131
|
-
2. Check that a vision-capable provider is connected by looking for ~/.local/share/opencode/auth.json with either an "opencode-go" entry (paid, fast) OR an "opencode" entry (free). If neither is present, tell me to run /connect and select either opencode-go or opencode (key from opencode.ai/auth).
|
|
132
|
-
3. Tell me to quit and restart opencode for the plugin to load.
|
|
133
|
-
|
|
134
|
-
After I restart and attach a screenshot, you should call the see_image tool to view it.
|
|
135
|
-
```
|
|
136
|
-
|
|
137
165
|
## License
|
|
138
166
|
|
|
139
167
|
MIT
|
package/index.ts
CHANGED
|
@@ -4,10 +4,6 @@ import os from "os"
|
|
|
4
4
|
import fs from "fs"
|
|
5
5
|
import type { Plugin } from "@opencode-ai/plugin"
|
|
6
6
|
|
|
7
|
-
// ─── Configuration (env-overridable) ────────────────────────────────────────
|
|
8
|
-
// Defaults target opencode-go's MiniMax M3. Users on other providers can
|
|
9
|
-
// override via environment variables without editing this file.
|
|
10
|
-
|
|
11
7
|
const ENDPOINT =
|
|
12
8
|
process.env.SEE_IMAGE_ENDPOINT ||
|
|
13
9
|
"https://opencode.ai/zen/go/v1/messages"
|
|
@@ -27,61 +23,6 @@ const EXT_MEDIA: Record<string, string> = {
|
|
|
27
23
|
bmp: "image/bmp",
|
|
28
24
|
}
|
|
29
25
|
|
|
30
|
-
// ─── Auth ───────────────────────────────────────────────────────────────────
|
|
31
|
-
// Resolves a usable API key + the endpoint + model to use. Falls back through
|
|
32
|
-
// a chain so users with any connected opencode subscription (paid opencode-go
|
|
33
|
-
// or free opencode Zen) get a working default with zero config.
|
|
34
|
-
function resolveAuth(): { key: string; endpoint: string; model: string } {
|
|
35
|
-
// 1. Explicit env vars win outright.
|
|
36
|
-
if (process.env.SEE_IMAGE_API_KEY) {
|
|
37
|
-
return { key: process.env.SEE_IMAGE_API_KEY, endpoint: ENDPOINT, model: MODEL }
|
|
38
|
-
}
|
|
39
|
-
|
|
40
|
-
// 2. Walk opencode's auth store and try the configured provider first,
|
|
41
|
-
// then a curated fallback chain (paid → free).
|
|
42
|
-
const authPath = path.join(os.homedir(), ".local/share/opencode/auth.json")
|
|
43
|
-
let auth: any = {}
|
|
44
|
-
try {
|
|
45
|
-
auth = JSON.parse(fs.readFileSync(authPath, "utf8"))
|
|
46
|
-
} catch {
|
|
47
|
-
// ignore — handled by the empty-auth path below
|
|
48
|
-
}
|
|
49
|
-
|
|
50
|
-
// Each candidate: [providerId, endpoint, defaultModel]
|
|
51
|
-
const candidates: Array<[string, string, string]> = [
|
|
52
|
-
[PROVIDER_ID, ENDPOINT, MODEL],
|
|
53
|
-
// Free fallback: OpenCode Zen's big-pickle supports vision at no cost.
|
|
54
|
-
["opencode", "https://opencode.ai/zen/v1/messages", "big-pickle"],
|
|
55
|
-
// Paid fallbacks on opencode-go:
|
|
56
|
-
["opencode-go", "https://opencode.ai/zen/go/v1/messages", "minimax-m3"],
|
|
57
|
-
]
|
|
58
|
-
|
|
59
|
-
const tried: string[] = []
|
|
60
|
-
for (const [pid, ep, mdl] of candidates) {
|
|
61
|
-
const entry = auth[pid]
|
|
62
|
-
const k = entry && (entry.key || entry.access)
|
|
63
|
-
if (k) {
|
|
64
|
-
// If the user pinned PROVIDER_ID but not MODEL, honor the candidate's
|
|
65
|
-
// default model only when provider matches the pinned one; otherwise
|
|
66
|
-
// keep the configured MODEL (may be set via env).
|
|
67
|
-
const useModel =
|
|
68
|
-
pid === PROVIDER_ID || !process.env.SEE_IMAGE_MODEL ? mdl : MODEL
|
|
69
|
-
return { key: k, endpoint: ep, model: useModel }
|
|
70
|
-
}
|
|
71
|
-
tried.push(pid)
|
|
72
|
-
}
|
|
73
|
-
|
|
74
|
-
throw new Error(
|
|
75
|
-
`see_image: no API key. Connect a provider in opencode via /connect — ` +
|
|
76
|
-
`either "opencode-go" (paid, fast MiniMax M3) or "opencode" (free, big-pickle). ` +
|
|
77
|
-
`Or set SEE_IMAGE_API_KEY explicitly. (Checked providers: ${tried.join(", ") || "none"} in ${authPath}.)`,
|
|
78
|
-
)
|
|
79
|
-
}
|
|
80
|
-
|
|
81
|
-
// ─── File resolution ────────────────────────────────────────────────────────
|
|
82
|
-
// When opencode rejects an image attachment, the model only sees a bare
|
|
83
|
-
// filename (no path). This resolves bare filenames by searching the places
|
|
84
|
-
// macOS / opencode tend to stash screenshots.
|
|
85
26
|
function resolveFilePath(name: string, cwd: string): string {
|
|
86
27
|
if (path.isAbsolute(name) && fs.existsSync(name)) return name
|
|
87
28
|
|
|
@@ -91,8 +32,6 @@ function resolveFilePath(name: string, cwd: string): string {
|
|
|
91
32
|
const tmpdir = process.env.TMPDIR || "/tmp"
|
|
92
33
|
const searchDirs: string[] = []
|
|
93
34
|
|
|
94
|
-
// macOS screenshot tool temp dirs (NSIRD_screencaptureui_<rand>) — this is
|
|
95
|
-
// where dragged screenshots actually land, not ~/Desktop.
|
|
96
35
|
const tempItems = path.join(tmpdir, "TemporaryItems")
|
|
97
36
|
if (fs.existsSync(tempItems)) {
|
|
98
37
|
try {
|
|
@@ -116,7 +55,6 @@ function resolveFilePath(name: string, cwd: string): string {
|
|
|
116
55
|
} catch {}
|
|
117
56
|
}
|
|
118
57
|
|
|
119
|
-
// Shallow recursive search in the top-level search dirs.
|
|
120
58
|
for (const dir of searchDirs) {
|
|
121
59
|
if (!dir || !fs.existsSync(dir)) continue
|
|
122
60
|
try {
|
|
@@ -133,103 +71,135 @@ function resolveFilePath(name: string, cwd: string): string {
|
|
|
133
71
|
)
|
|
134
72
|
}
|
|
135
73
|
|
|
136
|
-
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
|
|
146
|
-
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
:
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
|
|
169
|
-
model,
|
|
170
|
-
max_tokens: 2048,
|
|
171
|
-
messages: [
|
|
172
|
-
{
|
|
173
|
-
role: "user",
|
|
174
|
-
content: [
|
|
175
|
-
{
|
|
176
|
-
type: "image",
|
|
177
|
-
source: { type: "base64", media_type: mediaType, data: b64 },
|
|
178
|
-
},
|
|
74
|
+
async function seeImageViaSDK(
|
|
75
|
+
client: any,
|
|
76
|
+
dataUrl: string,
|
|
77
|
+
mediaType: string,
|
|
78
|
+
prompt: string,
|
|
79
|
+
): Promise<{ text: string; model: string; provider: string }> {
|
|
80
|
+
const envProvider = process.env.SEE_IMAGE_PROVIDER
|
|
81
|
+
const envModel = process.env.SEE_IMAGE_MODEL
|
|
82
|
+
const candidates: Array<{ providerID: string; modelID: string }> = []
|
|
83
|
+
if (envProvider && envModel) {
|
|
84
|
+
candidates.push({ providerID: envProvider, modelID: envModel })
|
|
85
|
+
}
|
|
86
|
+
candidates.push({ providerID: "opencode-go", modelID: "minimax-m3" })
|
|
87
|
+
candidates.push({ providerID: "opencode", modelID: "big-pickle" })
|
|
88
|
+
|
|
89
|
+
const errors: string[] = []
|
|
90
|
+
|
|
91
|
+
for (const { providerID, modelID } of candidates) {
|
|
92
|
+
let sessionID: string | undefined
|
|
93
|
+
try {
|
|
94
|
+
const sessionRes = await client.session.create({ body: {} })
|
|
95
|
+
sessionID = sessionRes.data?.id
|
|
96
|
+
if (!sessionID) {
|
|
97
|
+
errors.push(`${providerID}/${modelID}: no session ID`)
|
|
98
|
+
continue
|
|
99
|
+
}
|
|
100
|
+
|
|
101
|
+
const result = await client.session.prompt({
|
|
102
|
+
path: { id: sessionID },
|
|
103
|
+
body: {
|
|
104
|
+
model: { providerID, modelID },
|
|
105
|
+
parts: [
|
|
106
|
+
{ type: "file", mime: mediaType, url: dataUrl },
|
|
179
107
|
{ type: "text", text: prompt },
|
|
180
108
|
],
|
|
109
|
+
tools: {},
|
|
110
|
+
system:
|
|
111
|
+
"You are a vision assistant. Describe the image accurately and concisely. Answer with text only.",
|
|
181
112
|
},
|
|
182
|
-
|
|
113
|
+
})
|
|
114
|
+
|
|
115
|
+
const parts = result.data?.parts ?? []
|
|
116
|
+
const text = (parts as any[])
|
|
117
|
+
.filter((p: any) => p.type === "text")
|
|
118
|
+
.map((p: any) => p.text)
|
|
119
|
+
.filter((t: any) => typeof t === "string" && t.length > 0)
|
|
120
|
+
.join("\n")
|
|
121
|
+
.trim()
|
|
122
|
+
|
|
123
|
+
if (text) {
|
|
124
|
+
return { text, model: modelID, provider: providerID }
|
|
125
|
+
}
|
|
126
|
+
errors.push(`${providerID}/${modelID}: no text in response`)
|
|
127
|
+
} catch (e: any) {
|
|
128
|
+
errors.push(`${providerID}/${modelID}: ${e?.message ?? e}`)
|
|
129
|
+
} finally {
|
|
130
|
+
if (sessionID) {
|
|
131
|
+
await client.session
|
|
132
|
+
.delete({ path: { id: sessionID } })
|
|
133
|
+
.catch(() => {})
|
|
134
|
+
}
|
|
183
135
|
}
|
|
136
|
+
}
|
|
184
137
|
|
|
185
|
-
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
|
|
189
|
-
"anthropic-version": API_VERSION,
|
|
190
|
-
"content-type": "application/json",
|
|
191
|
-
"user-agent": USER_AGENT,
|
|
192
|
-
},
|
|
193
|
-
body: JSON.stringify(body),
|
|
194
|
-
})
|
|
138
|
+
throw new Error(
|
|
139
|
+
`see_image: SDK vision call failed for all candidates. ${errors.join("; ")}`,
|
|
140
|
+
)
|
|
141
|
+
}
|
|
195
142
|
|
|
196
|
-
|
|
197
|
-
|
|
198
|
-
|
|
199
|
-
|
|
200
|
-
|
|
201
|
-
|
|
143
|
+
async function seeImageViaHTTP(
|
|
144
|
+
b64: string,
|
|
145
|
+
mediaType: string,
|
|
146
|
+
prompt: string,
|
|
147
|
+
): Promise<{ text: string; model: string; provider: string }> {
|
|
148
|
+
const key = process.env.SEE_IMAGE_API_KEY!
|
|
149
|
+
const body = {
|
|
150
|
+
model: MODEL,
|
|
151
|
+
max_tokens: 2048,
|
|
152
|
+
messages: [
|
|
153
|
+
{
|
|
154
|
+
role: "user",
|
|
155
|
+
content: [
|
|
156
|
+
{
|
|
157
|
+
type: "image",
|
|
158
|
+
source: { type: "base64", media_type: mediaType, data: b64 },
|
|
159
|
+
},
|
|
160
|
+
{ type: "text", text: prompt },
|
|
161
|
+
],
|
|
162
|
+
},
|
|
163
|
+
],
|
|
164
|
+
}
|
|
202
165
|
|
|
203
|
-
|
|
204
|
-
|
|
205
|
-
|
|
206
|
-
|
|
207
|
-
|
|
208
|
-
|
|
209
|
-
|
|
210
|
-
|
|
211
|
-
|
|
212
|
-
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
|
|
216
|
-
|
|
166
|
+
const res = await fetch(ENDPOINT, {
|
|
167
|
+
method: "POST",
|
|
168
|
+
headers: {
|
|
169
|
+
"x-api-key": key,
|
|
170
|
+
"anthropic-version": API_VERSION,
|
|
171
|
+
"content-type": "application/json",
|
|
172
|
+
"user-agent": USER_AGENT,
|
|
173
|
+
},
|
|
174
|
+
body: JSON.stringify(body),
|
|
175
|
+
})
|
|
176
|
+
|
|
177
|
+
if (!res.ok) {
|
|
178
|
+
const errText = await res.text()
|
|
179
|
+
throw new Error(
|
|
180
|
+
`see_image: HTTP vision call to "${MODEL}" failed: HTTP ${res.status}, ${errText.slice(0, 300)}`,
|
|
181
|
+
)
|
|
182
|
+
}
|
|
217
183
|
|
|
218
|
-
|
|
219
|
-
|
|
220
|
-
|
|
221
|
-
|
|
184
|
+
const data: any = await res.json()
|
|
185
|
+
const text = data?.content
|
|
186
|
+
?.map((c: any) => c.text)
|
|
187
|
+
.filter((t: any) => typeof t === "string" && t.length > 0)
|
|
188
|
+
.join("\n")
|
|
189
|
+
.trim()
|
|
190
|
+
|
|
191
|
+
if (!text) {
|
|
192
|
+
throw new Error(
|
|
193
|
+
`see_image: model "${MODEL}" returned no text. Response: ${JSON.stringify(data).slice(0, 300)}`,
|
|
194
|
+
)
|
|
195
|
+
}
|
|
222
196
|
|
|
223
|
-
|
|
224
|
-
|
|
225
|
-
})
|
|
197
|
+
return { text, model: MODEL, provider: PROVIDER_ID }
|
|
198
|
+
}
|
|
226
199
|
|
|
227
|
-
|
|
228
|
-
// Injected via experimental.chat.system.transform so the triggering logic
|
|
229
|
-
// ships with the plugin — no separate SKILL.md install needed.
|
|
230
|
-
const SYSTEM_INSTRUCTIONS = `# See Image (vision bridge) — opencode-see-image plugin
|
|
200
|
+
const SYSTEM_INSTRUCTIONS = `# See Image (vision bridge), opencode-see-image plugin
|
|
231
201
|
|
|
232
|
-
You have access to a \`see_image\` tool. The current model may not support image input directly. When a user attaches a screenshot or image, opencode rejects it and you only receive an error string containing the **filename
|
|
202
|
+
You have access to a \`see_image\` tool. The current model may not support image input directly. When a user attaches a screenshot or image, opencode rejects it and you only receive an error string containing the **filename**, no path, no pixels. Use \`see_image\` to actually view it.
|
|
233
203
|
|
|
234
204
|
## When to use \`see_image\`
|
|
235
205
|
|
|
@@ -238,13 +208,13 @@ Use ONLY when one of these is true:
|
|
|
238
208
|
2. The user references an image/screenshot they expect you to see ("see this", "look at this", "can you see this", ".png"/".jpg")
|
|
239
209
|
3. The user pastes an image path they want you to inspect
|
|
240
210
|
|
|
241
|
-
Do NOT use \`see_image\` for reading text files
|
|
211
|
+
Do NOT use \`see_image\` for reading text files, use the \`read\` tool for those.
|
|
242
212
|
|
|
243
213
|
## How to use it
|
|
244
214
|
|
|
245
215
|
1. **Extract the filename** from the error string (the quoted name), or use the path the user gave.
|
|
246
216
|
2. **Call \`see_image\`** with \`filePath\` set to the bare filename (it auto-locates) or an absolute path. Pass an optional \`question\` if the user asked something specific.
|
|
247
|
-
3. **Answer using the returned description** as if you saw the image. Be natural
|
|
217
|
+
3. **Answer using the returned description** as if you saw the image. Be natural, don't mention that you used another model unless asked.
|
|
248
218
|
|
|
249
219
|
## Important
|
|
250
220
|
|
|
@@ -252,8 +222,139 @@ Do NOT use \`see_image\` for reading text files — use the \`read\` tool for th
|
|
|
252
222
|
- If the tool cannot find the file, tell the user the filename and ask for a full path or to drag the file into the project directory.
|
|
253
223
|
- To inspect a specific detail, pass a targeted \`question\` (e.g. "What error is shown in the terminal?").`
|
|
254
224
|
|
|
255
|
-
|
|
225
|
+
const PKG_NAME = "opencode-see-image"
|
|
226
|
+
const REGISTRY_LATEST = `https://registry.npmjs.org/${PKG_NAME}/latest`
|
|
227
|
+
|
|
228
|
+
function currentVersion(): string | null {
|
|
229
|
+
try {
|
|
230
|
+
const here = new URL(".", import.meta.url)
|
|
231
|
+
const pkgPath = new URL("package.json", here)
|
|
232
|
+
const pkg = JSON.parse(fs.readFileSync(pkgPath, "utf8"))
|
|
233
|
+
return pkg.version ?? null
|
|
234
|
+
} catch {
|
|
235
|
+
return null
|
|
236
|
+
}
|
|
237
|
+
}
|
|
238
|
+
|
|
239
|
+
function semverGt(a: string, b: string): boolean {
|
|
240
|
+
const pa = a.split(".").map((n) => parseInt(n, 10) || 0)
|
|
241
|
+
const pb = b.split(".").map((n) => parseInt(n, 10) || 0)
|
|
242
|
+
for (let i = 0; i < 3; i++) {
|
|
243
|
+
const x = pa[i] ?? 0
|
|
244
|
+
const y = pb[i] ?? 0
|
|
245
|
+
if (x > y) return true
|
|
246
|
+
if (x < y) return false
|
|
247
|
+
}
|
|
248
|
+
return false
|
|
249
|
+
}
|
|
250
|
+
|
|
251
|
+
async function maybeAutoUpdate(
|
|
252
|
+
client: any,
|
|
253
|
+
$: any,
|
|
254
|
+
log: (msg: string, level?: string) => void,
|
|
255
|
+
) {
|
|
256
|
+
const current = currentVersion()
|
|
257
|
+
if (!current) return
|
|
258
|
+
|
|
259
|
+
let latest: string
|
|
260
|
+
try {
|
|
261
|
+
const res = await fetch(REGISTRY_LATEST, {
|
|
262
|
+
headers: { accept: "application/json" },
|
|
263
|
+
})
|
|
264
|
+
if (!res.ok) return
|
|
265
|
+
const data: any = await res.json()
|
|
266
|
+
latest = data?.version
|
|
267
|
+
if (!latest) return
|
|
268
|
+
} catch {
|
|
269
|
+
return
|
|
270
|
+
}
|
|
271
|
+
|
|
272
|
+
if (!semverGt(latest, current)) return
|
|
273
|
+
|
|
274
|
+
log(`update available: ${current} -> ${latest}; running bun update`, "info")
|
|
275
|
+
|
|
276
|
+
const cacheDir = path.join(os.homedir(), ".cache/opencode")
|
|
277
|
+
try {
|
|
278
|
+
await $`cd ${cacheDir} && bun update ${PKG_NAME} --no-save`.quiet()
|
|
279
|
+
} catch (e: any) {
|
|
280
|
+
log(`bun update failed: ${e?.message ?? e}`, "warn")
|
|
281
|
+
return
|
|
282
|
+
}
|
|
283
|
+
|
|
284
|
+
try {
|
|
285
|
+
await client?.tui?.showToast?.({
|
|
286
|
+
body: {
|
|
287
|
+
message: `${PKG_NAME} updated to ${latest}, restart opencode to apply`,
|
|
288
|
+
variant: "success",
|
|
289
|
+
},
|
|
290
|
+
})
|
|
291
|
+
} catch {
|
|
292
|
+
log(`update applied: ${current} -> ${latest}; restart opencode to load`, "info")
|
|
293
|
+
}
|
|
294
|
+
}
|
|
295
|
+
|
|
256
296
|
const SeeImagePlugin: Plugin = async (ctx) => {
|
|
297
|
+
const { client, $ } = ctx
|
|
298
|
+
|
|
299
|
+
const log = (message: string, level: string = "info") => {
|
|
300
|
+
try {
|
|
301
|
+
client?.app?.log?.({ body: { service: PKG_NAME, level, message } })
|
|
302
|
+
} catch {}
|
|
303
|
+
}
|
|
304
|
+
|
|
305
|
+
maybeAutoUpdate(client, $, log).catch(() => {})
|
|
306
|
+
|
|
307
|
+
const seeImageTool = tool({
|
|
308
|
+
description:
|
|
309
|
+
'See an image/screenshot that the current model cannot view. Use when the user attaches an image and you get a "this model does not support image input" / "Cannot read" error, or when a screenshot/image is referenced ("see this", "can you see", .png/.jpg). Routes the image to a vision-capable model and returns a detailed textual description you can reason about as if you saw it. Pass filePath as an absolute path OR a bare filename (auto-located in macOS screenshot temp dirs, ~/Desktop, ~/Downloads, cwd).',
|
|
310
|
+
args: {
|
|
311
|
+
filePath: tool.schema
|
|
312
|
+
.string()
|
|
313
|
+
.describe(
|
|
314
|
+
'Path to the image. Absolute path, or a bare filename like "Screenshot 2026-06-18 at 17.32.24.png" to auto-locate.',
|
|
315
|
+
),
|
|
316
|
+
question: tool.schema
|
|
317
|
+
.string()
|
|
318
|
+
.optional()
|
|
319
|
+
.describe(
|
|
320
|
+
"Optional specific question about the image. Defaults to a general detailed description.",
|
|
321
|
+
),
|
|
322
|
+
},
|
|
323
|
+
async execute(args, context) {
|
|
324
|
+
const fullPath = resolveFilePath(args.filePath, context.directory)
|
|
325
|
+
const ext = path.extname(fullPath).slice(1).toLowerCase()
|
|
326
|
+
const mediaType = EXT_MEDIA[ext] || "image/png"
|
|
327
|
+
|
|
328
|
+
const buf = fs.readFileSync(fullPath)
|
|
329
|
+
const b64 = Buffer.from(buf).toString("base64")
|
|
330
|
+
const dataUrl = `data:${mediaType};base64,${b64}`
|
|
331
|
+
|
|
332
|
+
const prompt =
|
|
333
|
+
args.question && args.question.trim().length > 0
|
|
334
|
+
? args.question
|
|
335
|
+
: "Describe this image in detail. If it is a screenshot, describe the UI, text content, and layout precisely. This description will be used by another model to answer the user, so be thorough and accurate."
|
|
336
|
+
|
|
337
|
+
let result: { text: string; model: string; provider: string }
|
|
338
|
+
|
|
339
|
+
if (process.env.SEE_IMAGE_API_KEY) {
|
|
340
|
+
result = await seeImageViaHTTP(b64, mediaType, prompt)
|
|
341
|
+
} else {
|
|
342
|
+
result = await seeImageViaSDK(client, dataUrl, mediaType, prompt)
|
|
343
|
+
}
|
|
344
|
+
|
|
345
|
+
context.metadata({
|
|
346
|
+
title: `see_image: ${path.basename(fullPath)}`,
|
|
347
|
+
metadata: {
|
|
348
|
+
model: result.model,
|
|
349
|
+
provider: result.provider,
|
|
350
|
+
file: fullPath,
|
|
351
|
+
},
|
|
352
|
+
})
|
|
353
|
+
|
|
354
|
+
return result.text
|
|
355
|
+
},
|
|
356
|
+
})
|
|
357
|
+
|
|
257
358
|
return {
|
|
258
359
|
tool: {
|
|
259
360
|
see_image: seeImageTool,
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "opencode-see-image",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.4.0",
|
|
4
4
|
"description": "Give non-vision opencode models the ability to see images/screenshots by routing them to a vision-capable model (MiniMax M3 via opencode-go by default).",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "index.ts",
|