@liustack/modlens 3.10.0 → 3.12.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +16 -0
- package/README.md +1 -1
- package/README.zh-CN.md +2 -2
- package/dist/main.js +306 -170
- package/docs/cli.md +5 -3
- package/docs/harness-setup.md +3 -3
- package/docs/output-schema.md +1 -1
- package/docs/security.md +9 -1
- package/docs/troubleshooting.md +58 -0
- package/dsh/index.js +109 -17
- package/package.json +2 -2
- package/skills/modlens/SKILL.md +5 -5
- package/skills/modlens/references/configure.md +2 -1
- package/skills/modlens/references/find-image.md +1 -1
- package/skills/modlens/references/onboard.md +4 -4
- package/skills/modlens/references/runtime.md +1 -1
- package/skills/modlens/scripts/run.ps1 +1 -1
- package/skills/modlens/scripts/run.sh +1 -1
package/docs/cli.md
CHANGED
|
@@ -30,6 +30,8 @@ Output is a fixed JSON shape:
|
|
|
30
30
|
"summary": "A workflow diagram with four nodes connected by labeled arrows.",
|
|
31
31
|
"ocr": { "full_text": "/shaping\nBEFORE YOU BUILD\n...", "lines": [] },
|
|
32
32
|
"layout": { "regions": [{ "reading_order": 1, "type": "title", "text": "/shaping" }] },
|
|
33
|
+
"semantics": { "scene": "workflow diagram", "entities": [], "relations": [] },
|
|
34
|
+
"visual": { "dominant_colors": ["white", "black"], "style": "flat", "notes": [] },
|
|
33
35
|
"uncertainty": []
|
|
34
36
|
},
|
|
35
37
|
"meta": {
|
|
@@ -65,7 +67,7 @@ Output is a fixed JSON shape:
|
|
|
65
67
|
`--extra-body` is how vendor-specific knobs get through, turning thinking off
|
|
66
68
|
being the common one. It applies to the three API providers and replaces the
|
|
67
69
|
configured `extraBody` for that run. Per-vendor spellings and the fields it
|
|
68
|
-
refuses to touch are in [Configuration](configure.md).
|
|
70
|
+
refuses to touch are in [Configuration](../skills/modlens/references/configure.md).
|
|
69
71
|
|
|
70
72
|
The default `-m` model depends on the provider:
|
|
71
73
|
|
|
@@ -88,11 +90,11 @@ The default `-m` model depends on the provider:
|
|
|
88
90
|
| `--harness <name>` | Force storage scope: `claude-code`, `pi`, `opencode`, `none` | auto-detect |
|
|
89
91
|
| `--cwd <path>` | Project directory the image was pasted in | current directory |
|
|
90
92
|
|
|
91
|
-
Five providers: `antigravity-cli` (no key), `gemini-api` (fastest free route), `openai` (any OpenAI-compatible multimodal endpoint), `anthropic`, and `claude-cli` (uses your existing Claude subscription). Without `-p`, a run tries every provider that is set up, inline API providers first (5-10s), then the agents; the first good result wins and `meta.attempts` records the rest. Harnesses granted via `reuse.<harness>` contribute reused engines to the same regions (pi credentials inline, agent CLIs behind), with no priority over the user's own; details and the `guards` deny/allow lists are in [Configuration](configure.md).
|
|
93
|
+
Five providers: `antigravity-cli` (no key), `gemini-api` (fastest free route), `openai` (any OpenAI-compatible multimodal endpoint), `anthropic`, and `claude-cli` (uses your existing Claude subscription). Without `-p`, a run tries every provider that is set up, inline API providers first (5-10s), then the agents; the first good result wins and `meta.attempts` records the rest. Harnesses granted via `reuse.<harness>` contribute reused engines to the same regions (pi credentials inline, agent CLIs behind), with no priority over the user's own; details and the `guards` deny/allow lists are in [Configuration](../skills/modlens/references/configure.md).
|
|
92
94
|
|
|
93
95
|
Other subcommands:
|
|
94
96
|
|
|
95
97
|
- `modlens guard [--model <id>]`: should the engine run for the active model at all? Exit 0 allow, 1 deny, verdict as JSON.
|
|
96
|
-
- `modlens config <init|set|show>`: keys are `provider`, `reuse.<claude|codex|opencode|pi|grok>`, `guards.<denyModels|allowModels|denyWhenUnknown>`, and `<provider>.<apiKey|baseUrl|model|extraBody>`.
|
|
98
|
+
- `modlens config <init|set|show>`: keys are `provider`, `proxy` (HTTP/HTTPS proxy for the API providers, `HTTPS_PROXY`/`HTTP_PROXY` also honored), `reuse.<claude|codex|opencode|pi|grok>`, `guards.<denyModels|allowModels|denyWhenUnknown>`, and `<provider>.<apiKey|baseUrl|model|proxy|extraBody>`.
|
|
97
99
|
- `modlens doctor`: Node and node:sqlite, provider readiness, the failover chains for this machine, the detected harness, the guard's rules with a live verdict, and the Reuse section with per-harness grant decisions and discovered vision. Spends no quota; `--json` for a machine-readable report.
|
|
98
100
|
|
package/docs/harness-setup.md
CHANGED
|
@@ -18,7 +18,7 @@ One catch with text-only models: once `models.json` declares `input_modalities:
|
|
|
18
18
|
|
|
19
19
|
## Claude Code, Pi, OpenCode
|
|
20
20
|
|
|
21
|
-
None of them
|
|
21
|
+
None of them hands the model a usable temp-file path the way Codex does (newer Claude Code builds do write pastes to their own `~/.claude/image-cache/`, injected as a path line only in the terminal entrypoint), but all three persist the user message locally before any gateway strips it:
|
|
22
22
|
|
|
23
23
|
| Harness | Storage | Notes |
|
|
24
24
|
| :-- | :-- | :-- |
|
|
@@ -53,7 +53,7 @@ OpenCode with DeepSeek: `opencode auth login`, pick DeepSeek and paste the key (
|
|
|
53
53
|
dsh is different from the other harnesses: modlens plugs in as a native tool, not a prompt-triggered skill. The package itself is a dsh bundle, so one command installs it into a profile:
|
|
54
54
|
|
|
55
55
|
```sh
|
|
56
|
-
npx -y @deepseek-ai/dsh plugin --profile web add @liustack/modlens
|
|
56
|
+
npx -y @deepseek-ai/dsh plugin --profile web add @liustack/modlens@latest
|
|
57
57
|
```
|
|
58
58
|
|
|
59
|
-
This registers a `read_image` tool whose schema reaches the model on every request (no trigger heuristics), runs the modlens CLI shipped inside the same package, and returns the structured evidence as the tool's canonical JSON output. Engines, reuse grants, and guard rules stay in `~/.modlens/config.json`, shared with every other harness. dsh is in developer preview and its plugin surface may change; the plugin
|
|
59
|
+
This registers a `read_image` tool whose schema reaches the model on every request (no trigger heuristics), runs the modlens CLI shipped inside the same package, and returns the structured evidence as the tool's canonical JSON output. Engines, reuse grants, and guard rules stay in `~/.modlens/config.json`, shared with every other harness. dsh is in developer preview and its plugin surface may change; the plugin keeps its touch small (raw tool registration, the llm adapter surface for the vision variants, the attachment reader, and one agent pre-step hook) and degrades loudly if any of them moves.
|
package/docs/output-schema.md
CHANGED
|
@@ -67,6 +67,6 @@ The CLI prints one JSON object to stdout:
|
|
|
67
67
|
}
|
|
68
68
|
```
|
|
69
69
|
|
|
70
|
-
Required fields: `summary`, `ocr`, `layout`, `semantics`, `uncertainty
|
|
70
|
+
Required fields: `summary`, `ocr`, `layout`, `semantics`, `visual`, `uncertainty` — every top-level field, `visual` included. (Earlier docs called `visual` optional; the enforced schema has always required it, so build to the schema.)
|
|
71
71
|
|
|
72
72
|
Changes from v1: pixel `bbox` coordinates and numeric `confidence` scores were removed. Vision models fabricate both, so v2 stops pretending to provide them. `layout.regions[].type` gained `code`.
|
package/docs/security.md
CHANGED
|
@@ -22,7 +22,15 @@ The `claude-cli` provider runs with `--allowedTools Read` only, so it can read l
|
|
|
22
22
|
|
|
23
23
|
Both subprocess providers also run in a throwaway directory created fresh per call and removed afterward. For a local image it holds a private copy of that one image and nothing else, and it is a real copy, never a hardlink, so a provider writing to its temp path cannot touch the original. For a remote image the directory is empty and the agent downloads into it. Without this, text inside an image could steer a broadly-permissioned agent into reading files next to the original, or whatever project the caller happened to be in. Passing `--workdir` opts out and runs where you point it.
|
|
24
24
|
|
|
25
|
-
This is exposure reduction, not an OS sandbox: the agent can still read absolute paths, reach the network, and spawn processes. Treat it as a narrower default, not a security boundary. For images you do not trust, prefer
|
|
25
|
+
This is exposure reduction, not an OS sandbox: the agent can still read absolute paths, reach the network, and spawn processes. Treat it as a narrower default, not a security boundary. For images you do not trust, prefer `-p gemini-api`, which downloads the bytes itself and runs no local agent. Remote URLs already prefer the inline region: the failover chain for a remote URL tries the inline API providers first and the agents last. Who actually fetches a remote URL differs per provider, and only a local download can be guarded locally:
|
|
26
|
+
|
|
27
|
+
| Provider | Who fetches a remote URL | Local guards |
|
|
28
|
+
| :-- | :-- | :-- |
|
|
29
|
+
| `gemini-api` | modlens downloads, sends bytes inline | private-address guards, magic-byte check, 25 MB cap |
|
|
30
|
+
| `openai`, `anthropic` | the URL is passed to the vendor, which fetches it | none locally; the vendor's own fetching policy applies |
|
|
31
|
+
| `antigravity-cli`, agent CLIs | the agent fetches on its own | none locally |
|
|
32
|
+
|
|
33
|
+
So the private-address guards, the magic-byte check, and the size cap protect exactly the paths where modlens itself downloads: every local file read, and gemini-api's remote fetch. An explicit `-p` pins one provider and overrides the chain.
|
|
26
34
|
|
|
27
35
|
## Image content is untrusted input
|
|
28
36
|
|
package/docs/troubleshooting.md
CHANGED
|
@@ -119,6 +119,64 @@ One known blind spot: storage detection reads the newest assistant turn recorded
|
|
|
119
119
|
|
|
120
120
|
Note that the hard refusal above only fires on an actual `denyModels` match against the explicit `MODLENS_MODEL` value. Storage detection and the `denyWhenUnknown` policy never block `analyze`, they only speak through `modlens guard`, whose deny is advice to the agent rather than a locked door.
|
|
121
121
|
|
|
122
|
+
## dsh says "declares no dsh.bundle — installed as a plain dependency"
|
|
123
|
+
|
|
124
|
+
The dsh profile installed an old modlens version. The `dsh.bundle` declaration
|
|
125
|
+
exists since 3.9.0, and pnpm v11's release-age gate (`minimumReleaseAge`,
|
|
126
|
+
quarantining recently published versions, with a 10-day window measured on pnpm 11.21) silently falls back to an older version when every
|
|
127
|
+
recent one is inside the window. That old version has no bundle declaration,
|
|
128
|
+
so dsh correctly treats it as a plain dependency and none of the tools appear.
|
|
129
|
+
|
|
130
|
+
The fix: name the version explicitly. pnpm applies the age gate when resolving
|
|
131
|
+
a range, but an explicit version or dist-tag skips it ([pnpm#9989](https://github.com/pnpm/pnpm/issues/9989), verified on pnpm 11.21), which is why the install
|
|
132
|
+
command carries `@latest`:
|
|
133
|
+
|
|
134
|
+
```sh
|
|
135
|
+
npx -y @deepseek-ai/dsh plugin --profile <name> add @liustack/modlens@latest
|
|
136
|
+
```
|
|
137
|
+
|
|
138
|
+
dsh's reconcile notices the bundle declaration on the new version and
|
|
139
|
+
activates it; restart dsh afterwards. Verify with
|
|
140
|
+
`npx -y @deepseek-ai/dsh plugin --profile <name> list` — the version shown
|
|
141
|
+
should be 3.9.0 or newer.
|
|
142
|
+
|
|
143
|
+
If a future pnpm closes that skip, the durable alternative is a one-time
|
|
144
|
+
exclusion in `~/.dsh/profiles/<name>/pnpm-workspace.yaml` — the bare package
|
|
145
|
+
name, not `name@version`, so it survives future releases:
|
|
146
|
+
|
|
147
|
+
```yaml
|
|
148
|
+
minimumReleaseAgeExclude:
|
|
149
|
+
- '@liustack/modlens'
|
|
150
|
+
```
|
|
151
|
+
|
|
152
|
+
then `npx -y @deepseek-ai/dsh plugin --profile <name> update @liustack/modlens`.
|
|
153
|
+
The trade-off is honest either way: an explicit `@latest` (or the exclusion)
|
|
154
|
+
opts modlens out of pnpm's supply-chain cooling-off window, so new releases
|
|
155
|
+
install immediately.
|
|
156
|
+
|
|
157
|
+
## fetch failed, or could not connect
|
|
158
|
+
|
|
159
|
+
```
|
|
160
|
+
Could not connect to generativelanguage.googleapis.com (UND_ERR_CONNECT_TIMEOUT). The request never reached the network. ...
|
|
161
|
+
```
|
|
162
|
+
|
|
163
|
+
The API request never left the machine. On networks that reach the internet
|
|
164
|
+
through a proxy this is expected: Node's fetch ignores the proxy environment
|
|
165
|
+
variables by default. modlens honors them once you ask it to route that way,
|
|
166
|
+
in either form:
|
|
167
|
+
|
|
168
|
+
```bash
|
|
169
|
+
HTTPS_PROXY=http://127.0.0.1:7890 modlens -i shot.png -p gemini-api # env (NO_PROXY honored too)
|
|
170
|
+
modlens config set proxy http://127.0.0.1:7890 # persistent, all API providers
|
|
171
|
+
modlens config set openai.proxy http://127.0.0.1:7890 # one provider only
|
|
172
|
+
```
|
|
173
|
+
|
|
174
|
+
The proxy applies to API provider requests only. The remote-image download
|
|
175
|
+
path keeps its direct, IP-pinned connection on purpose: its SSRF guards
|
|
176
|
+
validate the exact address being contacted, and a proxy would blind them. On
|
|
177
|
+
a proxied machine, prefer local files or let the failover chain hand remote
|
|
178
|
+
URLs to a provider that fetches them upstream.
|
|
179
|
+
|
|
122
180
|
## Config file problems
|
|
123
181
|
|
|
124
182
|
```
|
package/dsh/index.js
CHANGED
|
@@ -24,11 +24,13 @@ const CLI_TIMEOUT_MS = 180_000
|
|
|
24
24
|
export const name = 'modlens'
|
|
25
25
|
export const inject = ['tools', 'agents', 'attachments', 'llm']
|
|
26
26
|
|
|
27
|
-
const MEDIA_EXT = {
|
|
27
|
+
export const MEDIA_EXT = {
|
|
28
28
|
'image/png': '.png',
|
|
29
29
|
'image/jpeg': '.jpg',
|
|
30
30
|
'image/webp': '.webp',
|
|
31
31
|
'image/gif': '.gif',
|
|
32
|
+
'image/heic': '.heic',
|
|
33
|
+
'image/heif': '.heif',
|
|
32
34
|
}
|
|
33
35
|
|
|
34
36
|
export function apply(ctx, config = {}) {
|
|
@@ -191,6 +193,84 @@ function registerVisionProvider(ctx, config) {
|
|
|
191
193
|
}
|
|
192
194
|
}
|
|
193
195
|
|
|
196
|
+
// The same pasted attachment rides every later step of its session, but the
|
|
197
|
+
// cache must never make a failure permanent or run the engine twice for
|
|
198
|
+
// concurrent steps. So it stores promises (concurrent readers join the first
|
|
199
|
+
// run), evicts failed reads on settle (a fixed config gets a fresh chance),
|
|
200
|
+
// and caps itself LRU-style so a long-lived Web profile cannot hoard
|
|
201
|
+
// evidence text forever.
|
|
202
|
+
const EVIDENCE_CACHE_LIMIT = 64
|
|
203
|
+
|
|
204
|
+
function cachedEvidence(ctx, adapter, block) {
|
|
205
|
+
const key = JSON.stringify(block.attachment ?? block)
|
|
206
|
+
const hit = adapter.evidenceCache.get(key)
|
|
207
|
+
if (hit !== undefined) {
|
|
208
|
+
// Refresh recency: Map iteration order is insertion order.
|
|
209
|
+
adapter.evidenceCache.delete(key)
|
|
210
|
+
adapter.evidenceCache.set(key, hit)
|
|
211
|
+
return hit
|
|
212
|
+
}
|
|
213
|
+
// Deliberately no caller signal: a shared entry must not die with its first
|
|
214
|
+
// caller (their abort used to cancel every concurrent joiner). A cancelled
|
|
215
|
+
// caller simply stops awaiting; the read finishes and the cache keeps it.
|
|
216
|
+
const pending = readImageBlock(ctx, block, undefined).then(
|
|
217
|
+
(evidence) => {
|
|
218
|
+
// Only evict our own entry: this promise may have been LRU-evicted and
|
|
219
|
+
// the key re-populated by a newer read meanwhile.
|
|
220
|
+
if (!evidence.ok && adapter.evidenceCache.get(key) === pending) {
|
|
221
|
+
adapter.evidenceCache.delete(key)
|
|
222
|
+
}
|
|
223
|
+
return evidence.block
|
|
224
|
+
},
|
|
225
|
+
(error) => {
|
|
226
|
+
// readImageBlock never rejects by contract; this is the belt for a
|
|
227
|
+
// future refactor breaking that, so a rejected promise cannot lodge in
|
|
228
|
+
// the cache forever.
|
|
229
|
+
if (adapter.evidenceCache.get(key) === pending) {
|
|
230
|
+
adapter.evidenceCache.delete(key)
|
|
231
|
+
}
|
|
232
|
+
return {
|
|
233
|
+
type: 'text',
|
|
234
|
+
text: `[A pasted image could not be read by modlens: ${
|
|
235
|
+
error instanceof Error ? error.message.slice(0, 300) : String(error)
|
|
236
|
+
}]`,
|
|
237
|
+
}
|
|
238
|
+
},
|
|
239
|
+
)
|
|
240
|
+
adapter.evidenceCache.set(key, pending)
|
|
241
|
+
while (adapter.evidenceCache.size > EVIDENCE_CACHE_LIMIT) {
|
|
242
|
+
adapter.evidenceCache.delete(adapter.evidenceCache.keys().next().value)
|
|
243
|
+
}
|
|
244
|
+
return pending
|
|
245
|
+
}
|
|
246
|
+
|
|
247
|
+
/**
|
|
248
|
+
* Wait on a shared promise without inheriting its lifetime: the caller's
|
|
249
|
+
* abort rejects THIS wait immediately, while the underlying read keeps
|
|
250
|
+
* running and lands in the cache for the retry.
|
|
251
|
+
*/
|
|
252
|
+
function abortableWait(promise, signal) {
|
|
253
|
+
if (!signal) return promise
|
|
254
|
+
return new Promise((resolve, reject) => {
|
|
255
|
+
if (signal.aborted) {
|
|
256
|
+
reject(signal.reason ?? new Error('aborted'))
|
|
257
|
+
return
|
|
258
|
+
}
|
|
259
|
+
const onAbort = () => reject(signal.reason ?? new Error('aborted'))
|
|
260
|
+
signal.addEventListener('abort', onAbort, { once: true })
|
|
261
|
+
promise.then(
|
|
262
|
+
(value) => {
|
|
263
|
+
signal.removeEventListener('abort', onAbort)
|
|
264
|
+
resolve(value)
|
|
265
|
+
},
|
|
266
|
+
(error) => {
|
|
267
|
+
signal.removeEventListener('abort', onAbort)
|
|
268
|
+
reject(error)
|
|
269
|
+
},
|
|
270
|
+
)
|
|
271
|
+
})
|
|
272
|
+
}
|
|
273
|
+
|
|
194
274
|
async function convertImagesToEvidence(ctx, messages, signal, adapter) {
|
|
195
275
|
const out = []
|
|
196
276
|
for (const message of messages) {
|
|
@@ -204,13 +284,7 @@ async function convertImagesToEvidence(ctx, messages, signal, adapter) {
|
|
|
204
284
|
content.push(block)
|
|
205
285
|
continue
|
|
206
286
|
}
|
|
207
|
-
|
|
208
|
-
let text = adapter.evidenceCache.get(key)
|
|
209
|
-
if (text === undefined) {
|
|
210
|
-
text = (await readImageBlock(ctx, block, signal)).text
|
|
211
|
-
adapter.evidenceCache.set(key, text)
|
|
212
|
-
}
|
|
213
|
-
content.push({ type: 'text', text })
|
|
287
|
+
content.push(await abortableWait(cachedEvidence(ctx, adapter, block), signal))
|
|
214
288
|
}
|
|
215
289
|
out.push({ ...message, content })
|
|
216
290
|
}
|
|
@@ -251,7 +325,7 @@ function registerAutoRead(ctx) {
|
|
|
251
325
|
content.push(block)
|
|
252
326
|
continue
|
|
253
327
|
}
|
|
254
|
-
content.push(await readImageBlock(ctx, block, payload.signal))
|
|
328
|
+
content.push((await readImageBlock(ctx, block, payload.signal)).block)
|
|
255
329
|
}
|
|
256
330
|
messages.push({ ...message, content })
|
|
257
331
|
}
|
|
@@ -259,6 +333,12 @@ function registerAutoRead(ctx) {
|
|
|
259
333
|
})
|
|
260
334
|
}
|
|
261
335
|
|
|
336
|
+
/**
|
|
337
|
+
* Read one image block into an evidence text block. Never throws: failures
|
|
338
|
+
* degrade to an explanatory block with `ok: false`, so callers can decide
|
|
339
|
+
* what a failure means (the pre-step keeps the step going, the cache refuses
|
|
340
|
+
* to memoize it).
|
|
341
|
+
*/
|
|
262
342
|
async function readImageBlock(ctx, block, signal) {
|
|
263
343
|
const { mkdtemp, rm, writeFile } = await import('node:fs/promises')
|
|
264
344
|
const { tmpdir } = await import('node:os')
|
|
@@ -275,9 +355,15 @@ async function readImageBlock(ctx, block, signal) {
|
|
|
275
355
|
"attachments.readImage returned no 'data' bytes; the dsh attachment shape may have changed",
|
|
276
356
|
)
|
|
277
357
|
}
|
|
278
|
-
dir = await mkdtemp(join(tmpdir(), 'modlens-dsh-'))
|
|
279
358
|
const mediaType = stored.ref?.mediaType ?? block.attachment?.mediaType
|
|
280
|
-
const
|
|
359
|
+
const ext = MEDIA_EXT[mediaType]
|
|
360
|
+
if (!ext) {
|
|
361
|
+
// Refusing beats disguising: a fake .png suffix would make the CLI (and
|
|
362
|
+
// the provider behind it) judge mislabelled bytes.
|
|
363
|
+
throw new Error(`unsupported pasted media type ${mediaType ?? '(none declared)'}`)
|
|
364
|
+
}
|
|
365
|
+
dir = await mkdtemp(join(tmpdir(), 'modlens-dsh-'))
|
|
366
|
+
const file = join(dir, `paste${ext}`)
|
|
281
367
|
await writeFile(file, Buffer.from(stored.data), { mode: 0o600 })
|
|
282
368
|
const cli = process.env.MODLENS_DSH_CLI || CLI_PATH
|
|
283
369
|
const { stdout, stderr, code } = await run(
|
|
@@ -290,15 +376,21 @@ async function readImageBlock(ctx, block, signal) {
|
|
|
290
376
|
}
|
|
291
377
|
const parsed = JSON.parse(stdout)
|
|
292
378
|
return {
|
|
293
|
-
|
|
294
|
-
|
|
379
|
+
ok: true,
|
|
380
|
+
block: {
|
|
381
|
+
type: 'text',
|
|
382
|
+
text: `[Pasted image, read by the modlens vision bridge]\n${renderEvidence(parsed.result)}`,
|
|
383
|
+
},
|
|
295
384
|
}
|
|
296
385
|
} catch (error) {
|
|
297
386
|
return {
|
|
298
|
-
|
|
299
|
-
|
|
300
|
-
|
|
301
|
-
|
|
387
|
+
ok: false,
|
|
388
|
+
block: {
|
|
389
|
+
type: 'text',
|
|
390
|
+
text: `[A pasted image could not be read by modlens: ${
|
|
391
|
+
error instanceof Error ? error.message.slice(0, 300) : String(error)
|
|
392
|
+
}. Tell the user, and suggest running \`npx @liustack/modlens doctor\`.]`,
|
|
393
|
+
},
|
|
302
394
|
}
|
|
303
395
|
} finally {
|
|
304
396
|
if (dir) {
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@liustack/modlens",
|
|
3
|
-
"version": "3.
|
|
3
|
+
"version": "3.12.0",
|
|
4
4
|
"description": "Plug-in vision for text-only LLMs, powered by the free Antigravity CLI",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
|
@@ -12,7 +12,7 @@
|
|
|
12
12
|
"typecheck": "tsc --noEmit",
|
|
13
13
|
"test": "vitest run",
|
|
14
14
|
"coverage": "vitest run --coverage",
|
|
15
|
-
"lint": "biome check src scripts",
|
|
15
|
+
"lint": "biome check src scripts dsh",
|
|
16
16
|
"format": "biome check --write src scripts",
|
|
17
17
|
"eval": "node evals/run.mjs",
|
|
18
18
|
"release": "node scripts/release.mjs",
|
package/skills/modlens/SKILL.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: modlens
|
|
3
|
-
description: "Plug-in vision for text-only models. Hard rule: when a file path or URL with an image extension (.png, .jpg, .jpeg, .webp, .gif, .
|
|
3
|
+
description: "Plug-in vision for text-only models. Hard rule: when a file path or URL with an image extension (.png, .jpg, .jpeg, .webp, .gif, .heic, .heif) appears anywhere in the conversation (typed by the user, injected as a `[Image: source: <path>]` line, or inside a tag) and you cannot see that image's content, run this skill on it before any other approach: no self-built OCR, no PIL, no tesseract. Also triggers on pasted-image placeholders such as `[Image #1]` and `[Unsupported Image]`. If you can actually see the image, do not use this skill. When unsure, run `modlens guard` before the first read of a session: a deny verdict means the active model has native vision and must read the image itself. Runs the modlens CLI to convert the image into structured JSON evidence: every word transcribed, layout regions, semantics, visual clues. Also use when the user asks how to install, configure, or switch modlens providers (Gemini API key, OpenAI-compatible endpoints, Claude API or Claude Code CLI)."
|
|
4
4
|
compatibility: Requires network access and one of node 22+/npx, bun/bunx, or a preinstalled modlens binary on PATH.
|
|
5
5
|
allowed-tools: Bash
|
|
6
6
|
---
|
|
@@ -20,11 +20,11 @@ powershell -ExecutionPolicy Bypass -File <skill-dir>\scripts\run.ps1 <args>
|
|
|
20
20
|
|
|
21
21
|
It resolves a working runtime (PATH `modlens`, then `npx`, then `bunx`) and forwards your arguments unchanged. Exit 78 means no runtime: relay the `nextSteps` from its stderr JSON instead of retrying.
|
|
22
22
|
|
|
23
|
-
If your harness forbids running scripts, reason through the same order by hand and run the first line that works (the pinned version is 3.
|
|
23
|
+
If your harness forbids running scripts, reason through the same order by hand and run the first line that works (the pinned version is 3.12.0):
|
|
24
24
|
|
|
25
|
-
1. A `modlens` on `PATH` whose major version is 3 and is at least 3.
|
|
26
|
-
2. Otherwise, if `npx` exists: `npx --yes --package @liustack/modlens@3.
|
|
27
|
-
3. Otherwise, if `bunx` exists: `bunx --bun @liustack/modlens@3.
|
|
25
|
+
1. A `modlens` on `PATH` whose major version is 3 and is at least 3.12.0: `modlens <args>`.
|
|
26
|
+
2. Otherwise, if `npx` exists: `npx --yes --package @liustack/modlens@3.12.0 modlens <args>`.
|
|
27
|
+
3. Otherwise, if `bunx` exists: `bunx --bun @liustack/modlens@3.12.0 <args>`.
|
|
28
28
|
4. Otherwise tell the user no JavaScript runtime was found and that installing Node 22.13+ (https://nodejs.org) or Bun (https://bun.sh) is the next step. Do not claim modlens itself failed.
|
|
29
29
|
|
|
30
30
|
`references/runtime.md` documents the pin and the diagnostic fields.
|
|
@@ -22,7 +22,8 @@ Everything lives under four top-level keys, all optional. This example shows eve
|
|
|
22
22
|
```json
|
|
23
23
|
{
|
|
24
24
|
"provider": "gemini-api",
|
|
25
|
-
"
|
|
25
|
+
"proxy": "http://127.0.0.1:7890",
|
|
26
|
+
"reuse": { "claude": true, "codex": true, "opencode": false, "pi": true, "grok": true },
|
|
26
27
|
"guards": {
|
|
27
28
|
"allowModels": ["deepseek-v4-*", "glm-5.*", "minimax-m2.5*", "qwen3-coder*"],
|
|
28
29
|
"denyModels": ["glm-*v*", "deepseek-vl*"],
|
|
@@ -18,7 +18,7 @@ Newer Claude Code builds write every pasted image to `~/.claude/image-cache/<ses
|
|
|
18
18
|
|
|
19
19
|
## Claude Code, Pi, or OpenCode (no usable path anywhere)
|
|
20
20
|
|
|
21
|
-
The image reads as `[Unsupported Image]`, a bare `[Image #1]`, or an attachment you simply cannot see. Whatever a gateway strips from the request, these harnesses persist user messages, image bytes included, in local session storage first: Claude Code and Pi in session JSONL files (`~/.claude/projects/`, `~/.pi/agent/sessions/`), OpenCode in a SQLite database (`~/.local/share/opencode/opencode.db`, read via node:sqlite, needs Node 22.
|
|
21
|
+
The image reads as `[Unsupported Image]`, a bare `[Image #1]`, or an attachment you simply cannot see. Whatever a gateway strips from the request, these harnesses persist user messages, image bytes included, in local session storage first: Claude Code and Pi in session JSONL files (`~/.claude/projects/`, `~/.pi/agent/sessions/`), OpenCode in a SQLite database (`~/.local/share/opencode/opencode.db`, read via node:sqlite, needs Node 22.13+; Bun cannot load node:sqlite, so a bunx-resolved launcher needs a real Node install for OpenCode recovery).
|
|
22
22
|
|
|
23
23
|
- Run `modlens recover-paste` from the project directory the conversation is happening in (`--count <n>` for several images). It detects which harness it is running inside and reads ONLY that harness's storage. In Claude Code it targets your exact session automatically via the injected `CLAUDE_CODE_SESSION_ID`; `--session <id>` is only needed to override.
|
|
24
24
|
- The output is JSON with real file paths, oldest to newest, so the LAST path is the most recent paste: analyze that one first. Entries carry `filename` when the harness stored one; match on it when the user's message mentions a name.
|
|
@@ -8,7 +8,7 @@ Run this flow when `~/.modlens/config.json` does not exist, or `modlens config s
|
|
|
8
8
|
modlens doctor --json
|
|
9
9
|
```
|
|
10
10
|
|
|
11
|
-
Read three things from the report:
|
|
11
|
+
Read three things from the report: each provider's `status` (`ready` means verified, `installed` means a CLI is on PATH with its sign-in unverified until the first real read, rendered as `[ok?]` in the text report), the Reuse section (per-harness decisions plus discovered logins and vision models, where the harness this conversation runs inside is itself the first reusable engine), and the guard state. Doctor spends no quota and makes no network calls.
|
|
12
12
|
|
|
13
13
|
## 2. Tell the user what their machine already has
|
|
14
14
|
|
|
@@ -16,9 +16,9 @@ One line per finding, plain words, in the user's language. Name concrete things,
|
|
|
16
16
|
|
|
17
17
|
- An engine is ready: "modlens is ready to go: <provider> is configured (via <env var / config file / existing login>)."
|
|
18
18
|
- Reusable vision found: "Your <harness> CLI is signed in and its model can read images. modlens can reuse it when needed, about <n> seconds per read, and it spends that account's quota."
|
|
19
|
-
- Nothing at all: "No vision engine is set up yet. The fastest free option is a Gemini API key (three minutes, no card)
|
|
19
|
+
- Nothing at all: "No vision engine is set up yet. The fastest free option is a Gemini API key (three minutes, no card). Antigravity CLI works with no sign-up at all."
|
|
20
20
|
|
|
21
|
-
Do not dump the raw doctor output on the user
|
|
21
|
+
Do not dump the raw doctor output on the user, summarize it. Do not describe options the machine does not have.
|
|
22
22
|
|
|
23
23
|
## 3. Ask before touching anything
|
|
24
24
|
|
|
@@ -27,7 +27,7 @@ Consent rules:
|
|
|
27
27
|
- One question per decision, never a bundled yes. Reusing Codex and reusing pi credentials are two questions (or one question with independent options), not one.
|
|
28
28
|
- Each question names the harness, whose quota it spends, and the accounting promise. Example wording: "Allow modlens to reuse your signed-in Codex CLI for image reads? Every reused read is labeled in the result so you always see whose quota was spent."
|
|
29
29
|
- The do-nothing outcome must be safe and stated: "If you skip this, modlens just uses the engines you configure yourself."
|
|
30
|
-
- If the user offers an API key, take exactly that key
|
|
30
|
+
- If the user offers an API key, take exactly that key, and never go looking for keys they did not hand over.
|
|
31
31
|
|
|
32
32
|
## 4. Apply only what was consented to
|
|
33
33
|
|
|
@@ -24,7 +24,7 @@ $ErrorActionPreference = 'Stop'
|
|
|
24
24
|
# package.json version, and the release script rewrites it on every bump.
|
|
25
25
|
$Package = '@liustack/modlens'
|
|
26
26
|
$Bin = 'modlens'
|
|
27
|
-
$Pinned = '3.
|
|
27
|
+
$Pinned = '3.12.0'
|
|
28
28
|
# -------------------------------------------------------------------------------
|
|
29
29
|
|
|
30
30
|
$NativeNote = 'no native artifact is published for this tool yet; phase A ships npm launch paths only'
|
|
@@ -22,7 +22,7 @@ set -eu
|
|
|
22
22
|
# package.json version, and the release script rewrites it on every bump.
|
|
23
23
|
PKG="@liustack/modlens"
|
|
24
24
|
BIN="modlens"
|
|
25
|
-
PINNED="3.
|
|
25
|
+
PINNED="3.12.0"
|
|
26
26
|
# -------------------------------------------------------------------------------
|
|
27
27
|
|
|
28
28
|
NATIVE_NOTE="no native artifact is published for this tool yet; phase A ships npm launch paths only"
|