@liustack/modlens 2.1.0 → 2.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +12 -1
- package/README.zh-CN.md +12 -1
- package/dist/main.js +206 -11
- package/package.json +1 -1
- package/skills/modlens/SKILL.md +8 -5
- package/skills/modlens/references/configure.md +85 -0
package/README.md
CHANGED
|
@@ -90,6 +90,10 @@ Batch mode works too: drop three illustrations at once, and the model announces
|
|
|
90
90
|
|
|
91
91
|

|
|
92
92
|
|
|
93
|
+
Stress test: a scatter plot of 128 models. ModLens pulls out the axes, the log scale, and the highlighted DeepSeek V4 Flash point at $0.028 and score 50, then walks through the cost-performance cutoff line. Dense charts are where vision models usually fold; this one holds.
|
|
94
|
+
|
|
95
|
+

|
|
96
|
+
|
|
93
97
|
## CLI reference
|
|
94
98
|
|
|
95
99
|
```bash
|
|
@@ -111,7 +115,7 @@ Reach for `-m gemini-3.1-pro-high` on dense screenshots or tricky documents. Out
|
|
|
111
115
|
|
|
112
116
|
## Providers and config
|
|
113
117
|
|
|
114
|
-
ModLens ships
|
|
118
|
+
ModLens ships five vision providers. `antigravity-cli` stays the default: zero keys, pure free quota.
|
|
115
119
|
|
|
116
120
|
| Provider | Needs | Typical speed | Notes |
|
|
117
121
|
| :-- | :-- | :-- | :-- |
|
|
@@ -119,6 +123,7 @@ ModLens ships four vision providers. `antigravity-cli` stays the default: zero k
|
|
|
119
123
|
| `gemini-api` | free AI Studio key | 5-10s | fastest free route, schema enforced server-side |
|
|
120
124
|
| `openai` | baseUrl + apiKey + model | endpoint-dependent | any OpenAI-compatible multimodal endpoint (qwen-vl, GLM, ...) |
|
|
121
125
|
| `anthropic` | `ANTHROPIC_API_KEY` | a few seconds | Claude Haiku by default, schema via forced tool call |
|
|
126
|
+
| `claude-cli` | Claude Code signed in | 20-45s | no key, rides your Claude subscription, Read-only permissions |
|
|
122
127
|
|
|
123
128
|
Config lives in `~/.modlens/config.json`. Environment variables override the file (`GEMINI_API_KEY`, `OPENAI_API_KEY`, `OPENAI_BASE_URL`, `ANTHROPIC_API_KEY`), and CLI flags override everything.
|
|
124
129
|
|
|
@@ -140,6 +145,12 @@ One catch: once text-only is declared, the Codex TUI **blocks Ctrl+V image paste
|
|
|
140
145
|
- **Drag the image file into the terminal**, or type its path. The path lands as plain text, and the modlens skill picks it up from there.
|
|
141
146
|
- Attach it with `codex exec -i image.png "..."`. The skill reads the path out of the message tag.
|
|
142
147
|
|
|
148
|
+
## Using it in Claude Code (gateway models)
|
|
149
|
+
|
|
150
|
+
No setup needed: drag the image file into the terminal, or type its path, and the skill takes over.
|
|
151
|
+
|
|
152
|
+
Paste is trickier. If you run a text-only model behind `ANTHROPIC_BASE_URL`, Claude Code never writes pasted images to a regular temp file and has no modality switch, so a pasted image reaches the model as a pathless `[Unsupported Image]` placeholder (lenient gateways like DeepSeek's Anthropic endpoint) or breaks the request outright ([#62009](https://github.com/anthropics/claude-code/issues/62009)). But the bytes are not gone: Claude Code appends every user message, images included, to the local session transcript before the gateway ever sees it. That is what `modlens recover-paste` exploits: it pulls the most recent pasted images back out of the transcript and prints real file paths, ready for `modlens -i`. The skill runs this automatically when it spots the placeholder. One honest caveat: the transcript layout is Claude Code internals with no compatibility promise; if recovery ever breaks, dragging the file still works everywhere.
|
|
153
|
+
|
|
143
154
|
## Why a bridge instead of a multimodal model?
|
|
144
155
|
|
|
145
156
|
- **Keep your model.** You picked DeepSeek-V4-Flash (or gpt-oss, or whatever else) for its price and its reasoning, not its eyesight. ModLens adds sight without touching that choice.
|
package/README.zh-CN.md
CHANGED
|
@@ -90,6 +90,10 @@ npx @liustack/modlens -i workflow.jpg
|
|
|
90
90
|
|
|
91
91
|

|
|
92
92
|
|
|
93
|
+
压力测试:一张 128 个模型的智能对成本散点图。ModLens 读出双轴、对数刻度,把高亮的 DeepSeek V4 Flash 精准拎出来(成本约 $0.028、智能指数 50),还讲明白了性价比斩杀线。密集图表是识图模型最容易露怯的地方,这一关它扛住了。
|
|
94
|
+
|
|
95
|
+

|
|
96
|
+
|
|
93
97
|
## CLI 参数
|
|
94
98
|
|
|
95
99
|
```bash
|
|
@@ -111,7 +115,7 @@ modlens -i <图片路径或 URL> [选项]
|
|
|
111
115
|
|
|
112
116
|
## Provider 与配置
|
|
113
117
|
|
|
114
|
-
ModLens
|
|
118
|
+
ModLens 内置五个视觉 provider,默认还是 `antigravity-cli`:零 key,纯免费额度。
|
|
115
119
|
|
|
116
120
|
| Provider | 需要什么 | 速度 | 说明 |
|
|
117
121
|
| :-- | :-- | :-- | :-- |
|
|
@@ -119,6 +123,7 @@ ModLens 内置四个视觉 provider,默认还是 `antigravity-cli`:零 key
|
|
|
119
123
|
| `gemini-api` | 免费 AI Studio key | 5-10 秒 | 最快的免费路线,服务端强制 schema |
|
|
120
124
|
| `openai` | baseUrl + apiKey + model | 看端点 | 任何 OpenAI 兼容的多模态端点(qwen-vl、GLM 等) |
|
|
121
125
|
| `anthropic` | `ANTHROPIC_API_KEY` | 几秒 | 默认 Claude Haiku,强制工具调用保 schema |
|
|
126
|
+
| `claude-cli` | Claude Code 已登录 | 20-45 秒 | 零 key,吃你的 Claude 订阅额度,只放行 Read 工具 |
|
|
122
127
|
|
|
123
128
|
配置放在 `~/.modlens/config.json`,环境变量能盖过它(`GEMINI_API_KEY`、`OPENAI_API_KEY`、`OPENAI_BASE_URL`、`ANTHROPIC_API_KEY`),CLI 参数最大。
|
|
124
129
|
|
|
@@ -140,6 +145,12 @@ Codex 只认 Responses API,DeepSeek 官方端点原生支持。先照着[官
|
|
|
140
145
|
- **把图片文件拖进终端**,或者手打路径。路径以纯文本形式落进消息,modlens skill 接着从这里接手。
|
|
141
146
|
- 用 `codex exec -i 图片.png "..."` skill 从这里把路径抠出来。
|
|
142
147
|
|
|
148
|
+
## 在 Claude Code 里用(网关接第三方模型)
|
|
149
|
+
|
|
150
|
+
不用任何配置:把图片文件拖进终端,或手打路径,skill 直接接手。
|
|
151
|
+
|
|
152
|
+
粘贴要多说两句。走 `ANTHROPIC_BASE_URL` 网关跑纯文本模型时,Claude Code 粘贴的图片从不写普通临时文件,也没有声明模型无视觉的开关,粘贴的图要么变成一个不带路径的 `[Unsupported Image]` 占位符到达模型(DeepSeek 的 Anthropic 兼容端点这类宽容网关),要么直接把请求搞挂([#62009](https://github.com/anthropics/claude-code/issues/62009))。但图片字节没有蒸发:Claude Code 在网关看到消息之前,就把每条用户消息(含图片)原样写进了本地会话记录。`modlens recover-paste` 干的就是这件事:从会话记录里把最近粘贴的图捞回来,落成真实文件路径,直接喂给 `modlens -i`。skill 看到占位符会自动跑这一步。一句老实话:会话记录格式是 Claude Code 的内部实现,没有兼容承诺,哪天捞不动了,拖文件永远是保底。
|
|
153
|
+
|
|
143
154
|
## 为什么外挂,而不是换多模态模型?
|
|
144
155
|
|
|
145
156
|
- **模型不用换。** 你选 DeepSeek-V4-Flash(或 gpt-oss,或别的什么)图的是价格和推理能力,不是视力。ModLens 只加视力,不碰这个选择。
|
package/dist/main.js
CHANGED
|
@@ -4,6 +4,7 @@ import * as fs from "fs";
|
|
|
4
4
|
import * as path from "path";
|
|
5
5
|
import { spawn } from "child_process";
|
|
6
6
|
import * as os from "os";
|
|
7
|
+
import * as crypto from "crypto";
|
|
7
8
|
const CONFIG_DIR = path.join(os.homedir(), ".modlens");
|
|
8
9
|
const CONFIG_PATH = path.join(CONFIG_DIR, "config.json");
|
|
9
10
|
const ENV_BINDINGS = {
|
|
@@ -79,7 +80,8 @@ const CONFIG_TEMPLATE = {
|
|
|
79
80
|
"antigravity-cli": { model: "gemini-3.6-flash-low" },
|
|
80
81
|
"gemini-api": { apiKey: "", model: "gemini-3.6-flash" },
|
|
81
82
|
openai: { baseUrl: "", apiKey: "", model: "" },
|
|
82
|
-
anthropic: { apiKey: "", model: "claude-haiku-4-5-20251001" }
|
|
83
|
+
anthropic: { apiKey: "", model: "claude-haiku-4-5-20251001" },
|
|
84
|
+
"claude-cli": { model: "haiku" }
|
|
83
85
|
}
|
|
84
86
|
};
|
|
85
87
|
function initConfigFile(configPath = CONFIG_PATH, force = false) {
|
|
@@ -269,11 +271,11 @@ function buildAntigravityInvocation(options) {
|
|
|
269
271
|
};
|
|
270
272
|
}
|
|
271
273
|
function parseAntigravityOutput(stdout) {
|
|
272
|
-
const envelope = parseEnvelope(stdout);
|
|
274
|
+
const envelope = parseEnvelope$1(stdout);
|
|
273
275
|
if (envelope.status && envelope.status !== "SUCCESS") {
|
|
274
276
|
throw new Error(`Antigravity CLI reported status ${envelope.status}.`);
|
|
275
277
|
}
|
|
276
|
-
const result = envelope.structured_output ?? (typeof envelope.response === "string" ? tryParseJson(envelope.response) : null);
|
|
278
|
+
const result = envelope.structured_output ?? (typeof envelope.response === "string" ? tryParseJson$1(envelope.response) : null);
|
|
277
279
|
if (result === null || result === void 0) {
|
|
278
280
|
throw new Error(
|
|
279
281
|
"Antigravity CLI output contains no structured result. Check that the model finished the task (auth, quota, timeout)."
|
|
@@ -288,14 +290,14 @@ function parseAntigravityOutput(stdout) {
|
|
|
288
290
|
}
|
|
289
291
|
};
|
|
290
292
|
}
|
|
291
|
-
function parseEnvelope(stdout) {
|
|
293
|
+
function parseEnvelope$1(stdout) {
|
|
292
294
|
const trimmed = stdout.trim();
|
|
293
|
-
let parsed = tryParseJson(trimmed);
|
|
295
|
+
let parsed = tryParseJson$1(trimmed);
|
|
294
296
|
if (parsed === null) {
|
|
295
297
|
const firstBrace = trimmed.indexOf("{");
|
|
296
298
|
const lastBrace = trimmed.lastIndexOf("}");
|
|
297
299
|
if (firstBrace >= 0 && lastBrace > firstBrace) {
|
|
298
|
-
parsed = tryParseJson(trimmed.slice(firstBrace, lastBrace + 1));
|
|
300
|
+
parsed = tryParseJson$1(trimmed.slice(firstBrace, lastBrace + 1));
|
|
299
301
|
}
|
|
300
302
|
}
|
|
301
303
|
if (!parsed || typeof parsed !== "object") {
|
|
@@ -303,7 +305,7 @@ function parseEnvelope(stdout) {
|
|
|
303
305
|
}
|
|
304
306
|
return parsed;
|
|
305
307
|
}
|
|
306
|
-
function tryParseJson(text) {
|
|
308
|
+
function tryParseJson$1(text) {
|
|
307
309
|
try {
|
|
308
310
|
return JSON.parse(text);
|
|
309
311
|
} catch {
|
|
@@ -432,7 +434,7 @@ Report your findings by calling the ${TOOL_NAME} tool.`;
|
|
|
432
434
|
});
|
|
433
435
|
if (!response.ok) {
|
|
434
436
|
const body = await response.text();
|
|
435
|
-
throw new Error(`Anthropic API error ${response.status}: ${truncate$
|
|
437
|
+
throw new Error(`Anthropic API error ${response.status}: ${truncate$3(body)}`);
|
|
436
438
|
}
|
|
437
439
|
const payload = await response.json();
|
|
438
440
|
const toolUse = payload.content?.find((block) => block.type === "tool_use");
|
|
@@ -448,7 +450,7 @@ Report your findings by calling the ${TOOL_NAME} tool.`;
|
|
|
448
450
|
}
|
|
449
451
|
};
|
|
450
452
|
}
|
|
451
|
-
function truncate$
|
|
453
|
+
function truncate$3(text) {
|
|
452
454
|
return text.length > 300 ? `${text.slice(0, 300)}...` : text;
|
|
453
455
|
}
|
|
454
456
|
const anthropicApiProvider = {
|
|
@@ -456,6 +458,92 @@ const anthropicApiProvider = {
|
|
|
456
458
|
defaultModel: ANTHROPIC_DEFAULT_MODEL,
|
|
457
459
|
execute: executeAnthropicApi
|
|
458
460
|
};
|
|
461
|
+
const CLAUDE_CLI_DEFAULT_MODEL = "haiku";
|
|
462
|
+
function buildClaudeCliInvocation(options) {
|
|
463
|
+
if (options.imageKind === "remote") {
|
|
464
|
+
throw new Error(
|
|
465
|
+
"claude-cli provider reads local files only. Download the image first, or use -p gemini-api for remote URLs."
|
|
466
|
+
);
|
|
467
|
+
}
|
|
468
|
+
const prompt = buildVisionPrompt({
|
|
469
|
+
imageSource: options.imageSource,
|
|
470
|
+
imageKind: "local",
|
|
471
|
+
extraPrompt: options.extraPrompt
|
|
472
|
+
});
|
|
473
|
+
const args = [
|
|
474
|
+
"-p",
|
|
475
|
+
prompt,
|
|
476
|
+
"--output-format",
|
|
477
|
+
"json",
|
|
478
|
+
"--json-schema",
|
|
479
|
+
visionResultSchemaJson(),
|
|
480
|
+
"--allowedTools",
|
|
481
|
+
"Read",
|
|
482
|
+
"--model",
|
|
483
|
+
options.model || options.settings?.model || CLAUDE_CLI_DEFAULT_MODEL
|
|
484
|
+
];
|
|
485
|
+
return {
|
|
486
|
+
command: options.providerBin || "claude",
|
|
487
|
+
args,
|
|
488
|
+
cwd: path.resolve(options.workdir || path.dirname(options.imageSource))
|
|
489
|
+
};
|
|
490
|
+
}
|
|
491
|
+
function parseClaudeCliOutput(stdout) {
|
|
492
|
+
const envelope = parseEnvelope(stdout);
|
|
493
|
+
if (envelope.is_error || envelope.subtype && envelope.subtype !== "success") {
|
|
494
|
+
throw new Error(
|
|
495
|
+
`Claude CLI reported ${envelope.subtype ?? "an error"}: ${truncate$2(envelope.result ?? "")}`
|
|
496
|
+
);
|
|
497
|
+
}
|
|
498
|
+
if (typeof envelope.result !== "string" || !envelope.result.trim()) {
|
|
499
|
+
throw new Error("Claude CLI output contains no result. Check login state (run: claude).");
|
|
500
|
+
}
|
|
501
|
+
let result;
|
|
502
|
+
try {
|
|
503
|
+
result = JSON.parse(envelope.result);
|
|
504
|
+
} catch {
|
|
505
|
+
throw new Error(`Claude CLI returned non-JSON result: ${truncate$2(envelope.result)}`);
|
|
506
|
+
}
|
|
507
|
+
return {
|
|
508
|
+
result,
|
|
509
|
+
meta: {
|
|
510
|
+
conversationId: envelope.session_id ?? null,
|
|
511
|
+
durationSeconds: typeof envelope.duration_ms === "number" ? envelope.duration_ms / 1e3 : null,
|
|
512
|
+
usage: envelope.usage ?? null
|
|
513
|
+
}
|
|
514
|
+
};
|
|
515
|
+
}
|
|
516
|
+
function parseEnvelope(stdout) {
|
|
517
|
+
const trimmed = stdout.trim();
|
|
518
|
+
let parsed = tryParseJson(trimmed);
|
|
519
|
+
if (parsed === null) {
|
|
520
|
+
const firstBrace = trimmed.indexOf("{");
|
|
521
|
+
const lastBrace = trimmed.lastIndexOf("}");
|
|
522
|
+
if (firstBrace >= 0 && lastBrace > firstBrace) {
|
|
523
|
+
parsed = tryParseJson(trimmed.slice(firstBrace, lastBrace + 1));
|
|
524
|
+
}
|
|
525
|
+
}
|
|
526
|
+
if (!parsed || typeof parsed !== "object") {
|
|
527
|
+
throw new Error("Failed to parse Claude CLI JSON output.");
|
|
528
|
+
}
|
|
529
|
+
return parsed;
|
|
530
|
+
}
|
|
531
|
+
function tryParseJson(text) {
|
|
532
|
+
try {
|
|
533
|
+
return JSON.parse(text);
|
|
534
|
+
} catch {
|
|
535
|
+
return null;
|
|
536
|
+
}
|
|
537
|
+
}
|
|
538
|
+
function truncate$2(text) {
|
|
539
|
+
return text.length > 300 ? `${text.slice(0, 300)}...` : text;
|
|
540
|
+
}
|
|
541
|
+
const claudeCliProvider = {
|
|
542
|
+
name: "claude-cli",
|
|
543
|
+
defaultModel: CLAUDE_CLI_DEFAULT_MODEL,
|
|
544
|
+
buildInvocation: buildClaudeCliInvocation,
|
|
545
|
+
parseOutput: parseClaudeCliOutput
|
|
546
|
+
};
|
|
459
547
|
const GEMINI_API_DEFAULT_MODEL = "gemini-3.6-flash";
|
|
460
548
|
const DEFAULT_BASE_URL = "https://generativelanguage.googleapis.com";
|
|
461
549
|
async function executeGeminiApi(options) {
|
|
@@ -615,7 +703,9 @@ const PROVIDERS = {
|
|
|
615
703
|
openai: openaiCompatProvider,
|
|
616
704
|
"openai-compat": openaiCompatProvider,
|
|
617
705
|
anthropic: anthropicApiProvider,
|
|
618
|
-
claude: anthropicApiProvider
|
|
706
|
+
claude: anthropicApiProvider,
|
|
707
|
+
"claude-cli": claudeCliProvider,
|
|
708
|
+
"claude-code": claudeCliProvider
|
|
619
709
|
};
|
|
620
710
|
function resolveProvider(providerName = "antigravity-cli") {
|
|
621
711
|
const normalized = providerName.trim().toLowerCase();
|
|
@@ -754,8 +844,89 @@ function runCommand(providerName, invocation, timeoutMs) {
|
|
|
754
844
|
});
|
|
755
845
|
});
|
|
756
846
|
}
|
|
847
|
+
const EXT_BY_MIME = {
|
|
848
|
+
"image/png": "png",
|
|
849
|
+
"image/jpeg": "jpg",
|
|
850
|
+
"image/webp": "webp",
|
|
851
|
+
"image/gif": "gif"
|
|
852
|
+
};
|
|
853
|
+
function projectSlug(cwd) {
|
|
854
|
+
return path.resolve(cwd).replace(/[/.]/g, "-");
|
|
855
|
+
}
|
|
856
|
+
function locateTranscript(cwd) {
|
|
857
|
+
const dir = path.join(os.homedir(), ".claude", "projects", projectSlug(cwd));
|
|
858
|
+
let entries;
|
|
859
|
+
try {
|
|
860
|
+
entries = fs.readdirSync(dir).filter((name) => name.endsWith(".jsonl"));
|
|
861
|
+
} catch {
|
|
862
|
+
throw new Error(
|
|
863
|
+
`No Claude Code transcripts found for this directory (${dir}). Run from the project the image was pasted in, or pass --transcript <path>.`
|
|
864
|
+
);
|
|
865
|
+
}
|
|
866
|
+
if (entries.length === 0) {
|
|
867
|
+
throw new Error(`No transcripts in ${dir}. Pass --transcript <path> to pick one manually.`);
|
|
868
|
+
}
|
|
869
|
+
const newest = entries.map((name) => {
|
|
870
|
+
const full = path.join(dir, name);
|
|
871
|
+
return { full, mtime: fs.statSync(full).mtimeMs };
|
|
872
|
+
}).sort((a, b) => b.mtime - a.mtime)[0];
|
|
873
|
+
return newest.full;
|
|
874
|
+
}
|
|
875
|
+
function extractUserImages(transcriptPath) {
|
|
876
|
+
let raw;
|
|
877
|
+
try {
|
|
878
|
+
raw = fs.readFileSync(transcriptPath, "utf-8");
|
|
879
|
+
} catch (error) {
|
|
880
|
+
throw new Error(`Cannot read transcript ${transcriptPath}: ${error.message}`);
|
|
881
|
+
}
|
|
882
|
+
const images = [];
|
|
883
|
+
for (const line of raw.split("\n")) {
|
|
884
|
+
if (!line.includes('"image"')) {
|
|
885
|
+
continue;
|
|
886
|
+
}
|
|
887
|
+
let parsed;
|
|
888
|
+
try {
|
|
889
|
+
parsed = JSON.parse(line);
|
|
890
|
+
} catch {
|
|
891
|
+
continue;
|
|
892
|
+
}
|
|
893
|
+
const message = parsed.message;
|
|
894
|
+
if (message?.role !== "user" || !Array.isArray(message.content)) {
|
|
895
|
+
continue;
|
|
896
|
+
}
|
|
897
|
+
for (const block of message.content) {
|
|
898
|
+
const source = block?.source;
|
|
899
|
+
if (block?.type === "image" && source?.type === "base64" && source.data) {
|
|
900
|
+
images.push({ mediaType: source.media_type ?? "image/png", data: source.data });
|
|
901
|
+
}
|
|
902
|
+
}
|
|
903
|
+
}
|
|
904
|
+
return images;
|
|
905
|
+
}
|
|
906
|
+
function recoverPastedImages(options = {}) {
|
|
907
|
+
const transcript = options.transcript ?? locateTranscript(options.cwd ?? process.cwd());
|
|
908
|
+
const count = Math.max(1, options.count ?? 1);
|
|
909
|
+
const outDir = options.outDir ?? path.join(os.tmpdir(), "modlens-paste");
|
|
910
|
+
const all = extractUserImages(transcript);
|
|
911
|
+
if (all.length === 0) {
|
|
912
|
+
throw new Error(
|
|
913
|
+
`No pasted images found in ${transcript}. The user may not have pasted any, or the transcript format changed; ask for a file path instead.`
|
|
914
|
+
);
|
|
915
|
+
}
|
|
916
|
+
fs.mkdirSync(outDir, { recursive: true });
|
|
917
|
+
const picked = all.slice(-count);
|
|
918
|
+
const images = picked.map((image) => {
|
|
919
|
+
const buffer = Buffer.from(image.data, "base64");
|
|
920
|
+
const hash = crypto.createHash("sha256").update(buffer).digest("hex").slice(0, 8);
|
|
921
|
+
const ext = EXT_BY_MIME[image.mediaType] ?? "png";
|
|
922
|
+
const filePath = path.join(outDir, `paste-${hash}.${ext}`);
|
|
923
|
+
fs.writeFileSync(filePath, buffer);
|
|
924
|
+
return { path: filePath, mediaType: image.mediaType, bytes: buffer.length };
|
|
925
|
+
});
|
|
926
|
+
return { transcript, images };
|
|
927
|
+
}
|
|
757
928
|
const program = new Command();
|
|
758
|
-
program.name("modlens").description("Plug-in vision for text-only LLMs: image in, structured JSON evidence out").version("2.
|
|
929
|
+
program.name("modlens").description("Plug-in vision for text-only LLMs: image in, structured JSON evidence out").version("2.3.0");
|
|
759
930
|
program.command("analyze", { isDefault: true }).description("Analyze an image into structured JSON evidence (default command)").requiredOption("-i, --input <path|url>", "Input image path or https URL").option("-o, --output <path>", "Write result JSON to a file").option("-m, --model <name>", "Provider model name").option("-p, --provider <name>", `Vision provider (${listProviders().join(", ")})`).option("--prompt <text>", "Extra focus for this image").option("--timeout <ms>", "Provider timeout in milliseconds", "180000").option("--provider-bin <path>", "Provider binary path (default: agy)").option("--workdir <path>", "Working directory for the provider").action(async (options) => {
|
|
760
931
|
try {
|
|
761
932
|
const timeoutMs = Number.parseInt(options.timeout, 10);
|
|
@@ -787,6 +958,30 @@ program.command("analyze", { isDefault: true }).description("Analyze an image in
|
|
|
787
958
|
process.exit(1);
|
|
788
959
|
}
|
|
789
960
|
});
|
|
961
|
+
program.command("recover-paste").description(
|
|
962
|
+
"Recover images pasted into Claude Code from the session transcript (they never hit disk otherwise)"
|
|
963
|
+
).option("--count <n>", "How many recent pasted images to recover", "1").option("--out-dir <path>", "Directory to write recovered images to").option("--transcript <path>", "Explicit transcript .jsonl (default: newest for cwd)").option("--cwd <path>", "Project directory the image was pasted in", process.cwd()).action(async (options) => {
|
|
964
|
+
try {
|
|
965
|
+
const count = Number.parseInt(options.count, 10);
|
|
966
|
+
if (!Number.isFinite(count) || count <= 0) {
|
|
967
|
+
throw new Error("Invalid --count. Use a positive integer.");
|
|
968
|
+
}
|
|
969
|
+
const result = recoverPastedImages({
|
|
970
|
+
count,
|
|
971
|
+
outDir: options.outDir,
|
|
972
|
+
transcript: options.transcript,
|
|
973
|
+
cwd: options.cwd
|
|
974
|
+
});
|
|
975
|
+
process.stdout.write(`${JSON.stringify(result, null, 2)}
|
|
976
|
+
`);
|
|
977
|
+
} catch (error) {
|
|
978
|
+
process.stderr.write(
|
|
979
|
+
`Error: ${error instanceof Error ? error.message : String(error)}
|
|
980
|
+
`
|
|
981
|
+
);
|
|
982
|
+
process.exit(1);
|
|
983
|
+
}
|
|
984
|
+
});
|
|
790
985
|
const config = program.command("config").description(`Manage ${CONFIG_PATH} (providers, keys, models)`);
|
|
791
986
|
config.command("init").description(`Create a starter config at ${CONFIG_PATH}`).option("--force", "Overwrite an existing config file").action((options) => {
|
|
792
987
|
try {
|
package/package.json
CHANGED
package/skills/modlens/SKILL.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: modlens
|
|
3
|
-
description: "Plug-in vision for text-only models. Use whenever the user shares an image (local path, screenshot, photo, chart, document scan, or image URL) and the active model cannot see images or has no vision tool. Runs the modlens CLI to convert the image into structured JSON evidence: OCR text, layout, semantics, visual clues."
|
|
3
|
+
description: "Plug-in vision for text-only models. Use whenever the user shares an image (local path, screenshot, photo, chart, document scan, or image URL) and the active model cannot see images or has no vision tool. Runs the modlens CLI to convert the image into structured JSON evidence: OCR text, layout, semantics, visual clues. Also use when the user asks how to install, configure, or switch modlens providers (Gemini API key, OpenAI-compatible endpoints, Claude API or Claude Code CLI)."
|
|
4
4
|
allowed-tools:
|
|
5
5
|
- Bash
|
|
6
6
|
---
|
|
@@ -12,6 +12,7 @@ Use this skill when:
|
|
|
12
12
|
- The user provides an image path or image URL and asks anything about it
|
|
13
13
|
- The active model has no native vision (text-only model in a coding agent)
|
|
14
14
|
- You need OCR text, layout, or chart/document structure as evidence before reasoning
|
|
15
|
+
- The user asks how to configure modlens, get an API key for it, or switch its provider: follow `references/configure.md` and run the commands for them
|
|
15
16
|
|
|
16
17
|
Do not use this skill for:
|
|
17
18
|
|
|
@@ -26,7 +27,7 @@ modlens --version
|
|
|
26
27
|
|
|
27
28
|
If `modlens` is missing, run it via `npx @liustack/modlens` instead.
|
|
28
29
|
|
|
29
|
-
ModLens supports
|
|
30
|
+
ModLens supports five vision providers. Check what is configured:
|
|
30
31
|
|
|
31
32
|
```bash
|
|
32
33
|
modlens config show
|
|
@@ -36,8 +37,9 @@ modlens config show
|
|
|
36
37
|
- **gemini-api**: needs `GEMINI_API_KEY` env or `modlens config set gemini-api.apiKey <key>` (free key from https://aistudio.google.com).
|
|
37
38
|
- **openai**: any OpenAI-compatible multimodal endpoint; needs baseUrl + apiKey + model via env (`OPENAI_BASE_URL`, `OPENAI_API_KEY`) or `modlens config set openai.<field> <value>`.
|
|
38
39
|
- **anthropic**: needs `ANTHROPIC_API_KEY` env or config; defaults to Claude Haiku.
|
|
40
|
+
- **claude-cli**: rides an existing Claude Code login (`claude`), no key, Read-only tool permissions, local files only.
|
|
39
41
|
|
|
40
|
-
`modlens config init` writes a starter config to `~/.modlens/config.json` when none exists.
|
|
42
|
+
`modlens config init` writes a starter config to `~/.modlens/config.json` when none exists. Full setup recipes per provider: `references/configure.md`.
|
|
41
43
|
|
|
42
44
|
## Command
|
|
43
45
|
|
|
@@ -55,7 +57,7 @@ Optional flags:
|
|
|
55
57
|
modlens -i <image> -o <output.json> -m <model> --prompt "<extra focus>" --timeout <ms>
|
|
56
58
|
```
|
|
57
59
|
|
|
58
|
-
Speed expectations: `gemini-api` typically 5-10 seconds, `antigravity-cli` 15-40 seconds (full agent
|
|
60
|
+
Speed expectations: `gemini-api` typically 5-10 seconds, `antigravity-cli` 15-40 seconds and `claude-cli` 20-45 seconds (full agent loops), `openai`/`anthropic` depend on the endpoint. For dense or hard images on antigravity-cli, try `-m gemini-3.1-pro-high`.
|
|
59
61
|
|
|
60
62
|
## Finding the image path in the chat
|
|
61
63
|
|
|
@@ -64,6 +66,7 @@ Harnesses rarely hand you a clean path. Look for these signals:
|
|
|
64
66
|
- Codex wraps every pasted or attached image in a text tag like
|
|
65
67
|
`<image name=[Image #1] path="/tmp/xxxx.png">`. Extract the `path` value and run modlens on it. Pasted images live in a temp file the harness already created.
|
|
66
68
|
- A placeholder like `image content omitted because you do not support image input` means the harness stripped an image for you. The path tag next to it still holds the real file. Use it.
|
|
69
|
+
- Claude Code never writes pasted images to a regular temp file. Behind a text-only gateway you will only see a placeholder like `[Unsupported Image]` or `[Image #1]`, with no path. When that happens, run `modlens recover-paste` (add `--count <n>` for several images): it pulls the pasted image bytes out of the local session transcript and prints the recovered file paths as JSON. Feed that path to `modlens -i`. If recovery fails (transcript format is Claude Code internals and may change), fall back to asking the user to drag the image file into the terminal or type its path.
|
|
67
70
|
- If the user mentions an image but no tag or path appears anywhere in the message, ask for the file path instead of guessing.
|
|
68
71
|
|
|
69
72
|
## Workflow
|
|
@@ -85,7 +88,7 @@ Top level: `{ image, provider, result, meta }`. Inside `result`:
|
|
|
85
88
|
- `visual`: colors and style clues
|
|
86
89
|
- `uncertainty[]`: what the vision engine was unsure about
|
|
87
90
|
|
|
88
|
-
Structure is enforced by schema on antigravity-cli (`--json-schema`), gemini-api (`responseJsonSchema`), and anthropic (forced tool call). The openai route uses a template prompt plus shape validation and fails loudly on mismatch.
|
|
91
|
+
Structure is enforced by schema on antigravity-cli and claude-cli (`--json-schema`), gemini-api (`responseJsonSchema`), and anthropic (forced tool call). The openai route uses a template prompt plus shape validation and fails loudly on mismatch.
|
|
89
92
|
|
|
90
93
|
## Failure Handling
|
|
91
94
|
|
|
@@ -0,0 +1,85 @@
|
|
|
1
|
+
# Configuring ModLens
|
|
2
|
+
|
|
3
|
+
Read this when the user asks how to set up, configure, or switch ModLens providers. Prefer running the commands for the user over explaining them.
|
|
4
|
+
|
|
5
|
+
## Where config lives
|
|
6
|
+
|
|
7
|
+
`~/.modlens/config.json`, managed by the CLI. Precedence: CLI flags > environment variables > config file > built-in defaults. The default provider with zero config is `antigravity-cli`.
|
|
8
|
+
|
|
9
|
+
```bash
|
|
10
|
+
modlens config init # write a starter config (refuses to overwrite; --force to redo)
|
|
11
|
+
modlens config show # effective file, API keys masked
|
|
12
|
+
modlens config set provider <name> # change the default provider
|
|
13
|
+
modlens config set <provider>.<field> <value> # fields: apiKey, baseUrl, model
|
|
14
|
+
```
|
|
15
|
+
|
|
16
|
+
`config set` writes the file with 0600 permissions.
|
|
17
|
+
|
|
18
|
+
## Provider setup recipes
|
|
19
|
+
|
|
20
|
+
### antigravity-cli (default, free, no key)
|
|
21
|
+
|
|
22
|
+
Needs Antigravity CLI installed and signed in:
|
|
23
|
+
|
|
24
|
+
```bash
|
|
25
|
+
curl -fsSL https://antigravity.google/cli/install.sh | bash
|
|
26
|
+
agy # user must complete browser sign-in themselves, then exit
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
Any free Google account works; no Google AI Pro needed. Sign-in cannot be automated, ask the user to run `agy` once.
|
|
30
|
+
|
|
31
|
+
### gemini-api (free key, fastest free route, 5-10s)
|
|
32
|
+
|
|
33
|
+
1. The user creates a key at https://aistudio.google.com (three minutes, no credit card, free tier does not expire).
|
|
34
|
+
2. Store it either way:
|
|
35
|
+
|
|
36
|
+
```bash
|
|
37
|
+
modlens config set gemini-api.apiKey <key>
|
|
38
|
+
# or environment: export GEMINI_API_KEY=<key>
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
Default model `gemini-3.6-flash` has vision on the free tier (about 10-15 requests/min, 1500/day). Free-tier data may be used by Google to improve products; mention this if the user handles sensitive images.
|
|
42
|
+
|
|
43
|
+
### openai (any OpenAI-compatible multimodal endpoint)
|
|
44
|
+
|
|
45
|
+
Needs three values. Example for DashScope qwen:
|
|
46
|
+
|
|
47
|
+
```bash
|
|
48
|
+
modlens config set openai.baseUrl https://dashscope.aliyuncs.com/compatible-mode/v1
|
|
49
|
+
modlens config set openai.apiKey <sk-key>
|
|
50
|
+
modlens config set openai.model qwen3.6-27b
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
For official OpenAI: baseUrl `https://api.openai.com/v1`, a vision-capable model. Environment equivalents: `OPENAI_BASE_URL`, `OPENAI_API_KEY`. The model must be multimodal; text-only models will fail or hallucinate. This route has no server-side schema enforcement, so occasional shape failures are surfaced as explicit errors; retry or switch provider.
|
|
54
|
+
|
|
55
|
+
### anthropic (Claude API key)
|
|
56
|
+
|
|
57
|
+
```bash
|
|
58
|
+
modlens config set anthropic.apiKey <sk-ant-key>
|
|
59
|
+
# or: export ANTHROPIC_API_KEY=<key>
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
Default model is Claude Haiku (`claude-haiku-4-5-20251001`). Schema is enforced through a forced tool call.
|
|
63
|
+
|
|
64
|
+
### claude-cli (Claude Code login, no key)
|
|
65
|
+
|
|
66
|
+
Rides an existing `claude` sign-in, so it costs the user's Claude subscription quota, not a separate API bill. Requires Claude Code installed and logged in (`claude --version` to check). Runs with `--allowedTools Read` only. Local image files only; for remote URLs use gemini-api instead. Default model alias `haiku`.
|
|
67
|
+
|
|
68
|
+
```bash
|
|
69
|
+
modlens config set provider claude-cli # make it the default if the user wants
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
## Choosing a provider for the user
|
|
73
|
+
|
|
74
|
+
- Wants zero setup and free: `antigravity-cli` (needs agy sign-in, 15-40s per image).
|
|
75
|
+
- Wants fast and free: `gemini-api` (three-minute key, 5-10s).
|
|
76
|
+
- Already pays for Claude: `claude-cli` (no extra key) or `anthropic` (API billing).
|
|
77
|
+
- Has a favorite multimodal endpoint (qwen, GLM, ...): `openai`.
|
|
78
|
+
|
|
79
|
+
## Troubleshooting
|
|
80
|
+
|
|
81
|
+
- Error names a missing env var or `config set` command: run exactly that.
|
|
82
|
+
- `Provider CLI not found: agy`: install Antigravity CLI or switch provider.
|
|
83
|
+
- `Claude CLI reported ...` or empty result: check `claude` login state.
|
|
84
|
+
- openai route `does not match the vision schema`: retry once, then switch to gemini-api or anthropic.
|
|
85
|
+
- `config init` refusing to run: the file exists; use `modlens config show` first, `--force` only if the user agrees to overwrite.
|