@liustack/modlens 2.1.0 → 2.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -90,6 +90,10 @@ Batch mode works too: drop three illustrations at once, and the model announces
90
90
 
91
91
  ![Text-only DeepSeek reading three images in one go via ModLens](https://raw.githubusercontent.com/liustack/modlens/main/assets/demo-codex-batch.png)
92
92
 
93
+ Stress test: a scatter plot of 128 models. ModLens pulls out the axes, the log scale, and the highlighted DeepSeek V4 Flash point at $0.028 and score 50, then walks through the cost-performance cutoff line. Dense charts are where vision models usually fold; this one holds.
94
+
95
+ ![Text-only DeepSeek reading a 128-model scatter plot via ModLens](https://raw.githubusercontent.com/liustack/modlens/main/assets/demo-codex-chart.png)
96
+
93
97
  ## CLI reference
94
98
 
95
99
  ```bash
@@ -111,7 +115,7 @@ Reach for `-m gemini-3.1-pro-high` on dense screenshots or tricky documents. Out
111
115
 
112
116
  ## Providers and config
113
117
 
114
- ModLens ships four vision providers. `antigravity-cli` stays the default: zero keys, pure free quota.
118
+ ModLens ships five vision providers. `antigravity-cli` stays the default: zero keys, pure free quota.
115
119
 
116
120
  | Provider | Needs | Typical speed | Notes |
117
121
  | :-- | :-- | :-- | :-- |
@@ -119,6 +123,7 @@ ModLens ships four vision providers. `antigravity-cli` stays the default: zero k
119
123
  | `gemini-api` | free AI Studio key | 5-10s | fastest free route, schema enforced server-side |
120
124
  | `openai` | baseUrl + apiKey + model | endpoint-dependent | any OpenAI-compatible multimodal endpoint (qwen-vl, GLM, ...) |
121
125
  | `anthropic` | `ANTHROPIC_API_KEY` | a few seconds | Claude Haiku by default, schema via forced tool call |
126
+ | `claude-cli` | Claude Code signed in | 20-45s | no key, rides your Claude subscription, Read-only permissions |
122
127
 
123
128
  Config lives in `~/.modlens/config.json`. Environment variables override the file (`GEMINI_API_KEY`, `OPENAI_API_KEY`, `OPENAI_BASE_URL`, `ANTHROPIC_API_KEY`), and CLI flags override everything.
124
129
 
@@ -140,6 +145,12 @@ One catch: once text-only is declared, the Codex TUI **blocks Ctrl+V image paste
140
145
  - **Drag the image file into the terminal**, or type its path. The path lands as plain text, and the modlens skill picks it up from there.
141
146
  - Attach it with `codex exec -i image.png "..."`. The skill reads the path out of the message tag.
142
147
 
148
+ ## Using it in Claude Code (gateway models)
149
+
150
+ No setup needed: drag the image file into the terminal, or type its path, and the skill takes over.
151
+
152
+ Paste is trickier. If you run a text-only model behind `ANTHROPIC_BASE_URL`, Claude Code never writes pasted images to a regular temp file and has no modality switch, so a pasted image reaches the model as a pathless `[Unsupported Image]` placeholder (lenient gateways like DeepSeek's Anthropic endpoint) or breaks the request outright ([#62009](https://github.com/anthropics/claude-code/issues/62009)). But the bytes are not gone: Claude Code appends every user message, images included, to the local session transcript before the gateway ever sees it. That is what `modlens recover-paste` exploits: it pulls the most recent pasted images back out of the transcript and prints real file paths, ready for `modlens -i`. The skill runs this automatically when it spots the placeholder. One honest caveat: the transcript layout is Claude Code internals with no compatibility promise; if recovery ever breaks, dragging the file still works everywhere.
153
+
143
154
  ## Why a bridge instead of a multimodal model?
144
155
 
145
156
  - **Keep your model.** You picked DeepSeek-V4-Flash (or gpt-oss, or whatever else) for its price and its reasoning, not its eyesight. ModLens adds sight without touching that choice.
package/README.zh-CN.md CHANGED
@@ -90,6 +90,10 @@ npx @liustack/modlens -i workflow.jpg
90
90
 
91
91
  ![纯文本 DeepSeek 通过 ModLens 一次读完三张图](https://raw.githubusercontent.com/liustack/modlens/main/assets/demo-codex-batch.png)
92
92
 
93
+ 压力测试:一张 128 个模型的智能对成本散点图。ModLens 读出双轴、对数刻度,把高亮的 DeepSeek V4 Flash 精准拎出来(成本约 $0.028、智能指数 50),还讲明白了性价比斩杀线。密集图表是识图模型最容易露怯的地方,这一关它扛住了。
94
+
95
+ ![纯文本 DeepSeek 通过 ModLens 读 128 个模型的散点图](https://raw.githubusercontent.com/liustack/modlens/main/assets/demo-codex-chart.png)
96
+
93
97
  ## CLI 参数
94
98
 
95
99
  ```bash
@@ -111,7 +115,7 @@ modlens -i <图片路径或 URL> [选项]
111
115
 
112
116
  ## Provider 与配置
113
117
 
114
- ModLens 内置四个视觉 provider,默认还是 `antigravity-cli`:零 key,纯免费额度。
118
+ ModLens 内置五个视觉 provider,默认还是 `antigravity-cli`:零 key,纯免费额度。
115
119
 
116
120
  | Provider | 需要什么 | 速度 | 说明 |
117
121
  | :-- | :-- | :-- | :-- |
@@ -119,6 +123,7 @@ ModLens 内置四个视觉 provider,默认还是 `antigravity-cli`:零 key
119
123
  | `gemini-api` | 免费 AI Studio key | 5-10 秒 | 最快的免费路线,服务端强制 schema |
120
124
  | `openai` | baseUrl + apiKey + model | 看端点 | 任何 OpenAI 兼容的多模态端点(qwen-vl、GLM 等) |
121
125
  | `anthropic` | `ANTHROPIC_API_KEY` | 几秒 | 默认 Claude Haiku,强制工具调用保 schema |
126
+ | `claude-cli` | Claude Code 已登录 | 20-45 秒 | 零 key,吃你的 Claude 订阅额度,只放行 Read 工具 |
122
127
 
123
128
  配置放在 `~/.modlens/config.json`,环境变量能盖过它(`GEMINI_API_KEY`、`OPENAI_API_KEY`、`OPENAI_BASE_URL`、`ANTHROPIC_API_KEY`),CLI 参数最大。
124
129
 
@@ -140,6 +145,12 @@ Codex 只认 Responses API,DeepSeek 官方端点原生支持。先照着[官
140
145
  - **把图片文件拖进终端**,或者手打路径。路径以纯文本形式落进消息,modlens skill 接着从这里接手。
141
146
  - 用 `codex exec -i 图片.png "..."` skill 从这里把路径抠出来。
142
147
 
148
+ ## 在 Claude Code 里用(网关接第三方模型)
149
+
150
+ 不用任何配置:把图片文件拖进终端,或手打路径,skill 直接接手。
151
+
152
+ 粘贴要多说两句。走 `ANTHROPIC_BASE_URL` 网关跑纯文本模型时,Claude Code 粘贴的图片从不写普通临时文件,也没有声明模型无视觉的开关,粘贴的图要么变成一个不带路径的 `[Unsupported Image]` 占位符到达模型(DeepSeek 的 Anthropic 兼容端点这类宽容网关),要么直接把请求搞挂([#62009](https://github.com/anthropics/claude-code/issues/62009))。但图片字节没有蒸发:Claude Code 在网关看到消息之前,就把每条用户消息(含图片)原样写进了本地会话记录。`modlens recover-paste` 干的就是这件事:从会话记录里把最近粘贴的图捞回来,落成真实文件路径,直接喂给 `modlens -i`。skill 看到占位符会自动跑这一步。一句老实话:会话记录格式是 Claude Code 的内部实现,没有兼容承诺,哪天捞不动了,拖文件永远是保底。
153
+
143
154
  ## 为什么外挂,而不是换多模态模型?
144
155
 
145
156
  - **模型不用换。** 你选 DeepSeek-V4-Flash(或 gpt-oss,或别的什么)图的是价格和推理能力,不是视力。ModLens 只加视力,不碰这个选择。
package/dist/main.js CHANGED
@@ -4,6 +4,7 @@ import * as fs from "fs";
4
4
  import * as path from "path";
5
5
  import { spawn } from "child_process";
6
6
  import * as os from "os";
7
+ import * as crypto from "crypto";
7
8
  const CONFIG_DIR = path.join(os.homedir(), ".modlens");
8
9
  const CONFIG_PATH = path.join(CONFIG_DIR, "config.json");
9
10
  const ENV_BINDINGS = {
@@ -79,7 +80,8 @@ const CONFIG_TEMPLATE = {
79
80
  "antigravity-cli": { model: "gemini-3.6-flash-low" },
80
81
  "gemini-api": { apiKey: "", model: "gemini-3.6-flash" },
81
82
  openai: { baseUrl: "", apiKey: "", model: "" },
82
- anthropic: { apiKey: "", model: "claude-haiku-4-5-20251001" }
83
+ anthropic: { apiKey: "", model: "claude-haiku-4-5-20251001" },
84
+ "claude-cli": { model: "haiku" }
83
85
  }
84
86
  };
85
87
  function initConfigFile(configPath = CONFIG_PATH, force = false) {
@@ -269,11 +271,11 @@ function buildAntigravityInvocation(options) {
269
271
  };
270
272
  }
271
273
  function parseAntigravityOutput(stdout) {
272
- const envelope = parseEnvelope(stdout);
274
+ const envelope = parseEnvelope$1(stdout);
273
275
  if (envelope.status && envelope.status !== "SUCCESS") {
274
276
  throw new Error(`Antigravity CLI reported status ${envelope.status}.`);
275
277
  }
276
- const result = envelope.structured_output ?? (typeof envelope.response === "string" ? tryParseJson(envelope.response) : null);
278
+ const result = envelope.structured_output ?? (typeof envelope.response === "string" ? tryParseJson$1(envelope.response) : null);
277
279
  if (result === null || result === void 0) {
278
280
  throw new Error(
279
281
  "Antigravity CLI output contains no structured result. Check that the model finished the task (auth, quota, timeout)."
@@ -288,14 +290,14 @@ function parseAntigravityOutput(stdout) {
288
290
  }
289
291
  };
290
292
  }
291
- function parseEnvelope(stdout) {
293
+ function parseEnvelope$1(stdout) {
292
294
  const trimmed = stdout.trim();
293
- let parsed = tryParseJson(trimmed);
295
+ let parsed = tryParseJson$1(trimmed);
294
296
  if (parsed === null) {
295
297
  const firstBrace = trimmed.indexOf("{");
296
298
  const lastBrace = trimmed.lastIndexOf("}");
297
299
  if (firstBrace >= 0 && lastBrace > firstBrace) {
298
- parsed = tryParseJson(trimmed.slice(firstBrace, lastBrace + 1));
300
+ parsed = tryParseJson$1(trimmed.slice(firstBrace, lastBrace + 1));
299
301
  }
300
302
  }
301
303
  if (!parsed || typeof parsed !== "object") {
@@ -303,7 +305,7 @@ function parseEnvelope(stdout) {
303
305
  }
304
306
  return parsed;
305
307
  }
306
- function tryParseJson(text) {
308
+ function tryParseJson$1(text) {
307
309
  try {
308
310
  return JSON.parse(text);
309
311
  } catch {
@@ -432,7 +434,7 @@ Report your findings by calling the ${TOOL_NAME} tool.`;
432
434
  });
433
435
  if (!response.ok) {
434
436
  const body = await response.text();
435
- throw new Error(`Anthropic API error ${response.status}: ${truncate$2(body)}`);
437
+ throw new Error(`Anthropic API error ${response.status}: ${truncate$3(body)}`);
436
438
  }
437
439
  const payload = await response.json();
438
440
  const toolUse = payload.content?.find((block) => block.type === "tool_use");
@@ -448,7 +450,7 @@ Report your findings by calling the ${TOOL_NAME} tool.`;
448
450
  }
449
451
  };
450
452
  }
451
- function truncate$2(text) {
453
+ function truncate$3(text) {
452
454
  return text.length > 300 ? `${text.slice(0, 300)}...` : text;
453
455
  }
454
456
  const anthropicApiProvider = {
@@ -456,6 +458,92 @@ const anthropicApiProvider = {
456
458
  defaultModel: ANTHROPIC_DEFAULT_MODEL,
457
459
  execute: executeAnthropicApi
458
460
  };
461
+ const CLAUDE_CLI_DEFAULT_MODEL = "haiku";
462
+ function buildClaudeCliInvocation(options) {
463
+ if (options.imageKind === "remote") {
464
+ throw new Error(
465
+ "claude-cli provider reads local files only. Download the image first, or use -p gemini-api for remote URLs."
466
+ );
467
+ }
468
+ const prompt = buildVisionPrompt({
469
+ imageSource: options.imageSource,
470
+ imageKind: "local",
471
+ extraPrompt: options.extraPrompt
472
+ });
473
+ const args = [
474
+ "-p",
475
+ prompt,
476
+ "--output-format",
477
+ "json",
478
+ "--json-schema",
479
+ visionResultSchemaJson(),
480
+ "--allowedTools",
481
+ "Read",
482
+ "--model",
483
+ options.model || options.settings?.model || CLAUDE_CLI_DEFAULT_MODEL
484
+ ];
485
+ return {
486
+ command: options.providerBin || "claude",
487
+ args,
488
+ cwd: path.resolve(options.workdir || path.dirname(options.imageSource))
489
+ };
490
+ }
491
+ function parseClaudeCliOutput(stdout) {
492
+ const envelope = parseEnvelope(stdout);
493
+ if (envelope.is_error || envelope.subtype && envelope.subtype !== "success") {
494
+ throw new Error(
495
+ `Claude CLI reported ${envelope.subtype ?? "an error"}: ${truncate$2(envelope.result ?? "")}`
496
+ );
497
+ }
498
+ if (typeof envelope.result !== "string" || !envelope.result.trim()) {
499
+ throw new Error("Claude CLI output contains no result. Check login state (run: claude).");
500
+ }
501
+ let result;
502
+ try {
503
+ result = JSON.parse(envelope.result);
504
+ } catch {
505
+ throw new Error(`Claude CLI returned non-JSON result: ${truncate$2(envelope.result)}`);
506
+ }
507
+ return {
508
+ result,
509
+ meta: {
510
+ conversationId: envelope.session_id ?? null,
511
+ durationSeconds: typeof envelope.duration_ms === "number" ? envelope.duration_ms / 1e3 : null,
512
+ usage: envelope.usage ?? null
513
+ }
514
+ };
515
+ }
516
+ function parseEnvelope(stdout) {
517
+ const trimmed = stdout.trim();
518
+ let parsed = tryParseJson(trimmed);
519
+ if (parsed === null) {
520
+ const firstBrace = trimmed.indexOf("{");
521
+ const lastBrace = trimmed.lastIndexOf("}");
522
+ if (firstBrace >= 0 && lastBrace > firstBrace) {
523
+ parsed = tryParseJson(trimmed.slice(firstBrace, lastBrace + 1));
524
+ }
525
+ }
526
+ if (!parsed || typeof parsed !== "object") {
527
+ throw new Error("Failed to parse Claude CLI JSON output.");
528
+ }
529
+ return parsed;
530
+ }
531
+ function tryParseJson(text) {
532
+ try {
533
+ return JSON.parse(text);
534
+ } catch {
535
+ return null;
536
+ }
537
+ }
538
+ function truncate$2(text) {
539
+ return text.length > 300 ? `${text.slice(0, 300)}...` : text;
540
+ }
541
+ const claudeCliProvider = {
542
+ name: "claude-cli",
543
+ defaultModel: CLAUDE_CLI_DEFAULT_MODEL,
544
+ buildInvocation: buildClaudeCliInvocation,
545
+ parseOutput: parseClaudeCliOutput
546
+ };
459
547
  const GEMINI_API_DEFAULT_MODEL = "gemini-3.6-flash";
460
548
  const DEFAULT_BASE_URL = "https://generativelanguage.googleapis.com";
461
549
  async function executeGeminiApi(options) {
@@ -615,7 +703,9 @@ const PROVIDERS = {
615
703
  openai: openaiCompatProvider,
616
704
  "openai-compat": openaiCompatProvider,
617
705
  anthropic: anthropicApiProvider,
618
- claude: anthropicApiProvider
706
+ claude: anthropicApiProvider,
707
+ "claude-cli": claudeCliProvider,
708
+ "claude-code": claudeCliProvider
619
709
  };
620
710
  function resolveProvider(providerName = "antigravity-cli") {
621
711
  const normalized = providerName.trim().toLowerCase();
@@ -754,8 +844,89 @@ function runCommand(providerName, invocation, timeoutMs) {
754
844
  });
755
845
  });
756
846
  }
847
+ const EXT_BY_MIME = {
848
+ "image/png": "png",
849
+ "image/jpeg": "jpg",
850
+ "image/webp": "webp",
851
+ "image/gif": "gif"
852
+ };
853
+ function projectSlug(cwd) {
854
+ return path.resolve(cwd).replace(/[/.]/g, "-");
855
+ }
856
+ function locateTranscript(cwd) {
857
+ const dir = path.join(os.homedir(), ".claude", "projects", projectSlug(cwd));
858
+ let entries;
859
+ try {
860
+ entries = fs.readdirSync(dir).filter((name) => name.endsWith(".jsonl"));
861
+ } catch {
862
+ throw new Error(
863
+ `No Claude Code transcripts found for this directory (${dir}). Run from the project the image was pasted in, or pass --transcript <path>.`
864
+ );
865
+ }
866
+ if (entries.length === 0) {
867
+ throw new Error(`No transcripts in ${dir}. Pass --transcript <path> to pick one manually.`);
868
+ }
869
+ const newest = entries.map((name) => {
870
+ const full = path.join(dir, name);
871
+ return { full, mtime: fs.statSync(full).mtimeMs };
872
+ }).sort((a, b) => b.mtime - a.mtime)[0];
873
+ return newest.full;
874
+ }
875
+ function extractUserImages(transcriptPath) {
876
+ let raw;
877
+ try {
878
+ raw = fs.readFileSync(transcriptPath, "utf-8");
879
+ } catch (error) {
880
+ throw new Error(`Cannot read transcript ${transcriptPath}: ${error.message}`);
881
+ }
882
+ const images = [];
883
+ for (const line of raw.split("\n")) {
884
+ if (!line.includes('"image"')) {
885
+ continue;
886
+ }
887
+ let parsed;
888
+ try {
889
+ parsed = JSON.parse(line);
890
+ } catch {
891
+ continue;
892
+ }
893
+ const message = parsed.message;
894
+ if (message?.role !== "user" || !Array.isArray(message.content)) {
895
+ continue;
896
+ }
897
+ for (const block of message.content) {
898
+ const source = block?.source;
899
+ if (block?.type === "image" && source?.type === "base64" && source.data) {
900
+ images.push({ mediaType: source.media_type ?? "image/png", data: source.data });
901
+ }
902
+ }
903
+ }
904
+ return images;
905
+ }
906
+ function recoverPastedImages(options = {}) {
907
+ const transcript = options.transcript ?? locateTranscript(options.cwd ?? process.cwd());
908
+ const count = Math.max(1, options.count ?? 1);
909
+ const outDir = options.outDir ?? path.join(os.tmpdir(), "modlens-paste");
910
+ const all = extractUserImages(transcript);
911
+ if (all.length === 0) {
912
+ throw new Error(
913
+ `No pasted images found in ${transcript}. The user may not have pasted any, or the transcript format changed; ask for a file path instead.`
914
+ );
915
+ }
916
+ fs.mkdirSync(outDir, { recursive: true });
917
+ const picked = all.slice(-count);
918
+ const images = picked.map((image) => {
919
+ const buffer = Buffer.from(image.data, "base64");
920
+ const hash = crypto.createHash("sha256").update(buffer).digest("hex").slice(0, 8);
921
+ const ext = EXT_BY_MIME[image.mediaType] ?? "png";
922
+ const filePath = path.join(outDir, `paste-${hash}.${ext}`);
923
+ fs.writeFileSync(filePath, buffer);
924
+ return { path: filePath, mediaType: image.mediaType, bytes: buffer.length };
925
+ });
926
+ return { transcript, images };
927
+ }
757
928
  const program = new Command();
758
- program.name("modlens").description("Plug-in vision for text-only LLMs: image in, structured JSON evidence out").version("2.1.0");
929
+ program.name("modlens").description("Plug-in vision for text-only LLMs: image in, structured JSON evidence out").version("2.3.0");
759
930
  program.command("analyze", { isDefault: true }).description("Analyze an image into structured JSON evidence (default command)").requiredOption("-i, --input <path|url>", "Input image path or https URL").option("-o, --output <path>", "Write result JSON to a file").option("-m, --model <name>", "Provider model name").option("-p, --provider <name>", `Vision provider (${listProviders().join(", ")})`).option("--prompt <text>", "Extra focus for this image").option("--timeout <ms>", "Provider timeout in milliseconds", "180000").option("--provider-bin <path>", "Provider binary path (default: agy)").option("--workdir <path>", "Working directory for the provider").action(async (options) => {
760
931
  try {
761
932
  const timeoutMs = Number.parseInt(options.timeout, 10);
@@ -787,6 +958,30 @@ program.command("analyze", { isDefault: true }).description("Analyze an image in
787
958
  process.exit(1);
788
959
  }
789
960
  });
961
+ program.command("recover-paste").description(
962
+ "Recover images pasted into Claude Code from the session transcript (they never hit disk otherwise)"
963
+ ).option("--count <n>", "How many recent pasted images to recover", "1").option("--out-dir <path>", "Directory to write recovered images to").option("--transcript <path>", "Explicit transcript .jsonl (default: newest for cwd)").option("--cwd <path>", "Project directory the image was pasted in", process.cwd()).action(async (options) => {
964
+ try {
965
+ const count = Number.parseInt(options.count, 10);
966
+ if (!Number.isFinite(count) || count <= 0) {
967
+ throw new Error("Invalid --count. Use a positive integer.");
968
+ }
969
+ const result = recoverPastedImages({
970
+ count,
971
+ outDir: options.outDir,
972
+ transcript: options.transcript,
973
+ cwd: options.cwd
974
+ });
975
+ process.stdout.write(`${JSON.stringify(result, null, 2)}
976
+ `);
977
+ } catch (error) {
978
+ process.stderr.write(
979
+ `Error: ${error instanceof Error ? error.message : String(error)}
980
+ `
981
+ );
982
+ process.exit(1);
983
+ }
984
+ });
790
985
  const config = program.command("config").description(`Manage ${CONFIG_PATH} (providers, keys, models)`);
791
986
  config.command("init").description(`Create a starter config at ${CONFIG_PATH}`).option("--force", "Overwrite an existing config file").action((options) => {
792
987
  try {
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@liustack/modlens",
3
- "version": "2.1.0",
3
+ "version": "2.3.0",
4
4
  "description": "Plug-in vision for text-only LLMs, powered by the free Antigravity CLI",
5
5
  "type": "module",
6
6
  "bin": {
@@ -1,6 +1,6 @@
1
1
  ---
2
2
  name: modlens
3
- description: "Plug-in vision for text-only models. Use whenever the user shares an image (local path, screenshot, photo, chart, document scan, or image URL) and the active model cannot see images or has no vision tool. Runs the modlens CLI to convert the image into structured JSON evidence: OCR text, layout, semantics, visual clues."
3
+ description: "Plug-in vision for text-only models. Use whenever the user shares an image (local path, screenshot, photo, chart, document scan, or image URL) and the active model cannot see images or has no vision tool. Runs the modlens CLI to convert the image into structured JSON evidence: OCR text, layout, semantics, visual clues. Also use when the user asks how to install, configure, or switch modlens providers (Gemini API key, OpenAI-compatible endpoints, Claude API or Claude Code CLI)."
4
4
  allowed-tools:
5
5
  - Bash
6
6
  ---
@@ -12,6 +12,7 @@ Use this skill when:
12
12
  - The user provides an image path or image URL and asks anything about it
13
13
  - The active model has no native vision (text-only model in a coding agent)
14
14
  - You need OCR text, layout, or chart/document structure as evidence before reasoning
15
+ - The user asks how to configure modlens, get an API key for it, or switch its provider: follow `references/configure.md` and run the commands for them
15
16
 
16
17
  Do not use this skill for:
17
18
 
@@ -26,7 +27,7 @@ modlens --version
26
27
 
27
28
  If `modlens` is missing, run it via `npx @liustack/modlens` instead.
28
29
 
29
- ModLens supports four vision providers. Check what is configured:
30
+ ModLens supports five vision providers. Check what is configured:
30
31
 
31
32
  ```bash
32
33
  modlens config show
@@ -36,8 +37,9 @@ modlens config show
36
37
  - **gemini-api**: needs `GEMINI_API_KEY` env or `modlens config set gemini-api.apiKey <key>` (free key from https://aistudio.google.com).
37
38
  - **openai**: any OpenAI-compatible multimodal endpoint; needs baseUrl + apiKey + model via env (`OPENAI_BASE_URL`, `OPENAI_API_KEY`) or `modlens config set openai.<field> <value>`.
38
39
  - **anthropic**: needs `ANTHROPIC_API_KEY` env or config; defaults to Claude Haiku.
40
+ - **claude-cli**: rides an existing Claude Code login (`claude`), no key, Read-only tool permissions, local files only.
39
41
 
40
- `modlens config init` writes a starter config to `~/.modlens/config.json` when none exists.
42
+ `modlens config init` writes a starter config to `~/.modlens/config.json` when none exists. Full setup recipes per provider: `references/configure.md`.
41
43
 
42
44
  ## Command
43
45
 
@@ -55,7 +57,7 @@ Optional flags:
55
57
  modlens -i <image> -o <output.json> -m <model> --prompt "<extra focus>" --timeout <ms>
56
58
  ```
57
59
 
58
- Speed expectations: `gemini-api` typically 5-10 seconds, `antigravity-cli` 15-40 seconds (full agent loop), `openai`/`anthropic` depend on the endpoint. For dense or hard images on antigravity-cli, try `-m gemini-3.1-pro-high`.
60
+ Speed expectations: `gemini-api` typically 5-10 seconds, `antigravity-cli` 15-40 seconds and `claude-cli` 20-45 seconds (full agent loops), `openai`/`anthropic` depend on the endpoint. For dense or hard images on antigravity-cli, try `-m gemini-3.1-pro-high`.
59
61
 
60
62
  ## Finding the image path in the chat
61
63
 
@@ -64,6 +66,7 @@ Harnesses rarely hand you a clean path. Look for these signals:
64
66
  - Codex wraps every pasted or attached image in a text tag like
65
67
  `<image name=[Image #1] path="/tmp/xxxx.png">`. Extract the `path` value and run modlens on it. Pasted images live in a temp file the harness already created.
66
68
  - A placeholder like `image content omitted because you do not support image input` means the harness stripped an image for you. The path tag next to it still holds the real file. Use it.
69
+ - Claude Code never writes pasted images to a regular temp file. Behind a text-only gateway you will only see a placeholder like `[Unsupported Image]` or `[Image #1]`, with no path. When that happens, run `modlens recover-paste` (add `--count <n>` for several images): it pulls the pasted image bytes out of the local session transcript and prints the recovered file paths as JSON. Feed that path to `modlens -i`. If recovery fails (transcript format is Claude Code internals and may change), fall back to asking the user to drag the image file into the terminal or type its path.
67
70
  - If the user mentions an image but no tag or path appears anywhere in the message, ask for the file path instead of guessing.
68
71
 
69
72
  ## Workflow
@@ -85,7 +88,7 @@ Top level: `{ image, provider, result, meta }`. Inside `result`:
85
88
  - `visual`: colors and style clues
86
89
  - `uncertainty[]`: what the vision engine was unsure about
87
90
 
88
- Structure is enforced by schema on antigravity-cli (`--json-schema`), gemini-api (`responseJsonSchema`), and anthropic (forced tool call). The openai route uses a template prompt plus shape validation and fails loudly on mismatch.
91
+ Structure is enforced by schema on antigravity-cli and claude-cli (`--json-schema`), gemini-api (`responseJsonSchema`), and anthropic (forced tool call). The openai route uses a template prompt plus shape validation and fails loudly on mismatch.
89
92
 
90
93
  ## Failure Handling
91
94
 
@@ -0,0 +1,85 @@
1
+ # Configuring ModLens
2
+
3
+ Read this when the user asks how to set up, configure, or switch ModLens providers. Prefer running the commands for the user over explaining them.
4
+
5
+ ## Where config lives
6
+
7
+ `~/.modlens/config.json`, managed by the CLI. Precedence: CLI flags > environment variables > config file > built-in defaults. The default provider with zero config is `antigravity-cli`.
8
+
9
+ ```bash
10
+ modlens config init # write a starter config (refuses to overwrite; --force to redo)
11
+ modlens config show # effective file, API keys masked
12
+ modlens config set provider <name> # change the default provider
13
+ modlens config set <provider>.<field> <value> # fields: apiKey, baseUrl, model
14
+ ```
15
+
16
+ `config set` writes the file with 0600 permissions.
17
+
18
+ ## Provider setup recipes
19
+
20
+ ### antigravity-cli (default, free, no key)
21
+
22
+ Needs Antigravity CLI installed and signed in:
23
+
24
+ ```bash
25
+ curl -fsSL https://antigravity.google/cli/install.sh | bash
26
+ agy # user must complete browser sign-in themselves, then exit
27
+ ```
28
+
29
+ Any free Google account works; no Google AI Pro needed. Sign-in cannot be automated, ask the user to run `agy` once.
30
+
31
+ ### gemini-api (free key, fastest free route, 5-10s)
32
+
33
+ 1. The user creates a key at https://aistudio.google.com (three minutes, no credit card, free tier does not expire).
34
+ 2. Store it either way:
35
+
36
+ ```bash
37
+ modlens config set gemini-api.apiKey <key>
38
+ # or environment: export GEMINI_API_KEY=<key>
39
+ ```
40
+
41
+ Default model `gemini-3.6-flash` has vision on the free tier (about 10-15 requests/min, 1500/day). Free-tier data may be used by Google to improve products; mention this if the user handles sensitive images.
42
+
43
+ ### openai (any OpenAI-compatible multimodal endpoint)
44
+
45
+ Needs three values. Example for DashScope qwen:
46
+
47
+ ```bash
48
+ modlens config set openai.baseUrl https://dashscope.aliyuncs.com/compatible-mode/v1
49
+ modlens config set openai.apiKey <sk-key>
50
+ modlens config set openai.model qwen3.6-27b
51
+ ```
52
+
53
+ For official OpenAI: baseUrl `https://api.openai.com/v1`, a vision-capable model. Environment equivalents: `OPENAI_BASE_URL`, `OPENAI_API_KEY`. The model must be multimodal; text-only models will fail or hallucinate. This route has no server-side schema enforcement, so occasional shape failures are surfaced as explicit errors; retry or switch provider.
54
+
55
+ ### anthropic (Claude API key)
56
+
57
+ ```bash
58
+ modlens config set anthropic.apiKey <sk-ant-key>
59
+ # or: export ANTHROPIC_API_KEY=<key>
60
+ ```
61
+
62
+ Default model is Claude Haiku (`claude-haiku-4-5-20251001`). Schema is enforced through a forced tool call.
63
+
64
+ ### claude-cli (Claude Code login, no key)
65
+
66
+ Rides an existing `claude` sign-in, so it costs the user's Claude subscription quota, not a separate API bill. Requires Claude Code installed and logged in (`claude --version` to check). Runs with `--allowedTools Read` only. Local image files only; for remote URLs use gemini-api instead. Default model alias `haiku`.
67
+
68
+ ```bash
69
+ modlens config set provider claude-cli # make it the default if the user wants
70
+ ```
71
+
72
+ ## Choosing a provider for the user
73
+
74
+ - Wants zero setup and free: `antigravity-cli` (needs agy sign-in, 15-40s per image).
75
+ - Wants fast and free: `gemini-api` (three-minute key, 5-10s).
76
+ - Already pays for Claude: `claude-cli` (no extra key) or `anthropic` (API billing).
77
+ - Has a favorite multimodal endpoint (qwen, GLM, ...): `openai`.
78
+
79
+ ## Troubleshooting
80
+
81
+ - Error names a missing env var or `config set` command: run exactly that.
82
+ - `Provider CLI not found: agy`: install Antigravity CLI or switch provider.
83
+ - `Claude CLI reported ...` or empty result: check `claude` login state.
84
+ - openai route `does not match the vision schema`: retry once, then switch to gemini-api or anthropic.
85
+ - `config init` refusing to run: the file exists; use `modlens config show` first, `--force` only if the user agrees to overwrite.