@liustack/modlens 2.0.0 → 2.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +52 -20
- package/README.zh-CN.md +60 -28
- package/dist/main.js +482 -11
- package/package.json +1 -1
- package/skills/modlens/SKILL.md +15 -9
package/README.md
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
<div align="center">
|
|
2
2
|
<img src="https://raw.githubusercontent.com/liustack/modlens/main/assets/banner.jpg" width="100%" alt="ModLens, plug-in vision for text-only LLMs" />
|
|
3
3
|
<h1>ModLens</h1>
|
|
4
|
-
<p><b>
|
|
4
|
+
<p><b>Free plug-in vision for your text-only LLM.</b></p>
|
|
5
5
|
<p>
|
|
6
6
|
<a href="https://www.npmjs.com/package/@liustack/modlens"><img src="https://img.shields.io/npm/v/@liustack/modlens" alt="npm"></a>
|
|
7
7
|
<a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-blue.svg" alt="License"></a>
|
|
@@ -9,12 +9,12 @@
|
|
|
9
9
|
<p><a href="./README.zh-CN.md">简体中文</a></p>
|
|
10
10
|
</div>
|
|
11
11
|
|
|
12
|
-
DeepSeek-V4-Flash
|
|
12
|
+
DeepSeek-V4-Flash gives you a lot of model for very little money: fast, strong, and its one real flaw is no multimodal. And it's not just DeepSeek. Every text-only model running inside Codex, Claude Code, Pi Agent, or OpenClaw hits the same wall.
|
|
13
13
|
|
|
14
|
-
|
|
14
|
+
ModLens fixes this the lightest way possible. It never touches your config and never adds a local proxy. It's just a vision plug-in, usable as a CLI or as an Agent Skill, that turns any image into structured visual evidence: text, layout, regions, entities, relations, visual clues. Under the hood it runs on [Antigravity CLI](https://antigravity.google) (`agy`), whose vision comes from free-quota Gemini 3.6 Flash. And Gemini's image understanding is famously good, good enough to embarrass most flagships, Fable 5 included. How it works:
|
|
15
15
|
|
|
16
16
|
```text
|
|
17
|
-
|
|
17
|
+
text-only model in your agent harness ──▶ modlens skill (auto-triggers on images)
|
|
18
18
|
│
|
|
19
19
|
▼
|
|
20
20
|
agy · Gemini 3.6 Flash (free quota)
|
|
@@ -23,8 +23,6 @@ your text-only model ──▶ modlens skill (auto-triggers on images)
|
|
|
23
23
|
structured JSON evidence ──▶ model answers with sight
|
|
24
24
|
```
|
|
25
25
|
|
|
26
|
-
Install the skill once and your agent starts handling images on its own. No model swap, no API key, no prompt surgery.
|
|
27
|
-
|
|
28
26
|
## Quick start
|
|
29
27
|
|
|
30
28
|
**1. Install Antigravity CLI and sign in** (one-time):
|
|
@@ -34,7 +32,7 @@ curl -fsSL https://antigravity.google/cli/install.sh | bash
|
|
|
34
32
|
agy # opens browser sign-in, then exit
|
|
35
33
|
```
|
|
36
34
|
|
|
37
|
-
**2. Install the skill.**
|
|
35
|
+
**2. Install the skill.** Just tell your agent (Claude Code, Codex, OpenClaw, Cursor, ...):
|
|
38
36
|
|
|
39
37
|
```text
|
|
40
38
|
Install the skill from https://github.com/liustack/modlens
|
|
@@ -46,7 +44,7 @@ or do it yourself:
|
|
|
46
44
|
npx -y skills add liustack/modlens
|
|
47
45
|
```
|
|
48
46
|
|
|
49
|
-
**3. Use it.**
|
|
47
|
+
**3. Use it.** Paste an image path into the CLI and ask anything. The skill fires on its own.
|
|
50
48
|
|
|
51
49
|
## See it work
|
|
52
50
|
|
|
@@ -69,14 +67,28 @@ Real output, truncated:
|
|
|
69
67
|
{ "language": "en", "text": "BEFORE YOU BUILD" }
|
|
70
68
|
]
|
|
71
69
|
},
|
|
72
|
-
"layout": {
|
|
70
|
+
"layout": {
|
|
71
|
+
"regions": [
|
|
72
|
+
{
|
|
73
|
+
"reading_order": 1,
|
|
74
|
+
"text": "/shaping BEFORE YOU BUILD",
|
|
75
|
+
"type": "other"
|
|
76
|
+
}
|
|
77
|
+
]
|
|
78
|
+
},
|
|
73
79
|
"uncertainty": []
|
|
74
80
|
},
|
|
75
81
|
"meta": { "model": "gemini-3.6-flash-low", "durationSeconds": 25.4 }
|
|
76
82
|
}
|
|
77
83
|
```
|
|
78
84
|
|
|
79
|
-
|
|
85
|
+
Here is the whole loop inside the Codex desktop app: drop in a tweet screenshot, and a text-only DeepSeek-V4-Flash reads all of it through ModLens: the caption, the engagement numbers (2.9K replies, 270K likes, 5M views), even the image's alt text. Where the resolution runs out, it says so instead of guessing.
|
|
86
|
+
|
|
87
|
+

|
|
88
|
+
|
|
89
|
+
Batch mode works too: drop three illustrations at once, and the model announces it will read them one by one through ModLens, then delivers all three descriptions in 21 seconds, design intent included.
|
|
90
|
+
|
|
91
|
+

|
|
80
92
|
|
|
81
93
|
## CLI reference
|
|
82
94
|
|
|
@@ -97,16 +109,36 @@ modlens -i <image-path-or-url> [options]
|
|
|
97
109
|
|
|
98
110
|
Reach for `-m gemini-3.1-pro-high` on dense screenshots or tricky documents. Output contract: [skills/modlens/references/output-schema.md](skills/modlens/references/output-schema.md).
|
|
99
111
|
|
|
112
|
+
## Providers and config
|
|
113
|
+
|
|
114
|
+
ModLens ships four vision providers. `antigravity-cli` stays the default: zero keys, pure free quota.
|
|
115
|
+
|
|
116
|
+
| Provider | Needs | Typical speed | Notes |
|
|
117
|
+
| :-- | :-- | :-- | :-- |
|
|
118
|
+
| `antigravity-cli` (default) | `agy` signed in | 15-40s | free quota, full agent loop |
|
|
119
|
+
| `gemini-api` | free AI Studio key | 5-10s | fastest free route, schema enforced server-side |
|
|
120
|
+
| `openai` | baseUrl + apiKey + model | endpoint-dependent | any OpenAI-compatible multimodal endpoint (qwen-vl, GLM, ...) |
|
|
121
|
+
| `anthropic` | `ANTHROPIC_API_KEY` | a few seconds | Claude Haiku by default, schema via forced tool call |
|
|
122
|
+
|
|
123
|
+
Config lives in `~/.modlens/config.json`. Environment variables override the file (`GEMINI_API_KEY`, `OPENAI_API_KEY`, `OPENAI_BASE_URL`, `ANTHROPIC_API_KEY`), and CLI flags override everything.
|
|
124
|
+
|
|
125
|
+
```bash
|
|
126
|
+
modlens config init # write a starter config
|
|
127
|
+
modlens config set gemini-api.apiKey <key> # saved with 0600 perms
|
|
128
|
+
modlens config show # keys come out masked
|
|
129
|
+
modlens config set provider gemini-api # switch the default provider
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
The free Gemini key takes three minutes at [aistudio.google.com](https://aistudio.google.com), no credit card. Or skip the manual work entirely and tell your agent: "configure modlens with my Gemini API key".
|
|
133
|
+
|
|
100
134
|
## Using it in Codex (DeepSeek and friends)
|
|
101
135
|
|
|
102
|
-
Codex speaks only the Responses API, and DeepSeek's official endpoint supports it natively. Start with the [official integration guide](https://api-docs.deepseek.com/quick_start/agent_integrations/codex/): its `models.json` declares deepseek-v4-flash as text-only (`input_modalities: ["text"]`), and that one line is what unlocks
|
|
136
|
+
Codex speaks only the Responses API, and DeepSeek's official endpoint supports it natively. Start with the [official integration guide](https://api-docs.deepseek.com/quick_start/agent_integrations/codex/): its `models.json` declares deepseek-v4-flash as text-only (`input_modalities: ["text"]`), and that one line is what unlocks the whole flow.
|
|
103
137
|
|
|
104
138
|
One catch: once text-only is declared, the Codex TUI **blocks Ctrl+V image paste outright** (`Model deepseek-v4-flash does not support image inputs`). The gate sits in the input box itself, so the image never makes it into the message. Two moves get around it, both verified end to end with deepseek-v4-flash:
|
|
105
139
|
|
|
106
140
|
- **Drag the image file into the terminal**, or type its path. The path lands as plain text, and the modlens skill picks it up from there.
|
|
107
|
-
- Attach it with `codex exec -i image.png "..."`.
|
|
108
|
-
|
|
109
|
-
Skip `models.json` (a bare custom-model config) and Codex assumes your model can see images, sending them raw, and whether that survives depends on the provider's patience. Dragging the file in is the one move that works everywhere, in every harness.
|
|
141
|
+
- Attach it with `codex exec -i image.png "..."`. The skill reads the path out of the message tag.
|
|
110
142
|
|
|
111
143
|
## Why a bridge instead of a multimodal model?
|
|
112
144
|
|
|
@@ -116,22 +148,22 @@ Skip `models.json` (a bare custom-model config) and Codex assumes your model can
|
|
|
116
148
|
|
|
117
149
|
ModSearch, ModLens's sibling project, plays the same trick for web search and page fetching: [liustack/modsearch](https://github.com/liustack/modsearch).
|
|
118
150
|
|
|
119
|
-
##
|
|
151
|
+
## Shameless plug
|
|
120
152
|
|
|
121
|
-
ModLens v2 was shaped, coded, and shipped with **[liustack](https://github.com/liustack/liustack)
|
|
153
|
+
This project runs on LIUSTACK Skills. ModLens v2 was shaped, coded, and shipped with **[liustack](https://github.com/liustack/liustack)** end to end: `shaping` before you build, `coding` while you build, `dig` when it breaks, `snapshot` when you hand off. Lighter than Superpowers, and sharper.
|
|
122
154
|
|
|
123
|
-
**ModLens
|
|
155
|
+
**ModLens gives your model eyes. LIUSTACK Skills gives your dev workflow wings:**
|
|
124
156
|
|
|
125
157
|
```bash
|
|
126
158
|
npx -y skills add liustack/liustack -g
|
|
127
159
|
```
|
|
128
160
|
|
|
129
|
-
⭐ Like
|
|
161
|
+
⭐ Like it? [Star ModLens](https://github.com/liustack/modlens) and [star liustack](https://github.com/liustack/liustack). Stars are how the next developer finds them.
|
|
130
162
|
|
|
131
163
|
## Security notes
|
|
132
164
|
|
|
133
|
-
- ModLens runs `agy` with `--dangerously-skip-permissions`, because print mode
|
|
134
|
-
- Vision output is evidence
|
|
165
|
+
- ModLens runs `agy` with `--dangerously-skip-permissions`, because print mode can fail in some setups without it. The prompt keeps the agent to reading that one image and tells it to treat image content as data, never as instructions. Even so, only point it at images you would open yourself, and run it inside a sandboxed workspace when you can.
|
|
166
|
+
- Vision output is evidence. Anything the engine cannot read lands in `uncertainty` instead of getting invented. Pixel bounding boxes and confidence scores were dropped in v2 because models fabricate them.
|
|
135
167
|
|
|
136
168
|
## Disclaimer
|
|
137
169
|
|
package/README.zh-CN.md
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
<div align="center">
|
|
2
2
|
<img src="https://raw.githubusercontent.com/liustack/modlens/main/assets/banner.jpg" width="100%" alt="ModLens,给纯文本 LLM 外挂视觉" />
|
|
3
3
|
<h1>ModLens</h1>
|
|
4
|
-
<p><b
|
|
4
|
+
<p><b>免费给你的大语言模型(纯文本 LLM)外挂视觉能力。</b></p>
|
|
5
5
|
<p>
|
|
6
6
|
<a href="https://www.npmjs.com/package/@liustack/modlens"><img src="https://img.shields.io/npm/v/@liustack/modlens" alt="npm"></a>
|
|
7
7
|
<a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-blue.svg" alt="License"></a>
|
|
@@ -9,12 +9,12 @@
|
|
|
9
9
|
<p><a href="./README.md">English</a></p>
|
|
10
10
|
</div>
|
|
11
11
|
|
|
12
|
-
DeepSeek-V4-Flash
|
|
12
|
+
DeepSeek-V4-Flash 碗大又好吃,速度快,性能强,要说唯一的缺点就是没有多模态。不仅 DeepSeek-V4-Flash,只要是纯文本语言模型,跑在 Codex、Claude Code、Pi Agent、OpenClaw 中,都有这个问题。
|
|
13
13
|
|
|
14
|
-
|
|
14
|
+
ModLens 用最轻量级方案解决这个问题。ModLens 不会入侵你的配置,也不会给你添加本地代理,ModLens 只是一个视觉外挂,有 cli 或 skill 两种模式。ModLens 能产出结构化的视觉证据:文字、版面、区块、实体、关系、视觉线索。ModLens 由 Antigravity [Antigravity CLI](https://antigravity.google)(`agy`)驱动,而 Antigravity 的视觉由免费额度的 Gemini 3.6 Flash 驱动。众所周知,Gemini 的识图能力,连 Fable 5 都吊打。原理如下:
|
|
15
15
|
|
|
16
16
|
```text
|
|
17
|
-
|
|
17
|
+
Agent Harness 中的纯文本模型 ──▶ modlens skill(遇到图片自动触发)
|
|
18
18
|
│
|
|
19
19
|
▼
|
|
20
20
|
agy · Gemini 3.6 Flash(免费额度)
|
|
@@ -23,8 +23,6 @@ DeepSeek-V4-Flash 便宜又能打,可惜是个瞎子。甩一张截图过去
|
|
|
23
23
|
结构化 JSON 证据 ──▶ 模型带着视力回答
|
|
24
24
|
```
|
|
25
25
|
|
|
26
|
-
skill 装一次,你的 agent 以后见着图片自己就处理了。模型不用换,API key 不用要,提示词也不用改。
|
|
27
|
-
|
|
28
26
|
## 快速开始
|
|
29
27
|
|
|
30
28
|
**1. 安装 Antigravity CLI 并登录**(一次性):
|
|
@@ -37,7 +35,7 @@ agy # 浏览器完成登录后退出
|
|
|
37
35
|
**2. 安装 skill。** 直接告诉你的 agent(Claude Code、Codex、OpenClaw、Cursor 等):
|
|
38
36
|
|
|
39
37
|
```text
|
|
40
|
-
|
|
38
|
+
安装这个 skill https://github.com/liustack/modlens
|
|
41
39
|
```
|
|
42
40
|
|
|
43
41
|
或者自己动手:
|
|
@@ -46,7 +44,7 @@ Install the skill from https://github.com/liustack/modlens
|
|
|
46
44
|
npx -y skills add liustack/modlens
|
|
47
45
|
```
|
|
48
46
|
|
|
49
|
-
**3. 用起来。**
|
|
47
|
+
**3. 用起来。** 在 cli 里粘贴个图片路径,随便问,skill 会自动触发。
|
|
50
48
|
|
|
51
49
|
## 看看效果
|
|
52
50
|
|
|
@@ -69,14 +67,28 @@ npx @liustack/modlens -i workflow.jpg
|
|
|
69
67
|
{ "language": "en", "text": "BEFORE YOU BUILD" }
|
|
70
68
|
]
|
|
71
69
|
},
|
|
72
|
-
"layout": {
|
|
70
|
+
"layout": {
|
|
71
|
+
"regions": [
|
|
72
|
+
{
|
|
73
|
+
"reading_order": 1,
|
|
74
|
+
"text": "/shaping BEFORE YOU BUILD",
|
|
75
|
+
"type": "other"
|
|
76
|
+
}
|
|
77
|
+
]
|
|
78
|
+
},
|
|
73
79
|
"uncertainty": []
|
|
74
80
|
},
|
|
75
81
|
"meta": { "model": "gemini-3.6-flash-low", "durationSeconds": 25.4 }
|
|
76
82
|
}
|
|
77
83
|
```
|
|
78
84
|
|
|
79
|
-
|
|
85
|
+
整条链路在 Codex 桌面 App 里跑起来是这样:丢一张推文截图,纯文本的 DeepSeek-V4-Flash 通过 ModLens 读出了全部内容:配文、互动数据(2.9K 回复、270K 点赞、5M 浏览)、连图片的 alt 文字都没放过。分辨率不够的地方,它老实说读不清,不瞎编。
|
|
86
|
+
|
|
87
|
+

|
|
88
|
+
|
|
89
|
+
批量也不在话下:一次丢三张插画进去,模型自己说「三张图我都用 modlens 视觉桥接逐张读取」,21 秒后逐张交卷,连画面的设计意图都点出来了。
|
|
90
|
+
|
|
91
|
+

|
|
80
92
|
|
|
81
93
|
## CLI 参数
|
|
82
94
|
|
|
@@ -84,19 +96,41 @@ npx @liustack/modlens -i workflow.jpg
|
|
|
84
96
|
modlens -i <图片路径或 URL> [选项]
|
|
85
97
|
```
|
|
86
98
|
|
|
87
|
-
| 参数
|
|
88
|
-
|
|
|
89
|
-
| `-i, --input <path\|url>` | 要解析的图片(必填)
|
|
90
|
-
| `-o, --output <path>`
|
|
91
|
-
| `-m, --model <name>`
|
|
92
|
-
| `-p, --provider <name>`
|
|
93
|
-
| `--prompt <text>`
|
|
94
|
-
| `--timeout <ms>`
|
|
95
|
-
| `--provider-bin <path>`
|
|
96
|
-
| `--workdir <path>`
|
|
99
|
+
| 参数 | 含义 | 默认值 |
|
|
100
|
+
| :------------------------ | :-------------------------------- | :--------------------- |
|
|
101
|
+
| `-i, --input <path\|url>` | 要解析的图片(必填) | |
|
|
102
|
+
| `-o, --output <path>` | 同时把 JSON 写入文件 | |
|
|
103
|
+
| `-m, --model <name>` | provider 模型 | `gemini-3.6-flash-low` |
|
|
104
|
+
| `-p, --provider <name>` | 视觉 provider | `antigravity-cli` |
|
|
105
|
+
| `--prompt <text>` | 额外关注点,比如 `"重点提取表格"` | |
|
|
106
|
+
| `--timeout <ms>` | provider 超时 | `180000` |
|
|
107
|
+
| `--provider-bin <path>` | provider 可执行文件 | `agy` |
|
|
108
|
+
| `--workdir <path>` | provider 运行目录 | |
|
|
97
109
|
|
|
98
110
|
截图信息密集或文档难啃,换成 `-m gemini-3.1-pro-high`。输出契约见 [skills/modlens/references/output-schema.md](skills/modlens/references/output-schema.md)。
|
|
99
111
|
|
|
112
|
+
## Provider 与配置
|
|
113
|
+
|
|
114
|
+
ModLens 内置四个视觉 provider,默认还是 `antigravity-cli`:零 key,纯免费额度。
|
|
115
|
+
|
|
116
|
+
| Provider | 需要什么 | 速度 | 说明 |
|
|
117
|
+
| :-- | :-- | :-- | :-- |
|
|
118
|
+
| `antigravity-cli`(默认) | `agy` 登录过 | 15-40 秒 | 免费额度,完整 agent 循环 |
|
|
119
|
+
| `gemini-api` | 免费 AI Studio key | 5-10 秒 | 最快的免费路线,服务端强制 schema |
|
|
120
|
+
| `openai` | baseUrl + apiKey + model | 看端点 | 任何 OpenAI 兼容的多模态端点(qwen-vl、GLM 等) |
|
|
121
|
+
| `anthropic` | `ANTHROPIC_API_KEY` | 几秒 | 默认 Claude Haiku,强制工具调用保 schema |
|
|
122
|
+
|
|
123
|
+
配置放在 `~/.modlens/config.json`,环境变量能盖过它(`GEMINI_API_KEY`、`OPENAI_API_KEY`、`OPENAI_BASE_URL`、`ANTHROPIC_API_KEY`),CLI 参数最大。
|
|
124
|
+
|
|
125
|
+
```bash
|
|
126
|
+
modlens config init # 生成配置骨架
|
|
127
|
+
modlens config set gemini-api.apiKey <key> # 落盘即 0600 权限
|
|
128
|
+
modlens config show # key 打码显示
|
|
129
|
+
modlens config set provider gemini-api # 换默认 provider
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
免费 Gemini key 去 [aistudio.google.com](https://aistudio.google.com) 领,三分钟,不要信用卡。嫌麻烦就直接跟你的 agent 说一句「帮我把 Gemini key 配进 modlens」,让它自己跑命令。
|
|
133
|
+
|
|
100
134
|
## 在 Codex 里用(DeepSeek 等纯文本模型)
|
|
101
135
|
|
|
102
136
|
Codex 只认 Responses API,DeepSeek 官方端点原生支持。先照着[官方集成文档](https://api-docs.deepseek.com/zh-cn/quick_start/agent_integrations/codex)配好:它的 `models.json` 把 deepseek-v4-flash 声明成纯文本(`input_modalities: ["text"]`),这一行就是打通下面整条链路的钥匙。
|
|
@@ -104,9 +138,7 @@ Codex 只认 Responses API,DeepSeek 官方端点原生支持。先照着[官
|
|
|
104
138
|
有个坑要注意:声明纯文本之后,Codex TUI 会**直接拦下 Ctrl+V 粘贴图片**(报错 `Model deepseek-v4-flash does not support image inputs`),闸门卡在输入框那一层,图片压根到不了消息里。能用的招数有两个,都拿 deepseek-v4-flash 端到端验证过:
|
|
105
139
|
|
|
106
140
|
- **把图片文件拖进终端**,或者手打路径。路径以纯文本形式落进消息,modlens skill 接着从这里接手。
|
|
107
|
-
- 用 `codex exec -i 图片.png "..."`
|
|
108
|
-
|
|
109
|
-
不配 `models.json`(裸的自定义模型配置)的话,Codex 会默认你的模型能看图,把图片原样发过去,能不能扛住全看服务商脾气好不好。拖文件这一招,换到哪个宿主都稳。
|
|
141
|
+
- 用 `codex exec -i 图片.png "..."` skill 从这里把路径抠出来。
|
|
110
142
|
|
|
111
143
|
## 为什么外挂,而不是换多模态模型?
|
|
112
144
|
|
|
@@ -116,11 +148,11 @@ Codex 只认 Responses API,DeepSeek 官方端点原生支持。先照着[官
|
|
|
116
148
|
|
|
117
149
|
姊妹项目 ModSearch 用同一招补上联网搜索和网页抓取:[liustack/modsearch](https://github.com/liustack/modsearch)。
|
|
118
150
|
|
|
119
|
-
##
|
|
151
|
+
## 插入一条硬广告
|
|
120
152
|
|
|
121
|
-
ModLens v2 从需求成形、编码到交付,全程用 **[liustack](https://github.com/liustack/liustack)**
|
|
153
|
+
本项目由 LIUSTACK Skills 驱动,ModLens v2 从需求成形、编码到交付,全程用 **[liustack](https://github.com/liustack/liustack)** 驱动:动手前 `shaping` 捋清楚,编码时 `coding` 上纪律,出问题 `dig` 挖根因,交接时 `snapshot` 留快照。比 Superpowers 更轻,也更强。
|
|
122
154
|
|
|
123
|
-
**ModLens 给你的模型装上眼睛,
|
|
155
|
+
**ModLens 给你的模型装上眼睛,LIUSTACK Skills 给你的开发工作流装上翅膀:**
|
|
124
156
|
|
|
125
157
|
```bash
|
|
126
158
|
npx -y skills add liustack/liustack -g
|
|
@@ -130,8 +162,8 @@ npx -y skills add liustack/liustack -g
|
|
|
130
162
|
|
|
131
163
|
## 安全说明
|
|
132
164
|
|
|
133
|
-
- ModLens 调用 `agy` 时带上 `--dangerously-skip-permissions`,因为 print
|
|
134
|
-
-
|
|
165
|
+
- ModLens 调用 `agy` 时带上 `--dangerously-skip-permissions`,因为 prompt/print 模式不带这个参数在某些场景会失败。提示词已经把 agent 限定在只读那一张图,并要求把图片内容当数据看,绝不当指令执行。即便如此,也只解析你自己敢打开的图片,尽量在沙箱化的工作目录里跑。
|
|
166
|
+
- 视觉输出是证据,引擎读不准的地方会进 `uncertainty`,而不是被编出来凑数。v2 把像素坐标和置信度分数都删了,因为模型会瞎编这两样。
|
|
135
167
|
|
|
136
168
|
## 免责声明
|
|
137
169
|
|
package/dist/main.js
CHANGED
|
@@ -3,8 +3,120 @@ import { Command } from "commander";
|
|
|
3
3
|
import * as fs from "fs";
|
|
4
4
|
import * as path from "path";
|
|
5
5
|
import { spawn } from "child_process";
|
|
6
|
+
import * as os from "os";
|
|
7
|
+
const CONFIG_DIR = path.join(os.homedir(), ".modlens");
|
|
8
|
+
const CONFIG_PATH = path.join(CONFIG_DIR, "config.json");
|
|
9
|
+
const ENV_BINDINGS = {
|
|
10
|
+
"gemini-api": { apiKey: "GEMINI_API_KEY" },
|
|
11
|
+
openai: { apiKey: "OPENAI_API_KEY", baseUrl: "OPENAI_BASE_URL" },
|
|
12
|
+
anthropic: { apiKey: "ANTHROPIC_API_KEY", baseUrl: "ANTHROPIC_BASE_URL" }
|
|
13
|
+
};
|
|
14
|
+
function loadConfigFile(configPath = CONFIG_PATH) {
|
|
15
|
+
let raw;
|
|
16
|
+
try {
|
|
17
|
+
raw = fs.readFileSync(configPath, "utf-8");
|
|
18
|
+
} catch {
|
|
19
|
+
return {};
|
|
20
|
+
}
|
|
21
|
+
try {
|
|
22
|
+
const parsed = JSON.parse(raw);
|
|
23
|
+
if (!parsed || typeof parsed !== "object") {
|
|
24
|
+
return {};
|
|
25
|
+
}
|
|
26
|
+
return parsed;
|
|
27
|
+
} catch (error) {
|
|
28
|
+
throw new Error(
|
|
29
|
+
`Failed to parse ${configPath}: ${error.message}. Fix or delete the file.`
|
|
30
|
+
);
|
|
31
|
+
}
|
|
32
|
+
}
|
|
33
|
+
function defaultProviderName(config2) {
|
|
34
|
+
return config2.provider?.trim() || "antigravity-cli";
|
|
35
|
+
}
|
|
36
|
+
function resolveProviderSettings(providerName, config2, env = process.env) {
|
|
37
|
+
const fromFile = config2.providers?.[providerName] ?? {};
|
|
38
|
+
const bindings = ENV_BINDINGS[providerName] ?? {};
|
|
39
|
+
const settings = { ...fromFile };
|
|
40
|
+
for (const [field, envName] of Object.entries(bindings)) {
|
|
41
|
+
const value = env[envName]?.trim();
|
|
42
|
+
if (value) {
|
|
43
|
+
settings[field] = value;
|
|
44
|
+
}
|
|
45
|
+
}
|
|
46
|
+
return settings;
|
|
47
|
+
}
|
|
48
|
+
function setConfigValue(dottedKey, value, configPath = CONFIG_PATH) {
|
|
49
|
+
const config2 = loadConfigFile(configPath);
|
|
50
|
+
if (dottedKey === "provider") {
|
|
51
|
+
config2.provider = value;
|
|
52
|
+
} else {
|
|
53
|
+
const dot = dottedKey.indexOf(".");
|
|
54
|
+
if (dot <= 0 || dot === dottedKey.length - 1) {
|
|
55
|
+
throw new Error(
|
|
56
|
+
`Invalid config key: ${dottedKey}. Use "provider" or "<provider>.<apiKey|baseUrl|model>".`
|
|
57
|
+
);
|
|
58
|
+
}
|
|
59
|
+
const providerName = dottedKey.slice(0, dot);
|
|
60
|
+
const field = dottedKey.slice(dot + 1);
|
|
61
|
+
if (!["apiKey", "baseUrl", "model"].includes(field)) {
|
|
62
|
+
throw new Error(`Unknown config field: ${field}. Use apiKey, baseUrl, or model.`);
|
|
63
|
+
}
|
|
64
|
+
config2.providers ??= {};
|
|
65
|
+
config2.providers[providerName] ??= {};
|
|
66
|
+
config2.providers[providerName][field] = value;
|
|
67
|
+
}
|
|
68
|
+
fs.mkdirSync(path.dirname(configPath), { recursive: true });
|
|
69
|
+
fs.writeFileSync(configPath, `${JSON.stringify(config2, null, 2)}
|
|
70
|
+
`, { mode: 384 });
|
|
71
|
+
try {
|
|
72
|
+
fs.chmodSync(configPath, 384);
|
|
73
|
+
} catch {
|
|
74
|
+
}
|
|
75
|
+
}
|
|
76
|
+
const CONFIG_TEMPLATE = {
|
|
77
|
+
provider: "antigravity-cli",
|
|
78
|
+
providers: {
|
|
79
|
+
"antigravity-cli": { model: "gemini-3.6-flash-low" },
|
|
80
|
+
"gemini-api": { apiKey: "", model: "gemini-3.6-flash" },
|
|
81
|
+
openai: { baseUrl: "", apiKey: "", model: "" },
|
|
82
|
+
anthropic: { apiKey: "", model: "claude-haiku-4-5-20251001" }
|
|
83
|
+
}
|
|
84
|
+
};
|
|
85
|
+
function initConfigFile(configPath = CONFIG_PATH, force = false) {
|
|
86
|
+
if (!force && fs.existsSync(configPath)) {
|
|
87
|
+
throw new Error(`${configPath} already exists. Use --force to overwrite.`);
|
|
88
|
+
}
|
|
89
|
+
fs.mkdirSync(path.dirname(configPath), { recursive: true });
|
|
90
|
+
fs.writeFileSync(configPath, `${JSON.stringify(CONFIG_TEMPLATE, null, 2)}
|
|
91
|
+
`, { mode: 384 });
|
|
92
|
+
try {
|
|
93
|
+
fs.chmodSync(configPath, 384);
|
|
94
|
+
} catch {
|
|
95
|
+
}
|
|
96
|
+
}
|
|
97
|
+
function renderConfig(config2) {
|
|
98
|
+
const masked = {
|
|
99
|
+
...config2,
|
|
100
|
+
providers: Object.fromEntries(
|
|
101
|
+
Object.entries(config2.providers ?? {}).map(([name, settings]) => [
|
|
102
|
+
name,
|
|
103
|
+
{
|
|
104
|
+
...settings,
|
|
105
|
+
...settings.apiKey ? { apiKey: maskKey(settings.apiKey) } : {}
|
|
106
|
+
}
|
|
107
|
+
])
|
|
108
|
+
)
|
|
109
|
+
};
|
|
110
|
+
return JSON.stringify(masked, null, 2);
|
|
111
|
+
}
|
|
112
|
+
function maskKey(key) {
|
|
113
|
+
if (key.length <= 8) {
|
|
114
|
+
return "****";
|
|
115
|
+
}
|
|
116
|
+
return `${key.slice(0, 6)}...${key.slice(-2)}`;
|
|
117
|
+
}
|
|
6
118
|
function buildVisionPrompt(options) {
|
|
7
|
-
const readInstruction = options.imageKind === "remote" ? `Fetch the image at this URL and analyze it: ${options.imageSource}` : `Read the image file at this path and analyze it: ${options.imageSource}`;
|
|
119
|
+
const readInstruction = options.imageKind === "inline" ? "Analyze the image attached to this message." : options.imageKind === "remote" ? `Fetch the image at this URL and analyze it: ${options.imageSource}` : `Read the image file at this path and analyze it: ${options.imageSource}`;
|
|
8
120
|
const basePrompt = `${readInstruction}
|
|
9
121
|
|
|
10
122
|
You are a vision parsing engine for a text-only LLM.
|
|
@@ -204,19 +316,320 @@ const antigravityCliProvider = {
|
|
|
204
316
|
buildInvocation: buildAntigravityInvocation,
|
|
205
317
|
parseOutput: parseAntigravityOutput
|
|
206
318
|
};
|
|
319
|
+
const MIME_BY_EXT = {
|
|
320
|
+
".jpg": "image/jpeg",
|
|
321
|
+
".jpeg": "image/jpeg",
|
|
322
|
+
".png": "image/png",
|
|
323
|
+
".webp": "image/webp",
|
|
324
|
+
".gif": "image/gif",
|
|
325
|
+
".heic": "image/heic",
|
|
326
|
+
".heif": "image/heif"
|
|
327
|
+
};
|
|
328
|
+
function mimeTypeFor(source) {
|
|
329
|
+
const ext = path.extname(new URL(source, "file:///").pathname).toLowerCase();
|
|
330
|
+
return MIME_BY_EXT[ext] ?? "image/jpeg";
|
|
331
|
+
}
|
|
332
|
+
function readLocalImageBase64(filePath) {
|
|
333
|
+
const data = fs.readFileSync(filePath).toString("base64");
|
|
334
|
+
return { data, mimeType: mimeTypeFor(filePath) };
|
|
335
|
+
}
|
|
336
|
+
async function fetchRemoteImageBase64(url, timeoutMs) {
|
|
337
|
+
const response = await fetch(url, { signal: AbortSignal.timeout(timeoutMs) });
|
|
338
|
+
if (!response.ok) {
|
|
339
|
+
throw new Error(`Failed to download image (${response.status}): ${url}`);
|
|
340
|
+
}
|
|
341
|
+
const buffer = Buffer.from(await response.arrayBuffer());
|
|
342
|
+
const contentType = response.headers.get("content-type")?.split(";")[0]?.trim();
|
|
343
|
+
return {
|
|
344
|
+
data: buffer.toString("base64"),
|
|
345
|
+
mimeType: contentType?.startsWith("image/") ? contentType : mimeTypeFor(url)
|
|
346
|
+
};
|
|
347
|
+
}
|
|
348
|
+
function extractJson(text) {
|
|
349
|
+
const trimmed = text.trim();
|
|
350
|
+
const direct = tryParse(trimmed);
|
|
351
|
+
if (direct !== null) {
|
|
352
|
+
return direct;
|
|
353
|
+
}
|
|
354
|
+
const fenced = /```(?:json)?\s*([\s\S]*?)```/i.exec(trimmed);
|
|
355
|
+
if (fenced) {
|
|
356
|
+
const parsed = tryParse(fenced[1].trim());
|
|
357
|
+
if (parsed !== null) {
|
|
358
|
+
return parsed;
|
|
359
|
+
}
|
|
360
|
+
}
|
|
361
|
+
const first = trimmed.indexOf("{");
|
|
362
|
+
const last = trimmed.lastIndexOf("}");
|
|
363
|
+
if (first >= 0 && last > first) {
|
|
364
|
+
return tryParse(trimmed.slice(first, last + 1));
|
|
365
|
+
}
|
|
366
|
+
return null;
|
|
367
|
+
}
|
|
368
|
+
function tryParse(text) {
|
|
369
|
+
try {
|
|
370
|
+
return JSON.parse(text);
|
|
371
|
+
} catch {
|
|
372
|
+
return null;
|
|
373
|
+
}
|
|
374
|
+
}
|
|
375
|
+
const ANTHROPIC_DEFAULT_MODEL = "claude-haiku-4-5-20251001";
|
|
376
|
+
const DEFAULT_BASE_URL$1 = "https://api.anthropic.com";
|
|
377
|
+
const TOOL_NAME = "report_vision_evidence";
|
|
378
|
+
async function executeAnthropicApi(options) {
|
|
379
|
+
const apiKey = options.settings?.apiKey;
|
|
380
|
+
if (!apiKey) {
|
|
381
|
+
throw new Error(
|
|
382
|
+
"anthropic provider needs an API key. Set ANTHROPIC_API_KEY, or run: modlens config set anthropic.apiKey <key>"
|
|
383
|
+
);
|
|
384
|
+
}
|
|
385
|
+
const model = options.model || options.settings?.model || ANTHROPIC_DEFAULT_MODEL;
|
|
386
|
+
const baseUrl = (options.settings?.baseUrl || DEFAULT_BASE_URL$1).replace(/\/$/, "");
|
|
387
|
+
const imageSource = options.imageKind === "remote" ? { type: "url", url: options.imageSource } : (() => {
|
|
388
|
+
const image = readLocalImageBase64(options.imageSource);
|
|
389
|
+
return {
|
|
390
|
+
type: "base64",
|
|
391
|
+
media_type: image.mimeType,
|
|
392
|
+
data: image.data
|
|
393
|
+
};
|
|
394
|
+
})();
|
|
395
|
+
const prompt = `${buildVisionPrompt({
|
|
396
|
+
imageSource: options.imageSource,
|
|
397
|
+
imageKind: "inline",
|
|
398
|
+
extraPrompt: options.extraPrompt
|
|
399
|
+
})}
|
|
400
|
+
|
|
401
|
+
Report your findings by calling the ${TOOL_NAME} tool.`;
|
|
402
|
+
const startedAt = Date.now();
|
|
403
|
+
const response = await fetch(`${baseUrl}/v1/messages`, {
|
|
404
|
+
method: "POST",
|
|
405
|
+
headers: {
|
|
406
|
+
"x-api-key": apiKey,
|
|
407
|
+
"anthropic-version": "2023-06-01",
|
|
408
|
+
"Content-Type": "application/json"
|
|
409
|
+
},
|
|
410
|
+
body: JSON.stringify({
|
|
411
|
+
model,
|
|
412
|
+
max_tokens: 4096,
|
|
413
|
+
tools: [
|
|
414
|
+
{
|
|
415
|
+
name: TOOL_NAME,
|
|
416
|
+
description: "Report the structured visual evidence extracted from the image.",
|
|
417
|
+
input_schema: VISION_RESULT_SCHEMA
|
|
418
|
+
}
|
|
419
|
+
],
|
|
420
|
+
tool_choice: { type: "tool", name: TOOL_NAME },
|
|
421
|
+
messages: [
|
|
422
|
+
{
|
|
423
|
+
role: "user",
|
|
424
|
+
content: [
|
|
425
|
+
{ type: "image", source: imageSource },
|
|
426
|
+
{ type: "text", text: prompt }
|
|
427
|
+
]
|
|
428
|
+
}
|
|
429
|
+
]
|
|
430
|
+
}),
|
|
431
|
+
signal: AbortSignal.timeout(options.timeoutMs)
|
|
432
|
+
});
|
|
433
|
+
if (!response.ok) {
|
|
434
|
+
const body = await response.text();
|
|
435
|
+
throw new Error(`Anthropic API error ${response.status}: ${truncate$2(body)}`);
|
|
436
|
+
}
|
|
437
|
+
const payload = await response.json();
|
|
438
|
+
const toolUse = payload.content?.find((block) => block.type === "tool_use");
|
|
439
|
+
if (!toolUse?.input) {
|
|
440
|
+
throw new Error("Anthropic API returned no tool_use block.");
|
|
441
|
+
}
|
|
442
|
+
return {
|
|
443
|
+
result: toolUse.input,
|
|
444
|
+
meta: {
|
|
445
|
+
conversationId: null,
|
|
446
|
+
durationSeconds: (Date.now() - startedAt) / 1e3,
|
|
447
|
+
usage: payload.usage ?? null
|
|
448
|
+
}
|
|
449
|
+
};
|
|
450
|
+
}
|
|
451
|
+
function truncate$2(text) {
|
|
452
|
+
return text.length > 300 ? `${text.slice(0, 300)}...` : text;
|
|
453
|
+
}
|
|
454
|
+
const anthropicApiProvider = {
|
|
455
|
+
name: "anthropic",
|
|
456
|
+
defaultModel: ANTHROPIC_DEFAULT_MODEL,
|
|
457
|
+
execute: executeAnthropicApi
|
|
458
|
+
};
|
|
459
|
+
const GEMINI_API_DEFAULT_MODEL = "gemini-3.6-flash";
|
|
460
|
+
const DEFAULT_BASE_URL = "https://generativelanguage.googleapis.com";
|
|
461
|
+
async function executeGeminiApi(options) {
|
|
462
|
+
const apiKey = options.settings?.apiKey;
|
|
463
|
+
if (!apiKey) {
|
|
464
|
+
throw new Error(
|
|
465
|
+
"gemini-api provider needs an API key. Set GEMINI_API_KEY, or run: modlens config set gemini-api.apiKey <key> (free key: https://aistudio.google.com)"
|
|
466
|
+
);
|
|
467
|
+
}
|
|
468
|
+
const model = options.model || options.settings?.model || GEMINI_API_DEFAULT_MODEL;
|
|
469
|
+
const baseUrl = (options.settings?.baseUrl || DEFAULT_BASE_URL).replace(/\/$/, "");
|
|
470
|
+
const image = options.imageKind === "remote" ? await fetchRemoteImageBase64(options.imageSource, options.timeoutMs) : readLocalImageBase64(options.imageSource);
|
|
471
|
+
const prompt = buildVisionPrompt({
|
|
472
|
+
imageSource: options.imageSource,
|
|
473
|
+
imageKind: "inline",
|
|
474
|
+
extraPrompt: options.extraPrompt
|
|
475
|
+
});
|
|
476
|
+
const startedAt = Date.now();
|
|
477
|
+
const response = await fetch(`${baseUrl}/v1beta/models/${model}:generateContent`, {
|
|
478
|
+
method: "POST",
|
|
479
|
+
headers: {
|
|
480
|
+
"x-goog-api-key": apiKey,
|
|
481
|
+
"Content-Type": "application/json"
|
|
482
|
+
},
|
|
483
|
+
body: JSON.stringify({
|
|
484
|
+
contents: [
|
|
485
|
+
{
|
|
486
|
+
parts: [
|
|
487
|
+
{ inline_data: { mime_type: image.mimeType, data: image.data } },
|
|
488
|
+
{ text: prompt }
|
|
489
|
+
]
|
|
490
|
+
}
|
|
491
|
+
],
|
|
492
|
+
generationConfig: {
|
|
493
|
+
responseMimeType: "application/json",
|
|
494
|
+
responseJsonSchema: VISION_RESULT_SCHEMA
|
|
495
|
+
}
|
|
496
|
+
}),
|
|
497
|
+
signal: AbortSignal.timeout(options.timeoutMs)
|
|
498
|
+
});
|
|
499
|
+
if (!response.ok) {
|
|
500
|
+
const body = await response.text();
|
|
501
|
+
throw new Error(`Gemini API error ${response.status}: ${truncate$1(body)}`);
|
|
502
|
+
}
|
|
503
|
+
const payload = await response.json();
|
|
504
|
+
const text = payload.candidates?.[0]?.content?.parts?.map((part) => part.text ?? "").join("");
|
|
505
|
+
if (!text) {
|
|
506
|
+
throw new Error("Gemini API returned no text candidate.");
|
|
507
|
+
}
|
|
508
|
+
let result;
|
|
509
|
+
try {
|
|
510
|
+
result = JSON.parse(text);
|
|
511
|
+
} catch {
|
|
512
|
+
throw new Error(`Gemini API returned non-JSON output: ${truncate$1(text)}`);
|
|
513
|
+
}
|
|
514
|
+
return {
|
|
515
|
+
result,
|
|
516
|
+
meta: {
|
|
517
|
+
conversationId: null,
|
|
518
|
+
durationSeconds: (Date.now() - startedAt) / 1e3,
|
|
519
|
+
usage: payload.usageMetadata ?? null
|
|
520
|
+
}
|
|
521
|
+
};
|
|
522
|
+
}
|
|
523
|
+
function truncate$1(text) {
|
|
524
|
+
return text.length > 300 ? `${text.slice(0, 300)}...` : text;
|
|
525
|
+
}
|
|
526
|
+
const geminiApiProvider = {
|
|
527
|
+
name: "gemini-api",
|
|
528
|
+
defaultModel: GEMINI_API_DEFAULT_MODEL,
|
|
529
|
+
execute: executeGeminiApi
|
|
530
|
+
};
|
|
531
|
+
async function executeOpenaiCompat(options) {
|
|
532
|
+
const apiKey = options.settings?.apiKey;
|
|
533
|
+
const baseUrl = options.settings?.baseUrl?.replace(/\/$/, "");
|
|
534
|
+
const model = options.model || options.settings?.model;
|
|
535
|
+
if (!apiKey || !baseUrl || !model) {
|
|
536
|
+
throw new Error(
|
|
537
|
+
"openai provider needs baseUrl, apiKey, and model. Set OPENAI_BASE_URL and OPENAI_API_KEY, or run: modlens config set openai.baseUrl <url> / openai.apiKey <key> / openai.model <name>"
|
|
538
|
+
);
|
|
539
|
+
}
|
|
540
|
+
const imageUrl = options.imageKind === "remote" ? options.imageSource : toDataUrl(readLocalImageBase64(options.imageSource));
|
|
541
|
+
const prompt = `${buildVisionPrompt({
|
|
542
|
+
imageSource: options.imageSource,
|
|
543
|
+
imageKind: "inline",
|
|
544
|
+
extraPrompt: options.extraPrompt
|
|
545
|
+
})}
|
|
546
|
+
|
|
547
|
+
Respond with ONE JSON object only, no markdown fences, no commentary. Fill this exact structure with your findings from the image (do not repeat this template literally, replace every value):
|
|
548
|
+
{"summary":"one paragraph describing the image","ocr":{"full_text":"all visible text","lines":[{"text":"one line","language":"en"}]},"layout":{"regions":[{"type":"title|subtitle|paragraph|list|table|chart|form|code|image|icon|other","reading_order":1,"text":"region text"}]},"semantics":{"scene":"what kind of scene","intent":"what the image is for","entities":[{"name":"entity","type":"kind","evidence":"where seen"}],"relations":[{"subject":"a","predicate":"relates to","object":"b"}]},"visual":{"dominant_colors":["color"],"style":"visual style","notes":["notable visual detail"]},"uncertainty":["anything unreadable or ambiguous"]}`;
|
|
549
|
+
const startedAt = Date.now();
|
|
550
|
+
const response = await fetch(`${baseUrl}/chat/completions`, {
|
|
551
|
+
method: "POST",
|
|
552
|
+
headers: {
|
|
553
|
+
Authorization: `Bearer ${apiKey}`,
|
|
554
|
+
"Content-Type": "application/json"
|
|
555
|
+
},
|
|
556
|
+
body: JSON.stringify({
|
|
557
|
+
model,
|
|
558
|
+
messages: [
|
|
559
|
+
{
|
|
560
|
+
role: "user",
|
|
561
|
+
content: [
|
|
562
|
+
{ type: "image_url", image_url: { url: imageUrl } },
|
|
563
|
+
{ type: "text", text: prompt }
|
|
564
|
+
]
|
|
565
|
+
}
|
|
566
|
+
]
|
|
567
|
+
}),
|
|
568
|
+
signal: AbortSignal.timeout(options.timeoutMs)
|
|
569
|
+
});
|
|
570
|
+
if (!response.ok) {
|
|
571
|
+
const body = await response.text();
|
|
572
|
+
throw new Error(`OpenAI-compatible API error ${response.status}: ${truncate(body)}`);
|
|
573
|
+
}
|
|
574
|
+
const payload = await response.json();
|
|
575
|
+
const text = payload.choices?.[0]?.message?.content;
|
|
576
|
+
if (!text) {
|
|
577
|
+
throw new Error("OpenAI-compatible API returned no message content.");
|
|
578
|
+
}
|
|
579
|
+
const result = extractJson(text);
|
|
580
|
+
if (result === null) {
|
|
581
|
+
throw new Error(`OpenAI-compatible API returned non-JSON output: ${truncate(text)}`);
|
|
582
|
+
}
|
|
583
|
+
const shaped = result;
|
|
584
|
+
if (typeof shaped.summary !== "string" || typeof shaped.ocr !== "object") {
|
|
585
|
+
throw new Error(
|
|
586
|
+
`OpenAI-compatible API returned JSON that does not match the vision schema (missing summary/ocr). Retry, or switch to gemini-api / anthropic for enforced schemas. Got: ${truncate(text)}`
|
|
587
|
+
);
|
|
588
|
+
}
|
|
589
|
+
return {
|
|
590
|
+
result,
|
|
591
|
+
meta: {
|
|
592
|
+
conversationId: null,
|
|
593
|
+
durationSeconds: (Date.now() - startedAt) / 1e3,
|
|
594
|
+
usage: payload.usage ?? null
|
|
595
|
+
}
|
|
596
|
+
};
|
|
597
|
+
}
|
|
598
|
+
function toDataUrl(image) {
|
|
599
|
+
return `data:${image.mimeType};base64,${image.data}`;
|
|
600
|
+
}
|
|
601
|
+
function truncate(text) {
|
|
602
|
+
return text.length > 300 ? `${text.slice(0, 300)}...` : text;
|
|
603
|
+
}
|
|
604
|
+
const openaiCompatProvider = {
|
|
605
|
+
name: "openai",
|
|
606
|
+
defaultModel: "",
|
|
607
|
+
execute: executeOpenaiCompat
|
|
608
|
+
};
|
|
207
609
|
const PROVIDERS = {
|
|
208
610
|
"antigravity-cli": antigravityCliProvider,
|
|
209
611
|
antigravity: antigravityCliProvider,
|
|
210
|
-
agy: antigravityCliProvider
|
|
612
|
+
agy: antigravityCliProvider,
|
|
613
|
+
"gemini-api": geminiApiProvider,
|
|
614
|
+
gemini: geminiApiProvider,
|
|
615
|
+
openai: openaiCompatProvider,
|
|
616
|
+
"openai-compat": openaiCompatProvider,
|
|
617
|
+
anthropic: anthropicApiProvider,
|
|
618
|
+
claude: anthropicApiProvider
|
|
211
619
|
};
|
|
212
620
|
function resolveProvider(providerName = "antigravity-cli") {
|
|
213
621
|
const normalized = providerName.trim().toLowerCase();
|
|
214
622
|
const provider = PROVIDERS[normalized];
|
|
215
623
|
if (!provider) {
|
|
216
|
-
throw new Error(
|
|
624
|
+
throw new Error(
|
|
625
|
+
`Unsupported provider: ${providerName}. Available: ${listProviders().join(", ")}`
|
|
626
|
+
);
|
|
217
627
|
}
|
|
218
628
|
return provider;
|
|
219
629
|
}
|
|
630
|
+
function listProviders() {
|
|
631
|
+
return [...new Set(Object.values(PROVIDERS).map((provider) => provider.name))];
|
|
632
|
+
}
|
|
220
633
|
const DEFAULT_TIMEOUT_MS = 18e4;
|
|
221
634
|
const KILL_GRACE_MS = 3e4;
|
|
222
635
|
async function analyzeImage(options) {
|
|
@@ -224,20 +637,35 @@ async function analyzeImage(options) {
|
|
|
224
637
|
if (resolvedInput.kind === "local") {
|
|
225
638
|
validateInputFile(resolvedInput.source);
|
|
226
639
|
}
|
|
227
|
-
const
|
|
640
|
+
const config2 = options.config ?? loadConfigFile();
|
|
641
|
+
const provider = resolveProvider(options.provider || defaultProviderName(config2));
|
|
642
|
+
const settings = resolveProviderSettings(provider.name, config2);
|
|
228
643
|
const timeoutMs = options.timeoutMs ?? DEFAULT_TIMEOUT_MS;
|
|
229
|
-
const model = options.model || provider.defaultModel;
|
|
230
|
-
const
|
|
644
|
+
const model = options.model || settings.model || provider.defaultModel;
|
|
645
|
+
const providerOptions = {
|
|
231
646
|
imageSource: resolvedInput.source,
|
|
232
647
|
imageKind: resolvedInput.kind,
|
|
233
648
|
model,
|
|
234
649
|
extraPrompt: options.prompt,
|
|
235
650
|
providerBin: options.providerBin,
|
|
236
651
|
workdir: options.workdir,
|
|
237
|
-
timeoutMs
|
|
238
|
-
|
|
239
|
-
|
|
240
|
-
|
|
652
|
+
timeoutMs,
|
|
653
|
+
settings
|
|
654
|
+
};
|
|
655
|
+
let parsed;
|
|
656
|
+
if (provider.execute) {
|
|
657
|
+
parsed = await provider.execute(providerOptions);
|
|
658
|
+
} else if (provider.buildInvocation && provider.parseOutput) {
|
|
659
|
+
const invocation = provider.buildInvocation(providerOptions);
|
|
660
|
+
const commandResult = await runCommand(
|
|
661
|
+
provider.name,
|
|
662
|
+
invocation,
|
|
663
|
+
timeoutMs + KILL_GRACE_MS
|
|
664
|
+
);
|
|
665
|
+
parsed = provider.parseOutput(commandResult.stdout);
|
|
666
|
+
} else {
|
|
667
|
+
throw new Error(`Provider ${provider.name} implements neither execute nor buildInvocation.`);
|
|
668
|
+
}
|
|
241
669
|
return {
|
|
242
670
|
image: resolvedInput.source,
|
|
243
671
|
provider: provider.name,
|
|
@@ -327,7 +755,8 @@ function runCommand(providerName, invocation, timeoutMs) {
|
|
|
327
755
|
});
|
|
328
756
|
}
|
|
329
757
|
const program = new Command();
|
|
330
|
-
program.name("modlens").description("Plug-in vision for text-only LLMs: image in, structured JSON evidence out").version("2.
|
|
758
|
+
program.name("modlens").description("Plug-in vision for text-only LLMs: image in, structured JSON evidence out").version("2.1.0");
|
|
759
|
+
program.command("analyze", { isDefault: true }).description("Analyze an image into structured JSON evidence (default command)").requiredOption("-i, --input <path|url>", "Input image path or https URL").option("-o, --output <path>", "Write result JSON to a file").option("-m, --model <name>", "Provider model name").option("-p, --provider <name>", `Vision provider (${listProviders().join(", ")})`).option("--prompt <text>", "Extra focus for this image").option("--timeout <ms>", "Provider timeout in milliseconds", "180000").option("--provider-bin <path>", "Provider binary path (default: agy)").option("--workdir <path>", "Working directory for the provider").action(async (options) => {
|
|
331
760
|
try {
|
|
332
761
|
const timeoutMs = Number.parseInt(options.timeout, 10);
|
|
333
762
|
if (!Number.isFinite(timeoutMs) || timeoutMs <= 0) {
|
|
@@ -358,4 +787,46 @@ program.name("modlens").description("Plug-in vision for text-only LLMs: image in
|
|
|
358
787
|
process.exit(1);
|
|
359
788
|
}
|
|
360
789
|
});
|
|
790
|
+
const config = program.command("config").description(`Manage ${CONFIG_PATH} (providers, keys, models)`);
|
|
791
|
+
config.command("init").description(`Create a starter config at ${CONFIG_PATH}`).option("--force", "Overwrite an existing config file").action((options) => {
|
|
792
|
+
try {
|
|
793
|
+
initConfigFile(CONFIG_PATH, Boolean(options.force));
|
|
794
|
+
process.stdout.write(
|
|
795
|
+
`Created ${CONFIG_PATH}
|
|
796
|
+
Fill in the apiKey fields you need, or use: modlens config set <provider>.apiKey <key>
|
|
797
|
+
`
|
|
798
|
+
);
|
|
799
|
+
} catch (error) {
|
|
800
|
+
process.stderr.write(
|
|
801
|
+
`Error: ${error instanceof Error ? error.message : String(error)}
|
|
802
|
+
`
|
|
803
|
+
);
|
|
804
|
+
process.exit(1);
|
|
805
|
+
}
|
|
806
|
+
});
|
|
807
|
+
config.command("set <key> <value>").description("Set a value, e.g. modlens config set gemini-api.apiKey <key>").action((key, value) => {
|
|
808
|
+
try {
|
|
809
|
+
setConfigValue(key, value);
|
|
810
|
+
process.stdout.write(`Saved ${key} to ${CONFIG_PATH}
|
|
811
|
+
`);
|
|
812
|
+
} catch (error) {
|
|
813
|
+
process.stderr.write(
|
|
814
|
+
`Error: ${error instanceof Error ? error.message : String(error)}
|
|
815
|
+
`
|
|
816
|
+
);
|
|
817
|
+
process.exit(1);
|
|
818
|
+
}
|
|
819
|
+
});
|
|
820
|
+
config.command("show").description("Print the effective config with API keys masked").action(() => {
|
|
821
|
+
try {
|
|
822
|
+
process.stdout.write(`${renderConfig(loadConfigFile())}
|
|
823
|
+
`);
|
|
824
|
+
} catch (error) {
|
|
825
|
+
process.stderr.write(
|
|
826
|
+
`Error: ${error instanceof Error ? error.message : String(error)}
|
|
827
|
+
`
|
|
828
|
+
);
|
|
829
|
+
process.exit(1);
|
|
830
|
+
}
|
|
831
|
+
});
|
|
361
832
|
program.parse();
|
package/package.json
CHANGED
package/skills/modlens/SKILL.md
CHANGED
|
@@ -22,23 +22,29 @@ Do not use this skill for:
|
|
|
22
22
|
|
|
23
23
|
```bash
|
|
24
24
|
modlens --version
|
|
25
|
-
agy --version
|
|
26
25
|
```
|
|
27
26
|
|
|
28
27
|
If `modlens` is missing, run it via `npx @liustack/modlens` instead.
|
|
29
28
|
|
|
30
|
-
|
|
29
|
+
ModLens supports four vision providers. Check what is configured:
|
|
31
30
|
|
|
32
31
|
```bash
|
|
33
|
-
|
|
32
|
+
modlens config show
|
|
34
33
|
```
|
|
35
34
|
|
|
36
|
-
|
|
35
|
+
- **antigravity-cli** (default, no key needed): needs `agy` installed and signed in. If `agy --version` fails: `curl -fsSL https://antigravity.google/cli/install.sh | bash`, then ask the user to run `agy` once and complete the Google sign-in (cannot be done non-interactively).
|
|
36
|
+
- **gemini-api**: needs `GEMINI_API_KEY` env or `modlens config set gemini-api.apiKey <key>` (free key from https://aistudio.google.com).
|
|
37
|
+
- **openai**: any OpenAI-compatible multimodal endpoint; needs baseUrl + apiKey + model via env (`OPENAI_BASE_URL`, `OPENAI_API_KEY`) or `modlens config set openai.<field> <value>`.
|
|
38
|
+
- **anthropic**: needs `ANTHROPIC_API_KEY` env or config; defaults to Claude Haiku.
|
|
39
|
+
|
|
40
|
+
`modlens config init` writes a starter config to `~/.modlens/config.json` when none exists.
|
|
37
41
|
|
|
38
42
|
## Command
|
|
39
43
|
|
|
40
44
|
```bash
|
|
41
45
|
modlens -i <image-path-or-url>
|
|
46
|
+
# pick a provider explicitly
|
|
47
|
+
modlens -i <image> -p gemini-api
|
|
42
48
|
# or without a global install
|
|
43
49
|
npx @liustack/modlens -i <image-path-or-url>
|
|
44
50
|
```
|
|
@@ -49,8 +55,7 @@ Optional flags:
|
|
|
49
55
|
modlens -i <image> -o <output.json> -m <model> --prompt "<extra focus>" --timeout <ms>
|
|
50
56
|
```
|
|
51
57
|
|
|
52
|
-
-
|
|
53
|
-
- A run typically takes 15-40 seconds. Do not treat silence as a hang before the timeout.
|
|
58
|
+
Speed expectations: `gemini-api` typically 5-10 seconds, `antigravity-cli` 15-40 seconds (full agent loop), `openai`/`anthropic` depend on the endpoint. For dense or hard images on antigravity-cli, try `-m gemini-3.1-pro-high`.
|
|
54
59
|
|
|
55
60
|
## Finding the image path in the chat
|
|
56
61
|
|
|
@@ -80,10 +85,11 @@ Top level: `{ image, provider, result, meta }`. Inside `result`:
|
|
|
80
85
|
- `visual`: colors and style clues
|
|
81
86
|
- `uncertainty[]`: what the vision engine was unsure about
|
|
82
87
|
|
|
83
|
-
Structure is enforced by
|
|
88
|
+
Structure is enforced by schema on antigravity-cli (`--json-schema`), gemini-api (`responseJsonSchema`), and anthropic (forced tool call). The openai route uses a template prompt plus shape validation and fails loudly on mismatch.
|
|
84
89
|
|
|
85
90
|
## Failure Handling
|
|
86
91
|
|
|
87
|
-
-
|
|
88
|
-
-
|
|
92
|
+
- `Provider CLI not found`: Antigravity CLI is not installed. Install it, or switch provider: `-p gemini-api`.
|
|
93
|
+
- Missing key errors name the exact env var and `config set` command to run. Relay that to the user.
|
|
94
|
+
- `does not match the vision schema` on the openai route: retry once, then switch to `-p gemini-api` or `-p anthropic` for enforced schemas.
|
|
89
95
|
- Timeouts: retry once with `--timeout 300000`. If it still fails, report the exact error instead of fabricating image content.
|