@liustack/modlens 2.7.11 → 3.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +186 -0
- package/README.md +137 -146
- package/README.zh-CN.md +142 -151
- package/SECURITY.md +17 -0
- package/dist/main.js +798 -401
- package/docs/commit.md +76 -0
- package/docs/harness-setup.md +43 -0
- package/docs/research-gemini-claude-skills.md +48 -0
- package/docs/security.md +31 -0
- package/docs/testing.md +29 -0
- package/docs/troubleshooting.md +132 -0
- package/package.json +13 -4
- package/skills/modlens/references/configure.md +2 -0
package/CHANGELOG.md
ADDED
|
@@ -0,0 +1,186 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## 3.0.0 - 2026-08-06
|
|
4
|
+
|
|
5
|
+
A code-review pass. Two user-facing bugs, a stack of doc corrections, and the tooling a public repo is expected to carry.
|
|
6
|
+
|
|
7
|
+
- BREAKING: requires Node 22.13+. The floor was Node 18 with a special note that OpenCode paste recovery needed 22.13 for `node:sqlite`. That split is gone: 22.13 is the single minimum, `node:sqlite` is always available, and the CI matrix now runs Node 22 and 24 (dropping 18 and 20). The defensive runtime guard in the OpenCode adapter and the `describe.skipIf` in its tests stay, harmless, in case someone runs below the stated floor. README (both languages), CONTRIBUTING, AGENTS, and the harness doc drop the per-feature Node caveat.
|
|
8
|
+
- New `evals/` scaffolding makes "every experiment leaves a reproducible artifact" a format and a tool rather than a slogan. Seed cases (dense bilingual text, a dense chart, a clean diagram, a stylized banner, and a dependency-free generated prompt-injection image) live under `evals/cases/`, and `pnpm eval` drives the built CLI over them, writing one evidence artifact per case (command, tool version and commit, provider and model, input SHA-256, raw output, expected points and scoring, latency, usage, errors and degradation) to a git-ignored `evals/results/<date>/`. It reports transcription and schema pass rates and a latency summary, and `--dry-run` validates cases without spending quota. Local and on-demand by design: it spends real quota and never runs in CI.
|
|
9
|
+
- New `modlens doctor` command diagnoses local config and routing without spending a byte of provider quota or making a network request. It reports the Node version against the 22.13 floor, node:sqlite availability, each provider's readiness (agy/claude on PATH, and whether each API key comes from env or the config file, with a copy-paste fix for what is missing), which provider will be selected and from which layer (flag, config, or default), the detected harness and whether the verdict came from process ancestry or an environment fingerprint, and the config file's path and permission bits. Add `--json` for a machine-readable report. Troubleshooting now opens by pointing at it.
|
|
10
|
+
- OpenCode paste recovery works on Windows again (issue #11). Matching a session compared `path.resolve(cwd)`, which is backslash-separated on Windows, against opencode's forward-slash `session.directory`, so the equality and both prefix checks missed every row and recovery returned nothing. The v2.8.0 change that turned `--session` into a directory-narrowing filter meant the old `--session` workaround stopped helping too. Both sides are normalized to forward slashes before matching now, with the LIKE wildcard escaping preserved.
|
|
11
|
+
- A provider that ignores SIGTERM on timeout is now actually killed. The SIGKILL backstop checked `child.killed`, which turns true the moment a signal is delivered, not when the process exits, so a child that trapped SIGTERM read as already dead and was never escalated. It now tracks whether the process has exited and sends SIGKILL when it has not.
|
|
12
|
+
- The published npm package now includes `docs/`, `CHANGELOG.md`, and `SECURITY.md`. The README links to the harness, security, and troubleshooting docs, but the `files` allow-list left them out of the tarball, so those links 404'd for anyone reading the package on npm.
|
|
13
|
+
- CI now runs on macOS as well as Linux across the Node 18/20/22 matrix, so a macOS-only regression (path handling, `mkdtemp`, file modes) is caught before release.
|
|
14
|
+
- `vitest` moves to 3.2.7 to match `@vitest/coverage-v8`, silencing the version-mismatch warning `pnpm coverage` printed when the two drifted.
|
|
15
|
+
- The `node:sqlite` requirement for OpenCode paste recovery is stated correctly: Node **22.13+**, not 22.5. The module was added in 22.5 but behind `--experimental-sqlite`, and only became available without a flag in 22.13. Both READMEs, CONTRIBUTING, the harness and troubleshooting docs, the runtime error message, and the CI comment now agree.
|
|
16
|
+
- Publishing happens in exactly one place now. `scripts/release.mjs` and the tag-triggered `release.yml` workflow both ran `npm publish` and created the GitHub Release, a race that could double-publish or leave a half-finished release. `release.mjs` keeps every guard, the version bump, commit, tag, and push, but stops at the tag: pushing it hands off to CI, which publishes with provenance and cuts the GitHub Release from the matching CHANGELOG section.
|
|
17
|
+
- The subprocess providers (`antigravity-cli`, `claude-cli`) now run in a throwaway directory holding only the input image, not in the image's own directory. `agy` runs with `--dangerously-skip-permissions`, so an injection in an image sitting beside other files could in principle steer the agent into reading them. Each call now hardlinks (or copies) the one image into a fresh `mkdtemp` directory, runs there, and removes it afterward. An explicit `--workdir` keeps the old behaviour.
|
|
18
|
+
- Structural schema validation now runs for every provider, not just `openai`. The shape check that caught half-filled results lived inside the OpenAI-compatible path, so a malformed result from gemini, anthropic, agy, or claude-cli could slip through if the server-side schema was not honoured. The check moved into `schema.ts`, and the analyzer runs it over every provider's result, naming the provider when a field is missing.
|
|
19
|
+
- Remote image downloads are now capped at 25 MB and their type is confirmed from the file header rather than guessed. `fetchRemoteImageBase64` read the whole response into memory with no ceiling (a memory-exhaustion vector) and trusted a server's `content-type`, defaulting to `image/jpeg` for anything else. It now rejects an oversized `content-length` up front, enforces the cap while streaming, sniffs the magic bytes (png/jpeg/gif/webp) as the authority over a faked extension or lying header, and refuses a type outside the allow list instead of relabelling it. Local reads go through the same type check.
|
|
20
|
+
- `recover-paste` no longer writes to a fixed, shared `<tmpdir>/modlens-paste` by default. A `recursive` mkdir does not re-apply its mode to a directory that already exists, so on a shared machine another user could pre-create that path and read every screenshot recovered into it. Each run now mints a private `mkdtemp` directory instead. An explicit `--out-dir` is still honoured, but an existing one is rejected unless it is a real directory (not a symlink), owned by the current user, with no group or world access. Recovered files are deliberately left in place for `modlens -i` to read.
|
|
21
|
+
|
|
22
|
+
- `config show` prints the effective config now, merging environment variables over the file and tagging each value file or env. Reading only the file hid keys set through `GEMINI_API_KEY` and the other bound vars, so the value modlens actually used never appeared.
|
|
23
|
+
- Local image paths containing `#` or `?` keep their real extension. Routing them through `new URL()` read the character as a fragment or query and dropped the extension, mislabelling the type as JPEG.
|
|
24
|
+
- The disclaimer no longer contradicts the MIT license it ships beside. It withholds warranty and endorsement without withholding the commercial-use right MIT grants, and points at the upstream engines' own terms.
|
|
25
|
+
- Docs caught up with the code. Both READMEs gained `--provider-bin`, `--workdir`, a per-provider default-model table, a full `recover-paste` flag table, and the `meta` output fields, with `MODLENS_HARNESS` and `--out-dir` written up in troubleshooting. The anthropic recipe warns that `ANTHROPIC_BASE_URL` can silently reroute a vision request to a text-only gateway. AGENTS.md drops three claims that had gone stale.
|
|
26
|
+
- Internals, all behavior-preserving: the duplicated JSON helpers (parse, extract, truncate) collapsed into one `util/json` module, and the 710-line `recoverPaste` split into per-harness modules. An always-true branch and a few lint findings cleared.
|
|
27
|
+
- Tooling: Biome for formatting and linting on the repo's 4-space style, a Node 18/20/22 CI matrix that skips the `node:sqlite` tests where the module is unavailable, `@vitest/coverage-v8` with a `coverage` script, tests for the CLI assembly, and a tag-triggered release workflow that publishes with provenance. Adds the collaboration files a public repo expects: CONTRIBUTING, SECURITY, CODE_OF_CONDUCT, issue and pull-request templates, and Dependabot.
|
|
28
|
+
- The Chinese README's plug now invites readers to the WeChat public account rather than installing the liustack skills.
|
|
29
|
+
- `release.mjs` matches the CHANGELOG again: version dots are escaped literally and the section ends at end of file, so the newest entry (and versions like `2.8.0`) match instead of being missed.
|
|
30
|
+
|
|
31
|
+
## 2.8.0 - 2026-08-06
|
|
32
|
+
|
|
33
|
+
- README rebuilt against how widely used projects actually write theirs: install command inside the first screen, a nav row and badges in the hero, short scannable highlights, and roughly 1,000 words instead of a long read. Harness specifics and security detail moved into `docs/harness-setup.md` and `docs/security.md`, with a Documentation table pointing at them.
|
|
34
|
+
|
|
35
|
+
## 2.7.11 - 2026-08-06
|
|
36
|
+
|
|
37
|
+
- Stops calling it OCR. A vision model reading an image is not OCR, which is a specific and different technology, and the word was borrowed for convenience across the README, the skill, and both articles. The prose now says what actually happens: every word in the image is transcribed. The `ocr` field in the output contract keeps its familiar name, with a note that a vision model does the reading.
|
|
38
|
+
|
|
39
|
+
## 2.7.10 - 2026-08-06
|
|
40
|
+
|
|
41
|
+
- README rewritten rather than patched again. The hero buried the one thing that sets this apart (you can paste) under a generic pitch, then repeated it in a feature list and again in its own section. The opening now leads with pasting, the feature list is gone as duplication, and the comparison against swapping models or running a vision MCP server sits where a reader weighing options will find it.
|
|
42
|
+
|
|
43
|
+
## 2.7.9 - 2026-08-06
|
|
44
|
+
|
|
45
|
+
- The flow diagram says something again. Replacing ASCII art with an abstract illustration removed the labels along with the alignment chore, which was a bad trade. It is now a rendered diagram with real labels, one per language, generated from HTML so nothing drifts.
|
|
46
|
+
|
|
47
|
+
## 2.7.8 - 2026-08-06
|
|
48
|
+
|
|
49
|
+
- Releases are now one command: `pnpm release <version|patch|minor|major>` refuses a dirty tree, a non-main branch, a duplicate tag, or a version with no CHANGELOG entry, then runs typecheck, tests, and build before anything irreversible happens, and finishes with tag, push, npm publish, and a GitHub release. Publishing by hand is how a version once reached npm with no changelog and no tag behind it.
|
|
50
|
+
- Every previously published version now has a git tag, reconstructed from the commit that carried it.
|
|
51
|
+
|
|
52
|
+
## 2.7.7 - 2026-08-05
|
|
53
|
+
|
|
54
|
+
- README: leads with a scannable feature block (paste support, evidence rather than a description, honest uncertainty, no model swap, zero-key start, four harnesses) and states requirements. Adds a comparison against swapping in a multimodal model and against vision MCP servers, our own weaknesses included.
|
|
55
|
+
- New `docs/troubleshooting.md`: every error this CLI prints, with cause and fix, linked from the README and the skill.
|
|
56
|
+
- The ASCII flow diagram is now a real illustration. Its alignment had needed repair across several releases, which is a poor trade for a picture.
|
|
57
|
+
- The Gemini CLI era research doc is marked historical so it is not read as current design.
|
|
58
|
+
|
|
59
|
+
## 2.7.6 - 2026-08-05
|
|
60
|
+
|
|
61
|
+
- `config init` now writes only the shape (`{"provider": "", "providers": {}}`) instead of all five providers with their fields pre-filled. Baked-in defaults in a config file silently outrank later changes to those defaults, and the placeholders hid the one decision that matters. The command prints what can be set instead.
|
|
62
|
+
|
|
63
|
+
## 2.7.5 - 2026-08-05
|
|
64
|
+
|
|
65
|
+
A verification pass on the 2.7.4 fixes (same external reviewer) found four that did not hold and three bugs the fixes themselves introduced. All seven are addressed here.
|
|
66
|
+
|
|
67
|
+
**Fixes that did not hold**
|
|
68
|
+
|
|
69
|
+
- Unreadable config files still became empty configs: the 2.7.4 edit never applied, because this file is indented differently from its sibling project. Permissions errors now surface.
|
|
70
|
+
- Harness detection still matched a flag's value: `node --require pi app.js` read as Pi. The script behind a node shim must now look like a path to a script.
|
|
71
|
+
- agy log evidence was scoped by file mtime alone, so a concurrent call or an older failure in the same file still misdiagnosed this run. Lines are now filtered by their own glog timestamps.
|
|
72
|
+
- The openai schema check only looked at top-level keys, so `{"ocr":{}}` passed. Nested required fields are checked.
|
|
73
|
+
|
|
74
|
+
**Bugs introduced by the 2.7.4 fixes**
|
|
75
|
+
|
|
76
|
+
- `transcriptBelongsTo` returned on the first recorded cwd, so a transcript whose first line matched could still hand over another project's images. Any matching line now decides, and a transcript with cwd lines that all mismatch is rejected.
|
|
77
|
+
- That check also read every transcript in full, then the image scan read it again. Each file is read once.
|
|
78
|
+
- The alias table added for config lookups was written by hand and did not match the real provider aliases (`claude` resolves to `anthropic`, not `claude-cli`, and `claude-code` and `openai-compat` were missing), so settings landed on the wrong provider. The table now comes from the provider registry.
|
|
79
|
+
- `--transcript` skipped harness validation, so `--harness bogus` silently parsed the file as Claude Code.
|
|
80
|
+
|
|
81
|
+
## 2.7.4 - 2026-08-05
|
|
82
|
+
|
|
83
|
+
Correctness and privacy pass after an external review (gpt-5.6-sol) that proved every finding with a probe.
|
|
84
|
+
|
|
85
|
+
**Recovering the wrong project's images**
|
|
86
|
+
|
|
87
|
+
- OpenCode directory matching passed the project path straight into SQL `LIKE`, where `_` and `%` are wildcards, so a path containing either matched other projects. Patterns are escaped now.
|
|
88
|
+
- `--session <id>` dropped the directory condition entirely, and session slugs are not unique across projects. The reviewer found two colliding slugs in a real local database. A session now narrows the directory match instead of replacing it.
|
|
89
|
+
- Claude Code and Pi directory slugs are lossy: `/tmp/project.alpha` and `/tmp/project-alpha` produce the same slug. Both harnesses record the real cwd inside the transcript, which is now checked before a transcript is trusted.
|
|
90
|
+
|
|
91
|
+
**Privacy**
|
|
92
|
+
|
|
93
|
+
- Recovered images landed as 0644 inside a 0755 directory, so on a shared `/tmp` any local user could read them. They are written 0600 into a 0700 directory, and re-chmodded because the filenames are content hashes and an existing file keeps its old mode.
|
|
94
|
+
|
|
95
|
+
**Correctness**
|
|
96
|
+
|
|
97
|
+
- A successful run could be reported as a timeout: the timer stayed armed while output drained, so a slow drain turned exit code 0 into a timeout error. It is cleared when the child exits.
|
|
98
|
+
- A timeout sent one SIGTERM and then waited, so an engine ignoring signals hung the CLI. It now settles immediately and escalates to SIGKILL.
|
|
99
|
+
- Output decoding kept no state across chunks, so a multi-byte character split across a chunk boundary became replacement characters.
|
|
100
|
+
- The OpenCode "needs Node 22.5" message was swallowed by an empty catch, leaving only "no pasted images". Setup problems now travel with the error.
|
|
101
|
+
- `--harness` was ignored when `--transcript` was given, so a copied Pi transcript was parsed as Claude Code. `--transcript <db>` also ignored `--cwd`.
|
|
102
|
+
- Harness detection scanned the first eight command tokens, so a command that merely mentioned "pi" in its arguments was detected as Pi. Only the executable, plus the script path behind a node shim, is read now.
|
|
103
|
+
- agy log evidence was accepted if the file was under two minutes old, which let a previous quota failure or a concurrent agy call misdiagnose an unrelated error. Evidence must now postdate the start of this run.
|
|
104
|
+
- The `claude-cli` provider inherited a 30 second kill grace meant for agy's own `--print-timeout`, silently extending `--timeout`. The grace applies only to engines with an internal deadline.
|
|
105
|
+
- The openai provider's "schema validation" accepted `{"summary":"x","ocr":null}` and anything missing layout, semantics, visual, or uncertainty. All required fields are checked.
|
|
106
|
+
- Settings saved under a provider alias (`config set gemini.apiKey`) were invisible once the name resolved to `gemini-api`.
|
|
107
|
+
- An unmapped image type was relabelled `.png`, so downstream tools reading the extension got the wrong type.
|
|
108
|
+
- `ENOENT` from spawn was always reported as a missing CLI, even when the real cause was a missing working directory.
|
|
109
|
+
- A config file that exists but cannot be read (permissions) silently became an empty config.
|
|
110
|
+
|
|
111
|
+
## 2.7.3 - 2026-08-05
|
|
112
|
+
|
|
113
|
+
- Fix: a failing `antigravity-cli` run now explains itself instead of reporting a bare exit code ([#3](https://github.com/liustack/modlens/issues/3), thanks @mtongle). Providers gained a `describeFailure` hook, and the agy provider uses it to surface agy's own error text and classify the two failures users actually hit: a locked OS keyring in headless sessions (the report's case, where agy claims to be signed out) and an exhausted weekly quota. Both messages end with the exact commands to switch to a keyless, quota-independent provider. Diagnosis only reads agy's log when this run produced an agy error envelope and the log is fresh, so stale logs cannot misdiagnose an unrelated failure.
|
|
114
|
+
- Docs: README leads with paste support, recommends the free AI Studio key over the slower agy default, and documents that the skill configures modlens for you on request.
|
|
115
|
+
|
|
116
|
+
## 2.7.2 - 2026-08-05
|
|
117
|
+
|
|
118
|
+
- Fix: runs with the `antigravity-cli` provider hung until the timeout killed them ([#1](https://github.com/liustack/modlens/issues/1), thanks @hawkmor for the diagnosis). agy exits cleanly but leaves a language server holding the inherited stdout pipe, so the child's `close` event never fires. The provider run now settles on `exit` plus a short drain window, and releases the pipes afterwards so a lingering descendant cannot keep the CLI process alive either.
|
|
119
|
+
|
|
120
|
+
## 2.7.1 - 2026-08-04
|
|
121
|
+
|
|
122
|
+
- Docs: per-harness skill discovery paths (`~/.claude/skills/`, `~/.codex/skills/`, `~/.agents/skills/` for Pi and OpenCode), OpenCode + DeepSeek setup recipe, and the four-harness live verification matrix (Claude Code session-id recovery, OpenCode full skill loop on DeepSeek, Pi store isolation, Codex refusal).
|
|
123
|
+
|
|
124
|
+
## 2.7.0 - 2026-08-04
|
|
125
|
+
|
|
126
|
+
- `recover-paste` now identifies the harness it is running inside before touching any storage: process ancestry first (the nearest known harness among parent processes, which also resolves nested setups to the innermost tool), env fingerprints second (`CLAUDECODE`, `PI_CODING_AGENT`, `CODEX_THREAD_ID`). Detection scopes recovery to that harness's store only, so another tool's stale sessions can never hijack a paste; Codex is refused outright with path-tag guidance. `--harness <name|none>` overrides, output gains a `detected` field.
|
|
127
|
+
- In Claude Code, recovery auto-targets the exact session from the injected `CLAUDE_CODE_SESSION_ID`, falling back to newest-image scanning when that transcript holds no images (subagent sessions).
|
|
128
|
+
|
|
129
|
+
## 2.6.1 - 2026-08-03
|
|
130
|
+
|
|
131
|
+
- Fix: opencode runs shell commands at the repo root while sessions record the directory they were launched in. Exact directory matching made recovery miss the real paste and fall through to stale Claude Code transcripts of the same project, recovering the wrong images (caught in a live session). Directories now match by prefix in both directions, and recovery is scoped to the single opencode session owning the newest image.
|
|
132
|
+
- Recovered entries report `filename` (the original attachment name) when the harness stored one.
|
|
133
|
+
- Skill: recovered paths are oldest to newest so analyze the last one first, match `filename` when present, run every command yourself instead of delegating to the user, and treat a `harness` value that differs from the harness you are running in as suspect.
|
|
134
|
+
|
|
135
|
+
## 2.6.0 - 2026-08-03
|
|
136
|
+
|
|
137
|
+
- `recover-paste` now supports OpenCode: pasted/attached images are read from its SQLite store (`~/.local/share/opencode/opencode.db`) via node:sqlite (Node 22.5+, lazy-loaded so older Nodes keep the JSONL harnesses). Recovery internals refactored into per-harness adapters (Claude Code, Pi, OpenCode) sharing one newest-image picker. Verified against a real opencode + deepseek session.
|
|
138
|
+
|
|
139
|
+
## 2.5.0 - 2026-08-03
|
|
140
|
+
|
|
141
|
+
- `recover-paste` now supports Pi (Armin Ronacher's coding agent) alongside Claude Code: both store pasted images as base64 in per-session JSONL files, and recovery auto-detects which harness owns the newest pasted image. Verified live against a real pi + deepseek session. Result JSON gains a `harness` field.
|
|
142
|
+
|
|
143
|
+
## 2.4.3 - 2026-08-03
|
|
144
|
+
|
|
145
|
+
- Docs: the Claude Code paste-recovery loop is now marked as verified end to end in a real DeepSeek-gateway session (placeholder spotted, file recovered by session id, image answered in full).
|
|
146
|
+
|
|
147
|
+
## 2.4.2 - 2026-08-03
|
|
148
|
+
|
|
149
|
+
- Project hygiene: CHANGELOG, GitHub Actions CI, AGENTS.md rewrite, testing guide rewrite, recover-paste and config command reference in READMEs, dead code removal, auto-externalized Node built-ins in the build.
|
|
150
|
+
|
|
151
|
+
## 2.4.1 - 2026-08-03
|
|
152
|
+
|
|
153
|
+
- Skill: path-finding is now a per-harness decision tree. Codex path tags never trigger transcript recovery, unknown harnesses are told to ask for a path instead of guessing.
|
|
154
|
+
|
|
155
|
+
## 2.4.0 - 2026-08-03
|
|
156
|
+
|
|
157
|
+
- `recover-paste --session <id>`: exact transcript targeting. Skills relay `${CLAUDE_SESSION_ID}` (substituted by Claude Code since v2.1.9); without it, recovery falls back to newest-image-timestamp scanning.
|
|
158
|
+
|
|
159
|
+
## 2.3.2 - 2026-08-03
|
|
160
|
+
|
|
161
|
+
- Tests co-located with sources, one module one `.test.ts` (31 to 50 tests). First direct coverage for `prompt` and `imageInput`.
|
|
162
|
+
- Skill explains why `recover-paste` takes no session id.
|
|
163
|
+
|
|
164
|
+
## 2.3.1 - 2026-08-03
|
|
165
|
+
|
|
166
|
+
- `recover-paste` locates the session by newest pasted-image timestamp instead of file mtime, immune to concurrent sessions in the same project.
|
|
167
|
+
|
|
168
|
+
## 2.3.0 - 2026-08-03
|
|
169
|
+
|
|
170
|
+
- New `recover-paste` command: recovers images pasted into Claude Code from the local session transcript (they never hit a regular temp file), prints real file paths as JSON.
|
|
171
|
+
|
|
172
|
+
## 2.2.0 - 2026-08-03
|
|
173
|
+
|
|
174
|
+
- New `claude-cli` provider: rides an existing Claude Code login, `--allowedTools Read` only, `--json-schema` enforced, haiku default.
|
|
175
|
+
- Skill routes configuration questions to `references/configure.md`.
|
|
176
|
+
|
|
177
|
+
## 2.1.0 - 2026-08-02
|
|
178
|
+
|
|
179
|
+
- Three direct-API providers: `gemini-api` (free AI Studio key, `responseJsonSchema`), `openai` (any OpenAI-compatible multimodal endpoint), `anthropic` (forced tool call, Claude Haiku default). 3-10s per image versus 15-40s agent loops.
|
|
180
|
+
- Layered config: `~/.modlens/config.json` via `config init/set/show` (0600, masked), env vars override the file, flags override everything.
|
|
181
|
+
|
|
182
|
+
## 2.0.0 - 2026-08-01
|
|
183
|
+
|
|
184
|
+
- Breaking: vision engine migrated from the discontinued Gemini CLI free tier to Antigravity CLI (`agy`).
|
|
185
|
+
- Provider layer (`buildInvocation` + `parseOutput`), schema-enforced structured output via `--json-schema`, no markdown scraping.
|
|
186
|
+
- Output contract v2: `result`/`meta` envelope; fabricated bbox and confidence fields dropped.
|
package/README.md
CHANGED
|
@@ -1,223 +1,214 @@
|
|
|
1
|
-
<
|
|
2
|
-
<img src="https://raw.githubusercontent.com/liustack/modlens/main/assets/banner.jpg" width="100%" alt="ModLens
|
|
3
|
-
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
1
|
+
<p align="center">
|
|
2
|
+
<img src="https://raw.githubusercontent.com/liustack/modlens/main/assets/banner.jpg" width="100%" alt="ModLens" />
|
|
3
|
+
</p>
|
|
4
|
+
|
|
5
|
+
<h1 align="center">ModLens</h1>
|
|
6
|
+
|
|
7
|
+
<p align="center"><b>Give a text-only model sight, and just paste the image.</b></p>
|
|
8
|
+
|
|
9
|
+
<p align="center">
|
|
10
|
+
<a href="./README.zh-CN.md">简体中文</a> ·
|
|
11
|
+
<a href="docs/troubleshooting.md">Troubleshooting</a> ·
|
|
12
|
+
<a href="skills/modlens/references/configure.md">Configuration</a> ·
|
|
13
|
+
<a href="skills/modlens/references/output-schema.md">Output contract</a> ·
|
|
14
|
+
<a href="docs/security.md">Security</a> ·
|
|
15
|
+
<a href="https://github.com/liustack/modsearch">ModSearch (web)</a>
|
|
16
|
+
</p>
|
|
17
|
+
|
|
18
|
+
<p align="center">
|
|
19
|
+
<a href="https://www.npmjs.com/package/@liustack/modlens"><img src="https://img.shields.io/npm/v/@liustack/modlens?style=flat-square&label=npm&color=cb3837" alt="npm"></a>
|
|
20
|
+
<a href="https://github.com/liustack/modlens/actions/workflows/ci.yml"><img src="https://img.shields.io/github/actions/workflow/status/liustack/modlens/ci.yml?branch=main&style=flat-square&label=ci" alt="CI"></a>
|
|
21
|
+
<a href="https://nodejs.org"><img src="https://img.shields.io/node/v/@liustack/modlens?style=flat-square" alt="Node.js"></a>
|
|
22
|
+
<a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-blue?style=flat-square" alt="License"></a>
|
|
23
|
+
</p>
|
|
11
24
|
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
25
|
+
```bash
|
|
26
|
+
npx -y skills add liustack/modlens # install the skill
|
|
27
|
+
npx @liustack/modlens -i screenshot.png # or just use the CLI
|
|
28
|
+
```
|
|
15
29
|
|
|
16
|
-
|
|
30
|
+
Models like DeepSeek-V4-Flash are cheap, fast, capable, and blind. Throw one a screenshot of an error and it sees nothing. ModLens hands the image to a real vision engine and brings back evidence your model can quote: every word transcribed, the layout mapped, the doubts declared. And **you just paste**: other bridges make you save a file and report its path, while ModLens pulls the pasted image straight back out of session storage.
|
|
17
31
|
|
|
18
|
-
|
|
32
|
+
## Highlights
|
|
19
33
|
|
|
20
|
-
|
|
34
|
+
- **Pasting works.** A pasted image never becomes a file, which is why other vision bridges cannot see it. ModLens recovers it from the harness's local session storage instead.
|
|
35
|
+
- **Evidence, not an impression.** Every word transcribed, layout cut into regions in reading order, entities and relations listed. Your model quotes specifics instead of trusting a vibe.
|
|
36
|
+
- **It says when it cannot read something.** Uncertain parts land in `uncertainty`. Pixel coordinates and confidence scores, the two things vision models fabricate most, were deliberately dropped.
|
|
37
|
+
- **Keep your model.** You picked it for price and reasoning, not eyesight. That choice stays.
|
|
38
|
+
- **Starts with no key.** The default engine (Antigravity CLI) needs none. A free Gemini key cuts a read to 5-10 seconds.
|
|
39
|
+
- **Install once, works everywhere.** Verified on real machines in Claude Code, Codex, Pi, and OpenCode.
|
|
21
40
|
|
|
22
|
-
|
|
41
|
+
## Installation
|
|
23
42
|
|
|
24
|
-
```
|
|
25
|
-
|
|
43
|
+
```bash
|
|
44
|
+
npx -y skills add liustack/modlens
|
|
26
45
|
```
|
|
27
46
|
|
|
28
|
-
Or
|
|
47
|
+
Or tell your agent: "Install the skill from https://github.com/liustack/modlens".
|
|
29
48
|
|
|
30
|
-
|
|
49
|
+
Then give it a vision engine. A free **[AI Studio](https://aistudio.google.com) Gemini key** is the fast answer (three minutes, no credit card, 5-10 seconds per image):
|
|
31
50
|
|
|
32
51
|
```bash
|
|
33
52
|
modlens config set gemini-api.apiKey <key>
|
|
34
53
|
modlens config set provider gemini-api
|
|
35
54
|
```
|
|
36
55
|
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
**3. Use it.** Paste an image, or throw a path at it, and ask anything. The skill fires on its own.
|
|
40
|
-
|
|
41
|
-
Requirements, in one line: Node 18+ (22.5+ for OpenCode paste recovery), macOS or Linux.
|
|
56
|
+
Skipping the sign-up is fine: **Antigravity CLI** works with no key, it is just slower (15-40 seconds) with a tight free quota:
|
|
42
57
|
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
ModLens takes the other route. Before those bytes are sent anywhere, the harness has already written them to local session storage. The skill goes there, pulls them back into a real file, and hands that to the vision engine. You do nothing, and the model gets the whole image instead of asking you for a path.
|
|
58
|
+
```bash
|
|
59
|
+
curl -fsSL https://antigravity.google/cli/install.sh | bash && agy # sign in, then exit
|
|
60
|
+
```
|
|
48
61
|
|
|
49
|
-
|
|
62
|
+
Requires Node 22.13+, macOS or Linux.
|
|
50
63
|
|
|
51
|
-
|
|
52
|
-
| :-- | :-- | :-- | :-- |
|
|
53
|
-
| Your chosen model | has to change | stays | stays |
|
|
54
|
-
| An image pasted into the chat | visible if the model supports it | out of reach, save a file and report the path | handled directly |
|
|
55
|
-
| What you get back | the model's own reading | usually a description | full transcription, layout regions, entities and relations, visual clues |
|
|
56
|
-
| Where it cannot read | may invent | may invent | says so in `uncertainty` |
|
|
57
|
-
| Cost | multimodal model pricing | usually per API call | agy's free quota, or a free Gemini key |
|
|
58
|
-
| Setup | change config, change model | install a server, edit config | one CLI or one skill |
|
|
64
|
+
## Usage
|
|
59
65
|
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
## See it work
|
|
66
|
+
With the skill installed you do not type commands: paste an image or drop a path, ask anything, and it fires on its own. By hand:
|
|
63
67
|
|
|
64
68
|
```bash
|
|
65
|
-
|
|
69
|
+
modlens -i screenshot.png # local image
|
|
70
|
+
modlens -i https://example.com/chart.png # remote image
|
|
71
|
+
modlens -i chart.png --prompt "focus on axes" # extra focus
|
|
72
|
+
modlens recover-paste # pull a pasted image into a file
|
|
66
73
|
```
|
|
67
74
|
|
|
68
|
-
|
|
75
|
+
Output is a fixed JSON shape:
|
|
69
76
|
|
|
70
77
|
```json
|
|
71
78
|
{
|
|
72
|
-
"image": "/
|
|
73
|
-
"provider": "
|
|
79
|
+
"image": "/path/to/screenshot.png",
|
|
80
|
+
"provider": "gemini-api",
|
|
74
81
|
"result": {
|
|
75
82
|
"summary": "A workflow diagram with four nodes connected by labeled arrows.",
|
|
76
|
-
"ocr": {
|
|
77
|
-
|
|
78
|
-
"lines": [
|
|
79
|
-
{ "language": "en", "text": "/shaping" },
|
|
80
|
-
{ "language": "en", "text": "BEFORE YOU BUILD" }
|
|
81
|
-
]
|
|
82
|
-
},
|
|
83
|
-
"layout": {
|
|
84
|
-
"regions": [
|
|
85
|
-
{
|
|
86
|
-
"reading_order": 1,
|
|
87
|
-
"text": "/shaping BEFORE YOU BUILD",
|
|
88
|
-
"type": "other"
|
|
89
|
-
}
|
|
90
|
-
]
|
|
91
|
-
},
|
|
83
|
+
"ocr": { "full_text": "/shaping\nBEFORE YOU BUILD\n...", "lines": [] },
|
|
84
|
+
"layout": { "regions": [{ "reading_order": 1, "type": "title", "text": "/shaping" }] },
|
|
92
85
|
"uncertainty": []
|
|
93
86
|
},
|
|
94
|
-
"meta": {
|
|
87
|
+
"meta": {
|
|
88
|
+
"generatedAt": "2026-08-06T12:00:00.000Z",
|
|
89
|
+
"model": "gemini-3.6-flash",
|
|
90
|
+
"conversationId": null,
|
|
91
|
+
"durationSeconds": 6.4,
|
|
92
|
+
"usage": { "promptTokenCount": 1234, "candidatesTokenCount": 567 }
|
|
93
|
+
}
|
|
95
94
|
}
|
|
96
95
|
```
|
|
97
96
|
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-

|
|
101
|
-
|
|
102
|
-
Batch mode works too: drop three illustrations at once, and the model announces it will read them one by one through ModLens, then delivers all three descriptions in 21 seconds, design intent included.
|
|
103
|
-
|
|
104
|
-

|
|
105
|
-
|
|
106
|
-
Stress test: a scatter plot of 128 models. ModLens pulls out the axes, the log scale, and the highlighted DeepSeek V4 Flash point at $0.028 and score 50, then walks through the cost-performance cutoff line. Dense charts are where vision models usually fold. This one holds.
|
|
97
|
+
`meta` records how the result was produced: when (`generatedAt`), which `model`, the provider's `conversationId` when it has one, wall-clock `durationSeconds`, and the raw `usage` the provider reported (shape varies by provider, `null` when none).
|
|
107
98
|
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
## CLI reference
|
|
99
|
+
## See it work
|
|
111
100
|
|
|
112
|
-
|
|
113
|
-
modlens -i <image-path-or-url> [options]
|
|
114
|
-
```
|
|
101
|
+
Unedited runs, all driving a text-only DeepSeek-V4-Flash.
|
|
115
102
|
|
|
116
|
-
|
|
117
|
-
| :-- | :-- | :-- |
|
|
118
|
-
| `-i, --input <path\|url>` | Image to analyze (required) | |
|
|
119
|
-
| `-o, --output <path>` | Also write JSON to a file | |
|
|
120
|
-
| `-m, --model <name>` | Provider model | `gemini-3.6-flash-low` |
|
|
121
|
-
| `-p, --provider <name>` | Vision provider | `antigravity-cli` |
|
|
122
|
-
| `--prompt <text>` | Extra focus, e.g. `"extract the table"` | |
|
|
123
|
-
| `--timeout <ms>` | Provider timeout | `180000` |
|
|
124
|
-
| `--provider-bin <path>` | Provider binary | `agy` |
|
|
125
|
-
| `--workdir <path>` | Working directory for the provider | |
|
|
103
|
+
A tweet screenshot in the Codex desktop app. It reads the caption, the engagement numbers (2.9K replies, 270K likes, 5M views), even the image's alt text. Where the resolution runs out, it says so instead of guessing.
|
|
126
104
|
|
|
127
|
-
|
|
105
|
+

|
|
128
106
|
|
|
129
|
-
|
|
107
|
+
Three images pasted at once. The model queues them up and reads them one by one, design intent included.
|
|
130
108
|
|
|
131
|
-
|
|
132
|
-
modlens recover-paste # newest pasted image, path printed as JSON
|
|
133
|
-
modlens recover-paste --count 3 # the three newest
|
|
134
|
-
modlens recover-paste --session <id> # exact session (skills pass ${CLAUDE_SESSION_ID})
|
|
135
|
-
modlens recover-paste --harness pi # force one harness's format
|
|
136
|
-
# --transcript <path> overrides everything; --cwd <dir> sets the project directory
|
|
137
|
-
```
|
|
109
|
+

|
|
138
110
|
|
|
139
|
-
|
|
111
|
+
The stress test: a scatter plot of 128 models. It identifies the chart, both axes, the log scale, and picks the one highlighted point out of the crowd with its coordinates (about $0.028, intelligence score 50). Dense charts are where vision models usually fold.
|
|
140
112
|
|
|
141
|
-
|
|
113
|
+

|
|
142
114
|
|
|
143
|
-
|
|
115
|
+
And the paste path, end to end: Claude Code on a DeepSeek gateway, two images pasted straight into the chat. The UI shows nothing but placeholders, the skill recovers both from session storage and reads them, down to the color values on a slide cover.
|
|
144
116
|
|
|
145
|
-
|
|
146
|
-
| :-- | :-- | :-- | :-- |
|
|
147
|
-
| `antigravity-cli` (default) | `agy` signed in | 15-40s | free quota, full agent loop, quota is tight (see below) |
|
|
148
|
-
| `gemini-api` (recommended) | free AI Studio key | 5-10s | fastest free route, schema enforced server-side |
|
|
149
|
-
| `openai` | baseUrl + apiKey + model | endpoint-dependent | any OpenAI-compatible multimodal endpoint (qwen-vl, GLM, ...) |
|
|
150
|
-
| `anthropic` | `ANTHROPIC_API_KEY` | a few seconds | Claude Haiku by default, schema via forced tool call |
|
|
151
|
-
| `claude-cli` | Claude Code signed in | 20-45s | no key, rides your Claude subscription, Read-only permissions |
|
|
117
|
+

|
|
152
118
|
|
|
153
|
-
|
|
119
|
+
## How it works
|
|
154
120
|
|
|
155
|
-
|
|
121
|
+

|
|
156
122
|
|
|
157
|
-
|
|
158
|
-
modlens config init # write a starter config
|
|
159
|
-
modlens config set gemini-api.apiKey <key> # saved with 0600 perms
|
|
160
|
-
modlens config show # keys come out masked
|
|
161
|
-
modlens config set provider gemini-api # switch the default provider
|
|
162
|
-
```
|
|
123
|
+
No magic, four steps:
|
|
163
124
|
|
|
164
|
-
The
|
|
125
|
+
1. The skill triggers when an image shows up: a path, a URL, or the bare placeholder a text-only model gets left with after a paste.
|
|
126
|
+
2. It runs the `modlens` CLI, which hands the image to a vision engine. Five to choose from, the free Antigravity CLI by default.
|
|
127
|
+
3. The engine's reading is forced into a fixed JSON schema: transcription, layout, semantics, uncertainty. Output that does not match the schema is rejected, never patched up.
|
|
128
|
+
4. Your model quotes the evidence and answers.
|
|
165
129
|
|
|
166
|
-
|
|
130
|
+
The paste trick is the part nobody else does. A pasted image is handled inside the client: encoded and sent the moment it lands, gone before any outside tool can touch it, which is why other bridges tell you to save a file and report the path. But before those bytes leave, the harness has already written them into its local session record. `recover-paste` reads them back from there: JSONL in Claude Code and Pi, SQLite in OpenCode, and Codex needs no recovery at all because its pastes already land as temp files. Details in [harness setup](docs/harness-setup.md).
|
|
167
131
|
|
|
168
|
-
|
|
132
|
+
| | Swap in a multimodal model | Other vision bridges (MCP servers etc.) | ModLens |
|
|
133
|
+
| :-- | :-- | :-- | :-- |
|
|
134
|
+
| Your chosen model | has to change | stays | stays |
|
|
135
|
+
| An image pasted into the chat | visible if the model supports it | out of reach | recovered and read |
|
|
136
|
+
| What you get back | the model's own reading | usually a description | transcription, layout regions, entities |
|
|
137
|
+
| Where it cannot read | may invent | may invent | says so in `uncertainty` |
|
|
138
|
+
| Cost | multimodal model pricing | usually per API call | agy's free quota or a free Gemini key |
|
|
169
139
|
|
|
170
|
-
|
|
140
|
+
The weaknesses, in the same place: agy's free tier is a weekly quota and heavy use hits the wall (a free Gemini key sidesteps it). Session storage layouts are each harness's internals with no compatibility promise, so if recovery ever breaks, dragging the file in still works everywhere.
|
|
171
141
|
|
|
172
|
-
|
|
142
|
+
## CLI reference
|
|
173
143
|
|
|
174
|
-
|
|
175
|
-
- Attach it with `codex exec -i image.png "..."`. The skill reads the path out of the message tag.
|
|
144
|
+
`modlens analyze` (the default command):
|
|
176
145
|
|
|
177
|
-
|
|
146
|
+
| Flag | Meaning | Default |
|
|
147
|
+
| :-- | :-- | :-- |
|
|
148
|
+
| `-i, --input <path\|url>` | Image to analyze (required) | |
|
|
149
|
+
| `-p, --provider <name>` | Vision provider | `antigravity-cli` |
|
|
150
|
+
| `-m, --model <name>` | Provider model | per provider (below) |
|
|
151
|
+
| `-o, --output <path>` | Also write JSON to a file | |
|
|
152
|
+
| `--prompt <text>` | Extra focus | |
|
|
153
|
+
| `--timeout <ms>` | Provider timeout | `180000` |
|
|
154
|
+
| `--provider-bin <path>` | Provider binary path | `agy` / `claude` |
|
|
155
|
+
| `--workdir <path>` | Working directory for the provider | image's directory |
|
|
178
156
|
|
|
179
|
-
|
|
157
|
+
The default `-m` model depends on the provider:
|
|
180
158
|
|
|
181
|
-
|
|
159
|
+
| Provider | Default model |
|
|
160
|
+
| :-- | :-- |
|
|
161
|
+
| `antigravity-cli` (default) | `gemini-3.6-flash-low` |
|
|
162
|
+
| `gemini-api` | `gemini-3.6-flash` |
|
|
163
|
+
| `anthropic` | `claude-haiku-4-5-20251001` |
|
|
164
|
+
| `claude-cli` | `haiku` |
|
|
165
|
+
| `openai` | none, `-m` is required |
|
|
182
166
|
|
|
183
|
-
|
|
167
|
+
`modlens recover-paste`:
|
|
184
168
|
|
|
185
|
-
|
|
169
|
+
| Flag | Meaning | Default |
|
|
170
|
+
| :-- | :-- | :-- |
|
|
171
|
+
| `--count <n>` | How many recent pasted images to recover | `1` |
|
|
172
|
+
| `--out-dir <path>` | Where to write recovered images | a fresh private `<tmpdir>/modlens-paste-*` per run |
|
|
173
|
+
| `--session <id>` | Session id for exact targeting | auto-detect |
|
|
174
|
+
| `--transcript <path>` | Explicit transcript `.jsonl` or `.db` (overrides `--session`) | |
|
|
175
|
+
| `--harness <name>` | Force storage scope: `claude-code`, `pi`, `opencode`, `none` | auto-detect |
|
|
176
|
+
| `--cwd <path>` | Project directory the image was pasted in | current directory |
|
|
186
177
|
|
|
187
|
-
`
|
|
178
|
+
Five providers: `antigravity-cli` (default, no key), `gemini-api` (fastest free route), `openai` (any OpenAI-compatible multimodal endpoint), `anthropic`, and `claude-cli` (rides your Claude subscription). Two more subcommands: `modlens config <init|set|show>`, and `modlens doctor` (checks Node, provider readiness, which provider will be selected and why, and the detected harness, without spending quota or touching the network. Add `--json` for a machine-readable report).
|
|
188
179
|
|
|
189
|
-
|
|
180
|
+
## Documentation
|
|
190
181
|
|
|
191
|
-
|
|
182
|
+
| Doc | Read it when |
|
|
183
|
+
| :-- | :-- |
|
|
184
|
+
| [Troubleshooting](docs/troubleshooting.md) | A command failed and the message needs decoding |
|
|
185
|
+
| [Configuration](skills/modlens/references/configure.md) | Setting a key, switching providers, fixing config |
|
|
186
|
+
| [Output contract](skills/modlens/references/output-schema.md) | Parsing the JSON or building on it |
|
|
187
|
+
| [Harness setup](docs/harness-setup.md) | Wiring it into Codex, Claude Code, Pi, or OpenCode |
|
|
188
|
+
| [Security](docs/security.md) | File permissions, image content as untrusted input |
|
|
189
|
+
| [CHANGELOG](CHANGELOG.md) | Finding what changed in a version |
|
|
190
|
+
| [AGENTS.md](AGENTS.md) | Working on this codebase |
|
|
192
191
|
|
|
193
|
-
##
|
|
192
|
+
## Contributing
|
|
194
193
|
|
|
195
|
-
|
|
196
|
-
- **Evidence beats pixels.** Text models reason best over structured text, not raw pixels. ModLens hands them the transcribed words, the segmented layout, and the extracted meaning, not a base64 blob.
|
|
197
|
-
- **Engines die, the bridge survives.** v1 ran on Gemini CLI's free tier until Google shut it down in June 2026. v2 moved to its successor, Antigravity CLI, behind the same provider interface, so the next engine swap costs one file, not a rewrite.
|
|
194
|
+
ModLens does not accept pull requests. It is a small tool with one pair of hands on it, and every line stays author-owned: that tight loop is what keeps it dependable. Two ways to contribute that genuinely help:
|
|
198
195
|
|
|
199
|
-
|
|
196
|
+
- **[Open an issue](https://github.com/liustack/modlens/issues).** Bugs, ideas, a confusing error, docs that read wrong. Issues get read and drive what gets built.
|
|
197
|
+
- **Fork it.** MIT means your copy is fully yours: rename it, rewire it, ship it.
|
|
200
198
|
|
|
201
199
|
## Shameless plug
|
|
202
200
|
|
|
203
|
-
This project runs on LIUSTACK Skills
|
|
204
|
-
|
|
205
|
-
**ModLens gives your model eyes. LIUSTACK Skills gives your dev workflow wings:**
|
|
201
|
+
This project runs on LIUSTACK Skills: `shaping` before you build, `coding` while you build, `dig` when it breaks, `snapshot` when you hand off. Lighter than Superpowers, and stronger.
|
|
206
202
|
|
|
207
203
|
```bash
|
|
208
204
|
npx -y skills add liustack/liustack -g
|
|
209
205
|
```
|
|
210
206
|
|
|
211
|
-
⭐
|
|
212
|
-
|
|
213
|
-
## Security notes
|
|
214
|
-
|
|
215
|
-
- ModLens runs `agy` with `--dangerously-skip-permissions`, because print mode can fail in some setups without it. The prompt keeps the agent to reading that one image and tells it to treat image content as data, never as instructions. Even so, only point it at images you would open yourself, and run it inside a sandboxed workspace when you can.
|
|
216
|
-
- Vision output is evidence. Anything the engine cannot read lands in `uncertainty` instead of getting invented. Pixel bounding boxes and confidence scores were dropped in v2 because models fabricate them.
|
|
207
|
+
⭐ If it helps, star [ModLens](https://github.com/liustack/modlens) and [liustack](https://github.com/liustack/liustack). Stars are how the next developer finds them.
|
|
217
208
|
|
|
218
209
|
## Disclaimer
|
|
219
210
|
|
|
220
|
-
|
|
211
|
+
Provided as-is under the MIT License below. The author makes no warranty and gives no endorsement for any particular use, commercial use included. Your use of upstream engines (Antigravity CLI, the Gemini, OpenAI, and Anthropic APIs, and any OpenAI-compatible endpoint) is governed by their own terms and quotas, which you are responsible for.
|
|
221
212
|
|
|
222
213
|
## License
|
|
223
214
|
|