@saifymatteo/pi-vision 0.0.0-stage → 0.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +661 -0
- package/NOTICE.md +5 -0
- package/README.md +42 -2
- package/examples/vision-toolkit.example.json +12 -0
- package/package.json +36 -4
- package/skills/vision-tools/LICENSE +21 -0
- package/skills/vision-tools/SKILL.md +134 -0
- package/skills/vision-tools/agents/openai.yaml +4 -0
- package/skills/vision-tools/bin/crop +94 -0
- package/skills/vision-tools/bin/detect +18 -0
- package/skills/vision-tools/bin/glance +98 -0
- package/skills/vision-tools/bin/ground +18 -0
- package/skills/vision-tools/bin/trace +134 -0
- package/skills/vision-tools/bin/vision-doctor +168 -0
- package/skills/vision-tools/detect.py +56 -0
- package/skills/vision-tools/ground.py +216 -0
- package/skills/vision-tools/references/gui.md +93 -0
- package/skills/vision-tools/references/long-screenshot-ocr.md +103 -0
- package/skills/vision-tools/references/restore-graphic.md +92 -0
- package/skills/vision-tools/references/restore-structure.md +47 -0
- package/skills/vision-tools/references/restore-ui.md +200 -0
- package/skills/vision-tools/scripts/dominant_colors.py +224 -0
- package/skills/vision-tools/scripts/extract_fg.py +278 -0
- package/skills/vision-tools/scripts/html_shot.py +111 -0
- package/skills/vision-tools/scripts/long_screenshot_ocr.py +1245 -0
- package/skills/vision-tools/scripts/pixel_diff.py +88 -0
- package/skills/vision-tools/vision_client.py +581 -0
- package/src/index.ts +728 -0
package/README.md
CHANGED
|
@@ -1,3 +1,43 @@
|
|
|
1
|
-
#
|
|
1
|
+
# @saifymatteo/pi-vision
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
## Purpose and registrations
|
|
4
|
+
|
|
5
|
+
Registers a `context` hook, not a tool or slash command. Preserves native images for image-capable session models by default. For text-only/unknown models it replaces image blocks with model-generated descriptions. Optional `describe` / `describe-old` policies self-describe all/older images using the current model on OpenAI Chat Completions connections.
|
|
6
|
+
|
|
7
|
+
## Configuration and services
|
|
8
|
+
|
|
9
|
+
Reads `<agent-dir>/extensions/vision-toolkit.json`: `model`, optional `fallbacks`, `imagePolicy` (native/describe/describe-old), `lang` (en/zh), and `providers` overrides. Example points at example.invalid and must be reviewed manually; it is never loaded automatically. Provider override keys support `$ENV_NAME` references (not brace syntax).
|
|
10
|
+
|
|
11
|
+
Resolution also consults agent `models.json`, `models-store.json`, `auth.json`, and optional `haoyiyin/9router.json`. None of those real files are bundled. An explicit provider/model ID and override can resolve without a static catalog entry. Requires an image-capable OpenAI Chat Completions service; no CLI. Self-description requires current-model credentials in auth.json.
|
|
12
|
+
|
|
13
|
+
## Bundled skill
|
|
14
|
+
|
|
15
|
+
The package bundles the `vision-tools` skill (`skills/vision-tools`): `bin/` launcher CLIs (`crop`, `detect`, `glance`, `ground`, `trace`, `vision-doctor`), Python tools under `scripts/`, reference guides, and `SKILL.md`. The extension's guidance messages point the agent at `vision-doctor` and the skill. Requires Python 3 with Pillow (PIL); the CLIs are Python scripts, not compiled binaries. Based on [Anionex/agent-vision-toolkit](https://github.com/Anionex/agent-vision-toolkit) (MIT), modified in this repository — see [NOTICE.md](NOTICE.md).
|
|
16
|
+
|
|
17
|
+
## Host limitations and security
|
|
18
|
+
|
|
19
|
+
The hook expects host-cloned request messages; rewriting should not mutate stored sessions. No config files are created. Missing/broken config leaves native-capable models native, but text-only models receive explicit failure markers rather than images. Image pixels and focus hints from surrounding user/assistant messages are transmitted to the model service; descriptions enter main-model context. Do not pass sensitive screenshots. Up to four concurrent requests, in-process cache, 180s per-request timeout and transient retries; configured HTTP failures can fail over. This is a lossy bridge, not an image security boundary.
|
|
20
|
+
|
|
21
|
+
Inherited caveats: cache keys include model ID but not provider/base URL; changing providers with the same ID can reuse cached descriptions until reload. The returned chain language is not propagated to individual candidates by the source. These behaviors were preserved, not repaired in this packaging work.
|
|
22
|
+
|
|
23
|
+
## Tests
|
|
24
|
+
|
|
25
|
+
From the repository root with its existing test dependencies available:
|
|
26
|
+
|
|
27
|
+
```sh
|
|
28
|
+
node --test packages/pi-vision/test/*.test.mjs
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
Tests use the shared test-only TypeScript/VM loader, fixtures, and host/service stubs. They do not prove compatibility with a live host, CLI, or model service.
|
|
32
|
+
|
|
33
|
+
## Running locally
|
|
34
|
+
|
|
35
|
+
From the repository root:
|
|
36
|
+
|
|
37
|
+
```sh
|
|
38
|
+
pi -e ./packages/pi-vision
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
Node >=24; host imports are peers, not bundled dependencies. Load only one copy: for an isolated session use `--no-extensions` with explicit `-e` paths so another installed copy is not also discovered. Set `PI_CODING_AGENT_DIR` to a disposable agent directory before launch; unset means the normal `~/.pi/agent` fallback. Extension config continues to resolve there, never relative to this package.
|
|
42
|
+
|
|
43
|
+
Examples are manually authored, sanitized reference documents, not runtime defaults. Review and manually create only the needed files in the isolated agent directory; do not copy live credentials or endpoints. No example is automatically loaded or copied. Loading an extension grants Pi-process OS permissions. Licensed AGPL-3.0-only; see [NOTICE.md](NOTICE.md).
|
package/package.json
CHANGED
|
@@ -1,6 +1,38 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@saifymatteo/pi-vision",
|
|
3
|
-
"version": "0.
|
|
4
|
-
"
|
|
5
|
-
"
|
|
6
|
-
|
|
3
|
+
"version": "0.1.1",
|
|
4
|
+
"license": "AGPL-3.0-only",
|
|
5
|
+
"type": "module",
|
|
6
|
+
"keywords": [
|
|
7
|
+
"pi-package"
|
|
8
|
+
],
|
|
9
|
+
"files": [
|
|
10
|
+
"src",
|
|
11
|
+
"examples",
|
|
12
|
+
"README.md",
|
|
13
|
+
"NOTICE.md",
|
|
14
|
+
"LICENSE",
|
|
15
|
+
"skills"
|
|
16
|
+
],
|
|
17
|
+
"pi": {
|
|
18
|
+
"extensions": [
|
|
19
|
+
"./src/index.ts"
|
|
20
|
+
],
|
|
21
|
+
"skills": [
|
|
22
|
+
"./skills"
|
|
23
|
+
]
|
|
24
|
+
},
|
|
25
|
+
"peerDependencies": {},
|
|
26
|
+
"engines": {
|
|
27
|
+
"node": ">=24"
|
|
28
|
+
},
|
|
29
|
+
"description": "Describe images for text-only models via a configured vision chain.",
|
|
30
|
+
"publishConfig": {
|
|
31
|
+
"access": "public"
|
|
32
|
+
},
|
|
33
|
+
"repository": {
|
|
34
|
+
"type": "git",
|
|
35
|
+
"url": "git+https://github.com/saifymatteo/pi-extensions.git",
|
|
36
|
+
"directory": "packages/pi-vision"
|
|
37
|
+
}
|
|
38
|
+
}
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Anionex
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,134 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: vision-tools
|
|
3
|
+
description: >-
|
|
4
|
+
Native-first workflows for screenshot reconstruction, diagram extraction,
|
|
5
|
+
screenshot-driven GUI work, long-screenshot OCR, and image processing
|
|
6
|
+
(crops, pixel differences, palette measurements, SVG tracing, foreground
|
|
7
|
+
extraction). Use when a task needs these workflows or concrete image
|
|
8
|
+
artifacts; ordinary image questions can be answered with native vision.
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
# Vision tools
|
|
12
|
+
|
|
13
|
+
Use the main model's native vision first. The tools here supplement it with
|
|
14
|
+
image artifacts, measurements, or an optional separate-model opinion; they
|
|
15
|
+
are not a required route for seeing images.
|
|
16
|
+
|
|
17
|
+
## Workflow
|
|
18
|
+
|
|
19
|
+
1. Inspect the image natively and identify what the task requires. If you only
|
|
20
|
+
have a description and need a missing detail, open the original image path.
|
|
21
|
+
If the file is gone, report the missing evidence.
|
|
22
|
+
2. Zoom only when a relevant detail is unresolved. Crop that region and read
|
|
23
|
+
the crop natively; use `ground` only if its estimated box would help locate
|
|
24
|
+
the target. Stop inspecting once you have evidence for the requested answer.
|
|
25
|
+
3. Use local image processing for numerical claims or reusable artifacts.
|
|
26
|
+
Treat model-generated boxes as estimates, not measured boundaries; verify
|
|
27
|
+
their target and coordinate space before acting on them.
|
|
28
|
+
4. Verify the result against the task: readable text for OCR, nodes and edges
|
|
29
|
+
for diagrams, the expected UI state after an action, or a rendered comparison
|
|
30
|
+
for visual reconstruction. A low pixel-diff score alone is not correctness.
|
|
31
|
+
|
|
32
|
+
For browser or accessible GUI work, prefer DOM/accessibility information for
|
|
33
|
+
controls and state. Use screenshot coordinates when that information is
|
|
34
|
+
unavailable or the task specifically concerns appearance.
|
|
35
|
+
|
|
36
|
+
## Local tools: use when their output is needed
|
|
37
|
+
|
|
38
|
+
Resolve `scripts/` paths relative to this skill directory. Use `--help` for
|
|
39
|
+
additional options. These tools do not use the vision API; HTML rendering can
|
|
40
|
+
load resources referenced by the page.
|
|
41
|
+
|
|
42
|
+
| Need | Command |
|
|
43
|
+
| --- | --- |
|
|
44
|
+
| Reusable crop or close-up | `crop shot.png --region X1,Y1,X2,Y2 -o detail.png` |
|
|
45
|
+
| Locate pixel differences | `python3 scripts/pixel_diff.py before.png after.png` |
|
|
46
|
+
| Measure a region's palette | `python3 scripts/dominant_colors.py shot.png --region X1,Y1,X2,Y2` |
|
|
47
|
+
| Vectorize a flat graphic | `trace icon.png -o icon.svg` |
|
|
48
|
+
| Extract a transparent foreground | `python3 scripts/extract_fg.py shot.png --region X1,Y1,X2,Y2 -o icon.png` |
|
|
49
|
+
| Render an HTML file | `python3 scripts/html_shot.py page.html --width 1440 --height 900 -o page.png` |
|
|
50
|
+
|
|
51
|
+
- `crop --scale 4` enlarges a crop; it adds no missing detail. Coordinates
|
|
52
|
+
measured on that output use the enlarged grid, not the original image.
|
|
53
|
+
- `pixel_diff` reports average intensity difference and ranked grid cells,
|
|
54
|
+
not the fraction of changed pixels or a semantic comparison. It rescales a
|
|
55
|
+
size-mismatched second image and reports that mismatch; match viewports
|
|
56
|
+
before judging visual fidelity.
|
|
57
|
+
- `dominant_colors` clusters sampled pixels. `--candidates '#F9FAFA,#F5F5F5'`
|
|
58
|
+
selects the closest supplied palette value; it does not recover original CSS.
|
|
59
|
+
- `trace` produces approximate vector paths from pixels, with filtering,
|
|
60
|
+
scaling and curve fitting. It is not an exact size/offset measurement tool.
|
|
61
|
+
It suits flat, high-contrast graphics; text becomes paths. Read the graphic
|
|
62
|
+
playbook before shipping its output.
|
|
63
|
+
- `extract_fg` has manual-region and centered-icon auto modes. Inspect the
|
|
64
|
+
alpha edges and complete foreground on light and dark backgrounds before use.
|
|
65
|
+
|
|
66
|
+
Use existing utilities when they fit. A small Pillow operation is also fine
|
|
67
|
+
when it is the simplest way to obtain an output these tools do not provide.
|
|
68
|
+
|
|
69
|
+
## Task playbooks: load only the matching one
|
|
70
|
+
|
|
71
|
+
| Task | Read |
|
|
72
|
+
| --- | --- |
|
|
73
|
+
| Long screenshot or chat-history transcription, chunking and boundary checks | `references/long-screenshot-ocr.md` |
|
|
74
|
+
| Rebuild or align a UI from a reference; fast approximation or close fidelity | `references/restore-ui.md` |
|
|
75
|
+
| Extract or rebuild an isolated graphic as PNG/SVG | `references/restore-graphic.md` |
|
|
76
|
+
| Convert a visual diagram into Mermaid, Graphviz or structured data | `references/restore-structure.md` |
|
|
77
|
+
| Operate a GUI from screenshots: calibrate, act, verify | `references/gui.md` |
|
|
78
|
+
|
|
79
|
+
## Optional model-backed CLIs
|
|
80
|
+
|
|
81
|
+
Use these only for a requested second opinion, a needed box/list contract,
|
|
82
|
+
scripted OCR, or a model without native image input. They send images to the
|
|
83
|
+
separately configured vision chain, not to the main agent model automatically.
|
|
84
|
+
Structured output does not guarantee better recognition or localization.
|
|
85
|
+
|
|
86
|
+
```bash
|
|
87
|
+
glance shot.png -q "Read the small status label."
|
|
88
|
+
glance shot.png --region X1,Y1,X2,Y2 --ocr
|
|
89
|
+
glance before.png after.png -q "What changed semantically?"
|
|
90
|
+
ground shot.png "the send button"
|
|
91
|
+
detect shot.png "buttons"
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
`ground` and `detect` return estimated boxes in original-image pixels, including
|
|
95
|
+
with `--region`. They can miss objects or select the wrong target, not just
|
|
96
|
+
round an edge incorrectly. `detect` is not an exhaustive inventory guarantee.
|
|
97
|
+
For multi-image questions, pass the images in one call so the model sees them
|
|
98
|
+
together. Native side-by-side inspection is also sufficient for semantic
|
|
99
|
+
comparison; use pixel diff when numerical differences matter.
|
|
100
|
+
|
|
101
|
+
The long-screenshot OCR script still uses `glance` for automated transcription.
|
|
102
|
+
Its `--split-only` mode makes chunks without API requests for native inspection.
|
|
103
|
+
Keep the `glance` client/config available if using the automated path.
|
|
104
|
+
|
|
105
|
+
### Configuration and failures
|
|
106
|
+
|
|
107
|
+
`extensions/vision-toolkit.json` under the Pi agent directory selects the CLI `model` and
|
|
108
|
+
`fallbacks`. For image-capable main models, `imagePolicy` defaults to `native`:
|
|
109
|
+
old and new images stay images, with no automatic description request.
|
|
110
|
+
`describe-old` and `describe` are legacy, explicit opt-ins for lossy image-to-text
|
|
111
|
+
conversion on supported connections. Text-only or unknown-capability models
|
|
112
|
+
still use the configured compatibility bridge. This policy does not route CLI
|
|
113
|
+
calls through the main model.
|
|
114
|
+
|
|
115
|
+
On a model-backed CLI configuration/API error, run `vision-doctor`. Use
|
|
116
|
+
`vision-doctor --smoke` only when a real diagnostic request is needed; it sends
|
|
117
|
+
a test image. Relay failures honestly. Continue with native inspection when
|
|
118
|
+
available; report if the scripted output cannot be produced. Keep provider
|
|
119
|
+
changes user-controlled: do not scrape credentials, hand-export secret overrides,
|
|
120
|
+
or silently switch to an unconfigured provider.
|
|
121
|
+
|
|
122
|
+
Local tools need their optional dependencies (Pillow, NumPy, vtracer or a
|
|
123
|
+
Chrome-family browser, depending on the tool). If a command is missing, use an
|
|
124
|
+
already available equivalent or report the missing capability; API diagnostics
|
|
125
|
+
do not fix local dependency errors.
|
|
126
|
+
|
|
127
|
+
## Image lifetime
|
|
128
|
+
|
|
129
|
+
Use temporary image paths directly for immediate inspection. Copy into durable
|
|
130
|
+
work storage only when the file must survive its owner or session, or be kept
|
|
131
|
+
with deliverables. Clipboard-image attachments are deleted at Pi session
|
|
132
|
+
shutdown; preserve them beforehand only if needed beyond that session.
|
|
133
|
+
|
|
134
|
+
Source: https://github.com/Anionex/agent-vision-toolkit
|
|
@@ -0,0 +1,94 @@
|
|
|
1
|
+
#!/usr/bin/env python3
|
|
2
|
+
"""crop: cut a pixel box out of an image into its own file.
|
|
3
|
+
|
|
4
|
+
The other CLIs zoom by re-cropping in memory (glance --region, trace
|
|
5
|
+
--region), but once the same box is needed twice — feed it to pixel_diff,
|
|
6
|
+
dominant_colors, trace, or another agent — it earns a file. Coordinates are
|
|
7
|
+
the same X1,Y1,X2,Y2 pixel boxes ground/detect print, and are clamped to
|
|
8
|
+
the image bounds exactly like the --region flags elsewhere in the toolkit.
|
|
9
|
+
"""
|
|
10
|
+
|
|
11
|
+
import argparse
|
|
12
|
+
import sys
|
|
13
|
+
# Windows consoles default to cp1252; descriptions can contain any Unicode.
|
|
14
|
+
if hasattr(sys.stdout, "reconfigure"):
|
|
15
|
+
sys.stdout.reconfigure(encoding="utf-8", errors="replace")
|
|
16
|
+
sys.stderr.reconfigure(encoding="utf-8", errors="replace")
|
|
17
|
+
from pathlib import Path
|
|
18
|
+
|
|
19
|
+
|
|
20
|
+
def parse_region(region: str) -> tuple[int, int, int, int]:
|
|
21
|
+
parts = region.split(",")
|
|
22
|
+
if len(parts) != 4:
|
|
23
|
+
raise ValueError("--region expects four integers: X1,Y1,X2,Y2 (pixels)")
|
|
24
|
+
try:
|
|
25
|
+
return (int(parts[0]), int(parts[1]), int(parts[2]), int(parts[3]))
|
|
26
|
+
except ValueError:
|
|
27
|
+
raise ValueError("--region expects four integers: X1,Y1,X2,Y2 (pixels)") from None
|
|
28
|
+
|
|
29
|
+
|
|
30
|
+
def clamp_box(box: tuple[int, int, int, int], width: int, height: int) -> tuple[int, int, int, int]:
|
|
31
|
+
x1, y1, x2, y2 = box
|
|
32
|
+
x1, x2 = sorted((x1, x2))
|
|
33
|
+
y1, y2 = sorted((y1, y2))
|
|
34
|
+
return (max(0, x1), max(0, y1), min(width, x2), min(height, y2))
|
|
35
|
+
|
|
36
|
+
|
|
37
|
+
def main() -> None:
|
|
38
|
+
parser = argparse.ArgumentParser(
|
|
39
|
+
prog="crop",
|
|
40
|
+
description="Cut a pixel box out of an image into its own file (local, no vision API)",
|
|
41
|
+
)
|
|
42
|
+
parser.add_argument("image", help="path to the image")
|
|
43
|
+
parser.add_argument("--region", metavar="X1,Y1,X2,Y2", required=True,
|
|
44
|
+
help="pixel box to cut out (e.g. a box from ground); clamped to the image bounds")
|
|
45
|
+
parser.add_argument("-o", "--output",
|
|
46
|
+
help="write the crop here (default: <image-stem>.crop.png next to the input)")
|
|
47
|
+
parser.add_argument("--scale", type=int, default=1, metavar="N",
|
|
48
|
+
help="upscale the cut-out N times (LANCZOS) before writing; "
|
|
49
|
+
"use 2-4 for small icons before ground/trace")
|
|
50
|
+
args = parser.parse_args()
|
|
51
|
+
if args.scale < 1:
|
|
52
|
+
parser.exit(1, "crop: --scale must be >= 1\n")
|
|
53
|
+
|
|
54
|
+
try:
|
|
55
|
+
from PIL import Image
|
|
56
|
+
except ImportError:
|
|
57
|
+
parser.exit(1, "crop: requires Pillow; install the optional dependency pillow first\n")
|
|
58
|
+
|
|
59
|
+
source = Path(args.image).expanduser()
|
|
60
|
+
if not source.is_file():
|
|
61
|
+
parser.exit(1, f"crop: Image not found: {source}\n")
|
|
62
|
+
|
|
63
|
+
try:
|
|
64
|
+
x1, y1, x2, y2 = parse_region(args.region)
|
|
65
|
+
except ValueError as exc:
|
|
66
|
+
parser.exit(1, f"crop: {exc}\n")
|
|
67
|
+
|
|
68
|
+
with Image.open(source) as image:
|
|
69
|
+
width, height = image.size
|
|
70
|
+
box = clamp_box((x1, y1, x2, y2), width, height)
|
|
71
|
+
if box[2] <= box[0] or box[3] <= box[1]:
|
|
72
|
+
parser.exit(1, f"crop: --region {args.region} is empty after clamping to {width}x{height}\n")
|
|
73
|
+
if box != (x1, y1, x2, y2):
|
|
74
|
+
print(f"note: region {x1},{y1},{x2},{y2} clamped to {box[0]},{box[1]},{box[2]},{box[3]}",
|
|
75
|
+
file=sys.stderr)
|
|
76
|
+
if args.output:
|
|
77
|
+
output = Path(args.output).expanduser()
|
|
78
|
+
elif args.scale > 1:
|
|
79
|
+
output = source.with_name(source.stem + f".crop@{args.scale}x.png")
|
|
80
|
+
else:
|
|
81
|
+
output = source.with_name(source.stem + ".crop.png")
|
|
82
|
+
try:
|
|
83
|
+
crop = image.crop(box)
|
|
84
|
+
if args.scale > 1:
|
|
85
|
+
crop = crop.resize((crop.width * args.scale, crop.height * args.scale),
|
|
86
|
+
resample=Image.LANCZOS)
|
|
87
|
+
crop.save(output)
|
|
88
|
+
except OSError as exc:
|
|
89
|
+
parser.exit(1, f"crop: cannot write {output}: {exc}\n")
|
|
90
|
+
print(f"wrote {output} ({(box[2] - box[0]) * args.scale}x{(box[3] - box[1]) * args.scale})")
|
|
91
|
+
|
|
92
|
+
|
|
93
|
+
if __name__ == "__main__":
|
|
94
|
+
main()
|
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
#!/usr/bin/env python3
|
|
2
|
+
"""detect: inventory the elements in an image and print pixel bounding boxes."""
|
|
3
|
+
|
|
4
|
+
from pathlib import Path
|
|
5
|
+
import sys
|
|
6
|
+
|
|
7
|
+
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
|
8
|
+
|
|
9
|
+
# Windows consoles default to cp1252; descriptions can contain any Unicode.
|
|
10
|
+
if hasattr(sys.stdout, "reconfigure"):
|
|
11
|
+
sys.stdout.reconfigure(encoding="utf-8", errors="replace")
|
|
12
|
+
sys.stderr.reconfigure(encoding="utf-8", errors="replace")
|
|
13
|
+
|
|
14
|
+
from detect import main # noqa: E402
|
|
15
|
+
|
|
16
|
+
|
|
17
|
+
if __name__ == "__main__":
|
|
18
|
+
main()
|
|
@@ -0,0 +1,98 @@
|
|
|
1
|
+
#!/usr/bin/env python3
|
|
2
|
+
"""glance: standalone image description, Q&A, and OCR CLI."""
|
|
3
|
+
|
|
4
|
+
import argparse
|
|
5
|
+
from pathlib import Path
|
|
6
|
+
import sys
|
|
7
|
+
|
|
8
|
+
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
|
9
|
+
|
|
10
|
+
# Windows consoles default to cp1252; descriptions can contain any Unicode.
|
|
11
|
+
if hasattr(sys.stdout, "reconfigure"):
|
|
12
|
+
sys.stdout.reconfigure(encoding="utf-8", errors="replace")
|
|
13
|
+
sys.stderr.reconfigure(encoding="utf-8", errors="replace")
|
|
14
|
+
|
|
15
|
+
from vision_client import ( # noqa: E402
|
|
16
|
+
VisionError,
|
|
17
|
+
describe_image,
|
|
18
|
+
image_path_to_data_url,
|
|
19
|
+
load_default_env,
|
|
20
|
+
)
|
|
21
|
+
|
|
22
|
+
|
|
23
|
+
def region_data_url(path, region):
|
|
24
|
+
try:
|
|
25
|
+
from PIL import Image
|
|
26
|
+
except ImportError:
|
|
27
|
+
raise VisionError("--region requires Pillow; install the optional dependency pillow first")
|
|
28
|
+
try:
|
|
29
|
+
x1, y1, x2, y2 = (int(v) for v in region.split(","))
|
|
30
|
+
except ValueError:
|
|
31
|
+
raise VisionError("--region expects four integers: X1,Y1,X2,Y2 (pixels)")
|
|
32
|
+
import base64
|
|
33
|
+
import io
|
|
34
|
+
try:
|
|
35
|
+
with Image.open(Path(path).expanduser()) as image:
|
|
36
|
+
width, height = image.size
|
|
37
|
+
box = (max(0, min(x1, x2)), max(0, min(y1, y2)),
|
|
38
|
+
min(width, max(x1, x2)), min(height, max(y1, y2)))
|
|
39
|
+
if box[2] <= box[0] or box[3] <= box[1]:
|
|
40
|
+
raise VisionError(f"--region {region} is empty after clamping to {width}x{height}")
|
|
41
|
+
buffer = io.BytesIO()
|
|
42
|
+
image.crop(box).save(buffer, format="PNG")
|
|
43
|
+
except (OSError, ValueError) as exc:
|
|
44
|
+
raise VisionError(f"Cannot read image: {path}") from exc
|
|
45
|
+
return "data:image/png;base64," + base64.b64encode(buffer.getvalue()).decode()
|
|
46
|
+
|
|
47
|
+
|
|
48
|
+
def build_prompt(args, count):
|
|
49
|
+
if args.ocr is not None:
|
|
50
|
+
extra = f" Additional requirements: {args.ocr}" if args.ocr else ""
|
|
51
|
+
scope = "these images" if count > 1 else "this image"
|
|
52
|
+
return (
|
|
53
|
+
f"Transcribe every piece of visible text in {scope} verbatim (titles, body text, labels, watermarks, etc.), "
|
|
54
|
+
"line by line, without omitting any characters. Do not rewrite, summarize, or translate the text, "
|
|
55
|
+
"and do not add any preamble, explanation, or extra content."
|
|
56
|
+
+ (" Label each image's text with its ordinal (Image 1, Image 2, ...)." if count > 1 else "")
|
|
57
|
+
+ extra
|
|
58
|
+
)
|
|
59
|
+
if args.query:
|
|
60
|
+
return args.query
|
|
61
|
+
if count > 1:
|
|
62
|
+
return ("Describe each image in detail (label them Image 1, Image 2, ...), "
|
|
63
|
+
"then point out the notable differences between them.")
|
|
64
|
+
return None
|
|
65
|
+
|
|
66
|
+
|
|
67
|
+
def main():
|
|
68
|
+
parser = argparse.ArgumentParser(
|
|
69
|
+
prog="glance",
|
|
70
|
+
description="Describe, answer questions about, or OCR an image with the configured vision model",
|
|
71
|
+
)
|
|
72
|
+
parser.add_argument("images", nargs="+", metavar="image",
|
|
73
|
+
help="path(s) to the image(s); pass several to compare them in one call")
|
|
74
|
+
group = parser.add_mutually_exclusive_group()
|
|
75
|
+
group.add_argument("-q", "--query", help="ask a question about the image(s)")
|
|
76
|
+
group.add_argument("--ocr", nargs="?", const="", metavar="EXTRA", help="transcribe all visible text verbatim")
|
|
77
|
+
parser.add_argument("--region", metavar="X1,Y1,X2,Y2",
|
|
78
|
+
help="crop to this pixel box (e.g. from ground) and send only the crop")
|
|
79
|
+
args = parser.parse_args()
|
|
80
|
+
load_default_env()
|
|
81
|
+
try:
|
|
82
|
+
if args.region and len(args.images) > 1:
|
|
83
|
+
raise VisionError("--region works with exactly one image")
|
|
84
|
+
urls = ([region_data_url(args.images[0], args.region)] if args.region
|
|
85
|
+
else [image_path_to_data_url(path) for path in args.images])
|
|
86
|
+
answer = describe_image(
|
|
87
|
+
urls,
|
|
88
|
+
build_prompt(args, len(urls)),
|
|
89
|
+
max_tokens=None, # no output cap; the vision model decides
|
|
90
|
+
apply_lang=args.ocr is None,
|
|
91
|
+
)
|
|
92
|
+
except VisionError as exc:
|
|
93
|
+
parser.exit(1, f"glance: {exc}\n")
|
|
94
|
+
print(answer)
|
|
95
|
+
|
|
96
|
+
|
|
97
|
+
if __name__ == "__main__":
|
|
98
|
+
main()
|
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
#!/usr/bin/env python3
|
|
2
|
+
"""ground: locate objects in an image and print pixel bounding boxes."""
|
|
3
|
+
|
|
4
|
+
from pathlib import Path
|
|
5
|
+
import sys
|
|
6
|
+
|
|
7
|
+
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
|
8
|
+
|
|
9
|
+
# Windows consoles default to cp1252; descriptions can contain any Unicode.
|
|
10
|
+
if hasattr(sys.stdout, "reconfigure"):
|
|
11
|
+
sys.stdout.reconfigure(encoding="utf-8", errors="replace")
|
|
12
|
+
sys.stderr.reconfigure(encoding="utf-8", errors="replace")
|
|
13
|
+
|
|
14
|
+
from ground import main # noqa: E402
|
|
15
|
+
|
|
16
|
+
|
|
17
|
+
if __name__ == "__main__":
|
|
18
|
+
main()
|
|
@@ -0,0 +1,134 @@
|
|
|
1
|
+
#!/usr/bin/env python3
|
|
2
|
+
"""trace: deterministic image-to-SVG tracing for exact shape geometry.
|
|
3
|
+
|
|
4
|
+
Runs fully local (no vision API, no credentials): coordinates come from the
|
|
5
|
+
actual pixels, so use this instead of asking a vision model for shapes,
|
|
6
|
+
sizes, or outlines. Flat, high-contrast graphics only; text becomes curves.
|
|
7
|
+
"""
|
|
8
|
+
|
|
9
|
+
import argparse
|
|
10
|
+
from pathlib import Path
|
|
11
|
+
import re
|
|
12
|
+
import sys
|
|
13
|
+
import tempfile
|
|
14
|
+
|
|
15
|
+
# Windows consoles default to cp1252; descriptions can contain any Unicode.
|
|
16
|
+
if hasattr(sys.stdout, "reconfigure"):
|
|
17
|
+
sys.stdout.reconfigure(encoding="utf-8", errors="replace")
|
|
18
|
+
sys.stderr.reconfigure(encoding="utf-8", errors="replace")
|
|
19
|
+
|
|
20
|
+
WHITE_FILLS = {"#ffffff", "#fff", "white"}
|
|
21
|
+
|
|
22
|
+
# Speckle filtering drops components under a fixed pixel area, so a 30px icon
|
|
23
|
+
# binarizes to nothing at 1x while the same icon enlarged traces cleanly.
|
|
24
|
+
# Enlarge anything smaller than this; full-size screenshots already clear it.
|
|
25
|
+
TARGET_MIN_SIDE = 256
|
|
26
|
+
|
|
27
|
+
|
|
28
|
+
def strip_background(svg):
|
|
29
|
+
"""Drop the leading full-canvas white path vtracer emits for the background."""
|
|
30
|
+
match = re.search(r"<path [^>]*/>", svg)
|
|
31
|
+
if match:
|
|
32
|
+
fill = re.search(r'fill="([^"]+)"', match.group(0))
|
|
33
|
+
if fill and fill.group(1).strip().lower() in WHITE_FILLS:
|
|
34
|
+
return svg.replace(match.group(0), "", 1)
|
|
35
|
+
return svg
|
|
36
|
+
|
|
37
|
+
|
|
38
|
+
def truncate_decimals(svg, places=2):
|
|
39
|
+
return re.sub(r"-?\d+\.\d{3,}", lambda m: f"{float(m.group()):.{places}f}", svg)
|
|
40
|
+
|
|
41
|
+
|
|
42
|
+
def write_svg(path, svg):
|
|
43
|
+
"""Write the exact UTF-8 payload whose byte count is reported to callers."""
|
|
44
|
+
payload = svg.encode("utf-8")
|
|
45
|
+
return Path(path).expanduser().write_bytes(payload)
|
|
46
|
+
|
|
47
|
+
|
|
48
|
+
def prepare_input(path, region, scale):
|
|
49
|
+
"""Crop/upscale with Pillow when asked; return the path vtracer reads and the scale used.
|
|
50
|
+
|
|
51
|
+
`scale` may be None, meaning "pick one from the size": small shapes are
|
|
52
|
+
enlarged until they survive speckle filtering. Without Pillow we cannot
|
|
53
|
+
measure the image, so that choice falls back to no upscale.
|
|
54
|
+
"""
|
|
55
|
+
try:
|
|
56
|
+
from PIL import Image
|
|
57
|
+
except ImportError:
|
|
58
|
+
if region or (scale or 1) != 1:
|
|
59
|
+
sys.exit("trace: --region/--scale require Pillow; install the optional dependency pillow first")
|
|
60
|
+
return str(path), 1
|
|
61
|
+
with Image.open(path) as image:
|
|
62
|
+
if region:
|
|
63
|
+
try:
|
|
64
|
+
x1, y1, x2, y2 = (int(v) for v in region.split(","))
|
|
65
|
+
except ValueError:
|
|
66
|
+
sys.exit("trace: --region expects four integers: X1,Y1,X2,Y2 (pixels)")
|
|
67
|
+
box = (max(0, min(x1, x2)), max(0, min(y1, y2)),
|
|
68
|
+
min(image.width, max(x1, x2)), min(image.height, max(y1, y2)))
|
|
69
|
+
if box[2] <= box[0] or box[3] <= box[1]:
|
|
70
|
+
sys.exit(f"trace: --region {region} is empty after clamping to {image.width}x{image.height}")
|
|
71
|
+
image = image.crop(box)
|
|
72
|
+
if scale is None:
|
|
73
|
+
shortest = max(min(image.width, image.height), 1)
|
|
74
|
+
scale = max(2 if region else 1, -(-TARGET_MIN_SIDE // shortest))
|
|
75
|
+
if not region and scale == 1:
|
|
76
|
+
return str(path), 1
|
|
77
|
+
if scale != 1:
|
|
78
|
+
image = image.resize((image.width * scale, image.height * scale), Image.LANCZOS)
|
|
79
|
+
handle = tempfile.NamedTemporaryFile(suffix=".png", delete=False)
|
|
80
|
+
image.save(handle.name)
|
|
81
|
+
return handle.name, scale
|
|
82
|
+
|
|
83
|
+
|
|
84
|
+
def main():
|
|
85
|
+
parser = argparse.ArgumentParser(
|
|
86
|
+
prog="trace",
|
|
87
|
+
description="Trace an image into SVG with exact pixel-derived geometry (local, deterministic)",
|
|
88
|
+
)
|
|
89
|
+
parser.add_argument("image", help="path to the image")
|
|
90
|
+
parser.add_argument("--region", metavar="X1,Y1,X2,Y2",
|
|
91
|
+
help="crop to this pixel box first (e.g. a box from ground)")
|
|
92
|
+
parser.add_argument("--scale", type=int, metavar="N",
|
|
93
|
+
help="upscale N times before tracing (default: enough to clear "
|
|
94
|
+
f"{TARGET_MIN_SIDE}px on the shorter side, so small icons survive)")
|
|
95
|
+
parser.add_argument("--polygon", action="store_true",
|
|
96
|
+
help="polygon mode for boxy diagrams/wireframes (default: spline for curved shapes)")
|
|
97
|
+
parser.add_argument("--color", action="store_true",
|
|
98
|
+
help="keep colors (default: black-and-white, which is far more compact)")
|
|
99
|
+
parser.add_argument("-o", "--output", help="write the SVG here instead of stdout")
|
|
100
|
+
args = parser.parse_args()
|
|
101
|
+
try:
|
|
102
|
+
import vtracer
|
|
103
|
+
except ImportError:
|
|
104
|
+
parser.exit(1, "trace: requires vtracer; install the optional dependency vtracer first\n")
|
|
105
|
+
source = Path(args.image).expanduser()
|
|
106
|
+
if not source.is_file():
|
|
107
|
+
parser.exit(1, f"trace: Image not found: {source}\n")
|
|
108
|
+
work, scale = prepare_input(source, args.region, args.scale)
|
|
109
|
+
with tempfile.NamedTemporaryFile(suffix=".svg", delete=False) as out:
|
|
110
|
+
vtracer.convert_image_to_svg_py(
|
|
111
|
+
work, out.name,
|
|
112
|
+
colormode="color" if args.color else "binary",
|
|
113
|
+
filter_speckle=8, corner_threshold=40,
|
|
114
|
+
mode="polygon" if args.polygon else "spline",
|
|
115
|
+
)
|
|
116
|
+
svg = Path(out.name).read_text()
|
|
117
|
+
svg = truncate_decimals(strip_background(svg))
|
|
118
|
+
paths = svg.count("<path")
|
|
119
|
+
if not paths:
|
|
120
|
+
# This is where an agent decides the tool cannot see the shape and goes
|
|
121
|
+
# back to guessing. Name the recoveries instead, cheapest first.
|
|
122
|
+
print("trace: 0 paths - nothing survived binarization. Try a larger --scale, "
|
|
123
|
+
"--region to crop closer to the shape, or pre-invert a light-on-dark image. "
|
|
124
|
+
"--color is a last resort: on an anti-aliased image it splits every gray "
|
|
125
|
+
"level into its own path.", file=sys.stderr)
|
|
126
|
+
if args.output:
|
|
127
|
+
bytes_written = write_svg(args.output, svg)
|
|
128
|
+
print(f"wrote {args.output} ({bytes_written} bytes, {paths} paths, traced at {scale}x)")
|
|
129
|
+
else:
|
|
130
|
+
print(svg)
|
|
131
|
+
|
|
132
|
+
|
|
133
|
+
if __name__ == "__main__":
|
|
134
|
+
main()
|