@saifymatteo/pi-vision 0.0.0-stage → 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,3 +1,43 @@
1
- # Temporary Holding Version
1
+ # @saifymatteo/pi-vision
2
2
 
3
- This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
3
+ ## Purpose and registrations
4
+
5
+ Registers a `context` hook, not a tool or slash command. Preserves native images for image-capable session models by default. For text-only/unknown models it replaces image blocks with model-generated descriptions. Optional `describe` / `describe-old` policies self-describe all/older images using the current model on OpenAI Chat Completions connections.
6
+
7
+ ## Configuration and services
8
+
9
+ Reads `<agent-dir>/extensions/vision-toolkit.json`: `model`, optional `fallbacks`, `imagePolicy` (native/describe/describe-old), `lang` (en/zh), and `providers` overrides. Example points at example.invalid and must be reviewed manually; it is never loaded automatically. Provider override keys support `$ENV_NAME` references (not brace syntax).
10
+
11
+ Resolution also consults agent `models.json`, `models-store.json`, `auth.json`, and optional `haoyiyin/9router.json`. None of those real files are bundled. An explicit provider/model ID and override can resolve without a static catalog entry. Requires an image-capable OpenAI Chat Completions service; no CLI. Self-description requires current-model credentials in auth.json.
12
+
13
+ ## Bundled skill
14
+
15
+ The package bundles the `vision-tools` skill (`skills/vision-tools`): `bin/` launcher CLIs (`crop`, `detect`, `glance`, `ground`, `trace`, `vision-doctor`), Python tools under `scripts/`, reference guides, and `SKILL.md`. The extension's guidance messages point the agent at `vision-doctor` and the skill. Requires Python 3 with Pillow (PIL); the CLIs are Python scripts, not compiled binaries. Based on [Anionex/agent-vision-toolkit](https://github.com/Anionex/agent-vision-toolkit) (MIT), modified in this repository — see [NOTICE.md](NOTICE.md).
16
+
17
+ ## Host limitations and security
18
+
19
+ The hook expects host-cloned request messages; rewriting should not mutate stored sessions. No config files are created. Missing/broken config leaves native-capable models native, but text-only models receive explicit failure markers rather than images. Image pixels and focus hints from surrounding user/assistant messages are transmitted to the model service; descriptions enter main-model context. Do not pass sensitive screenshots. Up to four concurrent requests, in-process cache, 180s per-request timeout and transient retries; configured HTTP failures can fail over. This is a lossy bridge, not an image security boundary.
20
+
21
+ Inherited caveats: cache keys include model ID but not provider/base URL; changing providers with the same ID can reuse cached descriptions until reload. The returned chain language is not propagated to individual candidates by the source. These behaviors were preserved, not repaired in this packaging work.
22
+
23
+ ## Tests
24
+
25
+ From the repository root with its existing test dependencies available:
26
+
27
+ ```sh
28
+ node --test packages/pi-vision/test/*.test.mjs
29
+ ```
30
+
31
+ Tests use the shared test-only TypeScript/VM loader, fixtures, and host/service stubs. They do not prove compatibility with a live host, CLI, or model service.
32
+
33
+ ## Running locally
34
+
35
+ From the repository root:
36
+
37
+ ```sh
38
+ pi -e ./packages/pi-vision
39
+ ```
40
+
41
+ Node >=24; host imports are peers, not bundled dependencies. Load only one copy: for an isolated session use `--no-extensions` with explicit `-e` paths so another installed copy is not also discovered. Set `PI_CODING_AGENT_DIR` to a disposable agent directory before launch; unset means the normal `~/.pi/agent` fallback. Extension config continues to resolve there, never relative to this package.
42
+
43
+ Examples are manually authored, sanitized reference documents, not runtime defaults. Review and manually create only the needed files in the isolated agent directory; do not copy live credentials or endpoints. No example is automatically loaded or copied. Loading an extension grants Pi-process OS permissions. Licensed AGPL-3.0-only; see [NOTICE.md](NOTICE.md).
@@ -0,0 +1,12 @@
1
+ {
2
+ "model": "example-vision/example-model",
3
+ "fallbacks": [],
4
+ "imagePolicy": "native",
5
+ "lang": "en",
6
+ "providers": {
7
+ "example-vision": {
8
+ "baseUrl": "https://example.invalid/v1",
9
+ "apiKey": "$PI_VISION_API_KEY"
10
+ }
11
+ }
12
+ }
package/package.json CHANGED
@@ -1,6 +1,38 @@
1
1
  {
2
2
  "name": "@saifymatteo/pi-vision",
3
- "version": "0.0.0-stage",
4
- "stub": true,
5
- "description": "Temporary package placeholder for staged publishing"
6
- }
3
+ "version": "0.1.0",
4
+ "license": "AGPL-3.0-only",
5
+ "type": "module",
6
+ "keywords": [
7
+ "pi-package"
8
+ ],
9
+ "files": [
10
+ "src",
11
+ "examples",
12
+ "README.md",
13
+ "NOTICE.md",
14
+ "LICENSE",
15
+ "skills"
16
+ ],
17
+ "pi": {
18
+ "extensions": [
19
+ "./src/index.ts"
20
+ ],
21
+ "skills": [
22
+ "./skills"
23
+ ]
24
+ },
25
+ "peerDependencies": {},
26
+ "engines": {
27
+ "node": ">=24"
28
+ },
29
+ "description": "Describe images for text-only models via a configured vision chain.",
30
+ "publishConfig": {
31
+ "access": "public"
32
+ },
33
+ "repository": {
34
+ "type": "git",
35
+ "url": "git+https://github.com/saifymatteo/pi-extensions.git",
36
+ "directory": "packages/pi-vision"
37
+ }
38
+ }
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Anionex
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,134 @@
1
+ ---
2
+ name: vision-tools
3
+ description: >-
4
+ Native-first workflows for screenshot reconstruction, diagram extraction,
5
+ screenshot-driven GUI work, long-screenshot OCR, and image processing
6
+ (crops, pixel differences, palette measurements, SVG tracing, foreground
7
+ extraction). Use when a task needs these workflows or concrete image
8
+ artifacts; ordinary image questions can be answered with native vision.
9
+ ---
10
+
11
+ # Vision tools
12
+
13
+ Use the main model's native vision first. The tools here supplement it with
14
+ image artifacts, measurements, or an optional separate-model opinion; they
15
+ are not a required route for seeing images.
16
+
17
+ ## Workflow
18
+
19
+ 1. Inspect the image natively and identify what the task requires. If you only
20
+ have a description and need a missing detail, open the original image path.
21
+ If the file is gone, report the missing evidence.
22
+ 2. Zoom only when a relevant detail is unresolved. Crop that region and read
23
+ the crop natively; use `ground` only if its estimated box would help locate
24
+ the target. Stop inspecting once you have evidence for the requested answer.
25
+ 3. Use local image processing for numerical claims or reusable artifacts.
26
+ Treat model-generated boxes as estimates, not measured boundaries; verify
27
+ their target and coordinate space before acting on them.
28
+ 4. Verify the result against the task: readable text for OCR, nodes and edges
29
+ for diagrams, the expected UI state after an action, or a rendered comparison
30
+ for visual reconstruction. A low pixel-diff score alone is not correctness.
31
+
32
+ For browser or accessible GUI work, prefer DOM/accessibility information for
33
+ controls and state. Use screenshot coordinates when that information is
34
+ unavailable or the task specifically concerns appearance.
35
+
36
+ ## Local tools: use when their output is needed
37
+
38
+ Resolve `scripts/` paths relative to this skill directory. Use `--help` for
39
+ additional options. These tools do not use the vision API; HTML rendering can
40
+ load resources referenced by the page.
41
+
42
+ | Need | Command |
43
+ | --- | --- |
44
+ | Reusable crop or close-up | `crop shot.png --region X1,Y1,X2,Y2 -o detail.png` |
45
+ | Locate pixel differences | `python3 scripts/pixel_diff.py before.png after.png` |
46
+ | Measure a region's palette | `python3 scripts/dominant_colors.py shot.png --region X1,Y1,X2,Y2` |
47
+ | Vectorize a flat graphic | `trace icon.png -o icon.svg` |
48
+ | Extract a transparent foreground | `python3 scripts/extract_fg.py shot.png --region X1,Y1,X2,Y2 -o icon.png` |
49
+ | Render an HTML file | `python3 scripts/html_shot.py page.html --width 1440 --height 900 -o page.png` |
50
+
51
+ - `crop --scale 4` enlarges a crop; it adds no missing detail. Coordinates
52
+ measured on that output use the enlarged grid, not the original image.
53
+ - `pixel_diff` reports average intensity difference and ranked grid cells,
54
+ not the fraction of changed pixels or a semantic comparison. It rescales a
55
+ size-mismatched second image and reports that mismatch; match viewports
56
+ before judging visual fidelity.
57
+ - `dominant_colors` clusters sampled pixels. `--candidates '#F9FAFA,#F5F5F5'`
58
+ selects the closest supplied palette value; it does not recover original CSS.
59
+ - `trace` produces approximate vector paths from pixels, with filtering,
60
+ scaling and curve fitting. It is not an exact size/offset measurement tool.
61
+ It suits flat, high-contrast graphics; text becomes paths. Read the graphic
62
+ playbook before shipping its output.
63
+ - `extract_fg` has manual-region and centered-icon auto modes. Inspect the
64
+ alpha edges and complete foreground on light and dark backgrounds before use.
65
+
66
+ Use existing utilities when they fit. A small Pillow operation is also fine
67
+ when it is the simplest way to obtain an output these tools do not provide.
68
+
69
+ ## Task playbooks: load only the matching one
70
+
71
+ | Task | Read |
72
+ | --- | --- |
73
+ | Long screenshot or chat-history transcription, chunking and boundary checks | `references/long-screenshot-ocr.md` |
74
+ | Rebuild or align a UI from a reference; fast approximation or close fidelity | `references/restore-ui.md` |
75
+ | Extract or rebuild an isolated graphic as PNG/SVG | `references/restore-graphic.md` |
76
+ | Convert a visual diagram into Mermaid, Graphviz or structured data | `references/restore-structure.md` |
77
+ | Operate a GUI from screenshots: calibrate, act, verify | `references/gui.md` |
78
+
79
+ ## Optional model-backed CLIs
80
+
81
+ Use these only for a requested second opinion, a needed box/list contract,
82
+ scripted OCR, or a model without native image input. They send images to the
83
+ separately configured vision chain, not to the main agent model automatically.
84
+ Structured output does not guarantee better recognition or localization.
85
+
86
+ ```bash
87
+ glance shot.png -q "Read the small status label."
88
+ glance shot.png --region X1,Y1,X2,Y2 --ocr
89
+ glance before.png after.png -q "What changed semantically?"
90
+ ground shot.png "the send button"
91
+ detect shot.png "buttons"
92
+ ```
93
+
94
+ `ground` and `detect` return estimated boxes in original-image pixels, including
95
+ with `--region`. They can miss objects or select the wrong target, not just
96
+ round an edge incorrectly. `detect` is not an exhaustive inventory guarantee.
97
+ For multi-image questions, pass the images in one call so the model sees them
98
+ together. Native side-by-side inspection is also sufficient for semantic
99
+ comparison; use pixel diff when numerical differences matter.
100
+
101
+ The long-screenshot OCR script still uses `glance` for automated transcription.
102
+ Its `--split-only` mode makes chunks without API requests for native inspection.
103
+ Keep the `glance` client/config available if using the automated path.
104
+
105
+ ### Configuration and failures
106
+
107
+ `extensions/vision-toolkit.json` under the Pi agent directory selects the CLI `model` and
108
+ `fallbacks`. For image-capable main models, `imagePolicy` defaults to `native`:
109
+ old and new images stay images, with no automatic description request.
110
+ `describe-old` and `describe` are legacy, explicit opt-ins for lossy image-to-text
111
+ conversion on supported connections. Text-only or unknown-capability models
112
+ still use the configured compatibility bridge. This policy does not route CLI
113
+ calls through the main model.
114
+
115
+ On a model-backed CLI configuration/API error, run `vision-doctor`. Use
116
+ `vision-doctor --smoke` only when a real diagnostic request is needed; it sends
117
+ a test image. Relay failures honestly. Continue with native inspection when
118
+ available; report if the scripted output cannot be produced. Keep provider
119
+ changes user-controlled: do not scrape credentials, hand-export secret overrides,
120
+ or silently switch to an unconfigured provider.
121
+
122
+ Local tools need their optional dependencies (Pillow, NumPy, vtracer or a
123
+ Chrome-family browser, depending on the tool). If a command is missing, use an
124
+ already available equivalent or report the missing capability; API diagnostics
125
+ do not fix local dependency errors.
126
+
127
+ ## Image lifetime
128
+
129
+ Use temporary image paths directly for immediate inspection. Copy into durable
130
+ work storage only when the file must survive its owner or session, or be kept
131
+ with deliverables. Clipboard-image attachments are deleted at Pi session
132
+ shutdown; preserve them beforehand only if needed beyond that session.
133
+
134
+ Source: https://github.com/Anionex/agent-vision-toolkit
@@ -0,0 +1,4 @@
1
+ interface:
2
+ display_name: "Agent Vision Toolkit"
3
+ short_description: "Native-first workflows and optional image processing"
4
+ default_prompt: "Use $vision-tools for native-first inspection, task-specific verification, and image-processing tools only when needed."
@@ -0,0 +1,94 @@
1
+ #!/usr/bin/env python3
2
+ """crop: cut a pixel box out of an image into its own file.
3
+
4
+ The other CLIs zoom by re-cropping in memory (glance --region, trace
5
+ --region), but once the same box is needed twice — feed it to pixel_diff,
6
+ dominant_colors, trace, or another agent — it earns a file. Coordinates are
7
+ the same X1,Y1,X2,Y2 pixel boxes ground/detect print, and are clamped to
8
+ the image bounds exactly like the --region flags elsewhere in the toolkit.
9
+ """
10
+
11
+ import argparse
12
+ import sys
13
+ # Windows consoles default to cp1252; descriptions can contain any Unicode.
14
+ if hasattr(sys.stdout, "reconfigure"):
15
+ sys.stdout.reconfigure(encoding="utf-8", errors="replace")
16
+ sys.stderr.reconfigure(encoding="utf-8", errors="replace")
17
+ from pathlib import Path
18
+
19
+
20
+ def parse_region(region: str) -> tuple[int, int, int, int]:
21
+ parts = region.split(",")
22
+ if len(parts) != 4:
23
+ raise ValueError("--region expects four integers: X1,Y1,X2,Y2 (pixels)")
24
+ try:
25
+ return (int(parts[0]), int(parts[1]), int(parts[2]), int(parts[3]))
26
+ except ValueError:
27
+ raise ValueError("--region expects four integers: X1,Y1,X2,Y2 (pixels)") from None
28
+
29
+
30
+ def clamp_box(box: tuple[int, int, int, int], width: int, height: int) -> tuple[int, int, int, int]:
31
+ x1, y1, x2, y2 = box
32
+ x1, x2 = sorted((x1, x2))
33
+ y1, y2 = sorted((y1, y2))
34
+ return (max(0, x1), max(0, y1), min(width, x2), min(height, y2))
35
+
36
+
37
+ def main() -> None:
38
+ parser = argparse.ArgumentParser(
39
+ prog="crop",
40
+ description="Cut a pixel box out of an image into its own file (local, no vision API)",
41
+ )
42
+ parser.add_argument("image", help="path to the image")
43
+ parser.add_argument("--region", metavar="X1,Y1,X2,Y2", required=True,
44
+ help="pixel box to cut out (e.g. a box from ground); clamped to the image bounds")
45
+ parser.add_argument("-o", "--output",
46
+ help="write the crop here (default: <image-stem>.crop.png next to the input)")
47
+ parser.add_argument("--scale", type=int, default=1, metavar="N",
48
+ help="upscale the cut-out N times (LANCZOS) before writing; "
49
+ "use 2-4 for small icons before ground/trace")
50
+ args = parser.parse_args()
51
+ if args.scale < 1:
52
+ parser.exit(1, "crop: --scale must be >= 1\n")
53
+
54
+ try:
55
+ from PIL import Image
56
+ except ImportError:
57
+ parser.exit(1, "crop: requires Pillow; install the optional dependency pillow first\n")
58
+
59
+ source = Path(args.image).expanduser()
60
+ if not source.is_file():
61
+ parser.exit(1, f"crop: Image not found: {source}\n")
62
+
63
+ try:
64
+ x1, y1, x2, y2 = parse_region(args.region)
65
+ except ValueError as exc:
66
+ parser.exit(1, f"crop: {exc}\n")
67
+
68
+ with Image.open(source) as image:
69
+ width, height = image.size
70
+ box = clamp_box((x1, y1, x2, y2), width, height)
71
+ if box[2] <= box[0] or box[3] <= box[1]:
72
+ parser.exit(1, f"crop: --region {args.region} is empty after clamping to {width}x{height}\n")
73
+ if box != (x1, y1, x2, y2):
74
+ print(f"note: region {x1},{y1},{x2},{y2} clamped to {box[0]},{box[1]},{box[2]},{box[3]}",
75
+ file=sys.stderr)
76
+ if args.output:
77
+ output = Path(args.output).expanduser()
78
+ elif args.scale > 1:
79
+ output = source.with_name(source.stem + f".crop@{args.scale}x.png")
80
+ else:
81
+ output = source.with_name(source.stem + ".crop.png")
82
+ try:
83
+ crop = image.crop(box)
84
+ if args.scale > 1:
85
+ crop = crop.resize((crop.width * args.scale, crop.height * args.scale),
86
+ resample=Image.LANCZOS)
87
+ crop.save(output)
88
+ except OSError as exc:
89
+ parser.exit(1, f"crop: cannot write {output}: {exc}\n")
90
+ print(f"wrote {output} ({(box[2] - box[0]) * args.scale}x{(box[3] - box[1]) * args.scale})")
91
+
92
+
93
+ if __name__ == "__main__":
94
+ main()
@@ -0,0 +1,18 @@
1
+ #!/usr/bin/env python3
2
+ """detect: inventory the elements in an image and print pixel bounding boxes."""
3
+
4
+ from pathlib import Path
5
+ import sys
6
+
7
+ sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
8
+
9
+ # Windows consoles default to cp1252; descriptions can contain any Unicode.
10
+ if hasattr(sys.stdout, "reconfigure"):
11
+ sys.stdout.reconfigure(encoding="utf-8", errors="replace")
12
+ sys.stderr.reconfigure(encoding="utf-8", errors="replace")
13
+
14
+ from detect import main # noqa: E402
15
+
16
+
17
+ if __name__ == "__main__":
18
+ main()
@@ -0,0 +1,98 @@
1
+ #!/usr/bin/env python3
2
+ """glance: standalone image description, Q&A, and OCR CLI."""
3
+
4
+ import argparse
5
+ from pathlib import Path
6
+ import sys
7
+
8
+ sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
9
+
10
+ # Windows consoles default to cp1252; descriptions can contain any Unicode.
11
+ if hasattr(sys.stdout, "reconfigure"):
12
+ sys.stdout.reconfigure(encoding="utf-8", errors="replace")
13
+ sys.stderr.reconfigure(encoding="utf-8", errors="replace")
14
+
15
+ from vision_client import ( # noqa: E402
16
+ VisionError,
17
+ describe_image,
18
+ image_path_to_data_url,
19
+ load_default_env,
20
+ )
21
+
22
+
23
+ def region_data_url(path, region):
24
+ try:
25
+ from PIL import Image
26
+ except ImportError:
27
+ raise VisionError("--region requires Pillow; install the optional dependency pillow first")
28
+ try:
29
+ x1, y1, x2, y2 = (int(v) for v in region.split(","))
30
+ except ValueError:
31
+ raise VisionError("--region expects four integers: X1,Y1,X2,Y2 (pixels)")
32
+ import base64
33
+ import io
34
+ try:
35
+ with Image.open(Path(path).expanduser()) as image:
36
+ width, height = image.size
37
+ box = (max(0, min(x1, x2)), max(0, min(y1, y2)),
38
+ min(width, max(x1, x2)), min(height, max(y1, y2)))
39
+ if box[2] <= box[0] or box[3] <= box[1]:
40
+ raise VisionError(f"--region {region} is empty after clamping to {width}x{height}")
41
+ buffer = io.BytesIO()
42
+ image.crop(box).save(buffer, format="PNG")
43
+ except (OSError, ValueError) as exc:
44
+ raise VisionError(f"Cannot read image: {path}") from exc
45
+ return "data:image/png;base64," + base64.b64encode(buffer.getvalue()).decode()
46
+
47
+
48
+ def build_prompt(args, count):
49
+ if args.ocr is not None:
50
+ extra = f" Additional requirements: {args.ocr}" if args.ocr else ""
51
+ scope = "these images" if count > 1 else "this image"
52
+ return (
53
+ f"Transcribe every piece of visible text in {scope} verbatim (titles, body text, labels, watermarks, etc.), "
54
+ "line by line, without omitting any characters. Do not rewrite, summarize, or translate the text, "
55
+ "and do not add any preamble, explanation, or extra content."
56
+ + (" Label each image's text with its ordinal (Image 1, Image 2, ...)." if count > 1 else "")
57
+ + extra
58
+ )
59
+ if args.query:
60
+ return args.query
61
+ if count > 1:
62
+ return ("Describe each image in detail (label them Image 1, Image 2, ...), "
63
+ "then point out the notable differences between them.")
64
+ return None
65
+
66
+
67
+ def main():
68
+ parser = argparse.ArgumentParser(
69
+ prog="glance",
70
+ description="Describe, answer questions about, or OCR an image with the configured vision model",
71
+ )
72
+ parser.add_argument("images", nargs="+", metavar="image",
73
+ help="path(s) to the image(s); pass several to compare them in one call")
74
+ group = parser.add_mutually_exclusive_group()
75
+ group.add_argument("-q", "--query", help="ask a question about the image(s)")
76
+ group.add_argument("--ocr", nargs="?", const="", metavar="EXTRA", help="transcribe all visible text verbatim")
77
+ parser.add_argument("--region", metavar="X1,Y1,X2,Y2",
78
+ help="crop to this pixel box (e.g. from ground) and send only the crop")
79
+ args = parser.parse_args()
80
+ load_default_env()
81
+ try:
82
+ if args.region and len(args.images) > 1:
83
+ raise VisionError("--region works with exactly one image")
84
+ urls = ([region_data_url(args.images[0], args.region)] if args.region
85
+ else [image_path_to_data_url(path) for path in args.images])
86
+ answer = describe_image(
87
+ urls,
88
+ build_prompt(args, len(urls)),
89
+ max_tokens=None, # no output cap; the vision model decides
90
+ apply_lang=args.ocr is None,
91
+ )
92
+ except VisionError as exc:
93
+ parser.exit(1, f"glance: {exc}\n")
94
+ print(answer)
95
+
96
+
97
+ if __name__ == "__main__":
98
+ main()
@@ -0,0 +1,18 @@
1
+ #!/usr/bin/env python3
2
+ """ground: locate objects in an image and print pixel bounding boxes."""
3
+
4
+ from pathlib import Path
5
+ import sys
6
+
7
+ sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
8
+
9
+ # Windows consoles default to cp1252; descriptions can contain any Unicode.
10
+ if hasattr(sys.stdout, "reconfigure"):
11
+ sys.stdout.reconfigure(encoding="utf-8", errors="replace")
12
+ sys.stderr.reconfigure(encoding="utf-8", errors="replace")
13
+
14
+ from ground import main # noqa: E402
15
+
16
+
17
+ if __name__ == "__main__":
18
+ main()
@@ -0,0 +1,134 @@
1
+ #!/usr/bin/env python3
2
+ """trace: deterministic image-to-SVG tracing for exact shape geometry.
3
+
4
+ Runs fully local (no vision API, no credentials): coordinates come from the
5
+ actual pixels, so use this instead of asking a vision model for shapes,
6
+ sizes, or outlines. Flat, high-contrast graphics only; text becomes curves.
7
+ """
8
+
9
+ import argparse
10
+ from pathlib import Path
11
+ import re
12
+ import sys
13
+ import tempfile
14
+
15
+ # Windows consoles default to cp1252; descriptions can contain any Unicode.
16
+ if hasattr(sys.stdout, "reconfigure"):
17
+ sys.stdout.reconfigure(encoding="utf-8", errors="replace")
18
+ sys.stderr.reconfigure(encoding="utf-8", errors="replace")
19
+
20
+ WHITE_FILLS = {"#ffffff", "#fff", "white"}
21
+
22
+ # Speckle filtering drops components under a fixed pixel area, so a 30px icon
23
+ # binarizes to nothing at 1x while the same icon enlarged traces cleanly.
24
+ # Enlarge anything smaller than this; full-size screenshots already clear it.
25
+ TARGET_MIN_SIDE = 256
26
+
27
+
28
+ def strip_background(svg):
29
+ """Drop the leading full-canvas white path vtracer emits for the background."""
30
+ match = re.search(r"<path [^>]*/>", svg)
31
+ if match:
32
+ fill = re.search(r'fill="([^"]+)"', match.group(0))
33
+ if fill and fill.group(1).strip().lower() in WHITE_FILLS:
34
+ return svg.replace(match.group(0), "", 1)
35
+ return svg
36
+
37
+
38
+ def truncate_decimals(svg, places=2):
39
+ return re.sub(r"-?\d+\.\d{3,}", lambda m: f"{float(m.group()):.{places}f}", svg)
40
+
41
+
42
+ def write_svg(path, svg):
43
+ """Write the exact UTF-8 payload whose byte count is reported to callers."""
44
+ payload = svg.encode("utf-8")
45
+ return Path(path).expanduser().write_bytes(payload)
46
+
47
+
48
+ def prepare_input(path, region, scale):
49
+ """Crop/upscale with Pillow when asked; return the path vtracer reads and the scale used.
50
+
51
+ `scale` may be None, meaning "pick one from the size": small shapes are
52
+ enlarged until they survive speckle filtering. Without Pillow we cannot
53
+ measure the image, so that choice falls back to no upscale.
54
+ """
55
+ try:
56
+ from PIL import Image
57
+ except ImportError:
58
+ if region or (scale or 1) != 1:
59
+ sys.exit("trace: --region/--scale require Pillow; install the optional dependency pillow first")
60
+ return str(path), 1
61
+ with Image.open(path) as image:
62
+ if region:
63
+ try:
64
+ x1, y1, x2, y2 = (int(v) for v in region.split(","))
65
+ except ValueError:
66
+ sys.exit("trace: --region expects four integers: X1,Y1,X2,Y2 (pixels)")
67
+ box = (max(0, min(x1, x2)), max(0, min(y1, y2)),
68
+ min(image.width, max(x1, x2)), min(image.height, max(y1, y2)))
69
+ if box[2] <= box[0] or box[3] <= box[1]:
70
+ sys.exit(f"trace: --region {region} is empty after clamping to {image.width}x{image.height}")
71
+ image = image.crop(box)
72
+ if scale is None:
73
+ shortest = max(min(image.width, image.height), 1)
74
+ scale = max(2 if region else 1, -(-TARGET_MIN_SIDE // shortest))
75
+ if not region and scale == 1:
76
+ return str(path), 1
77
+ if scale != 1:
78
+ image = image.resize((image.width * scale, image.height * scale), Image.LANCZOS)
79
+ handle = tempfile.NamedTemporaryFile(suffix=".png", delete=False)
80
+ image.save(handle.name)
81
+ return handle.name, scale
82
+
83
+
84
+ def main():
85
+ parser = argparse.ArgumentParser(
86
+ prog="trace",
87
+ description="Trace an image into SVG with exact pixel-derived geometry (local, deterministic)",
88
+ )
89
+ parser.add_argument("image", help="path to the image")
90
+ parser.add_argument("--region", metavar="X1,Y1,X2,Y2",
91
+ help="crop to this pixel box first (e.g. a box from ground)")
92
+ parser.add_argument("--scale", type=int, metavar="N",
93
+ help="upscale N times before tracing (default: enough to clear "
94
+ f"{TARGET_MIN_SIDE}px on the shorter side, so small icons survive)")
95
+ parser.add_argument("--polygon", action="store_true",
96
+ help="polygon mode for boxy diagrams/wireframes (default: spline for curved shapes)")
97
+ parser.add_argument("--color", action="store_true",
98
+ help="keep colors (default: black-and-white, which is far more compact)")
99
+ parser.add_argument("-o", "--output", help="write the SVG here instead of stdout")
100
+ args = parser.parse_args()
101
+ try:
102
+ import vtracer
103
+ except ImportError:
104
+ parser.exit(1, "trace: requires vtracer; install the optional dependency vtracer first\n")
105
+ source = Path(args.image).expanduser()
106
+ if not source.is_file():
107
+ parser.exit(1, f"trace: Image not found: {source}\n")
108
+ work, scale = prepare_input(source, args.region, args.scale)
109
+ with tempfile.NamedTemporaryFile(suffix=".svg", delete=False) as out:
110
+ vtracer.convert_image_to_svg_py(
111
+ work, out.name,
112
+ colormode="color" if args.color else "binary",
113
+ filter_speckle=8, corner_threshold=40,
114
+ mode="polygon" if args.polygon else "spline",
115
+ )
116
+ svg = Path(out.name).read_text()
117
+ svg = truncate_decimals(strip_background(svg))
118
+ paths = svg.count("<path")
119
+ if not paths:
120
+ # This is where an agent decides the tool cannot see the shape and goes
121
+ # back to guessing. Name the recoveries instead, cheapest first.
122
+ print("trace: 0 paths - nothing survived binarization. Try a larger --scale, "
123
+ "--region to crop closer to the shape, or pre-invert a light-on-dark image. "
124
+ "--color is a last resort: on an anti-aliased image it splits every gray "
125
+ "level into its own path.", file=sys.stderr)
126
+ if args.output:
127
+ bytes_written = write_svg(args.output, svg)
128
+ print(f"wrote {args.output} ({bytes_written} bytes, {paths} paths, traced at {scale}x)")
129
+ else:
130
+ print(svg)
131
+
132
+
133
+ if __name__ == "__main__":
134
+ main()