@anionex/dsh-vision-toolkit 0.1.32 → 0.1.33

package/README.i18n.yaml

@@ -2,5 +2,5 @@
 # side as of the last confirmed-consistent state. Both languages carry equal authority;
 # after editing either side, bring the other along and re-record with:
 #   pnpm run verify-translation-pairing --write dsh-vision-toolkit/README.md
-README.md: f590f56c6253c25a8066445419c1ec490936b8e8
-README.zh.md: 7896e4d8121c59bfd7e0c03942c9d1936716bd06
+README.md: 5083c3bc9c77ab79eed171c5fbdeb428df274f77
+README.zh.md: 979935b0fad2fcdeccf290372dcb5b12176a2ecf

package/README.md

@@ -16,33 +16,27 @@
 [![MIT](https://img.shields.io/badge/license-MIT-0B7285?style=flat-square)](LICENSE)
 [![DSH](https://img.shields.io/badge/DSH-Web%20%2B%20Headless-5B4CF0?style=flat-square)](cordis.patch.yml)
 
-
 **A more powerful vision toolkit—give text-only models in DeepSeek Harness eyes: image Q&A, long-screenshot OCR, UI restoration, and GUI visual tasks in one toolkit and Skill.**
 
 🚀 Paste an image and ask directly | Install with one command | Built-in free vision | Broad use cases
 
-<p align="center">
-  <a href="#highlights">Highlights</a> | <a href="#quick-start-three-steps">Quick start</a> | <a href="#common-workflows">Common workflows</a> | <a href="#toolbox">Toolbox</a> | <a href="#configuration-and-limits">Configuration</a> | <a href="#troubleshooting">Troubleshooting</a> | <a href="#development-and-community">Community</a>
-</p>
+[Highlights](#highlights) | [Quick start](#quick-start-three-steps) | [Toolbox](#toolbox) | [Configuration and limits](#configuration-and-limits) | [Troubleshooting](#troubleshooting) | [Community](#development-and-community)
 
 🌐 **English** | [中文](README.zh.md)
 
 </div>
 
-If you use DeepSeek or another text-only model in DeepSeek Harness (DSH), you may have run into the same problems: the model cannot see a screenshot, generic descriptions miss the point, buttons have no usable coordinates, and a rebuilt page can look “close enough” without a way to measure the remaining difference.
-
 🏆 This project is the first comprehensive vision-tool plugin in the DeepSeek Harness ecosystem: it was initiated before internal beta and built during the beta with reference to [`agent-vision-toolkit`](https://github.com/Anionex/agent-vision-toolkit).
 
 > **Original work:** The system and division of responsibilities behind these visual tools, together with the `vision-skills` Skill, were personally created and continuously refined by the author through long-term real-world use and repeated iteration.
 
 ## Highlights
 
-- **Paste and use it immediately.** Paste an image in DSH Web and the text-only route switches to its `(Vision Toolkit)` variant without manual path copying or model changes.
-- **A seamless image workflow.** Native thumbnails, session history, and workspace paths stay intact; Web can preview artifacts and Headless can continue using the same structured results.
+- **Paste an image and ask directly.** In DSH Web, pasting an image switches the text-only model to its `(Vision Toolkit)` variant automatically — no manual path copying or model changes. Native thumbnails, session history, and workspace paths stay intact; Web can preview artifacts.
 - **One command to install.** The built-in free Gemini 3.7 Flash vision service is ready after installation, with no API key required.
-- **Built-in free vision.** The shared service works immediately after installation with a quota of **300 images per machine per day**.
-- **Vision guided by intent.** The agent extracts evidence for the task at hand, such as “Where is the error?” or “Where is the button?”, instead of returning a generic caption.
-- **A complete screenshot-to-verification loop.** Reference images, HTML screenshots, difference regions, and pixel comparison work together for UI restoration.
+- **Built-in free vision quota.** The shared service works immediately after installation with a quota of **100 images per machine per day**.
+- **Not just a caption — the content that matters.** The model does not produce a generic description; it extracts evidence around the current task, such as “Where is the error?” or “Where is the button?”.
+- **A battle-tested visual-task methodology.** The bundled Skill tells the agent what to look at for different visual tasks, which tool to choose, how to proceed, and how to verify the result.
 
 [`agent-vision-toolkit`](https://github.com/Anionex/agent-vision-toolkit) gives an agent more than image captions: it can read, locate, crop, trace, rebuild, and verify visual work. DSH Vision Toolkit is its native DeepSeek Harness integration, bringing that workflow into Web and Headless Profiles.
 
@@ -51,7 +45,7 @@ This project has two layers:
 1. **Visual tools and a Skill:** the agent learns when to inspect, ground, OCR, crop, trace, or compare pixels.
 2. **Native DSH integration:** those capabilities live inside Profiles, sessions, Settings, Artifacts, and the Web UI, with a free Gemini 3.7 Flash vision service ready after installation.
 
-> **Install and use it immediately.** The default setup includes a free Gemini 3.7 Flash vision service and requires no API key. Cropping, pixel diffing, color analysis, foreground extraction, SVG tracing, and HTML screenshots run locally without spending vision API requests.
+> **Install and use it immediately.** The default setup includes a free Gemini 3.7 Flash vision service and requires no API key.
 
 ```sh
 dsh plugin --profile web add @anionex/dsh-vision-toolkit
@@ -59,22 +53,18 @@ dsh plugin --profile web add @anionex/dsh-vision-toolkit
 
 **Upstream toolkit:** [Anionex/agent-vision-toolkit](https://github.com/Anionex/agent-vision-toolkit) · **Project website:** [agent-vision.anionex.me](https://agent-vision.anionex.me)
 
-<details>
-<summary><strong>Table of contents</strong></summary>
+**Contents**
 
 - [Highlights](#highlights)
 - [Recent updates](#recent-updates)
 - [Who it is for](#who-it-is-for)
 - [See it in action](#see-it-in-action)
 - [Quick start: three steps](#quick-start-three-steps)
-- [Common workflows](#common-workflows)
 - [Toolbox](#toolbox)
 - [Configuration and limits](#configuration-and-limits)
 - [Troubleshooting](#troubleshooting)
 - [Development and community](#development-and-community)
 
-</details>
-
 ## Recent updates
 
 - **2026-08-16 · Windows Python:** Added Microsoft Store Python support, fixing first-time isolated-runtime setup failures for affected Windows users.
@@ -86,14 +76,18 @@ dsh plugin --profile web add @anionex/dsh-vision-toolkit
 
 ## Who it is for
 
-| The problem | What Vision Toolkit delivers |
-|---|---|
-| **A text-only model cannot see a screenshot** | Paste an image in DSH Web; the plugin obtains visual evidence and returns the task-relevant parts to the text model |
-| **The description is long but misses the point** | Ask “Where is the error?” or “What color is the submit button?” and receive an answer focused on that question |
-| **The model knows an element exists but cannot act on it** | Get original-image pixel coordinates and an optional labeled or numbered preview |
-| **Long-screenshot OCR skips or duplicates lines** | Split and audit the image while preserving Markdown, chunks, manifests, and resumable run state |
-| **UI restoration is judged by feel** | Compare the reference and implementation screenshots to get a difference percentage, ranked regions, a heatmap, and JSON |
-| **Screenshot assets cannot be reused** | Produce a crop, transparent PNG, color palette, or editable SVG instead of stopping at prose |
+1. Want an interaction experience similar to a multimodal model: paste an image directly and ask a question or make a request.
+2. Want more than image Q&A — complete more complex, high-value visual tasks such as turning a sketch into a front-end page, converting an image into HTML, or extracting chat messages from long screenshots; more scenarios are added over time.
+
+The bundled `vision-skills` Skill carries the complete upstream playbooks, explaining when to use each workflow, in what order to call the tools, and how to verify the result:
+
+| Playbook | What the agent learns to do |
+| --- | --- |
+| [Read long screenshots, chat histories, and scrolling pages](assets/skill/references/long-screenshot-ocr.md) | Find low-content cut bands, OCR each chunk in order, preserve chat speakers/timestamps/quotes, merge only duplicated overlap, and surface risky boundaries for verification |
+| [Rebuild a UI from a screenshot or design](assets/skill/references/restore-ui.md) | Reuse project components and assets first, then combine code-native UI, extracted visuals, rendered screenshots, and visual comparison to align a page or component |
+| [Restore an icon, logo, illustration, or other graphic](assets/skill/references/restore-graphic.md) | Extract a transparent PNG from the source image, or rebuild an editable/scalable SVG when needed, then verify shape, color, and alpha edges |
+| [Turn a sketch, diagram, or whiteboard into structured code](assets/skill/references/restore-structure.md) | Recover nodes, labels, connections, and directions as editable Mermaid, Graphviz, or another structured representation |
+| [Operate a GUI from screenshots](assets/skill/references/gui.md) | Locate a control, perform one action, capture the screen again, and verify the resulting state before continuing |
 
 ## See it in action
 
@@ -112,6 +106,8 @@ dsh plugin --profile web add @anionex/dsh-vision-toolkit
   <img src="assets/upstream/infographic-result.webp" width="49%" alt="Editable HTML and CSS reconstruction created from the reference screenshot" />
 </p>
 
+> Prompt example: “(Use vision-skills) Rebuild this image into HTML.”
+
 *Left: the reference screenshot. Right: an editable HTML/CSS result. The result can continue into screenshot rendering and pixel comparison instead of ending as an image description.*
 
 ### Sketch to working interface
@@ -123,15 +119,19 @@ dsh plugin --profile web add @anionex/dsh-vision-toolkit
 
 *Left: a hand-drawn reference. Right: the working interface reconstructed from it.*
 
-### Turn “looks close” into a verifiable result
+> Prompt example: “(Use vision-skills) Turn this sketch into a working front-end page.”
 
-The repository includes a reproducible UI-restoration example: the agent renders the reference and implementation, then uses difference regions, a heatmap, and a JSON report to guide the next correction.
+### Fast UI restoration: an approximate first pass
 
-<p>
-  <img src="examples/ui-restoration/assets/initial.png" width="49%" alt="Initial UI implementation with measurable layout and styling differences" />
-  <img src="examples/ui-restoration/assets/implementation.png" width="49%" alt="UI implementation after visual diagnosis and pixel comparison" />
+<p align="center">
+  <img src="assets/upstream/ui-fast-restore-reference.webp" width="49%" alt="Original YouMind homepage used as the fast UI restoration reference" />
+  <img src="assets/upstream/ui-fast-restore-result.webp" width="49%" alt="Approximate YouMind homepage produced with fast UI restoration mode" />
 </p>
 
+> Prompt example: “(Use vision-skills) Quickly rebuild this image into HTML.”
+
+*Left: the original page. Right: a fast reconstruction that preserves the main layout, content, and visual hierarchy while allowing approximate colors and library icons. Fast mode targets a first screenshot in about three minutes.*
+
 ## Quick start: three steps
 
 ### 1. Install
@@ -163,23 +163,12 @@ Crop this icon and convert it to SVG.
 Rebuild the page from reference.png. After each pass, render it and run a pixel diff until the major differences are gone.
 ```
 
-## Common workflows
-
-| Task | Recommended workflow |
-|---|---|
-| Image Q&A or screenshot debugging | Inspect → answer around the current question → locate details when needed |
-| Find a button, icon, or text region | Ground the target → return pixel box → create a labeled preview |
-| Extract an icon from a screenshot | Ground → crop → trace to SVG |
-| Read a long webpage screenshot | Split → OCR → merge Markdown → audit boundaries |
-| Recreate a page or component | Reference → implementation → HTML screenshot → pixel diff → iterate |
-| Extract brand visuals | Crop region → analyze dominant colors → extract foreground → export transparent PNG |
-
 ## Toolbox
 
 The plugin provides 10 tools that can be called independently or composed into a workflow:
 
 | Tool | Best question to ask | Main result |
-|---|---|---|
+| --- | --- | --- |
 | `vision_glance` | “What is happening in this image?” | Focused answer, description, OCR, or multi-image comparison |
 | `vision_ground` | “Where is the thing I need?” | Original pixel coordinates and optional boxed preview |
 | `vision_detect` | “Which buttons, icons, or elements are present?” | Numbered element inventory, coordinates, and optional preview |
@@ -197,10 +186,18 @@ For a long HTML document, pass `fullPage=true`. The requested width and height r
 
 ## How it works
 
-The plugin keeps image understanding and deterministic local image processing in one Agent workflow. Expand the flow below for the implementation boundary.
+The plugin keeps image understanding and deterministic local image processing in one Agent workflow. The diagram below shows the implementation boundary.
+
+### Descriptions that keep the task in view
+
+Most vision bridges for text-only models ask a multimodal model for a generic description and hand it to the text model, adding a semantic layer where information is lost. Vision Toolkit instead recovers **why the agent wants to look at the image**: the user message or the model's stated reason becomes a focus hint passed to the vision model. The result is a task-aware description that emphasizes what matters for the current step — with fewer tokens, higher accuracy, and faster responses.
+
+<p align="center">
+  <img src="assets/upstream/focus-hint-comparison-1.webp" width="49%" alt="Generic image descriptions compared with task-aware vision using a focus hint - part 1" />
+  <img src="assets/upstream/focus-hint-comparison-2.webp" width="49%" alt="Generic image descriptions compared with task-aware vision using a focus hint - part 2" />
+</p>
 
-<details>
-<summary><strong>Architecture and image-input behavior</strong></summary>
+**Architecture and image-input behavior**
 
 ```mermaid
 flowchart LR
@@ -226,8 +223,6 @@ recorded in `assets/skill/UPSTREAM.json` and `patches/vision-tools-dsh.patch`.
 
 For routes that DSH positively identifies as text-only, the plugin registers a sibling `<model> (Vision Toolkit)` variant. By default, pasting an image in DSH Web switches to that variant and gives the model both a reusable workspace path and a visual description focused on the current task.
 
-</details>
-
 ## Configuration and limits
 
 ### Built-in free service
@@ -245,8 +240,8 @@ Requests that still use the previous `qwen/qwen3.6-27b` model name remain compat
 This is a shared zero-configuration entry point, not an unlimited private endpoint. Request safeguards include:
 
 | Limit | Current value |
-|---|---:|
-| Daily quota | 300 images per machine per day |
+| --- | --- |
+| Daily quota | 100 images per machine per day |
 | Images per request | Up to 5 |
 | Image size | 4 MiB per image |
 | Decoded pixels | 20,000,000 per image |
@@ -278,119 +273,17 @@ OpenAI Chat Completions-compatible endpoints and Anthropic Messages are supporte
 
 For a trusted internal endpoint that uses a self-signed certificate or MITM proxy, start the DSH process with `VISION_SSL_VERIFY=0`. The plugin forwards that value to the isolated Python runtime; certificate verification remains enabled when the variable is unset or has any other value. The false values `false`, `off`, `no`, `none`, and `disabled` are also accepted, case-insensitively.
 
-### Requirements
-
-- A DeepSeek Harness Web or Headless Profile.
-- Node.js `^22.19.0` or `>=24.0.0`.
-- Python 3.11+ is usually not needed in advance: the plugin prefers a system Python and otherwise downloads a pinned standalone Python 3.13 automatically, preparing its own isolated environment. Only that first automatic download needs network access.
-- Only `vision_html_screenshot` requires Chrome, Chromium, or Edge.
-- Inputs must be PNG, JPEG, GIF, or WebP files in the session workspace, the platform temporary directory, or an explicitly allowed directory.
-
 ### Configure the Python runtime
 
-By default the plugin picks a system Python 3.11+, or downloads a standalone Python when none is found; most users never need to configure this section. The rest is for advanced setups where automatic discovery fails, a specific interpreter is required, or an external runtime is used.
-
-The packaged `managed` runtime creates its own isolated virtual environment. `runtime.python` selects the Python executable used to bootstrap or refresh that environment; it does not replace the managed environment with the interpreter's global site-packages. Set it when automatic discovery fails or when several Python installations exist. The override is also used by `runtime.mode: external`.
-
-Python 3.11 or newer is required; the automatically downloaded standalone Python is 3.13.15 and, like a system interpreter, is only used to bootstrap the isolated environment. Without an override, the plugin tries `python3` then `python` on macOS/Linux, and `python`, `py -3`, then `python3` on Windows, before falling back to the standalone download. A configured value is passed as one executable name or path, not as a shell command with arguments, so use `py` (not `py -3`) for the Windows launcher; use an absolute path when you need a specific version.
+Most users never need to configure the Python runtime: the plugin prefers a system Python 3.11+ and otherwise downloads a pinned standalone Python automatically.
 
-Configure it in the Profile patch:
-
-```yaml
-- id: vision-toolkit
-  config:
-    runtime:
-      # macOS/Linux system Python
-      python: python3
-      # Or a project-local environment:
-      # python: /absolute/path/to/project/.venv/bin/python
-      # Windows venv (forward slashes also work in YAML):
-      # python: C:/Users/you/project/.venv/Scripts/python.exe
-      # Windows launcher, when its default Python is 3.11+:
-      # python: py
-```
-
-For a managed runtime, create the project-local interpreter and point `runtime.python` at it. The plugin installs the locked dependencies into its own managed cache, so installing the lockfile into this bootstrap environment is optional:
-
-```sh
-python3 --version                         # must report 3.11 or newer
-uv venv .venv --python 3.13
-```
-
-For `runtime.mode: external`, install the locked dependencies using the `runtime/requirements.lock` from the **DSH Vision Toolkit plugin** checkout, then point `runtime.agentVisionToolkitPath` at a separate exact `agent-vision-toolkit` snapshot. The packaged `vendor/agent-vision-toolkit` directory is such a snapshot when it has not been modified:
-
-```sh
-uv pip install --python .venv/bin/python \
-  -r /absolute/path/to/dsh-vision-toolkit/runtime/requirements.lock
-```
-
-```yaml
-- id: vision-toolkit
-  config:
-    runtime:
-      mode: external
-      python: /absolute/path/to/dsh-vision-toolkit/.venv/bin/python
-      agentVisionToolkitPath: /absolute/path/to/dsh-vision-toolkit/vendor/agent-vision-toolkit
-```
-
-On Windows, use `py -3 --version` for the version check and `.venv\Scripts\python.exe` plus `runtime\requirements.lock` in the corresponding commands:
-
-```powershell
-py -3 --version                         # must report 3.11 or newer
-uv venv .venv --python 3.13
-# External mode only; use the plugin checkout's absolute lockfile path:
-uv pip install --python .venv\Scripts\python.exe -r C:\absolute\path\to\dsh-vision-toolkit\runtime\requirements.lock
-```
-
-Point `runtime.python` at the same interpreter, save the Profile patch, and restart the Web Profile. Then open **Settings → Vision Toolkit**: the Runtime panel should show the resolved interpreter and Python version, and **Run health check** plus **Test vision model** should complete without the Python-version error. A final smoke test is to place a PNG/JPEG in the session workspace and call `vision_glance`.
-
-The path fence automatically allows the session workspace and the platform temporary directory. On macOS/Linux the temporary root is `/tmp`. On Windows it is `TEMP`, then `TMP`, with the operating-system fallback if neither is set; model-generated `/tmp/...` paths are translated to that Windows directory before the normal realpath fence runs. No `allowedDirs` entry is needed for these platform temporary paths.
-
-Use `allowedDirs` only for additional trusted input roots outside the workspace and platform temporary directory:
-
-```yaml
-- id: vision-toolkit
-  config:
-    allowedDirs:
-      # macOS/Linux example
-      - /srv/vision-inputs
-      # Windows example (use this instead on Windows)
-      # - D:/vision-inputs
-```
-
-`allowedDirs` is an input allowlist, not the managed runtime cache. The managed runtime keeps its own files under `$DSH_HOME/cache/dsh-vision-toolkit` (or `~/.dsh/cache/dsh-vision-toolkit` when `DSH_HOME` is unset); that directory does not need to be added. Environment-variable forms such as `$env:TEMP` and `%TEMP%` are not expanded inside `allowedDirs`, so configure extra roots with real absolute paths.
-
-<details>
-<summary><strong>Install, upgrade, disable, and uninstall</strong></summary>
-
-```sh
-dsh plugin --profile web update @anionex/dsh-vision-toolkit
-dsh plugin --profile web remove @anionex/dsh-vision-toolkit
-```
-
-If you are migrating from the retired `@dsh-external/dsh-vision-toolkit`, remove the old package first and install `@anionex/dsh-vision-toolkit`.
-
-To disable the bundle temporarily, set this in the Profile patch:
-
-```yaml
-- id: vision-toolkit
-  disabled: true
-```
-
-Restart the Web Profile and refresh the page after enabling or upgrading the Web plugin.
-
-</details>
-
-### Plugin updates
-
-In **Settings → Vision Toolkit**, **Check for updates** queries the Profile's npm registry. For a direct registry installation, **Update and restart** installs only the exact version you confirmed, verifies it, and restarts an explicitly opted-in POSIX Web process on a fixed `--port`. Local/workspace/file/git/URL installs, Windows, dynamic ports, read-only Profiles, and manager-owned processes remain check-only.
-
-The updater revalidates the Profile before mutation, snapshots the original manifest and lockfile, and holds a token-owned cross-process lock. The current Web process exits only after the restart helper confirms that the backup is readable and the lock handoff succeeded. When the Profile was already operational, the replacement must report both the target plugin version and a ready runtime; failed replacements restore the original manifest/lockfile and rebuild dependencies with a frozen lockfile before retrying the previous exact version. If automatic recovery itself fails, the backup and lock are preserved and their paths are written to `$DSH_HOME/logs/vision-toolkit-restart.log`. Detached restart requires `DSH_VISION_TOOLKIT_ALLOW_DETACHED_RESTART=1`; unsaved Settings or API-key input blocks installation.
+For advanced setups — overriding `runtime.python`, using `runtime.mode: external`, verifying the runtime, or allowing additional input directories — see [Python runtime configuration](docs/python-runtime.md).
 
 ## Troubleshooting
 
 | Problem | What to do |
-|---|---|
+| --- | --- |
+| The vision-model test fails with `Vision API returned an incompatible response structure` | The base URL usually needs a path prefix. Local OpenAI-compatible services such as LM Studio and Ollama should be entered as `http://127.0.0.1:1234/v1` (include `/v1`); the plugin appends `/chat/completions`, and a port-only address hits an unknown endpoint and returns this error |
 | Pasting an image still says the model does not support image input | Restart the Web Profile, refresh the page, and confirm the selected route has the `(Vision Toolkit)` suffix. You can also place the image in the session workspace and invoke `/vision-skills` |
 | The free service returns 429 | Wait for the `Retry-After` interval, or switch to your own endpoint when you need stable higher volume |
 | The image exceeds a size or pixel limit | Crop or resize it first; the error identifies whether bytes or decoded pixels caused the rejection |
@@ -399,9 +292,11 @@ The updater revalidates the Profile before mutation, snapshots the original mani
 | Chrome is not found | Install Chrome, Chromium, or Edge. Only HTML screenshot rendering is unavailable; the other tools still work |
 | An artifact cannot be previewed | Use **Open file** or the workspace path in the result. Preview URLs exist only while the Web route is available |
 
-## Project status and limitations
+## FAQ
+
+**Will adding a vision model significantly increase costs?**
 
-The current release focuses on screenshot understanding, visual grounding, OCR, asset extraction, UI restoration, and pixel-level verification. It is not a video, audio, or camera-input system and does not automatically click GUI controls. Interactive box editing, remote service clusters, model voting, and cross-session visual caches are also outside the current scope.
+No. Each inspection sends only the necessary intent and the image to the multimodal model, and context does not accumulate across calls, so the added cost stays small. To reduce it further, a locally deployed small multimodal side model (for example the Gemma 4 or Qwen 3.5/3.6 series) can provide the vision capability.
 
 ## Development and community
 
@@ -415,7 +310,7 @@ The current release focuses on screenshot understanding, visual grounding, OCR,
   <img src="assets/community-group-qr.png" alt="QR code for the agent-vision-toolkit community group" width="240" />
 </p>
 
-I'm <a href="https://anionex.me/">anionex</a>, an AI-native developer who once ranked <strong>No. 3</strong> on GitHub's global developer trending list, with more than 16k stars across my projects. If you would like to follow my future work, <a href="https://github.com/Anionex">follow me on GitHub</a>.
+I'm [anionex](https://anionex.me/), an AI-native developer who once ranked **No. 3** on GitHub's global developer trending list, with more than 16k stars across my projects. If you would like to follow my future work, [follow me on GitHub](https://github.com/Anionex).
 
 [`agent-vision-toolkit`](https://github.com/Anionex/agent-vision-toolkit) was created by [Anionex](https://anionex.me/). This repository maintains its native DeepSeek Harness integration.