@bismawy/pi-vision-watcher 1.0.8 → 1.0.10

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (3) hide show
  1. package/README.md +140 -143
  2. package/package.json +2 -2
  3. package/src/index.ts +9 -5
package/README.md CHANGED
@@ -1,143 +1,140 @@
1
- # 👁ī¸ pi-vision-watcher
2
-
3
- **Give text-only [pi](https://github.com/earendil-works/pi-coding-agent) models vision**
4
-
5
- Describe images using an authenticated vision model of your choice, then seamlessly hand off the text descriptions to non-vision models.
6
-
7
- [![pi extension](https://img.shields.io/badge/pi-extension-blueviolet)](https://github.com/earendil-works/pi-coding-agent)
8
- [![npm](https://img.shields.io/npm/v/@bismawy/pi-vision-watcher)](https://www.npmjs.com/package/@bismawy/pi-vision-watcher)
9
- [![license](https://img.shields.io/badge/license-MIT-blue)](./LICENSE)
10
-
11
- ---
12
-
13
- ## The Problem
14
-
15
- Some of the best coding models are text-only. When you attach a screenshot, diagram, or UI mock, they either ignore it or fail the request entirely. Switching models just to read an image interrupts your workflow.
16
-
17
- ## The Solution
18
-
19
- `pi-vision-watcher` bridges this gap automatically:
20
- - **Interactive Picker:** Pick any vision model from your authenticated providers with `/vision-watcher`.
21
- - **Automatic Handoff:** Whenever a non-vision model receives an image (via paste, attachment, or the `read` tool), the image is described behind the scenes and swapped for rich descriptive text before reaching the model.
22
- - **Batched & Cached:** Uses a DataLoader pattern so multiple images in a turn coalesce into a **single batched vision request**, cached by SHA-256 hash.
23
-
24
- ---
25
-
26
- ## ✨ Features
27
-
28
- - đŸŽ¯ **Connected-Only Model Picker** — `/vision-watcher` filters out unconfigured providers, showing only models you actually have credentials for (`/login`, `models.json`, or environment variables). Vision-capable models are highlighted with 👁ī¸.
29
- - ⚡ **DataLoader Batching** — Multiple images from parallel `read` calls or multi-image attachments merge into ONE batched vision call during the tool-result phase, eliminating latency bottlenecks.
30
- - 🧠 **Thinking & Reasoning Support** — Configure reasoning effort (`/vision-watcher thinking <level>`) for reasoning-capable vision models (e.g. OpenAI o-series, Claude, DeepSeek).
31
- - 🔄 **Fallback Chains** — Specify backup vision models that automatically take over if your primary describer is unavailable or encounters rate limits.
32
- - 🚀 **Paste-Time Prewarm (Opt-in)** — Describe pasted images the moment the path lands in the prompt editor before you even press Enter.
33
- - đŸ“Ŧ **Async Clipboard Fallback (Opt-in)** — Races direct reads against asynchronous description delivery to prevent stalling.
34
- - 💾 **LRU Hash Caching** — Prevents duplicate calls for identical images across conversation turns.
35
- - 🛡ī¸ **Graceful Degradation** — Never crashes your agent turn. If a description fails, a clean `[Image: description unavailable]` placeholder is provided and logged to `~/.pi/agent/logs/pi-vision-watcher/errors.log`.
36
-
37
- ---
38
-
39
- ## đŸ“Ļ Install
40
-
41
- ```bash
42
- pi install npm:@bismawy/pi-vision-watcher
43
- ```
44
-
45
- Alternatively, install directly from GitHub:
46
-
47
- ```bash
48
- pi install git:github.com/bismawy/pi-vision-watcher
49
- ```
50
-
51
- Then run `/reload` in Pi (or restart Pi).
52
-
53
- ---
54
-
55
- ## 🎮 Usage
56
-
57
- ### Interactive Commands
58
-
59
- | Command | Description |
60
- |---|---|
61
- | `/vision-watcher` | Open the interactive TUI picker to select your vision model |
62
- | `/vision-watcher model <provider/id>` | Set the vision describer model directly |
63
- | `/vision-watcher status` | View active configuration and handoff status |
64
- | `/vision-watcher enable` / `disable` | Toggle extension on or off |
65
- | `/vision-watcher auto on` / `off` | Toggle automatic handoff for all non-vision models |
66
- | `/vision-watcher add <provider/id>` | Force handoff for a specific model (e.g. weak vision models) |
67
- | `/vision-watcher remove <provider/id>` | Remove model from forced handoff list |
68
- | `/vision-watcher thinking <level>` | Set describer thinking level (`off`, `minimal`, `low`, `medium`, `high`, `xhigh`, `max`) |
69
- | `/vision-watcher timeout <ms>` | Set the base per-image description timeout in ms (default 45000) |
70
- | `/vision-watcher prewarm on` / `off` | Enable paste-time prewarming in TUI editor |
71
- | `/vision-watcher fallback on` / `off` | Enable async pasted-path description injection |
72
- | `/vision-watcher clear` | Clear configured vision model |
73
- | `/vision-watcher help` | Display full command reference |
74
-
75
- ---
76
-
77
- ## ⚙ī¸ Configuration
78
-
79
- Configuration is stored at `~/.pi/agent/extensions/pi-vision-watcher.json`:
80
-
81
- ```json
82
- {
83
- "enabled": true,
84
- "visionModel": "openai/gpt-4o",
85
- "fallbackModels": [],
86
- "autoHandoff": true,
87
- "handoffModels": [],
88
- "thinking": false,
89
- "thinkingLevel": "medium",
90
- "describeTimeoutMs": 45000,
91
- "prewarmPastedImages": false,
92
- "asyncClipboardHandoff": false,
93
- "maxTokens": null,
94
- "cacheMax": 50,
95
- "maxDescriptionLines": 0
96
- }
97
- ```
98
-
99
- | Field | Default | Description |
100
- |---|---|---|
101
- | `enabled` | `true` | Master switch for vision handoff. |
102
- | `visionModel` | `null` | Primary describer as `provider/id` (`null` = handoff inactive). |
103
- | `fallbackModels` | `[]` | List of fallback `provider/id` models tried in order if primary fails. |
104
- | `autoHandoff` | `true` | Automatically apply handoff to all models lacking native vision. |
105
- | `handoffModels` | `[]` | Additional models forced to receive handoff even if vision-capable. |
106
- | `thinking` / `thinkingLevel` | `false` / `"medium"` | Reasoning effort for vision models that support thinking. |
107
- | `describeTimeoutMs` | `45000` | Base per-image timeout in ms before failing over to fallback models. |
108
- | `prewarmPastedImages` | `false` | Describe images immediately upon pasting into the prompt. |
109
- | `asyncClipboardHandoff` | `false` | Asynchronous injection fallback for pasted image paths. |
110
- | `maxTokens` | `null` | Output token cap for descriptions (`null` = model default). |
111
- | `cacheMax` | `50` | Maximum number of described images cached per session. |
112
- | `maxDescriptionLines` | `0` | Truncate description lines (`0` = unbounded). |
113
-
114
- ---
115
-
116
- ## 🔍 Troubleshooting & Logs
117
-
118
- - **Error Logs:** Detailed failure traces, timestamps, and config snapshots are recorded in `~/.pi/agent/logs/pi-vision-watcher/errors.log`.
119
- - **Transient Retries:** Failed descriptions are never cached permanently — the next turn automatically re-attempts description generation.
120
-
121
- ---
122
-
123
- ## 🛠ī¸ Development
124
-
125
- ```bash
126
- pnpm install
127
- pnpm test # Run Vitest unit tests
128
- pnpm typecheck # Run TypeScript type check
129
- ```
130
-
131
- ---
132
-
133
- ## 📜 Credits & License
134
-
135
- `pi-vision-watcher` is inspired by and forked from [`pi-vision-handoff`](https://github.com/monotykamary/pi-vision-handoff) by [Tom X Nguyen](https://github.com/monotykamary) (originating from the concept in `pi-umans-provider`).
136
-
137
- **Key Enhancements:**
138
- - Filters picker to only authenticated/connected models.
139
- - Added thinking & reasoning controls for modern reasoning vision models.
140
- - Multi-model fallback chain support.
141
- - Streamlined settings and UI badging.
142
-
143
- Released under the [MIT License](./LICENSE).
1
+ # 👁ī¸ @bismawy/pi-vision-watcher
2
+
3
+ **Give text-only [pi](https://github.com/earendil-works/pi-coding-agent) models vision capabilities.**
4
+
5
+ Seamlessly inspect, describe, and convert visual inputs (screenshots, mockups, terminal errors, clipboard pastes) into structured descriptions using your preferred vision model, and hand them off to text-only coding models without interrupting your workflow.
6
+
7
+ [![pi extension](https://img.shields.io/badge/pi-extension-blueviolet)](https://github.com/earendil-works/pi-coding-agent)
8
+ [![npm](https://img.shields.io/npm/v/@bismawy/pi-vision-watcher)](https://www.npmjs.com/package/@bismawy/pi-vision-watcher)
9
+ [![license](https://img.shields.io/badge/license-MIT-blue)](./LICENSE)
10
+
11
+ ![pi-vision-watcher](https://raw.githubusercontent.com/bismawy/pi-vision-watcher/main/assets/screenshot.webp)
12
+
13
+ ---
14
+
15
+ ## ⚡ Quick Start
16
+
17
+ ### 1. Installation
18
+ ```bash
19
+ pi install npm:@bismawy/pi-vision-watcher
20
+ ```
21
+ *(Or install directly from Git: `pi install git:github.com/bismawy/pi-vision-watcher`)*
22
+
23
+ ### 2. Select Vision Model
24
+ Open the interactive TUI selector to choose your vision describer model from your connected providers:
25
+ ```bash
26
+ /vision-watcher
27
+ ```
28
+ *(You can also set it directly: `/vision-watcher model openai/gpt-4o`)*
29
+
30
+ ### 3. Work Seamlessly
31
+ Switch to any text-only model in Pi (e.g. DeepSeek, Claude text-only, local models). Whenever you paste an image, attach a file, or the agent runs `read` on an image, `pi-vision-watcher` describes it automatically in the background.
32
+
33
+ ---
34
+
35
+ ## 🚀 Key Capabilities
36
+
37
+ - đŸŽ¯ **Connected-Only Interactive Picker:** Shows only vision-capable models from providers where you actually have active credentials (`/login`, `models.json`, or environment variables).
38
+ - ⚡ **DataLoader Batching & SHA-256 Cache:** Automatically groups multiple images across parallel tool calls or multi-file prompts into a single batched describer request. Cached images are never re-described.
39
+ - 🛡ī¸ **Proactive False-Vision Healing:** Aggregator providers often mistakenly flag models (like DeepSeek V4) as multimodal, causing HTTP 400 errors (`This model does not support image`). `pi-vision-watcher` proactively forces handoff for these models and auto-heals `models.json` `modelOverrides` in-process.
40
+ - 🧠 **Thinking & Reasoning Controls:** Adjust reasoning levels (`off`, `minimal`, `low`, `medium`, `high`, `xhigh`, `max`) for reasoning-capable vision models (o-series, Claude, DeepSeek).
41
+ - 🔄 **Multi-Model Fallback Chains:** Automatically falls back to backup vision models if your primary provider is rate-limited or unavailable.
42
+
43
+ ---
44
+
45
+ ## 🕹ī¸ Command Reference
46
+
47
+ | Command | Action |
48
+ |---|---|
49
+ | `/vision-watcher` | Open interactive TUI picker for connected vision models |
50
+ | `/vision-watcher model <provider/id>` | Set primary vision describer directly |
51
+ | `/vision-watcher status` | View current configuration and active model status |
52
+ | `/vision-watcher auto <on\|off>` | Toggle automatic handoff for non-vision models (default: `on`) |
53
+ | `/vision-watcher add <provider/id>` | Force handoff on a specific model |
54
+ | `/vision-watcher remove <provider/id>` | Remove model from forced handoff list |
55
+ | `/vision-watcher thinking <level>` | Configure reasoning effort for vision models |
56
+ | `/vision-watcher enable` / `disable` | Toggle extension active state |
57
+ | `/vision-watcher help` | Show in-CLI command documentation |
58
+
59
+ ---
60
+
61
+ ## 📖 Deep Dive & Advanced Configuration
62
+
63
+ <details>
64
+ <summary><b>⚙ī¸ Configuration File Schema (<code>pi-vision-watcher.json</code>)</b></summary>
65
+
66
+ Configuration is stored at `~/.pi/agent/extensions/pi-vision-watcher.json`:
67
+
68
+ ```json
69
+ {
70
+ "enabled": true,
71
+ "visionModel": "openai/gpt-4o",
72
+ "fallbackModels": [],
73
+ "autoHandoff": true,
74
+ "handoffModels": [],
75
+ "thinking": false,
76
+ "thinkingLevel": "medium",
77
+ "describeTimeoutMs": 45000,
78
+ "prewarmPastedImages": false,
79
+ "asyncClipboardHandoff": false,
80
+ "maxTokens": null,
81
+ "cacheMax": 50,
82
+ "maxDescriptionLines": 0
83
+ }
84
+ ```
85
+
86
+ | Field | Type | Default | Description |
87
+ |---|---|---|---|
88
+ | `enabled` | `boolean` | `true` | Master switch for handoff processing. |
89
+ | `visionModel` | `string \| null` | `null` | Primary describer model ref (`provider/id`). |
90
+ | `fallbackModels` | `string[]` | `[]` | Ordered backup models if the primary model fails. |
91
+ | `autoHandoff` | `boolean` | `true` | Automatically describe images for models lacking native vision. |
92
+ | `handoffModels` | `string[]` | `[]` | Specific model IDs forced to receive descriptions. |
93
+ | `thinking` | `boolean` | `false` | Enable reasoning tokens for vision model. |
94
+ | `thinkingLevel` | `string` | `"medium"` | Reasoning effort (`minimal`, `low`, `medium`, `high`, `xhigh`, `max`). |
95
+ | `describeTimeoutMs` | `number` | `45000` | Per-batch timeout before aborting or triggering fallbacks. |
96
+ | `prewarmPastedImages` | `boolean` | `false` | Start describing clipboard images immediately upon pasting in prompt. |
97
+ | `asyncClipboardHandoff` | `boolean` | `false` | Async clipboard injection fallback mechanism. |
98
+ | `maxTokens` | `number \| null` | `null` | Max output tokens for descriptions (`null` = model default). |
99
+ | `cacheMax` | `number` | `50` | Maximum cached image hashes per session. |
100
+ | `maxDescriptionLines` | `number` | `0` | Truncate lines in description block (`0` = full description). |
101
+
102
+ </details>
103
+
104
+ <details>
105
+ <summary><b>🔍 Troubleshooting, Recovery & Diagnostics</b></summary>
106
+
107
+ ### Structured Error Logging
108
+ If a vision call fails, errors are appended with stack traces and request metadata to:
109
+ ```text
110
+ ~/.pi/agent/logs/pi-vision-watcher/errors.log
111
+ ```
112
+ Failures degrade gracefully to `[Image: description unavailable]` without breaking the agent turn.
113
+
114
+ ### False-Vision Auto-Recovery
115
+ When a model falsely advertises image capability and returns an HTTP 400 rejection:
116
+ 1. `pi-vision-watcher` captures the error in the `message_end` event.
117
+ 2. It automatically updates `~/.pi/agent/models.json` under `providers.<name>.modelOverrides.<model>.input = ["text"]`.
118
+ 3. It triggers an in-process registry refresh so subsequent turns use handoff naturally.
119
+
120
+ </details>
121
+
122
+ <details>
123
+ <summary><b>🛠ī¸ Development & Testing</b></summary>
124
+
125
+ ```bash
126
+ bun install
127
+ bun run test # Run Vitest test suite (240+ unit tests)
128
+ bun run typecheck # Run TypeScript compiler check
129
+ bun run lint:dead # Scan for unused exports with Knip
130
+ ```
131
+
132
+ </details>
133
+
134
+ ---
135
+
136
+ ## 📜 License & Acknowledgments
137
+
138
+ - Built for the **[pi coding agent](https://github.com/earendil-works/pi-coding-agent)** ecosystem.
139
+ - Evolved from concepts in `pi-vision-handoff` by Tom X Nguyen and `pi-umans-provider`.
140
+ - Distributed under the **[MIT License](./LICENSE)**.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@bismawy/pi-vision-watcher",
3
- "version": "1.0.8",
3
+ "version": "1.0.10",
4
4
  "description": "Give text-only pi models vision — describe images with a vision model you pick via an interactive picker, then hand off the text description to non-vision models",
5
5
  "type": "module",
6
6
  "author": "bismawy",
@@ -57,7 +57,7 @@
57
57
  "extensions": [
58
58
  "./vision-watcher.ts"
59
59
  ],
60
- "image": "https://raw.githubusercontent.com/bismawy/pi-vision-watcher/main/assets/screenshot.png"
60
+ "image": "https://raw.githubusercontent.com/bismawy/pi-vision-watcher/main/assets/screenshot.webp"
61
61
  },
62
62
  "peerDependencies": {
63
63
  "@earendil-works/pi-ai": "*",
package/src/index.ts CHANGED
@@ -242,20 +242,24 @@ export function formatModelRef(provider: string, id: string): string {
242
242
  * passes the autoHandoff vision check, so the raw image reaches the provider
243
243
  * and 400s — we learn from that error and force handoff for the model. */
244
244
  export function isImageNotSupportedError(text: string): boolean {
245
- return /not support (?:the )?image|image[s]? (?:input[s]? )?(?:is |are )?not supported|does not accept image|unsupported (?:image|multimodal)/i.test(
245
+ return /not support (?:the )?image|image[s]? (?:input[s]? )?(?:is |are )?not supported|does not accept image|unsupported (?:image|multimodal)|only (?:supports|accepts) text|unsupported content type[\s\S]{0,40}image/i.test(
246
246
  text,
247
247
  );
248
248
  }
249
249
 
250
250
  /** Models whose registry entries commonly declare image input while the
251
- * backend rejects images (DeepSeek V4 via aggregator proxies). Conservative:
252
- * VL / Vision / Janus variants keep image input. Used so autoHandoff covers
253
- * them on the FIRST send — waiting for a 400 is too late. */
251
+ * backend rejects images (DeepSeek V4, GLM 4/5 non-V via aggregator
252
+ * proxies). Conservative: VL / Vision / Janus / *V variants keep image
253
+ * input. Used so autoHandoff covers them on the FIRST send — waiting for a
254
+ * 400 is too late. */
254
255
  export function isKnownTextOnlyFalselyVision(id: string | undefined | null): boolean {
255
256
  if (!id) return false;
256
257
  const n = id.toLowerCase();
257
258
  if (n.includes("vl") || n.includes("vision") || n.includes("janus")) return false;
258
- return /deepseek[-_./]*v4/.test(n);
259
+ if (/\dv(?=[-_./\s]|$)/.test(n)) return false; // glm-4v, glm-4.5v — vision variants
260
+ if (/deepseek[-_./]*v4/.test(n)) return true;
261
+ if (/glm[-_./]*[45]/.test(n)) return true;
262
+ return false;
259
263
  }
260
264
 
261
265
  /** Extract the provider error string from a finalized assistant message.