dsh-click 0.3.28 → 0.3.29

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,19 @@
1
1
  # Changelog
2
2
 
3
+ ## [0.3.29] - 2026-11-13
4
+
5
+ ### Added
6
+
7
+ - **`captureMethod` and `scope` on `screen_shot` results.** `captureMethod` names the path that produced the pixels (`printwindow-fullcontent` / `printwindow` / `copyfromscreen`), and `scope` says what was captured (`window` / `primary` / `virtual`). `scope: "virtual"` captures the whole virtual desktop (`SM_XVIRTUALSCREEN` family); it is opt-in, defaults to `primary`, and is marked experimental because the multi-monitor/negative-origin path is not fully verified.
8
+ - **Minimized windows are reported honestly.** `app_list` marks them `minimized: true` and omits `rect` (Windows reports the `-32000` placeholder rectangle while iconified), and `screen_shot` / `screen_read` refuse them with `WINDOW_MINIMIZED` instead of capturing a broken frame.
9
+
10
+ ### Fixed
11
+
12
+ - **Window screenshots no longer depend on occlusion (the main #6/#35 bug).** The helper now tries `PrintWindow(hwnd, hdc, PW_RENDERFULLCONTENT)` first, falls back to `PrintWindow(0)`, and only then to `CopyFromScreen`. A captured frame is rejected as blank by sampled checks on the 40px-inset 16x16 grid (quantized colour count and luminance standard deviation) plus a 99%-dark test on a 32x32 full-frame grid. Measured on Windows 11: a fully covered Explorer window produced byte-identical `PrintWindow(2)` frames across two different occluders, while `CopyFromScreen` captured the occluder and changed between runs. The colour-count rule is applied only to the no-flag fallback: a legitimate all-white Chrome page measured 3 quantized colours with a luminance sd of 9.6, so applying that rule to `PrintWindow(2)` would reject valid frames.
13
+ - **One target resolver for `snapshot` / `tree` / `shot`.** `snapshot` used to read only `args.windowId`, so `{"op":"snapshot","args":{"target":{"windowId":N}}}` silently reported the foreground window. All three now read `args.target`, with the legacy top-level `windowId` kept as a fallback. A `snapshot` with neither fails with `NO_TARGET` instead of guessing.
14
+ - **`shot` without a target documents and reports the primary screen**, not the foreground window; `screen_read` keeps the foreground default because it is a tree walk.
15
+ - **A window capture and its `shotHash` now come from one frame.** The hash used to be computed from a second capture, which could disagree with the shipped image and with the capture path that was reported.
16
+
3
17
  ## [0.3.28] - 2026-11-13
4
18
 
5
19
  ### Fixed
@@ -79,8 +93,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
79
93
  - Correct the release date in the previous section, which was stamped with the literal string `undefined` by the release stamper. No content or behaviour change; the version is bumped only because npm will not republish an existing version.
80
94
 
81
95
 
82
- ## [Unreleased]
83
-
84
96
  ## [0.3.17] - 2026-10-04
85
97
 
86
98
  undefined
package/README.en.md CHANGED
@@ -97,11 +97,11 @@ Every PerryLink plugin is listed in the built-in DSH Desktop market: **Market
97
97
 
98
98
  `dsh-click` splits the native desktop into an auditable observation face and action face:
99
99
 
100
- - **`screen_shot`** — captures a window (or the primary screen) and downscales it to `maxScreenshotSide`. With a vision-capable model the result carries the image; text-only models get a text description and keep working.
100
+ - **`screen_shot`** — captures a window (or the primary screen) and downscales it to `maxScreenshotSide`. Window captures take `PrintWindow(PW_RENDERFULLCONTENT)` first, so they do not depend on other windows stacked on top; `captureMethod` reports the path that actually produced the pixels. With a vision-capable model the result carries the image; text-only models get a text description and keep working.
101
101
  - **`screen_read`** — the structured observation: the accessibility tree (element ids, types, names, rectangles, supported patterns) plus pixel-location hints with colors — plain text, no image model required.
102
102
  - **`screen_find`** — for applications that expose no accessibility tree, finds clickable text or a described visual target via optional OCR (`ocr.*`, tesseract probed at mount) or visual grounding, returning screen coordinates plus an `observationId`; read-only, never needs approval.
103
103
  - **`click` / `type` / `scroll` / `key`** — window-scoped actions addressed by element id or coordinates. Delivery prefers UIA invoke, falls back to posted window messages, and never steals foreground focus by default; `type` takes the UIA `ValuePattern` fast path and otherwise injects real keystrokes through `SendInput` (CJK-safe); `key` reports honestly when an app ignores posted input.
104
- - **`app_list` / `app_launch`** — enumerate running applications and their visible windows; launch one by name or executable path.
104
+ - **`app_list` / `app_launch`** — enumerate running applications and their visible windows (minimized windows carry `minimized: true` and no `rect`); launch one by name or executable path.
105
105
 
106
106
  Every mutating action crosses one shared safety boundary:
107
107
 
@@ -133,6 +133,15 @@ model harness
133
133
  | `app_list` | ✅ | — | Running applications and their visible windows |
134
134
  | `app_launch` | | ✅ | By name or executable path, with optional arguments |
135
135
 
136
+ ### Screenshots and coordinate space
137
+
138
+ - **A target means a window capture; no target means the primary screen** (`snapshot.windowId = 0`, `title = "primary screen"`), not the foreground window. For the whole virtual desktop (multiple monitors, negative origins) pass `scope: "virtual"`; that mode is marked **experimental** and its multi-monitor geometry is not fully verified yet.
139
+ - **Window captures try `PrintWindow(hwnd, hdc, PW_RENDERFULLCONTENT)` first**: the window paints itself, so the image does not depend on what is stacked on top. On failure or a blank frame the helper falls back to `PrintWindow(0)` and then to `CopyFromScreen`. `captureMethod` (`printwindow-fullcontent` / `printwindow` / `copyfromscreen`) says which path produced the image.
140
+ - **`width` / `height` are the bitmap size, not the window size**: they are already downscaled to `maxScreenshotSide`, while `snapshot.rect` is always the physical window (or screen) rectangle. Map with `bitmapX = (screenX - rect.x) * width / rect.width`.
141
+ - **Every rectangle is in physical pixels**: DPI awareness is set when the helper starts (Per-Monitor V2), and `app_list`, `screen_read`, `screen_shot`, and `click` share one coordinate space.
142
+ - **Minimized windows**: `app_list` marks them `minimized: true` and returns no `rect` (Windows reports the `-32000` placeholder rect while iconified); `screen_shot` / `screen_read` fail with `WINDOW_MINIMIZED`, so restore the window first.
143
+
144
+
136
145
  ### Examples
137
146
 
138
147
  These are the plugin’s real entry points; install once, then call one.
package/README.es.md CHANGED
@@ -97,11 +97,11 @@ Todos los plugins de PerryLink pueden explorarse en el mercado integrado de DSH
97
97
 
98
98
  `dsh-click` separa el escritorio nativo en una cara de observación y otra de acción, ambas auditables:
99
99
 
100
- - **`screen_shot`** — captura una ventana (o la pantalla principal) y la escala a `maxScreenshotSide`. Con un modelo capaz de ver imágenes el resultado lleva la imagen; los modelos de solo texto reciben una descripción textual y siguen funcionando.
100
+ - **`screen_shot`** — captura una ventana (o la pantalla principal) y la escala a `maxScreenshotSide`. Las capturas de ventana usan primero `PrintWindow(PW_RENDERFULLCONTENT)`, así que no dependen de otras ventanas superpuestas; `captureMethod` indica la ruta que produjo los píxeles. Con un modelo capaz de ver imágenes el resultado lleva la imagen; los modelos de solo texto reciben una descripción textual y siguen funcionando.
101
101
  - **`screen_read`** — la observación estructurada: el árbol de accesibilidad (ids de elemento, tipos, nombres, rectángulos, patrones soportados) más pistas de posición de píxeles con colores; texto plano, sin necesidad de modelo de imagen.
102
102
  - **`screen_find`** — para aplicaciones sin árbol de accesibilidad, busca texto clicable o un objetivo visual descrito mediante OCR opcional (`ocr.*`, tesseract sondeado al montar) o grounding visual, y devuelve coordenadas de pantalla más un `observationId`; es de solo lectura y nunca pide aprobación.
103
103
  - **`click` / `type` / `scroll` / `key`** — acciones con ámbito de ventana direccionadas por id de elemento o coordenadas. La entrega prefiere la invocación UIA y recurre a mensajes de ventana publicados, y por defecto nunca roba el foco de primer plano; `type` usa la vía rápida `ValuePattern` de UIA y, si falla, inyecta pulsaciones reales con `SendInput` (seguro para CJK); `key` informa con honestidad cuando una app ignora la entrada publicada.
104
- - **`app_list` / `app_launch`** — enumera las aplicaciones en ejecución y sus ventanas visibles; lanza una por nombre o ruta del ejecutable.
104
+ - **`app_list` / `app_launch`** — enumera las aplicaciones en ejecución y sus ventanas visibles (las minimizadas llevan `minimized: true` y no traen `rect`); lanza una por nombre o ruta del ejecutable.
105
105
 
106
106
  Toda acción mutante cruza la misma frontera de seguridad:
107
107
 
@@ -133,6 +133,15 @@ modelo harness
133
133
  | `app_list` | ✅ | — | Aplicaciones en ejecución y sus ventanas visibles |
134
134
  | `app_launch` | | ✅ | Por nombre o ruta del ejecutable, con argumentos opcionales |
135
135
 
136
+ ### Capturas y espacio de coordenadas
137
+
138
+ - **Con `target` se captura una ventana; sin `target` se captura la pantalla principal** (`snapshot.windowId = 0`, `title = "primary screen"`), no la ventana en primer plano. Para todo el escritorio virtual (varios monitores, con origen negativo) pasa `scope: "virtual"`; ese modo está marcado como **experimental** y su geometría multimonitor aún no está verificada por completo.
139
+ - **La captura de ventana prueba primero `PrintWindow(hwnd, hdc, PW_RENDERFULLCONTENT)`**: la ventana se dibuja a sí misma, así que la imagen no depende de lo que haya encima. Si falla o sale en blanco, el helper retrocede a `PrintWindow(0)` y luego a `CopyFromScreen`. `captureMethod` (`printwindow-fullcontent` / `printwindow` / `copyfromscreen`) dice qué ruta produjo la imagen.
140
+ - **`width` / `height` son el tamaño del bitmap, no el de la ventana**: ya vienen reducidos a `maxScreenshotSide`, mientras que `snapshot.rect` es siempre el rectángulo físico de la ventana (o de la pantalla). Conversión: `bitmapX = (screenX - rect.x) * width / rect.width`.
141
+ - **Todos los rectángulos están en píxeles físicos**: la conciencia de DPI se fija al arrancar el helper (Per-Monitor V2), y `app_list`, `screen_read`, `screen_shot` y `click` comparten el mismo espacio de coordenadas.
142
+ - **Ventanas minimizadas**: `app_list` las marca con `minimized: true` y no devuelve `rect` (Windows da el rectángulo comodín `-32000` mientras están minimizadas); `screen_shot` / `screen_read` fallan con `WINDOW_MINIMIZED`; restaura la ventana primero.
143
+
144
+
136
145
  ### Ejemplos
137
146
 
138
147
  Son los puntos de entrada reales del plugin; instala una vez y llama a uno.
package/README.hi.md CHANGED
@@ -97,11 +97,11 @@ dsh plugin --profile <name> add dsh-click
97
97
 
98
98
  `dsh-click` नेटिव डेस्कटॉप को दो ऑडिट-योग्य पक्षों में बाँटता है — अवलोकन और कार्रवाई:
99
99
 
100
- - **`screen_shot`** — विंडो (या मुख्य स्क्रीन) की तस्वीर लेकर `maxScreenshotSide` तक छोटा करता है। जो मॉडल तस्वीरें लेता है उसे परिणाम में चित्र मिलता है; केवल-टेक्स्ट मॉडल को टेक्स्ट विवरण मिलता है और वे भी चलते रहते हैं।
100
+ - **`screen_shot`** — विंडो (या मुख्य स्क्रीन) की तस्वीर लेकर `maxScreenshotSide` तक छोटा करता है। विंडो कैप्चर पहले `PrintWindow(PW_RENDERFULLCONTENT)` से होते हैं, इसलिए ऊपर रखी विंडो से प्रभावित नहीं होते; `captureMethod` बताता है कि पिक्सेल असल में किस रास्ते से आए। जो मॉडल तस्वीरें लेता है उसे परिणाम में चित्र मिलता है; केवल-टेक्स्ट मॉडल को टेक्स्ट विवरण मिलता है और वे भी चलते रहते हैं।
101
101
  - **`screen_read`** — संरचित अवलोकन: accessibility tree (तत्व id, प्रकार, नाम, आयत, समर्थित पैटर्न) और रंगों सहित pixel स्थिति-संकेत — शुद्ध टेक्स्ट, कोई इमेज मॉडल ज़रूरी नहीं।
102
102
  - **`screen_find`** — जिन ऐप्स में accessibility tree नहीं है, उनके लिए वैकल्पिक OCR (`ocr.*`, माउंट पर tesseract जाँचा जाता है) या visual grounding से क्लिक-योग्य टेक्स्ट/वर्णित लक्ष्य खोजता है और स्क्रीन निर्देशांक तथा `observationId` लौटाता है; यह केवल-पठन है और कभी मंज़ूरी नहीं माँगता।
103
103
  - **`click` / `type` / `scroll` / `key`** — तत्व id या निर्देशांक से पता किए गए विंडो-स्कोप कार्रवाई। डिलीवरी UIA invoke पसंद करती है और posted विंडो संदेशों पर लौटती है, तथा डिफ़ॉल्ट रूप से foreground फ़ोकस कभी नहीं छीनती; `type` पहले UIA `ValuePattern` तेज़ रास्ता लेता है, फिर `SendInput` से असली कीस्ट्रोक डालता है (CJK-सुरक्षित); posted इनपुट अनदेखा करने वाले ऐप्स पर `key` ईमानदारी से विफलता बताता है।
104
- - **`app_list` / `app_launch`** — चल रहे ऐप्स और उनकी दिखने वाली विंडो सूचीबद्ध करता है; नाम या executable पथ से ऐप लॉन्च करता है।
104
+ - **`app_list` / `app_launch`** — चल रहे ऐप्स और उनकी दिखने वाली विंडो सूचीबद्ध करता है (minimized विंडो पर `minimized: true` होता है और कोई `rect` नहीं आता); नाम या executable पथ से ऐप लॉन्च करता है।
105
105
 
106
106
  हर बदलाव वाली कार्रवाई एक ही सुरक्षा सीमा से गुज़रती है:
107
107
 
@@ -133,6 +133,15 @@ dsh plugin --profile <name> add dsh-click
133
133
  | `app_list` | ✅ | — | चल रहे ऐप्स और उनकी दिखने वाली विंडो |
134
134
  | `app_launch` | | ✅ | नाम या executable पथ से, वैकल्पिक आर्ग्युमेंट के साथ |
135
135
 
136
+ ### स्क्रीनशॉट और निर्देशांक स्थान
137
+
138
+ - **`target` हो तो विंडो कैप्चर होती है; `target` न हो तो मुख्य स्क्रीन** (`snapshot.windowId = 0`, `title = "primary screen"`), foreground विंडो नहीं। पूरे virtual desktop (कई मॉनिटर, ऋणात्मक मूल) के लिए `scope: "virtual"` भेजें; यह मोड **experimental** है और बहु-मॉनिटर ज्यामिति की पूरी पुष्टि नहीं हुई है।
139
+ - **विंडो कैप्चर पहले `PrintWindow(hwnd, hdc, PW_RENDERFULLCONTENT)` आज़माते हैं**: विंडो स्वयं चित्रित होती है, इसलिए ऊपर रखी विंडो प्रभावित नहीं करती। विफलता या खाली फ़्रेम पर helper `PrintWindow(0)` और फिर `CopyFromScreen` पर लौटता है। `captureMethod` (`printwindow-fullcontent` / `printwindow` / `copyfromscreen`) बताता है कि चित्र किस रास्ते से बना।
140
+ - **`width` / `height` बिटमैप का आकार हैं, विंडो का नहीं**: ये `maxScreenshotSide` तक घटाए जा चुके हैं, जबकि `snapshot.rect` हमेशा विंडो (या स्क्रीन) का भौतिक आयत है। रूपांतरण: `bitmapX = (screenX - rect.x) * width / rect.width`.
141
+ - **सभी आयत भौतिक पिक्सेल में हैं**: DPI जागरूकता helper शुरू होते ही सेट होती है (Per-Monitor V2), और `app_list`, `screen_read`, `screen_shot` तथा `click` एक ही निर्देशांक स्थान साझा करते हैं।
142
+ - **minimized विंडो**: `app_list` उन्हें `minimized: true` से चिह्नित करता है और `rect` नहीं देता (Windows minimized अवस्था में `-32000` प्लेसहोल्डर आयत देता है); `screen_shot` / `screen_read` स्पष्ट रूप से `WINDOW_MINIMIZED` देकर विफल होते हैं — पहले विंडो बहाल करें।
143
+
144
+
136
145
  ### उदाहरण
137
146
 
138
147
  ये प्लगइन के वास्तविक प्रवेश बिंदु हैं; एक बार इंस्टॉल करके एक चलाएँ।
package/README.md CHANGED
@@ -97,11 +97,11 @@ dsh plugin --profile <name> add dsh-click
97
97
 
98
98
  `dsh-click` 把原生桌面拆成可审计的观察面与动作面:
99
99
 
100
- - **`screen_shot`** —— 截取窗口(或主屏幕),按 `maxScreenshotSide` 缩放。模型支持图像时结果附带图片;纯文本模型则得到文字描述,照常可用。
100
+ - **`screen_shot`** —— 截取窗口(或主屏幕),按 `maxScreenshotSide` 缩放。窗口截图优先走 `PrintWindow(PW_RENDERFULLCONTENT)`,不受其它窗口遮挡影响;结果里的 `captureMethod` 写明实际路径。模型支持图像时结果附带图片;纯文本模型则得到文字描述,照常可用。
101
101
  - **`screen_read`** —— 结构化观察:无障碍树(元素 id、类型、名称、矩形、支持的模式)加像素位置提示与颜色 —— 纯文本,无需图像模型。
102
102
  - **`screen_find`** —— 对没有无障碍树的应用,用可选 OCR(`ocr.*`,挂载时探测 tesseract)或视觉 grounding 按文字/描述找可点目标,返回屏幕坐标与 `observationId`;只读,不需要审批。
103
103
  - **`click` / `type` / `scroll` / `key`** —— 以元素 id 或坐标寻址的窗口级操作。优先 UIA invoke,回退到 post 窗口消息,且默认绝不抢占前台焦点;`type` 走 UIA `ValuePattern` 快路径,失败再用 `SendInput` 注入真实按键(CJK 安全);`key` 对忽略 post 输入的应用会大声拒绝。
104
- - **`app_list` / `app_launch`** —— 枚举运行中的应用及其可见窗口;按名称或可执行路径启动应用。
104
+ - **`app_list` / `app_launch`** —— 枚举运行中的应用及其可见窗口(最小化窗口标 `minimized: true` 且不返回 `rect`);按名称或可执行路径启动应用。
105
105
 
106
106
  每个变更性操作都穿过同一条安全边界:
107
107
 
@@ -133,6 +133,15 @@ dsh plugin --profile <name> add dsh-click
133
133
  | `app_list` | ✅ | — | 运行中的应用及其可见窗口 |
134
134
  | `app_launch` | | ✅ | 按名称或可执行路径启动,可带参数 |
135
135
 
136
+ ### 截图与坐标空间
137
+
138
+ - **带 `target` 才是窗口截图;不带 `target` 抓的是主屏**(`snapshot.windowId = 0`、`title = "primary screen"`),不是前台窗口。要整个虚拟桌面(多屏,含负坐标)请传 `scope: "virtual"`;该模式标注为**实验性**,多屏几何尚未完整验证。
139
+ - **窗口截图先走 `PrintWindow(hwnd, hdc, PW_RENDERFULLCONTENT)`**:窗口自己绘制,不受其它窗口遮挡影响;失败或空图时依次回退 `PrintWindow(0)` 与 `CopyFromScreen`。结果里的 `captureMethod` 为 `printwindow-fullcontent` / `printwindow` / `copyfromscreen`,直接说明这张图是怎么来的。
140
+ - **`width` / `height` 是位图尺寸,不是窗口尺寸**:它们已被 `maxScreenshotSide` 下采样;`snapshot.rect` 始终是物理像素的窗口(或屏幕)矩形。换算方式:`bitmapX = (screenX - rect.x) * width / rect.width`。
141
+ - **所有矩形都是物理像素**:DPI 感知在 helper 启动时设置(Per-Monitor V2),`app_list`、`screen_read`、`screen_shot` 与 `click` 共用同一坐标空间。
142
+ - **最小化窗口**:`app_list` 标记 `minimized: true` 并且不返回 `rect`(Windows 在最小化时给的是 `-32000` 占位矩形);`screen_shot` / `screen_read` 对它明确报 `WINDOW_MINIMIZED`,请先还原窗口。
143
+
144
+
136
145
  ### 示例
137
146
 
138
147
  以下为插件真实入口;安装后调用一个即可。
package/README.pt.md CHANGED
@@ -97,11 +97,11 @@ Todo plugin PerryLink está listado no mercado integrado do DSH Desktop: **Marke
97
97
 
98
98
  O `dsh-click` separa o desktop nativo numa face de observação e noutra de ação, ambas auditáveis:
99
99
 
100
- - **`screen_shot`** — captura uma janela (ou a tela principal) e a reduz para `maxScreenshotSide`. Com um modelo que aceita imagens o resultado leva a imagem; modelos só de texto recebem uma descrição textual e continuam funcionando.
100
+ - **`screen_shot`** — captura uma janela (ou a tela principal) e a reduz para `maxScreenshotSide`. Capturas de janela usam primeiro `PrintWindow(PW_RENDERFULLCONTENT)`, sem depender de outras janelas sobrepostas; `captureMethod` informa o caminho que produziu os píxeis. Com um modelo que aceita imagens o resultado leva a imagem; modelos só de texto recebem uma descrição textual e continuam funcionando.
101
101
  - **`screen_read`** — a observação estruturada: a árvore de acessibilidade (ids de elemento, tipos, nomes, retângulos, padrões suportados) mais dicas de posição de píxeis com cores; texto puro, sem precisar de modelo de imagem.
102
102
  - **`screen_find`** — para aplicações sem árvore de acessibilidade, encontra texto clicável ou um alvo visual descrito via OCR opcional (`ocr.*`, tesseract sondado ao montar) ou grounding visual, devolvendo coordenadas de tela mais um `observationId`; é só leitura e nunca pede aprovação.
103
103
  - **`click` / `type` / `scroll` / `key`** — ações com escopo de janela endereçadas por id de elemento ou coordenadas. A entrega prefere invocação UIA e recorre a mensagens de janela postadas, e por padrão nunca rouba o foco de primeiro plano; `type` usa o caminho rápido `ValuePattern` do UIA e, se falhar, injeta teclas reais via `SendInput` (seguro para CJK); `key` reporta com honestidade quando uma app ignora a entrada postada.
104
- - **`app_list` / `app_launch`** — enumera aplicações em execução e as suas janelas visíveis; lança uma por nome ou caminho do executável.
104
+ - **`app_list` / `app_launch`** — enumera aplicações em execução e as suas janelas visíveis (as minimizadas levam `minimized: true` e não trazem `rect`); lança uma por nome ou caminho do executável.
105
105
 
106
106
  Toda ação mutante cruza a mesma fronteira de segurança:
107
107
 
@@ -133,6 +133,15 @@ modelo harness
133
133
  | `app_list` | ✅ | — | Aplicações em execução e as suas janelas visíveis |
134
134
  | `app_launch` | | ✅ | Por nome ou caminho do executável, com argumentos opcionais |
135
135
 
136
+ ### Capturas e espaço de coordenadas
137
+
138
+ - **Com `target` captura-se uma janela; sem `target` captura-se a tela principal** (`snapshot.windowId = 0`, `title = "primary screen"`), não a janela em primeiro plano. Para todo o desktop virtual (vários monitores, com origem negativa) passe `scope: "virtual"`; esse modo está marcado como **experimental** e a geometria multimonitor ainda não está totalmente verificada.
139
+ - **A captura de janela tenta primeiro `PrintWindow(hwnd, hdc, PW_RENDERFULLCONTENT)`**: a janela desenha-se a si mesma, pelo que a imagem não depende do que está por cima. Se falhar ou vier em branco, o helper recua para `PrintWindow(0)` e depois para `CopyFromScreen`. `captureMethod` (`printwindow-fullcontent` / `printwindow` / `copyfromscreen`) diz qual caminho produziu a imagem.
140
+ - **`width` / `height` são o tamanho do bitmap, não o da janela**: já vêm reduzidos a `maxScreenshotSide`, enquanto `snapshot.rect` é sempre o retângulo físico da janela (ou da tela). Conversão: `bitmapX = (screenX - rect.x) * width / rect.width`.
141
+ - **Todos os retângulos estão em píxeis físicos**: a consciência de DPI é definida quando o helper arranca (Per-Monitor V2), e `app_list`, `screen_read`, `screen_shot` e `click` partilham o mesmo espaço de coordenadas.
142
+ - **Janelas minimizadas**: `app_list` marca-as com `minimized: true` e não devolve `rect` (o Windows dá o retângulo marcador `-32000` enquanto minimizadas); `screen_shot` / `screen_read` falham com `WINDOW_MINIMIZED`; restaure a janela primeiro.
143
+
144
+
136
145
  ### Exemplos
137
146
 
138
147
  Estes são os pontos de entrada reais do plugin; instale uma vez e chame um.
package/lib/index.js CHANGED
@@ -623,7 +623,7 @@ var ObservationStore = class {
623
623
  * @module dsh-click/version
624
624
  */
625
625
  /** Plugin version advertised in the helper-protocol handshake. */
626
- const VERSION = "0.3.28";
626
+ const VERSION = "0.3.29";
627
627
  /** The helper protocol revision; bump only on incompatible wire changes. */
628
628
  const HELPER_PROTOCOL_VERSION = 1;
629
629
  //#endregion
@@ -668,12 +668,16 @@ function expectWindow(value, op) {
668
668
  const processId = typeof processIdRaw === "number" ? processIdRaw : null;
669
669
  const executablePath = record["executablePath"];
670
670
  if (executablePath !== null && typeof executablePath !== "string") throw new DshClickError(`helper "${op}" returned malformed executablePath`, "BAD_HELPER_RESPONSE");
671
+ const minimized = record["minimized"] === true;
672
+ const rectValue = record["rect"];
673
+ if (rectValue !== null && minimized) throw new DshClickError(`helper "${op}" returned a rect for a minimized window`, "BAD_HELPER_RESPONSE");
671
674
  return {
672
675
  windowId: expectNumber(record, "windowId", op),
673
676
  processId,
674
677
  title: expectString(record, "title", op),
675
678
  className: expectString(record, "className", op),
676
- rect: expectRect(record["rect"], op),
679
+ rect: rectValue === null ? null : expectRect(rectValue, op),
680
+ minimized,
677
681
  executablePath,
678
682
  visible: record["visible"] !== false
679
683
  };
@@ -696,6 +700,18 @@ function expectSnapshot(value, op) {
696
700
  elementCount: expectNumber(record, "elementCount", op)
697
701
  };
698
702
  }
703
+ /** Validate the helper's capture-path report. */
704
+ function expectCaptureMethod(record, op) {
705
+ const value = expectString(record, "captureMethod", op);
706
+ if (value !== "printwindow-fullcontent" && value !== "printwindow" && value !== "copyfromscreen") throw new DshClickError(`helper "${op}" returned an unknown captureMethod`, "BAD_HELPER_RESPONSE");
707
+ return value;
708
+ }
709
+ /** Validate the helper's capture-scope report. */
710
+ function expectCaptureScope(record, op) {
711
+ const value = expectString(record, "scope", op);
712
+ if (value !== "window" && value !== "primary" && value !== "virtual") throw new DshClickError(`helper "${op}" returned an unknown scope`, "BAD_HELPER_RESPONSE");
713
+ return value;
714
+ }
699
715
  /** Validate one helper element record. */
700
716
  function expectElement(value, op) {
701
717
  const record = expectRecordValue(value, op);
@@ -820,10 +836,11 @@ var HelperBackend = class {
820
836
  if (!Array.isArray(result)) throw new DshClickError("helper \"windows\" returned a non-array result", "BAD_HELPER_RESPONSE");
821
837
  return result.map((item) => expectWindow(item, "windows"));
822
838
  }
823
- async shot(ref, maxSide, signal) {
839
+ async shot(ref, maxSide, scope, signal) {
824
840
  const record = expectRecordValue(await this.invoke("shot", {
825
841
  target: ref,
826
842
  maxSide,
843
+ scope,
827
844
  maxElements: this.config.maxElements,
828
845
  maxDepth: this.config.maxTreeDepth
829
846
  }, signal), "shot");
@@ -831,6 +848,8 @@ var HelperBackend = class {
831
848
  pngBase64: expectString(record, "pngBase64", "shot"),
832
849
  width: expectNumber(record, "width", "shot"),
833
850
  height: expectNumber(record, "height", "shot"),
851
+ captureMethod: expectCaptureMethod(record, "shot"),
852
+ scope: expectCaptureScope(record, "shot"),
834
853
  snapshot: expectSnapshot(record["snapshot"], "shot")
835
854
  };
836
855
  }
@@ -1251,7 +1270,8 @@ function observedWindowInfo(info, maxTextLength) {
1251
1270
  processId: info.processId,
1252
1271
  title: sanitizeVisible(info.title, maxTextLength),
1253
1272
  className: sanitizeVisible(info.className, maxTextLength),
1254
- rect: info.rect,
1273
+ rect: info.rect === null ? null : { ...info.rect },
1274
+ minimized: info.minimized,
1255
1275
  executablePath: info.executablePath === null ? null : sanitizePath(info.executablePath, maxTextLength),
1256
1276
  visible: info.visible
1257
1277
  };
@@ -1279,8 +1299,8 @@ function backendGuard(services) {
1279
1299
  if (!services.backend.available) throw new Error(`dsh-click backend unavailable on ${services.backend.platform}: ${services.backend.unavailableReason ?? "unknown reason"}`);
1280
1300
  }
1281
1301
  /** Plain-text fallback description for a captured screenshot. */
1282
- function shotDescription(window, width, height) {
1283
- return `${width}x${height} screenshot of ${windowLine(window)}; run screen_read on the same window for the structured element list and pixel positions.`;
1302
+ function shotDescription(window, width, height, captureMethod) {
1303
+ return `${width}x${height} bitmap via ${captureMethod} of ${windowLine(window)}; the bitmap is downscaled to the requested longest side, while the window rectangle is ${window.rect.width}x${window.rect.height} physical pixels (snapshot.rect) - map screen coordinates with rect.width/width. Run screen_read on the same window for the structured element list and pixel positions.`;
1284
1304
  }
1285
1305
  /**
1286
1306
  * `screen_shot` — capture the addressed window (or the primary screen).
@@ -1295,11 +1315,11 @@ function screenShotTool(services) {
1295
1315
  const { ctx, config, backend, observations } = services;
1296
1316
  return defineTool({
1297
1317
  name: "screen_shot",
1298
- description: "Capture a screenshot of a desktop window (or the primary screen when no window is given). Returns an observationId that later actions cite in `basedOn`. When the current model accepts images the result includes the image; otherwise it includes a text description. Read-only: never needs approval.",
1318
+ description: "Capture a screenshot of a desktop window, or of the primary screen when no target is given (a target-less shot never captures the foreground window). Window captures ask the window to paint itself with PrintWindow(PW_RENDERFULLCONTENT) first, so they are independent of other windows stacked on top; captureMethod reports which path produced the pixels. Returns an observationId that later actions cite in `basedOn`. When the current model accepts images the result includes the image; otherwise it includes a text description. Read-only: never needs approval.",
1299
1319
  parameters: {
1300
1320
  target: {
1301
1321
  type: "object",
1302
- description: "Which window to capture (windowId, windowTitle, or processId); omitted = the foreground window.",
1322
+ description: "Which window to capture (windowId, windowTitle, or processId); omitted = the primary screen.",
1303
1323
  properties: {
1304
1324
  windowId: {
1305
1325
  type: "integer",
@@ -1316,9 +1336,14 @@ function screenShotTool(services) {
1316
1336
  },
1317
1337
  additionalProperties: false
1318
1338
  },
1339
+ scope: {
1340
+ type: "string",
1341
+ enum: ["primary", "virtual"],
1342
+ description: "Only used without a target: capture the primary screen (default) or the whole virtual desktop (experimental - multi-monitor and negative-origin geometry are not fully verified)."
1343
+ },
1319
1344
  maxSide: {
1320
1345
  type: "integer",
1321
- description: "Longest side in pixels; larger captures are downscaled."
1346
+ description: "Longest side of the bitmap; larger captures are downscaled (the window rectangle in snapshot.rect stays in physical pixels)."
1322
1347
  }
1323
1348
  },
1324
1349
  output: {
@@ -1364,6 +1389,22 @@ function screenShotTool(services) {
1364
1389
  },
1365
1390
  additionalProperties: false
1366
1391
  },
1392
+ captureMethod: {
1393
+ type: "string",
1394
+ enum: [
1395
+ "printwindow-fullcontent",
1396
+ "printwindow",
1397
+ "copyfromscreen"
1398
+ ]
1399
+ },
1400
+ scope: {
1401
+ type: "string",
1402
+ enum: [
1403
+ "window",
1404
+ "primary",
1405
+ "virtual"
1406
+ ]
1407
+ },
1367
1408
  description: { type: "string" }
1368
1409
  },
1369
1410
  additionalProperties: false
@@ -1384,13 +1425,14 @@ function screenShotTool(services) {
1384
1425
  timeoutMs: config.helperTimeoutMs + 15e3,
1385
1426
  async execute(args, exec) {
1386
1427
  backendGuard(services);
1387
- const target = args.target ?? {};
1388
- const requestedSide = args.maxSide;
1428
+ const parsed = args;
1429
+ const target = parsed.target ?? {};
1430
+ const requestedSide = parsed.maxSide;
1389
1431
  const maxSide = requestedSide === void 0 ? config.maxScreenshotSide : Math.min(requestedSide, config.maxScreenshotSide);
1390
- const shot = await backend.shot(target, maxSide, exec.signal);
1432
+ const shot = await backend.shot(target, maxSide, parsed.scope ?? "primary", exec.signal);
1391
1433
  const sanitized = observedWindow(shot.snapshot, config.maxTextLength);
1392
1434
  const record = observations.record(shot.snapshot);
1393
- const description = shotDescription(sanitized, shot.width, shot.height);
1435
+ const description = shotDescription(sanitized, shot.width, shot.height, shot.captureMethod);
1394
1436
  let image;
1395
1437
  const attachments = ctx.get("attachments");
1396
1438
  if (config.imageMode !== "text" && attachments !== void 0 && await sessionAcceptsImages(exec.agent, ctx, exec.signal)) try {
@@ -1422,6 +1464,8 @@ function screenShotTool(services) {
1422
1464
  observationId: record.id,
1423
1465
  window: sanitized,
1424
1466
  ...image !== void 0 ? { image } : {},
1467
+ captureMethod: shot.captureMethod,
1468
+ scope: shot.scope,
1425
1469
  description
1426
1470
  };
1427
1471
  }
@@ -1603,7 +1647,7 @@ function screenFindTool(services) {
1603
1647
  parameters: {
1604
1648
  target: {
1605
1649
  type: "object",
1606
- description: "Which window to search (windowId, windowTitle, or processId); omitted = the foreground window.",
1650
+ description: "Which window to search (windowId, windowTitle, or processId); omitted = the primary screen (a target-less capture never uses the foreground window).",
1607
1651
  properties: {
1608
1652
  windowId: {
1609
1653
  type: "integer",
@@ -1626,7 +1670,7 @@ function screenFindTool(services) {
1626
1670
  },
1627
1671
  maxSide: {
1628
1672
  type: "integer",
1629
- description: "Longest side in pixels; larger captures are downscaled."
1673
+ description: "Longest side of the bitmap; larger captures are downscaled (coordinates returned are still screen pixels)."
1630
1674
  }
1631
1675
  },
1632
1676
  output: {
@@ -1705,7 +1749,7 @@ function screenFindTool(services) {
1705
1749
  const parsed = args;
1706
1750
  const requestedSide = parsed.maxSide;
1707
1751
  const maxSide = requestedSide === void 0 ? config.maxScreenshotSide : Math.min(requestedSide, config.maxScreenshotSide);
1708
- const shot = await backend.shot(parsed.target ?? {}, maxSide, exec.signal);
1752
+ const shot = await backend.shot(parsed.target ?? {}, maxSide, "primary", exec.signal);
1709
1753
  const sanitized = observedWindow(shot.snapshot, config.maxTextLength);
1710
1754
  const record = observations.record(shot.snapshot);
1711
1755
  const targets = [];
@@ -2095,7 +2139,7 @@ function appListTool(services) {
2095
2139
  const { config, backend } = services;
2096
2140
  return defineTool({
2097
2141
  name: "app_list",
2098
- description: "List running desktop applications and their visible windows (windowId, title, process id, executable path). Read-only: never needs approval. Use the returned windowIds as screen_shot/screen_read targets.",
2142
+ description: "List running desktop applications and their visible windows (windowId, title, process id, executable path). Read-only: never needs approval. Use the returned windowIds as screen_shot/screen_read targets. Minimized windows are listed with minimized: true and no rect (Windows reports a -32000 placeholder while iconified); restore them before capturing.",
2099
2143
  parameters: {},
2100
2144
  output: {
2101
2145
  schema: {
@@ -2122,7 +2166,7 @@ function appListTool(services) {
2122
2166
  processId: { oneOf: [{ type: "integer" }, { type: "null" }] },
2123
2167
  title: { type: "string" },
2124
2168
  className: { type: "string" },
2125
- rect: {
2169
+ rect: { oneOf: [{
2126
2170
  type: "object",
2127
2171
  properties: {
2128
2172
  x: { type: "integer" },
@@ -2131,7 +2175,8 @@ function appListTool(services) {
2131
2175
  height: { type: "integer" }
2132
2176
  },
2133
2177
  additionalProperties: false
2134
- },
2178
+ }, { type: "null" }] },
2179
+ minimized: { type: "boolean" },
2135
2180
  executablePath: { oneOf: [{ type: "string" }, { type: "null" }] },
2136
2181
  visible: { type: "boolean" }
2137
2182
  },
@@ -2150,7 +2195,10 @@ function appListTool(services) {
2150
2195
  const lines = [`${result.apps.length} running application(s) with windows:`];
2151
2196
  for (const app of result.apps) {
2152
2197
  lines.push(`- ${app.name} (pid ${app.processId}, ${app.executablePath ?? "unknown executable"})`);
2153
- for (const window of app.windows) lines.push(` - [windowId ${window.windowId}] ${JSON.stringify(window.title)} at (${window.rect.x}, ${window.rect.y}) ${window.rect.width}x${window.rect.height}`);
2198
+ for (const window of app.windows) {
2199
+ const where = window.minimized ? "minimized (no rect; screen_shot refuses it until restored)" : window.rect === null ? "no rect reported" : `at (${window.rect.x}, ${window.rect.y}) ${window.rect.width}x${window.rect.height}`;
2200
+ lines.push(` - [windowId ${window.windowId}] ${JSON.stringify(window.title)} ${where}`);
2201
+ }
2154
2202
  }
2155
2203
  return [{
2156
2204
  type: "text",
@@ -10,7 +10,7 @@
10
10
  */
11
11
  import type { SubprocessRuntime } from '@deepseek-ai/dsh-subprocess';
12
12
  import type { ResolvedConfig } from '../config.js';
13
- import { type ActionOutcome, type AppInfo, type ClickRequest, type DesktopBackend, type KeyRequest, type LaunchOutcome, type Screenshot, type ScrollRequest, type Tree, type WindowInfo, type WindowRef, type WindowSnapshot, type TypeRequest } from './types.js';
13
+ import { type ActionOutcome, type AppInfo, type ClickRequest, type DesktopBackend, type KeyRequest, type LaunchOutcome, type Screenshot, type ScreenScope, type ScrollRequest, type Tree, type WindowInfo, type WindowRef, type WindowSnapshot, type TypeRequest } from './types.js';
14
14
  /** The Windows helper backend over `ctx.subprocess`. */
15
15
  export declare class HelperBackend implements DesktopBackend {
16
16
  private readonly config;
@@ -27,7 +27,7 @@ export declare class HelperBackend implements DesktopBackend {
27
27
  /** Resolve (and cache) the powershell executable through the subprocess seam. */
28
28
  private resolvePowershell;
29
29
  listWindows(signal?: AbortSignal): Promise<WindowInfo[]>;
30
- shot(ref: WindowRef, maxSide: number, signal?: AbortSignal): Promise<Screenshot>;
30
+ shot(ref: WindowRef, maxSide: number, scope: ScreenScope, signal?: AbortSignal): Promise<Screenshot>;
31
31
  tree(ref: WindowRef, maxElements: number, maxDepth: number, includePixels: boolean, signal?: AbortSignal): Promise<Tree>;
32
32
  snapshot(windowId: number, signal?: AbortSignal): Promise<WindowSnapshot>;
33
33
  click(request: ClickRequest, focusFallback: boolean, signal?: AbortSignal): Promise<ActionOutcome>;
@@ -1 +1 @@
1
- {"version":3,"file":"runner.d.ts","sourceRoot":"","sources":["../../../src/platform/runner.ts"],"names":[],"mappings":"AAAA;;;;;;;;;GASG;AAMH,OAAO,KAAK,EAAE,iBAAiB,EAAE,MAAM,6BAA6B,CAAA;AACpE,OAAO,KAAK,EAAE,cAAc,EAAE,MAAM,cAAc,CAAA;AAElD,OAAO,EAML,KAAK,aAAa,EAClB,KAAK,OAAO,EACZ,KAAK,YAAY,EACjB,KAAK,cAAc,EAInB,KAAK,UAAU,EACf,KAAK,aAAa,EAGlB,KAAK,UAAU,EACf,KAAK,aAAa,EAClB,KAAK,IAAI,EACT,KAAK,UAAU,EACf,KAAK,SAAS,EACd,KAAK,cAAc,EACnB,KAAK,WAAW,EACjB,MAAM,YAAY,CAAA;AA6HnB,wDAAwD;AACxD,qBAAa,aAAc,YAAW,cAAc;IAShD,OAAO,CAAC,QAAQ,CAAC,MAAM;IACvB,OAAO,CAAC,QAAQ,CAAC,UAAU;IAT7B,gEAAgE;IAChE,OAAO,CAAC,UAAU,CAA6B;IAE/C,QAAQ,CAAC,SAAS,QAAO;IAEzB,mEAAmE;IACnE,uEAAuE;gBAEpD,MAAM,EAAE,cAAc,EACtB,UAAU,EAAE,iBAAiB;IAGhD,QAAQ,CAAC,QAAQ,WAAU;IAE3B,wEAAwE;YAC1D,MAAM;IAsDpB,iFAAiF;IACjF,OAAO,CAAC,iBAAiB;IAanB,WAAW,CAAC,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,UAAU,EAAE,CAAC;IAMxD,IAAI,CAAC,GAAG,EAAE,SAAS,EAAE,OAAO,EAAE,MAAM,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,UAAU,CAAC;IAgBhF,IAAI,CAAC,GAAG,EAAE,SAAS,EAAE,WAAW,EAAE,MAAM,EAAE,QAAQ,EAAE,MAAM,EAAE,aAAa,EAAE,OAAO,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,IAAI,CAAC;IAexH,QAAQ,CAAC,QAAQ,EAAE,MAAM,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,cAAc,CAAC;IASzE,KAAK,CAAC,OAAO,EAAE,YAAY,EAAE,aAAa,EAAE,OAAO,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,aAAa,CAAC;IAKlG,IAAI,CAAC,OAAO,EAAE,WAAW,EAAE,aAAa,EAAE,OAAO,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,aAAa,CAAC;IAKhG,MAAM,CAAC,OAAO,EAAE,aAAa,EAAE,aAAa,EAAE,OAAO,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,aAAa,CAAC;IAKpG,GAAG,CAAC,OAAO,EAAE,UAAU,EAAE,aAAa,EAAE,OAAO,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,aAAa,CAAC;IAK9F,IAAI,CAAC,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,OAAO,EAAE,CAAC;IAoB9C,MAAM,CAAC,IAAI,EAAE,MAAM,EAAE,IAAI,EAAE,SAAS,MAAM,EAAE,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,aAAa,CAAC;CAYlG"}
1
+ {"version":3,"file":"runner.d.ts","sourceRoot":"","sources":["../../../src/platform/runner.ts"],"names":[],"mappings":"AAAA;;;;;;;;;GASG;AAMH,OAAO,KAAK,EAAE,iBAAiB,EAAE,MAAM,6BAA6B,CAAA;AACpE,OAAO,KAAK,EAAE,cAAc,EAAE,MAAM,cAAc,CAAA;AAElD,OAAO,EAML,KAAK,aAAa,EAClB,KAAK,OAAO,EAGZ,KAAK,YAAY,EACjB,KAAK,cAAc,EAInB,KAAK,UAAU,EACf,KAAK,aAAa,EAGlB,KAAK,UAAU,EACf,KAAK,WAAW,EAChB,KAAK,aAAa,EAClB,KAAK,IAAI,EACT,KAAK,UAAU,EACf,KAAK,SAAS,EACd,KAAK,cAAc,EACnB,KAAK,WAAW,EACjB,MAAM,YAAY,CAAA;AAuJnB,wDAAwD;AACxD,qBAAa,aAAc,YAAW,cAAc;IAShD,OAAO,CAAC,QAAQ,CAAC,MAAM;IACvB,OAAO,CAAC,QAAQ,CAAC,UAAU;IAT7B,gEAAgE;IAChE,OAAO,CAAC,UAAU,CAA6B;IAE/C,QAAQ,CAAC,SAAS,QAAO;IAEzB,mEAAmE;IACnE,uEAAuE;gBAEpD,MAAM,EAAE,cAAc,EACtB,UAAU,EAAE,iBAAiB;IAGhD,QAAQ,CAAC,QAAQ,WAAU;IAE3B,wEAAwE;YAC1D,MAAM;IAsDpB,iFAAiF;IACjF,OAAO,CAAC,iBAAiB;IAanB,WAAW,CAAC,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,UAAU,EAAE,CAAC;IAMxD,IAAI,CAAC,GAAG,EAAE,SAAS,EAAE,OAAO,EAAE,MAAM,EAAE,KAAK,EAAE,WAAW,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,UAAU,CAAC;IAmBpG,IAAI,CAAC,GAAG,EAAE,SAAS,EAAE,WAAW,EAAE,MAAM,EAAE,QAAQ,EAAE,MAAM,EAAE,aAAa,EAAE,OAAO,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,IAAI,CAAC;IAexH,QAAQ,CAAC,QAAQ,EAAE,MAAM,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,cAAc,CAAC;IASzE,KAAK,CAAC,OAAO,EAAE,YAAY,EAAE,aAAa,EAAE,OAAO,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,aAAa,CAAC;IAKlG,IAAI,CAAC,OAAO,EAAE,WAAW,EAAE,aAAa,EAAE,OAAO,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,aAAa,CAAC;IAKhG,MAAM,CAAC,OAAO,EAAE,aAAa,EAAE,aAAa,EAAE,OAAO,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,aAAa,CAAC;IAKpG,GAAG,CAAC,OAAO,EAAE,UAAU,EAAE,aAAa,EAAE,OAAO,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,aAAa,CAAC;IAK9F,IAAI,CAAC,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,OAAO,EAAE,CAAC;IAoB9C,MAAM,CAAC,IAAI,EAAE,MAAM,EAAE,IAAI,EAAE,SAAS,MAAM,EAAE,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,aAAa,CAAC;CAYlG"}
@@ -24,6 +24,20 @@ export interface WindowRef {
24
24
  /** Owning process id. */
25
25
  processId?: number;
26
26
  }
27
+ /** Which screen region a target-less capture covers. */
28
+ export type ScreenScope = 'primary' | 'virtual';
29
+ /** What a capture actually covered: one window, the primary screen, or the virtual desktop. */
30
+ export type CaptureScope = 'window' | ScreenScope;
31
+ /**
32
+ * Which helper path produced the pixels.
33
+ *
34
+ * `printwindow-fullcontent` is `PrintWindow(hwnd, hdc, PW_RENDERFULLCONTENT)`:
35
+ * the window paints itself, so the image does not depend on what is stacked on
36
+ * top of it. `printwindow` is the legacy flag-0 call (some old windows only
37
+ * paint through it). `copyfromscreen` is the last-resort screen-region copy and
38
+ * is only correct while the window is unobscured.
39
+ */
40
+ export type CaptureMethod = 'printwindow-fullcontent' | 'printwindow' | 'copyfromscreen';
27
41
  /** Screen-space rectangle, integer pixel coordinates. */
28
42
  export interface Rect {
29
43
  x: number;
@@ -37,7 +51,14 @@ export interface WindowInfo {
37
51
  processId: number | null;
38
52
  title: string;
39
53
  className: string;
40
- rect: Rect;
54
+ /**
55
+ * Physical screen rectangle, or `null` for a minimized window: Windows reports
56
+ * the `CW_USEDEFAULT` placeholder (`-32000,-32000 314x50`) while iconified,
57
+ * which must never reach coordinate math.
58
+ */
59
+ rect: Rect | null;
60
+ /** True while the window is iconified (`IsIconic`); such a window cannot be captured. */
61
+ minimized: boolean;
41
62
  executablePath: string | null;
42
63
  visible: boolean;
43
64
  }
@@ -90,8 +111,18 @@ export interface WindowSnapshot {
90
111
  export interface Screenshot {
91
112
  /** PNG bytes, base64-encoded. */
92
113
  pngBase64: string;
114
+ /**
115
+ * Bitmap width after the `maxSide` downscale. This is NOT the window width:
116
+ * `snapshot.rect` always carries the physical window/screen rectangle, so a
117
+ * screen-to-bitmap mapping is `bitmapX = (screenX - rect.x) * width / rect.width`.
118
+ */
93
119
  width: number;
120
+ /** Bitmap height after the `maxSide` downscale (see {@link width}). */
94
121
  height: number;
122
+ /** Which capture path produced the pixels. */
123
+ captureMethod: CaptureMethod;
124
+ /** What was captured. `virtual` is experimental (multi-monitor geometry untested). */
125
+ scope: CaptureScope;
95
126
  snapshot: WindowSnapshot;
96
127
  }
97
128
  /** An accessibility tree read plus pixel hints. */
@@ -169,8 +200,12 @@ export interface DesktopBackend {
169
200
  readonly unavailableReason?: string;
170
201
  /** List every visible top-level window across running applications. */
171
202
  listWindows(signal?: AbortSignal): Promise<WindowInfo[]>;
172
- /** Capture a screenshot of the addressed window (or the primary screen). */
173
- shot(ref: WindowRef, maxSide: number, signal?: AbortSignal): Promise<Screenshot>;
203
+ /**
204
+ * Capture a screenshot of the addressed window, or of a screen region when
205
+ * the ref names no window (the helper never substitutes the foreground
206
+ * window here; the primary screen is the target-less default).
207
+ */
208
+ shot(ref: WindowRef, maxSide: number, scope: ScreenScope, signal?: AbortSignal): Promise<Screenshot>;
174
209
  /** Read the accessibility tree plus pixel hints of the addressed window. */
175
210
  tree(ref: WindowRef, maxElements: number, maxDepth: number, includePixels: boolean, signal?: AbortSignal): Promise<Tree>;
176
211
  /** Re-capture fresh identity + hashes for one window (the stale check). */
@@ -1 +1 @@
1
- {"version":3,"file":"types.d.ts","sourceRoot":"","sources":["../../../src/platform/types.ts"],"names":[],"mappings":"AAAA;;;;;;;;GAQG;AAEH,OAAO,KAAK,EAAE,YAAY,EAAE,MAAM,cAAc,CAAA;AAEhD,iFAAiF;AACjF,qBAAa,aAAc,SAAQ,KAAK;IAGT,QAAQ,CAAC,IAAI,EAAE,MAAM;IAFlD,+DAA+D;IAC/D,4DAA4D;gBAChD,OAAO,EAAE,MAAM,EAAW,IAAI,EAAE,MAAM;CAInD;AAED,+DAA+D;AAC/D,MAAM,WAAW,SAAS;IACxB,8CAA8C;IAC9C,QAAQ,CAAC,EAAE,MAAM,CAAA;IACjB,+EAA+E;IAC/E,WAAW,CAAC,EAAE,MAAM,CAAA;IACpB,yBAAyB;IACzB,SAAS,CAAC,EAAE,MAAM,CAAA;CACnB;AAED,yDAAyD;AACzD,MAAM,WAAW,IAAI;IACnB,CAAC,EAAE,MAAM,CAAA;IACT,CAAC,EAAE,MAAM,CAAA;IACT,KAAK,EAAE,MAAM,CAAA;IACb,MAAM,EAAE,MAAM,CAAA;CACf;AAED,wCAAwC;AACxC,MAAM,WAAW,UAAU;IACzB,QAAQ,EAAE,MAAM,CAAA;IAChB,SAAS,EAAE,MAAM,GAAG,IAAI,CAAA;IACxB,KAAK,EAAE,MAAM,CAAA;IACb,SAAS,EAAE,MAAM,CAAA;IACjB,IAAI,EAAE,IAAI,CAAA;IACV,cAAc,EAAE,MAAM,GAAG,IAAI,CAAA;IAC7B,OAAO,EAAE,OAAO,CAAA;CACjB;AAED,4DAA4D;AAC5D,MAAM,WAAW,WAAW;IAC1B,kEAAkE;IAClE,SAAS,EAAE,MAAM,CAAA;IACjB,6CAA6C;IAC7C,WAAW,EAAE,MAAM,CAAA;IACnB,8CAA8C;IAC9C,IAAI,EAAE,MAAM,CAAA;IACZ,uCAAuC;IACvC,YAAY,EAAE,MAAM,CAAA;IACpB,gDAAgD;IAChD,IAAI,EAAE,IAAI,CAAA;IACV,sCAAsC;IACtC,OAAO,EAAE,OAAO,CAAA;IAChB,mGAAmG;IACnG,QAAQ,EAAE,MAAM,EAAE,CAAA;CACnB;AAED,uEAAuE;AACvE,MAAM,WAAW,SAAS;IACxB,kDAAkD;IAClD,KAAK,EAAE,MAAM,CAAA;IACb,2BAA2B;IAC3B,CAAC,EAAE,MAAM,CAAA;IACT,2BAA2B;IAC3B,CAAC,EAAE,MAAM,CAAA;IACT,sDAAsD;IACtD,KAAK,EAAE,MAAM,CAAA;CACd;AAED,oFAAoF;AACpF,MAAM,WAAW,cAAc;IAC7B,QAAQ,EAAE,MAAM,CAAA;IAChB,SAAS,EAAE,MAAM,CAAA;IACjB,cAAc,EAAE,MAAM,GAAG,IAAI,CAAA;IAC7B,KAAK,EAAE,MAAM,CAAA;IACb,SAAS,EAAE,MAAM,CAAA;IACjB,IAAI,EAAE,IAAI,CAAA;IACV,0DAA0D;IAC1D,UAAU,EAAE,OAAO,CAAA;IACnB,4EAA4E;IAC5E,QAAQ,EAAE,MAAM,CAAA;IAChB,2CAA2C;IAC3C,QAAQ,EAAE,MAAM,CAAA;IAChB,iDAAiD;IACjD,YAAY,EAAE,MAAM,CAAA;CACrB;AAED,uEAAuE;AACvE,MAAM,WAAW,UAAU;IACzB,iCAAiC;IACjC,SAAS,EAAE,MAAM,CAAA;IACjB,KAAK,EAAE,MAAM,CAAA;IACb,MAAM,EAAE,MAAM,CAAA;IACd,QAAQ,EAAE,cAAc,CAAA;CACzB;AAED,mDAAmD;AACnD,MAAM,WAAW,IAAI;IACnB,QAAQ,EAAE,cAAc,CAAA;IACxB,QAAQ,EAAE,WAAW,EAAE,CAAA;IACvB,MAAM,EAAE,SAAS,EAAE,CAAA;CACpB;AAED,uCAAuC;AACvC,MAAM,WAAW,OAAO;IACtB,SAAS,EAAE,MAAM,CAAA;IACjB,IAAI,EAAE,MAAM,CAAA;IACZ,cAAc,EAAE,MAAM,GAAG,IAAI,CAAA;IAC7B,OAAO,EAAE,UAAU,EAAE,CAAA;CACtB;AAED,iEAAiE;AACjE,MAAM,WAAW,YAAY;IAC3B,QAAQ,EAAE,MAAM,CAAA;IAChB,SAAS,CAAC,EAAE,MAAM,CAAA;IAClB,CAAC,CAAC,EAAE,MAAM,CAAA;IACV,CAAC,CAAC,EAAE,MAAM,CAAA;IACV,MAAM,EAAE,MAAM,GAAG,OAAO,CAAA;CACzB;AAED,2DAA2D;AAC3D,MAAM,WAAW,WAAW;IAC1B,QAAQ,EAAE,MAAM,CAAA;IAChB,SAAS,EAAE,MAAM,CAAA;IACjB,IAAI,EAAE,MAAM,CAAA;IACZ,4EAA4E;IAC5E,QAAQ,EAAE,OAAO,CAAA;CAClB;AAED,wBAAwB;AACxB,MAAM,WAAW,aAAa;IAC5B,QAAQ,EAAE,MAAM,CAAA;IAChB,SAAS,CAAC,EAAE,MAAM,CAAA;IAClB,SAAS,EAAE,IAAI,GAAG,MAAM,GAAG,SAAS,GAAG,WAAW,CAAA;IAClD,2CAA2C;IAC3C,MAAM,EAAE,MAAM,CAAA;CACf;AAED,0DAA0D;AAC1D,MAAM,WAAW,UAAU;IACzB,QAAQ,EAAE,MAAM,CAAA;IAChB,IAAI,EAAE,MAAM,CAAA;CACb;AAED,6EAA6E;AAC7E,MAAM,WAAW,aAAa;IAC5B,QAAQ,EAAE,MAAM,CAAA;IAChB,6CAA6C;IAC7C,MAAM,EAAE,MAAM,CAAA;IACd,wCAAwC;IACxC,SAAS,EAAE,KAAK,GAAG,QAAQ,GAAG,MAAM,CAAA;IACpC,aAAa,EAAE,YAAY,CAAA;IAC3B,YAAY,EAAE,YAAY,CAAA;IAC1B,2EAA2E;IAC3E,QAAQ,CAAC,EAAE,OAAO,CAAA;IAClB,gCAAgC;IAChC,MAAM,CAAC,EAAE,MAAM,CAAA;CAChB;AAED,4CAA4C;AAC5C,MAAM,WAAW,aAAa;IAC5B,SAAS,EAAE,MAAM,CAAA;IACjB,cAAc,EAAE,MAAM,GAAG,IAAI,CAAA;CAC9B;AAED;;;;GAIG;AACH,MAAM,WAAW,cAAc;IAC7B,4EAA4E;IAC5E,QAAQ,CAAC,QAAQ,EAAE,MAAM,CAAA;IACzB,iFAAiF;IACjF,QAAQ,CAAC,SAAS,EAAE,OAAO,CAAA;IAC3B,kDAAkD;IAClD,QAAQ,CAAC,iBAAiB,CAAC,EAAE,MAAM,CAAA;IAEnC,uEAAuE;IACvE,WAAW,CAAC,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,UAAU,EAAE,CAAC,CAAA;IACxD,4EAA4E;IAC5E,IAAI,CAAC,GAAG,EAAE,SAAS,EAAE,OAAO,EAAE,MAAM,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,UAAU,CAAC,CAAA;IAChF,4EAA4E;IAC5E,IAAI,CAAC,GAAG,EAAE,SAAS,EAAE,WAAW,EAAE,MAAM,EAAE,QAAQ,EAAE,MAAM,EAAE,aAAa,EAAE,OAAO,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,IAAI,CAAC,CAAA;IACxH,2EAA2E;IAC3E,QAAQ,CAAC,QAAQ,EAAE,MAAM,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,cAAc,CAAC,CAAA;IACzE,gDAAgD;IAChD,KAAK,CAAC,OAAO,EAAE,YAAY,EAAE,aAAa,EAAE,OAAO,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,aAAa,CAAC,CAAA;IAClG,yCAAyC;IACzC,IAAI,CAAC,OAAO,EAAE,WAAW,EAAE,aAAa,EAAE,OAAO,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,aAAa,CAAC,CAAA;IAChG,2CAA2C;IAC3C,MAAM,CAAC,OAAO,EAAE,aAAa,EAAE,aAAa,EAAE,OAAO,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,aAAa,CAAC,CAAA;IACpG,mDAAmD;IACnD,GAAG,CAAC,OAAO,EAAE,UAAU,EAAE,aAAa,EAAE,OAAO,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,aAAa,CAAC,CAAA;IAC9F,yCAAyC;IACzC,IAAI,CAAC,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,OAAO,EAAE,CAAC,CAAA;IAC9C,8CAA8C;IAC9C,MAAM,CAAC,IAAI,EAAE,MAAM,EAAE,IAAI,EAAE,SAAS,MAAM,EAAE,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,aAAa,CAAC,CAAA;CAC5F;AAED,iEAAiE;AACjE,MAAM,WAAW,aAAa;IAC5B,QAAQ,EAAE,MAAM,CAAA;IAChB,aAAa,EAAE,MAAM,CAAA;IACrB,SAAS,EAAE,MAAM,CAAA;IACjB,EAAE,EAAE,MAAM,CAAA;IACV,IAAI,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAA;CAC9B;AAED,kFAAkF;AAClF,MAAM,MAAM,cAAc,GACtB;IAAE,EAAE,EAAE,IAAI,CAAC;IAAC,MAAM,EAAE,OAAO,CAAA;CAAE,GAC7B;IAAE,EAAE,EAAE,KAAK,CAAC;IAAC,KAAK,EAAE;QAAE,IAAI,EAAE,MAAM,CAAC;QAAC,OAAO,EAAE,MAAM,CAAA;KAAE,CAAA;CAAE,CAAA;AAE3D,gFAAgF;AAChF,wBAAgB,YAAY,CAAC,MAAM,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,EAAE,KAAK,EAAE,MAAM,EAAE,EAAE,EAAE,MAAM,GAAG,MAAM,CAM/F;AAED,yEAAyE;AACzE,wBAAgB,YAAY,CAAC,MAAM,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,EAAE,KAAK,EAAE,MAAM,EAAE,EAAE,EAAE,MAAM,GAAG,MAAM,CAM/F;AAED,+EAA+E;AAC/E,wBAAgB,YAAY,CAAC,MAAM,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,EAAE,KAAK,EAAE,MAAM,EAAE,EAAE,EAAE,MAAM,GAAG,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAMhH;AAED,8EAA8E;AAC9E,wBAAgB,iBAAiB,CAAC,KAAK,EAAE,OAAO,EAAE,EAAE,EAAE,MAAM,GAAG,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAKrF;AAED,oFAAoF;AACpF,wBAAgB,kBAAkB,CAAC,KAAK,EAAE,OAAO,EAAE,EAAE,EAAE,MAAM,GAAG,YAAY,CAW3E"}
1
+ {"version":3,"file":"types.d.ts","sourceRoot":"","sources":["../../../src/platform/types.ts"],"names":[],"mappings":"AAAA;;;;;;;;GAQG;AAEH,OAAO,KAAK,EAAE,YAAY,EAAE,MAAM,cAAc,CAAA;AAEhD,iFAAiF;AACjF,qBAAa,aAAc,SAAQ,KAAK;IAGT,QAAQ,CAAC,IAAI,EAAE,MAAM;IAFlD,+DAA+D;IAC/D,4DAA4D;gBAChD,OAAO,EAAE,MAAM,EAAW,IAAI,EAAE,MAAM;CAInD;AAED,+DAA+D;AAC/D,MAAM,WAAW,SAAS;IACxB,8CAA8C;IAC9C,QAAQ,CAAC,EAAE,MAAM,CAAA;IACjB,+EAA+E;IAC/E,WAAW,CAAC,EAAE,MAAM,CAAA;IACpB,yBAAyB;IACzB,SAAS,CAAC,EAAE,MAAM,CAAA;CACnB;AAED,wDAAwD;AACxD,MAAM,MAAM,WAAW,GAAG,SAAS,GAAG,SAAS,CAAA;AAE/C,+FAA+F;AAC/F,MAAM,MAAM,YAAY,GAAG,QAAQ,GAAG,WAAW,CAAA;AAEjD;;;;;;;;GAQG;AACH,MAAM,MAAM,aAAa,GAAG,yBAAyB,GAAG,aAAa,GAAG,gBAAgB,CAAA;AAExF,yDAAyD;AACzD,MAAM,WAAW,IAAI;IACnB,CAAC,EAAE,MAAM,CAAA;IACT,CAAC,EAAE,MAAM,CAAA;IACT,KAAK,EAAE,MAAM,CAAA;IACb,MAAM,EAAE,MAAM,CAAA;CACf;AAED,wCAAwC;AACxC,MAAM,WAAW,UAAU;IACzB,QAAQ,EAAE,MAAM,CAAA;IAChB,SAAS,EAAE,MAAM,GAAG,IAAI,CAAA;IACxB,KAAK,EAAE,MAAM,CAAA;IACb,SAAS,EAAE,MAAM,CAAA;IACjB;;;;OAIG;IACH,IAAI,EAAE,IAAI,GAAG,IAAI,CAAA;IACjB,yFAAyF;IACzF,SAAS,EAAE,OAAO,CAAA;IAClB,cAAc,EAAE,MAAM,GAAG,IAAI,CAAA;IAC7B,OAAO,EAAE,OAAO,CAAA;CACjB;AAED,4DAA4D;AAC5D,MAAM,WAAW,WAAW;IAC1B,kEAAkE;IAClE,SAAS,EAAE,MAAM,CAAA;IACjB,6CAA6C;IAC7C,WAAW,EAAE,MAAM,CAAA;IACnB,8CAA8C;IAC9C,IAAI,EAAE,MAAM,CAAA;IACZ,uCAAuC;IACvC,YAAY,EAAE,MAAM,CAAA;IACpB,gDAAgD;IAChD,IAAI,EAAE,IAAI,CAAA;IACV,sCAAsC;IACtC,OAAO,EAAE,OAAO,CAAA;IAChB,mGAAmG;IACnG,QAAQ,EAAE,MAAM,EAAE,CAAA;CACnB;AAED,uEAAuE;AACvE,MAAM,WAAW,SAAS;IACxB,kDAAkD;IAClD,KAAK,EAAE,MAAM,CAAA;IACb,2BAA2B;IAC3B,CAAC,EAAE,MAAM,CAAA;IACT,2BAA2B;IAC3B,CAAC,EAAE,MAAM,CAAA;IACT,sDAAsD;IACtD,KAAK,EAAE,MAAM,CAAA;CACd;AAED,oFAAoF;AACpF,MAAM,WAAW,cAAc;IAC7B,QAAQ,EAAE,MAAM,CAAA;IAChB,SAAS,EAAE,MAAM,CAAA;IACjB,cAAc,EAAE,MAAM,GAAG,IAAI,CAAA;IAC7B,KAAK,EAAE,MAAM,CAAA;IACb,SAAS,EAAE,MAAM,CAAA;IACjB,IAAI,EAAE,IAAI,CAAA;IACV,0DAA0D;IAC1D,UAAU,EAAE,OAAO,CAAA;IACnB,4EAA4E;IAC5E,QAAQ,EAAE,MAAM,CAAA;IAChB,2CAA2C;IAC3C,QAAQ,EAAE,MAAM,CAAA;IAChB,iDAAiD;IACjD,YAAY,EAAE,MAAM,CAAA;CACrB;AAED,uEAAuE;AACvE,MAAM,WAAW,UAAU;IACzB,iCAAiC;IACjC,SAAS,EAAE,MAAM,CAAA;IACjB;;;;OAIG;IACH,KAAK,EAAE,MAAM,CAAA;IACb,uEAAuE;IACvE,MAAM,EAAE,MAAM,CAAA;IACd,8CAA8C;IAC9C,aAAa,EAAE,aAAa,CAAA;IAC5B,sFAAsF;IACtF,KAAK,EAAE,YAAY,CAAA;IACnB,QAAQ,EAAE,cAAc,CAAA;CACzB;AAED,mDAAmD;AACnD,MAAM,WAAW,IAAI;IACnB,QAAQ,EAAE,cAAc,CAAA;IACxB,QAAQ,EAAE,WAAW,EAAE,CAAA;IACvB,MAAM,EAAE,SAAS,EAAE,CAAA;CACpB;AAED,uCAAuC;AACvC,MAAM,WAAW,OAAO;IACtB,SAAS,EAAE,MAAM,CAAA;IACjB,IAAI,EAAE,MAAM,CAAA;IACZ,cAAc,EAAE,MAAM,GAAG,IAAI,CAAA;IAC7B,OAAO,EAAE,UAAU,EAAE,CAAA;CACtB;AAED,iEAAiE;AACjE,MAAM,WAAW,YAAY;IAC3B,QAAQ,EAAE,MAAM,CAAA;IAChB,SAAS,CAAC,EAAE,MAAM,CAAA;IAClB,CAAC,CAAC,EAAE,MAAM,CAAA;IACV,CAAC,CAAC,EAAE,MAAM,CAAA;IACV,MAAM,EAAE,MAAM,GAAG,OAAO,CAAA;CACzB;AAED,2DAA2D;AAC3D,MAAM,WAAW,WAAW;IAC1B,QAAQ,EAAE,MAAM,CAAA;IAChB,SAAS,EAAE,MAAM,CAAA;IACjB,IAAI,EAAE,MAAM,CAAA;IACZ,4EAA4E;IAC5E,QAAQ,EAAE,OAAO,CAAA;CAClB;AAED,wBAAwB;AACxB,MAAM,WAAW,aAAa;IAC5B,QAAQ,EAAE,MAAM,CAAA;IAChB,SAAS,CAAC,EAAE,MAAM,CAAA;IAClB,SAAS,EAAE,IAAI,GAAG,MAAM,GAAG,SAAS,GAAG,WAAW,CAAA;IAClD,2CAA2C;IAC3C,MAAM,EAAE,MAAM,CAAA;CACf;AAED,0DAA0D;AAC1D,MAAM,WAAW,UAAU;IACzB,QAAQ,EAAE,MAAM,CAAA;IAChB,IAAI,EAAE,MAAM,CAAA;CACb;AAED,6EAA6E;AAC7E,MAAM,WAAW,aAAa;IAC5B,QAAQ,EAAE,MAAM,CAAA;IAChB,6CAA6C;IAC7C,MAAM,EAAE,MAAM,CAAA;IACd,wCAAwC;IACxC,SAAS,EAAE,KAAK,GAAG,QAAQ,GAAG,MAAM,CAAA;IACpC,aAAa,EAAE,YAAY,CAAA;IAC3B,YAAY,EAAE,YAAY,CAAA;IAC1B,2EAA2E;IAC3E,QAAQ,CAAC,EAAE,OAAO,CAAA;IAClB,gCAAgC;IAChC,MAAM,CAAC,EAAE,MAAM,CAAA;CAChB;AAED,4CAA4C;AAC5C,MAAM,WAAW,aAAa;IAC5B,SAAS,EAAE,MAAM,CAAA;IACjB,cAAc,EAAE,MAAM,GAAG,IAAI,CAAA;CAC9B;AAED;;;;GAIG;AACH,MAAM,WAAW,cAAc;IAC7B,4EAA4E;IAC5E,QAAQ,CAAC,QAAQ,EAAE,MAAM,CAAA;IACzB,iFAAiF;IACjF,QAAQ,CAAC,SAAS,EAAE,OAAO,CAAA;IAC3B,kDAAkD;IAClD,QAAQ,CAAC,iBAAiB,CAAC,EAAE,MAAM,CAAA;IAEnC,uEAAuE;IACvE,WAAW,CAAC,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,UAAU,EAAE,CAAC,CAAA;IACxD;;;;OAIG;IACH,IAAI,CAAC,GAAG,EAAE,SAAS,EAAE,OAAO,EAAE,MAAM,EAAE,KAAK,EAAE,WAAW,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,UAAU,CAAC,CAAA;IACpG,4EAA4E;IAC5E,IAAI,CAAC,GAAG,EAAE,SAAS,EAAE,WAAW,EAAE,MAAM,EAAE,QAAQ,EAAE,MAAM,EAAE,aAAa,EAAE,OAAO,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,IAAI,CAAC,CAAA;IACxH,2EAA2E;IAC3E,QAAQ,CAAC,QAAQ,EAAE,MAAM,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,cAAc,CAAC,CAAA;IACzE,gDAAgD;IAChD,KAAK,CAAC,OAAO,EAAE,YAAY,EAAE,aAAa,EAAE,OAAO,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,aAAa,CAAC,CAAA;IAClG,yCAAyC;IACzC,IAAI,CAAC,OAAO,EAAE,WAAW,EAAE,aAAa,EAAE,OAAO,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,aAAa,CAAC,CAAA;IAChG,2CAA2C;IAC3C,MAAM,CAAC,OAAO,EAAE,aAAa,EAAE,aAAa,EAAE,OAAO,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,aAAa,CAAC,CAAA;IACpG,mDAAmD;IACnD,GAAG,CAAC,OAAO,EAAE,UAAU,EAAE,aAAa,EAAE,OAAO,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,aAAa,CAAC,CAAA;IAC9F,yCAAyC;IACzC,IAAI,CAAC,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,OAAO,EAAE,CAAC,CAAA;IAC9C,8CAA8C;IAC9C,MAAM,CAAC,IAAI,EAAE,MAAM,EAAE,IAAI,EAAE,SAAS,MAAM,EAAE,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,aAAa,CAAC,CAAA;CAC5F;AAED,iEAAiE;AACjE,MAAM,WAAW,aAAa;IAC5B,QAAQ,EAAE,MAAM,CAAA;IAChB,aAAa,EAAE,MAAM,CAAA;IACrB,SAAS,EAAE,MAAM,CAAA;IACjB,EAAE,EAAE,MAAM,CAAA;IACV,IAAI,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAA;CAC9B;AAED,kFAAkF;AAClF,MAAM,MAAM,cAAc,GACtB;IAAE,EAAE,EAAE,IAAI,CAAC;IAAC,MAAM,EAAE,OAAO,CAAA;CAAE,GAC7B;IAAE,EAAE,EAAE,KAAK,CAAC;IAAC,KAAK,EAAE;QAAE,IAAI,EAAE,MAAM,CAAC;QAAC,OAAO,EAAE,MAAM,CAAA;KAAE,CAAA;CAAE,CAAA;AAE3D,gFAAgF;AAChF,wBAAgB,YAAY,CAAC,MAAM,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,EAAE,KAAK,EAAE,MAAM,EAAE,EAAE,EAAE,MAAM,GAAG,MAAM,CAM/F;AAED,yEAAyE;AACzE,wBAAgB,YAAY,CAAC,MAAM,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,EAAE,KAAK,EAAE,MAAM,EAAE,EAAE,EAAE,MAAM,GAAG,MAAM,CAM/F;AAED,+EAA+E;AAC/E,wBAAgB,YAAY,CAAC,MAAM,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,EAAE,KAAK,EAAE,MAAM,EAAE,EAAE,EAAE,MAAM,GAAG,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAMhH;AAED,8EAA8E;AAC9E,wBAAgB,iBAAiB,CAAC,KAAK,EAAE,OAAO,EAAE,EAAE,EAAE,MAAM,GAAG,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAKrF;AAED,oFAAoF;AACpF,wBAAgB,kBAAkB,CAAC,KAAK,EAAE,OAAO,EAAE,EAAE,EAAE,MAAM,GAAG,YAAY,CAW3E"}
@@ -1 +1 @@
1
- {"version":3,"file":"tools.d.ts","sourceRoot":"","sources":["../../src/tools.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;GAWG;AAEH,OAAO,KAAK,EAAE,OAAO,EAAE,MAAM,qBAAqB,CAAA;AAKlD,OAAO,EAAE,cAAc,EAAE,MAAM,cAAc,CAAA;AAC7C,OAAO,KAAK,EAAE,cAAc,EAAE,MAAM,aAAa,CAAA;AAEjD,OAAO,EAAE,gBAAgB,EAAE,MAAM,cAAc,CAAA;AAG/C,OAAO,EAA0B,KAAK,WAAW,EAAE,KAAK,uBAAuB,EAAE,MAAM,mBAAmB,CAAA;AAC1G,OAAO,KAAK,EAAE,cAAc,EAA0B,IAAI,EAAyC,MAAM,qBAAqB,CAAA;AAE9H,wEAAwE;AACxE,MAAM,WAAW,YAAY;IAC3B,4BAA4B;IAC5B,QAAQ,CAAC,GAAG,EAAE,OAAO,CAAA;IACrB,8BAA8B;IAC9B,QAAQ,CAAC,MAAM,EAAE,cAAc,CAAA;IAC/B,kCAAkC;IAClC,QAAQ,CAAC,OAAO,EAAE,cAAc,CAAA;IAChC,6BAA6B;IAC7B,QAAQ,CAAC,YAAY,EAAE,gBAAgB,CAAA;IACvC,2BAA2B;IAC3B,QAAQ,CAAC,OAAO,EAAE,cAAc,CAAA;IAChC,0CAA0C;IAC1C,QAAQ,CAAC,GAAG,EAAE,WAAW,CAAA;IACzB,8CAA8C;IAC9C,QAAQ,CAAC,SAAS,EAAE,uBAAuB,CAAA;CAC5C;AAED,4EAA4E;AAC5E,MAAM,WAAW,mBAAmB;IAClC,QAAQ,EAAE,MAAM,CAAA;IAChB,SAAS,EAAE,MAAM,CAAA;IACjB,KAAK,EAAE,MAAM,CAAA;IACb,SAAS,EAAE,MAAM,CAAA;IACjB,cAAc,EAAE,MAAM,GAAG,IAAI,CAAA;IAC7B,IAAI,EAAE,IAAI,CAAA;IACV,UAAU,EAAE,OAAO,CAAA;CACpB;AAyFD;;;;;;;;GAQG;AACH,wBAAgB,cAAc,CAAC,QAAQ,EAAE,YAAY,mDAoIpD;AAED;;;;;;;GAOG;AACH,wBAAgB,cAAc,CAAC,QAAQ,EAAE,YAAY,mDA+IpD;AAED;;;;;;;;;;GAUG;AACH,wBAAgB,cAAc,CAAC,QAAQ,EAAE,YAAY,mDAsJpD;AAiED;;;;;;;GAOG;AACH,wBAAgB,SAAS,CAAC,QAAQ,EAAE,YAAY,mDAoD/C;AAED;;;;;;;GAOG;AACH,wBAAgB,QAAQ,CAAC,QAAQ,EAAE,YAAY,mDAyC9C;AAED;;;;;;GAMG;AACH,wBAAgB,UAAU,CAAC,QAAQ,EAAE,YAAY,mDA0ChD;AAED;;;;;;;GAOG;AACH,wBAAgB,OAAO,CAAC,QAAQ,EAAE,YAAY,mDAkC7C;AAED;;;;;;GAMG;AACH,wBAAgB,WAAW,CAAC,QAAQ,EAAE,YAAY,mDA+EjD;AAED;;;;;;GAMG;AACH,wBAAgB,aAAa,CAAC,QAAQ,EAAE,YAAY,mDAwCnD;AAED,oDAAoD;AACpD,wBAAgB,QAAQ,CAAC,QAAQ,EAAE,YAAY,qDAY9C"}
1
+ {"version":3,"file":"tools.d.ts","sourceRoot":"","sources":["../../src/tools.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;GAWG;AAEH,OAAO,KAAK,EAAE,OAAO,EAAE,MAAM,qBAAqB,CAAA;AAKlD,OAAO,EAAE,cAAc,EAAE,MAAM,cAAc,CAAA;AAC7C,OAAO,KAAK,EAAE,cAAc,EAAE,MAAM,aAAa,CAAA;AAEjD,OAAO,EAAE,gBAAgB,EAAE,MAAM,cAAc,CAAA;AAG/C,OAAO,EAA0B,KAAK,WAAW,EAAE,KAAK,uBAAuB,EAAE,MAAM,mBAAmB,CAAA;AAC1G,OAAO,KAAK,EAAE,cAAc,EAA0B,IAAI,EAAyC,MAAM,qBAAqB,CAAA;AAE9H,wEAAwE;AACxE,MAAM,WAAW,YAAY;IAC3B,4BAA4B;IAC5B,QAAQ,CAAC,GAAG,EAAE,OAAO,CAAA;IACrB,8BAA8B;IAC9B,QAAQ,CAAC,MAAM,EAAE,cAAc,CAAA;IAC/B,kCAAkC;IAClC,QAAQ,CAAC,OAAO,EAAE,cAAc,CAAA;IAChC,6BAA6B;IAC7B,QAAQ,CAAC,YAAY,EAAE,gBAAgB,CAAA;IACvC,2BAA2B;IAC3B,QAAQ,CAAC,OAAO,EAAE,cAAc,CAAA;IAChC,0CAA0C;IAC1C,QAAQ,CAAC,GAAG,EAAE,WAAW,CAAA;IACzB,8CAA8C;IAC9C,QAAQ,CAAC,SAAS,EAAE,uBAAuB,CAAA;CAC5C;AAED,4EAA4E;AAC5E,MAAM,WAAW,mBAAmB;IAClC,QAAQ,EAAE,MAAM,CAAA;IAChB,SAAS,EAAE,MAAM,CAAA;IACjB,KAAK,EAAE,MAAM,CAAA;IACb,SAAS,EAAE,MAAM,CAAA;IACjB,cAAc,EAAE,MAAM,GAAG,IAAI,CAAA;IAC7B,IAAI,EAAE,IAAI,CAAA;IACV,UAAU,EAAE,OAAO,CAAA;CACpB;AA0FD;;;;;;;;GAQG;AACH,wBAAgB,cAAc,CAAC,QAAQ,EAAE,YAAY,mDAiJpD;AAED;;;;;;;GAOG;AACH,wBAAgB,cAAc,CAAC,QAAQ,EAAE,YAAY,mDA+IpD;AAED;;;;;;;;;;GAUG;AACH,wBAAgB,cAAc,CAAC,QAAQ,EAAE,YAAY,mDAsJpD;AAiED;;;;;;;GAOG;AACH,wBAAgB,SAAS,CAAC,QAAQ,EAAE,YAAY,mDAoD/C;AAED;;;;;;;GAOG;AACH,wBAAgB,QAAQ,CAAC,QAAQ,EAAE,YAAY,mDAyC9C;AAED;;;;;;GAMG;AACH,wBAAgB,UAAU,CAAC,QAAQ,EAAE,YAAY,mDA0ChD;AAED;;;;;;;GAOG;AACH,wBAAgB,OAAO,CAAC,QAAQ,EAAE,YAAY,mDAkC7C;AAED;;;;;;GAMG;AACH,wBAAgB,WAAW,CAAC,QAAQ,EAAE,YAAY,mDAuFjD;AAED;;;;;;GAMG;AACH,wBAAgB,aAAa,CAAC,QAAQ,EAAE,YAAY,mDAwCnD;AAED,oDAAoD;AACpD,wBAAgB,QAAQ,CAAC,QAAQ,EAAE,YAAY,qDAY9C"}
@@ -7,7 +7,7 @@
7
7
  * @module dsh-click/version
8
8
  */
9
9
  /** Plugin version advertised in the helper-protocol handshake. */
10
- export declare const VERSION = "0.3.28";
10
+ export declare const VERSION = "0.3.29";
11
11
  /** The helper protocol revision; bump only on incompatible wire changes. */
12
12
  export declare const HELPER_PROTOCOL_VERSION = 1;
13
13
  //# sourceMappingURL=version.d.ts.map
@@ -1,4 +1,4 @@
1
- # dsh-click native helper (Windows PowerShell 5.1).
1
+ # dsh-click native helper (Windows PowerShell 5.1).
2
2
  #
3
3
  # One request = one process: JSON request on stdin, JSON response on stdout
4
4
  # (exit code 0 even for handled failures; the error envelope is the response).
@@ -6,7 +6,8 @@
6
6
  # any other stream, and nothing here steals foreground focus unless the
7
7
  # `focusFallback` flag is explicitly true (config-gated, approval-gated).
8
8
  #
9
- # Operations: windows, apps, shot, tree, snapshot, click, type, scroll, key, launch.
9
+ # Operations: windows, apps, metrics, shot, tree, snapshot, click, type, scroll,
10
+ # key, launch.
10
11
  # The helper is NOT a security boundary by itself - the harness-side
11
12
  # freshness check, approval gate, and process-identity check are. This file
12
13
  # only refuses to invent global input: actions address one window by handle.
@@ -150,6 +151,13 @@ public static class DshClickWin32 {
150
151
  [DllImport("user32.dll")]
151
152
  public static extern int GetSystemMetrics(int index);
152
153
 
154
+ // PrintWindow: ask a window to paint itself into a DC. Flag 2 is
155
+ // PW_RENDERFULLCONTENT (Windows 8.1+), which makes DWM/DirectComposition
156
+ // windows render their real content instead of a blank or chrome-only
157
+ // frame; flag 0 is the legacy path for windows that ignore the newer flag.
158
+ [DllImport("user32.dll", SetLastError = true)]
159
+ public static extern bool PrintWindow(IntPtr hWnd, IntPtr hdcBlt, uint nFlags);
160
+
153
161
  // DPI awareness. The helper is hosted by powershell.exe, whose manifest
154
162
  // declares no DPI awareness, so GetWindowRect/GetSystemMetrics/GetClientRect
155
163
  // return logical pixels while UIA BoundingRectangle and the screen DC are
@@ -346,8 +354,9 @@ function Resolve-Window($target) {
346
354
  $hwnd = ConvertTo-Hwnd $target.windowId
347
355
  if ($hwnd -ne [IntPtr]::Zero) {
348
356
  $rect = Get-WindowRectInfo $hwnd
349
- if ($rect.width -le 0 -and $rect.height -le 0) {
357
+ if ($rect.width -le 0 -or $rect.height -le 0) {
350
358
  # A handle that no longer resolves to a window.
359
+ $script:ErrorCode = 'WINDOW_NOT_FOUND'
351
360
  throw "window $($target.windowId) not found"
352
361
  }
353
362
  return $hwnd
@@ -357,6 +366,7 @@ function Resolve-Window($target) {
357
366
  foreach ($candidate in [DshClickWin32]::EnumVisibleWindows()) {
358
367
  if ([DshClickWin32]::WindowTitle($candidate) -like "*$title*") { return $candidate }
359
368
  }
369
+ $script:ErrorCode = 'WINDOW_NOT_FOUND'
360
370
  throw "no visible window matches title '$title'"
361
371
  }
362
372
  $processId = $target.processId
@@ -366,13 +376,41 @@ function Resolve-Window($target) {
366
376
  [void][DshClickWin32]::GetWindowThreadProcessId($candidate, [ref]$pidValue)
367
377
  if ([int]$pidValue -eq [int]$processId) { return $candidate }
368
378
  }
379
+ $script:ErrorCode = 'WINDOW_NOT_FOUND'
369
380
  throw "no visible window owned by process $processId"
370
381
  }
371
382
  $foreground = [DshClickWin32]::GetForegroundWindow()
372
- if ($foreground -eq [IntPtr]::Zero) { throw 'no foreground window and no target given' }
383
+ if ($foreground -eq [IntPtr]::Zero) {
384
+ $script:ErrorCode = 'WINDOW_NOT_FOUND'
385
+ throw 'no foreground window and no target given'
386
+ }
373
387
  return $foreground
374
388
  }
375
389
 
390
+ # One target resolver for the observer ops. `tree`/`shot` always read
391
+ # `args.target`; `snapshot` historically read `args.windowId`. Both spellings
392
+ # now go through here, so `{"op":"snapshot","args":{"target":{"windowId":N}}}`
393
+ # addresses window N instead of silently falling back to the foreground window.
394
+ # `args.target` wins when both are present; the legacy top-level `windowId` is
395
+ # only a fallback, so old callers keep working.
396
+ function Resolve-OpTarget($opArgs) {
397
+ $target = @{}
398
+ if ($opArgs.target -ne $null) {
399
+ $candidate = $opArgs.target
400
+ if ($candidate.windowId -ne $null) { $target.windowId = $candidate.windowId }
401
+ if ($candidate.windowTitle -ne $null) { $target.windowTitle = $candidate.windowTitle }
402
+ if ($candidate.processId -ne $null) { $target.processId = $candidate.processId }
403
+ }
404
+ if ($target.windowId -eq $null -and $target.windowTitle -eq $null -and $target.processId -eq $null) {
405
+ if ($opArgs.windowId -ne $null) { $target.windowId = $opArgs.windowId }
406
+ }
407
+ return $target
408
+ }
409
+
410
+ function Test-TargetPresent($target) {
411
+ return ($target.windowId -ne $null) -or ($target.windowTitle -ne $null) -or ($target.processId -ne $null)
412
+ }
413
+
376
414
  # ---------------------------------------------------------------------------
377
415
  # Accessibility walk (canonical: the same walk builds treeHash everywhere).
378
416
  # ---------------------------------------------------------------------------
@@ -458,30 +496,214 @@ function Get-TreeHashText($nodes) {
458
496
  # ---------------------------------------------------------------------------
459
497
  # Screenshot.
460
498
  # ---------------------------------------------------------------------------
461
- function Get-WindowBitmap($hwnd) {
499
+ # A window is captured in three steps, best first:
500
+ # 1. PrintWindow(hwnd, hdc, 2) PW_RENDERFULLCONTENT - the window paints its
501
+ # own content, so identity and pixels do not depend on occlusion/Z-order.
502
+ # 2. PrintWindow(hwnd, hdc, 0) legacy windows that ignore the newer flag.
503
+ # 3. CopyFromScreen last resort; correct only while unobscured.
504
+ # PrintWindow reports success for frames that carry no content at all (a blank
505
+ # Electron frame, or a Chromium window that paints only its title-bar buttons),
506
+ # so every candidate goes through Test-BitmapLooksBlank before it is accepted.
507
+ # Measured on Windows 11: PW(2) carries content for WinUI3/Electron/Chromium;
508
+ # PW(0/1) is blank or chrome-only for the latter two. The thresholds below are
509
+ # fitted to that matrix: a pure-color check alone is fooled by a gray frame with
510
+ # three title-bar buttons, so the color-count floor is 6 quantized colors.
511
+ $script:CaptureInset = 40 # px trimmed from each edge before sampling
512
+ $script:CaptureGrid = 16 # interior sample grid (16x16)
513
+ $script:CaptureDarkGrid = 32 # full-frame sample grid for the dark-ratio test
514
+
515
+ function Get-BitmapLuminance($color) {
516
+ return (0.299 * [double]$color.R) + (0.587 * [double]$color.G) + (0.114 * [double]$color.B)
517
+ }
518
+
519
+ # True when a captured frame carries no usable content: a flat interior, an
520
+ # almost entirely black frame, or (on the no-flag PrintWindow path only) a few
521
+ # flat colors with chrome painted in a corner.
522
+ #
523
+ # The color-count rule is `strict` because a legitimate all-white page can also
524
+ # sample as 3 quantized colors: measured on a 2582x1550 Chrome new-tab window,
525
+ # PrintWindow(2) gives 3 colors but a luminance sd of 9.6, while the chrome-only
526
+ # frame the no-flag path paints on Chromium gives 4-5 colors with sd 13-14. The
527
+ # two are indistinguishable by color count alone, so the count rule is applied
528
+ # only where chrome-only frames are known to come from (PrintWindow flag 0).
529
+ # A blank all-black frame is caught on every path by the sd/dark-ratio rules.
530
+ function Test-BitmapLooksBlank($bitmap, $strict) {
531
+ if ($null -eq $bitmap -or $bitmap.Width -le 0 -or $bitmap.Height -le 0) { return $true }
532
+ $width = $bitmap.Width
533
+ $height = $bitmap.Height
534
+
535
+ # Interior region: drop the outer band (border, shadow, title-bar buttons)
536
+ # where a failed PrintWindow can still paint a plausible-looking frame.
537
+ $insetX = [Math]::Min($script:CaptureInset, [int][Math]::Floor(($width - 1) / 2))
538
+ $insetY = [Math]::Min($script:CaptureInset, [int][Math]::Floor(($height - 1) / 2))
539
+ $left = $insetX
540
+ $top = $insetY
541
+ $right = $width - 1 - $insetX
542
+ $bottom = $height - 1 - $insetY
543
+ if (($right - $left) -lt 8 -or ($bottom - $top) -lt 8) {
544
+ # Tiny window: the inset leaves nothing to sample, so use the full frame.
545
+ $left = 0; $top = 0; $right = $width - 1; $bottom = $height - 1
546
+ }
547
+ $spanX = [Math]::Max(1, $right - $left)
548
+ $spanY = [Math]::Max(1, $bottom - $top)
549
+ $stepX = [Math]::Max(1, [int][Math]::Ceiling($spanX / $script:CaptureGrid))
550
+ $stepY = [Math]::Max(1, [int][Math]::Ceiling($spanY / $script:CaptureGrid))
551
+
552
+ $colors = @{}
553
+ $sum = 0.0
554
+ $sumSquares = 0.0
555
+ $count = 0
556
+ for ($x = $left; $x -le $right; $x += $stepX) {
557
+ for ($y = $top; $y -le $bottom; $y += $stepY) {
558
+ $color = $bitmap.GetPixel($x, $y)
559
+ # 5-bit quantization: UI colors cluster, and an exact ARGB count is both
560
+ # noisy and slower than needed for a blank/non-blank verdict.
561
+ $key = ([int]($color.R -band 0xF8) -shl 16) -bor ([int]($color.G -band 0xF8) -shl 8) -bor [int]($color.B -band 0xF8)
562
+ $colors[$key] = $true
563
+ $luminance = Get-BitmapLuminance $color
564
+ $sum += $luminance
565
+ $sumSquares += $luminance * $luminance
566
+ $count += 1
567
+ }
568
+ }
569
+ if ($count -eq 0) { return $true }
570
+ $mean = $sum / $count
571
+ $variance = [Math]::Max(0.0, ($sumSquares / $count) - ($mean * $mean))
572
+ $deviation = [Math]::Sqrt($variance)
573
+ if ($deviation -lt 2) { return $true }
574
+ if ($strict -and $colors.Count -le 6) { return $true }
575
+
576
+ # Full-frame dark ratio: a PrintWindow frame that painted nothing at all.
577
+ $dark = 0
578
+ $samples = 0
579
+ $darkStepX = [Math]::Max(1, [int][Math]::Ceiling($width / $script:CaptureDarkGrid))
580
+ $darkStepY = [Math]::Max(1, [int][Math]::Ceiling($height / $script:CaptureDarkGrid))
581
+ for ($x = 0; $x -lt $width; $x += $darkStepX) {
582
+ for ($y = 0; $y -lt $height; $y += $darkStepY) {
583
+ if ((Get-BitmapLuminance $bitmap.GetPixel($x, $y)) -lt 8) { $dark += 1 }
584
+ $samples += 1
585
+ }
586
+ }
587
+ if ($samples -gt 0 -and (($dark * 100) -ge (99 * $samples))) { return $true }
588
+ return $false
589
+ }
590
+
591
+ function New-CaptureBitmap($width, $height) {
592
+ if ($width -le 0 -or $height -le 0) { throw "cannot allocate a ${width}x${height} capture bitmap" }
593
+ return New-Object System.Drawing.Bitmap($width, $height)
594
+ }
595
+
596
+ # One PrintWindow attempt; returns $null when the API refuses, otherwise the
597
+ # bitmap (which may still be blank - the caller decides).
598
+ function Get-PrintWindowFrame($hwnd, $rectInfo, $flags) {
599
+ $bitmap = New-CaptureBitmap $rectInfo.width $rectInfo.height
600
+ $graphics = [System.Drawing.Graphics]::FromImage($bitmap)
601
+ $hdc = [IntPtr]::Zero
602
+ $painted = $false
603
+ try {
604
+ $hdc = $graphics.GetHdc()
605
+ $painted = [DshClickWin32]::PrintWindow($hwnd, $hdc, [uint32]$flags)
606
+ } catch {
607
+ $painted = $false
608
+ } finally {
609
+ if ($hdc -ne [IntPtr]::Zero) {
610
+ try { $graphics.ReleaseHdc($hdc) } catch { }
611
+ }
612
+ $graphics.Dispose()
613
+ }
614
+ if (-not $painted) {
615
+ $bitmap.Dispose()
616
+ return $null
617
+ }
618
+ return $bitmap
619
+ }
620
+
621
+ # Capture one window: returns @{ bitmap; method; rect }. The caller owns the
622
+ # bitmap and must dispose it. Minimized windows are refused up front: IsIconic
623
+ # windows report a placeholder rect, and PrintWindow paints only the small
624
+ # window-icon band even when it answers true, so there is no honest image.
625
+ function Get-WindowCapture($hwnd) {
462
626
  $rectInfo = Get-WindowRectInfo $hwnd
463
- if ($rectInfo.width -le 0 -or $rectInfo.height -le 0) { throw "window $hwnd has no drawable rect" }
464
- $bitmap = New-Object System.Drawing.Bitmap($rectInfo.width, $rectInfo.height)
627
+ if ($rectInfo.width -le 0 -or $rectInfo.height -le 0) {
628
+ $script:ErrorCode = 'WINDOW_NOT_FOUND'
629
+ throw "window $hwnd has no drawable rect"
630
+ }
631
+ if ([DshClickWin32]::IsIconic($hwnd)) {
632
+ $script:ErrorCode = 'WINDOW_MINIMIZED'
633
+ throw "window $hwnd is minimized; a minimized window has no capturable pixels - restore it, then re-observe"
634
+ }
635
+
636
+ $candidate = Get-PrintWindowFrame $hwnd $rectInfo 2 # PW_RENDERFULLCONTENT
637
+ if ($candidate -ne $null) {
638
+ if (-not (Test-BitmapLooksBlank $candidate $false)) {
639
+ return @{ bitmap = $candidate; method = 'printwindow-fullcontent'; rect = $rectInfo }
640
+ }
641
+ $candidate.Dispose()
642
+ }
643
+ $candidate = Get-PrintWindowFrame $hwnd $rectInfo 0 # legacy PrintWindow
644
+ if ($candidate -ne $null) {
645
+ if (-not (Test-BitmapLooksBlank $candidate $true)) {
646
+ return @{ bitmap = $candidate; method = 'printwindow'; rect = $rectInfo }
647
+ }
648
+ $candidate.Dispose()
649
+ }
650
+
651
+ # Screen-region copy: the old path, kept for windows that refuse PrintWindow.
652
+ $bitmap = New-CaptureBitmap $rectInfo.width $rectInfo.height
465
653
  $graphics = [System.Drawing.Graphics]::FromImage($bitmap)
466
654
  try {
467
655
  $graphics.CopyFromScreen($rectInfo.x, $rectInfo.y, 0, 0, (New-Object System.Drawing.Size($rectInfo.width, $rectInfo.height)))
656
+ } catch {
657
+ $bitmap.Dispose()
658
+ throw
468
659
  } finally {
469
660
  $graphics.Dispose()
470
661
  }
471
- return $bitmap
662
+ return @{ bitmap = $bitmap; method = 'copyfromscreen'; rect = $rectInfo }
472
663
  }
473
664
 
474
- function Get-PrimaryBitmap() {
475
- $width = [DshClickWin32]::GetSystemMetrics(0) # SM_CXSCREEN
476
- $height = [DshClickWin32]::GetSystemMetrics(1) # SM_CYSCREEN
477
- $bitmap = New-Object System.Drawing.Bitmap($width, $height)
665
+ # Resolve the requested screen rectangle. `virtual` is the whole desktop
666
+ # (negative origins on multi-monitor setups) and stays experimental: only the
667
+ # single-monitor case is covered by the verification runs, and BitBlt across
668
+ # negative virtual-screen coordinates is untested here. If the virtual metrics
669
+ # are unusable, fall back to the primary screen rather than failing the call.
670
+ function Get-ScreenMetrics($scope) {
671
+ if ($scope -eq 'virtual') {
672
+ $virtualX = [int][DshClickWin32]::GetSystemMetrics(76) # SM_XVIRTUALSCREEN
673
+ $virtualY = [int][DshClickWin32]::GetSystemMetrics(77) # SM_YVIRTUALSCREEN
674
+ $virtualWidth = [int][DshClickWin32]::GetSystemMetrics(78) # SM_CXVIRTUALSCREEN
675
+ $virtualHeight = [int][DshClickWin32]::GetSystemMetrics(79) # SM_CYVIRTUALSCREEN
676
+ if ($virtualWidth -gt 0 -and $virtualHeight -gt 0) {
677
+ return @{ scope = 'virtual'; x = $virtualX; y = $virtualY; width = $virtualWidth; height = $virtualHeight }
678
+ }
679
+ }
680
+ return @{
681
+ scope = 'primary'; x = 0; y = 0
682
+ width = [int][DshClickWin32]::GetSystemMetrics(0) # SM_CXSCREEN
683
+ height = [int][DshClickWin32]::GetSystemMetrics(1) # SM_CYSCREEN
684
+ }
685
+ }
686
+
687
+ # Capture a screen region: returns @{ bitmap; method; scope; rect }.
688
+ function Get-ScreenCapture($scope) {
689
+ $metrics = Get-ScreenMetrics $scope
690
+ if ($metrics.width -le 0 -or $metrics.height -le 0) { throw 'the screen reports no drawable area' }
691
+ $bitmap = New-CaptureBitmap $metrics.width $metrics.height
478
692
  $graphics = [System.Drawing.Graphics]::FromImage($bitmap)
479
693
  try {
480
- $graphics.CopyFromScreen(0, 0, 0, 0, (New-Object System.Drawing.Size($width, $height)))
694
+ $graphics.CopyFromScreen($metrics.x, $metrics.y, 0, 0, (New-Object System.Drawing.Size($metrics.width, $metrics.height)))
695
+ } catch {
696
+ $bitmap.Dispose()
697
+ throw
481
698
  } finally {
482
699
  $graphics.Dispose()
483
700
  }
484
- return $bitmap
701
+ return @{
702
+ bitmap = $bitmap
703
+ method = 'copyfromscreen'
704
+ scope = $metrics.scope
705
+ rect = @{ x = $metrics.x; y = $metrics.y; width = $metrics.width; height = $metrics.height }
706
+ }
485
707
  }
486
708
 
487
709
  function Resize-BitmapIfNeeded($bitmap, $maxSide) {
@@ -651,12 +873,17 @@ function Invoke-OpWindows {
651
873
  foreach ($hwnd in [DshClickWin32]::EnumVisibleWindows()) {
652
874
  $pidValue = 0
653
875
  [void][DshClickWin32]::GetWindowThreadProcessId($hwnd, [ref]$pidValue)
876
+ # A minimized window still enumerates, but GetWindowRect then returns the
877
+ # CW_USEDEFAULT placeholder (-32000,-32000 314x50). Report the state and
878
+ # suppress the placeholder rect instead of handing it to coordinate math.
879
+ $minimized = [DshClickWin32]::IsIconic($hwnd)
654
880
  $windows += @{
655
881
  windowId = [int64]$hwnd
656
882
  processId = if ([int]$pidValue -gt 0) { [int]$pidValue } else { $null }
657
883
  title = [DshClickWin32]::WindowTitle($hwnd)
658
884
  className = [DshClickWin32]::WindowClass($hwnd)
659
- rect = (Get-WindowRectInfo $hwnd)
885
+ rect = if ($minimized) { $null } else { Get-WindowRectInfo $hwnd }
886
+ minimized = $minimized
660
887
  executablePath = $null
661
888
  visible = $true
662
889
  }
@@ -682,12 +909,14 @@ function Invoke-OpApps {
682
909
  $pidInt = [int]$pidValue
683
910
  if ($pidInt -le 0) { continue }
684
911
  if (-not $byPid.ContainsKey($pidInt)) { $byPid[$pidInt] = @() }
912
+ $minimized = [DshClickWin32]::IsIconic($hwnd)
685
913
  $byPid[$pidInt] += @{
686
914
  windowId = [int64]$hwnd
687
915
  processId = $pidInt
688
916
  title = [DshClickWin32]::WindowTitle($hwnd)
689
917
  className = [DshClickWin32]::WindowClass($hwnd)
690
- rect = (Get-WindowRectInfo $hwnd)
918
+ rect = if ($minimized) { $null } else { Get-WindowRectInfo $hwnd }
919
+ minimized = $minimized
691
920
  executablePath = $null
692
921
  visible = $true
693
922
  }
@@ -712,91 +941,129 @@ function Invoke-OpApps {
712
941
  return $apps
713
942
  }
714
943
 
715
- function Get-SnapshotRecord($hwnd, $maxElements, $maxDepth) {
716
- $facts = Get-ProcessFacts $hwnd
717
- $rect = Get-WindowRectInfo $hwnd
718
- $foreground = ([DshClickWin32]::GetForegroundWindow() -eq $hwnd)
719
- $script:TreeNodes = @()
720
- $element = Get-UiaElement $hwnd
721
- Walk-UiaTree $element 0 $maxDepth $maxElements $false
722
- $treeHash = Get-StringHash (Get-TreeHashText $script:TreeNodes)
723
- $bitmap = Get-WindowBitmap $hwnd
724
- $shotHash = $null
944
+ function Get-SnapshotRecord($hwnd, $maxElements, $maxDepth, $capture) {
945
+ $ownsCapture = $false
946
+ if ($null -eq $capture) {
947
+ $capture = Get-WindowCapture $hwnd
948
+ $ownsCapture = $true
949
+ }
725
950
  try {
726
- $shotHash = Get-BitmapHash $bitmap
951
+ $facts = Get-ProcessFacts $hwnd
952
+ $rect = Get-WindowRectInfo $hwnd
953
+ $foreground = ([DshClickWin32]::GetForegroundWindow() -eq $hwnd)
954
+ $script:TreeNodes = @()
955
+ $element = Get-UiaElement $hwnd
956
+ Walk-UiaTree $element 0 $maxDepth $maxElements $false
957
+ $treeHash = Get-StringHash (Get-TreeHashText $script:TreeNodes)
958
+ # The hash is taken from the same frame the caller will ship as the
959
+ # screenshot (or compare later), not from a second capture: two captures
960
+ # can disagree on a busy window and on which capture path succeeded.
961
+ $shotHash = Get-BitmapHash $capture.bitmap
962
+ return @{
963
+ windowId = [int64]$hwnd
964
+ processId = $facts.pid
965
+ executablePath = $facts.executablePath
966
+ title = [DshClickWin32]::WindowTitle($hwnd)
967
+ className = [DshClickWin32]::WindowClass($hwnd)
968
+ rect = $rect
969
+ foreground = $foreground
970
+ treeHash = $treeHash
971
+ shotHash = $shotHash
972
+ elementCount = $script:TreeNodes.Count
973
+ }
727
974
  } finally {
728
- $bitmap.Dispose()
729
- }
730
- return @{
731
- windowId = [int64]$hwnd
732
- processId = $facts.pid
733
- executablePath = $facts.executablePath
734
- title = [DshClickWin32]::WindowTitle($hwnd)
735
- className = [DshClickWin32]::WindowClass($hwnd)
736
- rect = $rect
737
- foreground = $foreground
738
- treeHash = $treeHash
739
- shotHash = $shotHash
740
- elementCount = $script:TreeNodes.Count
975
+ if ($ownsCapture) { $capture.bitmap.Dispose() }
741
976
  }
742
977
  }
743
978
 
744
979
  function Invoke-OpSnapshot($opArgs) {
745
- $hwnd = Resolve-Window @{ windowId = $opArgs.windowId }
980
+ $target = Resolve-OpTarget $opArgs
981
+ if (-not (Test-TargetPresent $target)) {
982
+ # This used to fall back to the foreground window without saying so, which
983
+ # reported a different window than the caller asked for. Refuse instead.
984
+ $script:ErrorCode = 'NO_TARGET'
985
+ throw 'snapshot requires windowId or target; it never guesses the foreground window'
986
+ }
987
+ $hwnd = Resolve-Window $target
746
988
  $maxElements = if ($opArgs.maxElements -ne $null) { [int]$opArgs.maxElements } else { 500 }
747
989
  $maxDepth = if ($opArgs.maxDepth -ne $null) { [int]$opArgs.maxDepth } else { 32 }
748
- return (Get-SnapshotRecord $hwnd $maxElements $maxDepth)
990
+ $capture = Get-WindowCapture $hwnd
991
+ try {
992
+ return (Get-SnapshotRecord $hwnd $maxElements $maxDepth $capture)
993
+ } finally {
994
+ $capture.bitmap.Dispose()
995
+ }
749
996
  }
750
997
 
751
998
  function Invoke-OpShot($opArgs) {
752
- $target = if ($opArgs.target -ne $null) { $opArgs.target } else { @{} }
999
+ $target = Resolve-OpTarget $opArgs
1000
+ $scope = if ($opArgs.scope -ne $null) { [string]$opArgs.scope } else { 'primary' }
1001
+ if ($scope -ne 'primary' -and $scope -ne 'virtual') {
1002
+ $script:ErrorCode = 'INVALID_SCOPE'
1003
+ throw "unknown scope '$scope' (expected 'primary' or 'virtual')"
1004
+ }
753
1005
  $maxSide = if ($opArgs.maxSide -ne $null) { [int]$opArgs.maxSide } else { 2560 }
754
1006
  $maxElements = if ($opArgs.maxElements -ne $null) { [int]$opArgs.maxElements } else { 500 }
755
1007
  $maxDepth = if ($opArgs.maxDepth -ne $null) { [int]$opArgs.maxDepth } else { 32 }
756
- $hasTarget = ($target.windowId -ne $null) -or ($target.windowTitle -ne $null) -or ($target.processId -ne $null)
757
- if ($hasTarget) {
758
- $hwnd = Resolve-Window $target
759
- $bitmap = Get-WindowBitmap $hwnd
760
- $snapshot = Get-SnapshotRecord $hwnd $maxElements $maxDepth
761
- } else {
762
- $hwnd = $null
763
- $bitmap = Get-PrimaryBitmap
764
- $snapshot = @{
765
- windowId = 0; processId = 0; executablePath = $null
766
- title = 'primary screen'; className = 'Screen'
767
- rect = @{ x = 0; y = 0; width = $bitmap.Width; height = $bitmap.Height }
768
- foreground = $true; treeHash = ''; shotHash = ''; elementCount = 0
769
- }
770
- }
1008
+ $capture = $null
1009
+ $bitmap = $null
771
1010
  try {
772
- $bitmap = Resize-BitmapIfNeeded $bitmap $maxSide
1011
+ if (Test-TargetPresent $target) {
1012
+ $hwnd = Resolve-Window $target
1013
+ $capture = Get-WindowCapture $hwnd
1014
+ $captureScope = 'window'
1015
+ $snapshot = Get-SnapshotRecord $hwnd $maxElements $maxDepth $capture
1016
+ } else {
1017
+ # No target means the screen, not the foreground window: the result says
1018
+ # `scope: primary` and its snapshot is titled "primary screen" (windowId 0).
1019
+ $capture = Get-ScreenCapture $scope
1020
+ $captureScope = $capture.scope
1021
+ $snapshot = @{
1022
+ windowId = 0; processId = 0; executablePath = $null
1023
+ title = if ($capture.scope -eq 'virtual') { 'virtual screen' } else { 'primary screen' }
1024
+ className = 'Screen'
1025
+ rect = $capture.rect
1026
+ foreground = $true; treeHash = ''; shotHash = ''; elementCount = 0
1027
+ }
1028
+ }
1029
+ $bitmap = $capture.bitmap
1030
+ if ([Math]::Max($bitmap.Width, $bitmap.Height) -gt $maxSide) {
1031
+ # Resize-BitmapIfNeeded disposes the original; the finally below disposes
1032
+ # whichever bitmap is current, and Dispose is idempotent.
1033
+ $bitmap = Resize-BitmapIfNeeded $capture.bitmap $maxSide
1034
+ }
773
1035
  $base64 = Get-BitmapBase64 $bitmap
774
1036
  return @{
775
1037
  pngBase64 = $base64
776
1038
  width = $bitmap.Width
777
1039
  height = $bitmap.Height
1040
+ captureMethod = $capture.method
1041
+ scope = $captureScope
778
1042
  snapshot = $snapshot
779
1043
  }
780
1044
  } finally {
781
- $bitmap.Dispose()
1045
+ if ($null -ne $bitmap) { $bitmap.Dispose() }
1046
+ elseif ($null -ne $capture) { $capture.bitmap.Dispose() }
782
1047
  }
783
1048
  }
784
1049
 
785
1050
  function Invoke-OpTree($opArgs) {
786
- $target = if ($opArgs.target -ne $null) { $opArgs.target } else { @{} }
1051
+ $target = Resolve-OpTarget $opArgs
787
1052
  $maxElements = if ($opArgs.maxElements -ne $null) { [int]$opArgs.maxElements } else { 500 }
788
1053
  $maxDepth = if ($opArgs.maxDepth -ne $null) { [int]$opArgs.maxDepth } else { 32 }
789
1054
  $includePixels = if ($opArgs.includePixels -ne $null) { [bool]$opArgs.includePixels } else { $true }
1055
+ # An empty target still means the foreground window here (documented for
1056
+ # screen_read); only `snapshot` refuses the guess.
790
1057
  $hwnd = Resolve-Window $target
791
- $snapshot = Get-SnapshotRecord $hwnd $maxElements $maxDepth
792
- $script:TreeNodes = @()
793
- $element = Get-UiaElement $hwnd
794
- Walk-UiaTree $element 0 $maxDepth $maxElements $includePixels
795
- $pixels = @()
796
- if ($includePixels) {
797
- $bitmap = Get-WindowBitmap $hwnd
798
- try {
799
- $rectInfo = Get-WindowRectInfo $hwnd
1058
+ $capture = Get-WindowCapture $hwnd
1059
+ try {
1060
+ $snapshot = Get-SnapshotRecord $hwnd $maxElements $maxDepth $capture
1061
+ $script:TreeNodes = @()
1062
+ $element = Get-UiaElement $hwnd
1063
+ Walk-UiaTree $element 0 $maxDepth $maxElements $includePixels
1064
+ $pixels = @()
1065
+ if ($includePixels) {
1066
+ $rectInfo = $capture.rect
800
1067
  $count = 0
801
1068
  foreach ($node in $script:TreeNodes) {
802
1069
  if ($count -ge 200) { break }
@@ -804,21 +1071,21 @@ function Invoke-OpTree($opArgs) {
804
1071
  if ($r.width -le 0 -or $r.height -le 0) { continue }
805
1072
  $cx = $r.x + [int]($r.width / 2) - $rectInfo.x
806
1073
  $cy = $r.y + [int]($r.height / 2) - $rectInfo.y
807
- $color = Get-PixelColorText $bitmap $cx $cy
1074
+ $color = Get-PixelColorText $capture.bitmap $cx $cy
808
1075
  if ($color -ne $null) {
809
1076
  $label = if ($node.name -ne '') { $node.name } else { "$($node.controlType)[$($node.elementId)]" }
810
1077
  $pixels += @{ label = $label; x = $r.x + [int]($r.width / 2); y = $r.y + [int]($r.height / 2); color = $color }
811
1078
  $count += 1
812
1079
  }
813
1080
  }
814
- } finally {
815
- $bitmap.Dispose()
816
1081
  }
817
- }
818
- return @{
819
- snapshot = $snapshot
820
- elements = @($script:TreeNodes)
821
- pixels = $pixels
1082
+ return @{
1083
+ snapshot = $snapshot
1084
+ elements = @($script:TreeNodes)
1085
+ pixels = $pixels
1086
+ }
1087
+ } finally {
1088
+ $capture.bitmap.Dispose()
822
1089
  }
823
1090
  }
824
1091
 
@@ -1045,6 +1312,9 @@ try {
1045
1312
  # Per-request focus-fallback verdict (set by Set-DshFocusFallback, consumed by
1046
1313
  # Get-ActionOutcome); one helper process serves exactly one request.
1047
1314
  $script:focusFallbackDetail = $null
1315
+ # Per-request failure code; a plain `throw` still defaults to HELPER_ERROR,
1316
+ # while the capture/scope guards set a precise code for the caller.
1317
+ $script:ErrorCode = $null
1048
1318
  switch ([string]$request.op) {
1049
1319
  'windows' { $result = Invoke-OpWindows }
1050
1320
  'metrics' { $result = Invoke-OpMetrics }
@@ -1064,6 +1334,7 @@ try {
1064
1334
  } catch {
1065
1335
  $message = $_.Exception.Message
1066
1336
  if ($message -eq $null -or $message.Length -eq 0) { $message = 'unknown helper failure' }
1067
- Write-DshError 'HELPER_ERROR' $message
1337
+ $code = if ($script:ErrorCode -ne $null -and [string]$script:ErrorCode -ne '') { [string]$script:ErrorCode } else { 'HELPER_ERROR' }
1338
+ Write-DshError $code $message
1068
1339
  exit 0
1069
1340
  }
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "dsh-click",
3
- "version": "0.3.28",
3
+ "version": "0.3.29",
4
4
  "description": "Cross-platform native desktop control tools for DeepSeek Harness (Windows first): screen_shot, screen_read (accessibility tree + pixel description for text-only models), click/type/scroll/key, and app_list/app_launch — every mutating action gated by approval, never stealing foreground focus, with stale-observation rejection and process-identity checks",
5
5
  "repository": {
6
6
  "type": "git",