dsh-click 0.3.28 → 0.3.29
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +14 -2
- package/README.en.md +11 -2
- package/README.es.md +11 -2
- package/README.hi.md +11 -2
- package/README.md +11 -2
- package/README.pt.md +11 -2
- package/lib/index.js +68 -20
- package/lib/types/platform/runner.d.ts +2 -2
- package/lib/types/platform/runner.d.ts.map +1 -1
- package/lib/types/platform/types.d.ts +38 -3
- package/lib/types/platform/types.d.ts.map +1 -1
- package/lib/types/tools.d.ts.map +1 -1
- package/lib/types/version.d.ts +1 -1
- package/native/win32/dsh-click-helper.ps1 +350 -79
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,19 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## [0.3.29] - 2026-11-13
|
|
4
|
+
|
|
5
|
+
### Added
|
|
6
|
+
|
|
7
|
+
- **`captureMethod` and `scope` on `screen_shot` results.** `captureMethod` names the path that produced the pixels (`printwindow-fullcontent` / `printwindow` / `copyfromscreen`), and `scope` says what was captured (`window` / `primary` / `virtual`). `scope: "virtual"` captures the whole virtual desktop (`SM_XVIRTUALSCREEN` family); it is opt-in, defaults to `primary`, and is marked experimental because the multi-monitor/negative-origin path is not fully verified.
|
|
8
|
+
- **Minimized windows are reported honestly.** `app_list` marks them `minimized: true` and omits `rect` (Windows reports the `-32000` placeholder rectangle while iconified), and `screen_shot` / `screen_read` refuse them with `WINDOW_MINIMIZED` instead of capturing a broken frame.
|
|
9
|
+
|
|
10
|
+
### Fixed
|
|
11
|
+
|
|
12
|
+
- **Window screenshots no longer depend on occlusion (the main #6/#35 bug).** The helper now tries `PrintWindow(hwnd, hdc, PW_RENDERFULLCONTENT)` first, falls back to `PrintWindow(0)`, and only then to `CopyFromScreen`. A captured frame is rejected as blank by sampled checks on the 40px-inset 16x16 grid (quantized colour count and luminance standard deviation) plus a 99%-dark test on a 32x32 full-frame grid. Measured on Windows 11: a fully covered Explorer window produced byte-identical `PrintWindow(2)` frames across two different occluders, while `CopyFromScreen` captured the occluder and changed between runs. The colour-count rule is applied only to the no-flag fallback: a legitimate all-white Chrome page measured 3 quantized colours with a luminance sd of 9.6, so applying that rule to `PrintWindow(2)` would reject valid frames.
|
|
13
|
+
- **One target resolver for `snapshot` / `tree` / `shot`.** `snapshot` used to read only `args.windowId`, so `{"op":"snapshot","args":{"target":{"windowId":N}}}` silently reported the foreground window. All three now read `args.target`, with the legacy top-level `windowId` kept as a fallback. A `snapshot` with neither fails with `NO_TARGET` instead of guessing.
|
|
14
|
+
- **`shot` without a target documents and reports the primary screen**, not the foreground window; `screen_read` keeps the foreground default because it is a tree walk.
|
|
15
|
+
- **A window capture and its `shotHash` now come from one frame.** The hash used to be computed from a second capture, which could disagree with the shipped image and with the capture path that was reported.
|
|
16
|
+
|
|
3
17
|
## [0.3.28] - 2026-11-13
|
|
4
18
|
|
|
5
19
|
### Fixed
|
|
@@ -79,8 +93,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
79
93
|
- Correct the release date in the previous section, which was stamped with the literal string `undefined` by the release stamper. No content or behaviour change; the version is bumped only because npm will not republish an existing version.
|
|
80
94
|
|
|
81
95
|
|
|
82
|
-
## [Unreleased]
|
|
83
|
-
|
|
84
96
|
## [0.3.17] - 2026-10-04
|
|
85
97
|
|
|
86
98
|
undefined
|
package/README.en.md
CHANGED
|
@@ -97,11 +97,11 @@ Every PerryLink plugin is listed in the built-in DSH Desktop market: **Market
|
|
|
97
97
|
|
|
98
98
|
`dsh-click` splits the native desktop into an auditable observation face and action face:
|
|
99
99
|
|
|
100
|
-
- **`screen_shot`** — captures a window (or the primary screen) and downscales it to `maxScreenshotSide`. With a vision-capable model the result carries the image; text-only models get a text description and keep working.
|
|
100
|
+
- **`screen_shot`** — captures a window (or the primary screen) and downscales it to `maxScreenshotSide`. Window captures take `PrintWindow(PW_RENDERFULLCONTENT)` first, so they do not depend on other windows stacked on top; `captureMethod` reports the path that actually produced the pixels. With a vision-capable model the result carries the image; text-only models get a text description and keep working.
|
|
101
101
|
- **`screen_read`** — the structured observation: the accessibility tree (element ids, types, names, rectangles, supported patterns) plus pixel-location hints with colors — plain text, no image model required.
|
|
102
102
|
- **`screen_find`** — for applications that expose no accessibility tree, finds clickable text or a described visual target via optional OCR (`ocr.*`, tesseract probed at mount) or visual grounding, returning screen coordinates plus an `observationId`; read-only, never needs approval.
|
|
103
103
|
- **`click` / `type` / `scroll` / `key`** — window-scoped actions addressed by element id or coordinates. Delivery prefers UIA invoke, falls back to posted window messages, and never steals foreground focus by default; `type` takes the UIA `ValuePattern` fast path and otherwise injects real keystrokes through `SendInput` (CJK-safe); `key` reports honestly when an app ignores posted input.
|
|
104
|
-
- **`app_list` / `app_launch`** — enumerate running applications and their visible windows; launch one by name or executable path.
|
|
104
|
+
- **`app_list` / `app_launch`** — enumerate running applications and their visible windows (minimized windows carry `minimized: true` and no `rect`); launch one by name or executable path.
|
|
105
105
|
|
|
106
106
|
Every mutating action crosses one shared safety boundary:
|
|
107
107
|
|
|
@@ -133,6 +133,15 @@ model harness
|
|
|
133
133
|
| `app_list` | ✅ | — | Running applications and their visible windows |
|
|
134
134
|
| `app_launch` | | ✅ | By name or executable path, with optional arguments |
|
|
135
135
|
|
|
136
|
+
### Screenshots and coordinate space
|
|
137
|
+
|
|
138
|
+
- **A target means a window capture; no target means the primary screen** (`snapshot.windowId = 0`, `title = "primary screen"`), not the foreground window. For the whole virtual desktop (multiple monitors, negative origins) pass `scope: "virtual"`; that mode is marked **experimental** and its multi-monitor geometry is not fully verified yet.
|
|
139
|
+
- **Window captures try `PrintWindow(hwnd, hdc, PW_RENDERFULLCONTENT)` first**: the window paints itself, so the image does not depend on what is stacked on top. On failure or a blank frame the helper falls back to `PrintWindow(0)` and then to `CopyFromScreen`. `captureMethod` (`printwindow-fullcontent` / `printwindow` / `copyfromscreen`) says which path produced the image.
|
|
140
|
+
- **`width` / `height` are the bitmap size, not the window size**: they are already downscaled to `maxScreenshotSide`, while `snapshot.rect` is always the physical window (or screen) rectangle. Map with `bitmapX = (screenX - rect.x) * width / rect.width`.
|
|
141
|
+
- **Every rectangle is in physical pixels**: DPI awareness is set when the helper starts (Per-Monitor V2), and `app_list`, `screen_read`, `screen_shot`, and `click` share one coordinate space.
|
|
142
|
+
- **Minimized windows**: `app_list` marks them `minimized: true` and returns no `rect` (Windows reports the `-32000` placeholder rect while iconified); `screen_shot` / `screen_read` fail with `WINDOW_MINIMIZED`, so restore the window first.
|
|
143
|
+
|
|
144
|
+
|
|
136
145
|
### Examples
|
|
137
146
|
|
|
138
147
|
These are the plugin’s real entry points; install once, then call one.
|
package/README.es.md
CHANGED
|
@@ -97,11 +97,11 @@ Todos los plugins de PerryLink pueden explorarse en el mercado integrado de DSH
|
|
|
97
97
|
|
|
98
98
|
`dsh-click` separa el escritorio nativo en una cara de observación y otra de acción, ambas auditables:
|
|
99
99
|
|
|
100
|
-
- **`screen_shot`** — captura una ventana (o la pantalla principal) y la escala a `maxScreenshotSide`. Con un modelo capaz de ver imágenes el resultado lleva la imagen; los modelos de solo texto reciben una descripción textual y siguen funcionando.
|
|
100
|
+
- **`screen_shot`** — captura una ventana (o la pantalla principal) y la escala a `maxScreenshotSide`. Las capturas de ventana usan primero `PrintWindow(PW_RENDERFULLCONTENT)`, así que no dependen de otras ventanas superpuestas; `captureMethod` indica la ruta que produjo los píxeles. Con un modelo capaz de ver imágenes el resultado lleva la imagen; los modelos de solo texto reciben una descripción textual y siguen funcionando.
|
|
101
101
|
- **`screen_read`** — la observación estructurada: el árbol de accesibilidad (ids de elemento, tipos, nombres, rectángulos, patrones soportados) más pistas de posición de píxeles con colores; texto plano, sin necesidad de modelo de imagen.
|
|
102
102
|
- **`screen_find`** — para aplicaciones sin árbol de accesibilidad, busca texto clicable o un objetivo visual descrito mediante OCR opcional (`ocr.*`, tesseract sondeado al montar) o grounding visual, y devuelve coordenadas de pantalla más un `observationId`; es de solo lectura y nunca pide aprobación.
|
|
103
103
|
- **`click` / `type` / `scroll` / `key`** — acciones con ámbito de ventana direccionadas por id de elemento o coordenadas. La entrega prefiere la invocación UIA y recurre a mensajes de ventana publicados, y por defecto nunca roba el foco de primer plano; `type` usa la vía rápida `ValuePattern` de UIA y, si falla, inyecta pulsaciones reales con `SendInput` (seguro para CJK); `key` informa con honestidad cuando una app ignora la entrada publicada.
|
|
104
|
-
- **`app_list` / `app_launch`** — enumera las aplicaciones en ejecución y sus ventanas visibles; lanza una por nombre o ruta del ejecutable.
|
|
104
|
+
- **`app_list` / `app_launch`** — enumera las aplicaciones en ejecución y sus ventanas visibles (las minimizadas llevan `minimized: true` y no traen `rect`); lanza una por nombre o ruta del ejecutable.
|
|
105
105
|
|
|
106
106
|
Toda acción mutante cruza la misma frontera de seguridad:
|
|
107
107
|
|
|
@@ -133,6 +133,15 @@ modelo harness
|
|
|
133
133
|
| `app_list` | ✅ | — | Aplicaciones en ejecución y sus ventanas visibles |
|
|
134
134
|
| `app_launch` | | ✅ | Por nombre o ruta del ejecutable, con argumentos opcionales |
|
|
135
135
|
|
|
136
|
+
### Capturas y espacio de coordenadas
|
|
137
|
+
|
|
138
|
+
- **Con `target` se captura una ventana; sin `target` se captura la pantalla principal** (`snapshot.windowId = 0`, `title = "primary screen"`), no la ventana en primer plano. Para todo el escritorio virtual (varios monitores, con origen negativo) pasa `scope: "virtual"`; ese modo está marcado como **experimental** y su geometría multimonitor aún no está verificada por completo.
|
|
139
|
+
- **La captura de ventana prueba primero `PrintWindow(hwnd, hdc, PW_RENDERFULLCONTENT)`**: la ventana se dibuja a sí misma, así que la imagen no depende de lo que haya encima. Si falla o sale en blanco, el helper retrocede a `PrintWindow(0)` y luego a `CopyFromScreen`. `captureMethod` (`printwindow-fullcontent` / `printwindow` / `copyfromscreen`) dice qué ruta produjo la imagen.
|
|
140
|
+
- **`width` / `height` son el tamaño del bitmap, no el de la ventana**: ya vienen reducidos a `maxScreenshotSide`, mientras que `snapshot.rect` es siempre el rectángulo físico de la ventana (o de la pantalla). Conversión: `bitmapX = (screenX - rect.x) * width / rect.width`.
|
|
141
|
+
- **Todos los rectángulos están en píxeles físicos**: la conciencia de DPI se fija al arrancar el helper (Per-Monitor V2), y `app_list`, `screen_read`, `screen_shot` y `click` comparten el mismo espacio de coordenadas.
|
|
142
|
+
- **Ventanas minimizadas**: `app_list` las marca con `minimized: true` y no devuelve `rect` (Windows da el rectángulo comodín `-32000` mientras están minimizadas); `screen_shot` / `screen_read` fallan con `WINDOW_MINIMIZED`; restaura la ventana primero.
|
|
143
|
+
|
|
144
|
+
|
|
136
145
|
### Ejemplos
|
|
137
146
|
|
|
138
147
|
Son los puntos de entrada reales del plugin; instala una vez y llama a uno.
|
package/README.hi.md
CHANGED
|
@@ -97,11 +97,11 @@ dsh plugin --profile <name> add dsh-click
|
|
|
97
97
|
|
|
98
98
|
`dsh-click` नेटिव डेस्कटॉप को दो ऑडिट-योग्य पक्षों में बाँटता है — अवलोकन और कार्रवाई:
|
|
99
99
|
|
|
100
|
-
- **`screen_shot`** — विंडो (या मुख्य स्क्रीन) की तस्वीर लेकर `maxScreenshotSide` तक छोटा करता है। जो मॉडल तस्वीरें लेता है उसे परिणाम में चित्र मिलता है; केवल-टेक्स्ट मॉडल को टेक्स्ट विवरण मिलता है और वे भी चलते रहते हैं।
|
|
100
|
+
- **`screen_shot`** — विंडो (या मुख्य स्क्रीन) की तस्वीर लेकर `maxScreenshotSide` तक छोटा करता है। विंडो कैप्चर पहले `PrintWindow(PW_RENDERFULLCONTENT)` से होते हैं, इसलिए ऊपर रखी विंडो से प्रभावित नहीं होते; `captureMethod` बताता है कि पिक्सेल असल में किस रास्ते से आए। जो मॉडल तस्वीरें लेता है उसे परिणाम में चित्र मिलता है; केवल-टेक्स्ट मॉडल को टेक्स्ट विवरण मिलता है और वे भी चलते रहते हैं।
|
|
101
101
|
- **`screen_read`** — संरचित अवलोकन: accessibility tree (तत्व id, प्रकार, नाम, आयत, समर्थित पैटर्न) और रंगों सहित pixel स्थिति-संकेत — शुद्ध टेक्स्ट, कोई इमेज मॉडल ज़रूरी नहीं।
|
|
102
102
|
- **`screen_find`** — जिन ऐप्स में accessibility tree नहीं है, उनके लिए वैकल्पिक OCR (`ocr.*`, माउंट पर tesseract जाँचा जाता है) या visual grounding से क्लिक-योग्य टेक्स्ट/वर्णित लक्ष्य खोजता है और स्क्रीन निर्देशांक तथा `observationId` लौटाता है; यह केवल-पठन है और कभी मंज़ूरी नहीं माँगता।
|
|
103
103
|
- **`click` / `type` / `scroll` / `key`** — तत्व id या निर्देशांक से पता किए गए विंडो-स्कोप कार्रवाई। डिलीवरी UIA invoke पसंद करती है और posted विंडो संदेशों पर लौटती है, तथा डिफ़ॉल्ट रूप से foreground फ़ोकस कभी नहीं छीनती; `type` पहले UIA `ValuePattern` तेज़ रास्ता लेता है, फिर `SendInput` से असली कीस्ट्रोक डालता है (CJK-सुरक्षित); posted इनपुट अनदेखा करने वाले ऐप्स पर `key` ईमानदारी से विफलता बताता है।
|
|
104
|
-
- **`app_list` / `app_launch`** — चल रहे ऐप्स और उनकी दिखने वाली विंडो सूचीबद्ध करता
|
|
104
|
+
- **`app_list` / `app_launch`** — चल रहे ऐप्स और उनकी दिखने वाली विंडो सूचीबद्ध करता है (minimized विंडो पर `minimized: true` होता है और कोई `rect` नहीं आता); नाम या executable पथ से ऐप लॉन्च करता है।
|
|
105
105
|
|
|
106
106
|
हर बदलाव वाली कार्रवाई एक ही सुरक्षा सीमा से गुज़रती है:
|
|
107
107
|
|
|
@@ -133,6 +133,15 @@ dsh plugin --profile <name> add dsh-click
|
|
|
133
133
|
| `app_list` | ✅ | — | चल रहे ऐप्स और उनकी दिखने वाली विंडो |
|
|
134
134
|
| `app_launch` | | ✅ | नाम या executable पथ से, वैकल्पिक आर्ग्युमेंट के साथ |
|
|
135
135
|
|
|
136
|
+
### स्क्रीनशॉट और निर्देशांक स्थान
|
|
137
|
+
|
|
138
|
+
- **`target` हो तो विंडो कैप्चर होती है; `target` न हो तो मुख्य स्क्रीन** (`snapshot.windowId = 0`, `title = "primary screen"`), foreground विंडो नहीं। पूरे virtual desktop (कई मॉनिटर, ऋणात्मक मूल) के लिए `scope: "virtual"` भेजें; यह मोड **experimental** है और बहु-मॉनिटर ज्यामिति की पूरी पुष्टि नहीं हुई है।
|
|
139
|
+
- **विंडो कैप्चर पहले `PrintWindow(hwnd, hdc, PW_RENDERFULLCONTENT)` आज़माते हैं**: विंडो स्वयं चित्रित होती है, इसलिए ऊपर रखी विंडो प्रभावित नहीं करती। विफलता या खाली फ़्रेम पर helper `PrintWindow(0)` और फिर `CopyFromScreen` पर लौटता है। `captureMethod` (`printwindow-fullcontent` / `printwindow` / `copyfromscreen`) बताता है कि चित्र किस रास्ते से बना।
|
|
140
|
+
- **`width` / `height` बिटमैप का आकार हैं, विंडो का नहीं**: ये `maxScreenshotSide` तक घटाए जा चुके हैं, जबकि `snapshot.rect` हमेशा विंडो (या स्क्रीन) का भौतिक आयत है। रूपांतरण: `bitmapX = (screenX - rect.x) * width / rect.width`.
|
|
141
|
+
- **सभी आयत भौतिक पिक्सेल में हैं**: DPI जागरूकता helper शुरू होते ही सेट होती है (Per-Monitor V2), और `app_list`, `screen_read`, `screen_shot` तथा `click` एक ही निर्देशांक स्थान साझा करते हैं।
|
|
142
|
+
- **minimized विंडो**: `app_list` उन्हें `minimized: true` से चिह्नित करता है और `rect` नहीं देता (Windows minimized अवस्था में `-32000` प्लेसहोल्डर आयत देता है); `screen_shot` / `screen_read` स्पष्ट रूप से `WINDOW_MINIMIZED` देकर विफल होते हैं — पहले विंडो बहाल करें।
|
|
143
|
+
|
|
144
|
+
|
|
136
145
|
### उदाहरण
|
|
137
146
|
|
|
138
147
|
ये प्लगइन के वास्तविक प्रवेश बिंदु हैं; एक बार इंस्टॉल करके एक चलाएँ।
|
package/README.md
CHANGED
|
@@ -97,11 +97,11 @@ dsh plugin --profile <name> add dsh-click
|
|
|
97
97
|
|
|
98
98
|
`dsh-click` 把原生桌面拆成可审计的观察面与动作面:
|
|
99
99
|
|
|
100
|
-
- **`screen_shot`** —— 截取窗口(或主屏幕),按 `maxScreenshotSide`
|
|
100
|
+
- **`screen_shot`** —— 截取窗口(或主屏幕),按 `maxScreenshotSide` 缩放。窗口截图优先走 `PrintWindow(PW_RENDERFULLCONTENT)`,不受其它窗口遮挡影响;结果里的 `captureMethod` 写明实际路径。模型支持图像时结果附带图片;纯文本模型则得到文字描述,照常可用。
|
|
101
101
|
- **`screen_read`** —— 结构化观察:无障碍树(元素 id、类型、名称、矩形、支持的模式)加像素位置提示与颜色 —— 纯文本,无需图像模型。
|
|
102
102
|
- **`screen_find`** —— 对没有无障碍树的应用,用可选 OCR(`ocr.*`,挂载时探测 tesseract)或视觉 grounding 按文字/描述找可点目标,返回屏幕坐标与 `observationId`;只读,不需要审批。
|
|
103
103
|
- **`click` / `type` / `scroll` / `key`** —— 以元素 id 或坐标寻址的窗口级操作。优先 UIA invoke,回退到 post 窗口消息,且默认绝不抢占前台焦点;`type` 走 UIA `ValuePattern` 快路径,失败再用 `SendInput` 注入真实按键(CJK 安全);`key` 对忽略 post 输入的应用会大声拒绝。
|
|
104
|
-
- **`app_list` / `app_launch`** ——
|
|
104
|
+
- **`app_list` / `app_launch`** —— 枚举运行中的应用及其可见窗口(最小化窗口标 `minimized: true` 且不返回 `rect`);按名称或可执行路径启动应用。
|
|
105
105
|
|
|
106
106
|
每个变更性操作都穿过同一条安全边界:
|
|
107
107
|
|
|
@@ -133,6 +133,15 @@ dsh plugin --profile <name> add dsh-click
|
|
|
133
133
|
| `app_list` | ✅ | — | 运行中的应用及其可见窗口 |
|
|
134
134
|
| `app_launch` | | ✅ | 按名称或可执行路径启动,可带参数 |
|
|
135
135
|
|
|
136
|
+
### 截图与坐标空间
|
|
137
|
+
|
|
138
|
+
- **带 `target` 才是窗口截图;不带 `target` 抓的是主屏**(`snapshot.windowId = 0`、`title = "primary screen"`),不是前台窗口。要整个虚拟桌面(多屏,含负坐标)请传 `scope: "virtual"`;该模式标注为**实验性**,多屏几何尚未完整验证。
|
|
139
|
+
- **窗口截图先走 `PrintWindow(hwnd, hdc, PW_RENDERFULLCONTENT)`**:窗口自己绘制,不受其它窗口遮挡影响;失败或空图时依次回退 `PrintWindow(0)` 与 `CopyFromScreen`。结果里的 `captureMethod` 为 `printwindow-fullcontent` / `printwindow` / `copyfromscreen`,直接说明这张图是怎么来的。
|
|
140
|
+
- **`width` / `height` 是位图尺寸,不是窗口尺寸**:它们已被 `maxScreenshotSide` 下采样;`snapshot.rect` 始终是物理像素的窗口(或屏幕)矩形。换算方式:`bitmapX = (screenX - rect.x) * width / rect.width`。
|
|
141
|
+
- **所有矩形都是物理像素**:DPI 感知在 helper 启动时设置(Per-Monitor V2),`app_list`、`screen_read`、`screen_shot` 与 `click` 共用同一坐标空间。
|
|
142
|
+
- **最小化窗口**:`app_list` 标记 `minimized: true` 并且不返回 `rect`(Windows 在最小化时给的是 `-32000` 占位矩形);`screen_shot` / `screen_read` 对它明确报 `WINDOW_MINIMIZED`,请先还原窗口。
|
|
143
|
+
|
|
144
|
+
|
|
136
145
|
### 示例
|
|
137
146
|
|
|
138
147
|
以下为插件真实入口;安装后调用一个即可。
|
package/README.pt.md
CHANGED
|
@@ -97,11 +97,11 @@ Todo plugin PerryLink está listado no mercado integrado do DSH Desktop: **Marke
|
|
|
97
97
|
|
|
98
98
|
O `dsh-click` separa o desktop nativo numa face de observação e noutra de ação, ambas auditáveis:
|
|
99
99
|
|
|
100
|
-
- **`screen_shot`** — captura uma janela (ou a tela principal) e a reduz para `maxScreenshotSide`. Com um modelo que aceita imagens o resultado leva a imagem; modelos só de texto recebem uma descrição textual e continuam funcionando.
|
|
100
|
+
- **`screen_shot`** — captura uma janela (ou a tela principal) e a reduz para `maxScreenshotSide`. Capturas de janela usam primeiro `PrintWindow(PW_RENDERFULLCONTENT)`, sem depender de outras janelas sobrepostas; `captureMethod` informa o caminho que produziu os píxeis. Com um modelo que aceita imagens o resultado leva a imagem; modelos só de texto recebem uma descrição textual e continuam funcionando.
|
|
101
101
|
- **`screen_read`** — a observação estruturada: a árvore de acessibilidade (ids de elemento, tipos, nomes, retângulos, padrões suportados) mais dicas de posição de píxeis com cores; texto puro, sem precisar de modelo de imagem.
|
|
102
102
|
- **`screen_find`** — para aplicações sem árvore de acessibilidade, encontra texto clicável ou um alvo visual descrito via OCR opcional (`ocr.*`, tesseract sondado ao montar) ou grounding visual, devolvendo coordenadas de tela mais um `observationId`; é só leitura e nunca pede aprovação.
|
|
103
103
|
- **`click` / `type` / `scroll` / `key`** — ações com escopo de janela endereçadas por id de elemento ou coordenadas. A entrega prefere invocação UIA e recorre a mensagens de janela postadas, e por padrão nunca rouba o foco de primeiro plano; `type` usa o caminho rápido `ValuePattern` do UIA e, se falhar, injeta teclas reais via `SendInput` (seguro para CJK); `key` reporta com honestidade quando uma app ignora a entrada postada.
|
|
104
|
-
- **`app_list` / `app_launch`** — enumera aplicações em execução e as suas janelas visíveis; lança uma por nome ou caminho do executável.
|
|
104
|
+
- **`app_list` / `app_launch`** — enumera aplicações em execução e as suas janelas visíveis (as minimizadas levam `minimized: true` e não trazem `rect`); lança uma por nome ou caminho do executável.
|
|
105
105
|
|
|
106
106
|
Toda ação mutante cruza a mesma fronteira de segurança:
|
|
107
107
|
|
|
@@ -133,6 +133,15 @@ modelo harness
|
|
|
133
133
|
| `app_list` | ✅ | — | Aplicações em execução e as suas janelas visíveis |
|
|
134
134
|
| `app_launch` | | ✅ | Por nome ou caminho do executável, com argumentos opcionais |
|
|
135
135
|
|
|
136
|
+
### Capturas e espaço de coordenadas
|
|
137
|
+
|
|
138
|
+
- **Com `target` captura-se uma janela; sem `target` captura-se a tela principal** (`snapshot.windowId = 0`, `title = "primary screen"`), não a janela em primeiro plano. Para todo o desktop virtual (vários monitores, com origem negativa) passe `scope: "virtual"`; esse modo está marcado como **experimental** e a geometria multimonitor ainda não está totalmente verificada.
|
|
139
|
+
- **A captura de janela tenta primeiro `PrintWindow(hwnd, hdc, PW_RENDERFULLCONTENT)`**: a janela desenha-se a si mesma, pelo que a imagem não depende do que está por cima. Se falhar ou vier em branco, o helper recua para `PrintWindow(0)` e depois para `CopyFromScreen`. `captureMethod` (`printwindow-fullcontent` / `printwindow` / `copyfromscreen`) diz qual caminho produziu a imagem.
|
|
140
|
+
- **`width` / `height` são o tamanho do bitmap, não o da janela**: já vêm reduzidos a `maxScreenshotSide`, enquanto `snapshot.rect` é sempre o retângulo físico da janela (ou da tela). Conversão: `bitmapX = (screenX - rect.x) * width / rect.width`.
|
|
141
|
+
- **Todos os retângulos estão em píxeis físicos**: a consciência de DPI é definida quando o helper arranca (Per-Monitor V2), e `app_list`, `screen_read`, `screen_shot` e `click` partilham o mesmo espaço de coordenadas.
|
|
142
|
+
- **Janelas minimizadas**: `app_list` marca-as com `minimized: true` e não devolve `rect` (o Windows dá o retângulo marcador `-32000` enquanto minimizadas); `screen_shot` / `screen_read` falham com `WINDOW_MINIMIZED`; restaure a janela primeiro.
|
|
143
|
+
|
|
144
|
+
|
|
136
145
|
### Exemplos
|
|
137
146
|
|
|
138
147
|
Estes são os pontos de entrada reais do plugin; instale uma vez e chame um.
|
package/lib/index.js
CHANGED
|
@@ -623,7 +623,7 @@ var ObservationStore = class {
|
|
|
623
623
|
* @module dsh-click/version
|
|
624
624
|
*/
|
|
625
625
|
/** Plugin version advertised in the helper-protocol handshake. */
|
|
626
|
-
const VERSION = "0.3.
|
|
626
|
+
const VERSION = "0.3.29";
|
|
627
627
|
/** The helper protocol revision; bump only on incompatible wire changes. */
|
|
628
628
|
const HELPER_PROTOCOL_VERSION = 1;
|
|
629
629
|
//#endregion
|
|
@@ -668,12 +668,16 @@ function expectWindow(value, op) {
|
|
|
668
668
|
const processId = typeof processIdRaw === "number" ? processIdRaw : null;
|
|
669
669
|
const executablePath = record["executablePath"];
|
|
670
670
|
if (executablePath !== null && typeof executablePath !== "string") throw new DshClickError(`helper "${op}" returned malformed executablePath`, "BAD_HELPER_RESPONSE");
|
|
671
|
+
const minimized = record["minimized"] === true;
|
|
672
|
+
const rectValue = record["rect"];
|
|
673
|
+
if (rectValue !== null && minimized) throw new DshClickError(`helper "${op}" returned a rect for a minimized window`, "BAD_HELPER_RESPONSE");
|
|
671
674
|
return {
|
|
672
675
|
windowId: expectNumber(record, "windowId", op),
|
|
673
676
|
processId,
|
|
674
677
|
title: expectString(record, "title", op),
|
|
675
678
|
className: expectString(record, "className", op),
|
|
676
|
-
rect: expectRect(
|
|
679
|
+
rect: rectValue === null ? null : expectRect(rectValue, op),
|
|
680
|
+
minimized,
|
|
677
681
|
executablePath,
|
|
678
682
|
visible: record["visible"] !== false
|
|
679
683
|
};
|
|
@@ -696,6 +700,18 @@ function expectSnapshot(value, op) {
|
|
|
696
700
|
elementCount: expectNumber(record, "elementCount", op)
|
|
697
701
|
};
|
|
698
702
|
}
|
|
703
|
+
/** Validate the helper's capture-path report. */
|
|
704
|
+
function expectCaptureMethod(record, op) {
|
|
705
|
+
const value = expectString(record, "captureMethod", op);
|
|
706
|
+
if (value !== "printwindow-fullcontent" && value !== "printwindow" && value !== "copyfromscreen") throw new DshClickError(`helper "${op}" returned an unknown captureMethod`, "BAD_HELPER_RESPONSE");
|
|
707
|
+
return value;
|
|
708
|
+
}
|
|
709
|
+
/** Validate the helper's capture-scope report. */
|
|
710
|
+
function expectCaptureScope(record, op) {
|
|
711
|
+
const value = expectString(record, "scope", op);
|
|
712
|
+
if (value !== "window" && value !== "primary" && value !== "virtual") throw new DshClickError(`helper "${op}" returned an unknown scope`, "BAD_HELPER_RESPONSE");
|
|
713
|
+
return value;
|
|
714
|
+
}
|
|
699
715
|
/** Validate one helper element record. */
|
|
700
716
|
function expectElement(value, op) {
|
|
701
717
|
const record = expectRecordValue(value, op);
|
|
@@ -820,10 +836,11 @@ var HelperBackend = class {
|
|
|
820
836
|
if (!Array.isArray(result)) throw new DshClickError("helper \"windows\" returned a non-array result", "BAD_HELPER_RESPONSE");
|
|
821
837
|
return result.map((item) => expectWindow(item, "windows"));
|
|
822
838
|
}
|
|
823
|
-
async shot(ref, maxSide, signal) {
|
|
839
|
+
async shot(ref, maxSide, scope, signal) {
|
|
824
840
|
const record = expectRecordValue(await this.invoke("shot", {
|
|
825
841
|
target: ref,
|
|
826
842
|
maxSide,
|
|
843
|
+
scope,
|
|
827
844
|
maxElements: this.config.maxElements,
|
|
828
845
|
maxDepth: this.config.maxTreeDepth
|
|
829
846
|
}, signal), "shot");
|
|
@@ -831,6 +848,8 @@ var HelperBackend = class {
|
|
|
831
848
|
pngBase64: expectString(record, "pngBase64", "shot"),
|
|
832
849
|
width: expectNumber(record, "width", "shot"),
|
|
833
850
|
height: expectNumber(record, "height", "shot"),
|
|
851
|
+
captureMethod: expectCaptureMethod(record, "shot"),
|
|
852
|
+
scope: expectCaptureScope(record, "shot"),
|
|
834
853
|
snapshot: expectSnapshot(record["snapshot"], "shot")
|
|
835
854
|
};
|
|
836
855
|
}
|
|
@@ -1251,7 +1270,8 @@ function observedWindowInfo(info, maxTextLength) {
|
|
|
1251
1270
|
processId: info.processId,
|
|
1252
1271
|
title: sanitizeVisible(info.title, maxTextLength),
|
|
1253
1272
|
className: sanitizeVisible(info.className, maxTextLength),
|
|
1254
|
-
rect: info.rect,
|
|
1273
|
+
rect: info.rect === null ? null : { ...info.rect },
|
|
1274
|
+
minimized: info.minimized,
|
|
1255
1275
|
executablePath: info.executablePath === null ? null : sanitizePath(info.executablePath, maxTextLength),
|
|
1256
1276
|
visible: info.visible
|
|
1257
1277
|
};
|
|
@@ -1279,8 +1299,8 @@ function backendGuard(services) {
|
|
|
1279
1299
|
if (!services.backend.available) throw new Error(`dsh-click backend unavailable on ${services.backend.platform}: ${services.backend.unavailableReason ?? "unknown reason"}`);
|
|
1280
1300
|
}
|
|
1281
1301
|
/** Plain-text fallback description for a captured screenshot. */
|
|
1282
|
-
function shotDescription(window, width, height) {
|
|
1283
|
-
return `${width}x${height}
|
|
1302
|
+
function shotDescription(window, width, height, captureMethod) {
|
|
1303
|
+
return `${width}x${height} bitmap via ${captureMethod} of ${windowLine(window)}; the bitmap is downscaled to the requested longest side, while the window rectangle is ${window.rect.width}x${window.rect.height} physical pixels (snapshot.rect) - map screen coordinates with rect.width/width. Run screen_read on the same window for the structured element list and pixel positions.`;
|
|
1284
1304
|
}
|
|
1285
1305
|
/**
|
|
1286
1306
|
* `screen_shot` — capture the addressed window (or the primary screen).
|
|
@@ -1295,11 +1315,11 @@ function screenShotTool(services) {
|
|
|
1295
1315
|
const { ctx, config, backend, observations } = services;
|
|
1296
1316
|
return defineTool({
|
|
1297
1317
|
name: "screen_shot",
|
|
1298
|
-
description: "Capture a screenshot of a desktop window
|
|
1318
|
+
description: "Capture a screenshot of a desktop window, or of the primary screen when no target is given (a target-less shot never captures the foreground window). Window captures ask the window to paint itself with PrintWindow(PW_RENDERFULLCONTENT) first, so they are independent of other windows stacked on top; captureMethod reports which path produced the pixels. Returns an observationId that later actions cite in `basedOn`. When the current model accepts images the result includes the image; otherwise it includes a text description. Read-only: never needs approval.",
|
|
1299
1319
|
parameters: {
|
|
1300
1320
|
target: {
|
|
1301
1321
|
type: "object",
|
|
1302
|
-
description: "Which window to capture (windowId, windowTitle, or processId); omitted = the
|
|
1322
|
+
description: "Which window to capture (windowId, windowTitle, or processId); omitted = the primary screen.",
|
|
1303
1323
|
properties: {
|
|
1304
1324
|
windowId: {
|
|
1305
1325
|
type: "integer",
|
|
@@ -1316,9 +1336,14 @@ function screenShotTool(services) {
|
|
|
1316
1336
|
},
|
|
1317
1337
|
additionalProperties: false
|
|
1318
1338
|
},
|
|
1339
|
+
scope: {
|
|
1340
|
+
type: "string",
|
|
1341
|
+
enum: ["primary", "virtual"],
|
|
1342
|
+
description: "Only used without a target: capture the primary screen (default) or the whole virtual desktop (experimental - multi-monitor and negative-origin geometry are not fully verified)."
|
|
1343
|
+
},
|
|
1319
1344
|
maxSide: {
|
|
1320
1345
|
type: "integer",
|
|
1321
|
-
description: "Longest side
|
|
1346
|
+
description: "Longest side of the bitmap; larger captures are downscaled (the window rectangle in snapshot.rect stays in physical pixels)."
|
|
1322
1347
|
}
|
|
1323
1348
|
},
|
|
1324
1349
|
output: {
|
|
@@ -1364,6 +1389,22 @@ function screenShotTool(services) {
|
|
|
1364
1389
|
},
|
|
1365
1390
|
additionalProperties: false
|
|
1366
1391
|
},
|
|
1392
|
+
captureMethod: {
|
|
1393
|
+
type: "string",
|
|
1394
|
+
enum: [
|
|
1395
|
+
"printwindow-fullcontent",
|
|
1396
|
+
"printwindow",
|
|
1397
|
+
"copyfromscreen"
|
|
1398
|
+
]
|
|
1399
|
+
},
|
|
1400
|
+
scope: {
|
|
1401
|
+
type: "string",
|
|
1402
|
+
enum: [
|
|
1403
|
+
"window",
|
|
1404
|
+
"primary",
|
|
1405
|
+
"virtual"
|
|
1406
|
+
]
|
|
1407
|
+
},
|
|
1367
1408
|
description: { type: "string" }
|
|
1368
1409
|
},
|
|
1369
1410
|
additionalProperties: false
|
|
@@ -1384,13 +1425,14 @@ function screenShotTool(services) {
|
|
|
1384
1425
|
timeoutMs: config.helperTimeoutMs + 15e3,
|
|
1385
1426
|
async execute(args, exec) {
|
|
1386
1427
|
backendGuard(services);
|
|
1387
|
-
const
|
|
1388
|
-
const
|
|
1428
|
+
const parsed = args;
|
|
1429
|
+
const target = parsed.target ?? {};
|
|
1430
|
+
const requestedSide = parsed.maxSide;
|
|
1389
1431
|
const maxSide = requestedSide === void 0 ? config.maxScreenshotSide : Math.min(requestedSide, config.maxScreenshotSide);
|
|
1390
|
-
const shot = await backend.shot(target, maxSide, exec.signal);
|
|
1432
|
+
const shot = await backend.shot(target, maxSide, parsed.scope ?? "primary", exec.signal);
|
|
1391
1433
|
const sanitized = observedWindow(shot.snapshot, config.maxTextLength);
|
|
1392
1434
|
const record = observations.record(shot.snapshot);
|
|
1393
|
-
const description = shotDescription(sanitized, shot.width, shot.height);
|
|
1435
|
+
const description = shotDescription(sanitized, shot.width, shot.height, shot.captureMethod);
|
|
1394
1436
|
let image;
|
|
1395
1437
|
const attachments = ctx.get("attachments");
|
|
1396
1438
|
if (config.imageMode !== "text" && attachments !== void 0 && await sessionAcceptsImages(exec.agent, ctx, exec.signal)) try {
|
|
@@ -1422,6 +1464,8 @@ function screenShotTool(services) {
|
|
|
1422
1464
|
observationId: record.id,
|
|
1423
1465
|
window: sanitized,
|
|
1424
1466
|
...image !== void 0 ? { image } : {},
|
|
1467
|
+
captureMethod: shot.captureMethod,
|
|
1468
|
+
scope: shot.scope,
|
|
1425
1469
|
description
|
|
1426
1470
|
};
|
|
1427
1471
|
}
|
|
@@ -1603,7 +1647,7 @@ function screenFindTool(services) {
|
|
|
1603
1647
|
parameters: {
|
|
1604
1648
|
target: {
|
|
1605
1649
|
type: "object",
|
|
1606
|
-
description: "Which window to search (windowId, windowTitle, or processId); omitted = the foreground window.",
|
|
1650
|
+
description: "Which window to search (windowId, windowTitle, or processId); omitted = the primary screen (a target-less capture never uses the foreground window).",
|
|
1607
1651
|
properties: {
|
|
1608
1652
|
windowId: {
|
|
1609
1653
|
type: "integer",
|
|
@@ -1626,7 +1670,7 @@ function screenFindTool(services) {
|
|
|
1626
1670
|
},
|
|
1627
1671
|
maxSide: {
|
|
1628
1672
|
type: "integer",
|
|
1629
|
-
description: "Longest side
|
|
1673
|
+
description: "Longest side of the bitmap; larger captures are downscaled (coordinates returned are still screen pixels)."
|
|
1630
1674
|
}
|
|
1631
1675
|
},
|
|
1632
1676
|
output: {
|
|
@@ -1705,7 +1749,7 @@ function screenFindTool(services) {
|
|
|
1705
1749
|
const parsed = args;
|
|
1706
1750
|
const requestedSide = parsed.maxSide;
|
|
1707
1751
|
const maxSide = requestedSide === void 0 ? config.maxScreenshotSide : Math.min(requestedSide, config.maxScreenshotSide);
|
|
1708
|
-
const shot = await backend.shot(parsed.target ?? {}, maxSide, exec.signal);
|
|
1752
|
+
const shot = await backend.shot(parsed.target ?? {}, maxSide, "primary", exec.signal);
|
|
1709
1753
|
const sanitized = observedWindow(shot.snapshot, config.maxTextLength);
|
|
1710
1754
|
const record = observations.record(shot.snapshot);
|
|
1711
1755
|
const targets = [];
|
|
@@ -2095,7 +2139,7 @@ function appListTool(services) {
|
|
|
2095
2139
|
const { config, backend } = services;
|
|
2096
2140
|
return defineTool({
|
|
2097
2141
|
name: "app_list",
|
|
2098
|
-
description: "List running desktop applications and their visible windows (windowId, title, process id, executable path). Read-only: never needs approval. Use the returned windowIds as screen_shot/screen_read targets.",
|
|
2142
|
+
description: "List running desktop applications and their visible windows (windowId, title, process id, executable path). Read-only: never needs approval. Use the returned windowIds as screen_shot/screen_read targets. Minimized windows are listed with minimized: true and no rect (Windows reports a -32000 placeholder while iconified); restore them before capturing.",
|
|
2099
2143
|
parameters: {},
|
|
2100
2144
|
output: {
|
|
2101
2145
|
schema: {
|
|
@@ -2122,7 +2166,7 @@ function appListTool(services) {
|
|
|
2122
2166
|
processId: { oneOf: [{ type: "integer" }, { type: "null" }] },
|
|
2123
2167
|
title: { type: "string" },
|
|
2124
2168
|
className: { type: "string" },
|
|
2125
|
-
rect: {
|
|
2169
|
+
rect: { oneOf: [{
|
|
2126
2170
|
type: "object",
|
|
2127
2171
|
properties: {
|
|
2128
2172
|
x: { type: "integer" },
|
|
@@ -2131,7 +2175,8 @@ function appListTool(services) {
|
|
|
2131
2175
|
height: { type: "integer" }
|
|
2132
2176
|
},
|
|
2133
2177
|
additionalProperties: false
|
|
2134
|
-
},
|
|
2178
|
+
}, { type: "null" }] },
|
|
2179
|
+
minimized: { type: "boolean" },
|
|
2135
2180
|
executablePath: { oneOf: [{ type: "string" }, { type: "null" }] },
|
|
2136
2181
|
visible: { type: "boolean" }
|
|
2137
2182
|
},
|
|
@@ -2150,7 +2195,10 @@ function appListTool(services) {
|
|
|
2150
2195
|
const lines = [`${result.apps.length} running application(s) with windows:`];
|
|
2151
2196
|
for (const app of result.apps) {
|
|
2152
2197
|
lines.push(`- ${app.name} (pid ${app.processId}, ${app.executablePath ?? "unknown executable"})`);
|
|
2153
|
-
for (const window of app.windows)
|
|
2198
|
+
for (const window of app.windows) {
|
|
2199
|
+
const where = window.minimized ? "minimized (no rect; screen_shot refuses it until restored)" : window.rect === null ? "no rect reported" : `at (${window.rect.x}, ${window.rect.y}) ${window.rect.width}x${window.rect.height}`;
|
|
2200
|
+
lines.push(` - [windowId ${window.windowId}] ${JSON.stringify(window.title)} ${where}`);
|
|
2201
|
+
}
|
|
2154
2202
|
}
|
|
2155
2203
|
return [{
|
|
2156
2204
|
type: "text",
|
|
@@ -10,7 +10,7 @@
|
|
|
10
10
|
*/
|
|
11
11
|
import type { SubprocessRuntime } from '@deepseek-ai/dsh-subprocess';
|
|
12
12
|
import type { ResolvedConfig } from '../config.js';
|
|
13
|
-
import { type ActionOutcome, type AppInfo, type ClickRequest, type DesktopBackend, type KeyRequest, type LaunchOutcome, type Screenshot, type ScrollRequest, type Tree, type WindowInfo, type WindowRef, type WindowSnapshot, type TypeRequest } from './types.js';
|
|
13
|
+
import { type ActionOutcome, type AppInfo, type ClickRequest, type DesktopBackend, type KeyRequest, type LaunchOutcome, type Screenshot, type ScreenScope, type ScrollRequest, type Tree, type WindowInfo, type WindowRef, type WindowSnapshot, type TypeRequest } from './types.js';
|
|
14
14
|
/** The Windows helper backend over `ctx.subprocess`. */
|
|
15
15
|
export declare class HelperBackend implements DesktopBackend {
|
|
16
16
|
private readonly config;
|
|
@@ -27,7 +27,7 @@ export declare class HelperBackend implements DesktopBackend {
|
|
|
27
27
|
/** Resolve (and cache) the powershell executable through the subprocess seam. */
|
|
28
28
|
private resolvePowershell;
|
|
29
29
|
listWindows(signal?: AbortSignal): Promise<WindowInfo[]>;
|
|
30
|
-
shot(ref: WindowRef, maxSide: number, signal?: AbortSignal): Promise<Screenshot>;
|
|
30
|
+
shot(ref: WindowRef, maxSide: number, scope: ScreenScope, signal?: AbortSignal): Promise<Screenshot>;
|
|
31
31
|
tree(ref: WindowRef, maxElements: number, maxDepth: number, includePixels: boolean, signal?: AbortSignal): Promise<Tree>;
|
|
32
32
|
snapshot(windowId: number, signal?: AbortSignal): Promise<WindowSnapshot>;
|
|
33
33
|
click(request: ClickRequest, focusFallback: boolean, signal?: AbortSignal): Promise<ActionOutcome>;
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"runner.d.ts","sourceRoot":"","sources":["../../../src/platform/runner.ts"],"names":[],"mappings":"AAAA;;;;;;;;;GASG;AAMH,OAAO,KAAK,EAAE,iBAAiB,EAAE,MAAM,6BAA6B,CAAA;AACpE,OAAO,KAAK,EAAE,cAAc,EAAE,MAAM,cAAc,CAAA;AAElD,OAAO,EAML,KAAK,aAAa,EAClB,KAAK,OAAO,
|
|
1
|
+
{"version":3,"file":"runner.d.ts","sourceRoot":"","sources":["../../../src/platform/runner.ts"],"names":[],"mappings":"AAAA;;;;;;;;;GASG;AAMH,OAAO,KAAK,EAAE,iBAAiB,EAAE,MAAM,6BAA6B,CAAA;AACpE,OAAO,KAAK,EAAE,cAAc,EAAE,MAAM,cAAc,CAAA;AAElD,OAAO,EAML,KAAK,aAAa,EAClB,KAAK,OAAO,EAGZ,KAAK,YAAY,EACjB,KAAK,cAAc,EAInB,KAAK,UAAU,EACf,KAAK,aAAa,EAGlB,KAAK,UAAU,EACf,KAAK,WAAW,EAChB,KAAK,aAAa,EAClB,KAAK,IAAI,EACT,KAAK,UAAU,EACf,KAAK,SAAS,EACd,KAAK,cAAc,EACnB,KAAK,WAAW,EACjB,MAAM,YAAY,CAAA;AAuJnB,wDAAwD;AACxD,qBAAa,aAAc,YAAW,cAAc;IAShD,OAAO,CAAC,QAAQ,CAAC,MAAM;IACvB,OAAO,CAAC,QAAQ,CAAC,UAAU;IAT7B,gEAAgE;IAChE,OAAO,CAAC,UAAU,CAA6B;IAE/C,QAAQ,CAAC,SAAS,QAAO;IAEzB,mEAAmE;IACnE,uEAAuE;gBAEpD,MAAM,EAAE,cAAc,EACtB,UAAU,EAAE,iBAAiB;IAGhD,QAAQ,CAAC,QAAQ,WAAU;IAE3B,wEAAwE;YAC1D,MAAM;IAsDpB,iFAAiF;IACjF,OAAO,CAAC,iBAAiB;IAanB,WAAW,CAAC,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,UAAU,EAAE,CAAC;IAMxD,IAAI,CAAC,GAAG,EAAE,SAAS,EAAE,OAAO,EAAE,MAAM,EAAE,KAAK,EAAE,WAAW,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,UAAU,CAAC;IAmBpG,IAAI,CAAC,GAAG,EAAE,SAAS,EAAE,WAAW,EAAE,MAAM,EAAE,QAAQ,EAAE,MAAM,EAAE,aAAa,EAAE,OAAO,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,IAAI,CAAC;IAexH,QAAQ,CAAC,QAAQ,EAAE,MAAM,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,cAAc,CAAC;IASzE,KAAK,CAAC,OAAO,EAAE,YAAY,EAAE,aAAa,EAAE,OAAO,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,aAAa,CAAC;IAKlG,IAAI,CAAC,OAAO,EAAE,WAAW,EAAE,aAAa,EAAE,OAAO,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,aAAa,CAAC;IAKhG,MAAM,CAAC,OAAO,EAAE,aAAa,EAAE,aAAa,EAAE,OAAO,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,aAAa,CAAC;IAKpG,GAAG,CAAC,OAAO,EAAE,UAAU,EAAE,aAAa,EAAE,OAAO,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,aAAa,CAAC;IAK9F,IAAI,CAAC,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,OAAO,EAAE,CAAC;IAoB9C,MAAM,CAAC,IAAI,EAAE,MAAM,EAAE,IAAI,EAAE,SAAS,MAAM,EAAE,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,aAAa,CAAC;CAYlG"}
|
|
@@ -24,6 +24,20 @@ export interface WindowRef {
|
|
|
24
24
|
/** Owning process id. */
|
|
25
25
|
processId?: number;
|
|
26
26
|
}
|
|
27
|
+
/** Which screen region a target-less capture covers. */
|
|
28
|
+
export type ScreenScope = 'primary' | 'virtual';
|
|
29
|
+
/** What a capture actually covered: one window, the primary screen, or the virtual desktop. */
|
|
30
|
+
export type CaptureScope = 'window' | ScreenScope;
|
|
31
|
+
/**
|
|
32
|
+
* Which helper path produced the pixels.
|
|
33
|
+
*
|
|
34
|
+
* `printwindow-fullcontent` is `PrintWindow(hwnd, hdc, PW_RENDERFULLCONTENT)`:
|
|
35
|
+
* the window paints itself, so the image does not depend on what is stacked on
|
|
36
|
+
* top of it. `printwindow` is the legacy flag-0 call (some old windows only
|
|
37
|
+
* paint through it). `copyfromscreen` is the last-resort screen-region copy and
|
|
38
|
+
* is only correct while the window is unobscured.
|
|
39
|
+
*/
|
|
40
|
+
export type CaptureMethod = 'printwindow-fullcontent' | 'printwindow' | 'copyfromscreen';
|
|
27
41
|
/** Screen-space rectangle, integer pixel coordinates. */
|
|
28
42
|
export interface Rect {
|
|
29
43
|
x: number;
|
|
@@ -37,7 +51,14 @@ export interface WindowInfo {
|
|
|
37
51
|
processId: number | null;
|
|
38
52
|
title: string;
|
|
39
53
|
className: string;
|
|
40
|
-
|
|
54
|
+
/**
|
|
55
|
+
* Physical screen rectangle, or `null` for a minimized window: Windows reports
|
|
56
|
+
* the `CW_USEDEFAULT` placeholder (`-32000,-32000 314x50`) while iconified,
|
|
57
|
+
* which must never reach coordinate math.
|
|
58
|
+
*/
|
|
59
|
+
rect: Rect | null;
|
|
60
|
+
/** True while the window is iconified (`IsIconic`); such a window cannot be captured. */
|
|
61
|
+
minimized: boolean;
|
|
41
62
|
executablePath: string | null;
|
|
42
63
|
visible: boolean;
|
|
43
64
|
}
|
|
@@ -90,8 +111,18 @@ export interface WindowSnapshot {
|
|
|
90
111
|
export interface Screenshot {
|
|
91
112
|
/** PNG bytes, base64-encoded. */
|
|
92
113
|
pngBase64: string;
|
|
114
|
+
/**
|
|
115
|
+
* Bitmap width after the `maxSide` downscale. This is NOT the window width:
|
|
116
|
+
* `snapshot.rect` always carries the physical window/screen rectangle, so a
|
|
117
|
+
* screen-to-bitmap mapping is `bitmapX = (screenX - rect.x) * width / rect.width`.
|
|
118
|
+
*/
|
|
93
119
|
width: number;
|
|
120
|
+
/** Bitmap height after the `maxSide` downscale (see {@link width}). */
|
|
94
121
|
height: number;
|
|
122
|
+
/** Which capture path produced the pixels. */
|
|
123
|
+
captureMethod: CaptureMethod;
|
|
124
|
+
/** What was captured. `virtual` is experimental (multi-monitor geometry untested). */
|
|
125
|
+
scope: CaptureScope;
|
|
95
126
|
snapshot: WindowSnapshot;
|
|
96
127
|
}
|
|
97
128
|
/** An accessibility tree read plus pixel hints. */
|
|
@@ -169,8 +200,12 @@ export interface DesktopBackend {
|
|
|
169
200
|
readonly unavailableReason?: string;
|
|
170
201
|
/** List every visible top-level window across running applications. */
|
|
171
202
|
listWindows(signal?: AbortSignal): Promise<WindowInfo[]>;
|
|
172
|
-
/**
|
|
173
|
-
|
|
203
|
+
/**
|
|
204
|
+
* Capture a screenshot of the addressed window, or of a screen region when
|
|
205
|
+
* the ref names no window (the helper never substitutes the foreground
|
|
206
|
+
* window here; the primary screen is the target-less default).
|
|
207
|
+
*/
|
|
208
|
+
shot(ref: WindowRef, maxSide: number, scope: ScreenScope, signal?: AbortSignal): Promise<Screenshot>;
|
|
174
209
|
/** Read the accessibility tree plus pixel hints of the addressed window. */
|
|
175
210
|
tree(ref: WindowRef, maxElements: number, maxDepth: number, includePixels: boolean, signal?: AbortSignal): Promise<Tree>;
|
|
176
211
|
/** Re-capture fresh identity + hashes for one window (the stale check). */
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"types.d.ts","sourceRoot":"","sources":["../../../src/platform/types.ts"],"names":[],"mappings":"AAAA;;;;;;;;GAQG;AAEH,OAAO,KAAK,EAAE,YAAY,EAAE,MAAM,cAAc,CAAA;AAEhD,iFAAiF;AACjF,qBAAa,aAAc,SAAQ,KAAK;IAGT,QAAQ,CAAC,IAAI,EAAE,MAAM;IAFlD,+DAA+D;IAC/D,4DAA4D;gBAChD,OAAO,EAAE,MAAM,EAAW,IAAI,EAAE,MAAM;CAInD;AAED,+DAA+D;AAC/D,MAAM,WAAW,SAAS;IACxB,8CAA8C;IAC9C,QAAQ,CAAC,EAAE,MAAM,CAAA;IACjB,+EAA+E;IAC/E,WAAW,CAAC,EAAE,MAAM,CAAA;IACpB,yBAAyB;IACzB,SAAS,CAAC,EAAE,MAAM,CAAA;CACnB;AAED,yDAAyD;AACzD,MAAM,WAAW,IAAI;IACnB,CAAC,EAAE,MAAM,CAAA;IACT,CAAC,EAAE,MAAM,CAAA;IACT,KAAK,EAAE,MAAM,CAAA;IACb,MAAM,EAAE,MAAM,CAAA;CACf;AAED,wCAAwC;AACxC,MAAM,WAAW,UAAU;IACzB,QAAQ,EAAE,MAAM,CAAA;IAChB,SAAS,EAAE,MAAM,GAAG,IAAI,CAAA;IACxB,KAAK,EAAE,MAAM,CAAA;IACb,SAAS,EAAE,MAAM,CAAA;IACjB,IAAI,EAAE,IAAI,CAAA;
|
|
1
|
+
{"version":3,"file":"types.d.ts","sourceRoot":"","sources":["../../../src/platform/types.ts"],"names":[],"mappings":"AAAA;;;;;;;;GAQG;AAEH,OAAO,KAAK,EAAE,YAAY,EAAE,MAAM,cAAc,CAAA;AAEhD,iFAAiF;AACjF,qBAAa,aAAc,SAAQ,KAAK;IAGT,QAAQ,CAAC,IAAI,EAAE,MAAM;IAFlD,+DAA+D;IAC/D,4DAA4D;gBAChD,OAAO,EAAE,MAAM,EAAW,IAAI,EAAE,MAAM;CAInD;AAED,+DAA+D;AAC/D,MAAM,WAAW,SAAS;IACxB,8CAA8C;IAC9C,QAAQ,CAAC,EAAE,MAAM,CAAA;IACjB,+EAA+E;IAC/E,WAAW,CAAC,EAAE,MAAM,CAAA;IACpB,yBAAyB;IACzB,SAAS,CAAC,EAAE,MAAM,CAAA;CACnB;AAED,wDAAwD;AACxD,MAAM,MAAM,WAAW,GAAG,SAAS,GAAG,SAAS,CAAA;AAE/C,+FAA+F;AAC/F,MAAM,MAAM,YAAY,GAAG,QAAQ,GAAG,WAAW,CAAA;AAEjD;;;;;;;;GAQG;AACH,MAAM,MAAM,aAAa,GAAG,yBAAyB,GAAG,aAAa,GAAG,gBAAgB,CAAA;AAExF,yDAAyD;AACzD,MAAM,WAAW,IAAI;IACnB,CAAC,EAAE,MAAM,CAAA;IACT,CAAC,EAAE,MAAM,CAAA;IACT,KAAK,EAAE,MAAM,CAAA;IACb,MAAM,EAAE,MAAM,CAAA;CACf;AAED,wCAAwC;AACxC,MAAM,WAAW,UAAU;IACzB,QAAQ,EAAE,MAAM,CAAA;IAChB,SAAS,EAAE,MAAM,GAAG,IAAI,CAAA;IACxB,KAAK,EAAE,MAAM,CAAA;IACb,SAAS,EAAE,MAAM,CAAA;IACjB;;;;OAIG;IACH,IAAI,EAAE,IAAI,GAAG,IAAI,CAAA;IACjB,yFAAyF;IACzF,SAAS,EAAE,OAAO,CAAA;IAClB,cAAc,EAAE,MAAM,GAAG,IAAI,CAAA;IAC7B,OAAO,EAAE,OAAO,CAAA;CACjB;AAED,4DAA4D;AAC5D,MAAM,WAAW,WAAW;IAC1B,kEAAkE;IAClE,SAAS,EAAE,MAAM,CAAA;IACjB,6CAA6C;IAC7C,WAAW,EAAE,MAAM,CAAA;IACnB,8CAA8C;IAC9C,IAAI,EAAE,MAAM,CAAA;IACZ,uCAAuC;IACvC,YAAY,EAAE,MAAM,CAAA;IACpB,gDAAgD;IAChD,IAAI,EAAE,IAAI,CAAA;IACV,sCAAsC;IACtC,OAAO,EAAE,OAAO,CAAA;IAChB,mGAAmG;IACnG,QAAQ,EAAE,MAAM,EAAE,CAAA;CACnB;AAED,uEAAuE;AACvE,MAAM,WAAW,SAAS;IACxB,kDAAkD;IAClD,KAAK,EAAE,MAAM,CAAA;IACb,2BAA2B;IAC3B,CAAC,EAAE,MAAM,CAAA;IACT,2BAA2B;IAC3B,CAAC,EAAE,MAAM,CAAA;IACT,sDAAsD;IACtD,KAAK,EAAE,MAAM,CAAA;CACd;AAED,oFAAoF;AACpF,MAAM,WAAW,cAAc;IAC7B,QAAQ,EAAE,MAAM,CAAA;IAChB,SAAS,EAAE,MAAM,CAAA;IACjB,cAAc,EAAE,MAAM,GAAG,IAAI,CAAA;IAC7B,KAAK,EAAE,MAAM,CAAA;IACb,SAAS,EAAE,MAAM,CAAA;IACjB,IAAI,EAAE,IAAI,CAAA;IACV,0DAA0D;IAC1D,UAAU,EAAE,OAAO,CAAA;IACnB,4EAA4E;IAC5E,QAAQ,EAAE,MAAM,CAAA;IAChB,2CAA2C;IAC3C,QAAQ,EAAE,MAAM,CAAA;IAChB,iDAAiD;IACjD,YAAY,EAAE,MAAM,CAAA;CACrB;AAED,uEAAuE;AACvE,MAAM,WAAW,UAAU;IACzB,iCAAiC;IACjC,SAAS,EAAE,MAAM,CAAA;IACjB;;;;OAIG;IACH,KAAK,EAAE,MAAM,CAAA;IACb,uEAAuE;IACvE,MAAM,EAAE,MAAM,CAAA;IACd,8CAA8C;IAC9C,aAAa,EAAE,aAAa,CAAA;IAC5B,sFAAsF;IACtF,KAAK,EAAE,YAAY,CAAA;IACnB,QAAQ,EAAE,cAAc,CAAA;CACzB;AAED,mDAAmD;AACnD,MAAM,WAAW,IAAI;IACnB,QAAQ,EAAE,cAAc,CAAA;IACxB,QAAQ,EAAE,WAAW,EAAE,CAAA;IACvB,MAAM,EAAE,SAAS,EAAE,CAAA;CACpB;AAED,uCAAuC;AACvC,MAAM,WAAW,OAAO;IACtB,SAAS,EAAE,MAAM,CAAA;IACjB,IAAI,EAAE,MAAM,CAAA;IACZ,cAAc,EAAE,MAAM,GAAG,IAAI,CAAA;IAC7B,OAAO,EAAE,UAAU,EAAE,CAAA;CACtB;AAED,iEAAiE;AACjE,MAAM,WAAW,YAAY;IAC3B,QAAQ,EAAE,MAAM,CAAA;IAChB,SAAS,CAAC,EAAE,MAAM,CAAA;IAClB,CAAC,CAAC,EAAE,MAAM,CAAA;IACV,CAAC,CAAC,EAAE,MAAM,CAAA;IACV,MAAM,EAAE,MAAM,GAAG,OAAO,CAAA;CACzB;AAED,2DAA2D;AAC3D,MAAM,WAAW,WAAW;IAC1B,QAAQ,EAAE,MAAM,CAAA;IAChB,SAAS,EAAE,MAAM,CAAA;IACjB,IAAI,EAAE,MAAM,CAAA;IACZ,4EAA4E;IAC5E,QAAQ,EAAE,OAAO,CAAA;CAClB;AAED,wBAAwB;AACxB,MAAM,WAAW,aAAa;IAC5B,QAAQ,EAAE,MAAM,CAAA;IAChB,SAAS,CAAC,EAAE,MAAM,CAAA;IAClB,SAAS,EAAE,IAAI,GAAG,MAAM,GAAG,SAAS,GAAG,WAAW,CAAA;IAClD,2CAA2C;IAC3C,MAAM,EAAE,MAAM,CAAA;CACf;AAED,0DAA0D;AAC1D,MAAM,WAAW,UAAU;IACzB,QAAQ,EAAE,MAAM,CAAA;IAChB,IAAI,EAAE,MAAM,CAAA;CACb;AAED,6EAA6E;AAC7E,MAAM,WAAW,aAAa;IAC5B,QAAQ,EAAE,MAAM,CAAA;IAChB,6CAA6C;IAC7C,MAAM,EAAE,MAAM,CAAA;IACd,wCAAwC;IACxC,SAAS,EAAE,KAAK,GAAG,QAAQ,GAAG,MAAM,CAAA;IACpC,aAAa,EAAE,YAAY,CAAA;IAC3B,YAAY,EAAE,YAAY,CAAA;IAC1B,2EAA2E;IAC3E,QAAQ,CAAC,EAAE,OAAO,CAAA;IAClB,gCAAgC;IAChC,MAAM,CAAC,EAAE,MAAM,CAAA;CAChB;AAED,4CAA4C;AAC5C,MAAM,WAAW,aAAa;IAC5B,SAAS,EAAE,MAAM,CAAA;IACjB,cAAc,EAAE,MAAM,GAAG,IAAI,CAAA;CAC9B;AAED;;;;GAIG;AACH,MAAM,WAAW,cAAc;IAC7B,4EAA4E;IAC5E,QAAQ,CAAC,QAAQ,EAAE,MAAM,CAAA;IACzB,iFAAiF;IACjF,QAAQ,CAAC,SAAS,EAAE,OAAO,CAAA;IAC3B,kDAAkD;IAClD,QAAQ,CAAC,iBAAiB,CAAC,EAAE,MAAM,CAAA;IAEnC,uEAAuE;IACvE,WAAW,CAAC,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,UAAU,EAAE,CAAC,CAAA;IACxD;;;;OAIG;IACH,IAAI,CAAC,GAAG,EAAE,SAAS,EAAE,OAAO,EAAE,MAAM,EAAE,KAAK,EAAE,WAAW,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,UAAU,CAAC,CAAA;IACpG,4EAA4E;IAC5E,IAAI,CAAC,GAAG,EAAE,SAAS,EAAE,WAAW,EAAE,MAAM,EAAE,QAAQ,EAAE,MAAM,EAAE,aAAa,EAAE,OAAO,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,IAAI,CAAC,CAAA;IACxH,2EAA2E;IAC3E,QAAQ,CAAC,QAAQ,EAAE,MAAM,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,cAAc,CAAC,CAAA;IACzE,gDAAgD;IAChD,KAAK,CAAC,OAAO,EAAE,YAAY,EAAE,aAAa,EAAE,OAAO,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,aAAa,CAAC,CAAA;IAClG,yCAAyC;IACzC,IAAI,CAAC,OAAO,EAAE,WAAW,EAAE,aAAa,EAAE,OAAO,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,aAAa,CAAC,CAAA;IAChG,2CAA2C;IAC3C,MAAM,CAAC,OAAO,EAAE,aAAa,EAAE,aAAa,EAAE,OAAO,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,aAAa,CAAC,CAAA;IACpG,mDAAmD;IACnD,GAAG,CAAC,OAAO,EAAE,UAAU,EAAE,aAAa,EAAE,OAAO,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,aAAa,CAAC,CAAA;IAC9F,yCAAyC;IACzC,IAAI,CAAC,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,OAAO,EAAE,CAAC,CAAA;IAC9C,8CAA8C;IAC9C,MAAM,CAAC,IAAI,EAAE,MAAM,EAAE,IAAI,EAAE,SAAS,MAAM,EAAE,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,aAAa,CAAC,CAAA;CAC5F;AAED,iEAAiE;AACjE,MAAM,WAAW,aAAa;IAC5B,QAAQ,EAAE,MAAM,CAAA;IAChB,aAAa,EAAE,MAAM,CAAA;IACrB,SAAS,EAAE,MAAM,CAAA;IACjB,EAAE,EAAE,MAAM,CAAA;IACV,IAAI,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAA;CAC9B;AAED,kFAAkF;AAClF,MAAM,MAAM,cAAc,GACtB;IAAE,EAAE,EAAE,IAAI,CAAC;IAAC,MAAM,EAAE,OAAO,CAAA;CAAE,GAC7B;IAAE,EAAE,EAAE,KAAK,CAAC;IAAC,KAAK,EAAE;QAAE,IAAI,EAAE,MAAM,CAAC;QAAC,OAAO,EAAE,MAAM,CAAA;KAAE,CAAA;CAAE,CAAA;AAE3D,gFAAgF;AAChF,wBAAgB,YAAY,CAAC,MAAM,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,EAAE,KAAK,EAAE,MAAM,EAAE,EAAE,EAAE,MAAM,GAAG,MAAM,CAM/F;AAED,yEAAyE;AACzE,wBAAgB,YAAY,CAAC,MAAM,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,EAAE,KAAK,EAAE,MAAM,EAAE,EAAE,EAAE,MAAM,GAAG,MAAM,CAM/F;AAED,+EAA+E;AAC/E,wBAAgB,YAAY,CAAC,MAAM,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,EAAE,KAAK,EAAE,MAAM,EAAE,EAAE,EAAE,MAAM,GAAG,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAMhH;AAED,8EAA8E;AAC9E,wBAAgB,iBAAiB,CAAC,KAAK,EAAE,OAAO,EAAE,EAAE,EAAE,MAAM,GAAG,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAKrF;AAED,oFAAoF;AACpF,wBAAgB,kBAAkB,CAAC,KAAK,EAAE,OAAO,EAAE,EAAE,EAAE,MAAM,GAAG,YAAY,CAW3E"}
|
package/lib/types/tools.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"tools.d.ts","sourceRoot":"","sources":["../../src/tools.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;GAWG;AAEH,OAAO,KAAK,EAAE,OAAO,EAAE,MAAM,qBAAqB,CAAA;AAKlD,OAAO,EAAE,cAAc,EAAE,MAAM,cAAc,CAAA;AAC7C,OAAO,KAAK,EAAE,cAAc,EAAE,MAAM,aAAa,CAAA;AAEjD,OAAO,EAAE,gBAAgB,EAAE,MAAM,cAAc,CAAA;AAG/C,OAAO,EAA0B,KAAK,WAAW,EAAE,KAAK,uBAAuB,EAAE,MAAM,mBAAmB,CAAA;AAC1G,OAAO,KAAK,EAAE,cAAc,EAA0B,IAAI,EAAyC,MAAM,qBAAqB,CAAA;AAE9H,wEAAwE;AACxE,MAAM,WAAW,YAAY;IAC3B,4BAA4B;IAC5B,QAAQ,CAAC,GAAG,EAAE,OAAO,CAAA;IACrB,8BAA8B;IAC9B,QAAQ,CAAC,MAAM,EAAE,cAAc,CAAA;IAC/B,kCAAkC;IAClC,QAAQ,CAAC,OAAO,EAAE,cAAc,CAAA;IAChC,6BAA6B;IAC7B,QAAQ,CAAC,YAAY,EAAE,gBAAgB,CAAA;IACvC,2BAA2B;IAC3B,QAAQ,CAAC,OAAO,EAAE,cAAc,CAAA;IAChC,0CAA0C;IAC1C,QAAQ,CAAC,GAAG,EAAE,WAAW,CAAA;IACzB,8CAA8C;IAC9C,QAAQ,CAAC,SAAS,EAAE,uBAAuB,CAAA;CAC5C;AAED,4EAA4E;AAC5E,MAAM,WAAW,mBAAmB;IAClC,QAAQ,EAAE,MAAM,CAAA;IAChB,SAAS,EAAE,MAAM,CAAA;IACjB,KAAK,EAAE,MAAM,CAAA;IACb,SAAS,EAAE,MAAM,CAAA;IACjB,cAAc,EAAE,MAAM,GAAG,IAAI,CAAA;IAC7B,IAAI,EAAE,IAAI,CAAA;IACV,UAAU,EAAE,OAAO,CAAA;CACpB;
|
|
1
|
+
{"version":3,"file":"tools.d.ts","sourceRoot":"","sources":["../../src/tools.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;GAWG;AAEH,OAAO,KAAK,EAAE,OAAO,EAAE,MAAM,qBAAqB,CAAA;AAKlD,OAAO,EAAE,cAAc,EAAE,MAAM,cAAc,CAAA;AAC7C,OAAO,KAAK,EAAE,cAAc,EAAE,MAAM,aAAa,CAAA;AAEjD,OAAO,EAAE,gBAAgB,EAAE,MAAM,cAAc,CAAA;AAG/C,OAAO,EAA0B,KAAK,WAAW,EAAE,KAAK,uBAAuB,EAAE,MAAM,mBAAmB,CAAA;AAC1G,OAAO,KAAK,EAAE,cAAc,EAA0B,IAAI,EAAyC,MAAM,qBAAqB,CAAA;AAE9H,wEAAwE;AACxE,MAAM,WAAW,YAAY;IAC3B,4BAA4B;IAC5B,QAAQ,CAAC,GAAG,EAAE,OAAO,CAAA;IACrB,8BAA8B;IAC9B,QAAQ,CAAC,MAAM,EAAE,cAAc,CAAA;IAC/B,kCAAkC;IAClC,QAAQ,CAAC,OAAO,EAAE,cAAc,CAAA;IAChC,6BAA6B;IAC7B,QAAQ,CAAC,YAAY,EAAE,gBAAgB,CAAA;IACvC,2BAA2B;IAC3B,QAAQ,CAAC,OAAO,EAAE,cAAc,CAAA;IAChC,0CAA0C;IAC1C,QAAQ,CAAC,GAAG,EAAE,WAAW,CAAA;IACzB,8CAA8C;IAC9C,QAAQ,CAAC,SAAS,EAAE,uBAAuB,CAAA;CAC5C;AAED,4EAA4E;AAC5E,MAAM,WAAW,mBAAmB;IAClC,QAAQ,EAAE,MAAM,CAAA;IAChB,SAAS,EAAE,MAAM,CAAA;IACjB,KAAK,EAAE,MAAM,CAAA;IACb,SAAS,EAAE,MAAM,CAAA;IACjB,cAAc,EAAE,MAAM,GAAG,IAAI,CAAA;IAC7B,IAAI,EAAE,IAAI,CAAA;IACV,UAAU,EAAE,OAAO,CAAA;CACpB;AA0FD;;;;;;;;GAQG;AACH,wBAAgB,cAAc,CAAC,QAAQ,EAAE,YAAY,mDAiJpD;AAED;;;;;;;GAOG;AACH,wBAAgB,cAAc,CAAC,QAAQ,EAAE,YAAY,mDA+IpD;AAED;;;;;;;;;;GAUG;AACH,wBAAgB,cAAc,CAAC,QAAQ,EAAE,YAAY,mDAsJpD;AAiED;;;;;;;GAOG;AACH,wBAAgB,SAAS,CAAC,QAAQ,EAAE,YAAY,mDAoD/C;AAED;;;;;;;GAOG;AACH,wBAAgB,QAAQ,CAAC,QAAQ,EAAE,YAAY,mDAyC9C;AAED;;;;;;GAMG;AACH,wBAAgB,UAAU,CAAC,QAAQ,EAAE,YAAY,mDA0ChD;AAED;;;;;;;GAOG;AACH,wBAAgB,OAAO,CAAC,QAAQ,EAAE,YAAY,mDAkC7C;AAED;;;;;;GAMG;AACH,wBAAgB,WAAW,CAAC,QAAQ,EAAE,YAAY,mDAuFjD;AAED;;;;;;GAMG;AACH,wBAAgB,aAAa,CAAC,QAAQ,EAAE,YAAY,mDAwCnD;AAED,oDAAoD;AACpD,wBAAgB,QAAQ,CAAC,QAAQ,EAAE,YAAY,qDAY9C"}
|
package/lib/types/version.d.ts
CHANGED
|
@@ -7,7 +7,7 @@
|
|
|
7
7
|
* @module dsh-click/version
|
|
8
8
|
*/
|
|
9
9
|
/** Plugin version advertised in the helper-protocol handshake. */
|
|
10
|
-
export declare const VERSION = "0.3.
|
|
10
|
+
export declare const VERSION = "0.3.29";
|
|
11
11
|
/** The helper protocol revision; bump only on incompatible wire changes. */
|
|
12
12
|
export declare const HELPER_PROTOCOL_VERSION = 1;
|
|
13
13
|
//# sourceMappingURL=version.d.ts.map
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
|
|
1
|
+
# dsh-click native helper (Windows PowerShell 5.1).
|
|
2
2
|
#
|
|
3
3
|
# One request = one process: JSON request on stdin, JSON response on stdout
|
|
4
4
|
# (exit code 0 even for handled failures; the error envelope is the response).
|
|
@@ -6,7 +6,8 @@
|
|
|
6
6
|
# any other stream, and nothing here steals foreground focus unless the
|
|
7
7
|
# `focusFallback` flag is explicitly true (config-gated, approval-gated).
|
|
8
8
|
#
|
|
9
|
-
# Operations: windows, apps, shot, tree, snapshot, click, type, scroll,
|
|
9
|
+
# Operations: windows, apps, metrics, shot, tree, snapshot, click, type, scroll,
|
|
10
|
+
# key, launch.
|
|
10
11
|
# The helper is NOT a security boundary by itself - the harness-side
|
|
11
12
|
# freshness check, approval gate, and process-identity check are. This file
|
|
12
13
|
# only refuses to invent global input: actions address one window by handle.
|
|
@@ -150,6 +151,13 @@ public static class DshClickWin32 {
|
|
|
150
151
|
[DllImport("user32.dll")]
|
|
151
152
|
public static extern int GetSystemMetrics(int index);
|
|
152
153
|
|
|
154
|
+
// PrintWindow: ask a window to paint itself into a DC. Flag 2 is
|
|
155
|
+
// PW_RENDERFULLCONTENT (Windows 8.1+), which makes DWM/DirectComposition
|
|
156
|
+
// windows render their real content instead of a blank or chrome-only
|
|
157
|
+
// frame; flag 0 is the legacy path for windows that ignore the newer flag.
|
|
158
|
+
[DllImport("user32.dll", SetLastError = true)]
|
|
159
|
+
public static extern bool PrintWindow(IntPtr hWnd, IntPtr hdcBlt, uint nFlags);
|
|
160
|
+
|
|
153
161
|
// DPI awareness. The helper is hosted by powershell.exe, whose manifest
|
|
154
162
|
// declares no DPI awareness, so GetWindowRect/GetSystemMetrics/GetClientRect
|
|
155
163
|
// return logical pixels while UIA BoundingRectangle and the screen DC are
|
|
@@ -346,8 +354,9 @@ function Resolve-Window($target) {
|
|
|
346
354
|
$hwnd = ConvertTo-Hwnd $target.windowId
|
|
347
355
|
if ($hwnd -ne [IntPtr]::Zero) {
|
|
348
356
|
$rect = Get-WindowRectInfo $hwnd
|
|
349
|
-
if ($rect.width -le 0 -
|
|
357
|
+
if ($rect.width -le 0 -or $rect.height -le 0) {
|
|
350
358
|
# A handle that no longer resolves to a window.
|
|
359
|
+
$script:ErrorCode = 'WINDOW_NOT_FOUND'
|
|
351
360
|
throw "window $($target.windowId) not found"
|
|
352
361
|
}
|
|
353
362
|
return $hwnd
|
|
@@ -357,6 +366,7 @@ function Resolve-Window($target) {
|
|
|
357
366
|
foreach ($candidate in [DshClickWin32]::EnumVisibleWindows()) {
|
|
358
367
|
if ([DshClickWin32]::WindowTitle($candidate) -like "*$title*") { return $candidate }
|
|
359
368
|
}
|
|
369
|
+
$script:ErrorCode = 'WINDOW_NOT_FOUND'
|
|
360
370
|
throw "no visible window matches title '$title'"
|
|
361
371
|
}
|
|
362
372
|
$processId = $target.processId
|
|
@@ -366,13 +376,41 @@ function Resolve-Window($target) {
|
|
|
366
376
|
[void][DshClickWin32]::GetWindowThreadProcessId($candidate, [ref]$pidValue)
|
|
367
377
|
if ([int]$pidValue -eq [int]$processId) { return $candidate }
|
|
368
378
|
}
|
|
379
|
+
$script:ErrorCode = 'WINDOW_NOT_FOUND'
|
|
369
380
|
throw "no visible window owned by process $processId"
|
|
370
381
|
}
|
|
371
382
|
$foreground = [DshClickWin32]::GetForegroundWindow()
|
|
372
|
-
if ($foreground -eq [IntPtr]::Zero) {
|
|
383
|
+
if ($foreground -eq [IntPtr]::Zero) {
|
|
384
|
+
$script:ErrorCode = 'WINDOW_NOT_FOUND'
|
|
385
|
+
throw 'no foreground window and no target given'
|
|
386
|
+
}
|
|
373
387
|
return $foreground
|
|
374
388
|
}
|
|
375
389
|
|
|
390
|
+
# One target resolver for the observer ops. `tree`/`shot` always read
|
|
391
|
+
# `args.target`; `snapshot` historically read `args.windowId`. Both spellings
|
|
392
|
+
# now go through here, so `{"op":"snapshot","args":{"target":{"windowId":N}}}`
|
|
393
|
+
# addresses window N instead of silently falling back to the foreground window.
|
|
394
|
+
# `args.target` wins when both are present; the legacy top-level `windowId` is
|
|
395
|
+
# only a fallback, so old callers keep working.
|
|
396
|
+
function Resolve-OpTarget($opArgs) {
|
|
397
|
+
$target = @{}
|
|
398
|
+
if ($opArgs.target -ne $null) {
|
|
399
|
+
$candidate = $opArgs.target
|
|
400
|
+
if ($candidate.windowId -ne $null) { $target.windowId = $candidate.windowId }
|
|
401
|
+
if ($candidate.windowTitle -ne $null) { $target.windowTitle = $candidate.windowTitle }
|
|
402
|
+
if ($candidate.processId -ne $null) { $target.processId = $candidate.processId }
|
|
403
|
+
}
|
|
404
|
+
if ($target.windowId -eq $null -and $target.windowTitle -eq $null -and $target.processId -eq $null) {
|
|
405
|
+
if ($opArgs.windowId -ne $null) { $target.windowId = $opArgs.windowId }
|
|
406
|
+
}
|
|
407
|
+
return $target
|
|
408
|
+
}
|
|
409
|
+
|
|
410
|
+
function Test-TargetPresent($target) {
|
|
411
|
+
return ($target.windowId -ne $null) -or ($target.windowTitle -ne $null) -or ($target.processId -ne $null)
|
|
412
|
+
}
|
|
413
|
+
|
|
376
414
|
# ---------------------------------------------------------------------------
|
|
377
415
|
# Accessibility walk (canonical: the same walk builds treeHash everywhere).
|
|
378
416
|
# ---------------------------------------------------------------------------
|
|
@@ -458,30 +496,214 @@ function Get-TreeHashText($nodes) {
|
|
|
458
496
|
# ---------------------------------------------------------------------------
|
|
459
497
|
# Screenshot.
|
|
460
498
|
# ---------------------------------------------------------------------------
|
|
461
|
-
|
|
499
|
+
# A window is captured in three steps, best first:
|
|
500
|
+
# 1. PrintWindow(hwnd, hdc, 2) PW_RENDERFULLCONTENT - the window paints its
|
|
501
|
+
# own content, so identity and pixels do not depend on occlusion/Z-order.
|
|
502
|
+
# 2. PrintWindow(hwnd, hdc, 0) legacy windows that ignore the newer flag.
|
|
503
|
+
# 3. CopyFromScreen last resort; correct only while unobscured.
|
|
504
|
+
# PrintWindow reports success for frames that carry no content at all (a blank
|
|
505
|
+
# Electron frame, or a Chromium window that paints only its title-bar buttons),
|
|
506
|
+
# so every candidate goes through Test-BitmapLooksBlank before it is accepted.
|
|
507
|
+
# Measured on Windows 11: PW(2) carries content for WinUI3/Electron/Chromium;
|
|
508
|
+
# PW(0/1) is blank or chrome-only for the latter two. The thresholds below are
|
|
509
|
+
# fitted to that matrix: a pure-color check alone is fooled by a gray frame with
|
|
510
|
+
# three title-bar buttons, so the color-count floor is 6 quantized colors.
|
|
511
|
+
$script:CaptureInset = 40 # px trimmed from each edge before sampling
|
|
512
|
+
$script:CaptureGrid = 16 # interior sample grid (16x16)
|
|
513
|
+
$script:CaptureDarkGrid = 32 # full-frame sample grid for the dark-ratio test
|
|
514
|
+
|
|
515
|
+
function Get-BitmapLuminance($color) {
|
|
516
|
+
return (0.299 * [double]$color.R) + (0.587 * [double]$color.G) + (0.114 * [double]$color.B)
|
|
517
|
+
}
|
|
518
|
+
|
|
519
|
+
# True when a captured frame carries no usable content: a flat interior, an
|
|
520
|
+
# almost entirely black frame, or (on the no-flag PrintWindow path only) a few
|
|
521
|
+
# flat colors with chrome painted in a corner.
|
|
522
|
+
#
|
|
523
|
+
# The color-count rule is `strict` because a legitimate all-white page can also
|
|
524
|
+
# sample as 3 quantized colors: measured on a 2582x1550 Chrome new-tab window,
|
|
525
|
+
# PrintWindow(2) gives 3 colors but a luminance sd of 9.6, while the chrome-only
|
|
526
|
+
# frame the no-flag path paints on Chromium gives 4-5 colors with sd 13-14. The
|
|
527
|
+
# two are indistinguishable by color count alone, so the count rule is applied
|
|
528
|
+
# only where chrome-only frames are known to come from (PrintWindow flag 0).
|
|
529
|
+
# A blank all-black frame is caught on every path by the sd/dark-ratio rules.
|
|
530
|
+
function Test-BitmapLooksBlank($bitmap, $strict) {
|
|
531
|
+
if ($null -eq $bitmap -or $bitmap.Width -le 0 -or $bitmap.Height -le 0) { return $true }
|
|
532
|
+
$width = $bitmap.Width
|
|
533
|
+
$height = $bitmap.Height
|
|
534
|
+
|
|
535
|
+
# Interior region: drop the outer band (border, shadow, title-bar buttons)
|
|
536
|
+
# where a failed PrintWindow can still paint a plausible-looking frame.
|
|
537
|
+
$insetX = [Math]::Min($script:CaptureInset, [int][Math]::Floor(($width - 1) / 2))
|
|
538
|
+
$insetY = [Math]::Min($script:CaptureInset, [int][Math]::Floor(($height - 1) / 2))
|
|
539
|
+
$left = $insetX
|
|
540
|
+
$top = $insetY
|
|
541
|
+
$right = $width - 1 - $insetX
|
|
542
|
+
$bottom = $height - 1 - $insetY
|
|
543
|
+
if (($right - $left) -lt 8 -or ($bottom - $top) -lt 8) {
|
|
544
|
+
# Tiny window: the inset leaves nothing to sample, so use the full frame.
|
|
545
|
+
$left = 0; $top = 0; $right = $width - 1; $bottom = $height - 1
|
|
546
|
+
}
|
|
547
|
+
$spanX = [Math]::Max(1, $right - $left)
|
|
548
|
+
$spanY = [Math]::Max(1, $bottom - $top)
|
|
549
|
+
$stepX = [Math]::Max(1, [int][Math]::Ceiling($spanX / $script:CaptureGrid))
|
|
550
|
+
$stepY = [Math]::Max(1, [int][Math]::Ceiling($spanY / $script:CaptureGrid))
|
|
551
|
+
|
|
552
|
+
$colors = @{}
|
|
553
|
+
$sum = 0.0
|
|
554
|
+
$sumSquares = 0.0
|
|
555
|
+
$count = 0
|
|
556
|
+
for ($x = $left; $x -le $right; $x += $stepX) {
|
|
557
|
+
for ($y = $top; $y -le $bottom; $y += $stepY) {
|
|
558
|
+
$color = $bitmap.GetPixel($x, $y)
|
|
559
|
+
# 5-bit quantization: UI colors cluster, and an exact ARGB count is both
|
|
560
|
+
# noisy and slower than needed for a blank/non-blank verdict.
|
|
561
|
+
$key = ([int]($color.R -band 0xF8) -shl 16) -bor ([int]($color.G -band 0xF8) -shl 8) -bor [int]($color.B -band 0xF8)
|
|
562
|
+
$colors[$key] = $true
|
|
563
|
+
$luminance = Get-BitmapLuminance $color
|
|
564
|
+
$sum += $luminance
|
|
565
|
+
$sumSquares += $luminance * $luminance
|
|
566
|
+
$count += 1
|
|
567
|
+
}
|
|
568
|
+
}
|
|
569
|
+
if ($count -eq 0) { return $true }
|
|
570
|
+
$mean = $sum / $count
|
|
571
|
+
$variance = [Math]::Max(0.0, ($sumSquares / $count) - ($mean * $mean))
|
|
572
|
+
$deviation = [Math]::Sqrt($variance)
|
|
573
|
+
if ($deviation -lt 2) { return $true }
|
|
574
|
+
if ($strict -and $colors.Count -le 6) { return $true }
|
|
575
|
+
|
|
576
|
+
# Full-frame dark ratio: a PrintWindow frame that painted nothing at all.
|
|
577
|
+
$dark = 0
|
|
578
|
+
$samples = 0
|
|
579
|
+
$darkStepX = [Math]::Max(1, [int][Math]::Ceiling($width / $script:CaptureDarkGrid))
|
|
580
|
+
$darkStepY = [Math]::Max(1, [int][Math]::Ceiling($height / $script:CaptureDarkGrid))
|
|
581
|
+
for ($x = 0; $x -lt $width; $x += $darkStepX) {
|
|
582
|
+
for ($y = 0; $y -lt $height; $y += $darkStepY) {
|
|
583
|
+
if ((Get-BitmapLuminance $bitmap.GetPixel($x, $y)) -lt 8) { $dark += 1 }
|
|
584
|
+
$samples += 1
|
|
585
|
+
}
|
|
586
|
+
}
|
|
587
|
+
if ($samples -gt 0 -and (($dark * 100) -ge (99 * $samples))) { return $true }
|
|
588
|
+
return $false
|
|
589
|
+
}
|
|
590
|
+
|
|
591
|
+
function New-CaptureBitmap($width, $height) {
|
|
592
|
+
if ($width -le 0 -or $height -le 0) { throw "cannot allocate a ${width}x${height} capture bitmap" }
|
|
593
|
+
return New-Object System.Drawing.Bitmap($width, $height)
|
|
594
|
+
}
|
|
595
|
+
|
|
596
|
+
# One PrintWindow attempt; returns $null when the API refuses, otherwise the
|
|
597
|
+
# bitmap (which may still be blank - the caller decides).
|
|
598
|
+
function Get-PrintWindowFrame($hwnd, $rectInfo, $flags) {
|
|
599
|
+
$bitmap = New-CaptureBitmap $rectInfo.width $rectInfo.height
|
|
600
|
+
$graphics = [System.Drawing.Graphics]::FromImage($bitmap)
|
|
601
|
+
$hdc = [IntPtr]::Zero
|
|
602
|
+
$painted = $false
|
|
603
|
+
try {
|
|
604
|
+
$hdc = $graphics.GetHdc()
|
|
605
|
+
$painted = [DshClickWin32]::PrintWindow($hwnd, $hdc, [uint32]$flags)
|
|
606
|
+
} catch {
|
|
607
|
+
$painted = $false
|
|
608
|
+
} finally {
|
|
609
|
+
if ($hdc -ne [IntPtr]::Zero) {
|
|
610
|
+
try { $graphics.ReleaseHdc($hdc) } catch { }
|
|
611
|
+
}
|
|
612
|
+
$graphics.Dispose()
|
|
613
|
+
}
|
|
614
|
+
if (-not $painted) {
|
|
615
|
+
$bitmap.Dispose()
|
|
616
|
+
return $null
|
|
617
|
+
}
|
|
618
|
+
return $bitmap
|
|
619
|
+
}
|
|
620
|
+
|
|
621
|
+
# Capture one window: returns @{ bitmap; method; rect }. The caller owns the
|
|
622
|
+
# bitmap and must dispose it. Minimized windows are refused up front: IsIconic
|
|
623
|
+
# windows report a placeholder rect, and PrintWindow paints only the small
|
|
624
|
+
# window-icon band even when it answers true, so there is no honest image.
|
|
625
|
+
function Get-WindowCapture($hwnd) {
|
|
462
626
|
$rectInfo = Get-WindowRectInfo $hwnd
|
|
463
|
-
if ($rectInfo.width -le 0 -or $rectInfo.height -le 0) {
|
|
464
|
-
|
|
627
|
+
if ($rectInfo.width -le 0 -or $rectInfo.height -le 0) {
|
|
628
|
+
$script:ErrorCode = 'WINDOW_NOT_FOUND'
|
|
629
|
+
throw "window $hwnd has no drawable rect"
|
|
630
|
+
}
|
|
631
|
+
if ([DshClickWin32]::IsIconic($hwnd)) {
|
|
632
|
+
$script:ErrorCode = 'WINDOW_MINIMIZED'
|
|
633
|
+
throw "window $hwnd is minimized; a minimized window has no capturable pixels - restore it, then re-observe"
|
|
634
|
+
}
|
|
635
|
+
|
|
636
|
+
$candidate = Get-PrintWindowFrame $hwnd $rectInfo 2 # PW_RENDERFULLCONTENT
|
|
637
|
+
if ($candidate -ne $null) {
|
|
638
|
+
if (-not (Test-BitmapLooksBlank $candidate $false)) {
|
|
639
|
+
return @{ bitmap = $candidate; method = 'printwindow-fullcontent'; rect = $rectInfo }
|
|
640
|
+
}
|
|
641
|
+
$candidate.Dispose()
|
|
642
|
+
}
|
|
643
|
+
$candidate = Get-PrintWindowFrame $hwnd $rectInfo 0 # legacy PrintWindow
|
|
644
|
+
if ($candidate -ne $null) {
|
|
645
|
+
if (-not (Test-BitmapLooksBlank $candidate $true)) {
|
|
646
|
+
return @{ bitmap = $candidate; method = 'printwindow'; rect = $rectInfo }
|
|
647
|
+
}
|
|
648
|
+
$candidate.Dispose()
|
|
649
|
+
}
|
|
650
|
+
|
|
651
|
+
# Screen-region copy: the old path, kept for windows that refuse PrintWindow.
|
|
652
|
+
$bitmap = New-CaptureBitmap $rectInfo.width $rectInfo.height
|
|
465
653
|
$graphics = [System.Drawing.Graphics]::FromImage($bitmap)
|
|
466
654
|
try {
|
|
467
655
|
$graphics.CopyFromScreen($rectInfo.x, $rectInfo.y, 0, 0, (New-Object System.Drawing.Size($rectInfo.width, $rectInfo.height)))
|
|
656
|
+
} catch {
|
|
657
|
+
$bitmap.Dispose()
|
|
658
|
+
throw
|
|
468
659
|
} finally {
|
|
469
660
|
$graphics.Dispose()
|
|
470
661
|
}
|
|
471
|
-
return $bitmap
|
|
662
|
+
return @{ bitmap = $bitmap; method = 'copyfromscreen'; rect = $rectInfo }
|
|
472
663
|
}
|
|
473
664
|
|
|
474
|
-
|
|
475
|
-
|
|
476
|
-
|
|
477
|
-
|
|
665
|
+
# Resolve the requested screen rectangle. `virtual` is the whole desktop
|
|
666
|
+
# (negative origins on multi-monitor setups) and stays experimental: only the
|
|
667
|
+
# single-monitor case is covered by the verification runs, and BitBlt across
|
|
668
|
+
# negative virtual-screen coordinates is untested here. If the virtual metrics
|
|
669
|
+
# are unusable, fall back to the primary screen rather than failing the call.
|
|
670
|
+
function Get-ScreenMetrics($scope) {
|
|
671
|
+
if ($scope -eq 'virtual') {
|
|
672
|
+
$virtualX = [int][DshClickWin32]::GetSystemMetrics(76) # SM_XVIRTUALSCREEN
|
|
673
|
+
$virtualY = [int][DshClickWin32]::GetSystemMetrics(77) # SM_YVIRTUALSCREEN
|
|
674
|
+
$virtualWidth = [int][DshClickWin32]::GetSystemMetrics(78) # SM_CXVIRTUALSCREEN
|
|
675
|
+
$virtualHeight = [int][DshClickWin32]::GetSystemMetrics(79) # SM_CYVIRTUALSCREEN
|
|
676
|
+
if ($virtualWidth -gt 0 -and $virtualHeight -gt 0) {
|
|
677
|
+
return @{ scope = 'virtual'; x = $virtualX; y = $virtualY; width = $virtualWidth; height = $virtualHeight }
|
|
678
|
+
}
|
|
679
|
+
}
|
|
680
|
+
return @{
|
|
681
|
+
scope = 'primary'; x = 0; y = 0
|
|
682
|
+
width = [int][DshClickWin32]::GetSystemMetrics(0) # SM_CXSCREEN
|
|
683
|
+
height = [int][DshClickWin32]::GetSystemMetrics(1) # SM_CYSCREEN
|
|
684
|
+
}
|
|
685
|
+
}
|
|
686
|
+
|
|
687
|
+
# Capture a screen region: returns @{ bitmap; method; scope; rect }.
|
|
688
|
+
function Get-ScreenCapture($scope) {
|
|
689
|
+
$metrics = Get-ScreenMetrics $scope
|
|
690
|
+
if ($metrics.width -le 0 -or $metrics.height -le 0) { throw 'the screen reports no drawable area' }
|
|
691
|
+
$bitmap = New-CaptureBitmap $metrics.width $metrics.height
|
|
478
692
|
$graphics = [System.Drawing.Graphics]::FromImage($bitmap)
|
|
479
693
|
try {
|
|
480
|
-
$graphics.CopyFromScreen(
|
|
694
|
+
$graphics.CopyFromScreen($metrics.x, $metrics.y, 0, 0, (New-Object System.Drawing.Size($metrics.width, $metrics.height)))
|
|
695
|
+
} catch {
|
|
696
|
+
$bitmap.Dispose()
|
|
697
|
+
throw
|
|
481
698
|
} finally {
|
|
482
699
|
$graphics.Dispose()
|
|
483
700
|
}
|
|
484
|
-
return
|
|
701
|
+
return @{
|
|
702
|
+
bitmap = $bitmap
|
|
703
|
+
method = 'copyfromscreen'
|
|
704
|
+
scope = $metrics.scope
|
|
705
|
+
rect = @{ x = $metrics.x; y = $metrics.y; width = $metrics.width; height = $metrics.height }
|
|
706
|
+
}
|
|
485
707
|
}
|
|
486
708
|
|
|
487
709
|
function Resize-BitmapIfNeeded($bitmap, $maxSide) {
|
|
@@ -651,12 +873,17 @@ function Invoke-OpWindows {
|
|
|
651
873
|
foreach ($hwnd in [DshClickWin32]::EnumVisibleWindows()) {
|
|
652
874
|
$pidValue = 0
|
|
653
875
|
[void][DshClickWin32]::GetWindowThreadProcessId($hwnd, [ref]$pidValue)
|
|
876
|
+
# A minimized window still enumerates, but GetWindowRect then returns the
|
|
877
|
+
# CW_USEDEFAULT placeholder (-32000,-32000 314x50). Report the state and
|
|
878
|
+
# suppress the placeholder rect instead of handing it to coordinate math.
|
|
879
|
+
$minimized = [DshClickWin32]::IsIconic($hwnd)
|
|
654
880
|
$windows += @{
|
|
655
881
|
windowId = [int64]$hwnd
|
|
656
882
|
processId = if ([int]$pidValue -gt 0) { [int]$pidValue } else { $null }
|
|
657
883
|
title = [DshClickWin32]::WindowTitle($hwnd)
|
|
658
884
|
className = [DshClickWin32]::WindowClass($hwnd)
|
|
659
|
-
rect = (Get-WindowRectInfo $hwnd
|
|
885
|
+
rect = if ($minimized) { $null } else { Get-WindowRectInfo $hwnd }
|
|
886
|
+
minimized = $minimized
|
|
660
887
|
executablePath = $null
|
|
661
888
|
visible = $true
|
|
662
889
|
}
|
|
@@ -682,12 +909,14 @@ function Invoke-OpApps {
|
|
|
682
909
|
$pidInt = [int]$pidValue
|
|
683
910
|
if ($pidInt -le 0) { continue }
|
|
684
911
|
if (-not $byPid.ContainsKey($pidInt)) { $byPid[$pidInt] = @() }
|
|
912
|
+
$minimized = [DshClickWin32]::IsIconic($hwnd)
|
|
685
913
|
$byPid[$pidInt] += @{
|
|
686
914
|
windowId = [int64]$hwnd
|
|
687
915
|
processId = $pidInt
|
|
688
916
|
title = [DshClickWin32]::WindowTitle($hwnd)
|
|
689
917
|
className = [DshClickWin32]::WindowClass($hwnd)
|
|
690
|
-
rect = (Get-WindowRectInfo $hwnd
|
|
918
|
+
rect = if ($minimized) { $null } else { Get-WindowRectInfo $hwnd }
|
|
919
|
+
minimized = $minimized
|
|
691
920
|
executablePath = $null
|
|
692
921
|
visible = $true
|
|
693
922
|
}
|
|
@@ -712,91 +941,129 @@ function Invoke-OpApps {
|
|
|
712
941
|
return $apps
|
|
713
942
|
}
|
|
714
943
|
|
|
715
|
-
function Get-SnapshotRecord($hwnd, $maxElements, $maxDepth) {
|
|
716
|
-
$
|
|
717
|
-
$
|
|
718
|
-
|
|
719
|
-
|
|
720
|
-
|
|
721
|
-
Walk-UiaTree $element 0 $maxDepth $maxElements $false
|
|
722
|
-
$treeHash = Get-StringHash (Get-TreeHashText $script:TreeNodes)
|
|
723
|
-
$bitmap = Get-WindowBitmap $hwnd
|
|
724
|
-
$shotHash = $null
|
|
944
|
+
function Get-SnapshotRecord($hwnd, $maxElements, $maxDepth, $capture) {
|
|
945
|
+
$ownsCapture = $false
|
|
946
|
+
if ($null -eq $capture) {
|
|
947
|
+
$capture = Get-WindowCapture $hwnd
|
|
948
|
+
$ownsCapture = $true
|
|
949
|
+
}
|
|
725
950
|
try {
|
|
726
|
-
$
|
|
951
|
+
$facts = Get-ProcessFacts $hwnd
|
|
952
|
+
$rect = Get-WindowRectInfo $hwnd
|
|
953
|
+
$foreground = ([DshClickWin32]::GetForegroundWindow() -eq $hwnd)
|
|
954
|
+
$script:TreeNodes = @()
|
|
955
|
+
$element = Get-UiaElement $hwnd
|
|
956
|
+
Walk-UiaTree $element 0 $maxDepth $maxElements $false
|
|
957
|
+
$treeHash = Get-StringHash (Get-TreeHashText $script:TreeNodes)
|
|
958
|
+
# The hash is taken from the same frame the caller will ship as the
|
|
959
|
+
# screenshot (or compare later), not from a second capture: two captures
|
|
960
|
+
# can disagree on a busy window and on which capture path succeeded.
|
|
961
|
+
$shotHash = Get-BitmapHash $capture.bitmap
|
|
962
|
+
return @{
|
|
963
|
+
windowId = [int64]$hwnd
|
|
964
|
+
processId = $facts.pid
|
|
965
|
+
executablePath = $facts.executablePath
|
|
966
|
+
title = [DshClickWin32]::WindowTitle($hwnd)
|
|
967
|
+
className = [DshClickWin32]::WindowClass($hwnd)
|
|
968
|
+
rect = $rect
|
|
969
|
+
foreground = $foreground
|
|
970
|
+
treeHash = $treeHash
|
|
971
|
+
shotHash = $shotHash
|
|
972
|
+
elementCount = $script:TreeNodes.Count
|
|
973
|
+
}
|
|
727
974
|
} finally {
|
|
728
|
-
$bitmap.Dispose()
|
|
729
|
-
}
|
|
730
|
-
return @{
|
|
731
|
-
windowId = [int64]$hwnd
|
|
732
|
-
processId = $facts.pid
|
|
733
|
-
executablePath = $facts.executablePath
|
|
734
|
-
title = [DshClickWin32]::WindowTitle($hwnd)
|
|
735
|
-
className = [DshClickWin32]::WindowClass($hwnd)
|
|
736
|
-
rect = $rect
|
|
737
|
-
foreground = $foreground
|
|
738
|
-
treeHash = $treeHash
|
|
739
|
-
shotHash = $shotHash
|
|
740
|
-
elementCount = $script:TreeNodes.Count
|
|
975
|
+
if ($ownsCapture) { $capture.bitmap.Dispose() }
|
|
741
976
|
}
|
|
742
977
|
}
|
|
743
978
|
|
|
744
979
|
function Invoke-OpSnapshot($opArgs) {
|
|
745
|
-
$
|
|
980
|
+
$target = Resolve-OpTarget $opArgs
|
|
981
|
+
if (-not (Test-TargetPresent $target)) {
|
|
982
|
+
# This used to fall back to the foreground window without saying so, which
|
|
983
|
+
# reported a different window than the caller asked for. Refuse instead.
|
|
984
|
+
$script:ErrorCode = 'NO_TARGET'
|
|
985
|
+
throw 'snapshot requires windowId or target; it never guesses the foreground window'
|
|
986
|
+
}
|
|
987
|
+
$hwnd = Resolve-Window $target
|
|
746
988
|
$maxElements = if ($opArgs.maxElements -ne $null) { [int]$opArgs.maxElements } else { 500 }
|
|
747
989
|
$maxDepth = if ($opArgs.maxDepth -ne $null) { [int]$opArgs.maxDepth } else { 32 }
|
|
748
|
-
|
|
990
|
+
$capture = Get-WindowCapture $hwnd
|
|
991
|
+
try {
|
|
992
|
+
return (Get-SnapshotRecord $hwnd $maxElements $maxDepth $capture)
|
|
993
|
+
} finally {
|
|
994
|
+
$capture.bitmap.Dispose()
|
|
995
|
+
}
|
|
749
996
|
}
|
|
750
997
|
|
|
751
998
|
function Invoke-OpShot($opArgs) {
|
|
752
|
-
$target =
|
|
999
|
+
$target = Resolve-OpTarget $opArgs
|
|
1000
|
+
$scope = if ($opArgs.scope -ne $null) { [string]$opArgs.scope } else { 'primary' }
|
|
1001
|
+
if ($scope -ne 'primary' -and $scope -ne 'virtual') {
|
|
1002
|
+
$script:ErrorCode = 'INVALID_SCOPE'
|
|
1003
|
+
throw "unknown scope '$scope' (expected 'primary' or 'virtual')"
|
|
1004
|
+
}
|
|
753
1005
|
$maxSide = if ($opArgs.maxSide -ne $null) { [int]$opArgs.maxSide } else { 2560 }
|
|
754
1006
|
$maxElements = if ($opArgs.maxElements -ne $null) { [int]$opArgs.maxElements } else { 500 }
|
|
755
1007
|
$maxDepth = if ($opArgs.maxDepth -ne $null) { [int]$opArgs.maxDepth } else { 32 }
|
|
756
|
-
$
|
|
757
|
-
|
|
758
|
-
$hwnd = Resolve-Window $target
|
|
759
|
-
$bitmap = Get-WindowBitmap $hwnd
|
|
760
|
-
$snapshot = Get-SnapshotRecord $hwnd $maxElements $maxDepth
|
|
761
|
-
} else {
|
|
762
|
-
$hwnd = $null
|
|
763
|
-
$bitmap = Get-PrimaryBitmap
|
|
764
|
-
$snapshot = @{
|
|
765
|
-
windowId = 0; processId = 0; executablePath = $null
|
|
766
|
-
title = 'primary screen'; className = 'Screen'
|
|
767
|
-
rect = @{ x = 0; y = 0; width = $bitmap.Width; height = $bitmap.Height }
|
|
768
|
-
foreground = $true; treeHash = ''; shotHash = ''; elementCount = 0
|
|
769
|
-
}
|
|
770
|
-
}
|
|
1008
|
+
$capture = $null
|
|
1009
|
+
$bitmap = $null
|
|
771
1010
|
try {
|
|
772
|
-
|
|
1011
|
+
if (Test-TargetPresent $target) {
|
|
1012
|
+
$hwnd = Resolve-Window $target
|
|
1013
|
+
$capture = Get-WindowCapture $hwnd
|
|
1014
|
+
$captureScope = 'window'
|
|
1015
|
+
$snapshot = Get-SnapshotRecord $hwnd $maxElements $maxDepth $capture
|
|
1016
|
+
} else {
|
|
1017
|
+
# No target means the screen, not the foreground window: the result says
|
|
1018
|
+
# `scope: primary` and its snapshot is titled "primary screen" (windowId 0).
|
|
1019
|
+
$capture = Get-ScreenCapture $scope
|
|
1020
|
+
$captureScope = $capture.scope
|
|
1021
|
+
$snapshot = @{
|
|
1022
|
+
windowId = 0; processId = 0; executablePath = $null
|
|
1023
|
+
title = if ($capture.scope -eq 'virtual') { 'virtual screen' } else { 'primary screen' }
|
|
1024
|
+
className = 'Screen'
|
|
1025
|
+
rect = $capture.rect
|
|
1026
|
+
foreground = $true; treeHash = ''; shotHash = ''; elementCount = 0
|
|
1027
|
+
}
|
|
1028
|
+
}
|
|
1029
|
+
$bitmap = $capture.bitmap
|
|
1030
|
+
if ([Math]::Max($bitmap.Width, $bitmap.Height) -gt $maxSide) {
|
|
1031
|
+
# Resize-BitmapIfNeeded disposes the original; the finally below disposes
|
|
1032
|
+
# whichever bitmap is current, and Dispose is idempotent.
|
|
1033
|
+
$bitmap = Resize-BitmapIfNeeded $capture.bitmap $maxSide
|
|
1034
|
+
}
|
|
773
1035
|
$base64 = Get-BitmapBase64 $bitmap
|
|
774
1036
|
return @{
|
|
775
1037
|
pngBase64 = $base64
|
|
776
1038
|
width = $bitmap.Width
|
|
777
1039
|
height = $bitmap.Height
|
|
1040
|
+
captureMethod = $capture.method
|
|
1041
|
+
scope = $captureScope
|
|
778
1042
|
snapshot = $snapshot
|
|
779
1043
|
}
|
|
780
1044
|
} finally {
|
|
781
|
-
$bitmap.Dispose()
|
|
1045
|
+
if ($null -ne $bitmap) { $bitmap.Dispose() }
|
|
1046
|
+
elseif ($null -ne $capture) { $capture.bitmap.Dispose() }
|
|
782
1047
|
}
|
|
783
1048
|
}
|
|
784
1049
|
|
|
785
1050
|
function Invoke-OpTree($opArgs) {
|
|
786
|
-
$target =
|
|
1051
|
+
$target = Resolve-OpTarget $opArgs
|
|
787
1052
|
$maxElements = if ($opArgs.maxElements -ne $null) { [int]$opArgs.maxElements } else { 500 }
|
|
788
1053
|
$maxDepth = if ($opArgs.maxDepth -ne $null) { [int]$opArgs.maxDepth } else { 32 }
|
|
789
1054
|
$includePixels = if ($opArgs.includePixels -ne $null) { [bool]$opArgs.includePixels } else { $true }
|
|
1055
|
+
# An empty target still means the foreground window here (documented for
|
|
1056
|
+
# screen_read); only `snapshot` refuses the guess.
|
|
790
1057
|
$hwnd = Resolve-Window $target
|
|
791
|
-
$
|
|
792
|
-
|
|
793
|
-
|
|
794
|
-
|
|
795
|
-
|
|
796
|
-
|
|
797
|
-
$
|
|
798
|
-
|
|
799
|
-
$rectInfo =
|
|
1058
|
+
$capture = Get-WindowCapture $hwnd
|
|
1059
|
+
try {
|
|
1060
|
+
$snapshot = Get-SnapshotRecord $hwnd $maxElements $maxDepth $capture
|
|
1061
|
+
$script:TreeNodes = @()
|
|
1062
|
+
$element = Get-UiaElement $hwnd
|
|
1063
|
+
Walk-UiaTree $element 0 $maxDepth $maxElements $includePixels
|
|
1064
|
+
$pixels = @()
|
|
1065
|
+
if ($includePixels) {
|
|
1066
|
+
$rectInfo = $capture.rect
|
|
800
1067
|
$count = 0
|
|
801
1068
|
foreach ($node in $script:TreeNodes) {
|
|
802
1069
|
if ($count -ge 200) { break }
|
|
@@ -804,21 +1071,21 @@ function Invoke-OpTree($opArgs) {
|
|
|
804
1071
|
if ($r.width -le 0 -or $r.height -le 0) { continue }
|
|
805
1072
|
$cx = $r.x + [int]($r.width / 2) - $rectInfo.x
|
|
806
1073
|
$cy = $r.y + [int]($r.height / 2) - $rectInfo.y
|
|
807
|
-
$color = Get-PixelColorText $bitmap $cx $cy
|
|
1074
|
+
$color = Get-PixelColorText $capture.bitmap $cx $cy
|
|
808
1075
|
if ($color -ne $null) {
|
|
809
1076
|
$label = if ($node.name -ne '') { $node.name } else { "$($node.controlType)[$($node.elementId)]" }
|
|
810
1077
|
$pixels += @{ label = $label; x = $r.x + [int]($r.width / 2); y = $r.y + [int]($r.height / 2); color = $color }
|
|
811
1078
|
$count += 1
|
|
812
1079
|
}
|
|
813
1080
|
}
|
|
814
|
-
} finally {
|
|
815
|
-
$bitmap.Dispose()
|
|
816
1081
|
}
|
|
817
|
-
|
|
818
|
-
|
|
819
|
-
|
|
820
|
-
|
|
821
|
-
|
|
1082
|
+
return @{
|
|
1083
|
+
snapshot = $snapshot
|
|
1084
|
+
elements = @($script:TreeNodes)
|
|
1085
|
+
pixels = $pixels
|
|
1086
|
+
}
|
|
1087
|
+
} finally {
|
|
1088
|
+
$capture.bitmap.Dispose()
|
|
822
1089
|
}
|
|
823
1090
|
}
|
|
824
1091
|
|
|
@@ -1045,6 +1312,9 @@ try {
|
|
|
1045
1312
|
# Per-request focus-fallback verdict (set by Set-DshFocusFallback, consumed by
|
|
1046
1313
|
# Get-ActionOutcome); one helper process serves exactly one request.
|
|
1047
1314
|
$script:focusFallbackDetail = $null
|
|
1315
|
+
# Per-request failure code; a plain `throw` still defaults to HELPER_ERROR,
|
|
1316
|
+
# while the capture/scope guards set a precise code for the caller.
|
|
1317
|
+
$script:ErrorCode = $null
|
|
1048
1318
|
switch ([string]$request.op) {
|
|
1049
1319
|
'windows' { $result = Invoke-OpWindows }
|
|
1050
1320
|
'metrics' { $result = Invoke-OpMetrics }
|
|
@@ -1064,6 +1334,7 @@ try {
|
|
|
1064
1334
|
} catch {
|
|
1065
1335
|
$message = $_.Exception.Message
|
|
1066
1336
|
if ($message -eq $null -or $message.Length -eq 0) { $message = 'unknown helper failure' }
|
|
1067
|
-
|
|
1337
|
+
$code = if ($script:ErrorCode -ne $null -and [string]$script:ErrorCode -ne '') { [string]$script:ErrorCode } else { 'HELPER_ERROR' }
|
|
1338
|
+
Write-DshError $code $message
|
|
1068
1339
|
exit 0
|
|
1069
1340
|
}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "dsh-click",
|
|
3
|
-
"version": "0.3.
|
|
3
|
+
"version": "0.3.29",
|
|
4
4
|
"description": "Cross-platform native desktop control tools for DeepSeek Harness (Windows first): screen_shot, screen_read (accessibility tree + pixel description for text-only models), click/type/scroll/key, and app_list/app_launch — every mutating action gated by approval, never stealing foreground focus, with stale-observation rejection and process-identity checks",
|
|
5
5
|
"repository": {
|
|
6
6
|
"type": "git",
|