@momasiku/dsh-pilot 0.5.10 → 0.5.11

package/README.md

@@ -116,6 +116,60 @@ is byte-identical to the previous one of the same session, the image is not atta
 (or stored) again — the result says `unchanged: true` and the coordinates from the
 frame the model already has still apply. `forceImage: true` overrides it.
 
+## Reading the control tree
+
+A screenshot costs a guess: the model finds a button in pixels, estimates its centre,
+clicks, and takes another frame to see whether the guess was right. Windows exposes the
+same control through UI Automation as a role, a name and a rectangle, so the click
+becomes the centre of that rectangle and nothing has to be looked at.
+
+```js
+desktop_control({ action: 'controls', title: 'Notepad' })
+// 4  [Button] '最小化' @ 887,1 45x39
+// 14 [Button] '新建会话' @ 12,54 256x24
+// 23 [Edit]   '搜索会话名称' @ 204,238 4x20  offscreen
+
+desktop_control({ action: 'click', target: 14, title: 'Notepad' })
+```
+
+**The tree is not a faster screenshot; it is a different trade.**
+
+| | screenshot | control tree |
+| --- | --- | --- |
+| one read | ~150 ms, ~1k tokens | ~130-300 ms, ~850 tokens (filtered) |
+| click target | estimated from pixels | the control's exact rectangle |
+| state | "looks disabled" | `enabled`, `offscreen`, `HasKeyboardFocus`, `IsSelected`, value |
+| availability | always | **not always** - see below |
+
+Per call the tree is slower or level. Per task it is much faster, because one read
+resolves several targets without another frame and without another round trip.
+
+**It does not always exist.** A canvas, a drawing surface, a custom-painted list and an
+Electron window that publishes nothing all give an empty or useless tree. That is why
+the screenshot path is still there and still the fallback: read the tree first, take a
+frame when the tree is empty or when you need to see what is drawn rather than what is
+declared.
+
+**Three things worth knowing.**
+
+- An index addresses one row of one listing. It is resolved against a fresh read at
+  click time, and the worker remembers the listing it handed out, so a row that changed
+  identity fails loudly instead of being clicked by position:
+  `the control at index 4 is not the one that was listed: expected [Button] 'OK', found [Button] 'Cancel'`.
+- `meaningfulOnly` (default `true`) returns only the roles a person can operate. On a
+  real 673-element window that is 77 elements and ~850 tokens instead of 9k - the rest
+  is layout panes with no name, no state and nothing to click. Pass `false` for the
+  whole tree.
+- `includeValues` is off by default: structure is not content. Password fields are
+  masked even when it is on.
+
+**When the thing you want has no tree of its own** - a chat transcript, a canvas, a
+custom-painted panel - do not look for it. Look for what surrounds it and subtract: the
+composer is an `Edit` with a rectangle, the window has a rectangle, so the transcript is
+what is left between them. Or skip the geometry: select the item in the tree, confirm
+with `HasKeyboardFocus` that the keyboard is where you want it, type, press Enter. That
+last step is something a screenshot cannot do - it can show a caret, not a promise.
+
 ## Why it is safe
 
 Giving a model hands is the part worth getting right:
@@ -178,6 +232,26 @@ and what `dsh plugin add` checks before it installs anything. Requirements:
 
 ### Recent changes
 
+**0.5.11** — the control tree, and two fixes that make a batch honest.
+
+- **`controls` reads the window's UI Automation tree**, and `target` clicks a listed
+  control's exact rectangle, so a named control no longer has to be found in pixels.
+  It takes no screenshot, so it pays no settle. Measured on a 673-element window:
+  **0.65 s** and **~850 tokens** filtered (77 operable elements) against ~9k
+  unfiltered. The worker remembers the listing it handed out, so an index whose row
+  changed identity fails instead of clicking whatever took its place.
+- **The settle was paid even when nothing was captured** (0.5.9). It exists so a frame
+  is not taken mid-repaint; `desktop_control` applied it on every call, capturing or
+  not. Measured **814 ms -> 12 ms** for a non-capturing action.
+- **A refused `focus` was reported as success** (0.5.9). The step recorded a note, said
+  `ok`, and the rest of the batch went to whichever window was actually in front. It
+  now checks the foreground window after the attempt — the state, not
+  `SetForegroundWindow`'s return value — and fails, naming both. `tolerant: true`
+  restores the old behaviour for a caller whose coordinates are right either way.
+- **The peer range now accepts the 0.2.1 host generation** (0.5.10). The 0.2.1 tuple
+  needed its own prerelease comparator: a prerelease only satisfies a range when a
+  comparator carries a prerelease tag on the same `major.minor.patch`.
+
 **0.5.8** — two fixes, and the package is now on npm as
 [`@momasiku/dsh-pilot`](https://www.npmjs.com/package/@momasiku/dsh-pilot).
 
@@ -224,7 +298,7 @@ coordinate contract for that frame.
 
 | parameter | type | meaning |
 | --- | --- | --- |
-| `action` | string, required | `click`, `doubleClick`, `rightClick`, `middleClick`, `move`, `drag`, `scroll`, `type`, `key`, `focus`, `windows` |
+| `action` | string, required | `click`, `doubleClick`, `rightClick`, `middleClick`, `move`, `drag`, `scroll`, `type`, `key`, `focus`, `windows`, `controls` |
 | `x`, `y` | integer | target in physical screen pixels; required by the pointer actions and by scroll-at-a-point |
 | `toX`, `toY` | integer | drag end point, or a nonzero value to make `scroll` horizontal |
 | `text` | string | text to insert for `type` |
@@ -236,6 +310,10 @@ coordinate contract for that frame.
 | `capture` | boolean | capture a fresh frame after acting (default `true`) |
 | `settleMs` | integer | wait before that capture so the screen can repaint (default 750) |
 | `forceImage` | boolean | attach the frame even when it repeats the previous frame of the session |
+| `target` | integer | pointer actions: click the control at this index from a `controls` listing instead of `x`/`y` |
+| `includeValues` | boolean | `controls`: also return each control's value (default `false`; passwords masked) |
+| `meaningfulOnly` | boolean | `controls`: only operable roles (default `true`) |
+| `tolerant` | boolean | `focus`: record a note instead of failing when Windows refuses the foreground |
 
 `type` delivers text as **one paste** (clipboard set, `Ctrl+V`, clipboard restored), so
 applications see a single insertion — multi-line text and CJK included — instead of a
@@ -243,6 +321,12 @@ burst of per-character keystrokes. `windows` returns every visible titled top-le
 window with its bounds, minimized state, and which is foreground: useful before
 clicking anything.
 
+`controls` returns the window's control tree as one line per control — `index [Role]
+'Name' @ left,top widthxheight` plus `disabled`/`offscreen`/`value` — and takes no
+screenshot (pass `capture: true` if you want one anyway). Feed an index back as
+`target` to click that control's exact centre. See **Reading the control tree** above
+for when a tree exists and when a frame is still the right tool.
+
 ### `desktop_sequence`
 
 | parameter | type | meaning |
@@ -447,6 +531,52 @@ DSH：desktop_sequence([focus "报表.xlsx", click 812,430, type "Q3 收入", ke
 
 **没变化的帧不重复发送**：每次截图都带 `frameHash`，与同会话上一帧逐字节相同时就不重复附图（也不重复存储），结果里写 `unchanged: true`，模型手上那张图的坐标继续有效；想看就传 `forceImage: true`。
 
+## 读控件树
+
+截图是要靠猜的：模型在像素里找到按钮、估个中心、点一下，再截一张图看猜没猜对。
+而 Windows 通过 UI Automation 直接把同一个控件交出来——角色、名字、矩形——于是点击
+就变成「那个矩形的中心」，什么也不用看。
+
+```js
+desktop_control({ action: 'controls', title: '记事本' })
+// 4  [Button] '最小化' @ 887,1 45x39
+// 14 [Button] '新建会话' @ 12,54 256x24
+// 23 [Edit]   '搜索会话名称' @ 204,238 4x20  offscreen
+
+desktop_control({ action: 'click', target: 14, title: '记事本' })
+```
+
+**控件树不是更快的截图，是另一种取舍。**
+
+| | 截图 | 控件树 |
+| --- | --- | --- |
+| 读一次 | ~150 ms，~1k token | ~130-300 ms，~850 token（过滤后） |
+| 点击位置 | 从像素里估 | 控件的精确矩形 |
+| 状态 | 「看着像禁用」 | `enabled`、`offscreen`、`HasKeyboardFocus`、`IsSelected`、值 |
+| 可用性 | 永远有 | **不一定有**，见下 |
+
+按单次调用看，树更慢或打平；按单个任务看，树快得多——一次读取能定位好几个目标，
+既不用多截一张图，也不用多跑一轮推理。
+
+**它不是总存在。** canvas、绘图面、自绘列表、以及什么都不上报的 Electron 窗口，
+给出的树要么是空的要么没意义。所以截图这条路还在，而且仍然是兜底：先读树，
+树为空、或者你需要看「画出来什么」而不是「声明了什么」时，再截一张图。
+
+**三个要知道的点。**
+
+- 索引只针对「某一次列表」。点击时会重新读一遍再解析，而且暖 worker 记着它发出去的
+  那份列表，所以某一行换了身份会明确报错，而不是按位置点下去：
+  `the control at index 4 is not the one that was listed: expected [Button] 'OK', found [Button] 'Cancel'`。
+- `meaningfulOnly`（默认 `true`）只返回人能操作的控件。在一个真实的 673 元素窗口上，
+  这是 77 个元素、约 850 token，而不是 9k——其余全是没名字、没状态、点了也没反应的
+  布局面板。要整棵树就传 `false`。
+- `includeValues` 默认关闭：结构不是内容。即使打开，密码框也会被打码。
+
+**当你要找的东西本身没有树**——聊天记录、画布、自绘面板——不要去找它。去找围着它的
+东西再做减法：输入框是个带矩形的 `Edit`，窗口有矩形，那么聊天区就是它们之间剩下的那块。
+或者干脆不用几何：在树里选中那一条，用 `HasKeyboardFocus` 确认键盘落点就是你要的地方，
+打字，回车。最后这一步是截图做不到的——截图能给你一根光标，给不了保证。
+
 ## 为什么安全
 
 给模型装上手，最该讲究的就是这一块：
@@ -486,6 +616,13 @@ dsh plugin --profile <你的profile> add "file:E:\path\to\dsh-pilot"
 
 ### 最近改动
 
+**0.5.11** —— 控件树，外加两个让批量操作「不说谎」的修复。
+
+- **`controls` 读窗口的 UI Automation 树**，`target` 点列表里那个控件的精确矩形，于是有名字的控件不必再去像素里找。它不截图，所以也不付 settle。在 673 个元素的窗口上实测：过滤后 **0.65 秒、约 850 token**（77 个可操作元素），不过滤约 9k。暖 worker 记着它发出去的那份列表，所以某一行换了身份会报错，而不是点到搬进来的那个。
+- **不截图的动作也在付 settle**（0.5.9）。settle 本是为了「别在重绘中途取帧」，但 `desktop_control` 每次调用都付。实测不截图的动作用时 **814 ms → 12 ms**。
+- **`focus` 被拒绝却报成功**（0.5.9）。那一步只记一条备注、仍然说 `ok`，于是批量剩下的步骤全打进了真正在前台的那个窗口。现在改成在尝试之后检查真实前台——看状态，不看 `SetForegroundWindow` 的返回值——不符就报错并说出两个窗口。`tolerant: true` 可以恢复旧行为，给「坐标反正都对」的调用方留口子。
+- **peer 范围开始接受 0.2.1 世代**（0.5.10）。`0.2.1` 这个元组需要它自己的预发布比较符：预发布版本只有在某个比较符与它同属一个 `major.minor.patch`、且自身带预发布标签时才放行。
+
 **0.5.8** —— 两个修复，同时包已发布到 npm：[`@momasiku/dsh-pilot`](https://www.npmjs.com/package/@momasiku/dsh-pilot)。
 
 - **显示器几何会变陈旧，而且是静默的。** 原本显示器清单来自 `System.Windows.Forms.Screen.AllScreens`，这是静态缓存，只有进程处理 `WM_DISPLAYCHANGE` 时才刷新；而暖 worker 是没有消息泵的 `-NonInteractive` 读取循环，缓存永远不会失效。实测：存活 13 小时的 worker 报 `<screens>primary WinDisc 0,0,1600,900</screens>`，而同一份结果里的 `<image_size>` 是 `1024x768` —— 同一份结果两个答案，而坐标契约正是从这份帧里读的。现在每次截图都用 `EnumDisplayMonitors` 重新枚举，虚拟屏改用 `GetSystemMetrics`，两者都不缓存。

package/lib/index.js

@@ -992,7 +992,7 @@ const POINTER_ACTIONS = new Set(['click', 'doubleClick', 'rightClick', 'middleCl
 const RISKS = ['low', 'medium', 'high'];
 const RISK_ORDER = { low: 0, medium: 1, high: 2 };
 /** Step fields forwarded verbatim to the action helper. */
-const STEP_FIELDS = ['x', 'y', 'toX', 'toY', 'amount', 'text', 'key', 'title', 'button', 'tolerant'];
+const STEP_FIELDS = ['x', 'y', 'toX', 'toY', 'amount', 'text', 'key', 'title', 'button', 'tolerant', 'target', 'includeValues', 'meaningfulOnly'];
 
 const SEQUENCE_STEP_RESULT_SCHEMA = {
 	type: 'object',
@@ -1252,7 +1252,8 @@ function apply(ctx, config) {
 	ctx.tools.register(defineTool({
 		name: 'desktop_control',
 		description: [
-			'Operate the live desktop: move the pointer, click, drag, scroll, type text, send keys, focus a window, or list windows.',
+			'Operate the live desktop: move the pointer, click, drag, scroll, type text, send keys, focus a window, read a window control tree, or list windows.',
+			'`controls` is the cheap way to hit a named control: it returns each control role, name and rectangle, so a click becomes the centre of that rectangle (pass its index as target) instead of a guess at pixels. Reach for a screenshot when the window exposes no usable tree - a canvas, a drawing surface, or an Electron window that publishes nothing - or when you need to see what is drawn rather than what is declared.',
 			'Coordinates are physical screen pixels and match screen_view images exactly, so read the target off a fresh screenshot and reuse those pixels verbatim.',
 			'Each press, keystroke, and window focus goes to whatever is focused on the real desktop, so call screen_view first and re-check after acting when the result matters.',
 			'Set capture true to receive a fresh image in the same result, which shows the effect of the action without a second round trip.',
@@ -1262,8 +1263,8 @@ function apply(ctx, config) {
 			action: {
 				type: 'string',
 				required: true,
-				enum: ['click', 'doubleClick', 'rightClick', 'middleClick', 'move', 'drag', 'scroll', 'type', 'key', 'focus', 'windows'],
-				description: 'click | doubleClick | rightClick | middleClick | move | drag | scroll | type | key | focus | windows.'
+				enum: ['click', 'doubleClick', 'rightClick', 'middleClick', 'move', 'drag', 'scroll', 'type', 'key', 'focus', 'windows', 'controls'],
+				description: 'click | doubleClick | rightClick | middleClick | move | drag | scroll | type | key | focus | windows | controls.'
 			},
 			x: { type: 'integer', description: 'Target x in physical screen pixels; required for the pointer actions and for scroll-at-a-point.' },
 			y: { type: 'integer', description: 'Target y in physical screen pixels; required for the pointer actions and for scroll-at-a-point.' },
@@ -1285,6 +1286,18 @@ function apply(ctx, config) {
 				type: 'boolean',
 				description: 'focus only: when Windows refuses to hand the window the foreground, record a note and continue instead of failing the step. Use it when the coordinates are right whichever window ends up in front.'
 			},
+			target: {
+				type: 'integer',
+				description: 'Click a control instead of a coordinate: the index number from a controls listing of the same window. The listing is re-read before the click, so an index that no longer resolves fails rather than hitting whatever moved into that slot.'
+			},
+			includeValues: {
+				type: 'boolean',
+				description: 'controls only: also return each control current value (edit boxes, text). Off by default, because a value is content rather than structure; password fields are always masked.'
+			},
+			meaningfulOnly: {
+				type: 'boolean',
+				description: 'controls only, default true: return just the controls a person can operate (buttons, edits, list rows, menus, tabs, check boxes...). Measured on a real 673-element window the filtered listing is 77 elements and about 850 tokens, against 9k for all of them, because the rest is layout panes with no name and nothing to click. Pass false when you need the whole tree - for example to see how a window is composed rather than what can be operated.'
+			},
 			capture: {
 				type: 'boolean',
 				description: 'Capture a fresh frame after the action and show it in this result. Default true.'
@@ -1319,6 +1332,10 @@ function apply(ctx, config) {
 					notes: { type: 'array', items: { type: 'string' } },
 					windows: { type: 'array', items: WINDOW_INFO_SCHEMA },
 					windowCount: { type: 'integer' },
+					controls: { type: 'array', items: { type: 'string' } },
+					controlCount: { type: 'integer' },
+					controlsTruncated: { type: 'boolean' },
+					controlsWindow: { type: 'string' },
 					actedAt: { type: 'string' },
 					capture: CAPTURE_SCHEMA,
 					capturePath: { type: 'string' },
@@ -1339,7 +1356,9 @@ function apply(ctx, config) {
 		isConcurrencySafe: () => false,
 		async execute(args, exec) {
 			warmWorker(config);
-			const wantsCapture = args.capture !== false;
+			// Reading the control tree is the alternative to taking a frame, so it does not
+			// take one unless asked: a caller that wants both still gets both.
+			const wantsCapture = args.capture === undefined ? args.action !== 'controls' : args.capture === true;
 			if (wantsCapture) await assertImageCapableRoute(ctx, exec, 'desktop_control with capture');
 
 			// The settle exists so a frame is not taken mid-repaint. With no frame to
@@ -1350,7 +1369,7 @@ function apply(ctx, config) {
 			const explicitSettle = Number.isSafeInteger(args.settleMs);
 			const settle = explicitSettle ? args.settleMs : (wantsCapture ? config.settleMs : 0);
 			const params = { action: args.action, settleMs: settle };
-			for (const field of ['x', 'y', 'toX', 'toY', 'amount', 'text', 'key', 'title', 'button', 'tolerant']) {
+			for (const field of ['x', 'y', 'toX', 'toY', 'amount', 'text', 'key', 'title', 'button', 'tolerant', 'target', 'includeValues', 'meaningfulOnly']) {
 				if (args[field] !== undefined && args[field] !== null) params[field] = args[field];
 			}
 
@@ -1359,7 +1378,7 @@ function apply(ctx, config) {
 			// Copy the declared fields only: the helper's own bookkeeping keys
 			// must not reach the closed output schema.
 			const value = { action: args.action };
-			for (const field of ['cursor', 'foregroundTitle', 'foregroundClass', 'foregroundProcess', 'notes', 'windowCount', 'actedAt']) {
+			for (const field of ['cursor', 'foregroundTitle', 'foregroundClass', 'foregroundProcess', 'notes', 'windowCount', 'actedAt', 'controls', 'controlCount', 'controlsTruncated', 'controlsWindow']) {
 				if (acted[field] !== undefined) value[field] = acted[field];
 			}
 			// The list is projected, not copied: the sensor's window entries carry

package/lib/scripts/_dsh-action.ps1

@@ -10,10 +10,20 @@
 # a hashtable instead of being printed.
 $ErrorActionPreference = 'Stop'
 
+# UI Automation: the control tree, for a caller that would rather read a control's
+# name and rectangle than guess its pixels. Loaded from here so every entry script
+# that dot-sources this file gets it for free.
+. (Join-Path $PSScriptRoot '_dsh-uia.ps1')
+
 # Set by Write-ActionError so a caller can forward the structured payload a thrown
 # failure carries (cancellation is not just an error message).
 $script:ActionStopPayload = $null
 
+# The last controls listing handed out, keyed by element index. A flat listing can
+# keep its length while its rows change, so an index that still resolves can still
+# point at a different control; this is what lets a click notice that.
+$script:UiListing = $null
+
 function Write-ActionError([string] $message) {
     $script:ActionStopPayload = [ordered]@{ ok = $false; error = $message }
     throw [System.Exception]::new($message)
@@ -156,6 +166,59 @@ Execute one action. $Config is the request object the callers already build
 (action, x, y, toX, toY, amount, text, key, title, button, hwnd, cancelFile,
 settleMs). Returns the same shape the one-shot script used to print.
 #>
+function Find-DshWindowHandle([string] $needle) {
+    # Foreground wins, otherwise the first visible match: the same rule focus uses,
+    # so a title means the same window to every action that takes one.
+    $lower = $needle.ToLowerInvariant()
+    $fallback = [IntPtr]::Zero
+    foreach ($handle in [DshInput]::TopLevelWindows()) {
+        if (-not [DshInput]::IsWindowVisible($handle)) { continue }
+        $title = [DshInput]::WindowTitle($handle)
+        if ($title.Length -eq 0) { continue }
+        $owner = [uint32] 0
+        $null = [DshInput]::GetWindowThreadProcessId($handle, [ref] $owner)
+        $processName = Get-ProcessNameSafe $owner
+        $hit = $title.ToLowerInvariant().Contains($lower)
+        if (-not $hit -and $null -ne $processName) { $hit = $processName.ToLowerInvariant().Contains($lower) }
+        if (-not $hit) { continue }
+        if ([DshInput]::IsForeground($handle)) { return $handle }
+        if ($fallback -eq [IntPtr]::Zero) { $fallback = $handle }
+    }
+    return $fallback
+}
+
+function Get-DshTargetPoint([string] $indexText, [IntPtr] $hwnd, [string] $title) {
+    # An index addresses a node only against the listing it came from. Resolving it
+    # again is what turns 'the button I saw' into 'the button that is there now',
+    # and an index that no longer resolves fails instead of clicking its successor.
+    $index = 0
+    if (-not [int]::TryParse($indexText, [ref] $index)) { Write-ActionError ('target must be the index of a control listed by ''controls'', got ''{0}''' -f $indexText) }
+    $handle = $hwnd
+    if ($handle -eq [IntPtr]::Zero) {
+        if ([string]::IsNullOrWhiteSpace($title)) { Write-ActionError ('target {0} needs a window: pass title or hwnd with it' -f $index) }
+        $handle = Find-DshWindowHandle $title
+        if ($handle -eq [IntPtr]::Zero) { Write-ActionError ('target {0} needs a window: no visible top-level window matches ''{1}''' -f $index, $title) }
+    }
+    $element = Resolve-DshUiaIndex -Handle $handle -Index $index
+    if ($null -eq $element) { Write-ActionError ('the tree changed: index {0} is past the end of the listing for ''{1}''. Read controls again.' -f $index, [DshInput]::WindowTitle($handle)) }
+    if ($null -ne $script:UiListing -and $script:UiListing.handle -eq [long] $handle -and $script:UiListing.items.ContainsKey($index)) {
+        # Compare identity, not just existence: a reorder keeps the index valid.
+        $gotRole = ''
+        $gotName = ''
+        try {
+            $gotRole = $element.Current.ControlType.ProgrammaticName -replace 'ControlType\.', ''
+            $gotName = [string] $element.Current.Name
+        } catch { }
+        $want = $script:UiListing.items[$index]
+        if ($gotRole -ne $want.role -or $gotName -ne $want.name) {
+            Write-ActionError ('the control at index {0} is not the one that was listed: expected [{1}] ''{2}'', found [{3}] ''{4}''. Read controls again.' -f $index, $want.role, $want.name, $gotRole, $gotName)
+        }
+    }
+    $centre = Get-DshUiaCentre -Element $element
+    if ($null -eq $centre) { Write-ActionError ('the control at index {0} has no on-screen rectangle - it may be scrolled out of view or collapsed' -f $index) }
+    return @($centre.x, $centre.y)
+}
+
 function Invoke-DshAction {
     param([object] $Config)
 
@@ -164,6 +227,16 @@ function Invoke-DshAction {
         return $null
     }
 
+    # x/y, or the centre of the control a target path addresses. One entry point so
+    # every pointer action accepts both spellings without repeating the branch.
+    function Get-AnchoredPoint([string] $what) {
+        $targetPath = [string] (Get-Param 'target')
+        if (-not [string]::IsNullOrWhiteSpace($targetPath)) {
+            return Get-DshTargetPoint $targetPath $script:TargetHwnd $windowTitle
+        }
+        return Get-Point (Get-Param 'x') (Get-Param 'y') $what
+    }
+
     $script:CancelFile = [string] (Get-Param 'cancelFile')
 
     # Optional target window. When present, every input action re-asserts it as the
@@ -199,7 +272,7 @@ function Invoke-DshAction {
 
 switch ($action) {
     'click' {
-        $point = Get-Point (Get-Param 'x') (Get-Param 'y') 'click'
+        $point = Get-AnchoredPoint 'click'
         $null = [DshInput]::SetCursorPos($point[0], $point[1])
         Start-Sleep -Milliseconds 80
         Assert-Alive
@@ -209,7 +282,7 @@ switch ($action) {
         [DshInput]::Button($button, $false)
     }
     'doubleClick' {
-        $point = Get-Point (Get-Param 'x') (Get-Param 'y') 'doubleClick'
+        $point = Get-AnchoredPoint 'doubleClick'
         $null = [DshInput]::SetCursorPos($point[0], $point[1])
         Start-Sleep -Milliseconds 80
         Assert-Alive
@@ -221,7 +294,7 @@ switch ($action) {
         [DshInput]::Button($button, $true); Start-Sleep -Milliseconds 40; [DshInput]::Button($button, $false)
     }
     'rightClick' {
-        $point = Get-Point (Get-Param 'x') (Get-Param 'y') 'rightClick'
+        $point = Get-AnchoredPoint 'rightClick'
         $null = [DshInput]::SetCursorPos($point[0], $point[1])
         Start-Sleep -Milliseconds 80
         Assert-Alive
@@ -231,7 +304,7 @@ switch ($action) {
         [DshInput]::Button('right', $false)
     }
     'middleClick' {
-        $point = Get-Point (Get-Param 'x') (Get-Param 'y') 'middleClick'
+        $point = Get-AnchoredPoint 'middleClick'
         $null = [DshInput]::SetCursorPos($point[0], $point[1])
         Start-Sleep -Milliseconds 80
         Assert-Alive
@@ -241,7 +314,7 @@ switch ($action) {
         [DshInput]::Button('middle', $false)
     }
     'move' {
-        $point = Get-Point (Get-Param 'x') (Get-Param 'y') 'move'
+        $point = Get-AnchoredPoint 'move'
         $null = [DshInput]::SetCursorPos($point[0], $point[1])
         # The move is queued, so an immediate read can still return the previous
         # position; wait (bounded) until the pointer is where the caller asked.
@@ -303,8 +376,47 @@ switch ($action) {
             catch { Write-ActionError "the keystroke '$keySpec' could not be delivered: $($_.Exception.Message)" }
         }
     }
-    'focus' {
-        # A refused hand-off used to be a note: the step reported ok and the batch
+    'controls' {
+        # Read the window's UI Automation tree and return it as text. This is the
+        # cheap half of seeing: a control's name and rectangle, no frame, no pixels.
+        # It returns its own result and returns early so it never pays the settle
+        # or the capture - the whole point is that no screenshot is needed.
+        $treeHandle = $script:TargetHwnd
+        if ($treeHandle -eq [IntPtr]::Zero) {
+            if ([string]::IsNullOrWhiteSpace($windowTitle)) { Write-ActionError 'controls requires a title, or an hwnd' }
+            $treeHandle = Find-DshWindowHandle $windowTitle
+            if ($treeHandle -eq [IntPtr]::Zero) { Write-ActionError ('no visible top-level window matches ''{0}''; pass an hwnd if the window is hidden to the tray' -f $windowTitle) }
+        }
+        $includeValues = $false
+        $ivRaw = Get-Param 'includeValues'
+        if ($null -ne $ivRaw) { $includeValues = [bool] $ivRaw }
+        $meaningfulOnly = $true
+        $moRaw = Get-Param 'meaningfulOnly'
+        if ($null -ne $moRaw) { $meaningfulOnly = [bool] $moRaw }
+        $tree = Get-DshUiaNodes -Handle $treeHandle -IncludeValues $includeValues -MeaningfulOnly $meaningfulOnly
+        if (-not $tree.ok) { Write-ActionError $tree.error }
+        $items = @{}
+        foreach ($n in $tree.nodes) { $items[[int] $n.index] = @{ role = [string] $n.role; name = [string] $n.name } }
+        $script:UiListing = @{ handle = [long] $treeHandle; items = $items }
+        $lines = Format-DshUiaNodes -Nodes $tree.nodes -Truncated $tree.truncated
+        $treeCursor = [DshInput]::Cursor()
+        $result = [ordered]@{
+            ok                = $true
+            action            = 'controls'
+            cursor            = [ordered]@{ x = $treeCursor.X; y = $treeCursor.Y }
+            foregroundTitle   = [DshInput]::ForegroundTitle()
+            foregroundClass   = [DshInput]::ForegroundClass()
+            foregroundProcess = Get-ProcessNameSafe ([uint32] [DshInput]::ForegroundPid())
+            controls          = $lines
+            controlCount      = $tree.nodes.Count
+            controlsTruncated = [bool] $tree.truncated
+            controlsWindow    = [DshInput]::WindowTitle($treeHandle)
+            notes             = $notes
+            actedAt           = [DateTimeOffset]::Now.ToString('o')
+        }
+        return $result
+    }
+    'focus' {        # A refused hand-off used to be a note: the step reported ok and the batch
         # carried on, so every later click and keystroke went to whatever window was
         # actually in front. That is the one failure this plugin must not report as
         # success, so it is an error now; tolerant keeps the old behaviour for a

package/lib/scripts/_dsh-uia.ps1

@@ -0,0 +1,156 @@
+# pilot / _dsh-uia.ps1 - read a window's UI Automation tree, and resolve a node by index.
+#
+# Why this exists: screen_view returns pixels. Finding a control in pixels costs the
+# model a guess at coordinates plus another frame to confirm the guess. UI Automation
+# hands over the control's role, name and bounding rectangle directly, so a click
+# becomes 'the centre of this rect' instead of 'where I think that button is'.
+#
+# The read uses FindAll with a CacheRequest, not TreeWalker. Measured on a 674-node
+# window: TreeWalker ignores the cache request and pays six or seven cross-process
+# reads per node (~2.7 s), while FindAll populates every property in one call (~270 ms).
+# The price is that a flat list has no parent/child structure, so a node is addressed
+# by its position in that list rather than by a path.
+#
+# Nothing is kept between calls: an index only means anything against the listing it
+# came from, and re-reading is the only way to find out that it changed.
+
+$script:DshUiaReady = $false
+
+function Initialize-DshUia {
+    if ($script:DshUiaReady) { return }
+    Add-Type -AssemblyName UIAutomationClient
+    Add-Type -AssemblyName UIAutomationTypes
+    $script:DshUiaReady = $true
+}
+
+<# Read one window's descendants. Every node carries the index that addresses it. #>
+function Get-DshUiaNodes {
+    param(
+        [Parameter(Mandatory = $true)][IntPtr] $Handle,
+        [int] $MaxNodes = 2000,
+        [bool] $IncludeValues = $false,
+        [bool] $MeaningfulOnly = $true
+    )
+
+    Initialize-DshUia
+    $AE = [System.Windows.Automation.AutomationElement]
+
+    $root = $AE::FromHandle($Handle)
+    if ($null -eq $root) { return [ordered]@{ ok = $false; error = 'the window exposes no automation element'; nodes = @(); truncated = $false; total = 0 } }
+
+    $cache = New-Object System.Windows.Automation.CacheRequest
+    foreach ($prop in @(
+        $AE::ControlTypeProperty, $AE::NameProperty, $AE::BoundingRectangleProperty,
+        $AE::IsEnabledProperty, $AE::IsOffscreenProperty, $AE::IsPasswordProperty
+    )) { $cache.Add($prop) }
+
+    $collection = $null
+    $cache.Push()
+    try {
+        $collection = $root.FindAll([System.Windows.Automation.TreeScope]::Descendants, [System.Windows.Automation.Condition]::TrueCondition)
+    } catch {
+        return [ordered]@{ ok = $false; error = ('the tree could not be read: ' + $_.Exception.Message); nodes = @(); truncated = $false; total = 0 }
+    } finally {
+        $cache.Pop()
+    }
+
+    $total = $collection.Count
+    $limit = $total
+    $truncated = $false
+    if ($limit -gt $MaxNodes) { $limit = $MaxNodes; $truncated = $true }
+
+    # Roles a person can operate. Measured on a 673-element window: the filtered
+    $interactiveRoles = @('Button','Edit','ComboBox','ListItem','MenuItem','TabItem','CheckBox','RadioButton','Hyperlink','SplitButton','Slider','Spinner','TreeItem','DataItem')
+    $bulkRoles = @('DataItem','TreeItem','ListItem','MenuItem')
+
+    $nodes = New-Object System.Collections.Generic.List[object]
+    for ($i = 0; $i -lt $limit; $i++) {
+        $el = $collection[$i]
+        try {
+            $r = $el.Cached.BoundingRectangle
+            $left = 0; $top = 0; $w = 0; $h = 0; $hasRect = $false
+            if (-not $r.IsEmpty) { $left = [int] $r.X; $top = [int] $r.Y; $w = [int] $r.Width; $h = [int] $r.Height; $hasRect = $true }
+            $value = $null
+            if ($IncludeValues) {
+                if ($el.Cached.IsPassword) {
+                    $value = '<password field>'
+                } else {
+                    try {
+                        $vp = $el.GetCurrentPattern([System.Windows.Automation.ValuePattern]::Pattern)
+                        if ($null -ne $vp) { $value = [string] $vp.Current.Value }
+                    } catch { }
+                }
+            }
+            $role = ($el.Cached.ControlType.ProgrammaticName -replace 'ControlType\.', '')
+            if ($MeaningfulOnly) {
+                if ($interactiveRoles -notcontains $role) { continue }
+                # A bulk container row with no name is addressable by index but carries
+                # nothing to reason about, and they are the bulk of a large list.
+                if ($bulkRoles -contains $role -and $name.Length -eq 0) { continue }
+            }
+            $nodes.Add([ordered]@{
+                index = $i
+                role = ($el.Cached.ControlType.ProgrammaticName -replace 'ControlType\.', '')
+                name = [string] $el.Cached.Name
+                left = $left; top = $top; width = $w; height = $h; hasRect = $hasRect
+                enabled = [bool] $el.Cached.IsEnabled
+                offscreen = [bool] $el.Cached.IsOffscreen
+                value = $value
+            })
+        } catch {
+            # A provider that throws mid-read must not abort the whole listing.
+        }
+    }
+
+    return [ordered]@{ ok = $true; nodes = $nodes; truncated = $truncated; total = $total }
+}
+
+<# Render the nodes as the text the model reads. One line per control. #>
+function Format-DshUiaNodes {
+    param([object] $Nodes, [bool] $Truncated, [int] $Total)
+
+    $lines = New-Object System.Collections.Generic.List[string]
+    foreach ($n in $Nodes) {
+        $name = [string] $n.name
+        if ($name.Length -gt 200) { $name = $name.Substring(0, 200) + '...' }
+        $rect = '(no rect)'
+        if ($n.hasRect) { $rect = ('{0},{1} {2}x{3}' -f $n.left, $n.top, $n.width, $n.height) }
+        $bits = @()
+        if (-not $n.enabled) { $bits += 'disabled' }
+        if ($n.offscreen) { $bits += 'offscreen' }
+        if ($null -ne $n.value) { $bits += ('value=' + ($n.value -replace '\s+', ' ')) }
+        $tail = ''
+        if ($bits.Count -gt 0) { $tail = '  ' + ($bits -join ' ') }
+        $lines.Add(('{0} [{1}] ''{2}'' @ {3}{4}' -f $n.index, $n.role, $name, $rect, $tail))
+    }
+    if ($Truncated) { $lines.Add(('... truncated at the node cap ({0} elements in total)' -f $Total)) }
+    return $lines.ToArray()
+}
+
+<# Resolve an index back to an element by re-reading the same listing. #>
+function Resolve-DshUiaIndex {
+    param([Parameter(Mandatory = $true)][IntPtr] $Handle, [Parameter(Mandatory = $true)][int] $Index)
+
+    Initialize-DshUia
+    $AE = [System.Windows.Automation.AutomationElement]
+    $root = $AE::FromHandle($Handle)
+    if ($null -eq $root) { return $null }
+    $all = $root.FindAll([System.Windows.Automation.TreeScope]::Descendants, [System.Windows.Automation.Condition]::TrueCondition)
+    if ($Index -lt 0 -or $Index -ge $all.Count) { return $null }
+    return $all[$Index]
+}
+
+<# Centre of an element's rectangle, or $null when it has no on-screen box. #>
+function Get-DshUiaCentre {
+    param([Parameter(Mandatory = $true)][object] $Element)
+    try {
+        $r = $Element.Current.BoundingRectangle
+        if ($r.IsEmpty) { return $null }
+        if ($r.Width -le 0 -or $r.Height -le 0) { return $null }
+        return [pscustomobject]@{
+            x = [int] ($r.X + $r.Width / 2)
+            y = [int] ($r.Y + $r.Height / 2)
+            left = [int] $r.X; top = [int] $r.Y; width = [int] $r.Width; height = [int] $r.Height
+        }
+    } catch { return $null }
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@momasiku/dsh-pilot",
-  "version": "0.5.10",
+  "version": "0.5.11",
   "description": "Desktop vision and control for DeepSeek Harness - the whole Windows machine, any application, not browser automation: screen_view captures the live screen as an image the model can see, desktop_control drives the mouse, keyboard, and windows against the exact same pixel space; desktop_sequence runs a whole batch of actions in one call and looks once at the end.",
   "license": "MIT",
   "author": "momasiku",