@deepseek-ai/dsh-experimental-cua-macos 0.2.2-rc.1 → 0.2.2-rc.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (81) hide show
  1. package/README.i18n.yaml +14 -14
  2. package/README.md +102 -67
  3. package/README.zh.md +102 -67
  4. package/cua-helper.json +7 -5
  5. package/lib/index.js +165 -110
  6. package/lib/types/ax-limits.d.ts +6 -0
  7. package/lib/types/ax-limits.d.ts.map +1 -0
  8. package/lib/types/ax-limits.js.map +1 -0
  9. package/lib/types/guidance.d.ts +4 -3
  10. package/lib/types/guidance.d.ts.map +1 -0
  11. package/lib/types/guidance.js.map +1 -0
  12. package/lib/types/held-keys.d.ts +9 -0
  13. package/lib/types/held-keys.d.ts.map +1 -0
  14. package/lib/types/held-keys.js.map +1 -0
  15. package/lib/types/index.d.ts +82 -53
  16. package/lib/types/index.d.ts.map +1 -0
  17. package/lib/types/index.js.map +1 -0
  18. package/lib/types/native-build.d.ts +16 -2
  19. package/lib/types/native-build.d.ts.map +1 -0
  20. package/lib/types/native-build.js.map +1 -0
  21. package/lib/types/permission-guide.d.ts.map +1 -0
  22. package/lib/types/permission-guide.js.map +1 -0
  23. package/lib/types/shared-native.d.ts.map +1 -0
  24. package/lib/types/shared-native.js.map +1 -0
  25. package/native/README.md +52 -30
  26. package/native/README.zh.md +52 -30
  27. package/native/Sources/CuaNative/AXAccess.swift +241 -51
  28. package/native/Sources/CuaNative/AXHover.swift +3 -3
  29. package/native/Sources/CuaNative/AXWorker.swift +118 -97
  30. package/native/Sources/CuaNative/HeldKeys.swift +114 -0
  31. package/native/Sources/CuaNative/KeyboardInput.swift +83 -64
  32. package/native/Sources/CuaNative/NativePlatform.swift +16 -6
  33. package/native/Sources/CuaNative/NativeProtocol.generated.swift +16 -11
  34. package/native/Sources/CuaNative/Permissions.swift +1 -0
  35. package/native/Sources/CuaNative/Protocol.swift +11 -3
  36. package/native/Sources/CuaNative/RequestAdmission.swift +11 -0
  37. package/native/Sources/CuaNative/RequestCoordinator.swift +15 -17
  38. package/native/Sources/CuaNative/RequestScheduler.swift +1 -0
  39. package/native/Sources/CuaNative/SharedHelperRuntime.swift +4 -3
  40. package/native/Sources/CuaNative/StandaloneApplication.swift +86 -0
  41. package/native/Sources/CuaNative/WindowCapture.swift +2 -0
  42. package/native/Sources/CuaNative/WindowGeometry.swift +2 -2
  43. package/native/Sources/CuaNative/WindowPointerInput.swift +2 -2
  44. package/native/bin/darwin-arm64/current.json +1 -1
  45. package/native/bin/darwin-arm64/{f0879ec551751eaefd3402b3980575207183f0bf60590e02fb1591e34b740644 → d896b6b6e45a717a7dbc7aa52d0b34eb70f1558c472b16b1272f80678b2ed4be}/DSH Computer Use.app/Contents/CodeResources +0 -0
  46. package/native/bin/darwin-arm64/d896b6b6e45a717a7dbc7aa52d0b34eb70f1558c472b16b1272f80678b2ed4be/DSH Computer Use.app/Contents/MacOS/dsh-cua-native +0 -0
  47. package/native/bin/darwin-x64/{848ea303966d41561060de50cc12e51e729f68dbb71035ea460b353b821cf34d → b68fcdb1474dad2b8694e1aa78d53bc963e6263fb3d12953502537a44df7f646}/DSH Computer Use.app/Contents/CodeResources +0 -0
  48. package/native/bin/darwin-x64/b68fcdb1474dad2b8694e1aa78d53bc963e6263fb3d12953502537a44df7f646/DSH Computer Use.app/Contents/MacOS/dsh-cua-native +0 -0
  49. package/native/bin/darwin-x64/current.json +1 -1
  50. package/native/build_app.py +72 -16
  51. package/native/protocol.schema.json +373 -84
  52. package/native/tests/swift/AXFailureTests.swift +67 -3
  53. package/native/tests/swift/AXTextFormatterTests.swift +0 -10
  54. package/native/tests/swift/ForegroundTests.swift +6 -33
  55. package/native/tests/swift/KeyboardChordTests.swift +137 -0
  56. package/native/tests/swift/ListActionsTests.swift +38 -0
  57. package/native/tests/swift/PerformActionTests.swift +106 -0
  58. package/native/tests/swift/ProtocolTests.swift +32 -2
  59. package/native/tests/swift/SharedHelperTests.swift +1 -1
  60. package/native/tests/swift/StandaloneApplicationTests.swift +136 -0
  61. package/native/upstream.json +1 -1
  62. package/package.json +46 -43
  63. package/src/ax-limits.ts +2 -0
  64. package/src/guidance.ts +21 -0
  65. package/src/held-keys.ts +42 -0
  66. package/src/index.ts +90 -0
  67. package/src/native-build.ts +113 -0
  68. package/src/permission-guide.ts +110 -0
  69. package/src/shared-native.ts +294 -0
  70. package/native/bin/darwin-arm64/f0879ec551751eaefd3402b3980575207183f0bf60590e02fb1591e34b740644/DSH Computer Use.app/Contents/MacOS/dsh-cua-native +0 -0
  71. package/native/bin/darwin-x64/848ea303966d41561060de50cc12e51e729f68dbb71035ea460b353b821cf34d/DSH Computer Use.app/Contents/MacOS/dsh-cua-native +0 -0
  72. /package/native/bin/darwin-arm64/{f0879ec551751eaefd3402b3980575207183f0bf60590e02fb1591e34b740644 → d896b6b6e45a717a7dbc7aa52d0b34eb70f1558c472b16b1272f80678b2ed4be}/DSH Computer Use.app/Contents/Info.plist +0 -0
  73. /package/native/bin/darwin-arm64/{f0879ec551751eaefd3402b3980575207183f0bf60590e02fb1591e34b740644 → d896b6b6e45a717a7dbc7aa52d0b34eb70f1558c472b16b1272f80678b2ed4be}/DSH Computer Use.app/Contents/Resources/DSHComputerUse.icns +0 -0
  74. /package/native/bin/darwin-arm64/{f0879ec551751eaefd3402b3980575207183f0bf60590e02fb1591e34b740644 → d896b6b6e45a717a7dbc7aa52d0b34eb70f1558c472b16b1272f80678b2ed4be}/DSH Computer Use.app/Contents/Resources/LICENSE.computer-use-icon +0 -0
  75. /package/native/bin/darwin-arm64/{f0879ec551751eaefd3402b3980575207183f0bf60590e02fb1591e34b740644 → d896b6b6e45a717a7dbc7aa52d0b34eb70f1558c472b16b1272f80678b2ed4be}/DSH Computer Use.app/Contents/Resources/LICENSE.cua-driver +0 -0
  76. /package/native/bin/darwin-arm64/{f0879ec551751eaefd3402b3980575207183f0bf60590e02fb1591e34b740644 → d896b6b6e45a717a7dbc7aa52d0b34eb70f1558c472b16b1272f80678b2ed4be}/DSH Computer Use.app/Contents/_CodeSignature/CodeResources +0 -0
  77. /package/native/bin/darwin-x64/{848ea303966d41561060de50cc12e51e729f68dbb71035ea460b353b821cf34d → b68fcdb1474dad2b8694e1aa78d53bc963e6263fb3d12953502537a44df7f646}/DSH Computer Use.app/Contents/Info.plist +0 -0
  78. /package/native/bin/darwin-x64/{848ea303966d41561060de50cc12e51e729f68dbb71035ea460b353b821cf34d → b68fcdb1474dad2b8694e1aa78d53bc963e6263fb3d12953502537a44df7f646}/DSH Computer Use.app/Contents/Resources/DSHComputerUse.icns +0 -0
  79. /package/native/bin/darwin-x64/{848ea303966d41561060de50cc12e51e729f68dbb71035ea460b353b821cf34d → b68fcdb1474dad2b8694e1aa78d53bc963e6263fb3d12953502537a44df7f646}/DSH Computer Use.app/Contents/Resources/LICENSE.computer-use-icon +0 -0
  80. /package/native/bin/darwin-x64/{848ea303966d41561060de50cc12e51e729f68dbb71035ea460b353b821cf34d → b68fcdb1474dad2b8694e1aa78d53bc963e6263fb3d12953502537a44df7f646}/DSH Computer Use.app/Contents/Resources/LICENSE.cua-driver +0 -0
  81. /package/native/bin/darwin-x64/{848ea303966d41561060de50cc12e51e729f68dbb71035ea460b353b821cf34d → b68fcdb1474dad2b8694e1aa78d53bc963e6263fb3d12953502537a44df7f646}/DSH Computer Use.app/Contents/_CodeSignature/CodeResources +0 -0
package/README.i18n.yaml CHANGED
@@ -9,20 +9,20 @@
9
9
  en: 022ca213425e11ee
10
10
  zh: d64631748ed91cc2
11
11
  /deepseek-ai-dsh-experimental-cua-macos/summary:
12
- en: 4549f02fc53854a5
13
- zh: 6f74f1130b6623a8
12
+ en: 7ffb2f01c0b05f32
13
+ zh: 74dac288fa7f7690
14
14
  /deepseek-ai-dsh-experimental-cua-macos/table-of-contents:
15
- en: 1c91307fdaebba7a
16
- zh: 910fba0aa793f47f
15
+ en: 1777e7bf6785f986
16
+ zh: e3daac1074ebb74d
17
17
  /deepseek-ai-dsh-experimental-cua-macos/use-this-package:
18
- en: ead96859cdf97de5
19
- zh: acaf62c479d5d6e8
18
+ en: 233423a8675de85d
19
+ zh: faf5cecb9999cfa7
20
20
  /deepseek-ai-dsh-experimental-cua-macos/use-this-package/connect-an-external-execution-environment:
21
- en: 786b8fd630b11e5f
22
- zh: 55143f15486afce8
21
+ en: 1bf907f257ef6f90
22
+ zh: ed71613b8de260aa
23
23
  /deepseek-ai-dsh-experimental-cua-macos/understand-the-implementation:
24
- en: 8b9ec208ca057a4a
25
- zh: e8dbb45891ad15b8
24
+ en: 4e82b85a5aa72eee
25
+ zh: 4e004b7a205d8688
26
26
  /deepseek-ai-dsh-experimental-cua-macos/further-exploration:
27
27
  en: ff9fac614001a98f
28
28
  zh: 329678941154aaea
@@ -39,14 +39,14 @@
39
39
  en: 2dbc22ffd1f8a717
40
40
  zh: d4458872e75305d9
41
41
  /deepseek-ai-dsh-experimental-cua-macos/model-experience/native-application-operations/token-effect:
42
- en: ab52b46d874fa8b9
43
- zh: e87d3620ef77cb06
42
+ en: 86b0b7e406627a47
43
+ zh: 53ad627fb3521f28
44
44
  /deepseek-ai-dsh-experimental-cua-macos/model-experience/native-application-operations/kv-cache-effect:
45
45
  en: a2c9acb5a4320dda
46
46
  zh: e8d1586af60215c4
47
47
  /deepseek-ai-dsh-experimental-cua-macos/known-limitations-and-deferred-work:
48
- en: 791c1ec9ff4b2978
49
- zh: a32d35de026f3285
48
+ en: b362f9357fe79293
49
+ zh: a74f0e9cffb647bc
50
50
  /deepseek-ai-dsh-experimental-cua-macos/known-limitations-and-deferred-work/dev-note:
51
51
  en: b836f01bd0bceb4a
52
52
  zh: 8122fa08b2aa95fa
package/README.md CHANGED
@@ -9,7 +9,7 @@ English | [中文](README.zh.md)
9
9
 
10
10
  ## Summary
11
11
 
12
- Inspect macOS permissions, select running applications and windows, read accessibility trees, capture screenshots and send input. The object API can be retained by an external persistent execution environment. The Swift helper uses macOS frameworks directly and adapts selected MIT-licensed Cua Driver behavior. This package owns neither a PTC runtime nor variable persistence.
12
+ Inspect macOS permissions, select running applications and windows, read accessibility trees, capture screenshots from a selected window, and send input. The object API can be retained by an external persistent execution environment. The Swift helper uses macOS frameworks directly and adapts selected MIT-licensed Cua Driver behavior. This package owns neither a PTC runtime nor variable persistence.
13
13
 
14
14
  ## Table of Contents
15
15
 
@@ -22,7 +22,7 @@ Inspect macOS permissions, select running applications and windows, read accessi
22
22
 
23
23
  The helper supplies bounded 64-pixel PNG application icons for permission prompts and saved-application lists. Icon reads do not launch an application or inspect its windows; icons do not determine reusable consent.
24
24
 
25
- The helper runs through LaunchServices as the independent application `DSH Computer Use.app`. Accessibility and Screen Recording permissions belong to this application. Permission guidance uses the responsible application’s bundle URL, identifier, filename, and system icon captured at startup; unavailable system attribution falls back to the independent helper’s own bundle. Dragging and Finder selection read the current on-disk bundle identifier and executable through the application-discovery reader. The page and guide both display the application’s `.app` filename. Existing connections retain their running application generation when another build updates `current.json`.
25
+ The helper runs through LaunchServices as an independent application: `DSH Computer Use.app` for release builds and `DSH Computer Use Dev.app` for source builds. Accessibility and Screen Recording permissions belong to this application. Permission guidance uses the responsible application’s bundle URL, identifier, filename, and system icon captured at startup; unavailable system attribution falls back to the independent helper’s own bundle. Dragging and Finder selection read the current on-disk bundle identifier and executable through the application-discovery reader. The page and guide both display the application’s `.app` filename. Existing connections retain their running application generation when another build updates `current.json`. When bundled inside another `.app`, it starts from a verified standalone copy; see [application storage](native/README.md#build-and-service).
26
26
 
27
27
  Permission requests open the selected privacy pane and a native floating guide. The guide selects the largest settings window once and follows that window by ID. It uses the normal window level, ordered immediately above the selected window so authentication dialogs remain above it. Placement reads window bounds without requesting Accessibility access. The guide stays hidden until the settings window is identified; a failed settings-open request displays manual navigation instructions independently. A material alpha mask gives the panel and its shadow rounded corners. The latest request replaces the shared guide, and only its owning connection can close it during cleanup. Navigation is attempted even when the application is unavailable, and the page reports that failure. Already-granted permissions open settings without a guide; explicit settings navigation dismisses previous guidance. A rejected settings-open request reports `SETTINGS_UNAVAILABLE`; missing or replaced application targets report `APPLICATION_UNAVAILABLE`. System Settings owns drag acceptance and any relaunch requirement.
28
28
 
@@ -33,21 +33,35 @@ The provider resolves guide and recovery copy from the current Client language,
33
33
  <a id="use-this-package"></a>
34
34
  ## Use this package
35
35
 
36
- This package is an optional macOS dependency. For local development, run `pnpm --dir packages/experimental/cua-macos run build:native` to generate an ad-hoc-signed application and `current.json` under `native/bin`. Release packaging requires `--identity "Developer ID Application: <name> (<team>)"` and notarization; see [native build and service](native/README.md#build-and-service). Installed packages preserve the distributed signature. Source preparation requires Swift and Python 3 from Apple Command Line Tools.
36
+ This package is an optional macOS dependency. For local development, run `pnpm --dir packages/experimental/cua-macos run build:native` to generate an ad-hoc-signed application and `current.json` under `native/bin`. Release packaging requires `--identity "Developer ID Application: <name> (<team>)"` and notarization; see [native build and service](native/README.md#build-and-service). Installed packages preserve the distributed signature. A package inside an Electron `app.asar` archive reads its application from the sibling `app.asar.unpacked` directory, which the Desktop installer populates; an installed package without the application fails preparation with an error naming the expected directory. Source preparation requires Swift and Python 3 from Apple Command Line Tools.
37
37
 
38
- For repository Web use, enable [Computer Use](../cua-bundle/README.md) in Plugins. This prepares the local executable on macOS 14 or later; Windows support is in development. Subsequent builds check it while the existing Web profile keeps the bundle selected. Disabled profiles do not prepare native components. Explicit source composition is also available:
38
+ For repository Web use, enable [Computer Use](../cua-bundle/README.md) in Plugins. This prepares the local executable on macOS 14 or later. Subsequent builds check it while the existing Web profile keeps the bundle selected. Disabled profiles do not prepare native components. Explicit source composition is also available:
39
39
 
40
40
  ```sh
41
41
  pnpm dsh headless --patch packages/experimental/cua-macos/examples/macos.cordis.yml --dump-config
42
42
  ```
43
43
 
44
- The dump command checks profile composition without a model request. For a real task, replace `--dump-config` with a prompt and configure the profile’s model credentials. The overlay mounts the computer-use registry and this provider; remove an existing competing provider before using it. `DSH Computer Use.app` needs Accessibility permission for AX/input and Screen Recording permission for capture; use the Computer Use permission page to open System Settings and add the helper. `getCapabilities` reports permissions and session lock state without opening a system dialog. Locked or unavailable desktop sessions reject UI observations and input before dispatch; the user must unlock macOS before resuming. `listApps` and `cua.getState` list running applications. `findApps(query)` searches standard macOS application directories by name, bundle ID or absolute path. `getApp` selects a unique name, bundle ID, path or running PID, obtains application consent, and launches a stopped app in the background when needed. A stopped application’s bundle identifier and executable path are read from disk before approval and rechecked before and after launch; applications without a bundle identifier use their canonical executable path for reusable consent. Selecting a stopped installation requests a new instance of that copy. The helper waits for the launched application to finish launching before returning its PID. A post-launch verification failure expires the pending handle but can leave the process running; inspect running applications before selecting a fresh handle. The returned object reads initial AX without activating the app. Launch cancellation may leave the application running. Model guidance starts with background observation and input, without activation during initialization or before each action. For hidden or minimized windows, try `app.show()` or `window.show()` and read fresh state. This requests unhide and unminimize without activation, raising or switching Spaces; app-level restoration selects the focused, main or sole AX window. The result names the selected `windowId`, or null when no window was selected, and reports `foregroundChanged` by comparing the front app and top window before and after the request. Null means foreground evidence was unavailable; later focus changes remain possible. If restoration fails or the window remains unusable, explain the needed foreground step, call `app.activate()` or an observed window’s `raise` action, then observe again. Necessary foreground access takes priority over preserving focus. Applications may change focus after input; background delivery does not guarantee an undisturbed user session.
44
+ The dump command checks profile composition without a model request. For a real task, replace `--dump-config` with a prompt and configure the profile’s model credentials. The overlay mounts the computer-use registry and this provider; remove an existing competing provider before using it. The helper needs Accessibility permission for AX/input and Screen Recording permission for capture; use the Computer Use permission page to open System Settings and add the helper. Source builds use the bundle filename `DSH Computer Use Dev.app` and declare `DSH Computer Use Dev` in Info.plist; select the development entry when granting access. `getCapabilities` returns the provider’s declared capabilities with observed permissions and session lock state. Locked or unavailable desktop sessions reject UI observations and input before dispatch; the user must unlock macOS before resuming. `listApps` and `cua.getState` list running applications. `findApps(query)` searches standard macOS application directories by name, bundle ID or absolute path. `getApp` selects a unique name, bundle ID, path or running PID, obtains application consent, and launches a stopped app in the background when needed. A stopped application’s bundle identifier and executable path are read from disk before approval and rechecked before and after launch; applications without a bundle identifier use their canonical executable path for reusable consent. Selecting a stopped installation requests a new instance of that copy. The helper waits for launch to complete before reporting the running state. A post-launch verification failure expires the pending handle but can leave the process running; inspect running applications before selecting a fresh handle. The returned object does not read AX or activate the app. Launch cancellation may leave the application running. Model guidance starts with background observation and input, without activation during initialization or before each action. For hidden or minimized windows, try `app.show()` or `window.show()` and read fresh state. This requests unhide and unminimize without activation, raising or switching Spaces; app-level restoration selects the focused, main or sole AX window. The result names the selected `windowId`, or null when no window was selected, and reports `foregroundChanged` by comparing the front app and top window before and after the request. Null means foreground evidence was unavailable; later focus changes remain possible. If restoration fails or the window remains unusable, explain the needed foreground step, call `app.activate()` or an observed window’s advertised `AXRaise` action, then observe again. Necessary foreground access takes priority over preserving focus. Applications may change focus after input; background delivery does not guarantee an undisturbed user session.
45
+
46
+ Source builds default to `sourceSigning: ad-hoc`, without creating certificates or keychains. Changed binaries can require new permission grants. To reduce repeated grants during rebuilds and across worktrees, select an existing certificate and its absolute keychain path in the DSH home’s `cordis.patch.yml`; `cua` is the entry ID loading `@deepseek-ai/dsh-experimental-cua-macos`:
47
+
48
+ ```yaml
49
+ - id: cua
50
+ config:
51
+ sourceSigning:
52
+ identity: "<certificate SHA-1 fingerprint>"
53
+ keychain: "/Users/<user>/.cua-macos-keys/dev/signing.keychain-db"
54
+ ```
55
+
56
+ `identity` is an existing certificate name or SHA-1 fingerprint; `keychain` is a required absolute path. Developers own creating and unlocking the keychain; builds only use the selected certificate. Preserve the entry’s other configuration fields when merging this patch. The repository’s [developer signing skill](../../../.agents/skills/dsh-cua-macos-signing/SKILL.md) guides Agents through creating, reusing, and removing identities under `~/.cua-macos-keys/`, with only their selection stored in DSH home.
45
57
 
46
58
  | Field | Default | Meaning |
47
59
  | --- | --- | --- |
48
60
  | `agentPreset` | `ptc` | Register tools and guidance only for Agents bound to this preset; `false` registers globally. |
49
61
  | `allowedApps` | `[]` | Legacy path/signature grants, imported once for removal; they do not authorize new bundle-ID identities. |
50
62
  | `helperPath` | package-owned executable | Absolute executable inside the signed helper `.app`; a bare binary fails with `NATIVE_APPLICATION_REQUIRED`. |
63
+ | `requireSystemPointerApproval` | `true` | Ask on the first explicit request to move the system pointer. With `false`, only explicitly requested moves use it. |
64
+ | `sourceSigning` | `ad-hoc` | Source signing: `ad-hoc`, or `{ identity, keychain }` for an existing certificate and keychain path. Ignored for installed packages and `helperPath`. |
51
65
  | `buildTimeoutMs` | `300000` | Maximum native preparation time. |
52
66
  | `timeoutMs` | `30000` | Per-operation deadline; cancellation can leave partial input. |
53
67
  | `connectTimeoutMs` | `5000` | Deadline for connecting to or starting the shared helper. |
@@ -58,26 +72,28 @@ The dump command checks profile composition without a model request. For a real
58
72
  | `maxClipboardBytes` | `33554432` | Maximum original clipboard bytes preserved during a paste; oversized contents reject before replacement. |
59
73
  | `pasteReadTimeoutMs` | `1000` | Clipboard read deadline after the Paste command returns; expiry reports `PASTE_NOT_CONSUMED`. |
60
74
  | `maxAppMatches` | `20` | Maximum installed-app search matches; broader queries reject and require a narrower name. |
61
- | `maxElements` / `maxDepth` | `2000` / `25` | Accessibility traversal limits. |
75
+ | `maxElements` / `maxDepth` | `5000` / `50` | Defaults for tree reads. Each read can raise or lower either value. Paste menu discovery always uses the configured limits. [Rationale](../../../.agents/notes/implemented/architecture/2026-11-09-cua-ax-traversal-defaults.md) |
62
76
  | `maxClickHoldMs` | `5000` | Maximum explicit click hold in milliseconds; accepts nonnegative integers. Zero only permits immediate release. |
63
77
  | `dragDurationMs` / `dragSteps` | `400` / `20` | Duration and movement segments for a drag. |
64
- | `textEventIntervalMs` | `0.25` | Delay after each Unicode down/up event, from 0 to 1000 ms. Chord timing is independent. |
78
+ | `textEventIntervalMs` | `0.25` | Delay after each Unicode down/up event, from 0 to 999 ms. Keystroke timing is independent. |
79
+
80
+ The [configuration catalog](../../../docs/config-catalog.md#deepseek-aidsh-experimental-cua-macos) lists all fields. Native tool calls use `{method, params}`; the complete method-specific schema is exposed to the model. `getApp` returns the `appHandle` used by every app-targeted operation, and `app.getWindow(windowId)` returns the object that owns screenshots and pointer input.
65
81
 
66
- The [configuration catalog](../../../docs/config-catalog.md#deepseek-aidsh-experimental-cua-macos) lists all fields. Native tool calls use `{method, params}`; the complete method-specific schema is exposed to the model. For example, `getApp` returns the `appHandle` needed by `listWindows`, `getState` and input operations.
82
+ `pressKey(key, modifiers?)` presses and releases one key. For shortcuts, pass held modifiers separately, as in `pressKey('a', ['command'])`. Pressing a modifier itself, as in `pressKey('shift')`, sets its flag on press and clears it on release. A single `+` or `plus` sends Shift+=. The macOS helper accepts `command`/`cmd`, `shift`, `option`/`alt`, `control`/`ctrl`, and `fn`, and maps them to their physical keys. Key names are platform-defined; keypad names (`KP_0` through `KP_9`, `KP_Enter`, and keypad operators) retain their physical key codes. `typeText` sends literal text, including its line breaks and tabs. Use the App object's `pressKey` for a short press.
67
83
 
68
- Keypad names (`KP_0` through `KP_9`, `KP_Enter`, and keypad operators) retain their physical key codes. Key chords use `+` between modifiers and the key. Use `+` for the plus key, `cmd++` for Command-plus, or the key name `plus`; an unfinished chord such as `cmd+` is rejected.
84
+ For a long press, call `app.sendKeyEvent('w', 'down')` inside `run_code`. The first down holds the key across subsequent tool calls. For an ordinary key already held by this execution, each additional down sends one key-down marked as a repeat to the original process. Choose the interval between calls. A control can filter synthetic repeats, so check its response. One up releases the key. Choose when to send each up: after W down, D down and D up, W stays down. When the program returns, exits with an error, or the user pauses it, the helper releases the remaining keys at the processes that received their downs. A later execution cannot release an earlier execution's keys.
69
85
 
70
86
  Agent disposal and plugin unload cancel and await the complete tool call, including image admission and attachment storage. An image write already in progress must settle before disposal returns; a canceled call cannot return a successful screenshot.
71
87
 
72
88
  Application consent is enforced before AX reads, window-title reads, screenshots, activation, and input through the CUA tool. The first selection asks the human for session access, denial, or saved access when profile persistence is available. Session access is stored by conversation and application identity and survives DSH restart and Computer Use disablement. Other conversations require their own consent. Application identities use the case-sensitive bundle identifier, so versions and copies at different paths share consent and revocation regardless of signing. A valid running bundle identifier remains sufficient when the operating system cannot supply an executable path. Available paths are still checked against the selected target; process-instance and window ownership checks remain mandatory. Without a bundle identifier, consent requires the canonical executable path. Restarting or updating at that path retains consent; moving or copying the executable to another path requires a new decision. This policy does not authenticate the publisher: another program declaring the same identifier, or replacing an approved executable at the same fallback path, inherits consent. Denial prevents repeated prompts in that session. Revocation reads and removes saved access in the same queue as grant creation, invalidates each affected Agent connection, and awaits its native cleanup without replaying input. Every app handle on those Agent connections becomes unusable; other applications need fresh selection, but their session and saved grants remain reusable. See the [application consent decision](../../../.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.md). These checks govern CUA calls; persistent PTC and separately authorized shell tools are not an operating-system sandbox.
73
89
 
74
- Raw `getState` exposes validated state fields and the complete text for the requested disclosure options. The native helper also returns structured element diagnostics for native checks; the tool projection omits that duplicate array from model text and structured tool values.
90
+ Raw `getState` exposes validated state fields and the complete text for the requested disclosure options. The native helper also returns structured element diagnostics with opaque string `elementId` values for native checks; the tool projection omits that duplicate array from model text and structured tool values.
75
91
 
76
- AX observations show menu-bar entries by default. Use `app.getAXState({ includeMenus: true })` to expand menu contents and `app.getAXState({ details: true })` for identifiers, global-point bounds and complete action names. These options also apply to raw `getState` and combined AX/screenshot observations. Every observation replaces that scope's indices; obtain fresh indices from the expanded output. Details do not bypass the configured node and depth limits. Truncation identifies the reached limit.
92
+ AX observations show menu-bar entries by default. Use `app.getAccessibilityTree({ includeMenus: true })` to expand menu contents and `app.getAccessibilityTree({ details: true })` for identifiers, global-point bounds and complete action names. These options also apply to raw `getState`. Every read replaces the application's current snapshot; use element IDs from that read. Pass `maxElements` or `maxDepth` to override either configured limit for one read. Truncation identifies the reached limit.
77
93
 
78
- Values and placeholders are separate. Named, stateful or actionable groups and scroll areas retain their own rows. Repeated names, descriptions and help text on one node are shown once. Values, placeholders and URLs keep their distinct meanings even when their text matches; body text shared by distinct nodes remains intact. Writability is explicit on relevant controls, including `AXIncrementor`; `unknown` means the system did not report it. Missing optional state fields do not imply false. UTF-16 ranges outside the observed value are marked unavailable. Custom action aliases omit AppKit target/selector diagnostics; ambiguous aliases require a complete name from `details:true`.
94
+ Values and placeholders are separate. Named, stateful or actionable groups and scroll areas retain their own rows. Repeated names, descriptions and help text on one node are shown once. Values, placeholders and URLs keep their distinct meanings even when their text matches; body text shared by distinct nodes remains intact. Writability is explicit on relevant controls, including `AXIncrementor`; `unknown` means the system did not report it. Missing optional state fields do not imply false. During tree observation, an AXValue or AXSelectedTextRange writability probe returning -25200 leaves that field unknown and preserves the remaining nodes. Actual writes remain strict; permission, communication and invalid-element errors still propagate. UTF-16 ranges outside the observed value are marked unavailable. Compact tree labels can omit AppKit target and selector details. Pass the exact name returned by `listActions(elementId)` to `performAction`.
79
95
 
80
- Observations emit by default and also return their value. In persistent PTC, use `void await app.getAXState()` for one emitted copy, or `emit:false` and one explicit print. Capturing a screenshot after an emitted tree can fail independently; retain the successful AX output.
96
+ Observations emit by default and also return their value. In persistent PTC, use `void await app.getAccessibilityTree()` for one emitted copy, or `emit:false` and one explicit print. Capturing a screenshot after an emitted tree can fail independently; retain the successful AX output.
81
97
 
82
98
  Screenshot diagnostics use `[cua-screenshot]` records keyed by the tool request ID. Native phase records return over the originating Agent connection and cover window/AX geometry checks, system capture, PNG/Base64 encoding and response writing. Host debug logs add dispatch, reply and cleanup timings. Each request retains at most 8192 bytes of complete diagnostic records, with explicit truncation; screenshot failures append these records to the error for Session export. Records contain stages, timings and error codes, not window titles, AX text or image bytes.
83
99
 
@@ -91,14 +107,19 @@ To run through persistent PTC, add `--patch packages/experimental/cua-macos/exam
91
107
  import { connectCua } from '@deepseek-ai/dsh-cua/bridge'
92
108
  const cua = connectCua(() => tools.cua, text => console.log(text))
93
109
  const app = await cua.getApp("com.apple.TextEdit")
94
- void await app.getScreenshot()
110
+ const windows = await app.getWindows()
111
+ if (windows.length === 0) throw new Error("Open a TextEdit window first")
112
+ const window = await app.getWindow(windows[0].windowId)
113
+ void await window.getScreenshot()
95
114
  ```
96
115
 
97
- Use app methods directly for single-window tasks. `getApp` reads initial AX state, so a screenshot needs no window enumeration or additional AX read. Use `app.getWindows()` and `app.getWindow(windowId)` when the task needs a particular window or default selection is ambiguous; choose by the observed title and bounds.
116
+ An independent floating panel has its own window ID. Select it from `app.getWindows()` to capture or click the panel.
98
117
 
99
- `getWindow` emits its initial AX tree. An owned window without an AX surface returns an empty snapshot and remains available for screenshots and coordinate input. Window objects retain separate snapshots, so reading one window does not invalidate another. Re-reading the same scope invalidates old indices. An AX node removed during observation rejects the read with `STALE_SNAPSHOT` and publishes no partial tree; read again without repeating the preceding input. App-scoped screenshots and coordinates reuse the latest AX observation: its focused on-screen, nonzero-size window, otherwise the sole such window. They do not query current focus again; read `app.getAXState()` to follow a focus change. Explicit window objects retain their selected target. Keyboard methods stay on `app` because input follows the app’s current focus.
118
+ Use the `cua` API to discover and select the application (`listApps`, `getState`, `findApps`, `getApp`); use the selected app to activate it, enumerate its windows, and select one. Window selection is required for screenshots and pointer input, not only when several windows exist. `getApp` and `app.getWindow(windowId)` return without reading AX; choose the window from `getWindows()` by its title and bounds.
100
119
 
101
- `hover(target)` first reveals advertised AX alternate UI without moving the system cursor. Entering a revealed descendant preserves its parent UI, and leaving restores the default UI. When unavailable, hover moves the visible system cursor to an AX element center or window-local point without pressing a button, including over disabled controls. It requires the selected app to be foreground and the target point unobscured; otherwise `FOREGROUND_REQUIRED` requires explicit activation or raising and a fresh observation. Completion, AX reads and screenshots do not restore the previous point. Wait for the application’s hover delay, observe the revealed content, then hover into it and click; successive calls can visit intermediate points. Other input or application changes can interrupt the interaction. [Continuous-interaction rationale](../../../.agents/notes/implemented/architecture/2026-10-13-cua-continuous-hover.md).
120
+ `getWindow` sends no native request; screenshots and pointer input reject a closed or foreign window. Read `getAccessibilityTree()` explicitly before element operations. Screenshots do not require a successful AX read. All coordinate input, including relative coordinates, requires a successful AX observation covering the window or a screenshot; absolute pixel coordinates require a delivered screenshot. An owned window without an AX surface returns an empty snapshot when explicitly observed. Within one API instance, the facade keeps one current accessibility snapshot per application handle, shared by every app and window object for that handle, including repeated selections. Reading any window replaces that snapshot; element operations use the latest snapshot ID, so use IDs from the latest tree and do not assume an older ID still identifies the same control. A failed read publishes no partial snapshot; read again without repeating the preceding input. Window objects own screenshots and pointer input. Keyboard methods stay on `app` because input follows the app’s current focus.
121
+
122
+ Ordinary `mouseMove(target)` defaults to AX hover. Without both `AXShowAlternateUI` and `AXShowDefaultUI` on an enabled target or ancestor, it returns `HOVER_UNAVAILABLE` without prompting. Only when the task requires physical movement should the main task call `win.mouseMove(target, { useSystemPointer: true })`. On approval the Host rechecks the window and point and dispatches this move. Pass `useSystemPointer: true` on every physical move. Later explicit moves reuse the decision until the session closes or DSH restarts. Omitting the option or passing `false` uses AX. Refusal sends no input.
102
123
 
103
124
  `paste(text, { format: "text" | "html" })` temporarily publishes the requested clipboard format and invokes the application’s enabled standard Paste menu command. It keeps the temporary format until a pasteboard reader requests its data, then restores every captured original format while it still owns the clipboard, preserving a later copy. Paste operations share a cross-process lock and cancellation joins helper cleanup. A forced process exit can prevent restoration. Unsupported Paste commands fail without typing fallback; `typeText` remains separate key-event input.
104
125
 
@@ -106,13 +127,15 @@ Use app methods directly for single-window tasks. `getApp` reads initial AX stat
106
127
 
107
128
  Screenshot capture requires an attachment store and a resolved model route declaring image input. The shared attachment admission validates canonical Base64, the raster and its configured capacity, then stores the image once. Native data remains available to programmatic callers; model content carries coordinate metadata and the durable image reference. Response byte limits apply before parsing.
108
129
 
109
- Absolute coordinates for hover, click, scroll and drag use pixels in the latest returned screenshot. Pass `relative: true` for integer thousandths of window width and height, from 0 to 999; omission or `false` uses pixels. Relative input can use a successful AX read or screenshot, while absolute input requires a screenshot. The Host maps the delivered image to native points, including attachment and model-request resizing. A value of 999 remains inside the window edge. Coordinate options do not apply to AX element indices. A width or height change since the latest observation returns `WINDOW_CHANGED` before input, including raw calls without a snapshot ID; read fresh AX or take a new screenshot before proceeding. Window movement alone preserves window-local coordinates. Capture and pointer input reject known disagreement between the selected window’s accessibility frame and displayed bounds with `WINDOW_CHANGED`, before posting pointer events. This includes Stage Manager transformations when AX retains the full window frame. Read fresh state; any activation or window raise remains explicit. Advertised AX actions remain available. Missing AX geometry retains the AXless path and does not establish that a transformed window can be operated.
130
+ Use `win.click(id)` to send mouse events at the center of an element, or `win.click([x, y])` to click a point. For both `click(id)` and `mouseMove(id)`, the element must belong to the selected window. Prefer `invoke(id)` for a background control’s default action. For mouse input, activate the application and observe the target first. Mouse clicks on system Open and Save sheets are routed to their verified panel-service window. On macOS 14.2 and later, a window screenshot excludes its attached window group so the image uses the selected window’s bounds.
131
+
132
+ Absolute coordinates for `mouseMove`, `click`, and `drag` use `[x, y]` tuples; `scroll` uses named `x` and `y`. All use pixels in the latest returned screenshot. Pass `relative: true` for integer thousandths of window width and height, from 0 to 999; omission or `false` uses pixels. Relative input can use a successful AX read or screenshot, while absolute input requires a screenshot. The Host maps the delivered image to native points, including attachment and model-request resizing. A value of 999 remains inside the window edge. Coordinate options do not apply to element IDs. A width or height change since the latest observation returns `WINDOW_CHANGED` before input, including raw coordinate calls without a snapshot ID; read fresh AX or take a new screenshot before proceeding. Window movement alone preserves window-local coordinates. Capture and pointer input reject known disagreement between the selected window’s accessibility frame and displayed bounds with `WINDOW_CHANGED`, before posting pointer events. This includes Stage Manager transformations when AX retains the full window frame. Read fresh state; any activation or window raise remains explicit. Advertised AX actions remain available. Missing AX geometry retains the AXless path and does not establish that a transformed window can be operated.
110
133
 
111
134
  Retained objects resolve the current tool binding on every call. The external runtime owns persistence, imports, cancellation and completion-value serialization. Screenshots return encoded bytes with their MIME type, pixel dimensions and window ID; the standard tool pipeline admits the image attachment exactly once. `emit: false` suppresses facade output only. A host that uses another dispatcher can build objects through `createCuaApi` from `./api`.
112
135
 
113
- AppKit sheets are selectable by their window IDs and remain visible in their document window’s tree. Keyboard input follows a system Open/Save panel only when the application’s focused sheet identifies one Apple panel-service window and that service still focuses the same panel. Unrelated or ambiguous remote recipients fail before dispatch.
136
+ An AppKit sheet can appear in its document window’s tree with a separate window ID. For pointer input on the sheet, use `app.getWindow(sheetId)` and an element ID from that window’s tree. Keyboard input follows a system Open/Save panel only when the application’s focused sheet identifies one Apple panel-service window and that service still focuses the same panel. Unrelated or ambiguous remote recipients fail before dispatch.
114
137
 
115
- Indexed clicks support a single right/middle click or a single/double left click. Click and drag options accept a button and modifier keys; a modified click uses pointer input. Drags hold that button and carry its modifiers through every movement and release. AX text marks disabled controls, `valueWritable=true`, and available `selected` state. The helper uses an advertised accessibility action for unmodified clicks when available; pointer delivery resolves the element’s current bounds and verifies its owning window. Actions report dispatch, not application success. A visual pointer or click marker can be rendered independently of input; AX actions do not require moving the system cursor. Window-directed mouse input can still change application focus, selection or dialogs. Read state after input. `STALE_APP` requires acquiring the application again; `STALE_SNAPSHOT` requires a fresh AX read. Request timeout/cancellation waits for native acknowledgement and preserves usable handles. Missing cleanup acknowledgement closes that Agent’s connection. A screenshot RPC still awaiting its system callback occupies the single capture slot; further screenshots return `CAPTURE_BUSY`, while AX and input remain available. A late capture result is discarded, and its real completion releases that slot. Only an unresponsive operation or failed cleanup can invalidate the shared process and all its handles. Never retry uncertain input automatically. Closing an Agent releases its connection and AX references, without undoing changes to applications.
138
+ Element clicks use opaque element IDs from the current snapshot. Left clicks support a single or double click; right and middle clicks support one. Click and drag options accept a button and modifier names. Drags hold that button and carry its modifiers through every movement and release. AX text marks disabled controls, `valueWritable=true`, and available `selected` state. Actions report dispatch, not application success. A visual pointer or click marker can be rendered independently of input; AX actions do not require moving the system cursor. Window-directed mouse input can still change application focus, selection or dialogs. Read state after input. `STALE_APP` requires acquiring the application again; `STALE_SNAPSHOT` requires a fresh AX read. Request timeout/cancellation waits for native acknowledgement and preserves usable handles. Missing cleanup acknowledgement closes that Agent’s connection. A screenshot RPC still awaiting its system callback occupies the single capture slot; further screenshots return `CAPTURE_BUSY`, while AX and input remain available. A late capture result is discarded, and its real completion releases that slot. Only an unresponsive operation or failed cleanup can invalidate the shared process and all its handles. Never retry uncertain input automatically. Closing an Agent releases its connection and AX references, without undoing changes to applications.
116
139
 
117
140
  -----
118
141
 
@@ -122,13 +145,13 @@ Indexed clicks support a single right/middle click or a single/double left click
122
145
  <details>
123
146
  <summary>Dispatch and resource ownership</summary>
124
147
 
125
- The object facade translates app/window methods into JSON operations. The bridge routes them through the normal tool pipeline, preserving authorization, nested logs and image admission. Agents and Hosts belonging to one user share a helper for the same native build. Each connection retains independent handles, snapshots, configuration and cancellation state; desktop operations use a global queue. The managed launcher transfers service lifetime to these connections, and the helper exits after the last connection finishes cleanup. Process locks cover both build identity and the real executable path, preventing duplicate startup and path collisions after a running executable is replaced. Native handles bind a process identity to its kernel start time, and snapshots retain native AX references only inside that helper. The same operation protocol can serve a future platform provider without adding platform logic to the object API.
148
+ The object facade translates app/window methods into JSON operations. Within one API instance it keeps one current accessibility snapshot per application handle, while the native helper retains its own application and optional-window snapshots behind it. The bridge routes operations through the normal tool pipeline, preserving authorization, nested logs and image admission. Agents and Hosts belonging to one user share a helper for the same native build. Each connection retains independent handles, snapshots, configuration and cancellation state; desktop operations use a global queue. The managed launcher transfers service lifetime to these connections, and the helper exits after the last connection finishes cleanup. Process locks cover both build identity and the real executable path, preventing duplicate startup and path collisions after a running executable is replaced. Native handles bind a process identity to its kernel start time, and snapshots retain native AX references only inside that helper.
126
149
 
127
150
  [Zod operation definitions](../cua/src/protocol.ts) own model parameters and separately declare Host resource fields and private continuations. Public data types derive from those definitions. `pnpm exec tsx packages/experimental/cua-macos/native/generate-protocol.ts` generates the native JSON Schema and Swift field/enum admission tables; the package test lane checks freshness and rejects Host-only fields at model admission. Native target validation, resource limits and operation behavior remain enforced at execution.
128
151
 
129
- [AX text formatting](native/Sources/CuaNative/AXTextFormatter.swift) consumes already-read node data. The AX walker alone assigns indices and retains native references, so formatting owns no second snapshot or target registry.
152
+ [AX text formatting](native/Sources/CuaNative/AXTextFormatter.swift) consumes already-read node data. The AX walker alone assigns opaque element IDs and retains native references, so formatting owns no second snapshot or target registry.
130
153
 
131
- Native request IDs match the logged tool call IDs. Successful activation and window-raise results retain foreground request details and immediate before/after focus observations in the raw tool result. An already-frontmost app skips redundant activation; a window raise is skipped only when the exact window is also focused, main, unminimized and first among visible normal windows. Unknown observations retain the requested operation. These observations do not establish who caused an unrelated focus change or verify completion.
154
+ Native request IDs match the logged tool call IDs. Successful activation results retain foreground request details and immediate before/after focus observations in the raw tool result. An already-frontmost app skips redundant activation. Unknown observations retain the requested operation. These observations do not establish who caused an unrelated focus change or verify completion.
132
155
 
133
156
  Node roles and metadata share one AX batch per observation. Group and scroll-area nodes contribute only children; unused metadata errors do not reject them. Unsupported batching retains individual attribute reads. AX traversal visits each native identity once, including proxy references that compare equal, so cyclic accessibility children cannot repeat whole subtrees. The [source manifest](native/upstream.json) records adapted upstream behavior and local Swift modules. AppKit runs on the main event loop; a separate serial executor owns AX references and synchronous input. A request retains exclusive execution through capture and response delivery. No invariant companion is published: there is no independently projected registry or durable evaluator state to reconcile. Protocol validation, target identity and resource cleanup are checked by their owning implementations.
134
157
 
@@ -157,53 +180,62 @@ The plugin contributes one `cua` operation tool with method-specific parameters.
157
180
  ##### Operation guidance (verbatim)
158
181
 
159
182
  ```markdown
160
- Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps or cua.getState to discover running applications, and findApps(query) to search installed applications by name, bundle ID or absolute path. getApp requests user consent before launching a stopped application in the background, or before observations and input on a running one. If access is denied or revoked, stop using that application until the user explicitly permits another authorization request. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. Raw getState returns AX text and snapshotId. Menu-bar entries are shown by default; includeMenus=true expands their contents. details=true adds identifiers, global-point bounds and complete action names. Both options also apply to getAXState. Each read replaces the indices for that app/window scope. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are pixels in the latest returned screenshot, measured from its top-left. Set relative:true for integer thousandths from 0 to 999 instead. The Host converts both modes to native window points; AX detail bounds remain global macOS points. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
161
-
162
- Initialize the object API once in persistent PTC. Reuse its variables in later calls; put temporary declarations inside { ... } blocks to avoid redeclaration errors. All object methods are async. getAXState returns text, not a structured state object. Its text separates values and placeholders, and reports subroles, help, URLs, disabled/focused/selected/expanded state and UTF-16 selection ranges when available. Writability is true, false or unknown on relevant controls; omitted state fields do not assert false. selection=unavailable means the reported range exceeds the observed text. Compact output omits ordinary Press/Pick (click), ShowMenu (right click) and ScrollToVisible actions; details=true lists every available action. Use the displayed action name with performSecondaryAction; ambiguous aliases require the complete name from details=true. Truncation reports the reached provider limit. Narrow the window or increase that provider limit before reading again.
163
-
164
- Use app methods directly for single-window tasks; getApp already reads initial AX state. Call app.getWindows() and await app.getWindow(windowId) only to target a particular window or resolve ambiguous window selection. App screenshots and coordinate actions use the latest AX observation: its focused on-screen, nonzero-size window, otherwise the sole such window. They do not refresh focus automatically. Read app.getAXState() to follow changed focus.
165
-
166
- CuaScreenshot contains { bytes: Uint8Array, mimeType: "image/png" | "image/jpeg" | "image/webp" | "image/gif", pixelWidth: number, pixelHeight: number, windowId: number }. Use mimeType to choose a file extension or declare an image; do not assume PNG.
167
-
168
- Object API signatures (options default to { emit: true }):
169
-
170
- - cua.getCapabilities(options?) -> { platform, permissions, sessionLocked, methods, coordinateSpace, browser }
171
- - cua.listApps(options?) -> Array<{ id, displayName, isRunning: true, pid }>
172
- - cua.findApps(query: string, options?) -> Array<{ id, displayName, path, isRunning, pid? }>
173
- - cua.getState(options?) -> { apps, browsers: [] }
174
- - cua.getApp(selector: string) -> App
175
- - app.activate() -> { dispatched: true, verified: false }
176
- - app.getWindows(options?) -> Array<{ windowId, pid, appName, title, bounds, isOnScreen, onCurrentSpace }>
177
- - app.getWindow(windowId: number) -> Window
178
- - app.typeText(text: string), app.pressKey(chord: string) -> action result
179
- - app.paste(text: string, options?: { format?: "text" | "html" }) -> action result
180
-
181
- App and Window methods:
182
-
183
- - show() -> { dispatched: true, verified: false, windowId: number | null, foregroundChanged: boolean | null }
184
- - getAXState(options?: { emit?: boolean, disableDiffing?: boolean, includeMenus?: boolean, details?: boolean }) -> string
185
- - getScreenshot(options?: { emit?: boolean }) -> CuaScreenshot
186
- - getAXStateAndScreenshot(options?) -> { state: string, screenshot: CuaScreenshot }
187
- - hover(target: number | [x, y], options?: { relative?: boolean }) -> { dispatched, verified, hoverRoute, systemPointerMoved }
188
- - click(target: number | [x, y], options?: { mouseButton?: "left" | "right" | "middle", clickCount?: 1 | 2, holdMs?: number, modifiers?: Array<"command" | "control" | "option" | "shift" | "fn">, relative?: boolean }) -> action result
189
- - setValue(elementIndex: number, value: string) -> action result
190
- - selectText(elementIndex: number, text: string, options?: { prefix?: string, suffix?: string, selectionType?: "text" | "cursor_before" | "cursor_after" }) -> action result
191
- - performSecondaryAction(elementIndex: number, action: string) -> action result
192
- - scroll({ x, y, deltaLinesX, deltaLinesY, relative? }) -> action result
193
- - drag(from: [x, y], to: [x, y], options?: { mouseButton?: "left" | "right" | "middle", modifiers?: Array<"command" | "control" | "option" | "shift" | "fn">, relative?: boolean }) -> action result
194
-
195
- Raw operation tools use { method, params }, including params: {} for parameterless methods. getApp selects without activating the app. Keep applications in the background for observation and routine input. Do not activate or raise an app during initialization, before each action, or merely because another app is foreground. Prefer advertised AX actions. If a window is hidden or minimized, try app.show() or window.show() and read fresh state. show requests unhide and unminimize without activation, raising or switching Spaces; it chooses the explicit, focused, main or sole AX window. A null windowId means no window was selected for restoration. foregroundChanged compares the front app and top window before and after the request; null means unavailable, and delayed focus changes remain possible. If restoration is unsupported or the window remains unusable, including another Space or Stage Manager, explain the needed foreground step, use app.activate() or an advertised raise action, and read fresh state. Completing the task takes priority when foreground access is necessary. Avoid repeated activation when the user switches away; continue background-capable steps and report what requires foreground access. Applications may still change focus in response to input.
196
-
197
- Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Absolute coordinate input requires a fresh screenshot of that window. Relative input can use a successful AX observation or screenshot. SCREENSHOT_REQUIRED means capture a fresh screenshot before using pixel coordinates. For click and hover, coordinate targets must be [x, y], for example hover([650, 400], {relative:true}). A bare number such as hover(650) is always an AX element index. The relative option applies only to coordinate targets, not AX indices. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to grant the required macOS system permissions for macOS Accessibility or Screen Recording. After granting access, observe fresh state before continuing; do not automatically replay prior input.
198
-
199
- Hover first uses advertised AX alternate UI without moving the system cursor, preserving revealed descendants until the pointer target leaves their region. When unavailable, it moves the visible system cursor without pressing a button; that route requires the target point to be unobscured in the foreground app. FOREGROUND_REQUIRED requires explicit activation or raising and a fresh observation before retrying. Returning, AX reads and screenshots do not restore the previous point. Wait for the application's hover delay, observe, then hover into the revealed controls before clicking. Use successive hover calls for intermediate points. The result reports hoverRoute and systemPointerMoved. Other input or application changes can interrupt the interaction; app objects do not reserve applications between calls.
200
-
201
- Use selectText only on controls advertising selectionWritable. It selects a unique literal occurrence, or places the cursor before/after it; prefix and suffix must be immediately adjacent. AMBIGUOUS_TEXT requires more context; TEXT_NOT_FOUND or SELECTION_CHANGED requires a fresh observation before deciding on input. Left clicks support single/double clicks; right and middle clicks support single clicks. Modified clicks use pointer input. Optional holdMs is a nonnegative integer in milliseconds for each press, including each press of a double-click. Zero releases without an additional hold. The configured maximum is stated below. It uses pointer input even for AX targets; omission preserves ordinary clicks. The operation deadline still applies, and cancellation or timeout releases the held button. Keypad names such as KP_0 and KP_Enter retain their physical key identity. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Paste delivers plain text or HTML through the application's enabled standard Paste menu command. Paste follows the target app's focused text input. Foreground requirements vary by application; for example, ChatGPT/Codex was observed to accept background Paste without reading clipboard data. PASTE_NOT_CONSUMED means the command was dispatched but no clipboard read was observed before the configured deadline. Inspect the target contents, explicitly activate the app and focus the intended input, then observe again before deciding whether to retry; a delayed paste may still run. Original clipboard formats are restored if the clipboard still belongs to this paste; a later copy is preserved. Unsupported Paste commands fail without typing fallback. CLIPBOARD_CHANGED means another writer intervened; observe before deciding on further input. Browser applications can be controlled with getApp like other native apps; browser tabs are unavailable. Installed-app search covers standard macOS application directories; use an absolute application path for another location. Launch does not restore a window or document, and cancellation may leave an application already started. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global. In persistent PTC, use the host-provided initialization snippet below once. Its installation-owned file URL works independently of the project directory. Retain app objects between calls. Observation methods emit by default and also return their value. Use void await app.getAXState() to avoid a duplicate REPL completion, or pass emit:false and print the returned text once. End screenshot calls with void await app.getScreenshot() or void await app.getAXStateAndScreenshot() so image bytes are not serialized as the program completion value.
183
+ macOS: accessibility tree operations are available at app or window level. Simulated keyboard input is available at app level. Simulated mouse input is available at window level. For pointer input on a sheet, select its window ID with app.getWindow(id). For menu items outside a window, use app.invoke(id). findApps searches /Applications, /System/Applications, and ~/Applications. Use the Computer Use permission page to grant the helper application Accessibility and Screen Recording permissions in System Settings. Native window and AX bounds use macOS points, while delivered screenshots carry their own pixel dimensions. show does not switch Spaces, and Stage Manager can present reduced window geometry; explain any required foreground step before calling app.activate() or performing an advertised AXRaise action.
184
+
185
+ Use cua to observe and operate native applications. Always use cua.getCapabilities() to confirm whether a specific API is available on the current platform; its reported capabilities take precedence over availability statements in these instructions. All methods are async; observation options default to { emit: true }, and emit: false suppresses sink output only, leaving returned values unchanged. Actions report dispatch evidence, not verified completion. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI actions and ask the user to unlock the desktop session; null or unknown means verify a usable unlocked desktop session first. cua.getApp requests user consent; if access is denied or revoked, stop using that application until the user explicitly permits another request, and never bypass a refusal or switch tools to avoid it. Never replay uncertain input without a fresh observation.
186
+
187
+ In persistent PTC, use the host-provided bridge initialization once and retain the returned objects; cua comes from that bridge and is not a preinstalled global. Use void await for observations that emit, and never stringify screenshot image bytes.
188
+
189
+ Signatures show resolved return values after await. ActionResult = { dispatched: boolean, foregroundChanged?: boolean | null }; foregroundChanged is omitted when not sampled and null when the sample is unavailable. JsonValue = string | number | boolean | null | JsonValue[] | { [key: string]: JsonValue }; numbers must be finite. App is the application object with readonly app.appHandle: string (opaque) and the methods listed below for app level. Window is the window object with readonly window.windowId: number and the methods listed below for window level. Neither object carries pid.
190
+
191
+ Setup and discovery:
192
+ - cua.getCapabilities(options?: { emit?: boolean }) -> { platform: string, permissions: { accessibility: boolean, screenRecording: boolean }, appLevelAccessibility: boolean, systemPointer: boolean, keyboard: ('app' | 'window')[], sessionLocked: boolean | null, methods: string[], coordinateSpace: 'screenshot-pixels', browser: false } — inspect reported permissions, sessionLocked, methods, and keyboard scopes without requesting permissions.
193
+ - cua.listApps(options?: { emit?: boolean }) -> { id: string, displayName: string, isRunning: true, pid: number }[] — enumerate running applications.
194
+ - cua.findApps(query: string, options?: { emit?: boolean }) -> ({ id: string, displayName: string, path: string } & ({ isRunning: true, pid: number } | { isRunning: false }))[] — search installed applications, including stopped matches; only running matches carry pid.
195
+ - cua.getState(options?: { emit?: boolean }) -> { apps: { id: string, displayName: string, isRunning: true, pid: number }[], browsers: [] } — summarize running applications; this is not an accessibility observation.
196
+ - cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and return without reading AX or activating the app.
197
+ - app.getWindows(options?: { emit?: boolean }) -> { windowId: number, appName: string, title: string, bounds: { x: number, y: number, width: number, height: number }, isOnScreen: boolean, onCurrentSpace: boolean | null }[] — enumerate windows with global native bounds; null onCurrentSpace means unknown Space membership.
198
+ - app.getWindow(windowId: number) -> Window — select a window by an ID from getWindows without reading AX. Providers reject closed or foreign windows when operating on them.
199
+ - app.activate() -> ActionResult — request explicit foreground activation; use only when foreground access is required.
200
+ - app.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide the app and restore its selected, focused, main, or sole window without activation.
201
+ - window.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide and restore that explicit window without activation.
202
+
203
+ ## Accessibility tree
204
+ Available at app or window level depending on the platform.
205
+
206
+ Read getAccessibilityTree explicitly before using element ids. Selection and screenshots do not read or replace the current AX snapshot. Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing.
207
+ - getAccessibilityTree(options?: { emit?: boolean, includeMenus?: boolean, details?: boolean, maxElements?: number, maxDepth?: number }) -> string — set the node count or depth for one read. Omitted limits use the provider settings. includeMenus expands menu contents and details adds identifiers, native bounds, and complete action names. macOS accepts maxElements up to 100000 and maxDepth up to 512. If the tree is truncated, select a narrower window or raise the reported limit on the next read. In persistent PTC, capture a large tree with const tree = await win.getAccessibilityTree({ emit: false }). Filter its lines in memory, or save it with await tools.write({ file_path: path, content: tree }) to a new workspace file. Search its contents with tools.grep({ path, pattern }) and use tools.read around a matching line. Print only the matches you need. Use element ids from the file only while that observation is current.
208
+ - listActions(elementId: string) -> string[] — list the actions an element advertises.
209
+ - invoke(elementId: string) -> ActionResult — invoke the element's default action; dispatched is false when it has none.
210
+ - performAction(elementId: string, action: string, argument?: JsonValue) -> ActionResult — perform an observed action with an optional direct JSON argument.
211
+ - setValue(elementId: string, value: JsonValue) -> ActionResult — write a control value.
212
+ - selectText(elementId: string, text: string, options?: { prefix?: string, suffix?: string, selectionType?: 'text' | 'cursor_before' | 'cursor_after' }) -> ActionResult — select one literal occurrence or place the cursor beside it, using adjacent context to disambiguate.
213
+
214
+ ## Simulated mouse
215
+ Use an explicit window for screenshots and pointer input. Before coordinate input, including relative coordinates, obtain a successful getAccessibilityTree observation covering that window or take its screenshot. Absolute pixel coordinates require a returned screenshot.
216
+ - getScreenshot(options?: { emit?: boolean }) -> { bytes: Uint8Array, mimeType: 'image/png' | 'image/jpeg' | 'image/webp' | 'image/gif', pixelWidth: number, pixelHeight: number, windowId: number } — capture the selected window as one encoded image; bytes is a JavaScript byte array, and mimeType identifies its encoding, so do not assume PNG.
217
+ - mouseMove(target: string | [number, number], options?: { relative?: boolean, useSystemPointer?: boolean }) -> { dispatched: boolean, foregroundChanged?: boolean | null, hoverRoute: 'accessibility' | 'system-pointer', systemPointerMoved: boolean } — request a mouse move to the target without pressing a button; inspect hoverRoute and systemPointerMoved, and expect no automatic pointer restoration. Default hover requires both AXShowAlternateUI and AXShowDefaultUI on an enabled element or ancestor. Without that route, HOVER_UNAVAILABLE is returned before sending input. Physical movement requires an unobscured foreground target. Explain any necessary activation and observe the target again.
218
+ - click(target: string | [number, number], options?: { mouseButton?: 'left' | 'right' | 'middle', clickCount?: 1 | 2, modifiers?: ('command' | 'control' | 'option' | 'shift' | 'fn')[], holdMs?: number, relative?: boolean }) -> ActionResult — click an element id or point, or perform a left-button double-click; holdMs controls the press duration of each press. click is a Window method: select a window with const win = await app.getWindow(windowId), then call win.click(target). For a background control’s default action, use app.invoke(id). For a mouse click on a background app, call app.activate() first and re-read the target. Both target forms send mouse-down and mouse-up events. click(id) uses the center of the full element bounds. If that center is outside the window, take a fresh win.getScreenshot() and use click([x, y]) on a visible part of the control in screenshot pixels. A background app can ignore the events even when dispatched is true. Verify the expected effect, such as a changed selection, before typing or retrying.
219
+ - scroll(options: { x: number, y: number, deltaLinesX: number, deltaLinesY: number, relative?: boolean }) -> ActionResult — scroll at a point; deltas are integers from -10 to 10 with at least one nonzero, and positive values reveal content above or left.
220
+ - drag(from: [number, number], to: [number, number], options?: { mouseButton?: 'left' | 'right' | 'middle', modifiers?: ('command' | 'control' | 'option' | 'shift' | 'fn')[], relative?: boolean }) -> ActionResult — drag between two points of one window.
221
+
222
+ A string target is an element id; coordinate targets are [x, y] and are pixels in the latest returned screenshot by default. With relative: true they are integer thousandths of the window width and height from 0 to 999, based on the current AX observation or screenshot. Left clicks support one or two presses; right and middle clicks support one. FOREGROUND_REQUIRED means explain the need, then explicitly activate the target and observe again; activation is never automatic. Read fresh state after actions, and treat WINDOW_CHANGED, STALE_SNAPSHOT, and SCREENSHOT_REQUIRED as requiring the stated fresh observation or capture before any retry.
223
+
224
+ ## Simulated keyboard
225
+ Available at app or window level depending on the platform.
226
+ - typeText(text: string) -> ActionResult — type literal text at the app's current focus; line endings send Return and tabs send Tab.
227
+ - paste(text: string, options?: { format?: 'text' | 'html' }) -> ActionResult — deliver text or HTML through the app's Paste command, restoring prior clipboard contents unless another writer changes them; a Paste failure is surfaced without substituting typeText, and PASTE_NOT_CONSUMED means inspect before any retry.
228
+ - pressKey(key: string, modifiers?: string[]) -> ActionResult — press and release one complete keystroke. For example pressKey('Tab', ['shift']) or pressKey('+'). Key and modifier names are platform-defined strings. Pass key and modifiers separately; do not pack them into a chord string.
229
+ - sendKeyEvent(key: string, direction: 'down' | 'up') -> ActionResult — send 'down' to hold a key across tool calls in one run_code, for example W down, D down, D up, W up. Holding a key on a physical keyboard can produce repeated key-down events after a delay. To send one such repeat for a non-modifier key this run already holds, send 'down' again. Each call posts one event. A control can filter repeats. Use the app's response to decide whether and when to repeat or release the key. When this run_code returns, throws, or the user pauses it, macOS releases only its remaining keys at their original recipients.
230
+
231
+ Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree maps to getState; getWindow has no raw method; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name.
232
+
233
+ Only if the task cannot be completed without moving the user’s pointer, call await win.mouseMove(target, { useSystemPointer: true }). Prefer an advertised accessibility action, a click on a visible control, or browser tools when possible. The question interrupts the user. When approval is disabled an explicit move can act immediately. Pass true on every physical move, even after approval. Omitting the option or passing false uses the default hover route without prompting. Approval or refusal lasts until this session closes or DSH restarts. A refusal sends no input. If the user dismisses the request, ask again only if they request it. Wait for an outstanding request to finish. Subagents cannot request physical movement.
202
234
  ```
203
235
 
204
236
  #### Token effect
205
237
 
206
- Enabling the plugin adds a fixed schema and guidance. AX observations, operation results and admitted images add model input. Retained objects are not serialized into prompts by this package.
238
+ Enabling the plugin adds a fixed schema and guidance. AX observations, operation results and admitted images add model input. Retained objects are not serialized into prompts by this package. A higher default or per-read override can include more AX text in model input when the window has more nodes to visit. Each visible auxiliary window, such as a status item, menu or tooltip, adds one window record to `getWindows()` and raw `getState` output; it does not automatically add a screenshot.
207
239
 
208
240
  #### KV Cache effect
209
241
 
@@ -211,6 +243,8 @@ Tool results append to Session history. Changing plugin configuration or tool pr
211
243
 
212
244
  ## Known Limitations and Deferred Work
213
245
 
246
+ The native operation deadline (`timeoutMs`), per-AX-call timeout (`axTimeoutMs`), and response size limit (`maxResponseBytes`) still apply to larger tree reads.
247
+
214
248
  The queue orders individual native requests. Holding an app object does not reserve its application across a Session; [continuous interaction](../../../.agents/notes/implemented/architecture/2026-10-13-cua-continuous-hover.md) grants no cross-call ownership. Capture RPCs that never return occupy the single capture slot; creating new app objects does not reset that system connection.
215
249
 
216
250
  <a id="known-limitations-and-deferred-work"></a>
@@ -218,11 +252,12 @@ The queue orders individual native requests. Holding an app object does not rese
218
252
  - macOS 14 is the minimum supported version; settings navigation, drag acceptance, and permission refresh require qualification on each supported release. If macOS requires quitting and reopening the helper after a Screen Recording change, its preflight check can remain denied until that restart. The guide can be dismissed without terminating other connections.
219
253
  - The guide closes when its selected settings window leaves the visible-window list, including when a Space switch or minimization removes it. Request permission again after returning to reopen the guide.
220
254
  - Only the macOS native implementation is provided. Windows support is in development; this package remains in `experimental` until that support is complete, then moves to the `computer-use` group. Browser tabs, AX diffs, unassociated system panels and accessory-only apps are unsupported.
255
+ - Element IDs belong to the application's current accessibility snapshot; a later read replaces them. Screenshots and pointer input require an explicit window object.
221
256
  - Coordinates use the returned screenshot pixels, including the title bar; `relative: true` selects integers from 0 to 999. Native point conversion is Host-owned. Window movement, application changes and other desktop users can race observations and input.
222
257
  - macOS Stage Manager can reduce background window bounds and screenshots to thumbnails. Full-window observation and coordinate input in this state are not guaranteed; model guidance requires explaining foreground needs and avoiding repeated focus reclamation.
223
258
  - Background input depends on private SkyLight symbols; macOS updates can make them unavailable. Validate each target OS version.
224
259
  - Observed failures include dispatched Unicode text not appearing in Blender and background Command+A not selecting all text in AppKit. Inspect application state after input; a successful tool result confirms dispatch only.
225
- - Source builds use ad-hoc signatures and may require authorization again after source changes. Release distribution needs a stable Developer ID signature and notarization; the build command accepts `--identity`, and installed packages preserve the distributed signature. The [permission page](../cua/README.md) offers helper authorization and target-application revocation.
260
+ - Source builds default to `ad-hoc` and can require new grants after binary changes. The [developer’s fixed identity](../../../.agents/skills/dsh-cua-macos-signing/SKILL.md) lives outside DSH home; deleting DSH home removes its selection, not the certificate or private key. Release distribution needs a stable Developer ID signature and notarization; the build command accepts `--identity`, and installed packages preserve the distributed signature. The [permission page](../cua/README.md) offers helper authorization and target-application revocation.
226
261
  - Release packaging does not automatically verify GUI startup, helper identity or exit cleanup. Release maintainers can run the [distribution smoke](native/README.md#build-and-service) on a logged-in Mac before publication; a successful package build does not establish those runtime results.
227
262
  - Saved permission rows show the name, path and icon of one approved copy, not an inventory of every installation sharing its identity. Removing that copy can leave its icon unavailable. Revocation still applies to every copy with the same identity.
228
263
  - If macOS does not expose a running application’s bundle identifier, the helper checks its reported bundle only when its executable matches the observed process path, then falls back to that path. Consent identity creation fails only when both the bundle identifier and actual executable path are unavailable; process-instance verification can still fail independently.