@deepseek-ai/dsh-experimental-cua-macos 0.0.0-stage → 0.2.1-alpha.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.i18n.yaml +52 -0
- package/README.md +225 -2
- package/README.zh.md +226 -0
- package/lib/index.js +573 -0
- package/lib/types/guidance.d.ts +4 -0
- package/lib/types/index.d.ts +19 -0
- package/lib/types/native-build.d.ts +47 -0
- package/lib/types/shared-native.d.ts +13 -0
- package/native/LICENSE.cua-driver +21 -0
- package/native/Package.swift +13 -0
- package/native/README.md +130 -0
- package/native/README.zh.md +131 -0
- package/native/Sources/CuaNative/AXAccess.swift +525 -0
- package/native/Sources/CuaNative/AXHover.swift +37 -0
- package/native/Sources/CuaNative/AXTextFormatter.swift +120 -0
- package/native/Sources/CuaNative/AXWorker.swift +489 -0
- package/native/Sources/CuaNative/ApplicationAuthorization.swift +69 -0
- package/native/Sources/CuaNative/ApplicationDiscovery.swift +92 -0
- package/native/Sources/CuaNative/ClipboardPaste.swift +138 -0
- package/native/Sources/CuaNative/KeyboardInput.swift +260 -0
- package/native/Sources/CuaNative/Main.swift +37 -0
- package/native/Sources/CuaNative/NativePlatform.swift +118 -0
- package/native/Sources/CuaNative/NativeProtocol.generated.swift +33 -0
- package/native/Sources/CuaNative/PIDEventPoster.swift +85 -0
- package/native/Sources/CuaNative/Protocol.swift +149 -0
- package/native/Sources/CuaNative/RequestAdmission.swift +58 -0
- package/native/Sources/CuaNative/RequestCoordinator.swift +251 -0
- package/native/Sources/CuaNative/RequestScheduler.swift +107 -0
- package/native/Sources/CuaNative/Runtime.swift +170 -0
- package/native/Sources/CuaNative/ScreenshotTrace.swift +46 -0
- package/native/Sources/CuaNative/SharedHelperProxy.swift +124 -0
- package/native/Sources/CuaNative/SharedHelperRuntime.swift +289 -0
- package/native/Sources/CuaNative/SharedHelperServer.swift +188 -0
- package/native/Sources/CuaNative/TextSelection.swift +38 -0
- package/native/Sources/CuaNative/WindowCapture.swift +165 -0
- package/native/Sources/CuaNative/WindowGeometry.swift +54 -0
- package/native/Sources/CuaNative/WindowPointerInput.swift +300 -0
- package/native/Sources/CuaNative/WindowSpaces.swift +133 -0
- package/native/protocol.schema.json +1030 -0
- package/native/tests/swift/AXFailureTests.swift +166 -0
- package/native/tests/swift/AXTextFormatterTests.swift +128 -0
- package/native/tests/swift/ApplicationDiscoveryTests.swift +59 -0
- package/native/tests/swift/ClipboardPasteTests.swift +136 -0
- package/native/tests/swift/ForegroundTests.swift +42 -0
- package/native/tests/swift/InputMappingTests.swift +34 -0
- package/native/tests/swift/PermissionIdentityTests.swift +24 -0
- package/native/tests/swift/PointerTimingTests.swift +10 -0
- package/native/tests/swift/ProtocolTests.swift +118 -0
- package/native/tests/swift/ScreenshotTraceTests.swift +23 -0
- package/native/tests/swift/SharedHelperTests.swift +90 -0
- package/native/tests/swift/SharedSocketTests.swift +57 -0
- package/native/tests/swift/SystemHoverTests.swift +27 -0
- package/native/tests/swift/TextSelectionTests.swift +47 -0
- package/native/tests/swift/WindowCaptureTests.swift +80 -0
- package/native/tests/swift/WindowGeometryTests.swift +105 -0
- package/native/tests/swift/WindowSpacesTests.swift +26 -0
- package/native/tests/test_swift.py +27 -0
- package/native/upstream.json +67 -0
- package/package.json +85 -3
package/README.i18n.yaml
ADDED
|
@@ -0,0 +1,52 @@
|
|
|
1
|
+
# Bilingual-pair consistency record for README.md (docs/i18n/README.md): per heading
|
|
2
|
+
# section, a hash of its English and Chinese blocks outside code blocks and generated regions.
|
|
3
|
+
# After editing either side, bring the other along and re-record with:
|
|
4
|
+
# pnpm run verify-translation-pairing --write packages/experimental/cua-macos/README.md
|
|
5
|
+
/:
|
|
6
|
+
en: a1503331d3f35516
|
|
7
|
+
zh: fcb8e5220ff732e1
|
|
8
|
+
/deepseek-ai-dsh-experimental-cua-macos:
|
|
9
|
+
en: 022ca213425e11ee
|
|
10
|
+
zh: d64631748ed91cc2
|
|
11
|
+
/deepseek-ai-dsh-experimental-cua-macos/summary:
|
|
12
|
+
en: 4549f02fc53854a5
|
|
13
|
+
zh: 6f74f1130b6623a8
|
|
14
|
+
/deepseek-ai-dsh-experimental-cua-macos/table-of-contents:
|
|
15
|
+
en: d152484eb41ac6b4
|
|
16
|
+
zh: 7b30b973a849247d
|
|
17
|
+
/deepseek-ai-dsh-experimental-cua-macos/use-this-package:
|
|
18
|
+
en: e2e1797ad6b73f25
|
|
19
|
+
zh: 35b7a06e471e433c
|
|
20
|
+
/deepseek-ai-dsh-experimental-cua-macos/use-this-package/connect-an-external-execution-environment:
|
|
21
|
+
en: e7909c9b43e44fbc
|
|
22
|
+
zh: 98949877d4ea3a86
|
|
23
|
+
/deepseek-ai-dsh-experimental-cua-macos/understand-the-implementation:
|
|
24
|
+
en: 8b9ec208ca057a4a
|
|
25
|
+
zh: e8dbb45891ad15b8
|
|
26
|
+
/deepseek-ai-dsh-experimental-cua-macos/further-exploration:
|
|
27
|
+
en: ff9fac614001a98f
|
|
28
|
+
zh: 329678941154aaea
|
|
29
|
+
/deepseek-ai-dsh-experimental-cua-macos/model-experience:
|
|
30
|
+
en: 215e7ba838619b7b
|
|
31
|
+
zh: a311da3843709f90
|
|
32
|
+
/deepseek-ai-dsh-experimental-cua-macos/model-experience/native-application-operations:
|
|
33
|
+
en: ab7cba4a42334ab9
|
|
34
|
+
zh: f77cd9d84624b6c2
|
|
35
|
+
/deepseek-ai-dsh-experimental-cua-macos/model-experience/native-application-operations/what-the-model-sees:
|
|
36
|
+
en: c221fa5ce647b1dd
|
|
37
|
+
zh: 9388c91a9f89e029
|
|
38
|
+
/deepseek-ai-dsh-experimental-cua-macos/model-experience/native-application-operations/what-the-model-sees/operation-guidance-verbatim:
|
|
39
|
+
en: 2dbc22ffd1f8a717
|
|
40
|
+
zh: d4458872e75305d9
|
|
41
|
+
/deepseek-ai-dsh-experimental-cua-macos/model-experience/native-application-operations/token-effect:
|
|
42
|
+
en: ab52b46d874fa8b9
|
|
43
|
+
zh: e87d3620ef77cb06
|
|
44
|
+
/deepseek-ai-dsh-experimental-cua-macos/model-experience/native-application-operations/kv-cache-effect:
|
|
45
|
+
en: a2c9acb5a4320dda
|
|
46
|
+
zh: e8d1586af60215c4
|
|
47
|
+
/deepseek-ai-dsh-experimental-cua-macos/known-limitations-and-deferred-work:
|
|
48
|
+
en: 9d8af4c39fe407a3
|
|
49
|
+
zh: afec49eedfd702cb
|
|
50
|
+
/deepseek-ai-dsh-experimental-cua-macos/known-limitations-and-deferred-work/dev-note:
|
|
51
|
+
en: b836f01bd0bceb4a
|
|
52
|
+
zh: 8122fa08b2aa95fa
|
package/README.md
CHANGED
|
@@ -1,3 +1,226 @@
|
|
|
1
|
-
|
|
1
|
+
---
|
|
2
|
+
description: "Inspect and operate native macOS applications through scoped app and window objects."
|
|
3
|
+
kind: "package-reference"
|
|
4
|
+
---
|
|
2
5
|
|
|
3
|
-
|
|
6
|
+
# @deepseek-ai/dsh-experimental-cua-macos
|
|
7
|
+
|
|
8
|
+
English | [中文](README.zh.md)
|
|
9
|
+
|
|
10
|
+
## Summary
|
|
11
|
+
|
|
12
|
+
Inspect macOS permissions, select running applications and windows, read accessibility trees, capture screenshots and send input. The object API can be retained by an external persistent execution environment. The Swift helper uses macOS frameworks directly and adapts selected MIT-licensed Cua Driver behavior. This package owns neither a PTC runtime nor variable persistence.
|
|
13
|
+
|
|
14
|
+
## Table of Contents
|
|
15
|
+
|
|
16
|
+
- [Use this package](#use-this-package)
|
|
17
|
+
- [Understand the implementation](#understand-the-implementation)
|
|
18
|
+
- [Further Exploration](#further-exploration)
|
|
19
|
+
- [Model Experience](#model-experience)
|
|
20
|
+
- [Known Limitations and Deferred Work](#known-limitations-and-deferred-work)
|
|
21
|
+
- [Dev Note](#dev-note)
|
|
22
|
+
|
|
23
|
+
-----
|
|
24
|
+
|
|
25
|
+
<a id="use-this-package"></a>
|
|
26
|
+
## Use this package
|
|
27
|
+
|
|
28
|
+
Enable this experimental provider through explicit source composition on macOS 14 or later. Windows support is in development. Shipped profiles do not mount it. Prepare a native helper and load the registry/provider overlay:
|
|
29
|
+
|
|
30
|
+
```sh
|
|
31
|
+
swift build --package-path packages/experimental/cua-macos/native --configuration release
|
|
32
|
+
cua_native_root="$PWD/packages/experimental/cua-macos/native"
|
|
33
|
+
export DSH_CUA_HELPER="$cua_native_root/.build/release/dsh-cua-native"
|
|
34
|
+
pnpm dsh headless --patch packages/experimental/cua-macos/examples/macos.cordis.yml --dump-config
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
The dump command checks profile composition without a model request. For a real task, replace `--dump-config` with a prompt and configure the profile’s model credentials. The overlay mounts the computer-use registry and this provider; remove an existing competing provider before using it. The responsible host/helper needs Accessibility permission for AX/input and Screen Recording permission for capture. `getCapabilities` reports permissions and session lock state without opening a system dialog. Locked or unavailable desktop sessions reject UI observations and input before dispatch; the user must unlock macOS before resuming. `listApps` and `cua.getState` list running applications. `findApps(query)` searches standard macOS application directories by name, bundle ID or absolute path. `getApp` selects a unique name, bundle ID, path or running PID, obtains application consent, and launches a stopped app in the background when needed. A stopped executable is identified before approval and checked again before and after launch; unsigned code only receives task access. The returned object reads initial AX without activating the app. Launch cancellation may leave the application running. Model guidance starts with background observation and input, without activation during initialization or before each action. For hidden or minimized windows, try `app.show()` or `window.show()` and read fresh state. This requests unhide and unminimize without activation, raising or switching Spaces; app-level restoration selects the focused, main or sole AX window. The result names the selected `windowId`, or null when no window was selected, and reports `foregroundChanged` by comparing the front app and top window before and after the request. Null means foreground evidence was unavailable; later focus changes remain possible. If restoration fails or the window remains unusable, explain the needed foreground step, call `app.activate()` or an observed window’s `raise` action, then observe again. Necessary foreground access takes priority over preserving focus. Applications may change focus after input; background delivery does not guarantee an undisturbed user session.
|
|
38
|
+
|
|
39
|
+
| Field | Default | Meaning |
|
|
40
|
+
| --- | --- | --- |
|
|
41
|
+
| `allowedApps` | `[]` | Live profile-local application consent, keyed by executable path and verified signing requirement. |
|
|
42
|
+
| `helperPath` | package-owned executable | Optional absolute native executable override. |
|
|
43
|
+
| `buildTimeoutMs` | `300000` | Maximum native preparation time. |
|
|
44
|
+
| `timeoutMs` | `30000` | Per-operation deadline; cancellation can leave partial input. |
|
|
45
|
+
| `connectTimeoutMs` | `5000` | Deadline for connecting to or starting the shared helper. |
|
|
46
|
+
| `screenshotTimeoutMs` | `5000` | Screenshot deadline including queue time, capped by `timeoutMs`. |
|
|
47
|
+
| `axTimeoutMs` | `3000` | Deadline per Accessibility system call, configured independently of the complete operation. |
|
|
48
|
+
| `maxRequestBytes` / `maxResponseBytes` | `1048576` / `33554432` | UTF-8 request and response limits. |
|
|
49
|
+
| `maxClipboardBytes` | `33554432` | Maximum original clipboard bytes preserved during a paste; oversized contents reject before replacement. |
|
|
50
|
+
| `pasteReadTimeoutMs` | `1000` | Clipboard read deadline after the Paste command returns; expiry reports `PASTE_NOT_CONSUMED`. |
|
|
51
|
+
| `maxAppMatches` | `20` | Maximum installed-app search matches; broader queries reject and require a narrower name. |
|
|
52
|
+
| `maxElements` / `maxDepth` | `2000` / `25` | Accessibility traversal limits. |
|
|
53
|
+
| `dragDurationMs` / `dragSteps` | `400` / `20` | Duration and movement segments for a drag. |
|
|
54
|
+
| `textEventIntervalMs` | `0.25` | Delay after each Unicode down/up event, from 0 to 1000 ms. Chord timing is independent. |
|
|
55
|
+
|
|
56
|
+
The [configuration catalog](../../../docs/config-catalog.md#deepseek-aidsh-experimental-cua-macos) lists all fields. Native tool calls use `{method, params}`; the complete method-specific schema is exposed to the model. For example, `getApp` returns the `appHandle` needed by `listWindows`, `getState` and input operations.
|
|
57
|
+
|
|
58
|
+
Keypad names (`KP_0` through `KP_9`, `KP_Enter`, and keypad operators) retain their physical key codes. Key chords use `+` between modifiers and the key. Use `+` for the plus key, `cmd++` for Command-plus, or the key name `plus`; an unfinished chord such as `cmd+` is rejected.
|
|
59
|
+
|
|
60
|
+
Agent disposal and plugin unload cancel and await the complete tool call, including image admission and attachment storage. An image write already in progress must settle before disposal returns; a canceled call cannot return a successful screenshot.
|
|
61
|
+
|
|
62
|
+
Application consent is enforced before AX reads, window-title reads, screenshots, activation, and input through the CUA tool. User Questions offers task access or denial, plus saved access when the application has a verifiable signature and the composition provides settings persistence. Task access is bound to the selected process handle; saved access is bound to its canonical executable path and designated code-signing requirement. Denial prevents repeated prompts in that task. Removing a saved `allowedApps` entry cancels affected operations and invalidates their helper handles; reauthorization requires fresh selection. Other handles on the same Agent connection also expire, while saved grants remain reusable. Agent/provider disposal joins cleanup of the corresponding native connection. See the [application consent decision](../../../.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.md). These checks govern CUA calls; persistent PTC and separately authorized shell tools retain their own execution authority.
|
|
63
|
+
|
|
64
|
+
Raw `getState` exposes validated state fields and the complete text for the requested disclosure options. The native helper also returns structured element diagnostics for native checks; the tool projection omits that duplicate array from model text and structured tool values.
|
|
65
|
+
|
|
66
|
+
AX observations show menu-bar entries by default. Use `app.getAXState({ includeMenus: true })` to expand menu contents and `app.getAXState({ details: true })` for identifiers, global-point bounds and complete action names. These options also apply to raw `getState` and combined AX/screenshot observations. Every observation replaces that scope's indices; obtain fresh indices from the expanded output. Details do not bypass the configured node and depth limits. Truncation identifies the reached limit.
|
|
67
|
+
|
|
68
|
+
Values and placeholders are separate. Named, stateful or actionable groups and scroll areas retain their own rows. Repeated names, descriptions and help text on one node are shown once. Values, placeholders and URLs keep their distinct meanings even when their text matches; body text shared by distinct nodes remains intact. Writability is explicit on relevant controls, including `AXIncrementor`; `unknown` means the system did not report it. Missing optional state fields do not imply false. UTF-16 ranges outside the observed value are marked unavailable. Custom action aliases omit AppKit target/selector diagnostics; ambiguous aliases require a complete name from `details:true`.
|
|
69
|
+
|
|
70
|
+
Observations emit by default and also return their value. In persistent PTC, use `void await app.getAXState()` for one emitted copy, or `emit:false` and one explicit print. Capturing a screenshot after an emitted tree can fail independently; retain the successful AX output.
|
|
71
|
+
|
|
72
|
+
Screenshot diagnostics use `[cua-screenshot]` records keyed by the tool request ID. Native phase records return over the originating Agent connection and cover window/AX geometry checks, system capture, PNG/Base64 encoding and response writing. Host debug logs add dispatch, reply and cleanup timings. Each request retains at most 8192 bytes of complete diagnostic records, with explicit truncation; screenshot failures append these records to the error for Session export. Records contain stages, timings and error codes, not window titles, AX text or image bytes.
|
|
73
|
+
|
|
74
|
+
Native preparation serializes compiler checks and publication with the shared file lock. It updates the package payload and runs a content-addressed copy under native/.build/prepared, preserving executable files used by earlier source-runtime processes. Identical builds reuse the same file; corrupted retained copies fail validation. Cancellation and preparation failure finish managed process cleanup before releasing the lock.
|
|
75
|
+
|
|
76
|
+
### Connect an external execution environment
|
|
77
|
+
|
|
78
|
+
To run through persistent PTC, add `--patch packages/experimental/cua-macos/examples/persistent-ptc.cordis.yml` after the macOS overlay. This provider keeps variables for the live Agent and applies the selected file sandbox policy. The caller imports the bridge and supplies the current authorized `tools.cua` binding. Persistent PTC receives an installation-owned bridge file URL in its CUA instructions, so the project does not need to install this package. The example below is for an external host that resolves the package from its own installation and supplies `tools`; the package does not install a global object or keep variables alive:
|
|
79
|
+
|
|
80
|
+
```javascript
|
|
81
|
+
import { connectCua } from '@deepseek-ai/dsh-cua/bridge'
|
|
82
|
+
const cua = connectCua(() => tools.cua, text => console.log(text))
|
|
83
|
+
const app = await cua.getApp("com.apple.TextEdit")
|
|
84
|
+
void await app.getScreenshot()
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
Use app methods directly for single-window tasks. `getApp` reads initial AX state, so a screenshot needs no window enumeration or additional AX read. Use `app.getWindows()` and `app.getWindow(windowId)` when the task needs a particular window or default selection is ambiguous; choose by the observed title and bounds.
|
|
88
|
+
|
|
89
|
+
`getWindow` emits its initial AX tree. An owned window without an AX surface returns an empty snapshot and remains available for screenshots and coordinate input. Window objects retain separate snapshots, so reading one window does not invalidate another. Re-reading the same scope invalidates old indices. An AX node removed during observation rejects the read with `STALE_SNAPSHOT` and publishes no partial tree; read again without repeating the preceding input. App-scoped screenshots and coordinates reuse the latest AX observation: its focused on-screen, nonzero-size window, otherwise the sole such window. They do not query current focus again; read `app.getAXState()` to follow a focus change. Explicit window objects retain their selected target. Keyboard methods stay on `app` because input follows the app’s current focus.
|
|
90
|
+
|
|
91
|
+
`hover(target)` first reveals advertised AX alternate UI without moving the system cursor. Entering a revealed descendant preserves its parent UI, and leaving restores the default UI. When unavailable, hover moves the visible system cursor to an AX element center or window-local point without pressing a button, including over disabled controls. It requires the selected app to be foreground and the target point unobscured; otherwise `FOREGROUND_REQUIRED` requires explicit activation or raising and a fresh observation. Completion, AX reads and screenshots do not restore the previous point. Wait for the application’s hover delay, observe the revealed content, then hover into it and click; successive calls can visit intermediate points. Other input or application changes can interrupt the interaction. [Continuous-interaction rationale](../../../.agents/notes/implemented/architecture/2026-10-13-cua-continuous-hover.md).
|
|
92
|
+
|
|
93
|
+
`paste(text, { format: "text" | "html" })` temporarily publishes the requested clipboard format and invokes the application’s enabled standard Paste menu command. It keeps the temporary format until a pasteboard reader requests its data, then restores every captured original format while it still owns the clipboard, preserving a later copy. Paste operations share a cross-process lock and cancellation joins helper cleanup. A forced process exit can prevent restoration. Unsupported Paste commands fail without typing fallback; `typeText` remains separate key-event input.
|
|
94
|
+
|
|
95
|
+
`selectText` selects a unique literal occurrence or places the cursor before/after it through writable `AXSelectedTextRange`. Adjacent prefix/suffix disambiguate repeats. Selection preserves the remaining text and formatting, and verifies the resulting UTF-16 range; unsupported, missing, ambiguous or changed text fails without keyboard fallback. AX text includes `selectionWritable`, selection ranges and selected text when available.
|
|
96
|
+
|
|
97
|
+
Screenshot capture requires an attachment store and a resolved model route declaring image input. The shared attachment admission validates canonical Base64, the raster and its configured capacity, then stores the image once. Native data remains available to programmatic callers; model content carries coordinate metadata and the durable image reference. Response byte limits apply before parsing.
|
|
98
|
+
|
|
99
|
+
Absolute coordinates for hover, click, scroll and drag use pixels in the latest returned screenshot. Pass `relative: true` for integer thousandths of window width and height; omission or `false` uses pixels. Relative input can use a successful AX read or screenshot, while absolute input requires a screenshot. The Host maps the delivered image to native points, including attachment and model-request resizing. A value of 1000 remains inside the window edge. Coordinate options do not apply to AX element indices. A width or height change since the latest observation returns `WINDOW_CHANGED` before input, including raw calls without a snapshot ID; read fresh AX or take a new screenshot before proceeding. Window movement alone preserves window-local coordinates. Capture and pointer input reject known disagreement between the selected window’s accessibility frame and displayed bounds with `WINDOW_CHANGED`, before posting pointer events. This includes Stage Manager transformations when AX retains the full window frame. Read fresh state; any activation or window raise remains explicit. Advertised AX actions remain available. Missing AX geometry retains the AXless path and does not establish that a transformed window can be operated.
|
|
100
|
+
|
|
101
|
+
Retained objects resolve the current tool binding on every call. The external runtime owns persistence, imports, cancellation and completion-value serialization. Screenshots return encoded bytes with their MIME type, pixel dimensions and window ID; the standard tool pipeline admits the image attachment exactly once. `emit: false` suppresses facade output only. A host that uses another dispatcher can build objects through `createCuaApi` from `./api`.
|
|
102
|
+
|
|
103
|
+
AppKit sheets are selectable by their window IDs and remain visible in their document window’s tree. Keyboard input follows a system Open/Save panel only when the application’s focused sheet identifies one Apple panel-service window and that service still focuses the same panel. Unrelated or ambiguous remote recipients fail before dispatch.
|
|
104
|
+
|
|
105
|
+
Indexed clicks support a single right/middle click or a single/double left click. Click and drag options accept a button and modifier keys; a modified click uses pointer input. Drags hold that button and carry its modifiers through every movement and release. AX text marks disabled controls, `valueWritable=true`, and available `selected` state. The helper uses an advertised accessibility action for unmodified clicks when available; pointer delivery resolves the element’s current bounds and verifies its owning window. Actions report dispatch, not application success. A visual pointer or click marker can be rendered independently of input; AX actions do not require moving the system cursor. Window-directed mouse input can still change application focus, selection or dialogs. Read state after input. `STALE_APP` requires acquiring the application again; `STALE_SNAPSHOT` requires a fresh AX read. Request timeout/cancellation waits for native acknowledgement and preserves usable handles. Missing cleanup acknowledgement closes that Agent’s connection. A screenshot RPC still awaiting its system callback occupies the single capture slot; further screenshots return `CAPTURE_BUSY`, while AX and input remain available. A late capture result is discarded, and its real completion releases that slot. Only an unresponsive operation or failed cleanup can invalidate the shared process and all its handles. Never retry uncertain input automatically. Closing an Agent releases its connection and AX references, without undoing changes to applications.
|
|
106
|
+
|
|
107
|
+
-----
|
|
108
|
+
|
|
109
|
+
<a id="understand-the-implementation"></a>
|
|
110
|
+
## Understand the implementation
|
|
111
|
+
|
|
112
|
+
<details>
|
|
113
|
+
<summary>Dispatch and resource ownership</summary>
|
|
114
|
+
|
|
115
|
+
The object facade translates app/window methods into JSON operations. The bridge routes them through the normal tool pipeline, preserving authorization, nested logs and image admission. Agents and Hosts belonging to one user share a helper for the same native build. Each connection retains independent handles, snapshots, configuration and cancellation state; desktop operations use a global queue. The managed launcher transfers service lifetime to these connections, and the helper exits after the last connection finishes cleanup. Process locks cover both build identity and the real executable path, preventing duplicate startup and path collisions after a running executable is replaced. Native handles bind a process identity to its kernel start time, and snapshots retain native AX references only inside that helper. The same operation protocol can serve a future platform provider without adding platform logic to the object API.
|
|
116
|
+
|
|
117
|
+
[Zod operation definitions](../cua/src/protocol.ts) own model parameters and separately declare Host resource fields and private continuations. Public data types derive from those definitions. `pnpm exec tsx packages/experimental/cua-macos/native/generate-protocol.ts` generates the native JSON Schema and Swift field/enum admission tables; the package test lane checks freshness and rejects Host-only fields at model admission. Native target validation, resource limits and operation behavior remain enforced at execution.
|
|
118
|
+
|
|
119
|
+
[AX text formatting](native/Sources/CuaNative/AXTextFormatter.swift) consumes already-read node data. The AX walker alone assigns indices and retains native references, so formatting owns no second snapshot or target registry.
|
|
120
|
+
|
|
121
|
+
Native request IDs match the logged tool call IDs. Successful activation and window-raise results retain foreground request details and immediate before/after focus observations in the raw tool result. An already-frontmost app skips redundant activation; a window raise is skipped only when the exact window is also focused, main, unminimized and first among visible normal windows. Unknown observations retain the requested operation. These observations do not establish who caused an unrelated focus change or verify completion.
|
|
122
|
+
|
|
123
|
+
Node roles and metadata share one AX batch per observation. Group and scroll-area nodes contribute only children; unused metadata errors do not reject them. Unsupported batching retains individual attribute reads. AX traversal visits each native identity once, including proxy references that compare equal, so cyclic accessibility children cannot repeat whole subtrees. The [source manifest](native/upstream.json) records adapted upstream behavior and local Swift modules. AppKit runs on the main event loop; a separate serial executor owns AX references and synchronous input. A request retains exclusive execution through capture and response delivery. No invariant companion is published: there is no independently projected registry or durable evaluator state to reconcile. Protocol validation, target identity and resource cleanup are checked by their owning implementations.
|
|
124
|
+
|
|
125
|
+
</details>
|
|
126
|
+
|
|
127
|
+
-----
|
|
128
|
+
|
|
129
|
+
<a id="further-exploration"></a>
|
|
130
|
+
## Further Exploration
|
|
131
|
+
|
|
132
|
+
- [Computer-use registration](../../computer-use/computer-use/README.md)
|
|
133
|
+
- [Native protocol](native/protocol.schema.json)
|
|
134
|
+
- [Object interfaces](../cua/src/types.ts)
|
|
135
|
+
|
|
136
|
+
-----
|
|
137
|
+
|
|
138
|
+
<a id="model-experience"></a>
|
|
139
|
+
## Model Experience
|
|
140
|
+
|
|
141
|
+
### Native application operations
|
|
142
|
+
|
|
143
|
+
#### What the model sees
|
|
144
|
+
|
|
145
|
+
The plugin contributes one `cua` operation tool with method-specific parameters. Standard PTC presentation exposes it as `tools.cua` through `run_code`. The following package-owned guidance is emitted verbatim:
|
|
146
|
+
|
|
147
|
+
##### Operation guidance (verbatim)
|
|
148
|
+
|
|
149
|
+
```markdown
|
|
150
|
+
Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps or cua.getState to discover running applications, and findApps(query) to search installed applications by name, bundle ID or absolute path. getApp requests user consent before launching a stopped application in the background, or before observations and input on a running one. If access is denied or revoked, stop using that application until the user explicitly grants access again. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. Raw getState returns AX text and snapshotId. Menu-bar entries are shown by default; includeMenus=true expands their contents. details=true adds identifiers, global-point bounds and complete action names. Both options also apply to getAXState. Each read replaces the indices for that app/window scope. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are pixels in the latest returned screenshot, measured from its top-left. Set relative:true for integer thousandths from 0 to 1000 instead. The Host converts both modes to native window points; AX detail bounds remain global macOS points. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
|
|
151
|
+
|
|
152
|
+
Initialize the object API once in persistent PTC. Reuse its variables in later calls; put temporary declarations inside { ... } blocks to avoid redeclaration errors. All object methods are async. getAXState returns text, not a structured state object. Its text separates values and placeholders, and reports subroles, help, URLs, disabled/focused/selected/expanded state and UTF-16 selection ranges when available. Writability is true, false or unknown on relevant controls; omitted state fields do not assert false. selection=unavailable means the reported range exceeds the observed text. Compact output omits ordinary Press/Pick (click), ShowMenu (right click) and ScrollToVisible actions; details=true lists every available action. Use the displayed action name with performSecondaryAction; ambiguous aliases require the complete name from details=true. Truncation reports the reached provider limit. Narrow the window or increase that provider limit before reading again.
|
|
153
|
+
|
|
154
|
+
Use app methods directly for single-window tasks; getApp already reads initial AX state. Call app.getWindows() and await app.getWindow(windowId) only to target a particular window or resolve ambiguous window selection. App screenshots and coordinate actions use the latest AX observation: its focused on-screen, nonzero-size window, otherwise the sole such window. They do not refresh focus automatically. Read app.getAXState() to follow changed focus.
|
|
155
|
+
|
|
156
|
+
CuaScreenshot contains { bytes: Uint8Array, mimeType: "image/png" | "image/jpeg" | "image/webp" | "image/gif", pixelWidth: number, pixelHeight: number, windowId: number }. Use mimeType to choose a file extension or declare an image; do not assume PNG.
|
|
157
|
+
|
|
158
|
+
Object API signatures (options default to { emit: true }):
|
|
159
|
+
|
|
160
|
+
- cua.getCapabilities(options?) -> { platform, permissions, sessionLocked, methods, coordinateSpace, browser }
|
|
161
|
+
- cua.listApps(options?) -> Array<{ id, displayName, isRunning: true, pid }>
|
|
162
|
+
- cua.findApps(query: string, options?) -> Array<{ id, displayName, path, isRunning, pid? }>
|
|
163
|
+
- cua.getState(options?) -> { apps, browsers: [] }
|
|
164
|
+
- cua.getApp(selector: string) -> App
|
|
165
|
+
- app.activate() -> { dispatched: true, verified: false }
|
|
166
|
+
- app.getWindows(options?) -> Array<{ windowId, pid, appName, title, bounds, isOnScreen, onCurrentSpace }>
|
|
167
|
+
- app.getWindow(windowId: number) -> Window
|
|
168
|
+
- app.typeText(text: string), app.pressKey(chord: string) -> action result
|
|
169
|
+
- app.paste(text: string, options?: { format?: "text" | "html" }) -> action result
|
|
170
|
+
|
|
171
|
+
App and Window methods:
|
|
172
|
+
|
|
173
|
+
- show() -> { dispatched: true, verified: false, windowId: number | null, foregroundChanged: boolean | null }
|
|
174
|
+
- getAXState(options?: { emit?: boolean, disableDiffing?: boolean, includeMenus?: boolean, details?: boolean }) -> string
|
|
175
|
+
- getScreenshot(options?: { emit?: boolean }) -> CuaScreenshot
|
|
176
|
+
- getAXStateAndScreenshot(options?) -> { state: string, screenshot: CuaScreenshot }
|
|
177
|
+
- hover(target: number | [x, y], options?: { relative?: boolean }) -> { dispatched, verified, hoverRoute, systemPointerMoved }
|
|
178
|
+
- click(target: number | [x, y], options?: { mouseButton?: "left" | "right" | "middle", clickCount?: 1 | 2, modifiers?: Array<"command" | "control" | "option" | "shift" | "fn">, relative?: boolean }) -> action result
|
|
179
|
+
- setValue(elementIndex: number, value: string) -> action result
|
|
180
|
+
- selectText(elementIndex: number, text: string, options?: { prefix?: string, suffix?: string, selectionType?: "text" | "cursor_before" | "cursor_after" }) -> action result
|
|
181
|
+
- performSecondaryAction(elementIndex: number, action: string) -> action result
|
|
182
|
+
- scroll({ x, y, deltaLinesX, deltaLinesY, relative? }) -> action result
|
|
183
|
+
- drag(from: [x, y], to: [x, y], options?: { mouseButton?: "left" | "right" | "middle", modifiers?: Array<"command" | "control" | "option" | "shift" | "fn">, relative?: boolean }) -> action result
|
|
184
|
+
|
|
185
|
+
Raw operation tools use { method, params }, including params: {} for parameterless methods. getApp selects without activating the app. Keep applications in the background for observation and routine input. Do not activate or raise an app during initialization, before each action, or merely because another app is foreground. Prefer advertised AX actions. If a window is hidden or minimized, try app.show() or window.show() and read fresh state. show requests unhide and unminimize without activation, raising or switching Spaces; it chooses the explicit, focused, main or sole AX window. A null windowId means no window was selected for restoration. foregroundChanged compares the front app and top window before and after the request; null means unavailable, and delayed focus changes remain possible. If restoration is unsupported or the window remains unusable, including another Space or Stage Manager, explain the needed foreground step, use app.activate() or an advertised raise action, and read fresh state. Completing the task takes priority when foreground access is necessary. Avoid repeated activation when the user switches away; continue background-capable steps and report what requires foreground access. Applications may still change focus in response to input.
|
|
186
|
+
|
|
187
|
+
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Absolute coordinate input requires a fresh screenshot of that window. Relative input can use a successful AX observation or screenshot. SCREENSHOT_REQUIRED means capture a fresh screenshot before using pixel coordinates. For click and hover, coordinate targets must be [x, y], for example hover([650, 400], {relative:true}). A bare number such as hover(650) is always an AX element index. The relative option applies only to coordinate targets, not AX indices. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to grant macOS Accessibility or Screen Recording access in System Settings. After granting access, observe fresh state before continuing; do not automatically replay prior input.
|
|
188
|
+
|
|
189
|
+
Hover first uses advertised AX alternate UI without moving the system cursor, preserving revealed descendants until the pointer target leaves their region. When unavailable, it moves the visible system cursor without pressing a button; that route requires the target point to be unobscured in the foreground app. FOREGROUND_REQUIRED requires explicit activation or raising and a fresh observation before retrying. Returning, AX reads and screenshots do not restore the previous point. Wait for the application's hover delay, observe, then hover into the revealed controls before clicking. Use successive hover calls for intermediate points. The result reports hoverRoute and systemPointerMoved. Other input or application changes can interrupt the interaction; app objects do not reserve applications between calls.
|
|
190
|
+
|
|
191
|
+
Use selectText only on controls advertising selectionWritable. It selects a unique literal occurrence, or places the cursor before/after it; prefix and suffix must be immediately adjacent. AMBIGUOUS_TEXT requires more context; TEXT_NOT_FOUND or SELECTION_CHANGED requires a fresh observation before deciding on input. Left clicks support single/double clicks; right and middle clicks support single clicks. Modified clicks use pointer input. Keypad names such as KP_0 and KP_Enter retain their physical key identity. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Paste delivers plain text or HTML through the application's enabled standard Paste menu command. Paste follows the target app's focused text input. Foreground requirements vary by application; for example, ChatGPT/Codex was observed to accept background Paste without reading clipboard data. PASTE_NOT_CONSUMED means the command was dispatched but no clipboard read was observed before the configured deadline. Inspect the target contents, explicitly activate the app and focus the intended input, then observe again before deciding whether to retry; a delayed paste may still run. Original clipboard formats are restored if the clipboard still belongs to this paste; a later copy is preserved. Unsupported Paste commands fail without typing fallback. CLIPBOARD_CHANGED means another writer intervened; observe before deciding on further input. Browser applications can be controlled with getApp like other native apps; browser tabs are unavailable. Installed-app search covers standard macOS application directories; use an absolute application path for another location. Launch does not restore a window or document, and cancellation may leave an application already started. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global. In persistent PTC, use the host-provided initialization snippet below once. Its installation-owned file URL works independently of the project directory. Retain app objects between calls. Observation methods emit by default and also return their value. Use void await app.getAXState() to avoid a duplicate REPL completion, or pass emit:false and print the returned text once. End screenshot calls with void await app.getScreenshot() or void await app.getAXStateAndScreenshot() so image bytes are not serialized as the program completion value.
|
|
192
|
+
```
|
|
193
|
+
|
|
194
|
+
#### Token effect
|
|
195
|
+
|
|
196
|
+
Enabling the plugin adds a fixed schema and guidance. AX observations, operation results and admitted images add model input. Retained objects are not serialized into prompts by this package.
|
|
197
|
+
|
|
198
|
+
#### KV Cache effect
|
|
199
|
+
|
|
200
|
+
Tool results append to Session history. Changing plugin configuration or tool presentation can change the request prefix; retaining an app object does not change existing history.
|
|
201
|
+
|
|
202
|
+
## Known Limitations and Deferred Work
|
|
203
|
+
|
|
204
|
+
The queue orders individual native requests. Holding an app object does not reserve its application across a Session; [continuous interaction](../../../.agents/notes/implemented/architecture/2026-10-13-cua-continuous-hover.md) grants no cross-call ownership. Capture RPCs that never return occupy the single capture slot; creating new app objects does not reset that system connection.
|
|
205
|
+
|
|
206
|
+
<a id="known-limitations-and-deferred-work"></a>
|
|
207
|
+
|
|
208
|
+
- Only the macOS native implementation is provided. Windows support is in development; this package remains in `experimental` until that support is complete, then moves to the `computer-use` group. Browser tabs, AX diffs, unassociated system panels and accessory-only apps are unsupported.
|
|
209
|
+
- Coordinates use the returned screenshot pixels, including the title bar; `relative: true` selects integer thousandths. Native point conversion is Host-owned. Window movement, application changes and other desktop users can race observations and input.
|
|
210
|
+
- macOS Stage Manager can reduce background window bounds and screenshots to thumbnails. Full-window observation and coordinate input in this state are not guaranteed; model guidance requires explaining foreground needs and avoiding repeated focus reclamation.
|
|
211
|
+
- Background input depends on private SkyLight symbols; macOS updates can make them unavailable. Validate each target OS version.
|
|
212
|
+
- Observed failures include dispatched Unicode text not appearing in Blender and background Command+A not selecting all text in AppKit. Inspect application state after input; a successful tool result confirms dispatch only.
|
|
213
|
+
- Native binaries require a local build. Signing, distribution, and system-permission onboarding remain deferred.
|
|
214
|
+
- Persistent PTC is an external dependency assumption, not an implementation in this package. Real-model task success requires separate evaluation beyond native and protocol fixtures.
|
|
215
|
+
|
|
216
|
+
-----
|
|
217
|
+
|
|
218
|
+
<a id="dev-note"></a>
|
|
219
|
+
### Dev Note
|
|
220
|
+
|
|
221
|
+
<details>
|
|
222
|
+
<summary>Working context for maintainers</summary>
|
|
223
|
+
|
|
224
|
+
None.
|
|
225
|
+
|
|
226
|
+
</details>
|
package/README.zh.md
ADDED
|
@@ -0,0 +1,226 @@
|
|
|
1
|
+
---
|
|
2
|
+
description: "通过按应用和窗口限定的对象,观察和操作原生 macOS 应用。"
|
|
3
|
+
kind: "package-reference"
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# @deepseek-ai/dsh-experimental-cua-macos
|
|
7
|
+
|
|
8
|
+
[English](README.md) | 中文
|
|
9
|
+
|
|
10
|
+
## 概述
|
|
11
|
+
|
|
12
|
+
检查 macOS 权限,选择运行中的应用和窗口,读取辅助功能树、截图并发送输入。外部持久执行环境可以保留这些对象。Swift 辅助进程直接使用 macOS 框架,并改写选取的 MIT 许可 Cua Driver 行为。本包不提供 PTC 运行时或变量持久化。
|
|
13
|
+
|
|
14
|
+
## 目录
|
|
15
|
+
|
|
16
|
+
- [使用本包](#use-this-package)
|
|
17
|
+
- [理解实现](#understand-the-implementation)
|
|
18
|
+
- [进一步阅读](#further-exploration)
|
|
19
|
+
- [模型体验](#model-experience)
|
|
20
|
+
- [已知限制与后续工作](#known-limitations-and-deferred-work)
|
|
21
|
+
- [开发备注](#dev-note)
|
|
22
|
+
|
|
23
|
+
-----
|
|
24
|
+
|
|
25
|
+
<a id="use-this-package"></a>
|
|
26
|
+
## 使用本包
|
|
27
|
+
|
|
28
|
+
该实验性提供方需要在 macOS 14 及以上通过源码配置显式启用。Windows 支持正在开发中。默认配置不加载它。先准备原生组件,再加载注册表和提供方覆盖层:
|
|
29
|
+
|
|
30
|
+
```sh
|
|
31
|
+
swift build --package-path packages/experimental/cua-macos/native --configuration release
|
|
32
|
+
cua_native_root="$PWD/packages/experimental/cua-macos/native"
|
|
33
|
+
export DSH_CUA_HELPER="$cua_native_root/.build/release/dsh-cua-native"
|
|
34
|
+
pnpm dsh headless --patch packages/experimental/cua-macos/examples/macos.cordis.yml --dump-config
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
配置输出命令不发起模型请求。执行真实任务时,将 `--dump-config` 替换为任务提示,并配置该 profile 的模型凭据。配置挂载 computer-use 注册表和本 provider;使用前需移除已有的竞争 provider。负责执行的宿主或辅助进程需要辅助功能权限来读取 AX 和输入,需要屏幕录制权限来截图。`getCapabilities` 查询权限和锁屏状态,不弹出系统授权窗口。桌面会话锁定或不可用时,界面观察与输入会在分派前拒绝,需由用户解锁 macOS 后继续。`listApps` 和 `cua.getState` 列出运行中的应用。`findApps(query)` 按名称、bundle ID 或绝对路径搜索 macOS 标准应用目录。`getApp` 按唯一名称、bundle ID、路径或运行中的 PID 选择应用,取得应用许可后,在需要时后台启动尚未运行的应用。启动前先确认可执行文件身份,授权后在启动前后再次核对;未签名代码只取得任务许可。返回的对象读取初始 AX,不自动激活应用。启动被取消时,应用可能已经运行。模型指引要求先在后台观察和输入,不在初始化或每次动作前激活应用。窗口隐藏或最小化时,先尝试 `app.show()` 或 `window.show()`,再重新观察。它请求取消隐藏和恢复最小化,不主动激活、抬起窗口或切换桌面空间;应用级恢复选择焦点窗口、主窗口或唯一 AX 窗口。结果通过 `windowId` 指明选中的窗口,未选中时为 null;`foregroundChanged` 比较操作前后的前台应用和最上层窗口,无法取得完整观察时为 null,返回后仍可能发生焦点变化。恢复失败或窗口仍不可用时,说明需要前台的步骤,再调用 `app.activate()` 或窗口的 `raise` 动作并重新观察。必须使用前台时,以完成任务为先。应用收到输入后仍可能改变焦点,后台投递不保证用户操作始终不受干扰。
|
|
38
|
+
|
|
39
|
+
| 字段 | 默认值 | 含义 |
|
|
40
|
+
| --- | --- | --- |
|
|
41
|
+
| `allowedApps` | `[]` | 当前配置档的动态应用授权,绑定可执行文件路径和已验证的代码签名要求。 |
|
|
42
|
+
| `helperPath` | 包内可执行文件 | 可选的原生可执行文件绝对路径覆盖。 |
|
|
43
|
+
| `buildTimeoutMs` | `300000` | 原生组件准备的最长时间。 |
|
|
44
|
+
| `timeoutMs` | `30000` | 单次操作截止时间,取消后可能已发生部分输入。 |
|
|
45
|
+
| `connectTimeoutMs` | `5000` | 连接或启动共享 helper 的截止时间。 |
|
|
46
|
+
| `screenshotTimeoutMs` | `5000` | 截图总期限,包含排队时间,且不超过 `timeoutMs`。 |
|
|
47
|
+
| `axTimeoutMs` | `3000` | 每次 Accessibility 系统调用的截止时间,与完整操作截止独立配置。 |
|
|
48
|
+
| `maxRequestBytes` / `maxResponseBytes` | `1048576` / `33554432` | UTF-8 请求与响应字节限制。 |
|
|
49
|
+
| `maxClipboardBytes` | `33554432` | 粘贴时保存原剪贴板内容的字节上限;超限会在替换前拒绝。 |
|
|
50
|
+
| `pasteReadTimeoutMs` | `1000` | Paste 命令返回后等待剪贴板读取的上限;到期返回 `PASTE_NOT_CONSUMED`。 |
|
|
51
|
+
| `maxAppMatches` | `20` | 已安装应用搜索的结果上限;范围过大的查询会被拒绝,需要缩小名称范围。 |
|
|
52
|
+
| `maxElements` / `maxDepth` | `2000` / `25` | 辅助功能树遍历限制。 |
|
|
53
|
+
| `dragDurationMs` / `dragSteps` | `400` / `20` | 拖动时长和移动分段数。 |
|
|
54
|
+
| `textEventIntervalMs` | `0.25` | 每个 Unicode 按下或抬起事件后的间隔,范围为 0 至 1000 毫秒。组合键的节奏独立。 |
|
|
55
|
+
|
|
56
|
+
[配置目录](../../../docs/config-catalog.zh.md#deepseek-aidsh-experimental-cua-macos)列出全部字段。原生工具调用使用 `{method, params}`,模型可以看到每个方法的完整参数定义。例如,`getApp` 返回 `listWindows`、`getState` 和输入操作所需的 `appHandle`。
|
|
57
|
+
|
|
58
|
+
数字键盘名称(`KP_0` 到 `KP_9`、`KP_Enter` 和数字键盘运算键)保留独立的物理键码。组合键用 `+` 分隔修饰键与按键。加号键写作 `+`,Command 加号写作 `cmd++`,也可以使用键名 `plus`;`cmd+` 这类缺少按键的输入会被拒绝。
|
|
59
|
+
|
|
60
|
+
Agent 销毁和插件卸载会取消并等待完整工具调用,包括图像接入检查和附件存储。已经开始的图像写入必须结束后,销毁才会返回;已取消的调用不会返回成功截图。
|
|
61
|
+
|
|
62
|
+
CUA 工具在读取 AX、窗口标题、截图、激活和输入之前检查应用授权。User Questions 提供任务级允许和拒绝选项;应用签名可验证且组合支持配置持久化时,也可保存授权。任务授权绑定所选进程句柄,长期授权绑定真实可执行路径和指定代码签名要求。拒绝后,同一任务不再反复询问。删除保存的 `allowedApps` 条目会取消相关操作并使原生组件中的句柄失效;重新授权需要重新选择应用。同一 Agent 连接中的其他句柄也会失效,已保存的授权仍可复用。Agent 或提供方销毁时等待对应连接的原生状态清理。设计理由见[应用授权决策](../../../.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.zh.md)。这些检查约束 CUA 调用;持久 PTC 和另行授权的 shell 工具保留各自的执行权限。
|
|
63
|
+
|
|
64
|
+
原始 `getState` 操作返回经过校验的状态字段,以及按本次披露选项生成的完整文本。原生 helper 还返回供原生检查使用的结构化元素诊断数组;工具投影会从模型文本和结构化工具值中移除这份重复数组。
|
|
65
|
+
|
|
66
|
+
AX 观察默认只展示菜单栏入口。使用 `app.getAXState({ includeMenus: true })` 展开菜单内容,使用 `app.getAXState({ details: true })` 读取标识符、全局 point 坐标和完整动作名。原始 `getState` 和 AX/截图组合观察也接受这两个选项。每次观察都会替换该范围的元素编号,展开后应使用新输出中的编号。详情不会绕过配置的节点数和深度上限;截断提示会指明触发的上限。
|
|
67
|
+
|
|
68
|
+
实际值与占位文字分别展示。有名称、状态或可用动作的分组和滚动区域保留独立行。同一节点中重复的名称、描述和帮助只显示一次。实际值、占位文字和 URL 即使文字相同也保留各自含义,不删除不同节点共有的正文。相关控件明确展示可写性,包括 `AXIncrementor`;`unknown` 表示系统未报告。缺少可选状态字段不代表 false。超出当前值长度的 UTF-16 选区标为不可用。自定义动作简称省略 AppKit 的 target/selector 诊断信息;同名动作需要使用 `details:true` 返回的完整名称。
|
|
69
|
+
|
|
70
|
+
观察方法默认自动输出,同时返回结果。在 persistent PTC 中使用 `void await app.getAXState()` 输出一次,或指定 `emit:false` 后自行打印一次。AX 输出后的截图可能单独失败,已成功取得的 AX 输出仍保留。
|
|
71
|
+
|
|
72
|
+
截图诊断使用带工具请求 ID 的 `[cua-screenshot]` 记录。原生阶段记录经本次 Agent 连接返回,覆盖窗口及 AX 几何检查、系统截图、PNG/Base64 编码和响应写出。Host 调试日志补充发送、响应和清理耗时。每次请求最多保留 8192 字节完整诊断记录,超限会明确标记;截图失败时附在错误中,保留到 Session trace。记录只含阶段、耗时和错误码,不含窗口标题、AX 正文或图片内容。
|
|
73
|
+
|
|
74
|
+
原生准备复用文件锁,串行检查编译和发布。构建更新包内产物,并从 native/.build/prepared 下按内容摘要区分的副本运行,保留旧源码进程使用的可执行文件。相同构建复用同一文件;保留的副本损坏时拒绝使用。取消或准备失败后,先完成受管进程清理再释放锁。
|
|
75
|
+
|
|
76
|
+
### 接入外部执行环境
|
|
77
|
+
|
|
78
|
+
使用 persistent PTC 时,在 macOS 配置后追加 `--patch packages/experimental/cua-macos/examples/persistent-ptc.cordis.yml`。该 provider 为存活的 Agent 保留变量,并遵守当前选择的文件沙箱策略。调用方导入适配模块,并提供当前执行中已授权的 `tools.cua` 绑定。persistent PTC 会在 CUA 指引中收到安装目录内 bridge 的文件 URL,项目无须安装本包。下面的例子用于能从自身安装目录解析本包、且已提供 `tools` 的外部宿主;本包不安装全局对象,也不负责保留变量:
|
|
79
|
+
|
|
80
|
+
```javascript
|
|
81
|
+
import { connectCua } from '@deepseek-ai/dsh-cua/bridge'
|
|
82
|
+
const cua = connectCua(() => tools.cua, text => console.log(text))
|
|
83
|
+
const app = await cua.getApp("com.apple.TextEdit")
|
|
84
|
+
void await app.getScreenshot()
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
单窗口任务直接使用 app 方法。`getApp` 已读取初始 AX,截图不需要枚举窗口或额外读取 AX。任务需要指定窗口或默认选窗存在歧义时,再调用 `app.getWindows()` 和 `app.getWindow(windowId)`,根据观察到的标题与位置选择目标。
|
|
88
|
+
|
|
89
|
+
`getWindow` 会输出初始 AX 树。已确认归属但没有 AX 界面的窗口返回空快照,仍可截图和发送坐标输入。不同窗口分别保留快照,读取一个窗口不会使另一个窗口失效;重新读取同一范围会使旧索引失效。观察期间节点被移除时,该次读取返回 `STALE_SNAPSHOT`,不发布部分树;可以重新读取,但不要重复此前的输入。应用级截图和坐标操作复用最近一次 AX 观察:优先使用其中在屏幕上且尺寸非零的焦点窗口,否则要求只有一个符合条件的窗口。它们不会重新查询即时焦点;需要跟随焦点变化时,先读取 `app.getAXState()`。显式窗口对象保持选定的目标。键盘方法保留在 `app` 上,因为输入跟随应用的当前焦点。
|
|
90
|
+
|
|
91
|
+
`hover(target)` 优先通过控件声明的 AX 动作展开替代界面,不移动系统鼠标。进入展开区域的子控件时保持父控件展开,离开后恢复默认界面。没有这条路径时,再将可见系统鼠标移到 AX 元素中心或窗口内坐标,不按鼠标按钮,也可用于禁用控件。所选应用必须在前台,且目标点未被遮挡,否则返回 `FOREGROUND_REQUIRED`,调用方需显式激活或抬起窗口并重新观察。调用结束、AX 读取和截图不恢复先前的位置。等待应用的悬停延迟后,观察出现的内容,再移入并点击;连续调用可经过中间位置。其他输入或应用自身变化可能中断交互。[连续交互的设计理由](../../../.agents/notes/implemented/architecture/2026-10-13-cua-continuous-hover.zh.md)。
|
|
92
|
+
|
|
93
|
+
`paste(text, { format: "text" | "html" })` 临时提供指定格式的剪贴板内容,并调用应用已启用的标准 Paste 菜单命令。临时格式保留到剪贴板读取方请求其数据,然后只在仍持有剪贴板时恢复保存的全部原格式,保留期间产生的新复制内容。粘贴操作共用跨进程锁,取消会等待辅助进程清理。强制终止进程可能阻止恢复。应用不支持 Paste 命令时明确失败,不自动改成逐字输入;`typeText` 仍单独提供按键事件输入。
|
|
94
|
+
|
|
95
|
+
`selectText` 通过可写的 `AXSelectedTextRange` 选中唯一的字面文本,或把光标放在它的前后。紧邻的前后文用于区分重复文本。选区操作保留其余文字和格式,并核对实际 UTF-16 范围;控件不支持、文本不存在、匹配不唯一或文字变化时明确失败,不自动改用键盘输入。AX 文本包含可用的 `selectionWritable`、选区范围和选中文字。
|
|
96
|
+
|
|
97
|
+
截图要求附件服务可用,并确认当前模型支持图片输入。共享附件接入校验规范 Base64、图像数据及配置容量,然后保存一次图片。程序调用方仍可取得原生数据,模型内容包含坐标元数据和持久图片引用。响应字节上限在解析前执行。
|
|
98
|
+
|
|
99
|
+
hover、click、scroll 和 drag 的绝对坐标默认使用最近返回截图中的像素。传入 `relative: true` 时使用窗口宽高的千分比整数;省略或传入 `false` 时使用像素。相对坐标可基于成功的 AX 读取或截图,绝对坐标必须先截图。Host 根据实际送入模型的图片尺寸换算原生 point,包含附件归一化和模型请求阶段的缩放。1000 映射到窗口边缘以内。坐标选项不适用于 AX 元素索引。窗口宽高与最近一次观察不同时,在输入前返回 `WINDOW_CHANGED`,原始调用即使不传快照 ID 也会检查;重新读取 AX 或截图后再操作。单纯移动窗口不使窗口内坐标失效。截图和鼠标输入会核对所选窗口的辅助功能框与显示边界;已知两者不一致时,在发送任何鼠标事件前返回 `WINDOW_CHANGED`。当 AX 保留完整窗口框时,这也会拦住 Stage Manager 变换后的窗口。需要重新读取状态;激活应用或抬起窗口仍须显式调用。元素声明的 AX 动作保持可用。缺少 AX 几何时保留无 AX 路径,不代表变换后的窗口一定可以操作。
|
|
100
|
+
|
|
101
|
+
保留的对象会在每次调用时获取当前工具绑定。外部运行时负责持久化、导入、取消和最终表达式值的序列化。截图返回编码字节及其 MIME 类型、像素尺寸和窗口 ID,标准工具链负责将图片作为附件传递一次。`emit: false` 只控制对象层的输出。使用其他分派器的宿主可以通过 `./api` 的 `createCuaApi` 创建对象。
|
|
102
|
+
|
|
103
|
+
AppKit sheet 可以通过其窗口 ID 单独选择,也会出现在所属文档窗口的控件树中。只有应用当前获得焦点的 sheet 指向唯一的 Apple 面板服务窗口,且该服务仍聚焦于同一面板时,键盘输入才转交系统打开或保存面板。无法确认或存在歧义的远端接收方会在输入前报错。
|
|
104
|
+
|
|
105
|
+
元素索引支持右键或中键单击以及左键单击、双击。点击和拖动选项可指定按键与修饰键;带修饰键的点击使用指针输入。拖动期间保持指定按键,并在移动和释放事件中携带修饰键。AX 文本标明禁用控件、`valueWritable=true` 和可读取的 `selected` 状态。不带修饰键的点击优先使用元素声明的辅助功能动作;需要发送鼠标事件时,会重新读取元素位置并核实所属窗口。动作返回只表示输入已分派。可视指针或点击标记可以独立绘制;AX 动作无需移动系统指针。窗口定向鼠标输入仍可能改变应用焦点、选区或对话框。输入后需要读取状态确认结果。`STALE_APP` 需要重新获取应用,`STALE_SNAPSHOT` 需要重新读取 AX。请求超时或取消会等待原生停止确认,保留仍可用的句柄;无法确认清理完成时关闭该 Agent 的连接。未返回系统回调的截图 RPC 占用唯一捕获槽位,新截图立即返回 `CAPTURE_BUSY`,AX 和输入仍可使用。迟到图片被丢弃,真实回调结束后释放槽位。只有操作无法停止或清理失败时,才可能使共享进程及所有旧句柄失效。不能自动重试结果不确定的输入。关闭 Agent 会释放其连接和 AX 引用,但不会撤销应用中的修改。
|
|
106
|
+
|
|
107
|
+
-----
|
|
108
|
+
|
|
109
|
+
<a id="understand-the-implementation"></a>
|
|
110
|
+
## 理解实现
|
|
111
|
+
|
|
112
|
+
<details>
|
|
113
|
+
<summary>操作分派与资源归属</summary>
|
|
114
|
+
|
|
115
|
+
对象接口将应用和窗口方法转换为 JSON 操作。适配模块通过标准工具链保留授权、嵌套日志和图片接入。同一用户的相同 helper 构建由多个 Agent 和 Host 共享。每条连接独立保存句柄、快照、配置和取消状态,桌面操作通过全局队列执行。受管启动器将服务生命期交给这些连接,最后一条连接清理完成后服务退出。进程锁覆盖构建身份及真实可执行路径,防止重复启动或覆盖运行版本后产生路径冲突。原生句柄绑定进程身份及内核启动时间,快照中的 AX 引用只保存在辅助进程内。未来平台可以实现相同操作协议,无需在对象接口中加入平台逻辑。
|
|
116
|
+
|
|
117
|
+
[Zod 操作定义](../cua/src/protocol.ts)负责模型参数,并单独声明 Host 资源字段和内部续接操作。公共数据类型从这些定义推导。`pnpm exec tsx packages/experimental/cua-macos/native/generate-protocol.ts` 生成原生 JSON Schema 和 Swift 字段、枚举校验表;包测试检查产物是否同步,并验证模型不能提交 Host 专属字段。原生目标验证、资源上限和操作行为仍在执行处检查。
|
|
118
|
+
|
|
119
|
+
[AX 文本格式化](native/Sources/CuaNative/AXTextFormatter.swift)只接收已读取的节点数据。AX 遍历器独自分配索引并保存原生引用,格式化不维护另一套快照或目标注册表。
|
|
120
|
+
|
|
121
|
+
原生请求 ID 与工具日志中的调用 ID 一致。成功的激活和窗口抬起操作会在原始工具结果中保留前台操作信息,以及操作前后的即时焦点观察。应用已在前台时跳过重复激活;窗口抬起还要求该窗口正是焦点主窗口、未最小化且位于可见普通窗口的最前方,才会跳过。无法确认时保留原请求。这些观察不能确认其他焦点切换的发起者,也不代表操作已经完成。
|
|
122
|
+
|
|
123
|
+
每次观察在同一批 AX 请求中读取节点角色和元数据。组节点和滚动区域节点只提供子节点,未使用的元数据错误不会导致它们被拒绝。不支持批量读取时保留逐个属性读取。AX 遍历对每个原生身份只访问一次,包括表示同一元素的不同代理引用,避免循环子节点重复展开整个子树。[源码清单](native/upstream.json)记录改写的上游行为和本地 Swift 模块。AppKit 使用主事件循环,独立串行执行器持有 AX 引用并发送同步输入。一条请求在截图和结果发送期间仍独占执行。本包不发布 invariant 入口,因为没有需要对照的独立注册表投影或持久执行器状态。协议验证、目标身份和资源清理由对应实现负责。
|
|
124
|
+
|
|
125
|
+
</details>
|
|
126
|
+
|
|
127
|
+
-----
|
|
128
|
+
|
|
129
|
+
<a id="further-exploration"></a>
|
|
130
|
+
## 进一步阅读
|
|
131
|
+
|
|
132
|
+
- [Computer-use 注册](../../computer-use/computer-use/README.zh.md)
|
|
133
|
+
- [原生协议](native/protocol.schema.json)
|
|
134
|
+
- [对象接口](../cua/src/types.ts)
|
|
135
|
+
|
|
136
|
+
-----
|
|
137
|
+
|
|
138
|
+
<a id="model-experience"></a>
|
|
139
|
+
## 模型体验
|
|
140
|
+
|
|
141
|
+
### 原生应用操作
|
|
142
|
+
|
|
143
|
+
#### 模型可见内容
|
|
144
|
+
|
|
145
|
+
插件提供一个 `cua` 操作工具,定义各方法的参数。标准 PTC 呈现通过 `run_code` 将它暴露为 `tools.cua`。以下是本包逐字输出的指导文本:
|
|
146
|
+
|
|
147
|
+
##### 操作指导文本(原文)
|
|
148
|
+
|
|
149
|
+
```markdown
|
|
150
|
+
Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps or cua.getState to discover running applications, and findApps(query) to search installed applications by name, bundle ID or absolute path. getApp requests user consent before launching a stopped application in the background, or before observations and input on a running one. If access is denied or revoked, stop using that application until the user explicitly grants access again. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. Raw getState returns AX text and snapshotId. Menu-bar entries are shown by default; includeMenus=true expands their contents. details=true adds identifiers, global-point bounds and complete action names. Both options also apply to getAXState. Each read replaces the indices for that app/window scope. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are pixels in the latest returned screenshot, measured from its top-left. Set relative:true for integer thousandths from 0 to 1000 instead. The Host converts both modes to native window points; AX detail bounds remain global macOS points. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
|
|
151
|
+
|
|
152
|
+
Initialize the object API once in persistent PTC. Reuse its variables in later calls; put temporary declarations inside { ... } blocks to avoid redeclaration errors. All object methods are async. getAXState returns text, not a structured state object. Its text separates values and placeholders, and reports subroles, help, URLs, disabled/focused/selected/expanded state and UTF-16 selection ranges when available. Writability is true, false or unknown on relevant controls; omitted state fields do not assert false. selection=unavailable means the reported range exceeds the observed text. Compact output omits ordinary Press/Pick (click), ShowMenu (right click) and ScrollToVisible actions; details=true lists every available action. Use the displayed action name with performSecondaryAction; ambiguous aliases require the complete name from details=true. Truncation reports the reached provider limit. Narrow the window or increase that provider limit before reading again.
|
|
153
|
+
|
|
154
|
+
Use app methods directly for single-window tasks; getApp already reads initial AX state. Call app.getWindows() and await app.getWindow(windowId) only to target a particular window or resolve ambiguous window selection. App screenshots and coordinate actions use the latest AX observation: its focused on-screen, nonzero-size window, otherwise the sole such window. They do not refresh focus automatically. Read app.getAXState() to follow changed focus.
|
|
155
|
+
|
|
156
|
+
CuaScreenshot contains { bytes: Uint8Array, mimeType: "image/png" | "image/jpeg" | "image/webp" | "image/gif", pixelWidth: number, pixelHeight: number, windowId: number }. Use mimeType to choose a file extension or declare an image; do not assume PNG.
|
|
157
|
+
|
|
158
|
+
Object API signatures (options default to { emit: true }):
|
|
159
|
+
|
|
160
|
+
- cua.getCapabilities(options?) -> { platform, permissions, sessionLocked, methods, coordinateSpace, browser }
|
|
161
|
+
- cua.listApps(options?) -> Array<{ id, displayName, isRunning: true, pid }>
|
|
162
|
+
- cua.findApps(query: string, options?) -> Array<{ id, displayName, path, isRunning, pid? }>
|
|
163
|
+
- cua.getState(options?) -> { apps, browsers: [] }
|
|
164
|
+
- cua.getApp(selector: string) -> App
|
|
165
|
+
- app.activate() -> { dispatched: true, verified: false }
|
|
166
|
+
- app.getWindows(options?) -> Array<{ windowId, pid, appName, title, bounds, isOnScreen, onCurrentSpace }>
|
|
167
|
+
- app.getWindow(windowId: number) -> Window
|
|
168
|
+
- app.typeText(text: string), app.pressKey(chord: string) -> action result
|
|
169
|
+
- app.paste(text: string, options?: { format?: "text" | "html" }) -> action result
|
|
170
|
+
|
|
171
|
+
App and Window methods:
|
|
172
|
+
|
|
173
|
+
- show() -> { dispatched: true, verified: false, windowId: number | null, foregroundChanged: boolean | null }
|
|
174
|
+
- getAXState(options?: { emit?: boolean, disableDiffing?: boolean, includeMenus?: boolean, details?: boolean }) -> string
|
|
175
|
+
- getScreenshot(options?: { emit?: boolean }) -> CuaScreenshot
|
|
176
|
+
- getAXStateAndScreenshot(options?) -> { state: string, screenshot: CuaScreenshot }
|
|
177
|
+
- hover(target: number | [x, y], options?: { relative?: boolean }) -> { dispatched, verified, hoverRoute, systemPointerMoved }
|
|
178
|
+
- click(target: number | [x, y], options?: { mouseButton?: "left" | "right" | "middle", clickCount?: 1 | 2, modifiers?: Array<"command" | "control" | "option" | "shift" | "fn">, relative?: boolean }) -> action result
|
|
179
|
+
- setValue(elementIndex: number, value: string) -> action result
|
|
180
|
+
- selectText(elementIndex: number, text: string, options?: { prefix?: string, suffix?: string, selectionType?: "text" | "cursor_before" | "cursor_after" }) -> action result
|
|
181
|
+
- performSecondaryAction(elementIndex: number, action: string) -> action result
|
|
182
|
+
- scroll({ x, y, deltaLinesX, deltaLinesY, relative? }) -> action result
|
|
183
|
+
- drag(from: [x, y], to: [x, y], options?: { mouseButton?: "left" | "right" | "middle", modifiers?: Array<"command" | "control" | "option" | "shift" | "fn">, relative?: boolean }) -> action result
|
|
184
|
+
|
|
185
|
+
Raw operation tools use { method, params }, including params: {} for parameterless methods. getApp selects without activating the app. Keep applications in the background for observation and routine input. Do not activate or raise an app during initialization, before each action, or merely because another app is foreground. Prefer advertised AX actions. If a window is hidden or minimized, try app.show() or window.show() and read fresh state. show requests unhide and unminimize without activation, raising or switching Spaces; it chooses the explicit, focused, main or sole AX window. A null windowId means no window was selected for restoration. foregroundChanged compares the front app and top window before and after the request; null means unavailable, and delayed focus changes remain possible. If restoration is unsupported or the window remains unusable, including another Space or Stage Manager, explain the needed foreground step, use app.activate() or an advertised raise action, and read fresh state. Completing the task takes priority when foreground access is necessary. Avoid repeated activation when the user switches away; continue background-capable steps and report what requires foreground access. Applications may still change focus in response to input.
|
|
186
|
+
|
|
187
|
+
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Absolute coordinate input requires a fresh screenshot of that window. Relative input can use a successful AX observation or screenshot. SCREENSHOT_REQUIRED means capture a fresh screenshot before using pixel coordinates. For click and hover, coordinate targets must be [x, y], for example hover([650, 400], {relative:true}). A bare number such as hover(650) is always an AX element index. The relative option applies only to coordinate targets, not AX indices. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to grant macOS Accessibility or Screen Recording access in System Settings. After granting access, observe fresh state before continuing; do not automatically replay prior input.
|
|
188
|
+
|
|
189
|
+
Hover first uses advertised AX alternate UI without moving the system cursor, preserving revealed descendants until the pointer target leaves their region. When unavailable, it moves the visible system cursor without pressing a button; that route requires the target point to be unobscured in the foreground app. FOREGROUND_REQUIRED requires explicit activation or raising and a fresh observation before retrying. Returning, AX reads and screenshots do not restore the previous point. Wait for the application's hover delay, observe, then hover into the revealed controls before clicking. Use successive hover calls for intermediate points. The result reports hoverRoute and systemPointerMoved. Other input or application changes can interrupt the interaction; app objects do not reserve applications between calls.
|
|
190
|
+
|
|
191
|
+
Use selectText only on controls advertising selectionWritable. It selects a unique literal occurrence, or places the cursor before/after it; prefix and suffix must be immediately adjacent. AMBIGUOUS_TEXT requires more context; TEXT_NOT_FOUND or SELECTION_CHANGED requires a fresh observation before deciding on input. Left clicks support single/double clicks; right and middle clicks support single clicks. Modified clicks use pointer input. Keypad names such as KP_0 and KP_Enter retain their physical key identity. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Paste delivers plain text or HTML through the application's enabled standard Paste menu command. Paste follows the target app's focused text input. Foreground requirements vary by application; for example, ChatGPT/Codex was observed to accept background Paste without reading clipboard data. PASTE_NOT_CONSUMED means the command was dispatched but no clipboard read was observed before the configured deadline. Inspect the target contents, explicitly activate the app and focus the intended input, then observe again before deciding whether to retry; a delayed paste may still run. Original clipboard formats are restored if the clipboard still belongs to this paste; a later copy is preserved. Unsupported Paste commands fail without typing fallback. CLIPBOARD_CHANGED means another writer intervened; observe before deciding on further input. Browser applications can be controlled with getApp like other native apps; browser tabs are unavailable. Installed-app search covers standard macOS application directories; use an absolute application path for another location. Launch does not restore a window or document, and cancellation may leave an application already started. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global. In persistent PTC, use the host-provided initialization snippet below once. Its installation-owned file URL works independently of the project directory. Retain app objects between calls. Observation methods emit by default and also return their value. Use void await app.getAXState() to avoid a duplicate REPL completion, or pass emit:false and print the returned text once. End screenshot calls with void await app.getScreenshot() or void await app.getAXStateAndScreenshot() so image bytes are not serialized as the program completion value.
|
|
192
|
+
```
|
|
193
|
+
|
|
194
|
+
#### Token 影响
|
|
195
|
+
|
|
196
|
+
启用插件会增加固定参数定义和指导文本。AX 观察、操作结果和已接入的图片会增加模型输入。本包不将保留的对象序列化进提示。
|
|
197
|
+
|
|
198
|
+
#### KV Cache 影响
|
|
199
|
+
|
|
200
|
+
工具结果追加到 Session 历史。修改插件配置或工具呈现方式可能改变请求前缀;保留应用对象不会改变已有历史。
|
|
201
|
+
|
|
202
|
+
## 已知限制与后续工作
|
|
203
|
+
|
|
204
|
+
队列只排序单次原生请求。持有 app 对象不会为 Session 保留应用,[连续交互](../../../.agents/notes/implemented/architecture/2026-10-13-cua-continuous-hover.zh.md)不分配跨调用所有权。一直不返回的系统截图 RPC 会占用唯一捕获槽位,重新创建 app 对象不会重置该系统连接。
|
|
205
|
+
|
|
206
|
+
<a id="known-limitations-and-deferred-work"></a>
|
|
207
|
+
|
|
208
|
+
- 只提供 macOS 原生实现,Windows 支持正在开发中。本包暂放在 `experimental`,待 Windows 支持完成后迁回 `computer-use` 组。不支持浏览器标签、启动应用、AX 差分、无法确认应用归属的系统面板,以及仅后台辅助类型的应用。
|
|
209
|
+
- 坐标使用包含标题栏的返回截图像素;`relative: true` 使用千分比整数,由 Host 换算原生 point。窗口移动、应用变化及其他桌面使用者的操作可能与观察和输入交错。
|
|
210
|
+
- macOS 台前调度可能将后台窗口的尺寸与截图变为缩略图,当前无法保证这种状态下的完整画面与坐标输入;模型指引要求说明前台需求并避免反复抢回焦点。
|
|
211
|
+
- 后台输入依赖私有 SkyLight 符号,macOS 升级可能使其不可用,需在目标系统版本验证。
|
|
212
|
+
- 实测 Blender Unicode 输入曾出现分派成功但界面未显示文本,后台 AppKit 的 Command+A 曾未选中全文。输入后必须检查应用状态;工具返回成功只表示已分派。
|
|
213
|
+
- 原生二进制需要本地构建。签名、分发和系统权限引导留待后续。
|
|
214
|
+
- 持久 PTC 是对外部运行环境的假设,本包不实现它。真实模型任务成功率需要在原生测试和协议测试之外单独评估。
|
|
215
|
+
|
|
216
|
+
-----
|
|
217
|
+
|
|
218
|
+
<a id="dev-note"></a>
|
|
219
|
+
### 开发备注
|
|
220
|
+
|
|
221
|
+
<details>
|
|
222
|
+
<summary>维护者工作背景</summary>
|
|
223
|
+
|
|
224
|
+
无。
|
|
225
|
+
|
|
226
|
+
</details>
|