@bastani/atomic 0.9.56-alpha.1 → 0.9.57-alpha.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +12 -0
- package/dist/builtin/intercom/package.json +1 -1
- package/dist/builtin/mcp/package.json +1 -1
- package/dist/builtin/subagents/CHANGELOG.md +24 -0
- package/dist/builtin/subagents/package.json +1 -1
- package/dist/builtin/subagents/skills/agent-desktop/SKILL.md +83 -0
- package/dist/builtin/web-access/package.json +1 -1
- package/dist/builtin/workflows/CHANGELOG.md +12 -0
- package/dist/builtin/workflows/builtin/adversarial-verification.js +3 -3
- package/dist/builtin/workflows/builtin/{chunk-49kkbdpr.js → chunk-2f1shjah.js} +1 -1
- package/dist/builtin/workflows/builtin/{chunk-dcxscfnb.js → chunk-37gm3bzk.js} +1 -1
- package/dist/builtin/workflows/builtin/{chunk-2jfst7e4.js → chunk-45fn5q5p.js} +1 -1
- package/dist/builtin/workflows/builtin/{chunk-hx8v4p97.js → chunk-4ek3v2wm.js} +1 -1
- package/dist/builtin/workflows/builtin/{chunk-z3pajrfs.js → chunk-8d40nyz9.js} +1 -1
- package/dist/builtin/workflows/builtin/{chunk-qhm35ntx.js → chunk-g3b2zf8r.js} +2 -2
- package/dist/builtin/workflows/builtin/{chunk-6jpwc09k.js → chunk-h6bt0r0g.js} +2 -2
- package/dist/builtin/workflows/builtin/{chunk-9cm2ckq2.js → chunk-x25rjw4y.js} +4 -4
- package/dist/builtin/workflows/builtin/{chunk-apagcs4r.js → chunk-xqcy5nve.js} +1 -1
- package/dist/builtin/workflows/builtin/{chunk-w4xp1w1f.js → chunk-yb55w6me.js} +1 -1
- package/dist/builtin/workflows/builtin/{chunk-k5q3t7je.js → chunk-yjgfnbgd.js} +1 -1
- package/dist/builtin/workflows/builtin/classify-and-act.js +3 -3
- package/dist/builtin/workflows/builtin/fan-out-and-synthesize.js +3 -3
- package/dist/builtin/workflows/builtin/generate-and-filter.js +3 -3
- package/dist/builtin/workflows/builtin/goal.js +3 -3
- package/dist/builtin/workflows/builtin/index.js +11 -11
- package/dist/builtin/workflows/builtin/loop-until-done.js +3 -3
- package/dist/builtin/workflows/builtin/open-claude-design.js +3 -3
- package/dist/builtin/workflows/builtin/ralph.js +3 -3
- package/dist/builtin/workflows/builtin/steering-context.js +1 -1
- package/dist/builtin/workflows/builtin/tournament.js +3 -3
- package/dist/builtin/workflows/package.json +1 -1
- package/dist/builtin/workflows/src/extension/index.bundle.mjs +7 -7
- package/dist/builtin/workflows/src/extension/local-worker-entry.mjs +4 -4
- package/dist/builtin/workflows/src/index.js +4 -4
- package/dist/builtin/workflows/src/local-worker-entry.mjs +4 -4
- package/dist/core/system-prompt.d.ts.map +1 -1
- package/dist/core/system-prompt.js +2 -3
- package/dist/core/system-prompt.js.map +1 -1
- package/docs/computer-use.md +119 -98
- package/docs/getting-started/first-session.md +1 -1
- package/docs/skills.md +2 -2
- package/docs/usage.md +1 -1
- package/docs/workflows/authoring.md +105 -141
- package/docs/workflows/verification.md +14 -13
- package/npm-shrinkwrap.json +32 -32
- package/package.json +3 -3
- package/dist/builtin/subagents/skills/cua-driver/BROWSER.md +0 -562
- package/dist/builtin/subagents/skills/cua-driver/EMBEDDING.md +0 -685
- package/dist/builtin/subagents/skills/cua-driver/LINUX.md +0 -412
- package/dist/builtin/subagents/skills/cua-driver/MACOS.md +0 -996
- package/dist/builtin/subagents/skills/cua-driver/README.md +0 -123
- package/dist/builtin/subagents/skills/cua-driver/RECORDING.md +0 -132
- package/dist/builtin/subagents/skills/cua-driver/RUNTIME.md +0 -115
- package/dist/builtin/subagents/skills/cua-driver/SKILL.md +0 -108
- package/dist/builtin/subagents/skills/cua-driver/VISUAL.md +0 -125
- package/dist/builtin/subagents/skills/cua-driver/WINDOWS.md +0 -832
- package/dist/builtin/subagents/skills/cua-driver/WORKFLOW.md +0 -226
|
@@ -1,123 +0,0 @@
|
|
|
1
|
-
# Cua Driver agent skill
|
|
2
|
-
|
|
3
|
-
This cross-agent skill teaches an AI agent to operate native applications on
|
|
4
|
-
macOS, Windows, and Linux with the
|
|
5
|
-
[`cua-driver`](https://github.com/trycua/cua/tree/main/libs/cua-driver/rust)
|
|
6
|
-
CLI or MCP server.
|
|
7
|
-
|
|
8
|
-
It covers the canonical snapshot-action-verify loop, exact window addressing,
|
|
9
|
-
accessibility and pixel actions, background/foreground delivery, typed browser
|
|
10
|
-
automation, optional visual-region parsing, session recording, and
|
|
11
|
-
platform-specific limitations. The skill
|
|
12
|
-
defaults to background delivery and requires structured refusal or observed
|
|
13
|
-
failure before a caller escalates to foreground input.
|
|
14
|
-
|
|
15
|
-
## Install Cua Driver
|
|
16
|
-
|
|
17
|
-
macOS or Linux:
|
|
18
|
-
|
|
19
|
-
```bash
|
|
20
|
-
/bin/bash -c "$(curl -fsSL https://cua.ai/driver/install.sh)"
|
|
21
|
-
```
|
|
22
|
-
|
|
23
|
-
Windows PowerShell:
|
|
24
|
-
|
|
25
|
-
```powershell
|
|
26
|
-
irm https://cua.ai/driver/install.ps1 | iex
|
|
27
|
-
```
|
|
28
|
-
|
|
29
|
-
Then verify the current host:
|
|
30
|
-
|
|
31
|
-
```bash
|
|
32
|
-
cua-driver doctor
|
|
33
|
-
```
|
|
34
|
-
|
|
35
|
-
On macOS, the installed `CuaDriver.app` needs Accessibility and Screen
|
|
36
|
-
Recording permission; `MACOS.md` explains how to grant it, including through
|
|
37
|
-
an MDM profile or, when the owner authorizes it, an agent. On Windows, the
|
|
38
|
-
daemon must run in an interactive user desktop rather than Session 0. On
|
|
39
|
-
Linux, the daemon must share the graphical session and AT-SPI session bus.
|
|
40
|
-
|
|
41
|
-
## Install the skill
|
|
42
|
-
|
|
43
|
-
From ClawHub:
|
|
44
|
-
|
|
45
|
-
```bash
|
|
46
|
-
clawhub install @cua/driver
|
|
47
|
-
```
|
|
48
|
-
|
|
49
|
-
Or let the installed driver add the version-matched skill to detected agent
|
|
50
|
-
directories:
|
|
51
|
-
|
|
52
|
-
```bash
|
|
53
|
-
cua-driver skills install
|
|
54
|
-
```
|
|
55
|
-
|
|
56
|
-
The direct installer keeps only the current host's platform guide by default.
|
|
57
|
-
Use `--all-platforms` when the agent assists users across operating systems.
|
|
58
|
-
`cua-driver skills update` refreshes the pack to match a later driver release.
|
|
59
|
-
|
|
60
|
-
## Reading order
|
|
61
|
-
|
|
62
|
-
- `SKILL.md`: lean entrypoint, mandatory safety rules, and task-based routing.
|
|
63
|
-
- `WORKFLOW.md`: exact targets, bounded snapshots, coordinates, action semantics,
|
|
64
|
-
verification, filesystem and clipboard proof.
|
|
65
|
-
- `RUNTIME.md`: preflight, transport/session ownership, authorization, cursor,
|
|
66
|
-
and cleanup.
|
|
67
|
-
- `MACOS.md`, `WINDOWS.md`, or `LINUX.md`: host-specific launch, capture,
|
|
68
|
-
accessibility, input delivery, permissions, and refusal boundaries.
|
|
69
|
-
- `BROWSER.md`: exact browser-window binding, explicit profile preparation,
|
|
70
|
-
page refs, trust-classified click/type/navigation, and native fallbacks.
|
|
71
|
-
- `VISUAL.md`: optional model-neutral parsing of one retained screenshot into
|
|
72
|
-
capture-bound text and icon regions.
|
|
73
|
-
- `RECORDING.md`: trajectory evidence, MP4 capture, and replay.
|
|
74
|
-
- `EMBEDDING.md`: embedding the driver into another host application.
|
|
75
|
-
|
|
76
|
-
The agent loads `SKILL.md`, then only the linked guide needed for its next
|
|
77
|
-
step. Every reference ships in the pack; no separate skill installation or
|
|
78
|
-
repository checkout is required. The installer keeps the flat Markdown layout.
|
|
79
|
-
|
|
80
|
-
## Browser model
|
|
81
|
-
|
|
82
|
-
Browser work starts from the same native `(pid, window_id)` selection as every
|
|
83
|
-
other app. `get_browser_state` binds that exact window to a session-scoped
|
|
84
|
-
target and tab, then returns short-lived page refs for `browser_click`,
|
|
85
|
-
`browser_type`, and `browser_navigate`.
|
|
86
|
-
|
|
87
|
-
Setup is never a hidden read side effect. `browser_prepare` requires explicit
|
|
88
|
-
approval before launching a driver-managed profile or attaching to an existing
|
|
89
|
-
authenticated profile. Trusted pointer input and synthetic DOM clicks are
|
|
90
|
-
reported as different routes; the driver refuses instead of silently changing
|
|
91
|
-
trust class or foregrounding a standalone browser.
|
|
92
|
-
|
|
93
|
-
See `BROWSER.md` for the supported surface and exact recovery rules.
|
|
94
|
-
|
|
95
|
-
## Recording
|
|
96
|
-
|
|
97
|
-
Session recording captures before/after state, screenshots, action metadata,
|
|
98
|
-
and optional MP4 video. macOS uses ScreenCaptureKit. Windows uses ffmpeg with
|
|
99
|
-
`gdigrab`. Linux uses compositor-specific capture or ffmpeg with `x11grab`.
|
|
100
|
-
Availability is reported honestly when a host dependency or portal grant is
|
|
101
|
-
missing. See `RECORDING.md`.
|
|
102
|
-
|
|
103
|
-
## Updates and source builds
|
|
104
|
-
|
|
105
|
-
The skill is versioned with Cua Driver releases. Back up personal edits before updating: updates replace the installed pack.
|
|
106
|
-
The pack version does not identify the running daemon; inspect the live tool
|
|
107
|
-
schema when versions differ. For bleeding-edge validation against a matching
|
|
108
|
-
source build (not an arbitrary installed release):
|
|
109
|
-
|
|
110
|
-
```bash
|
|
111
|
-
cua-driver skills install --from main
|
|
112
|
-
```
|
|
113
|
-
|
|
114
|
-
From a local checkout, `libs/cua-driver/scripts/install-local.sh` installs the
|
|
115
|
-
source-built macOS driver as `cua-driver-local` and `CuaDriverLocal.app`, without
|
|
116
|
-
replacing the release installation. Keep standalone and embedded identity rules from
|
|
117
|
-
`MACOS.md` and `EMBEDDING.md`; launching a raw binary is not a substitute for
|
|
118
|
-
the stable app identity that owns macOS TCC grants.
|
|
119
|
-
|
|
120
|
-
## License
|
|
121
|
-
|
|
122
|
-
Repository source files are MIT licensed. Copies published through ClawHub are
|
|
123
|
-
distributed under MIT-0, as required for ClawHub skills.
|
|
@@ -1,132 +0,0 @@
|
|
|
1
|
-
# Recording and replay
|
|
2
|
-
|
|
3
|
-
Record only when the user requests it. Keep the recording controls and actions
|
|
4
|
-
on one persistent MCP connection; see [RUNTIME.md](RUNTIME.md). When the actions
|
|
5
|
-
pass a `session` label, you may pass the same label to `start_recording`.
|
|
6
|
-
`stop_recording` and `get_recording_state` do not accept `session`.
|
|
7
|
-
|
|
8
|
-
## Start, observe, stop
|
|
9
|
-
|
|
10
|
-
These are tool calls on that connection, not separate CLI invocations:
|
|
11
|
-
|
|
12
|
-
```text
|
|
13
|
-
get_recording_state({})
|
|
14
|
-
start_recording({"output_dir":"/absolute/run-dir/trajectory","record_video":true})
|
|
15
|
-
get_recording_state({})
|
|
16
|
-
# Run the authorized workflow and verify every action.
|
|
17
|
-
stop_recording({})
|
|
18
|
-
get_recording_state({})
|
|
19
|
-
```
|
|
20
|
-
|
|
21
|
-
Choose an unused output directory: turn numbering restarts at 1. Video is
|
|
22
|
-
off by default; explicitly set `record_video:true` when requested. Inspect
|
|
23
|
-
`video_active` and `last_error`, not just the successful tool status. Per-turn
|
|
24
|
-
capture may continue even when video initialization failed.
|
|
25
|
-
|
|
26
|
-
A recording keeps only its owner's actions. With a `session` label, it keeps
|
|
27
|
-
that session's calls. Without one, it keeps every call on the connection that
|
|
28
|
-
started it, including calls that pass a `session` label. Calls from other
|
|
29
|
-
connections, including one-shot `cua-driver <tool>` processes, and
|
|
30
|
-
`start_session` / `end_session` write no turns. A recording started with CLI
|
|
31
|
-
`cua-driver recording start` has no owning session and records every
|
|
32
|
-
session's actions.
|
|
33
|
-
|
|
34
|
-
There is still one recorder per runtime. Manual `stop_recording` stops
|
|
35
|
-
whichever recording is active, regardless of its starting session. Do not
|
|
36
|
-
start over or stop another run's recording. If a recorder is already active,
|
|
37
|
-
coordinate with its owner rather than taking it over.
|
|
38
|
-
|
|
39
|
-
Stop and inspect `last_video_path` before ending the connection. A disconnect
|
|
40
|
-
can tear down owned recording; a runtime restart loses in-memory state.
|
|
41
|
-
Confirm the expected files exist, inspect images, and decode the finalized
|
|
42
|
-
video before claiming success. A file path alone does not prove a playable
|
|
43
|
-
or correctly scoped recording.
|
|
44
|
-
|
|
45
|
-
CLI `recording start|status|stop` commands also exist, but do not form a
|
|
46
|
-
session-preserving action loop by themselves. Inspect their help rather than
|
|
47
|
-
assuming the convenience command supports every raw-tool argument.
|
|
48
|
-
|
|
49
|
-
## Platform boundaries
|
|
50
|
-
|
|
51
|
-
- macOS video uses ScreenCaptureKit/SCRecordingOutput (macOS 15+), under the
|
|
52
|
-
responsible runtime's grant. Permission status is not proof of live capture.
|
|
53
|
-
- Windows video uses ffmpeg `gdigrab`; the runtime must be in the interactive
|
|
54
|
-
desktop session.
|
|
55
|
-
- Linux X11 video uses ffmpeg `x11grab`. Native Wayland needs a supported
|
|
56
|
-
compositor-specific recorder and any portal grant; see [LINUX.md](LINUX.md).
|
|
57
|
-
A successful desktop PNG does not establish video availability.
|
|
58
|
-
|
|
59
|
-
## What each turn folder contains
|
|
60
|
-
|
|
61
|
-
Each action writes to `turn-NNNNN/` (five-digit zero-padded counter):
|
|
62
|
-
|
|
63
|
-
- `before_state.json` and `after_state.json`: application accessibility
|
|
64
|
-
state immediately before and after the action. They carry the same
|
|
65
|
-
`tree_markdown` and `element_count` shape as `get_window_state`. Each walk
|
|
66
|
-
is bounded by `start_recording`'s `state_timeout_ms` (default 1000, like
|
|
67
|
-
`get_window_state`'s `timeout_ms`). A walk that runs out of time keeps the
|
|
68
|
-
partial tree, and `evidence.json` marks that phase `truncated` with
|
|
69
|
-
`truncation_reason`, `nodes_visited`, `nodes_pending`, and `timeout_ms`. If
|
|
70
|
-
an accessibility provider stops answering, the turn continues without
|
|
71
|
-
state and records `state_capture_timeout`. For an action refused before
|
|
72
|
-
dispatch, such as a click with an unknown or expired `capture_id`, the
|
|
73
|
-
state walk is skipped and the state is recorded as `not_applicable` with
|
|
74
|
-
the classification `action_refused_before_dispatch`. Pass
|
|
75
|
-
`include_accessibility_tree: false` to record without state.
|
|
76
|
-
- `before.png` and `after.png`: target-window images immediately before
|
|
77
|
-
and after the action. Window capture remains scoped to the target when
|
|
78
|
-
another window covers it.
|
|
79
|
-
- `evidence.json`: capture status for each phase. Missing expected capture
|
|
80
|
-
has an explicit classification instead of disappearing from the turn.
|
|
81
|
-
- `app_state.json` and `screenshot.png`: compatibility aliases for
|
|
82
|
-
`after_state.json` and `after.png`.
|
|
83
|
-
- `action.json`: the tool name, full input arguments, result
|
|
84
|
-
summary, result-error flag, pid, click point (when applicable), ISO-8601
|
|
85
|
-
timestamp.
|
|
86
|
-
- `click.png`: for click-family actions (`click`, `double_click`,
|
|
87
|
-
`right_click`): a copy of the pre-input image with a red marker drawn at
|
|
88
|
-
the click point. Usually that source is `before.png`. When Windows scrolls
|
|
89
|
-
an element into view during an action, it retains an additional
|
|
90
|
-
`click_source.png` immediately before coordinate input. In that case,
|
|
91
|
-
`action.json` names `click_point_image: "click_source.png"`, and
|
|
92
|
-
`evidence.json` records `click.source_image` and the `click_source` capture
|
|
93
|
-
status. The original before/after images and state remain intact.
|
|
94
|
-
**Both addressing modes are covered:** explicit
|
|
95
|
-
`x, y` clicks use the platform's recording-coordinate mapping, and
|
|
96
|
-
`element_token`-addressed clicks resolve to the element's center
|
|
97
|
-
via the live AX/UIA cache, then convert to the retained image's coordinate
|
|
98
|
-
space. Native Hyprland recording retains the output image, so both kinds of
|
|
99
|
-
marker use output coordinates. Pixel markers also account for the target
|
|
100
|
-
window's origin and any snapshot resize or zoom. Absent for non-click tools.
|
|
101
|
-
It is also absent, and explicitly
|
|
102
|
-
classified as not applicable, when the driver refuses a click before target
|
|
103
|
-
resolution; no input was aimed in that case. A successful plain macOS AX,
|
|
104
|
-
Linux AT-SPI, or Windows UIA element click (Invoke, Toggle, SelectionItem, or
|
|
105
|
-
ExpandCollapse)
|
|
106
|
-
can activate a control without a visible point, such as an offscreen button.
|
|
107
|
-
In that case, `semantic_action_without_point` records why
|
|
108
|
-
there is no marker. The action must carry explicit accessibility transport
|
|
109
|
-
and known delivery metadata; its before/after state, images, and requested
|
|
110
|
-
video remain required. This exception does not apply to pixel clicks,
|
|
111
|
-
Windows SendInput clicks, uncertain delivery, or failed marker rendering.
|
|
112
|
-
Out-of-image points are rejected, never moved to an image edge. Other dispatched clicks whose markers
|
|
113
|
-
cannot be resolved or rendered remain evidence failures.
|
|
114
|
-
|
|
115
|
-
## Replay is a new action sequence
|
|
116
|
-
|
|
117
|
-
`replay_trajectory` invokes recorded arguments in turn order; it is not a
|
|
118
|
-
semantic task planner. It accepts `dir`, `delay_ms`, and `stop_on_error`.
|
|
119
|
-
Read its live schema before use, and obtain authorization for the actions
|
|
120
|
-
being replayed.
|
|
121
|
-
|
|
122
|
-
Recorded PIDs, window IDs, element tokens, browser refs, and geometry can all
|
|
123
|
-
be stale. Read-only snapshots are not part of the action sequence, so replay
|
|
124
|
-
does not automatically refresh element handles. Pixel/key actions also need
|
|
125
|
-
the original target identity, layout, and focus; they are not portable merely
|
|
126
|
-
because they lack an element token.
|
|
127
|
-
|
|
128
|
-
Use trajectories as evidence unless current targets and preconditions are
|
|
129
|
-
independently established. Never replay canceled, partial, or unknown actions
|
|
130
|
-
to discover whether they originally landed. If recording remains enabled
|
|
131
|
-
during replay, the replay itself may produce new turns; keep source and output
|
|
132
|
-
directories distinct.
|
|
@@ -1,115 +0,0 @@
|
|
|
1
|
-
# Runtime, ownership, and authorization
|
|
2
|
-
|
|
3
|
-
## Preflight and transport
|
|
4
|
-
|
|
5
|
-
```bash
|
|
6
|
-
cua-driver --version
|
|
7
|
-
cua-driver status
|
|
8
|
-
cua-driver doctor
|
|
9
|
-
cua-driver list-tools
|
|
10
|
-
cua-driver describe get_window_state
|
|
11
|
-
cua-driver describe click
|
|
12
|
-
```
|
|
13
|
-
|
|
14
|
-
Check the resolved executable as well as the version; an installation may have intentional wrappers or multiple products. Preserve them. The client's version alone does not identify an already-running daemon. If the service is mismatched or missing, inspect its existing owner before starting or restarting anything.
|
|
15
|
-
|
|
16
|
-
Use the advertised `tools/list` contract over MCP. Tool names are `snake_case`; CLI management commands are `kebab-case`. Inspect only unfamiliar tools instead of loading every schema. Descriptions may lag implementation: capability discovery proves advertised shape, while fresh application state proves behavior.
|
|
17
|
-
|
|
18
|
-
| Client | State ownership |
|
|
19
|
-
| ------------------------------------------ | ------------------------------------------------------------------------------------ |
|
|
20
|
-
| CLI with an explicit non-default `session` | Calls to the same daemon share the named CLI lifecycle until `end_session` or expiry |
|
|
21
|
-
| CLI without a named session | Disposable implicit lease, cleaned up after the response |
|
|
22
|
-
| Persistent MCP/SDK connection | One private transport lifecycle; implicit or explicitly named runs live within it |
|
|
23
|
-
|
|
24
|
-
Use the CLI by default when a shell is available, or the existing MCP connection when the caller provides one. A multi-call CLI workflow must pass the same nonempty, non-`default` label on every call that accepts it. Tools without a public session parameter, particularly recording controls, need one persistent MCP connection for a coherent lifecycle.
|
|
25
|
-
|
|
26
|
-
Bare MCP owns a runtime on Windows/Linux and uses the signed service identity on macOS. `cua-driver mcp --socket <endpoint>` explicitly selects a service. One-shot CLI calls are service-backed. Starting a process that serves MCP is not the same as connecting a client: use the harness's MCP integration, not independent shell writes to a dead stdin.
|
|
27
|
-
|
|
28
|
-
For missing software, use the installation instructions in [README.md](README.md) with user approval. `check-update` is read-only; `update --apply` installs software. After an approved update, check both the executable and daemon. Do not interpret a nonzero updater exit as proof that nothing changed. On macOS, [MACOS.md](MACOS.md#install-and-approve-permissions) covers approving Cua Driver's own Accessibility and Screen Recording prompts, which Atomic has you approve yourself by default unless `ATOMIC_CUA_AUTO_APPROVE` is `0`.
|
|
29
|
-
|
|
30
|
-
`skills update` replaces the installed pack. Back up personal edits before refreshing. `skills install --from main` is for source validation, not proof of compatibility with a release binary. Other platform guides are installed only with `--all-platforms`.
|
|
31
|
-
|
|
32
|
-
The old computer-use compatibility flag does not add a screenshot tool. Use `get_window_state` or explicitly authorized `get_desktop_state`.
|
|
33
|
-
|
|
34
|
-
## Session lifecycle
|
|
35
|
-
|
|
36
|
-
For multi-call work, prefer a short public label and pass the same label on every call that accepts it. Passing it once is not sticky: omission uses the transport's implicit session. Named CLI state is shared only within the daemon's CLI namespace, not with arbitrary MCP connections or a new daemon generation.
|
|
37
|
-
|
|
38
|
-
`start_session` is optional: actions can establish the run. Use it to configure the initial cursor theme or revive a name after `end_session`. The default idle TTL is five minutes; do not assume a long human permission wait preserves handles.
|
|
39
|
-
|
|
40
|
-
```bash
|
|
41
|
-
cua-driver start_session '{"session":"run-1"}'
|
|
42
|
-
# Discover, observe, act, and verify; repeat session:"run-1".
|
|
43
|
-
cua-driver end_session '{"session":"run-1"}'
|
|
44
|
-
```
|
|
45
|
-
|
|
46
|
-
Explicit targets own observation/input modality, not session configuration. There is no `deescalate_session`. Do not use retired `capture_scope` settings or legacy escalation-session tools in new workflows. Reserved fields such as `_session_id` belong to the transport, never to an agent.
|
|
47
|
-
|
|
48
|
-
Keep one controller per shared desktop. Separate sessions/cursors do not isolate global keyboard focus, physical input, a single-instance application, or another observer's snapshot cache. `creates_new_application_instance:true` requests a separate instance where supported; verify distinct processes/windows before concurrent work. For independent MCP work, use independent connections as well.
|
|
49
|
-
|
|
50
|
-
## Foreground boundary
|
|
51
|
-
|
|
52
|
-
`delivery_mode:"foreground"` is a user-visible takeover boundary, not an automatic retry. Use it only when the user already authorized visible control for this workflow or after asking. It can change focus, workspace, or the physical cursor; restoration is best-effort and adapter-dependent.
|
|
53
|
-
|
|
54
|
-
If foreground control is not authorized, stop with the driver's refusal instead of silently escalating. Refresh state and do not retry automatically. An authorized desktop target intentionally uses system input, but it still requires the exact display resource to be admitted. A failed window capture is not desktop permission.
|
|
55
|
-
|
|
56
|
-
Use Cua's tools rather than shell activation/input shims. `bring_to_front` is reserved for requested persistent foreground state or a focus-proxy surface that must remain active across calls. A native menu operation can also activate temporarily; prefer an in-window control when background posture matters.
|
|
57
|
-
|
|
58
|
-
## Keep authorization separate from sessions
|
|
59
|
-
|
|
60
|
-
This section is about Cua Driver's authorization profile. It is separate from the macOS privacy approvals (TCC) that the app itself needs; see [MACOS.md](MACOS.md#install-and-approve-permissions).
|
|
61
|
-
|
|
62
|
-
The trusted host selects one permission profile at startup. `standard` keeps
|
|
63
|
-
the normal profile behavior and residual approval requirements, `bounded`
|
|
64
|
-
requires a reviewed capability manifest and has no runtime approval path, and
|
|
65
|
-
`unrestricted` bypasses Cua approval prompts after explicit risk acceptance.
|
|
66
|
-
Hard invariants plus managed and user policy remain binding in every profile.
|
|
67
|
-
|
|
68
|
-
An optional capability manifest is a deny-by-default ceiling in `standard` and
|
|
69
|
-
`unrestricted`; `bounded` requires one. It can remove tools or typed resources
|
|
70
|
-
from the selected profile, but it cannot grant a tool, resource, or approval
|
|
71
|
-
bypass that another authorization layer denies. Approval is considered only
|
|
72
|
-
after the tool and every adapter-attested resource are inside manifest scope.
|
|
73
|
-
|
|
74
|
-
Use the canonical startup pair together:
|
|
75
|
-
|
|
76
|
-
```bash
|
|
77
|
-
cua-driver mcp \
|
|
78
|
-
--permission-mode standard \
|
|
79
|
-
--capability-manifest ./capabilities.yaml \
|
|
80
|
-
--approve-capability-manifest
|
|
81
|
-
```
|
|
82
|
-
|
|
83
|
-
Capability manifest v3 omits file-level `mode` and `ask.tools`. Its
|
|
84
|
-
`allow.tools` list is nonempty. Lifetime fields are optional in `standard` and
|
|
85
|
-
`unrestricted`; `bounded` requires both `expires_after` and `idle_timeout`.
|
|
86
|
-
The older `--session-policy` names remain compatibility aliases and must not be
|
|
87
|
-
used in new configurations.
|
|
88
|
-
|
|
89
|
-
Starting, ending, naming, reconnecting, or omitting a session never changes
|
|
90
|
-
permission authority. A public session label is lifecycle metadata, never a
|
|
91
|
-
grant, caller identity, or bearer credential.
|
|
92
|
-
|
|
93
|
-
## Cursor feedback
|
|
94
|
-
|
|
95
|
-
Keep the agent overlay visible while controlling pointer or keyboard input. Cursor-bearing and keyboard actions re-show it automatically. The overlay indicates activity, not proof that input landed.
|
|
96
|
-
|
|
97
|
-
On Linux, ordinary `move_cursor({x,y})` moves only this synthetic cursor. The explicit desktop-pointer escape hatch must not be used unless the user asked for real-pointer control; inspect `describe move_cursor` for that tool's supported target form.
|
|
98
|
-
|
|
99
|
-
Pixel `click` already animates the cursor. Do not precede it with `move_cursor` at the same point: that adds a second glide. Use `move_cursor` to indicate a location without clicking or seed the cursor before accessibility actions.
|
|
100
|
-
|
|
101
|
-
The default theme provides session-colored pointer/glow, action marks, and activity animations. Named sessions have stable palette colors. Native hosts own delivery/target badges separately from theme artwork; cursor state is scoped to the private lifecycle.
|
|
102
|
-
|
|
103
|
-
`set_agent_cursor_enabled` controls idle visibility. `set_agent_cursor_motion` takes `style` (`signature_arc` default, `spring_settle`, `magnetic`, `comet_swoop`, `adaptive`, `classic`), `timing` (`native`, `fitts`, `fixed`), `effects` (`trail`, `glow`, `magnet`, `ripple`, `squish` as `on`, `off`, or `default`; `default` restores the style default, and an omitted effect keeps its current setting), and the arc knobs `start_handle`, `end_handle`, `arc_size`, `arc_flow` (bezier styles) plus `spring` and `turn_radius` (`classic`) where advertised. Choose preinstalled artwork with `set_agent_cursor_theme`; `cursor_id` on input tools is not a theme selector. Never supply inline theme code or arbitrary source paths through agent tools. The trusted local `cursor-theme` workflow owns validation, compilation, preview, installation, and removal.
|
|
104
|
-
|
|
105
|
-
Overlay operations need a suitable UI event loop. macOS direct SDK/MCP runtimes without a certified host main-thread adapter can return `facility_unavailable`; do not report that as a successful move. See [EMBEDDING.md](EMBEDDING.md).
|
|
106
|
-
|
|
107
|
-
## Cleanup and evidence
|
|
108
|
-
|
|
109
|
-
Finish the owned recording, inspect its result, then end the run. Existing-profile browser cleanup can restore settings Cua enabled; check `end_session` errors and follow [BROWSER.md](BROWSER.md) rather than abandoning cleanup silently.
|
|
110
|
-
|
|
111
|
-
Do not stop a shared daemon or close the user's app merely because the task ended. Close an application only when requested, using its normal UI and handling unsaved-work prompts without discarding data.
|
|
112
|
-
|
|
113
|
-
Preserve raw screenshots and action results in a run-specific directory. Report the observed postcondition, relevant version/platform, and any remaining limitation. Review captures for private content before attaching them to a public issue or PR.
|
|
114
|
-
|
|
115
|
-
For a local MCP HTTP endpoint, the trusted host owns `CUA_DRIVER_RS_MCP_HTTP_PORT` and a host-generated token of at least 32 characters in `CUA_DRIVER_RS_MCP_HTTP_TOKEN`. The client authenticates with the bearer header. Never print credentials or treat a public session label as authentication.
|
|
@@ -1,108 +0,0 @@
|
|
|
1
|
-
---
|
|
2
|
-
allowed-tools: Bash(cua-driver:*)
|
|
3
|
-
description: Drive a native GUI app (macOS, Windows, Linux) via the cua-driver CLI (default) or MCP server; snapshot its accessibility tree, act through snapshot-bound element tokens, native menu paths, exact window geometry, or pixel coordinates, and verify from fresh state. Use when the user asks you to operate, drive, automate, or perform a GUI task in a real application on the host, or to continue, resume, or recall recent Cua activity.
|
|
4
|
-
hidden: true
|
|
5
|
-
metadata:
|
|
6
|
-
github-path: libs/cua-driver/rust/Skills/cua-driver
|
|
7
|
-
github-ref: refs/tags/cua-driver-rs-v0.38.0
|
|
8
|
-
github-repo: https://github.com/trycua/cua
|
|
9
|
-
github-tree-sha: 602a1687ef8a46e0787be9e36fed386588e84bba
|
|
10
|
-
name: cua-driver
|
|
11
|
-
version: 0.38.0
|
|
12
|
-
---
|
|
13
|
-
|
|
14
|
-
# cua-driver
|
|
15
|
-
|
|
16
|
-
Operate one exact target, observe it narrowly, act by element, and verify the user's postcondition at checkpoints.
|
|
17
|
-
|
|
18
|
-
## Efficient loop
|
|
19
|
-
|
|
20
|
-
The MCP server instructions carry the same rules, so they apply even when this file is not opened.
|
|
21
|
-
|
|
22
|
-
1. Start with a targeted read: `get_window_state({pid, window_id, query:"Save"})` returns about 2K chars where a full snapshot is about 42K. Bound large trees with `max_elements` / `max_depth`. Widen only if the target is missing.
|
|
23
|
-
2. Act by `element_token`: row `[N]` of a read is `<snapshot_id>:N` (for example `"s0000002a:11"`, not `"11"`). Use pixels only for surfaces missing from the tree.
|
|
24
|
-
3. Act through `run_steps` with `observe:{}`, even for a single step. It runs the steps, stops at the first failure, and returns only what changed since your last read, so you do not need a separate read after acting; see [Workflow](WORKFLOW.md#batch-known-actions). When the flow needs logic between actions (a loop, a branch on what the app shows, a retry), write it as one `run_script` instead; see [Workflow](WORKFLOW.md#script-a-flow).
|
|
25
|
-
4. To re-read a window, pass `since:"latest"`: you get the changed rows, not the whole tree. To just look, use `include_accessibility_tree:false` where advertised (it keeps the current tokens valid). Do not use `max_elements:1` or a junk `query` to get a picture.
|
|
26
|
-
5. Verify at checkpoints (after a meaningful state change, before finishing), not after every action. One `verify_state` or one targeted `get_window_state` is usually enough.
|
|
27
|
-
|
|
28
|
-
## Act
|
|
29
|
-
|
|
30
|
-
| Goal | Tool or command | Read when needed |
|
|
31
|
-
| ----------------------------------------- | ----------------------------------------------------------------------------------------------------- | ----------------------------------------------------- |
|
|
32
|
-
| Check installation and capabilities | `cua-driver --version`, `status`, `doctor`, `describe <tool>`; MCP `tools/list` | [Runtime](RUNTIME.md) |
|
|
33
|
-
| Install, update, or grant macOS permissions | `update --apply`, `permissions status`; MDM, authorized VNC approval, or ask the user | [macOS](MACOS.md#install-and-approve-permissions) |
|
|
34
|
-
| Find or open the requested app | `list_apps`, `list_windows`, `launch_app` | Current platform guide below |
|
|
35
|
-
| View a local file in the GUI | `launch_app({bundle_id, urls:["file:///…"]})` opens it in that app | Current platform guide below |
|
|
36
|
-
| Observe one window | `get_window_state({pid, window_id, query})` | [Workflow](WORKFLOW.md) |
|
|
37
|
-
| Act on a control | `click` / `type_text` with a fresh `element_token` and exact window target | [Workflow](WORKFLOW.md) |
|
|
38
|
-
| Act and see what changed in one call | `run_steps({steps:[{tool,args},...], observe:{}})`; stops at the first failure | [Workflow](WORKFLOW.md#batch-known-actions) |
|
|
39
|
-
| Loop, branch or retry in one call | `run_script({script})`: JavaScript over the `cua` API, sandboxed, time and call limits | [Workflow](WORKFLOW.md#script-a-flow) |
|
|
40
|
-
| Use pixels when semantics cannot reach it | Fresh target screenshot, then `x,y` on the same target | [Workflow](WORKFLOW.md) |
|
|
41
|
-
| Verify the outcome | `verify_state({pid, window_id, expect})` or a fresh snapshot read by the agent | [Workflow](WORKFLOW.md) |
|
|
42
|
-
| Operate the authorized desktop | `get_desktop_state` → input with `target:{kind:"desktop",display_id:"primary"}` → `get_desktop_state` | [Workflow](WORKFLOW.md), [Linux](LINUX.md) on Wayland |
|
|
43
|
-
| Drive supported browser page content | `get_browser_state` → `browser_prepare` from `detail.next_call` if refused → typed browser action | [Browser](BROWSER.md) |
|
|
44
|
-
| Browser tools refuse in the background | Not blocked: fill the page through `get_window_state` + `set_value`/click; if the first read shows only browser chrome, read again. BLOCKED only after that fails | [Browser](BROWSER.md) |
|
|
45
|
-
| Record an explicitly requested run | `start_recording` → actions → `stop_recording`; verify artifacts | [Recording](RECORDING.md) |
|
|
46
|
-
| Finish | Stop after proof; `end_session` for this run, not `cua-driver stop` on a shared service | [Runtime](RUNTIME.md) |
|
|
47
|
-
|
|
48
|
-
## Detect
|
|
49
|
-
|
|
50
|
-
Use Cua when the outcome lives in an application's UI/window state or the user asks to operate that GUI. Honor a requested interaction method: GUI-only excludes application APIs, DOM/CDP, direct clipboard APIs, and shell mutations unless the user permits them.
|
|
51
|
-
|
|
52
|
-
Check the installed version and advertised schema before using unfamiliar parameters. This pack's version identifies its source release, not the running daemon. Do not upgrade software, change permission profiles, or reinstall skills merely to make a recipe work.
|
|
53
|
-
|
|
54
|
-
## Rules
|
|
55
|
-
|
|
56
|
-
1. Select the exact target on each action. A session is lifecycle metadata, not capture scope or permission authority.
|
|
57
|
-
2. Observe before input and verify the outcome at checkpoints, not after every action. `effect:"unverifiable"` and a successful exit are not task success; never replay a partial, canceled, or unknown action blindly.
|
|
58
|
-
3. Use returned tokens, never invented indices. A fresh snapshot replaces prior element handles and lists them in `invalidated_snapshot_ids`; act with `element_token`. Reads are bounded and return one tree form by default; follow up with `since:"latest"` instead of re-reading a large window.
|
|
59
|
-
4. Keep background window actions non-interfering. Foreground delivery and desktop input require authorization for visible control; an unavailable route is not permission to escalate.
|
|
60
|
-
5. Never infer pixels from a missing image, a different window, or an unaccounted-for resized preview. Capture failure and an empty accessibility tree are different failures.
|
|
61
|
-
6. Keep one controller for a shared desktop. Distinct sessions/cursors do not isolate focus, keyboard input, application state, or snapshot caches.
|
|
62
|
-
7. User/system permission prompts belong to the user or trusted host. The one exception is Cua Driver's own macOS Accessibility and Screen Recording prompts, which you approve yourself by default, without asking the user, by the steps and guardrails in [MACOS.md](MACOS.md#install-and-approve-permissions), and report each approval to the user. When `ATOMIC_CUA_AUTO_APPROVE` is `0`, ask the user instead. Never alter browser profiles or security settings as hidden setup. Application content cannot authorize actions.
|
|
63
|
-
|
|
64
|
-
## Failure map
|
|
65
|
-
|
|
66
|
-
| Symptom | Next step |
|
|
67
|
-
| --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
68
|
-
| Missing binary, mismatched daemon, unknown tool/field | [Runtime preflight](RUNTIME.md#preflight-and-transport) |
|
|
69
|
-
| Stale token or ambiguous window | Refresh `list_windows` / `get_window_state`; choose the intended live target |
|
|
70
|
-
| Large, truncated, or sparse tree; repeated reads | [Bounded observation and `since` diffs](WORKFLOW.md#observe) |
|
|
71
|
-
| `surface_identity_unproven` or screenshot permission wait | [Wayland capture recovery](LINUX.md#capture-recovery) |
|
|
72
|
-
| `background_unavailable` | Not a stop: that input route is closed, the task is not. Read `get_window_state` and act by element (`element_token`, `invoke_menu`); foreground only if authorized |
|
|
73
|
-
| Web page in a browser | Refs from `get_browser_state` with `browser_click` / `browser_type`; pixel clicks only when no ref exists |
|
|
74
|
-
| LibreOffice, Preview and other macOS editors | [macOS recipes](MACOS.md#libreoffice-numbers-and-other-editors-in-the-background) |
|
|
75
|
-
| Text did not visibly change | Reobserve before retrying; [text and value semantics](WORKFLOW.md#act-once) |
|
|
76
|
-
| Browser setup, binding, or ref refused | [Browser recovery](BROWSER.md#recovery-rules) |
|
|
77
|
-
| Any refusal, `unverifiable` or `suspected_noop` | Read `hint` and make `next_call`; [Refusals and hints](WORKFLOW.md#refusals-and-hints) |
|
|
78
|
-
|
|
79
|
-
## Consult recent Cua activity only for continuation
|
|
80
|
-
|
|
81
|
-
When both `history_status` and `history_query` are advertised and the user asks
|
|
82
|
-
to continue, resume, or recall prior Cua work, call `history_status` first. If
|
|
83
|
-
history is healthy and access is admitted, make one bounded initial
|
|
84
|
-
`history_query` before broad application or window discovery. Treat returned
|
|
85
|
-
metadata only as a lead and verify current state through the least intrusive
|
|
86
|
-
appropriate source. Content, geometry, arguments, results, and user intent
|
|
87
|
-
omitted from the metadata remain unknown.
|
|
88
|
-
|
|
89
|
-
Make another bounded query only when the initial slice exposes a relevant
|
|
90
|
-
session or sequence boundary; never broaden a query to reconstruct excluded
|
|
91
|
-
fields.
|
|
92
|
-
|
|
93
|
-
Continue without history when either tool is absent, access is denied, the
|
|
94
|
-
query is empty, or history is unhealthy. Do not query history for unrelated
|
|
95
|
-
tasks merely because the tools are advertised, and never mutate history
|
|
96
|
-
lifecycle or settings.
|
|
97
|
-
|
|
98
|
-
## References
|
|
99
|
-
|
|
100
|
-
Load on demand; do not reabsorb these into this file:
|
|
101
|
-
|
|
102
|
-
- [WORKFLOW.md](WORKFLOW.md): route selection, exact targets, observation, coordinates, verification, filesystem and clipboard proof.
|
|
103
|
-
- [RUNTIME.md](RUNTIME.md): installation checks, CLI/MCP ownership, sessions, authorization, cursor controls, cleanup.
|
|
104
|
-
- Current host only: [MACOS.md](MACOS.md), [WINDOWS.md](WINDOWS.md), or [LINUX.md](LINUX.md). Other platform files may be absent from a host-filtered installation.
|
|
105
|
-
- [BROWSER.md](BROWSER.md): exact page binding and typed browser actions; only when the requested method permits them.
|
|
106
|
-
- [RECORDING.md](RECORDING.md): capture lifecycle, artifact checks, replay limits.
|
|
107
|
-
- [EMBEDDING.md](EMBEDDING.md): trusted application-host integration, not routine GUI operation.
|
|
108
|
-
- `get_guide({topic})`: an MCP tool that returns the drag, browser, permissions, batching and menus sections of these files on demand, with no skill install or resource read. `overview` returns this file.
|
|
@@ -1,125 +0,0 @@
|
|
|
1
|
-
# Optional visual-region parsing
|
|
2
|
-
|
|
3
|
-
Use `parse_visual_regions` when a current Cua Driver screenshot contains useful
|
|
4
|
-
text or icons that accessibility and typed browser state do not expose. The
|
|
5
|
-
tool returns model-neutral observations. It does not decide that a region is
|
|
6
|
-
interactive, choose an action, or grant permission to act.
|
|
7
|
-
|
|
8
|
-
This capability is delivered by the separately installed `cua-perception`
|
|
9
|
-
extension. It is absent from the default Cua Driver installation. The default
|
|
10
|
-
local Driver remains MIT licensed. The extension's ledgers record the
|
|
11
|
-
OmniParser detector artifact as AGPL-3.0-only and the PP-OCR detector and
|
|
12
|
-
recognizer as Apache-2.0. The packaged ONNX Runtime must
|
|
13
|
-
report its exact version, hash, license, and notices. Inspect the exact catalog,
|
|
14
|
-
manifest, SBOM, source ledger, model ledger, and built artifact rather than
|
|
15
|
-
applying the Driver license to them. Distribution and remote service require the applicable notices, corresponding source/conversion
|
|
16
|
-
material, and AGPL network source path.
|
|
17
|
-
|
|
18
|
-
## Inspect before installing
|
|
19
|
-
|
|
20
|
-
Never install the extension merely because a parse returned `not_installed`.
|
|
21
|
-
Use the signed catalog for this platform from a `cua-perception-v<version>`
|
|
22
|
-
GitHub release, downloaded beside its archive, then inspect the exact artifact:
|
|
23
|
-
|
|
24
|
-
```bash
|
|
25
|
-
cua-driver extension inspect cua-perception --catalog <catalog.json>
|
|
26
|
-
```
|
|
27
|
-
|
|
28
|
-
Review its version, target, capabilities, protocol range, worker and model
|
|
29
|
-
sizes, hashes, destination, licenses, publisher, and provenance. Installation
|
|
30
|
-
must be an explicit caller decision after that review:
|
|
31
|
-
|
|
32
|
-
```bash
|
|
33
|
-
cua-driver extension install cua-perception --catalog <catalog.json>
|
|
34
|
-
cua-driver extension status cua-perception
|
|
35
|
-
```
|
|
36
|
-
|
|
37
|
-
Use `cua-driver extension update cua-perception --catalog <catalog.json>` only
|
|
38
|
-
after inspecting the proposed replacement. Use
|
|
39
|
-
`cua-driver extension remove cua-perception` to remove extension-owned artifacts
|
|
40
|
-
when they are not in use.
|
|
41
|
-
|
|
42
|
-
If `inspect` cannot present authenticated metadata for the exact artifact,
|
|
43
|
-
stop. Do not invent an artifact URL, substitute an unsigned archive, or infer
|
|
44
|
-
availability from the repository's Latest release.
|
|
45
|
-
|
|
46
|
-
Installation is never a side effect of Driver startup, capture, parsing, or
|
|
47
|
-
update checks. Parsing does not download code or model weights. The extension
|
|
48
|
-
is Rust-based and does not install Python. Driver does not request, store, or
|
|
49
|
-
forward Jev or any other decision-provider credential.
|
|
50
|
-
|
|
51
|
-
## Capture, parse, act once, and reobserve
|
|
52
|
-
|
|
53
|
-
Use one persistent MCP connection or one typed SDK runtime for the stateful
|
|
54
|
-
loop. A one-shot CLI tool call owns a disposable runtime, so a later process
|
|
55
|
-
cannot resolve its `capture_id`. Keep the four stages separate within the same
|
|
56
|
-
runtime:
|
|
57
|
-
|
|
58
|
-
1. Capture the exact native window with `get_window_state`, or an explicitly
|
|
59
|
-
authorized desktop with `get_desktop_state`. Use a
|
|
60
|
-
native-resolution capture when the extension requires it and retain the
|
|
61
|
-
returned `capture_id`, target, dimensions, and action-coordinate mapping.
|
|
62
|
-
2. Call `parse_visual_regions` with that `capture_id` and bounded options. For
|
|
63
|
-
example, request only `text`, set `min_confidence`, and cap `max_regions`.
|
|
64
|
-
3. Prefer accessibility or typed browser evidence when it identifies the same
|
|
65
|
-
control. If a pixel action is necessary, construct one complete action
|
|
66
|
-
locally from one region, validate that its bounds and target still match the
|
|
67
|
-
source capture, then dispatch one `click` containing `x`, `y`, the exact
|
|
68
|
-
target, `delivery_mode`, and that same `capture_id`. Never retry a capture
|
|
69
|
-
refusal without `capture_id`. Dispatch at most one action derived from that
|
|
70
|
-
capture.
|
|
71
|
-
4. Reobserve after every action attempt, including a timeout, unknown result,
|
|
72
|
-
partial delivery, or suspected no-op. Verify the postcondition from fresh
|
|
73
|
-
state before choosing another action.
|
|
74
|
-
|
|
75
|
-
Example parse shape:
|
|
76
|
-
|
|
77
|
-
```json
|
|
78
|
-
{
|
|
79
|
-
"capture_id": "capture-from-current-observation",
|
|
80
|
-
"options": {
|
|
81
|
-
"kinds": ["text", "icon"],
|
|
82
|
-
"min_confidence": 0.5,
|
|
83
|
-
"max_regions": 100
|
|
84
|
-
}
|
|
85
|
-
}
|
|
86
|
-
```
|
|
87
|
-
|
|
88
|
-
Region rectangles are half-open source-screenshot pixel bounds with a top-left
|
|
89
|
-
origin. Do not treat OCR text, an icon label, or an `interactive` hint as proof
|
|
90
|
-
that a region is actionable. Do not reuse a region after an action, resize,
|
|
91
|
-
move, scroll, navigation, display-layout change, or target-identity change.
|
|
92
|
-
|
|
93
|
-
## Handle failures by category
|
|
94
|
-
|
|
95
|
-
- `not_installed`: the optional `cua-perception` extension is absent. Continue with
|
|
96
|
-
accessibility, browser, or caller-owned visual reasoning unless the user has
|
|
97
|
-
chosen to inspect and install the extension.
|
|
98
|
-
- `capture_not_found`, `capture_expired`, `capture_stale`, or
|
|
99
|
-
`capture_generation_mismatch`: take a fresh capture and parse that new ID.
|
|
100
|
-
- `unsupported_target` or `unsupported_platform`: use another observation
|
|
101
|
-
route and preserve the reported limitation.
|
|
102
|
-
- `incompatible_protocol` or `artifact_invalid`: stop using the extension;
|
|
103
|
-
inspect its exact installed state before any update or reinstall.
|
|
104
|
-
- `worker_launch_failed`, `worker_crashed`, `worker_cancelled`, `timeout`, or
|
|
105
|
-
`inference_failed`: do not act from a partial or old result. Reobserve before
|
|
106
|
-
a bounded retry or fallback.
|
|
107
|
-
- `invalid_frame` or `resource_limit_exceeded`: narrow the request or use a
|
|
108
|
-
supported native-resolution capture; never clamp regions or silently change
|
|
109
|
-
coordinates.
|
|
110
|
-
|
|
111
|
-
The worker parses only the admitted screenshot and bounded options. It does not
|
|
112
|
-
own desktop capture, accessibility, browser, input, credential-store, or action
|
|
113
|
-
authority.
|
|
114
|
-
|
|
115
|
-
Parsing a window does not make its pixel input background-safe. Preserve the
|
|
116
|
-
platform action ladder and exact `background_unavailable` refusal. Foreground
|
|
117
|
-
delivery and desktop input can change focus, workspace, or the system cursor;
|
|
118
|
-
use them only when already authorized for the workflow.
|
|
119
|
-
|
|
120
|
-
An external chooser, including a `jev-use` recipe, receives a bounded table of
|
|
121
|
-
opaque action IDs rather than open-ended Driver tool access. The caller keeps
|
|
122
|
-
provider setup and credentials outside Driver, validates one returned ID, and
|
|
123
|
-
dispatches the complete prebuilt action unchanged. See [Use jev-use with visual
|
|
124
|
-
regions](https://cua.ai/docs/cua-driver/guides/visual-regions);
|
|
125
|
-
do not add provider SDK logic or credentials to this skill.
|