@bastani/atomic 0.9.56 → 0.9.57-alpha.1

package/dist/builtin/subagents/skills/cua-driver/WORKFLOW.md

@@ -1,226 +0,0 @@
-# Observe, act, verify
-
-## Choose the route
-
-Name the user's postcondition and any method constraint before acting.
-
-1. For a non-GUI outcome, use a caller-provided application API, filesystem operation, or CLI when permitted; independently read back its result. An MCP-only client must not assume it has a shell.
-2. For exact window geometry, prefer `set_window_frame` and verify with `list_windows`. For a known native menu path, use `invoke_menu` and verify the command's effect. It can temporarily activate the target, so apply the foreground authorization boundary.
-3. For supported browser page content, use [BROWSER.md](BROWSER.md) when page-aware automation is permitted. GUI-only or native-input proof stays on the native route.
-4. For native controls, prefer snapshot-bound accessibility actions. If semantics cannot reach the target, use a fresh, valid screenshot and the window pixel route.
-5. If the selected background route refuses or visibly fails, consider explicitly authorized foreground delivery for that action.
-6. If exact-window observation or delivery is unavailable, use the desktop route only when the task and authorization admit full-display capture and visible input.
-
-Skip a route whose prerequisite is demonstrably absent: no valid window image means no window-pixel attempt. Do not exhaust impossible actions just to reach a later rung. A refusal never grants broader authority.
-
-## Select the target
-
-Discover running applications and their windows before assuming a remembered PID still exists. Use `launch_app` when the user requests or implies launch. Select the intended window from its returned `windows` or `list_windows({pid})`; never blindly take the first item.
-
-A cold launch may return a process before a window appears. Retry bounded window discovery, not repeated launches. Empty discovery does not prove the process exited. For closure, absence from `list_windows` proves no listed windows, not process termination.
-
-Choose the target on each action:
-
-```json
-{ "target": { "kind": "window", "pid": 844, "window_id": 10725 } }
-```
-
-```json
-{ "target": { "kind": "desktop", "display_id": "primary" } }
-```
-
-These objects are argument fragments for tools advertising `target`, not standalone commands. Replace example IDs and coordinates with observed values. Do not combine `target` with flat `scope`, `pid`, or `window_id` input fields. Observation tools and semantic-only tools such as `get_window_state` and `set_value` retain their own `pid`/`window_id` schema.
-
-`list_windows.z_index` uses larger integers for windows nearer the front. A null value is unknown, not zero; array order is not stacking evidence. Titles, bounds, and application identity help select among returned candidates but never replace an exact identifier.
-
-## Observe
-
-`get_window_state({pid, window_id})` requests the accessibility tree and a grounding screenshot by default. Start with `query` (or `max_elements`) instead of a full read, and verify at checkpoints rather than after every action. Check what actually came back: permission, backing-store, or surface-identity failures can leave usable tree data without an image. `screenshot_error` and `screenshot_frame_valid:false` are not empty-tree signals.
-
-By default the tree comes back **once**, as compact `tree_markdown` (`tree_format:"markdown"`). A row `[N]` is addressed with `element_token` `<snapshot_id>:N`; the `snapshot_id` is in the response header and in `structuredContent`. Use `tree_format:"elements"` for the structured `elements` array when you need explicit tokens, frames, parents, or values, or `"both"` for the old double payload (about twice the size). `full_output:true` restores the previous full response, including the platform's walk limits. In `elements`, rows may contain `element_token`, role, label, value, actions, parent, depth, enabled/selected state, `frame` (screen coordinates, the space of `scope:"desktop"` actions), and `screenshot_frame` (pixels of the screenshot in the same response, the space of window-local pointer `x`/`y`). Missing fields are unknown.
-
-- Use `query` to project matching rows plus ancestors without renumbering their indices.
-- A read walks at most 250 nodes by default. When it stops there the response says `Tree truncated at max_elements=…` (`truncated:true`, `truncation_hint`). Pass a larger `max_elements`, or narrow with `query` / `max_depth`. Truncation does not prove absence.
-- After the first read, pass `since:"latest"` (or `since:<snapshot_id>`) to get only what changed. `latest` is this window's most recent read with the same `query`, `max_elements` and `max_depth`. You get: `+` added, `~` changed, `-` removed rows (removed ids are the old snapshot's), or `no change since …; focused element is …` (focus is reported on macOS only). The response carries a new `snapshot_id`: use it in tokens (`<new id>:N`). Rows not listed keep their `[N]` unless a `reindexed:` line says otherwise. An unknown, expired, other-window, or differently-scoped `since` returns a full read, and `since_status` says why. A diff reads the rendered tree rows only; it does not prove the screenshot is unchanged.
-- `_note`, `background_input`, and (Linux) `frame_note` are omitted unless `verbose:true`; a degraded snapshot keeps `background_input`.
-- `include_screenshot:false` returns the tree only. On macOS the window keeps the pixel frame of its last screenshot while its size is unchanged, so `x,y` read off that screenshot still work; elsewhere, and after a resize, take a new screenshot before a pixel action.
-- Where advertised, `include_accessibility_tree:false` requests capture without a tree walk. It keeps the current snapshot's rows and tokens valid. `display_only:true` with it is for live previews: it also leaves your pixel frame alone, and its image cannot ground actions. Use it when you only need to look; do not use `max_elements:1` or a `query` that matches nothing for that, because those replace the snapshot with an almost empty one.
-- Row numbers belong to one read. A narrower re-read (smaller `max_elements`, `max_depth`, another `query`) can stop before a row you saw earlier, and `<new snapshot_id>:N` for that row is then out of range. Use rows printed in the read whose `snapshot_id` you use.
-- `capture_mode` is deprecated and ignored. Do not change configuration to repair a sparse tree.
-- Use `screenshot_out_file` to save a PNG instead of inlining it, then actually read the image. Use an absolute, run-scoped output path.
-
-The accessibility model may lag or disagree with rendered state: Electron text shims can echo writes, Catalyst may expose placeholders, and virtualized rows may have unusable frames. Cross-check the relevant visible outcome. An empty tree can also mean an unavailable accessibility bridge; inspect `degraded_reason` and the platform guide. Retry once for lazy initialization, not indefinitely.
-
-## Act once
-
-Use the `element_token` `<snapshot_id>:N` for row `[N]` of the latest snapshot of the intended window, for example `"s0000002a:11"`. A bare `"11"` is accepted only together with `window_id` and then means row 11 of that window's current snapshot. Do not mix a row number from one read with the `snapshot_id` of another. A later snapshot can invalidate a pending action, including when another agent observes the same window.
-
-Example CLI window action, with IDs and token replaced from the preceding response:
-
-```bash
-cua-driver click '{"target":{"kind":"window","pid":844,"window_id":10725},"element_token":"s0000002a:14","session":"run-1"}'
-```
-
-| Intent                          | Arguments in addition to exact target/session                                  |
-| ------------------------------- | ------------------------------------------------------------------------------ |
-| Activate a control              | `click` with `element_token`                                                   |
-| Double/right click              | `double_click` / `right_click`; inspect the host schema and delivery result    |
-| Insert text                     | `type_text` with `element_token` and `text`                                    |
-| Focus a visible field and type  | `type_text` with `x,y,text`, when that form is advertised                      |
-| Replace an exposed native value | `set_value` with `pid,element_token,value` (its own schema, no `target`)       |
-| Send a key or combination       | `press_key` with `key`, or `hotkey` with `keys`                                |
-| Scroll / drag                   | `scroll` takes `direction` and `amount` (wheel notches); signed `dx`/`dy` is converted (positive `dy` is down; up to 50 is notches, more is pixels at 100 px per notch). Inspect `describe drag` |
-
-Text insertion and value replacement are different intents. Setting a field does not prove a form submission, navigation, or rename committed. Open a collapsed search/input control and reobserve before typing into it; one focus-click may not both open and focus it. Inspect the existing value/selection before replacing content.
-
-If a text action returns `unverifiable`, take a fresh snapshot before retrying. A deferred provider can publish after the call unwinds, so retrying immediately may duplicate text. If renderer focus is missing, the advertised `type_text` pixel form focuses then types in one call. For minimized windows, prefer an exposed semantic commit control; do not assume Return or a value write commits, and do not silently restore the window.
-
-Keep `delivery_mode:"background"` as the default for window input. The route may use accessibility hit-testing even when addressed by pixels: pixel coordinates do not promise physical pointer delivery. Read the returned `route` instead of inferring it from the tool name.
-
-## Batch known actions
-
-`run_steps` is the default way to act. When the next action or actions are decided, send them as one `run_steps` call with `observe:{}` instead of one call per action followed by a read. Even a single step pays off: the action and the look at its result become one call. The batch runs the same tools in order, stops at the first failure, and returns per-step status plus at most one bounded observation. Typical fit: fill several fields, then press a button, then read the result. Do not batch across a point where the answer decides the next step, or when a step reshuffles the window and invalidates element tokens used by later steps (use pixel targets after it, or split the batch there).
-
-```bash
-cua-driver run_steps '{"session":"run-1","steps":[
-  {"tool":"set_value","args":{"pid":844,"element_token":"s0000002a:14","value":"Ada"}},
-  {"tool":"set_value","args":{"pid":844,"element_token":"s0000002a:15","value":"Lovelace"}},
-  {"tool":"click","args":{"target":{"kind":"window","pid":844,"window_id":10725},"element_token":"s0000002a:21"}},
-  {"tool":"press_key","args":{"pid":844,"key":"return"}}
- ],"delay_ms":100,"observe":{}}'
-```
-
-- `tool` is one of `click`, `double_click`, `right_click`, `set_value`, `type_text`, `press_key`, `hotkey`, `scroll`, `drag`, `move_cursor` (`press` and `type` are accepted for `press_key` and `type_text`, `triple_click` is a click with `count:3`, and a key name such as `down` as the tool is a `press_key`); `args` are exactly that tool's arguments. Observation tools do not run inside a batch: a read between actions is skipped (it could not change the later steps), a `get_window_state` as the last step is the observation (merged with `observe`), and a trailing `zoom` runs after it. Run `describe run_steps` and `describe <tool>` for schemas. Up to 64 steps. `delivery_mode` on `run_steps` is the default for steps that set none.
-- Name the window once: `pid` and `window_id` (or `app`/`window`) on `run_steps` itself are the default for every step that names none.
-- A step whose `element_token` went stale because an earlier step re-rendered the window is retried once on the element with the same role and label in a fresh read, when exactly one matches; the step's line says so.
-- Every step is validated before the first runs, so a malformed step changes nothing. Each step then passes the same session, permission, capability-manifest and approval checks as a direct call; a batch grants nothing a single call lacks, and a refused step ends the batch like any other failure.
-- A batch has one session. Set `session` on `run_steps`; a step may repeat it but not name another.
-- `observe` is optional and reads once, after the last executed step (also after a failure). Pass `true` or `get_window_state` arguments; `pid`/`window_id` come from the last step that names both. Defaults: `since:"latest"` (only what changed since your last read of that window with the same view; a full read if there is none), `include_screenshot:false`, `max_elements:250`. Pass `include_screenshot:true` to see the window, or `since:null` for a full read. Omit `observe` to read nothing.
-- Read `steps[].ok` and `failed_step`. Steps before a failure did run and are not rolled back; steps after it did not. Observe before repairing, as for a single `unverifiable` action. When a step fails because another window of the app holds focus (a dialog opened mid-batch), the end read is of that window, even without `observe`; when a pixel step names a window with no screenshot yet, the end read includes its screenshot. A step refused because its app ignores background input (Electron, LibreOffice) says how to re-send it; set `foreground_fallback:true` on `run_steps` to retry such steps once in the foreground (the window is activated and the pointer may move).
-
-## Script a flow
-
-Use `run_script` when the flow needs logic between actions: a loop over rows, a branch on what the app shows, a retry, or a value read in one window and typed into another. The script is JavaScript, the body of an async function; it runs in ONE call and only its `return` value, console output and a per-call log come back. Use `run_steps` for a fixed list of steps you have already decided, and plain calls when the next step needs your judgement.
-
-```bash
-cua-driver run_script '{"session":"run-1","script":"const app = await cua.getApp(\"Calculator\"); for (const key of [\"7\", \"multiply\", \"6\", \"equals\"]) await app.click({role: \"button\", name: key}); const [display] = await app.query({role: \"statictext\"}); return display.value;"}'
-```
-
-- The only API is `cua`: `getApp`, `launch`, `listApps`, `listWindows`, `sleep`, and on an app `getState`, `query`, `waitFor`, `verify` and the actions (`click`, `typeText`, `pressKey`, `setValue`, `scroll`, `drag`, ...). A target is `{role, name, nth?}` (found fresh, waits up to 3 s), an `element_token`, or `[x, y]` window pixels. `describe run_script` lists the full API.
-- A failed driver call throws; catch it to recover. An uncaught error names the script line, the call and the reason.
-- Every driver call passes the same session, permission, capability-manifest and approval checks as a direct call, so a script grants nothing the same direct calls lack. It cannot call `run_steps`, `run_script`, `set_config` or tools outside its allowlist.
-- Sandbox: QuickJS with no filesystem, network, process, timers or environment. Limits: `timeout_ms` wall time (default 30000, max 120000), `max_calls` driver calls (default 100, max 500), a 32 MiB heap, a bounded stack, and a 64 KiB script. The script stops when the request is cancelled.
-- `run_script` is on by default. Whoever runs the driver can turn it off with `CUA_DRIVER_DISABLE_RUN_SCRIPT=1` or `"disable_run_script": true` in `~/.cua-driver/config.json` (the older `CUA_DRIVER_EXPERIMENTAL_SCRIPT=0` and `"experimental_script": false` also work). `set_config` cannot change it. When it is off, it is not in `tools/list`.
-
-## Pixel coordinates
-
-Ground window actions on the PNG from that exact `get_window_state`; ground desktop actions on `get_desktop_state`. Origins are top-left, increasing downward. The driver handles its own window capture scaling; do not add window offsets to window-local input.
-
-The harness may downsample the displayed preview independently of the returned PNG. Use the returned dimensions and the original file. If measuring on a resized preview, account for its exact scale in both axes; do not assume the preview is native resolution. Do not guess from accessibility frames or another app's geometry.
-
-After movement, resize, navigation, or a competing desktop interaction, reobserve. `zoom` takes a region as corners, `x1,y1` (top-left) to `x2,y2` (bottom-right); `x,y,width,height` is converted. Preserve the `from_zoom` mapping on the supported follow-up action. A manually cropped image requires its crop offset; an untracked crop is not an action coordinate source.
-
-For tiny targets, inspect at full resolution or annotate a copy without changing its dimensions. Keep the raw evidence unmodified. `debug_image_out`, where advertised, captures an action diagnostic; it is not a pre-action approval step.
-
-## Desktop loop
-
-Use the full-display path when authorized and the narrower route cannot satisfy the task. It is global input to the visible desktop, not background targeting. Ensure the desired app/field is visible and focused before keyboard input; stop if the user or another controller changes it.
-
-```bash
-cua-driver get_desktop_state '{"session":"run-1","screenshot_out_file":"/absolute/run-dir/before.png"}'
-# Read before.png; replace x/y with the observed target.
-cua-driver click '{"target":{"kind":"desktop","display_id":"primary"},"x":600,"y":300,"session":"run-1"}'
-cua-driver get_desktop_state '{"session":"run-1","screenshot_out_file":"/absolute/run-dir/after.png"}'
-# Read after.png and verify the intended outcome.
-```
-
-Desktop calls use the portable `display_id:"primary"` target. Do not assume arbitrary display IDs work. A desktop action does not change the modality or authority of later calls; the next action can target a window again.
-
-## Verify and stop
-
-For an expressible exact-window postcondition, use `verify_state` with bounded predicates and stable samples. Its result is `satisfied`, `unsatisfied`, or `unknown`. The driver does not interpret its optional final image; the agent does. Read `unknown_reason`: ambiguous matches, unavailable observations, untrusted web state, or insufficient stable samples are not success.
-
-```bash
-cua-driver verify_state '{"pid":844,"window_id":10725,"expect":[{"element":{"selector":{"label_contains":"Saved"},"exists":true}}],"include_screenshot":true,"session":"run-1"}'
-```
-
-This example proves a matching trusted element exists, not that every application has a meaningful “Saved” indicator. `label_contains` matches an element's accessible name, which for label-less text is its value. On macOS a selector also reaches display-only text (static text, read-only values) when no addressable control matches it. Text inside web content stays `unknown` with `untrusted_source`; read it from a fresh snapshot instead. Choose predicates that establish this task. Use fresh `get_window_state` for outcomes the predicate language cannot express, and fresh `get_desktop_state` for desktop proof. `verify_state` remains an exact-window tool; a previous desktop action does not disable it.
-
-Action facts are not task outcomes:
-
-| Field                | Meaning for the next decision                                                                         |
-| -------------------- | ----------------------------------------------------------------------------------------------------- |
-| `effect:"confirmed"` | The action has publishable readback; still check the task postcondition                               |
-| `partial`            | Only the reported delivered portion landed; inspect before repair                                     |
-| `unverifiable`       | Delivery cannot prove effect; observe before retrying                                                 |
-| `suspected_noop`     | Available evidence suggests no useful change                                                          |
-| `refused`            | The route deliberately did not deliver                                                                |
-| `route`              | Public actuator class: `accessibility`, `synthetic_events`, `global_input`, `dom`, or `trusted_input` |
-| `escalation`         | Suggested next route/reason, never authorization or an automatic retry                                |
-| `hint`               | One sentence on what to do next; see [Refusals and hints](#refusals-and-hints)                        |
-| `next_call`          | `{tool, arguments}` safe to run as is; see [Refusals and hints](#refusals-and-hints)                  |
-
-Successful action escalation uses `target` (`pixel`, `foreground`, or `page`) and `reason` (`route_unavailable`, `delivery_failed`, `effect_unconfirmed`, `suspected_noop`, or `permission_required`). Error diagnostics may have a different shape; inspect their code and current schema. Missing action facts are not confirmation.
-
-### Refusals and hints
-
-A refusal, an `unverifiable` action and a `suspected_noop` carry two fields that say what to try next, on every tool:
-
-- `hint`: one short sentence. Read it before you give up or reach for pixels.
-- `next_call`: `{tool, arguments}`, a call that is safe to run as is. It is a read (`get_window_state`, `list_windows`, `get_browser_state`, `check_permissions`) or a call that keeps the user's window where it is. Make it, then decide.
-
-A foreground retry is never a `next_call`. When the `hint` names `delivery_mode:"foreground"`, it comes with a condition: only if the user allows the window to come forward. Without that, the step is blocked: say so and why; do not force it.
-
-The fields sit next to `code` on an error result, next to `refusal` on a browser refusal, and on the `ActionResult` itself. In `run_steps` a failed step's report carries them; in `run_script` the hint is appended to the failed call's error message. A tool that already set its own `hint` or `next_call` keeps it. `next_call` is advice, not an automatic retry: never replay a partial or unknown action without a fresh read.
-
-After three single action calls in a row in one session (not counting reads), the result of the third also carries a `hint` to chain the next ones with `run_steps`, and again after every third. It is advice with no `next_call`, and it is never sent for a step that already runs inside `run_steps` or `run_script`. Every model turn costs seconds, so chain actions whenever you already know the next few steps.
-
-When the postcondition is visibly satisfied, stop acting. For media, selection alone is not playback: check the title, playing/pause indicator, and elapsed progress after preroll/buffering. For closure, verify the requested window disappeared. State only what the evidence proves.
-
-Never replay canceled, partial, or unknown actions automatically. An interrupted transport may have delivered input before losing its response. Report the interruption accurately and obtain fresh state before any further action.
-
-## Filesystem outcomes and GUI fallbacks
-
-When the requested outcome is a filesystem change and the caller has a
-headless filesystem or command capability, use that semantic route. Enumerate the
-exact source set, decide the destination-conflict policy before changing
-anything, perform one batch-safe operation, then independently read back both
-source and destination manifests. Do not open a file manager merely to mimic a
-move, copy, or rename that the caller can execute and verify directly.
-
-If the caller has no such capability, use the file manager as a GUI fallback
-and keep each claim narrow:
-
-1. After entering an inline rename and setting its value, commit it with the
-   platform's confirmation key, then take a fresh snapshot. Value readback from
-   the inline editor proves only that the editor changed; it does not prove the
-   filesystem rename committed.
-2. For a multi-selection, use the platform modifier (`cmd` on macOS, `ctrl` on
-   Windows/Linux). With authorization for visible control on macOS and Windows, issue that modified click with
-   `delivery_mode:"foreground"` so the target observes physical modifier state;
-   a refused background attempt is an escalation signal, not a failed action to
-   trust or repeat. Re-snapshot before the next operation. Continue only when
-   every intended item is selected and the prior selection was preserved.
-3. After a cross-window drag or paste, verify the destination contains the
-   complete expected set and the source reflects copy-versus-move semantics.
-   A delivered drag, keypress, or menu action is not file-operation proof.
-4. If a destination conflict presents an unrecognized policy or ambiguous
-   partial result, stop that GUI path and surface the unresolved state instead
-   of retrying blindly.
-
-### Clipboard outcomes and GUI fallbacks
-
-When the requested postcondition is an exact value on the system clipboard,
-rather than the literal gesture of selecting and copying it, keep the operation
-semantic. Read the value from the narrowest typed source, call
-`clipboard_write`, then prove the real clipboard state with `clipboard_read`.
-For browser content, this means reading the page with `get_browser_state` and
-writing the exact observed text; a passive page-text ref does not need to be
-clicked first.
-
-Use visual selection followed by the platform copy hotkey only when the user
-explicitly asks for that gesture, the source cannot expose the value
-semantically, or direct clipboard tools are unavailable. Treat that as a GUI
-fallback: re-snapshot before acting, verify the selected range when the
-application exposes it, and escalate only the delivery step that cannot land
-in the background.