@bastani/atomic 0.9.28-alpha.3 → 0.9.28-alpha.5

package/dist/builtin/subagents/skills/cua-driver/RECORDING.md

@@ -1,86 +1,89 @@
-# Recording & replaying trajectories
-
-> **Cross-platform.** Recording is available on macOS (native
-> ScreenCaptureKit), Windows (ffmpeg + `gdigrab`), and Linux (ffmpeg +
-> `x11grab`). Replay is cross-platform as long as the recorded artifacts
-> are present.
-
-Session-scoped capture of action sequences + pre/post state, suitable
-for demos, regression diffs, and training data. Invoked only when the
-user explicitly asks to record — the skill does not auto-enable this.
-
-`start_recording` turns on a session-scoped trajectory recorder. While
-enabled, every action-tool call (`click`, `right_click`, `scroll`,
-`type_text`, `press_key`, `hotkey`, `set_value`) writes a numbered
-turn folder under a caller-chosen output directory. Read-only tools
-(`get_window_state`, `list_windows`, `screenshot`, `list_apps`,
-permission probes, agent-cursor getters / setters, and the recording
-controls themselves) are not recorded.
-
-**Video on by default.** `start_recording` also captures the main
-display to `<output_dir>/recording.mp4` (H.264 / 30 fps) for the
-lifetime of the session. The mp4 is finalized on `stop_recording`. Opt
-out with `record_video: false` when you don't want video.
-
-**macOS — native ScreenCaptureKit, zero-config.** On macOS the daemon's
-recorder uses `SCStream` + `SCRecordingOutput`, so it inherits the daemon's
-Screen Recording grant — no separate
-subprocess prompt, no fast-fail, no second TCC dance. Requires macOS
-15.0+ (SCRecordingOutput introduced in macOS 15). No ffmpeg needed.
-
-**Windows / Linux — ffmpeg subprocess.** Outside macOS the recorder
-shells to ffmpeg with `gdigrab` (Windows) or `x11grab` (Linux). The
-binary needs to be on PATH (`winget install Gyan.FFmpeg` /
-`apt install ffmpeg`); when missing, the per-turn capture continues
-without video and `last_error` carries the install hint. ffmpeg
-startup failures fast-fail with a stderr tail in the error.
-
-## Start / stop
-
-Two equivalent surfaces: the `start_recording` / `stop_recording` MCP
-tools, or the friendlier `cua-driver recording` subcommand group
-(wraps both with human-readable output).
+# Recording and replay
 
-```
-cua-driver recording start ~/cua-trajectories/run-1
-# … run the workflow …
-cua-driver recording status    # -> enabled / disabled, next_turn, output_dir
-cua-driver recording stop      # -> "Recording stopped. (video → recording.mp4)"
-```
+Record only when the user requests it. Keep the recording controls and actions
+on one persistent MCP connection; see [RUNTIME.md](RUNTIME.md). When the actions
+pass a `session` label, you may pass the same label to `start_recording`.
+`stop_recording` and `get_recording_state` do not accept `session`.
 
-Raw-tool equivalent:
+## Start, observe, stop
 
-```
-cua-driver start_recording '{"output_dir":"~/cua-trajectories/run-1"}'
-cua-driver get_recording_state
-cua-driver stop_recording '{}'
+These are tool calls on that connection, not separate CLI invocations:
+
+```text
+get_recording_state({})
+start_recording({"output_dir":"/absolute/run-dir/trajectory","record_video":true})
+get_recording_state({})
+# Run the authorized workflow and verify every action.
+stop_recording({})
+get_recording_state({})
 ```
 
-The `recording` subcommands require a running daemon (`cua-driver
-serve &`) because recording state is per-process. `output_dir` expands
-`~` and is created (with intermediates) if missing. Turn numbering
-starts at `1` every time recording is (re-)enabled, regardless of any
-existing contents in the directory. State lives in memory only — a
-daemon restart resets to disabled.
+Choose an unused output directory: turn numbering restarts at 1. Video is
+off by default; explicitly set `record_video:true` when requested. Inspect
+`video_active` and `last_error`, not just the successful tool status. Per-turn
+capture may continue even when video initialization failed.
+
+A recording keeps only its owner's actions. With a `session` label, it keeps
+that session's calls. Without one, it keeps every call on the connection that
+started it, including calls that pass a `session` label. Calls from other
+connections, including one-shot `cua-driver <tool>` processes, and
+`start_session` / `end_session` write no turns. A recording started with CLI
+`cua-driver recording start` has no owning session and records every
+session's actions.
+
+There is still one recorder per runtime. Manual `stop_recording` stops
+whichever recording is active, regardless of its starting session. Do not
+start over or stop another run's recording. If a recorder is already active,
+coordinate with its owner rather than taking it over.
+
+Stop and inspect `last_video_path` before ending the connection. A disconnect
+can tear down owned recording; a runtime restart loses in-memory state.
+Confirm the expected files exist, inspect images, and decode the finalized
+video before claiming success. A file path alone does not prove a playable
+or correctly scoped recording.
+
+CLI `recording start|status|stop` commands also exist, but do not form a
+session-preserving action loop by themselves. Inspect their help rather than
+assuming the convenience command supports every raw-tool argument.
+
+## Platform boundaries
+
+- macOS video uses ScreenCaptureKit/SCRecordingOutput (macOS 15+), under the
+  responsible runtime's grant. Permission status is not proof of live capture.
+- Windows video uses ffmpeg `gdigrab`; the runtime must be in the interactive
+  desktop session.
+- Linux X11 video uses ffmpeg `x11grab`. Native Wayland needs a supported
+  compositor-specific recorder and any portal grant; see [LINUX.md](LINUX.md).
+  A successful desktop PNG does not establish video availability.
 
 ## What each turn folder contains
 
 Each action writes to `turn-NNNNN/` (five-digit zero-padded counter):
 
-- `before_state.json` and `after_state.json` — application accessibility
+- `before_state.json` and `after_state.json`: application accessibility
   state immediately before and after the action. They carry the same
-  `tree_markdown` and `element_count` shape as `get_window_state`.
-- `before.png` and `after.png` — target-window images immediately before
+  `tree_markdown` and `element_count` shape as `get_window_state`. Each walk
+  is bounded by `start_recording`'s `state_timeout_ms` (default 1000, like
+  `get_window_state`'s `timeout_ms`). A walk that runs out of time keeps the
+  partial tree, and `evidence.json` marks that phase `truncated` with
+  `truncation_reason`, `nodes_visited`, `nodes_pending`, and `timeout_ms`. If
+  an accessibility provider stops answering, the turn continues without
+  state and records `state_capture_timeout`. For an action refused before
+  dispatch, such as a click with an unknown or expired `capture_id`, the
+  state walk is skipped and the state is recorded as `not_applicable` with
+  the classification `action_refused_before_dispatch`. Pass
+  `include_accessibility_tree: false` to record without state.
+- `before.png` and `after.png`: target-window images immediately before
   and after the action. Window capture remains scoped to the target when
   another window covers it.
-- `evidence.json` — capture status for each phase. Missing expected capture
+- `evidence.json`: capture status for each phase. Missing expected capture
   has an explicit classification instead of disappearing from the turn.
-- `app_state.json` and `screenshot.png` — compatibility aliases for
+- `app_state.json` and `screenshot.png`: compatibility aliases for
   `after_state.json` and `after.png`.
-- `action.json` — the tool name, full input arguments, result
+- `action.json`: the tool name, full input arguments, result
   summary, result-error flag, pid, click point (when applicable), ISO-8601
   timestamp.
-- `click.png` — for click-family actions (`click`, `double_click`,
+- `click.png`: for click-family actions (`click`, `double_click`,
   `right_click`): a copy of the pre-input image with a red marker drawn at
   the click point. Usually that source is `before.png`. When Windows scrolls
   an element into view during an action, it retains an additional
@@ -90,7 +93,7 @@ Each action writes to `turn-NNNNN/` (five-digit zero-padded counter):
   status. The original before/after images and state remain intact.
   **Both addressing modes are covered:** explicit
   `x, y` clicks use the platform's recording-coordinate mapping, and
-  `element_index`-addressed clicks resolve to the element's center
+  `element_token`-addressed clicks resolve to the element's center
   via the live AX/UIA cache, then convert to the retained image's coordinate
   space. Native Hyprland recording retains the output image, so both kinds of
   marker use output coordinates. Pixel markers also account for the target
@@ -109,54 +112,21 @@ Each action writes to `turn-NNNNN/` (five-digit zero-padded counter):
   Out-of-image points are rejected, never moved to an image edge. Other dispatched clicks whose markers
   cannot be resolved or rendered remain evidence failures.
 
-## When to use it
-
-- Demos and screen recordings — play the turn folder back to show
-  exactly what the agent saw and what it did.
-- Replay for regression — re-run the same sequence against a future
-  build and diff the new trajectory against the saved one.
-- Training data collection — each turn is a
-  `(state, action, next_state)` triple ready for offline learning.
+## Replay is a new action sequence
 
-## When to invoke it
+`replay_trajectory` invokes recorded arguments in turn order; it is not a
+semantic task planner. It accepts `dir`, `delay_ms`, and `stop_on_error`.
+Read its live schema before use, and obtain authorization for the actions
+being replayed.
 
-This skill does **not** auto-enable recording. The client invokes
-`start_recording` explicitly when the user asks to capture a session.
-If the user says "record this session" or similar, call
-`start_recording({output_dir:…})` before the first action (video on
-by default; pass `record_video: false` to opt out), and
-`stop_recording({})` when done.
-
-## Replaying a recorded trajectory
-
-`replay_trajectory({dir})` walks `<dir>/turn-NNNNN/` folders in
-lexical order, reads each `action.json`, and re-invokes the recorded
-tool with its recorded `arguments`. Optional knobs: `delay_ms`
-(pacing between turns, default 500) and `stop_on_error` (halt on
-first failure, default true).
-
-```
-cua-driver recording start ~/cua-trajectories/demo1
-# … run the workflow …
-cua-driver recording stop
-# Later: replay against a new build.
-cua-driver replay_trajectory '{"dir":"~/cua-trajectories/demo1","delay_ms":500}'
-```
+Recorded PIDs, window IDs, element tokens, browser refs, and geometry can all
+be stale. Read-only snapshots are not part of the action sequence, so replay
+does not automatically refresh element handles. Pixel/key actions also need
+the original target identity, layout, and focus; they are not portable merely
+because they lack an element token.
 
-Important caveat: **element_index doesn't survive across sessions**.
-Indices are assigned fresh on every `get_window_state` snapshot,
-keyed on `(pid, window_id)`, so a recorded
-`click({pid, window_id, element_index: 14})` from yesterday won't
-resolve today — the pid is usually different, the window_id always
-is. The call returns `Invalid element_index` or `No cached AX
-state`. Pixel clicks (`click({pid, x, y})`) and keyboard tools
-(`press_key`, `hotkey`, `type_text` without element_index) replay cleanly; element-indexed actions require a
-live snapshot that replay doesn't currently re-emit (read-only tools
-like `get_window_state` aren't recorded). For a reliable replay, either
-compose the trajectory from pixel + keyboard primitives, or capture
-it as a regression artifact (compare the failure/success pattern
-across builds) rather than a re-driving script.
-
-If recording is still enabled while replay runs, the replay is
-itself recorded into the current output directory — that's the
-intended regression-diff workflow.
+Use trajectories as evidence unless current targets and preconditions are
+independently established. Never replay canceled, partial, or unknown actions
+to discover whether they originally landed. If recording remains enabled
+during replay, the replay itself may produce new turns; keep source and output
+directories distinct.

package/dist/builtin/subagents/skills/cua-driver/RUNTIME.md

@@ -0,0 +1,113 @@
+# Runtime, ownership, and authorization
+
+## Preflight and transport
+
+```bash
+cua-driver --version
+cua-driver status
+cua-driver doctor
+cua-driver list-tools
+cua-driver describe get_window_state
+cua-driver describe click
+```
+
+Check the resolved executable as well as the version; an installation may have intentional wrappers or multiple products. Preserve them. The client's version alone does not identify an already-running daemon. If the service is mismatched or missing, inspect its existing owner before starting or restarting anything.
+
+Use the advertised `tools/list` contract over MCP. Tool names are `snake_case`; CLI management commands are `kebab-case`. Inspect only unfamiliar tools instead of loading every schema. Descriptions may lag implementation: capability discovery proves advertised shape, while fresh application state proves behavior.
+
+| Client                                     | State ownership                                                                      |
+| ------------------------------------------ | ------------------------------------------------------------------------------------ |
+| CLI with an explicit non-default `session` | Calls to the same daemon share the named CLI lifecycle until `end_session` or expiry |
+| CLI without a named session                | Disposable implicit lease, cleaned up after the response                             |
+| Persistent MCP/SDK connection              | One private transport lifecycle; implicit or explicitly named runs live within it    |
+
+Use the CLI by default when a shell is available, or the existing MCP connection when the caller provides one. A multi-call CLI workflow must pass the same nonempty, non-`default` label on every call that accepts it. Tools without a public session parameter, particularly recording controls, need one persistent MCP connection for a coherent lifecycle.
+
+Bare MCP owns a runtime on Windows/Linux and uses the signed service identity on macOS. `cua-driver mcp --socket <endpoint>` explicitly selects a service. One-shot CLI calls are service-backed. Starting a process that serves MCP is not the same as connecting a client: use the harness's MCP integration, not independent shell writes to a dead stdin.
+
+For missing software, use the installation instructions in [README.md](README.md) with user approval. `check-update` is read-only; `update --apply` installs software. After an approved update, check both the executable and daemon. Do not interpret a nonzero updater exit as proof that nothing changed.
+
+`skills update` replaces the installed pack. Back up personal edits before refreshing. `skills install --from main` is for source validation, not proof of compatibility with a release binary. Other platform guides are installed only with `--all-platforms`.
+
+The old computer-use compatibility flag does not add a screenshot tool. Use `get_window_state` or explicitly authorized `get_desktop_state`.
+
+## Session lifecycle
+
+For multi-call work, prefer a short public label and pass the same label on every call that accepts it. Passing it once is not sticky: omission uses the transport's implicit session. Named CLI state is shared only within the daemon's CLI namespace, not with arbitrary MCP connections or a new daemon generation.
+
+`start_session` is optional: actions can establish the run. Use it to configure the initial cursor theme or revive a name after `end_session`. The default idle TTL is five minutes; do not assume a long human permission wait preserves handles.
+
+```bash
+cua-driver start_session '{"session":"run-1"}'
+# Discover, observe, act, and verify; repeat session:"run-1".
+cua-driver end_session '{"session":"run-1"}'
+```
+
+Explicit targets own observation/input modality, not session configuration. There is no `deescalate_session`. Do not use retired `capture_scope` settings or legacy escalation-session tools in new workflows. Reserved fields such as `_session_id` belong to the transport, never to an agent.
+
+Keep one controller per shared desktop. Separate sessions/cursors do not isolate global keyboard focus, physical input, a single-instance application, or another observer's snapshot cache. `creates_new_application_instance:true` requests a separate instance where supported; verify distinct processes/windows before concurrent work. For independent MCP work, use independent connections as well.
+
+## Foreground boundary
+
+`delivery_mode:"foreground"` is a user-visible takeover boundary, not an automatic retry. Use it only when the user already authorized visible control for this workflow or after asking. It can change focus, workspace, or the physical cursor; restoration is best-effort and adapter-dependent.
+
+If foreground control is not authorized, stop with the driver's refusal instead of silently escalating. Refresh state and do not retry automatically. An authorized desktop target intentionally uses system input, but it still requires the exact display resource to be admitted. A failed window capture is not desktop permission.
+
+Use Cua's tools rather than shell activation/input shims. `bring_to_front` is reserved for requested persistent foreground state or a focus-proxy surface that must remain active across calls. A native menu operation can also activate temporarily; prefer an in-window control when background posture matters.
+
+## Keep authorization separate from sessions
+
+The trusted host selects one permission profile at startup. `standard` keeps
+the normal profile behavior and residual approval requirements, `bounded`
+requires a reviewed capability manifest and has no runtime approval path, and
+`unrestricted` bypasses Cua approval prompts after explicit risk acceptance.
+Hard invariants plus managed and user policy remain binding in every profile.
+
+An optional capability manifest is a deny-by-default ceiling in `standard` and
+`unrestricted`; `bounded` requires one. It can remove tools or typed resources
+from the selected profile, but it cannot grant a tool, resource, or approval
+bypass that another authorization layer denies. Approval is considered only
+after the tool and every adapter-attested resource are inside manifest scope.
+
+Use the canonical startup pair together:
+
+```bash
+cua-driver mcp \
+  --permission-mode standard \
+  --capability-manifest ./capabilities.yaml \
+  --approve-capability-manifest
+```
+
+Capability manifest v3 omits file-level `mode` and `ask.tools`. Its
+`allow.tools` list is nonempty. Lifetime fields are optional in `standard` and
+`unrestricted`; `bounded` requires both `expires_after` and `idle_timeout`.
+The older `--session-policy` names remain compatibility aliases and must not be
+used in new configurations.
+
+Starting, ending, naming, reconnecting, or omitting a session never changes
+permission authority. A public session label is lifecycle metadata, never a
+grant, caller identity, or bearer credential.
+
+## Cursor feedback
+
+Keep the agent overlay visible while controlling pointer or keyboard input. Cursor-bearing and keyboard actions re-show it automatically. The overlay indicates activity, not proof that input landed.
+
+On Linux, ordinary `move_cursor({x,y})` moves only this synthetic cursor. The explicit desktop-pointer escape hatch must not be used unless the user asked for real-pointer control; inspect `describe move_cursor` for that tool's supported target form.
+
+Pixel `click` already animates the cursor. Do not precede it with `move_cursor` at the same point: that adds a second glide. Use `move_cursor` to indicate a location without clicking or seed the cursor before accessibility actions.
+
+The default theme provides session-colored pointer/glow, action marks, and activity animations. Named sessions have stable palette colors. Native hosts own delivery/target badges separately from theme artwork; cursor state is scoped to the private lifecycle.
+
+`set_agent_cursor_enabled` controls idle visibility. `set_agent_cursor_motion` takes `style` (`signature_arc` default, `spring_settle`, `magnetic`, `comet_swoop`, `adaptive`, `classic`), `timing` (`native`, `fitts`, `fixed`), `effects` (`trail`, `glow`, `magnet`, `ripple`, `squish` booleans; null restores the style default), and the arc knobs `start_handle`, `end_handle`, `arc_size`, `arc_flow` (bezier styles) plus `spring` and `turn_radius` (`classic`) where advertised. Choose preinstalled artwork with `set_agent_cursor_theme`; `cursor_id` on input tools is not a theme selector. Never supply inline theme code or arbitrary source paths through agent tools. The trusted local `cursor-theme` workflow owns validation, compilation, preview, installation, and removal.
+
+Overlay operations need a suitable UI event loop. macOS direct SDK/MCP runtimes without a certified host main-thread adapter can return `facility_unavailable`; do not report that as a successful move. See [EMBEDDING.md](EMBEDDING.md).
+
+## Cleanup and evidence
+
+Finish the owned recording, inspect its result, then end the run. Existing-profile browser cleanup can restore settings Cua enabled; check `end_session` errors and follow [BROWSER.md](BROWSER.md) rather than abandoning cleanup silently.
+
+Do not stop a shared daemon or close the user's app merely because the task ended. Close an application only when requested, using its normal UI and handling unsaved-work prompts without discarding data.
+
+Preserve raw screenshots and action results in a run-specific directory. Report the observed postcondition, relevant version/platform, and any remaining limitation. Review captures for private content before attaching them to a public issue or PR.
+
+For a local MCP HTTP endpoint, the trusted host owns `CUA_DRIVER_RS_MCP_HTTP_PORT` and a host-generated token of at least 32 characters in `CUA_DRIVER_RS_MCP_HTTP_TOKEN`. The client authenticates with the bearer header. Never print credentials or treat a public session label as authentication.