@qwen-code/qwen-code 0.23.3 → 0.23.4-dev.codemode-responses.20260914.87437db784

package/bundled/computer-use/references/windows-linux.md

@@ -0,0 +1,247 @@
+# Windows and Linux Computer Use
+
+Use this workflow only when `computer.getPlatform()` returned `windows` or
+`linux`. The shared entrypoint has already initialized `computer`.
+
+## API surface
+
+```ts
+type WindowTarget = { pid: number; windowId: number };
+type ElementTarget = {
+  pid: number;
+  windowId?: number;
+  elementToken: string;
+};
+type CoordinateTarget = WindowTarget & { x: number; y: number };
+type PointOrElementTarget = CoordinateTarget | ElementTarget;
+type ExactActionTarget = WindowTarget | ElementTarget;
+type DeliveryOptions = { deliveryMode?: DeliveryMode };
+
+type ComputerUse = {
+  click: (
+    args: PointOrElementTarget &
+      DeliveryOptions & { button?: MouseButton; count?: number },
+  ) => Promise<object>;
+  doubleClick: (
+    args: PointOrElementTarget & DeliveryOptions,
+  ) => Promise<object>;
+  rightClick: (
+    args: PointOrElementTarget & DeliveryOptions & { modifier?: string[] },
+  ) => Promise<object>;
+  drag: (
+    args: WindowTarget &
+      DeliveryOptions & {
+        fromX: number;
+        fromY: number;
+        toX: number;
+        toY: number;
+      },
+  ) => Promise<object>;
+  observeWindow: (
+    args: WindowTarget & {
+      disableDiff?: boolean;
+      includeScreenshot?: boolean;
+      maxTextChars?: number;
+    },
+  ) => Promise<WindowObservation>;
+  listApps: () => Promise<Array<App>>;
+  listWindows: (args: {
+    pid: number;
+    onScreenOnly?: boolean;
+  }) => Promise<Array<Window>>;
+  performSecondaryAction: (
+    args: ElementTarget & { action: string },
+  ) => Promise<object>;
+  pressKey: (
+    args: ExactActionTarget &
+      DeliveryOptions & { key: string; modifiers?: string[] },
+  ) => Promise<object>;
+  hotkey: (
+    args: ExactActionTarget & DeliveryOptions & { keys: string[] },
+  ) => Promise<object>;
+  scroll: (
+    args: PointOrElementTarget &
+      DeliveryOptions & { direction: Direction; amount?: number },
+  ) => Promise<object>;
+  setValue: (args: ElementTarget & { value: string }) => Promise<object>;
+  typeText: (
+    args: ExactActionTarget & DeliveryOptions & { text: string },
+  ) => Promise<object>;
+  close: () => Promise<void>;
+};
+
+type App = {
+  name?: string;
+  bundle_id?: string;
+  pid?: number;
+  running?: boolean;
+  launch_path?: string;
+};
+
+type Window = {
+  window_id: number;
+  title?: string;
+  is_on_screen?: boolean;
+  on_current_space?: boolean;
+};
+
+type WindowObservation = {
+  pid: number;
+  windowId: number;
+  mode: 'full' | 'diff' | 'no_change';
+  resyncReason?: string;
+  text: string;
+  elements: Array<Element>;
+  screenshot?: Screenshot;
+};
+
+type Element = {
+  element_token?: string;
+  role?: string;
+  label?: string;
+  automation_id?: string;
+  value?: unknown;
+  actions?: string[];
+};
+
+type Screenshot = {
+  images: Array<{ mimeType: string; dataBase64: string }>;
+};
+
+type Direction = 'up' | 'down' | 'left' | 'right';
+type MouseButton = 'left' | 'right' | 'middle';
+type DeliveryMode = 'background' | 'foreground';
+```
+
+## Workflow
+
+### 1. Initialize
+
+Start by getting the state for the app and window you want to use. When the task
+names an app, filter for that name directly:
+
+The snippets below show individual steps. When the app and visible window
+unambiguously match the task, combine discovery, selection, and observation in
+one `node_repl` call. Print candidate lists for selection when they are ambiguous.
+
+```js
+var apps = await computer.listApps();
+var matches = apps.filter(
+  (app) => app.name === 'Target App' || app.bundle_id === 'com.example.target',
+);
+nodeRepl.write(JSON.stringify(matches));
+```
+
+Use the matching app's observed PID as `targetPid`, then list its visible windows:
+
+```js
+var windows = await computer.listWindows({
+  pid: targetPid,
+  onScreenOnly: true,
+});
+nodeRepl.write(JSON.stringify(windows));
+```
+
+Select the intended document or dialog from the returned titles and window IDs;
+the first entry is not necessarily the task window. Set `target` to
+`{ pid: targetPid, windowId: selectedWindowId }` using that observed ID, then read
+its state:
+
+```js
+var state = await computer.observeWindow(target);
+nodeRepl.write(state.text); // This will return the accessibility tree
+```
+
+If you cannot identify an app from the task, prior context, or builtin apps,
+start by discovering the available apps:
+
+```js
+var apps = await computer.listApps();
+nodeRepl.write(JSON.stringify(apps));
+```
+
+After performing one or more UI actions, call `observeWindow(...)` before
+deciding what to do next. This keeps you in the current UI state. Read the
+current actionable `element_token` values from `state.elements`; tokens for
+unchanged elements remain valid across `diff` and `no_change` observations.
+
+For token efficiency, when appropriate, the accessibility tree will be returned
+as a diff from the most previous accessibility tree, listing only the elements
+that were removed, added, or changed. Prefer this default diff output; pass
+`disableDiff: true` only when you need a fresh full accessibility tree. If you
+disregard the text from a previous call to `observeWindow`, such as when you only
+emit the screenshot, get the full tree next time you inspect AX text.
+`state.elements` remains the current full actionable element list when
+`state.text` is a diff or reports no change.
+
+Returned text defaults to at most 12,000 characters. Set `maxTextChars` (minimum 512) to adjust the limit. When text is truncated, filter `state.elements` for
+the controls you need or request `disableDiff: true` with a larger
+`maxTextChars`. Do not repeatedly print the whole array. An omitted row does not
+prove absence. Incomplete captures retain only currently issued action tokens;
+after a read failure, use only tokens from the latest observation.
+
+### 2. Actions using the exact window
+
+Choose the delivery mode and current window before batching actions:
+
+- End the input batch when opening a dialog or menu. For a new dialog, call `listWindows({ pid, onScreenOnly: true })` and observe the matching returned window ID before typing. Do not guess an ID if no matching window is returned. For ordinary menus that remain in the current window, refresh that window's observation and use its current menu tokens.
+- An AX action can return an error after the app has already opened or closed a window. Check the current windows and state before retrying; do not assume the error means nothing happened or repeat the old token immediately.
+
+Batch actions whose target remains the same, then fetch the latest state:
+
+```js
+await computer.click({ pid: target.pid, elementToken });
+await computer.setValue({ pid: target.pid, elementToken, value: 'openai.com' });
+await computer.pressKey({ ...target, key: 'Enter' });
+await computer.typeText({ ...target, text: 'hello' });
+await computer.scroll({
+  pid: target.pid,
+  elementToken,
+  direction: 'down',
+  amount: 1,
+});
+await computer.performSecondaryAction({
+  pid: target.pid,
+  elementToken,
+  action: 'Show Menu',
+});
+nodeRepl.write((await computer.observeWindow(target)).text);
+```
+
+Notes:
+
+- Prefer `element_token`-based actions over coordinate actions. If AX actions or AX text are unavailable or behave unexpectedly, switch to screenshots, coordinate clicks, and key presses.
+- For window-targeted coordinate actions, use pixels in the PNG returned by `observeWindow` for that exact window, with its top-left corner as `(0, 0)`. AX `frame` values are screen-space logical points, not PNG pixels; do not pass an AX frame's center directly as `x`/`y`. Use a current element token or locate the control in the screenshot. The driver already accounts for display scale and image downscaling.
+- `doubleClick` and `rightClick` invoke their dedicated SDK actions; `rightClick` also accepts optional modifiers.
+- If the UI is not behaving as expected, try fetching the latest `observeWindow(...)` state to make sure you have the latest context.
+- Prefer using accessibility text over screenshots for efficiency, but if the interface is not fully working or not providing enough context, make sure to fetch a screenshot to get more context. The accessibility interface may be incomplete in some applications, so a screenshot helps fully understand what is going on.
+- `performSecondaryAction` invokes an accessibility action that an element exposes besides a normal click, such as expanding a disclosure row, showing a menu, incrementing a control, or cancelling something. It requires an action actually exposed for that element in the accessibility text. Do not guess action names.
+- `pressKey` presses one key and accepts optional modifiers; `hotkey` sends a key combination such as `{ keys: ['ctrl', 'c'] }`. Single-key examples include `"a"`, `"Enter"`, `"Tab"`, and `"Up"`.
+- Take care when passing strings containing `\n` or `\r` to `typeText`, as it simulates pressing the return key. Many apps with message composers or forms will respond by sending the message or submitting the form rather than inserting a newline.
+
+## Reading screenshots
+
+Request a screenshot with the observation, read its accessibility text, and
+emit each returned image:
+
+`includeScreenshot: true` is the parameter that requests a screenshot.
+
+```js
+var state = await computer.observeWindow({
+  ...target,
+  includeScreenshot: true,
+});
+nodeRepl.write(state.text);
+for (const image of state.screenshot?.images ?? []) {
+  await nodeRepl.emitImage(`data:${image.mimeType};base64,${image.dataBase64}`);
+}
+```
+
+When all Computer Use work is complete:
+
+```js
+await computer.close();
+globalThis.computer = undefined;
+```
+
+Reset the Node REPL only when no other persistent state is needed.

package/bundled/goal-draft/SKILL.md

@@ -66,7 +66,7 @@ Use exactly these labels, in this order. Keep the whole objective on one line wh
 Outcome: <one sentence: what is true when done>
 Done when: 1) <command> exits 0 and its output shows <…> (paste that line); 2) <file/state assertion provable via read or grep>; 3) …
 Must not: <files not to touch; tests/thresholds not to weaken; irreversible actions not to take>
-Budget: <user's stopping agreement; otherwise stop as blocked after 20 turns, and mark that default [ASSUMPTION] in Context>
+Budget: <user's advisory stopping agreement; otherwise stop as blocked after 20 turns, and mark that default [ASSUMPTION] in Context>
 On block: propose blocked with the exact blocker and the decision a human must make; never claim completion without evidence for every Done-when item
 Context: <only facts the agent cannot derive: paths, branch, environment, earlier decisions>
 ```
@@ -80,18 +80,18 @@ Rules of thumb:
 - Prefer "the smallest safe change in `<scope>`" over open-ended refactors.
 - Put anything that must not change on the way into Must not — this is what stops the loop from deleting a failing test to "pass".
 - Keep it short: everything the agent can derive from the workspace stays out. Aim for under ~1200 characters.
-- `Budget` is a stopping agreement for the model, not a runtime-enforced turn or wall-clock limit. Do not claim that writing it configures a timer or changes the Goal token budget. Preserve a user-specified budget; otherwise mark the default `[ASSUMPTION]` in Context.
+- `Budget` is a stopping agreement for the model, not a runtime-enforced turn or wall-clock limit. Do not claim that writing it configures a timer or changes the Goal token budget. When the user needs an enforced ceiling, tell them to set `model.goalMaxTurns` or `model.goalMaxActiveMinutes` in their settings -- never write the setting into the objective, which the Goal's own model cannot act on -- and say that either takes effect after a restart and only for Goals created afterwards -- neither bounds a Goal that is already running. Preserve a user-specified budget; otherwise mark the default `[ASSUMPTION]` in Context.
 
 For example, an audit's Done-when checks can require a report covering the agreed scenarios, observed results and evidence for each scenario, and reproduction steps for each confirmed defect (or an explicit "no confirmed defects" result). Ground the scenarios and report destination before offering the objective for use.
 
 ### Weak → strong
 
-| Weak                          | Strong                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
-| ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| make checkout faster          | Outcome: checkout API p95 is below 250 ms on the documented slow path. Done when: 1) `npm run bench:checkout` exits 0 and prints a p95 below 250 (paste the line); 2) `npm test` exits 0. Must not: change the benchmark, skip tests, touch files outside `src/checkout`. Budget: stop as blocked after 20 turns. On block: report the measured p95 and what blocks it. Context: [ASSUMPTION] the 20-turn budget is the drafter's default, not the user's. |
-| keep handling the PR comments | Outcome: every unresolved review thread on PR #123 is fixed or answered. Done when: 1) the review-threads query shows zero unresolved threads (paste the count); 2) CI on the head commit is green (paste the check summary). Must not: force-push, resolve a thread without replying to it. Budget: stop as blocked after 30 turns. On block: list the threads that need a maintainer decision.                                                           |
-| clean up the auth module      | Not a goal — "clean" has no check. Ask what would be observable (zero lint warnings in `src/auth`? a file count? a coverage threshold?) or offer a refactor plan instead.                                                                                                                                                                                                                                                                                  |
-| get the release out           | Not a goal as written — publishing is irreversible. Either narrow it to a checkable pre-release state (tag exists, changelog entry present, `npm run release:dry-run` exits 0) and put "do not publish" in Must not, or leave publishing to a human.                                                                                                                                                                                                       |
+| Weak                          | Strong                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
+| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| make checkout faster          | Outcome: checkout API p95 is below 250 ms on the documented slow path. Done when: 1) `npm run bench:checkout` exits 0 and prints a p95 below 250 (paste the line); 2) `npm test` exits 0. Must not: change the benchmark, skip tests, touch files outside `src/checkout`. Budget: as model guidance, stop as blocked after 20 turns. On block: report the measured p95 and what blocks it. Context: [ASSUMPTION] the 20-turn budget is the drafter's default, not the user's. |
+| keep handling the PR comments | Outcome: every unresolved review thread on PR #123 is fixed or answered. Done when: 1) the review-threads query shows zero unresolved threads (paste the count); 2) CI on the head commit is green (paste the check summary). Must not: force-push, resolve a thread without replying to it. Budget: as model guidance, stop as blocked after 30 turns. On block: list the threads that need a maintainer decision.                                                           |
+| clean up the auth module      | Not a goal — "clean" has no check. Ask what would be observable (zero lint warnings in `src/auth`? a file count? a coverage threshold?) or offer a refactor plan instead.                                                                                                                                                                                                                                                                                                     |
+| get the release out           | Not a goal as written — publishing is irreversible. Either narrow it to a checkable pre-release state (tag exists, changelog entry present, `npm run release:dry-run` exits 0) and put "do not publish" in Must not, or leave publishing to a human.                                                                                                                                                                                                                          |
 
 ## Step 5 — self-check, then hand off
 
@@ -114,10 +114,10 @@ Then hand off, and nothing else:
 
 **If the `propose_goal` tool is available and no Goal is active**, call it with the objective on one line. The user approves or declines it in a dialog; only their approval sets the Goal. If they decline you will not be told why: stop, do not ask about it, and do not propose the same or a reworded objective again. After approval, acknowledge it in one sentence and end the turn — the Goal runtime starts the first Goal turn on its own.
 
-**Otherwise** (Web Shell or another ACP client, headless, the tool is disabled, or a Goal is active), print:
+**Otherwise** (a client without Goal proposal support, headless, the tool is disabled, or a Goal is active), print:
 
 1. The objective in a fenced code block.
 2. One line the user can run as-is: `/goal set <objective on one line>` (or `/goal edit …` when tightening the active goal). Print it as plain text with no code markers, so it can be copied verbatim.
-3. One sentence explaining that the draft has not been applied and the user must run the command to apply it; include any `[ASSUMPTION]` items. Do not promise a dialog in Web Shell or other ACP sessions.
+3. One sentence explaining that the draft has not been applied and the user must run the command to apply it; include any `[ASSUMPTION]` items. Do not promise a dialog when the tool is unavailable.
 
 Do not run /goal yourself. Do not begin the task. Stop and wait for the user.