@logictan/dsh-desktop-agent 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Logic Tan
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,73 @@
+# @logictan/dsh-desktop-agent
+
+A DSH desktop agent that drives **one window on your own desktop** through the
+Cua Driver tools, using **screenshots as its primary sense**.
+
+Each step captures the target window, sends the picture to a vision-capable model
+from DSH's own model directory, and performs the click, key, or text entry the
+model chose. Nothing is hard-coded to a particular model and there is no second
+API key: every call goes through `ctx.llm`.
+
+## Why screenshots instead of the accessibility tree
+
+The accessibility tree is the right sense for ordinary applications, and this
+plugin still uses it — but it cannot see a game. A game is a self-drawn surface:
+it exposes no meaningful actionable nodes, and most native games additionally
+filter input that is routed by process id. For those targets the only channel that
+carries any information is the picture itself.
+
+So the plugin routes by capability rather than by configuration:
+
+| The resolved model declares `image` | Channel |
+| --- | --- |
+| yes | screenshot + a JSON action |
+| no, or the field is absent | the driver's element table and markdown tree |
+
+"Absent" is treated as "cannot see" on purpose. The host rewrites an image sent to
+a route that does not accept one into placeholder text, after which the model is
+describing a picture it never received — and answering confidently about it.
+
+## Requirements
+
+The profile must mount the Cua Driver computer-use provider; the plugin dispatches
+to the `cua_driver_native__*` tools it publishes and has no driver of its own.
+
+## Settings
+
+`Settings → Plugins → desktop-agent → configuration`:
+
+| Field | Meaning |
+| --- | --- |
+| Vision provider / model | The route used for each decision, chosen from the models that declare image input. All empty = the session's own current route. |
+| Vision reasoning effort | Optional, and only offered for models that declare one. |
+| Max steps | Actions per run; defaults to 40. |
+| Screenshot long edge | Pixels; defaults to 1568. Lower saves tokens, but too low hides the controls. |
+| Delivery | `background` (default) never steals focus; `foreground` is required by surfaces that filter per-process-routed input, such as canvas apps and games. |
+
+## Tool
+
+`desktop_agent({ app, goal, windowId })` resolves one window, runs the
+observe/decide/act loop until `DONE`/`BLOCKED` or a cap is reached, and returns a
+structured trace. `app` accepts a bundle id, an application name, or a window
+title; omitting it uses the frontmost window.
+
+Only on-screen windows are eligible. A covered window cannot be captured or acted
+on, so the tool fails with the window named rather than screenshotting whatever
+happened to be in front.
+
+## Scope
+
+v1 drives one window at a time, in the foreground of that window's own desktop,
+and is meant for tasks whose decisions are measured in seconds — turn-based and
+strategy games, simulations, settings panels, file managers.
+
+It is not for real-time action games: one vision decision costs 300–800 ms, an
+order of magnitude away from human reaction time. It does not read or write
+process memory, and it makes no attempt to evade anti-cheat protection.
+
+## Coordinates
+
+Action coordinates are pixels in the screenshot the model was shown, measured
+from its top-left corner. The driver resolves them against the capture the caller
+last saw, so every step re-observes before it acts — a new capture of a window
+also invalidates the element tokens of the previous one.

package/cordis.patch.yml

@@ -0,0 +1,18 @@
+# dsh-desktop-agent bundle patch layer.
+#
+# One bare plugin row. The package ships two halves — the host half
+# (exports ".") registers the `desktop_agent` model tool and exports the
+# `Config` schema the Plugins page derives its form from, the browser half
+# (exports "./client") registers the Plugins-page configuration card.
+#
+# The row must stay a BARE package name: the web plugin table locates each
+# package's `dsh.client` manifest from the specifier of the loader row that
+# mounts it, and that lookup accepts only a bare name — a subpath row resolves
+# to no manifest, so the browser half would silently never load.
+#
+# Row id `desktop-agent` equals the host half's `export const name` in
+# src/index.js. It is checked by hand against every other row id in this
+# repository; `aggregate.mjs --check` does not validate id uniqueness.
+- insert:
+    - id: desktop-agent
+      name: '@logictan/dsh-desktop-agent'

package/lib/actions.js

@@ -0,0 +1,195 @@
+/**
+ * The action vocabulary and its projection onto the Cua Driver tools.
+ *
+ * One decision from the model becomes exactly one driver call. The vocabulary is
+ * deliberately the driver's own, minus everything this plugin cannot verify or
+ * does not need: no `drag`, no clipboard, no recording, no browser family. Each
+ * entry here maps to a tool the driver actually exposes, so a decision the model
+ * can express is a decision that can be executed.
+ *
+ * Coordinates travel as screenshot pixels in the frame the last observation
+ * carried — see `geometry.js` for why that frame, and only that frame, is
+ * authoritative.
+ *
+ * @module @logictan/dsh-desktop-agent/actions
+ */
+import { isFrame, within } from './geometry.js';
+
+/** Every action kind the decision protocol accepts. */
+export const ACTION_KINDS = [
+  'click',
+  'double_click',
+  'right_click',
+  'type_text',
+  'press_key',
+  'hotkey',
+  'scroll',
+  'set_value',
+  'launch_app',
+  'wait',
+  'done',
+  'blocked',
+];
+
+/** Mouse actions, each addressed by a screenshot pixel. */
+const POINTER_KINDS = ['click', 'double_click', 'right_click'];
+
+/** Actions that end the run rather than reaching the driver. */
+const TERMINAL_KINDS = ['done', 'blocked'];
+
+/** Driver tool name per action kind. */
+const TOOL_BY_KIND = {
+  click: 'cua_driver_native__click',
+  double_click: 'cua_driver_native__double_click',
+  right_click: 'cua_driver_native__right_click',
+  type_text: 'cua_driver_native__type_text',
+  press_key: 'cua_driver_native__press_key',
+  hotkey: 'cua_driver_native__hotkey',
+  scroll: 'cua_driver_native__scroll',
+  set_value: 'cua_driver_native__set_value',
+  launch_app: 'cua_driver_native__launch_app',
+};
+
+/**
+ * Validate one decision and build the driver call it maps to.
+ *
+ * Validation is deliberately strict about the fields an action cannot run
+ * without. A model that names `click` and forgets the point has not made an
+ * ambiguous decision, it has made an incomplete one, and reporting that back is
+ * what lets the next decision correct it — silently defaulting to the window
+ * centre would perform an action nobody asked for.
+ *
+ * @param decision - the parsed decision.
+ * @param context - the observation frame and the target the action applies to.
+ * @returns the driver tool name and its arguments, or a terminal outcome.
+ * @throws {Error} when the decision is missing a field its action requires.
+ */
+export function plan(decision, context) {
+  const { kind } = decision;
+  if (!ACTION_KINDS.includes(kind)) {
+    throw new Error(`desktop_agent: "${kind}" is not an action. Choose one of: ${ACTION_KINDS.join(', ')}.`);
+  }
+  if (TERMINAL_KINDS.includes(kind)) return { terminal: kind };
+  if (kind === 'wait') {
+    const ms = decision.ms ?? 1000;
+    if (!Number.isFinite(ms) || ms < 0) {
+      throw new Error('desktop_agent: wait needs a non-negative number of milliseconds.');
+    }
+    return { wait: ms };
+  }
+
+  if (kind === 'launch_app') {
+    const args = {};
+    if (typeof decision.bundleId === 'string' && decision.bundleId !== '') args.bundle_id = decision.bundleId;
+    if (typeof decision.name === 'string' && decision.name !== '') args.name = decision.name;
+    if (args.bundle_id === undefined && args.name === undefined) {
+      throw new Error('desktop_agent: launch_app needs either bundleId or name.');
+    }
+    return { tool: TOOL_BY_KIND.launch_app, args };
+  }
+
+  const args = { pid: context.target.pid, window_id: context.target.windowId };
+  if (context.deliveryMode === 'foreground') args.delivery_mode = 'foreground';
+
+  if (POINTER_KINDS.includes(kind)) {
+    const point = requirePoint(decision, context);
+    const modifiers = Array.isArray(decision.modifiers) ? decision.modifiers : [];
+
+    // A MODIFIED double click is expressed as a two-count click: click is the
+    // only pointer tool whose schema carries both `count` and `modifier`
+    // (double_click carries modifier but no count), so one call can say "twice,
+    // with this modifier" without splitting the gesture across two tools.
+    if (kind === 'double_click' && modifiers.length > 0) {
+      args.x = point.x;
+      args.y = point.y;
+      args.count = 2;
+      args.modifier = modifiers;
+      return { tool: TOOL_BY_KIND.click, args };
+    }
+
+    args.x = point.x;
+    args.y = point.y;
+    if (kind === 'click' && Number.isInteger(decision.count) && decision.count > 1) args.count = decision.count;
+    if (kind !== 'double_click' && modifiers.length > 0) args.modifier = modifiers;
+    return { tool: TOOL_BY_KIND[kind], args };
+  }
+
+  if (kind === 'scroll') {
+    if (typeof decision.direction !== 'string') throw new Error('desktop_agent: scroll needs a direction.');
+    args.direction = decision.direction;
+    if (Number.isInteger(decision.amount)) args.amount = decision.amount;
+    if (typeof decision.by === 'string' && decision.by !== '') args.by = decision.by;
+    // A wheel event lands on whatever is under the cursor, which is the only way
+    // to reach a nested scroller that never takes keyboard focus. When the
+    // decision names a point, route through the pixel-wheel path.
+    if (decision.x !== undefined && decision.y !== undefined) {
+      const point = requirePoint(decision, context);
+      args.x = point.x;
+      args.y = point.y;
+    }
+    return { tool: TOOL_BY_KIND.scroll, args };
+  }
+
+  if (kind === 'type_text') {
+    if (typeof decision.text !== 'string' || decision.text === '') {
+      throw new Error('desktop_agent: type_text needs the text to type.');
+    }
+    args.text = decision.text;
+    return { tool: TOOL_BY_KIND.type_text, args };
+  }
+
+  if (kind === 'press_key') {
+    if (typeof decision.key !== 'string' || decision.key === '') {
+      throw new Error('desktop_agent: press_key needs a key.');
+    }
+    args.key = decision.key;
+    if (Array.isArray(decision.modifiers) && decision.modifiers.length > 0) args.modifiers = decision.modifiers;
+    return { tool: TOOL_BY_KIND.press_key, args };
+  }
+
+  if (kind === 'hotkey') {
+    if (!Array.isArray(decision.keys) || decision.keys.length < 2) {
+      throw new Error('desktop_agent: hotkey needs at least one modifier and one key, e.g. ["cmd", "s"].');
+    }
+    args.keys = decision.keys;
+    return { tool: TOOL_BY_KIND.hotkey, args };
+  }
+
+  // set_value: addressed by the element token from the latest AX observation.
+  if (typeof decision.token !== 'string' || decision.token === '') {
+    throw new Error('desktop_agent: set_value needs the element token of the target.');
+  }
+  if (typeof decision.value !== 'string') throw new Error('desktop_agent: set_value needs a value.');
+  args.element_token = decision.token;
+  args.value = decision.value;
+  return { tool: TOOL_BY_KIND.set_value, args };
+}
+
+/**
+ * Read and bound-check a decision's point against the observation frame.
+ *
+ * @param decision - the decision carrying `x` and `y`.
+ * @param context - the observation frame the point is read in.
+ * @returns the integral point.
+ * @throws {Error} when the point is missing, non-numeric, or outside the frame.
+ */
+function requirePoint(decision, context) {
+  const { x, y } = decision;
+  if (!isFrame(context.frame)) {
+    throw new Error(
+      'desktop_agent: no screenshot frame is available for this action, so a coordinate cannot be placed. ' +
+        'This run is on the accessibility channel; use set_value with an element token instead.',
+    );
+  }
+  if (!Number.isFinite(x) || !Number.isFinite(y)) {
+    throw new Error('desktop_agent: this action needs numeric x and y in the screenshot coordinate space.');
+  }
+  const point = { x: Math.round(x), y: Math.round(y) };
+  if (!within(point, context.frame)) {
+    throw new Error(
+      `desktop_agent: the point (${point.x}, ${point.y}) lies outside the ${context.frame.width}×${context.frame.height} screenshot; ` +
+        're-read the coordinates from the screenshot you were shown.',
+    );
+  }
+  return point;
+}

package/lib/ask.js

@@ -0,0 +1,87 @@
+/**
+ * The one model call this plugin makes: ask for the next action.
+ *
+ * The call goes through the host's own `ctx.llm`, so there is no second API key,
+ * no second provider configuration, and nothing for config sync to strip. It is
+ * also what makes the channel decision real: the SAME route that will answer is
+ * the one the plugin asks about image support before it decides whether to send a
+ * screenshot or an element table.
+ *
+ * @module @logictan/dsh-desktop-agent/ask
+ */
+import { BlockAssembler, createUserMessage } from '@deepseek-ai/dsh-llm';
+import { INSTRUCTION, MAX_TOKENS, decisionPayload } from './request.js';
+import { NO_ROUTE_MESSAGE } from './route.js';
+
+/**
+ * The producer-owned source kind stamped on every message this plugin sends.
+ *
+ * Session format v4 refuses the retired `{ kind: 'plugin', plugin }` wrapper at
+ * native admission, so the identity rides `kind` itself under the host's
+ * documented `plugin:` prefix for a producer that is not a first-party plugin.
+ */
+export const SOURCE = { kind: 'plugin:dsh-desktop-agent' };
+
+/**
+ * Ask the routed model for one decision.
+ *
+ * @param input - the `ctx.llm` service, the route, the goal, the observation, the
+ *   history, and the caller's cancellation signal.
+ * @returns the model's raw answer text.
+ * @throws {Error} an actionable message when the call failed or produced no text.
+ */
+export async function ask(input) {
+  const { llm, route, goal, observation, history, signal } = input;
+
+  if (route.provider === '' || route.model === '') {
+    throw new Error(NO_ROUTE_MESSAGE);
+  }
+
+  const assembler = new BlockAssembler();
+  const options = {
+    provider: route.provider,
+    model: route.model,
+    messages: [
+      createUserMessage({ content: [{ type: 'text', text: INSTRUCTION }], source: SOURCE }),
+      createUserMessage({ content: contentOf(observation, decisionPayload({ goal, observation, history })), source: SOURCE }),
+    ],
+    maxTokens: MAX_TOKENS,
+    ...(route.reasoningEffort === '' ? {} : { reasoningEffort: route.reasoningEffort }),
+    ...(signal === undefined ? {} : { signal }),
+  };
+
+  for await (const chunk of llm.stream(options)) assembler.push(chunk);
+
+  const finish = assembler.finish;
+  if (finish.kind === 'error' || finish.kind === 'aborted') {
+    throw new Error(`desktop_agent: the decision call failed (${finish.failure.message}).`);
+  }
+
+  const text = assembler
+    .blocks()
+    .filter((block) => block.type === 'text')
+    .map((block) => block.text)
+    .join('');
+
+  if (text.trim() === '') throw new Error('desktop_agent: the model returned no decision.');
+  return text;
+}
+
+/**
+ * Build the content blocks of the data message.
+ *
+ * The screenshot is carried as a durable attachment reference, which is the only
+ * shape the harness admits an image in. The bytes are committed through
+ * `ctx.attachments` first: an inline base64 block is not a valid request part,
+ * and a reference the store never issued would be refused at the provider
+ * boundary.
+ *
+ * @param observation - the observation the decision is made against.
+ * @param payload - the serializable data payload.
+ * @returns the content blocks, image first.
+ */
+function contentOf(observation, payload) {
+  const text = { type: 'text', text: JSON.stringify(payload) };
+  if (observation.channel !== 'vision') return [text];
+  return [{ type: 'image', attachment: observation.attachment }, text];
+}