@remcp/runtime 0.2.48 → 0.2.50

package/CHANGELOG.md

@@ -2,6 +2,20 @@
 
 ## Unreleased
 
+## 0.2.50 — 2026-09-20
+
+- Lockstep release with the ReMCP client after the hosted MCP/plugin contract was hardened for current OpenAI/MCP Apps requirements.
+- Computer-use tools retain the 83-tool runtime surface with AI-oriented descriptions, conditional JSON Schema validation, structured outputs, dynamic capability advertising, and semantic Accessibility/DOM-first routing.
+- `browser_navigate` adds `action=new_tab` so CDP automation can create the first page target without a shell/browser-address-bar workaround.
+- Live Linux verification covers Tesseract OCR targeting, XDG RemoteDesktop keyboard/pointer input, Unicode `type_text`, and the complete 39-tool computer-use smoke.
+
+## 0.2.49 — 2026-09-20
+
+- Terminal process tools now return schema-validated structured session state, output ranges, match results and termination facts alongside their human-readable text.
+- The runtime catalog and hosted tool contract align on bounded structured outputs and accurate read-only/destructive/idempotent/open-world annotations for the expanded computer-use surface.
+- Preview references remain optional presentation metadata rather than replacing the source result used for model reasoning.
+- Lockstep patch release with the ReMCP client at `0.2.49`.
+
 ## 0.2.48 — 2026-09-20
 
 - Migrate the local stdio runtime to the official `@modelcontextprotocol/server` v2 serving entry while preserving legacy 2025-era client compatibility.
@@ -13,6 +27,10 @@
 - Added high-level `computer_snapshot` and `computer_action` primitives so agents can prefer semantic Accessibility/DOM targets and fall back to coordinates/screenshots only when necessary.
 - Chrome DevTools Protocol endpoints are restricted to loopback addresses. New filesystem/document operations continue through the existing resolved-path allowlist, and launch/service/power operations reuse the runtime command policy.
 - Fixed metadata output truncation once the JSON tool contract exceeded the operating-system pipe buffer by waiting for stdout writes to flush before exit.
+- Terminal session tools now expose schema-validated structured `pid`, status, exit information, output ranges/matches, and session lists alongside their human-readable text, so hosted agents can chain process calls without parsing prose.
+- Tool descriptions now state the intended routing boundary for exact-path vs glob reads, singular vs batch filesystem operations, full-screen vs targeted/semantic screenshots, and local runtime settings vs hosted device labels.
+- Desktop action schemas and common guards now reject targetless `ui_action`/`window_action`, coordinate-less pointer moves, zero-motion scrolls, incomplete cross-backend `computer_action` targets, and connectivity tests without host/port before platform adapters can act. Empty `ui_find`/`browser_find` calls are redirected to their snapshot tools, and `wait_for_ui` requires an explicit state/condition instead of silently acting as a no-op.
+- `take_screenshot` is correctly advertised as non-read-only/non-idempotent because kept or oversized captures can persist timestamped files; selector-scoped `browser_snapshot` screenshots restore the original page scroll position before returning so the inspection path remains read-only.
 - No browser/document rendering stack was added; the runtime remains dependency-light and uses native OS facilities plus existing qpdf/poppler/zip helpers when available.
 
 ## 0.2.47 — 2026-09-19

package/README.md

@@ -49,9 +49,11 @@ support and emits `notifications/tools/list_changed` when that capability set ch
 | Diagnostics | `service`, `event_log`, `network`, `installed_apps`, `environment`, `audio`, `power_action`, `record_screen` |
 | Documents | `read_document`, `edit_spreadsheet`, `edit_document`, `pdf_action` |
 
-The hosted ReMCP endpoint adds eight account/relay tools — `list_devices`, `ping_device`,
+The hosted ReMCP endpoint adds eleven account/presentation tools — `list_devices`, `ping_device`,
 `who_am_i`, `create_pairing_command`, `get_recent_tool_calls`, `get_usage_statistics`,
-`get_configuration`, and `rename_device` — for a 91-tool release contract.
+`get_configuration`, `render_file_preview`, `render_image_preview`, `render_terminal_preview`, and
+`rename_device` — for a 94-tool MCP Apps catalog. Plain MCP clients receive 91 tools because the
+three presentation-only render tools are omitted unless the UI extension is negotiated.
 
 `computer_snapshot` and `computer_action` keep OCR inside the compact high-level surface: when a local `tesseract` binary is available, snapshot can return bounded OCR text/boxes and click targeting can fall back through Accessibility → browser DOM/CDP → OCR → coordinates. OCR is optional and never adds another MCP tool or bundled OCR dependency.
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@remcp/runtime",
-  "version": "0.2.48",
+  "version": "0.2.50",
   "description": "First-party ReMCP local device runtime: filesystem, terminal, computer-use, browser, diagnostics and document tools over MCP.",
   "type": "module",
   "license": "MIT",

package/src/catalog.mjs

@@ -16,6 +16,72 @@ export const TEXT_OUTPUT_SCHEMA = {
   additionalProperties: false,
 };
 
+const NULLABLE_NUMBER_SCHEMA = { anyOf:[{ type:'number' }, { type:'null' }] };
+const NULLABLE_STRING_SCHEMA = { anyOf:[{ type:'string' }, { type:'null' }] };
+const TERMINAL_SESSION_PROPERTIES = {
+  pid:{ type:'number', description:'Process/session id used by follow-up terminal tools.' },
+  status:{ type:'string', description:'running or exited, with an exit code/signal when known.' },
+  runtimeMs:{ type:'number' },
+  lines:{ type:'number', description:'Total output lines produced by the session.' },
+  exited:{ type:'boolean' },
+  exitCode:NULLABLE_NUMBER_SCHEMA,
+  signal:NULLABLE_STRING_SCHEMA,
+};
+function terminalOutputSchema(extraProperties = {}, extraRequired = []) {
+  return {
+    type:'object',
+    properties:{ ...TERMINAL_SESSION_PROPERTIES, ...extraProperties },
+    required:['pid','status','runtimeMs','lines','exited','exitCode','signal', ...extraRequired],
+    additionalProperties:false,
+  };
+}
+const START_PROCESS_OUTPUT_SCHEMA = terminalOutputSchema({
+  command:{ type:'string' },
+  output:{ type:'string' },
+  partial:{ type:'string' },
+  warning:{ type:'string' },
+}, ['command','output']);
+const READ_PROCESS_OUTPUT_SCHEMA = terminalOutputSchema({
+  range:{ type:'string' },
+  output:{ type:'string' },
+  explicitOffset:{ type:'boolean' },
+}, ['range','output','explicitOffset']);
+const WAIT_PROCESS_OUTPUT_SCHEMA = terminalOutputSchema({
+  pattern:{ type:'string' },
+  matched:{ type:'boolean' },
+  bufferedMatch:{ type:'boolean' },
+  timeoutMs:{ type:'number' },
+  output:{ type:'string' },
+}, ['pattern','matched','bufferedMatch','timeoutMs','output']);
+const INTERACT_PROCESS_OUTPUT_SCHEMA = terminalOutputSchema({
+  output:{ type:'string' },
+}, ['output']);
+const TERMINATE_PROCESS_OUTPUT_SCHEMA = terminalOutputSchema({
+  terminated:{ type:'boolean' },
+  alreadyExited:{ type:'boolean' },
+  escalated:{ type:'boolean' },
+}, ['terminated','alreadyExited','escalated']);
+const LIST_SESSIONS_OUTPUT_SCHEMA = {
+  type:'object',
+  properties:{
+    sessions:{
+      type:'array',
+      items:{
+        type:'object',
+        properties:{
+          ...TERMINAL_SESSION_PROPERTIES,
+          blocked:{ type:'string' },
+          command:{ type:'string' },
+        },
+        required:['pid','status','runtimeMs','lines','exited','exitCode','signal','blocked','command'],
+        additionalProperties:false,
+      },
+    },
+  },
+  required:['sessions'],
+  additionalProperties:false,
+};
+
 const readOnly = { readOnlyHint: true, destructiveHint: false, idempotentHint: true, openWorldHint: false };
 const readOnlyNonIdempotent = { readOnlyHint: true, destructiveHint: false, idempotentHint: false, openWorldHint: false };
 const additive = { readOnlyHint: false, destructiveHint: false, idempotentHint: true, openWorldHint: false };
@@ -43,7 +109,7 @@ export const toolDefinitions = [
   {
     name: 'read_files',
     title: 'Read files by glob',
-    description: 'Read every file matching a glob under a directory in one call, each section prefixed with its path and line count. Use this to load a whole project area into context quickly instead of one read per file.',
+    description: 'Read files discovered by one glob under a directory, each section prefixed with its path and line count. Use this when the exact file list is not already known. If you already know the exact paths, use read_multiple_files; for one known path use read_file.',
     inputSchema: {
       type: 'object',
       properties: {
@@ -61,7 +127,7 @@ export const toolDefinitions = [
   {
     name: 'read_multiple_files',
     title: 'Read multiple files',
-    description: 'Read several text files in one call. Each file is returned separately and a failure to read one file does not stop the others.',
+    description: 'Read several explicitly named text files in one call. Use this when the exact paths are already known; each file is returned separately and one failure does not stop the others. Do not use this for glob discovery: use read_files instead. For one path use read_file.',
     inputSchema: {
       type: 'object',
       properties: {
@@ -173,7 +239,7 @@ export const toolDefinitions = [
   {
     name: 'write_file',
     title: 'Write file',
-    description: 'Create a file or change its full content. Parent directories are created automatically. Replaces the file by default; use mode "append" to add to the end. For binary data pass encoding-free base64 through write_binary instead.',
+    description: 'Create or replace exactly one text file. Parent directories are created automatically; mode "append" adds to the end. Use write_files for two or more independent files, and write_binary for binary bytes.',
     inputSchema: {
       type: 'object',
       properties: {
@@ -190,7 +256,7 @@ export const toolDefinitions = [
   {
     name: 'write_files',
     title: 'Write multiple files',
-    description: 'Create or replace many files in one call, each with its own path, content, and optional mode. Use this to scaffold a project or apply a multi-file change without one round trip per file.',
+    description: 'Create or replace two or more text files in one batch, each with its own path, content, and optional mode. Use this for scaffolding or coherent multi-file writes; for one path use write_file.',
     inputSchema: {
       type: 'object',
       properties: {
@@ -361,7 +427,7 @@ export const toolDefinitions = [
   {
     name: 'move_file',
     title: 'Move or rename',
-    description: 'Move or rename a file or directory. Replaces an existing destination file by default; pass overwrite false to refuse instead.',
+    description: 'Move or rename exactly one file or directory from one source to one destination. Replaces an existing destination file by default; pass overwrite false to refuse instead. Use move_paths for two or more moves.',
     inputSchema: {
       type: 'object',
       properties: {
@@ -378,7 +444,7 @@ export const toolDefinitions = [
   {
     name: 'copy_file',
     title: 'Copy file',
-    description: 'Copy one file to a new path, replacing the destination by default. Pass overwrite false to refuse an existing destination. Directories are not copied recursively.',
+    description: 'Copy exactly one regular file to a new path. Directories are not copied recursively. Use copy_paths for directories or two or more copy operations; pass overwrite false to refuse an existing destination.',
     inputSchema: {
       type: 'object',
       properties: {
@@ -395,7 +461,7 @@ export const toolDefinitions = [
   {
     name: 'copy_paths',
     title: 'Copy paths',
-    description: 'Copy many files or whole directories in one call, each with its own source and destination. Directories are copied recursively.',
+    description: 'Copy two or more paths, or copy a directory recursively, in one call. Each item has its own source and destination. For exactly one regular file use copy_file.',
     inputSchema: {
       type: 'object',
       properties: {
@@ -423,7 +489,7 @@ export const toolDefinitions = [
   {
     name: 'move_paths',
     title: 'Move paths',
-    description: 'Move or rename many files or whole directories in one call, each with its own source and destination. Falls back to copy-and-delete across filesystems.',
+    description: 'Move or rename two or more files/directories in one batch, each with its own source and destination. Falls back to copy-and-delete across filesystems. For one source/destination pair use move_file.',
     inputSchema: {
       type: 'object',
       properties: {
@@ -451,7 +517,7 @@ export const toolDefinitions = [
   {
     name: 'delete_path',
     title: 'Delete path',
-    description: 'Delete a file or a directory on the computer. Directories are removed with their contents unless recursive is false. The filesystem root is refused.',
+    description: 'Delete exactly one file or directory. Directories are removed with their contents unless recursive is false; the filesystem root is refused. Use delete_paths for two or more targets, or move_to_trash when deletion should be reversible.',
     inputSchema: {
       type: 'object',
       properties: {
@@ -467,7 +533,7 @@ export const toolDefinitions = [
   {
     name: 'delete_paths',
     title: 'Delete paths',
-    description: 'Delete many files and directories in one call, reporting each result. Use move_to_trash instead when the deletion should be reversible.',
+    description: 'Delete two or more files/directories in one batch and report each result. For one target use delete_path. Use move_to_trash instead when deletion should be reversible.',
     inputSchema: {
       type: 'object',
       properties: {
@@ -516,7 +582,7 @@ export const toolDefinitions = [
   {
     name: 'take_screenshot',
     title: 'Take screenshot',
-    description: 'Capture the screen of the paired computer and return it as an image, for GUI work, visual checks, and demonstrating what is on screen. Uses the standard desktop screenshot portal on Wayland, with grim, gnome-screenshot, spectacle, scrot, ImageMagick import, screencapture, or PowerShell fallbacks depending on the platform.',
+    description: 'Capture the entire desktop only when full-screen pixels are actually needed. Prefer ui_snapshot or browser_snapshot for semantic inspection, and screenshot_region for one window, monitor, or rectangle. This tool uses the platform screenshot portal/utility and returns an image.',
     inputSchema: {
       type: 'object',
       properties: {
@@ -525,7 +591,8 @@ export const toolDefinitions = [
       },
       additionalProperties: false,
     },
-    annotations: readOnly,
+    // keep=true and oversized fallbacks persist a timestamped PNG, so a repeated call can add files.
+    annotations: { readOnlyHint: false, destructiveHint: false, idempotentHint: false, openWorldHint: false },
     handler: fileToolHandlers.take_screenshot,
   },
   {
@@ -606,6 +673,7 @@ export const toolDefinitions = [
       additionalProperties: false,
     },
     annotations: command,
+    outputSchema: START_PROCESS_OUTPUT_SCHEMA,
     handler: terminalToolHandlers.start_process,
   },
   {
@@ -626,6 +694,7 @@ export const toolDefinitions = [
     // Reading without an explicit offset advances the session cursor, so retries can return
     // different output even after the process exits. It never modifies user files.
     annotations: readOnlyNonIdempotent,
+    outputSchema: READ_PROCESS_OUTPUT_SCHEMA,
     handler: terminalToolHandlers.read_process_output,
   },
   {
@@ -643,6 +712,7 @@ export const toolDefinitions = [
       additionalProperties: false,
     },
     annotations: command,
+    outputSchema: INTERACT_PROCESS_OUTPUT_SCHEMA,
     handler: terminalToolHandlers.interact_with_process,
   },
   {
@@ -660,6 +730,7 @@ export const toolDefinitions = [
       additionalProperties: false,
     },
     annotations: readOnlyNonIdempotent,
+    outputSchema: WAIT_PROCESS_OUTPUT_SCHEMA,
     handler: terminalToolHandlers.wait_for_process_output,
   },
   {
@@ -675,6 +746,7 @@ export const toolDefinitions = [
       additionalProperties: false,
     },
     annotations: { readOnlyHint: false, destructiveHint: true, idempotentHint: true, openWorldHint: false },
+    outputSchema: TERMINATE_PROCESS_OUTPUT_SCHEMA,
     handler: terminalToolHandlers.force_terminate,
   },
   {
@@ -683,6 +755,7 @@ export const toolDefinitions = [
     description: 'List terminal sessions started during this ReMCP runtime session with their status and how long they have been running.',
     inputSchema: { type: 'object', properties: {}, additionalProperties: false },
     annotations: readOnly,
+    outputSchema: LIST_SESSIONS_OUTPUT_SCHEMA,
     handler: terminalToolHandlers.list_sessions,
   },
   {
@@ -741,7 +814,7 @@ export const toolDefinitions = [
   {
     name: 'set_config_value',
     title: 'Change runtime setting',
-    description: 'Change one of this runtime’s own preferences on this computer: telemetryEnabled, maxReadLines, maxBufferedLines or maxOutputBytes. The change applies immediately and is saved to runtime.json. Access roots, blocked commands, the command guardrail, the shell and the write limit cannot be set through MCP — they stay with the person at this computer.',
+    description: 'Change one of this local runtime’s own preferences on the selected computer: telemetryEnabled, maxReadLines, maxBufferedLines or maxOutputBytes. This does not rename the paired device; use hosted rename_device for that label. Access roots, blocked commands, shell/guardrails and the write limit remain local-only.',
     inputSchema: {
       type: 'object',
       properties: {

package/src/extended/browser.mjs

@@ -310,10 +310,12 @@ export async function browserSnapshot(args) {
 
     await session.send('Page.enable');
     const selector = optionalString(args.selector);
+    let restoreScroll = null;
     if (selector) {
       const bounds = await evaluate(session, `new Promise((resolve,reject) => {
         const el=document.querySelector(${JSON.stringify(selector)});
         if(!el){reject(new Error('Screenshot element not found'));return;}
+        const originalScrollX=scrollX, originalScrollY=scrollY;
         const initial=el.getBoundingClientRect();
         const desiredTop=Math.max(0, scrollY + initial.top - Math.max(0,(innerHeight-initial.height)/2));
         window.scrollTo({top:desiredTop,left:scrollX,behavior:'instant'});
@@ -328,6 +330,8 @@ export async function browserSnapshot(args) {
             viewport_height:innerHeight,
             scroll_x:scrollX,
             scroll_y:scrollY,
+            original_scroll_x:originalScrollX,
+            original_scroll_y:originalScrollY,
             fully_visible:r.x>=0 && r.y>=0 && r.right<=innerWidth && r.bottom<=innerHeight
           });
         }));
@@ -335,7 +339,9 @@ export async function browserSnapshot(args) {
       if (!bounds || !Number.isFinite(Number(bounds.width)) || !Number.isFinite(Number(bounds.height)) || bounds.width <= 0 || bounds.height <= 0) {
         throw new Error('Screenshot element has invalid bounds');
       }
-      payload.screenshot_target = { selector, ...bounds };
+      restoreScroll = { x:Number(bounds.original_scroll_x || 0), y:Number(bounds.original_scroll_y || 0) };
+      const { original_scroll_x: _originalX, original_scroll_y: _originalY, ...reportedBounds } = bounds;
+      payload.screenshot_target = { selector, ...reportedBounds };
     }
     payload.screenshot = {
       scope: 'viewport',
@@ -343,11 +349,18 @@ export async function browserSnapshot(args) {
       height: await evaluate(session, 'innerHeight').catch(() => null),
     };
     const result = jsonResult(payload);
-    const captured = await session.send('Page.captureScreenshot', {
-      format: 'png',
-      fromSurface: true,
-      captureBeyondViewport: false,
-    });
+    let captured;
+    try {
+      captured = await session.send('Page.captureScreenshot', {
+        format: 'png',
+        fromSurface: true,
+        captureBeyondViewport: false,
+      });
+    } finally {
+      if (restoreScroll) {
+        await evaluate(session, `window.scrollTo({left:${restoreScroll.x},top:${restoreScroll.y},behavior:'instant'})`).catch(() => {});
+      }
+    }
     const data = String(captured?.data || '');
     if (!data) throw new Error('CDP returned no screenshot data');
     if (Buffer.byteLength(data, 'base64') > 8 * 1024 * 1024) throw new Error('Browser screenshot exceeds the 8 MiB inline image limit');
@@ -393,6 +406,7 @@ function findExpression(args) {
 }
 
 export async function browserFind(args) {
+  if (!(args.selector || args.text || args.role)) throw new Error('browser_find requires selector, text, or role; use browser_snapshot to enumerate page structure');
   return withTarget(args, async (session, target) => {
     const matches = await evaluate(session, findExpression(args));
     return jsonResult({ target_id: target.id, count: Array.isArray(matches) ? matches.length : 0, matches: Array.isArray(matches) ? matches : [] });

package/src/extended/catalog.mjs

@@ -34,14 +34,14 @@ const AI_TOOL_DESCRIPTIONS = Object.freeze({
   window_action: 'Use this to focus, minimize, maximize, restore, move, resize or close one known top-level window. Select it by id/PID/app/title from list_windows; do not use pointer coordinates for window management when this semantic tool can express the action.',
   launch_app: 'Use this to start a desktop application by executable/path or friendly application name without a shell. Prefer it over start_process for GUI application launch; use start_process for terminal commands, build tools and shell pipelines.',
   ui_snapshot: 'Use this to inspect native desktop Accessibility/UI Automation semantics for the active app before clicking or typing. Prefer scope=active; use scope=desktop only for cross-application discovery. For a Chromium web page with CDP available, prefer browser_snapshot for page DOM semantics.',
-  ui_find: 'Use this after ui_snapshot when you need a small set of native UI elements by role/name/label/AutomationId. Prefer returned id or compact label for subsequent ui_action; refresh only when the UI has changed.',
+  ui_find: 'Use this after ui_snapshot when you need a small set of native UI elements by role/name/label/AutomationId. It requires an element criterion; use ui_snapshot, not an empty ui_find, to enumerate UI. Prefer returned id or compact label for subsequent ui_action; refresh only when the UI has changed.',
   ui_action: 'Use this to invoke, focus, edit, select, toggle, expand/collapse, range-edit or scroll a native accessible UI element. Prefer this over pointer clicks because semantic actions survive layout changes; fall back to pointer only when accessibility cannot perform the requested action.',
   type_text: 'Use this for reliable Unicode text entry into a native semantic target or focused control. Prefer method=auto so Accessibility or clipboard restoration preserves the requested characters regardless of keyboard layout. Use method=keys only for deliberate physical key-by-key input; printable output then follows the current keyboard layout on the computer. Use keyboard for shortcuts, navigation keys and control keys.',
   keyboard: 'Use this for shortcuts and individual key presses such as Ctrl/Cmd+C, Tab, Enter or Escape. Do not use it to enter normal prose when type_text is available. On Wayland auto uses an already-authorized portal; backend=portal explicitly requests consent.',
   pointer: 'Use this only when an action truly requires screen coordinates or a semantic UI/browser action is unavailable. Prefer ui_action for native controls and browser_action for web controls; pointer is the coordinate fallback for move/click/button-down/button-up.',
   drag_drop: 'Use this for a real drag gesture between two UI element ids or coordinates. Prefer element ids from ui_find/ui_snapshot because their current bounds are resolved at execution time; use coordinates only when no semantic endpoints exist.',
   scroll: 'Use this for native wheel scrolling of a desktop region or semantic UI target. Prefer browser_action scroll_into_view for a known web element and ui_action scroll_into_view for a known accessible desktop element.',
-  wait_for_ui: 'Use this after a desktop action to wait for a native UI element/window/focus/text condition instead of sleeping or polling screenshots. Prefer semantic presence/absence/change conditions and set a bounded timeout.',
+  wait_for_ui: 'Use this after a desktop action to wait for an explicit native UI state or condition instead of sleeping or polling screenshots. Presence/absence waits need a semantic target; use state=changed for a broad tree change and set a bounded timeout.',
   clipboard: 'Use this for explicit text clipboard read/write/clear operations. Do not use clipboard as a typing workaround directly; type_text already uses safe clipboard paste with restoration when needed.',
   display_inventory: 'Use this before monitor-specific screenshots, cross-monitor geometry or coordinate work to obtain monitor bounds, scale and primary-display information.',
   screenshot_region: 'Use this when pixels are required for visual verification but only one window, monitor or rectangle matters. Prefer this over a full-desktop screenshot; use ui_snapshot/browser_snapshot first when semantic structure is sufficient.',
@@ -50,8 +50,8 @@ const AI_TOOL_DESCRIPTIONS = Object.freeze({
   notification: 'Use this only when the user wants a visible native desktop notification on the paired computer. Do not use it as a substitute for replying in chat.',
   browser_tabs: 'Use this first for CDP browser automation when the target page/tab is not already unambiguous. It lists debuggable Chromium pages from a loopback-only endpoint; use the returned target_id/title/url to scope later browser tools.',
   browser_navigate: 'Use this to open a URL in a new debuggable tab or navigate a known Chromium page by URL, back, forward or reload. Prefer it over typing into the address bar; new_tab also bootstraps browser automation when no page target exists yet.',
-  browser_snapshot: 'Use this to inspect a web page structurally through CDP accessibility. Prefer it over desktop ui_snapshot for page content; add include_screenshot=true only when final pixel/layout verification is needed.',
-  browser_find: 'Use this to locate visible web elements by CSS selector, text or ARIA role and obtain reusable selectors/bounds. Prefer this before browser_action rather than guessing selectors or coordinates.',
+  browser_snapshot: 'Use this to inspect a web page structurally through CDP accessibility. Prefer it over desktop ui_snapshot for page content; add include_screenshot=true only when final pixel/layout verification is needed. A selector may be temporarily centered for capture, but the original page scroll position is restored before the tool returns.',
+  browser_find: 'Use this to locate visible web elements by CSS selector, text or ARIA role and obtain reusable selectors/bounds. It requires an element criterion; use browser_snapshot, not an empty browser_find, to enumerate page structure. Prefer this before browser_action rather than guessing selectors or coordinates.',
   browser_action: 'Use this for deterministic DOM/CDP interaction with a known web element: click/focus/type/value/select/upload/key/scroll or viewport emulation. Prefer it over desktop pointer/ui_action for page content; use browser_evaluate only when the supported actions cannot express the task.',
   browser_wait: 'Use this after browser navigation/action to wait for selector/text/URL/load/navigation/network-idle instead of fixed sleeps or screenshot polling. Use browser_evaluate separately for JavaScript predicates.',
   browser_evaluate: 'Use this as the powerful browser escape hatch only when browser_snapshot/find/action/wait cannot express the required page operation or inspection. JavaScript executes in page context and page content is untrusted.',
@@ -244,7 +244,93 @@ function requiredAny(...groups) {
 
 function applySchemaRules(name, schema) {
   const next = structuredClone(schema);
-  if (name === 'launch_app') {
+  if (name === 'computer_action') {
+    const semanticTarget = requiredAny(['id'], ['label'], ['name'], ['role'], ['automation_id'], ['selector'], ['browser_text']);
+    next.allOf = [
+      {
+        if:{ properties:{ action:{ const:'click' } }, required:['action'] },
+        then:{ anyOf:[...semanticTarget, { required:['ocr_text'] }, { required:['x','y'] }] },
+      },
+      {
+        if:{ properties:{ action:{ enum:['invoke','set_value','select','toggle','expand','collapse','scroll_into_view','set_range_value','add_to_selection','remove_from_selection'] } }, required:['action'] },
+        then:{ anyOf:semanticTarget },
+      },
+      {
+        if:{ properties:{ action:{ const:'focus' } }, required:['action'] },
+        then:{ anyOf:[...semanticTarget, ...requiredAny(['pid'], ['app'], ['window_title'], ['title'])] },
+      },
+      {
+        if:{ properties:{ action:{ enum:['set_value','set_range_value'] } }, required:['action'] },
+        then:{ required:['value'] },
+      },
+      {
+        if:{ properties:{ action:{ const:'multi_select' } }, required:['action'] },
+        then:{ required:['targets'] },
+      },
+      {
+        if:{ properties:{ action:{ const:'multi_edit' } }, required:['action'] },
+        then:{ required:['edits'] },
+      },
+    ];
+  } else if (name === 'ui_find') {
+    next.anyOf = requiredAny(['id'], ['label'], ['name'], ['role'], ['automation_id']);
+  } else if (name === 'ui_action') {
+    next.allOf = [
+      { anyOf:requiredAny(['id'], ['label'], ['name'], ['role'], ['automation_id']) },
+      {
+        if:{ properties:{ action:{ enum:['set_value','set_range_value'] } }, required:['action'] },
+        then:{ required:['value'] },
+      },
+    ];
+  } else if (name === 'wait_for_ui') {
+    next.anyOf = requiredAny(['state'], ['condition']);
+    next.allOf = [
+      {
+        if:{ properties:{ condition:{ const:'text_exists' } }, required:['condition'] },
+        then:{ required:['text'] },
+      },
+      {
+        if:{ properties:{ condition:{ const:'active_window' } }, required:['condition'] },
+        then:{ anyOf:requiredAny(['text'], ['name'], ['window_title']) },
+      },
+      {
+        if:{ properties:{ condition:{ enum:['element_exists','element_enabled'] } }, required:['condition'] },
+        then:{ anyOf:requiredAny(['id'], ['label'], ['name'], ['role'], ['automation_id']) },
+      },
+      {
+        if:{ properties:{ state:{ enum:['present','absent'] } }, required:['state'] },
+        then:{ anyOf:requiredAny(['id'], ['label'], ['name'], ['role'], ['automation_id']) },
+      },
+    ];
+  } else if (name === 'window_action') {
+    next.allOf = [
+      { anyOf:requiredAny(['id'], ['pid'], ['app'], ['title']) },
+      {
+        if:{ properties:{ action:{ const:'move' } }, required:['action'] },
+        then:{ required:['x','y'] },
+      },
+      {
+        if:{ properties:{ action:{ const:'resize' } }, required:['action'] },
+        then:{ required:['width','height'] },
+      },
+      {
+        if:{ properties:{ action:{ const:'move_resize' } }, required:['action'] },
+        then:{ required:['x','y','width','height'] },
+      },
+    ];
+  } else if (name === 'pointer') {
+    next.allOf = [{
+      if:{ properties:{ action:{ const:'move' } }, required:['action'] },
+      then:{ required:['x','y'] },
+    }];
+  } else if (name === 'scroll') {
+    next.anyOf = requiredAny(['delta_x'], ['delta_y'], ['delta'], ['direction']);
+  } else if (name === 'network') {
+    next.allOf = [{
+      if:{ properties:{ action:{ const:'test' } }, required:['action'] },
+      then:{ required:['host','port'] },
+    }];
+  } else if (name === 'launch_app') {
     next.anyOf = requiredAny(['app'], ['path']);
   } else if (name === 'keyboard') {
     next.anyOf = requiredAny(['shortcut'], ['key'], ['keys']);
@@ -259,6 +345,8 @@ function applySchemaRules(name, schema) {
       { required:['x','y','width','height'] },
       ...requiredAny(['window_id'], ['pid'], ['app'], ['title'], ['monitor'], ['monitor_index']),
     ];
+  } else if (name === 'browser_find') {
+    next.anyOf = requiredAny(['selector'], ['text'], ['role']);
   } else if (name === 'browser_navigate') {
     next.allOf = [{
       if:{ properties:{ action:{ enum:['url','new_tab'] } }, required:['action'] },
@@ -464,7 +552,7 @@ export const extendedToolDefinitions = [
 
   define('browser_tabs', 'Browser tabs', 'List debuggable Chrome, Edge, or Chromium page targets from a loopback-only Chrome DevTools Protocol endpoint.', o({ ...browserTarget }), readOnlyLive, browserHandlers.browser_tabs, ['browser_cdp']),
   define('browser_navigate', 'Browser navigate', 'Open a new debuggable tab or navigate a selected browser page by URL, history back/forward, or reload through Chrome DevTools Protocol.', o({ ...browserTarget, action:e(['url','new_tab','back','forward','reload'], 'Default url when url is provided, otherwise reload. new_tab creates the first page target when needed.'), url:s('Destination URL for action=url or action=new_tab.'), wait:b('Wait for the page load event; default true.'), ignore_cache:b('Reload without cache when action=reload.') }), openMutatingNonDestructive, browserHandlers.browser_navigate, ['browser_cdp']),
-  define('browser_snapshot', 'Browser snapshot', 'Return the selected browser page accessibility tree through Chrome DevTools Protocol and, when requested, attach a real rendered viewport PNG for visual verification. A selector recenters the target before capture and returns its exact viewport bounds.', o({ ...browserTarget, max_nodes:n('Maximum AX nodes; default 1500.'), include_screenshot:b('Attach a real CDP-rendered viewport PNG screenshot; default false.'), selector:s('When include_screenshot=true, center this CSS-selected element before capture and report its viewport bounds.') }), readOnlyLive, browserHandlers.browser_snapshot, ['browser_cdp']),
+  define('browser_snapshot', 'Browser snapshot', 'Return the selected browser page accessibility tree through Chrome DevTools Protocol and, when requested, attach a real rendered viewport PNG for visual verification. A selector is temporarily recentered for capture, its exact viewport bounds are returned, and the prior scroll position is restored before return.', o({ ...browserTarget, max_nodes:n('Maximum AX nodes; default 1500.'), include_screenshot:b('Attach a real CDP-rendered viewport PNG screenshot; default false.'), selector:s('When include_screenshot=true, center this CSS-selected element before capture and report its viewport bounds.') }), readOnlyLive, browserHandlers.browser_snapshot, ['browser_cdp']),
   define('browser_find', 'Find browser element', 'Find visible DOM elements by CSS selector, text, or ARIA role and return reusable selectors plus text, value, and bounding boxes.', o({ ...browserTarget, selector:s('CSS selector.'), text:s('Visible text substring.'), role:s('ARIA role.'), limit:n() }), readOnlyLive, browserHandlers.browser_find, ['browser_cdp']),
   define('browser_action', 'Browser element action', 'Click, focus, type/set a value, select, scroll to, upload files, press a key, or set an exact responsive-test viewport in a debuggable browser page using DOM/CDP semantics.', o({ ...browserTarget, action:e(['click','focus','type','set_value','select','scroll_into_view','upload','press','set_viewport']), selector:s(), text:s(), value:s(), text_value:s(), option:s(), path:s(), paths:{type:'array',items:{type:'string'}}, key:s(), width:n('CSS viewport width for set_viewport.'), height:n('CSS viewport height for set_viewport.'), device_scale_factor:n('Device scale factor for set_viewport; default 1.'), mobile:b('Enable mobile emulation for set_viewport.') }, ['action']), openMutating, browserHandlers.browser_action, ['browser_cdp']),
   define('browser_wait', 'Wait for browser', 'Wait for DOM state, page text, URL, completed loading, a navigation away from the current URL, or a short network-idle period in a debuggable browser page. Use browser_evaluate separately for JavaScript predicates or page-specific inspection.', o({ ...browserTarget, condition:e(['selector','text','url_contains','load','navigation','network_idle']), selector:s(), text:s(), value:s(), poll_ms:n(), idle_ms:n('Required zero-in-flight network quiet window for network_idle; default 500 ms.') }), readOnlyLive, browserHandlers.browser_wait, ['browser_cdp']),

package/src/extended/desktop.mjs

@@ -273,7 +273,15 @@ export async function listWindows(args = {}) {
     return { ...window, monitor: monitor?.name ?? null, monitor_index: monitor?.index ?? null };
   }));
 }
-export async function windowAction(args = {}) { return adapter.windowAction(args); }
+export async function windowAction(args = {}) {
+  const hasTarget = Boolean(args.id || Number.isInteger(Number(args.pid)) || args.app || args.title);
+  if (!hasTarget) throw new Error('window_action requires id, pid, app, or title');
+  const action = optionalString(args.action);
+  if (action === 'move' && ![args.x,args.y].every(value => Number.isFinite(Number(value)))) throw new Error('window_action move requires x and y');
+  if (action === 'resize' && ![args.width,args.height].every(value => Number.isFinite(Number(value)))) throw new Error('window_action resize requires width and height');
+  if (action === 'move_resize' && ![args.x,args.y,args.width,args.height].every(value => Number.isFinite(Number(value)))) throw new Error('window_action move_resize requires x, y, width and height');
+  return adapter.windowAction(args);
+}
 export async function uiSnapshot(args = {}) {
   const explicitlyScoped = args.pid != null || args.app || args.window_title || args.windowTitle;
   const activeOnly = args.active_only != null
@@ -285,10 +293,18 @@ export async function uiSnapshot(args = {}) {
   return jsonResult(enrichUiPayload(value, effective));
 }
 export async function uiAction(args = {}) {
-  return adapter.uiAction(withResolvedUiLabel(args));
+  const resolved = withResolvedUiLabel(args);
+  const hasTarget = Boolean(resolved.id || resolved.label != null || resolved.name || resolved.role || resolved.automation_id || resolved.automationId);
+  if (!hasTarget) throw new Error('ui_action requires id, label, name, role, or automation_id');
+  const action = optionalString(resolved.action);
+  if (['set_value','set_range_value'].includes(action) && resolved.value == null) throw new Error(`ui_action ${action} requires value`);
+  return adapter.uiAction(resolved);
 }
 export async function keyboard(args = {}) { return adapter.keyboard(args); }
-export async function pointer(args = {}) { return adapter.pointer(args); }
+export async function pointer(args = {}) {
+  if (args.action === 'move' && ![args.x,args.y].every(value => Number.isFinite(Number(value)))) throw new Error('pointer move requires x and y');
+  return adapter.pointer(args);
+}
 export async function clipboard(args = {}) { return adapter.clipboard(args); }
 export async function displayInventory(args = {}) { return adapter.displayInventory(args); }
 export async function cursorPosition(args = {}) { return typeof adapter.cursorPosition === 'function' ? adapter.cursorPosition(args) : jsonResult({ x:null, y:null }); }
@@ -334,6 +350,8 @@ export async function screenshotRegion(args = {}) {
 export async function notification(args = {}) { return adapter.notification(args); }
 export async function scroll(args = {}) {
   const direction = optionalString(args.direction);
+  const hasDelta = [args.delta_x,args.delta_y,args.delta].some(value => value != null && Number.isFinite(Number(value)));
+  if (!direction && !hasDelta) throw new Error('scroll requires direction or a delta');
   const times = clamp(args.wheel_times, 1, 1, 50);
   let deltaX = Number(args.delta_x || 0);
   let deltaY = Number(args.delta_y ?? args.delta ?? 0);
@@ -403,6 +421,8 @@ async function uiMatches(args = {}) {
 }
 
 export async function uiFind(args = {}) {
+  const hasSelector = Boolean(args.id || args.label != null || args.name || args.role || args.automation_id || args.automationId);
+  if (!hasSelector) throw new Error('ui_find requires id, label, name, role, or automation_id; use ui_snapshot to enumerate UI');
   const nodes = await uiMatches(args);
   return jsonResult({ count: nodes.length, nodes });
 }
@@ -473,11 +493,17 @@ export async function waitForUi(args = {}) {
   const timeoutMs = clamp(args.timeout_ms, 10_000, 100, 120_000);
   const pollMs = clamp(args.poll_ms, 250, 50, 5000);
   const rawCondition = optionalString(args.condition);
+  const rawState = optionalString(args.state);
+  if (!rawCondition && !rawState) throw new Error('wait_for_ui requires state or condition');
   const aliases = { text:'text_exists', window:'active_window', element:'element_exists', enabled:'element_enabled', focused:'focused_element' };
   const condition = rawCondition
     ? requireEnum(aliases[rawCondition] || rawCondition, 'condition', ['text_exists','active_window','element_exists','element_enabled','focused_element'])
     : null;
-  const state = condition ? null : requireEnum(args.state || 'present', 'state', ['present', 'absent', 'changed']);
+  const state = condition ? null : requireEnum(rawState, 'state', ['present', 'absent', 'changed']);
+  const hasSelector = Boolean(args.id || args.label != null || args.name || args.role || args.automation_id || args.automationId);
+  if (['element_exists','element_enabled'].includes(condition) && !hasSelector) throw new Error(`wait_for_ui condition=${condition} requires a semantic target`);
+  if (condition === 'active_window' && !(args.text || args.name || args.window_title || args.windowTitle)) throw new Error('wait_for_ui condition=active_window requires text, name, or window_title');
+  if (['present','absent'].includes(state) && !hasSelector) throw new Error(`wait_for_ui state=${state} requires a semantic target`);
   const deadline = Date.now() + timeoutMs;
   const startedAt = Date.now();
   let attempts = 0;

package/src/tools/terminal.mjs

@@ -17,7 +17,7 @@ import {
   waitForProcessExit,
   waitForProcessActivity,
 } from '../sessions.mjs';
-import { clampInteger, fail, requireInteger, requireString, text } from '../util.mjs';
+import { clampInteger, fail, requireInteger, requireString, structured, text } from '../util.mjs';
 
 function shellCommand() {
   if (runtimeConfig.defaultShell) return runtimeConfig.defaultShell;
@@ -32,7 +32,15 @@ function shellArgs(command) {
 
 function describeSession(session) {
   const status = session.exited ? `exited${session.signal ? ` (${session.signal})` : session.exitCode === null ? '' : ` (code ${session.exitCode})`}` : 'running';
-  return { pid: session.pid, status, runtimeMs: (session.finishedAt || Date.now()) - session.startedAt, lines: totalLines(session) };
+  return {
+    pid: session.pid,
+    status,
+    runtimeMs: (session.finishedAt || Date.now()) - session.startedAt,
+    lines: totalLines(session),
+    exited: Boolean(session.exited),
+    exitCode: session.exitCode ?? null,
+    signal: session.signal ?? null,
+  };
 }
 
 function abortError() {
@@ -90,8 +98,16 @@ export async function startProcessTool(args, extra = {}) {
   const partial = session.partial;
   session.cursor = session.droppedLines + session.lines.length;
   session.lastPartialRead = session.partial || null;
-  const warning = verdict.note ? `${verdict.note}\n` : '';
-  return text(`${warning}${[headline, output, partial].filter(Boolean).join('\n')}`);
+  const warning = verdict.note ? String(verdict.note) : '';
+  const rendered = [warning, headline, output, partial].filter(Boolean).join('\n');
+  const status = describeSession(session);
+  return structured({
+    ...status,
+    command,
+    output,
+    ...(partial ? { partial } : {}),
+    ...(warning ? { warning } : {}),
+  }, rendered);
 }
 
 export async function readProcessOutputTool(args, extra = {}) {
@@ -124,7 +140,13 @@ export async function readProcessOutputTool(args, extra = {}) {
   }
   const status = describeSession(session);
   const header = `pid ${pid} ${status.status} · lines ${range}`;
-  return text(`${header}\n${slice.join('\n')}`);
+  const output = slice.join('\n');
+  return structured({
+    ...status,
+    range,
+    output,
+    explicitOffset:hasOffset,
+  }, `${header}\n${output}`);
 }
 
 function compileWaiter(pattern) {
@@ -164,7 +186,15 @@ export async function waitForProcessOutputTool(args, extra = {}) {
   const headline = matched
     ? `pid ${pid} ${status.status} · pattern matched${bufferedMatch ? ' (already buffered)' : ''}`
     : `pid ${pid} ${status.status} · pattern not matched within ${timeoutMs}ms`;
-  return text([headline, slice.join('\n')].filter(Boolean).join('\n'));
+  const output = slice.join('\n');
+  return structured({
+    ...status,
+    pattern,
+    matched,
+    bufferedMatch,
+    timeoutMs,
+    output,
+  }, [headline, output].filter(Boolean).join('\n'));
 }
 
 export async function interactWithProcessTool(args, extra = {}) {
@@ -195,34 +225,49 @@ export async function interactWithProcessTool(args, extra = {}) {
   }
   const slice = readNewOutput(session);
   const status = describeSession(session);
-  return text([`pid ${pid} ${status.status}`, slice.join('\n')].filter(Boolean).join('\n'));
+  const output = slice.join('\n');
+  return structured({
+    ...status,
+    output,
+  }, [`pid ${pid} ${status.status}`, output].filter(Boolean).join('\n'));
 }
 
 export async function forceTerminateTool(args) {
   const pid = requireInteger(args.pid, 'pid');
   const session = getProcessSession(pid);
   if (!session) fail(`No ReMCP session with pid ${pid}`);
-  if (session.exited) return text(`Process ${pid} already exited.`);
+  if (session.exited) {
+    const status = describeSession(session);
+    return structured({ ...status, terminated:false, alreadyExited:true, escalated:false }, `Process ${pid} already exited.`);
+  }
   killSessionTree(session, 'SIGTERM');
   const deadline = Date.now() + 2000;
   while (!session.exited && Date.now() < deadline) await waitForProcessActivity(session, 100);
+  let escalated = false;
   if (!session.exited) {
+    escalated = true;
     killSessionTree(session, 'SIGKILL');
     await waitForProcessActivity(session, 1000);
   }
   const status = describeSession(session);
-  return text(`Terminated session ${pid}${status.status.startsWith('exited') ? '' : ' (still running)'}. Status: ${status.status}.`);
+  return structured(
+    { ...status, terminated:Boolean(session.exited), alreadyExited:false, escalated },
+    `Terminated session ${pid}${status.status.startsWith('exited') ? '' : ' (still running)'}. Status: ${status.status}.`,
+  );
 }
 
 export async function listSessionsTool() {
   const sessions = listProcessSessions();
-  if (!sessions.length) return text('No active terminal sessions.');
-  const rows = sessions.map(session => {
+  const facts = sessions.map(session => {
     const status = describeSession(session);
     const blocked = session.exited ? '' : session.partial ? 'blocked-possibly' : 'idle-or-running';
-    return `pid ${status.pid} · ${status.status} · ${Math.round(status.runtimeMs / 1000)}s · ${blocked} · ${session.command.slice(0, 120)}`;
+    return { ...status, blocked, command:session.command.slice(0, 120) };
   });
-  return text(rows.join('\n'));
+  if (!facts.length) return structured({ sessions:[] }, 'No active terminal sessions.');
+  const rows = facts.map(session =>
+    `pid ${session.pid} · ${session.status} · ${Math.round(session.runtimeMs / 1000)}s · ${session.blocked} · ${session.command}`
+  );
+  return structured({ sessions:facts }, rows.join('\n'));
 }
 
 export const terminalToolHandlers = {