mobile-debug-mcp 0.31.0 → 0.32.0

package/docs/benchmarks/interaction-latency/final.md

@@ -0,0 +1,17 @@
+# Interaction latency benchmark
+
+Commit: d3e6ae720db1ad1da294795790689df5a80b56b0 (working tree candidate). Fixture: 1.0, SHA-256 98244deeeba0a21da7d99a3e4b43c97806d6f8a8787ad7fe38249e5d311a6fb6.
+
+Environment: Pixel_9_Pro, Android 16, API 36. Three warm-ups and ten measured runs per mode. Failed runs excluded.
+
+| Mode | p50 ms | p95 ms | Failures |
+|---|---:|---:|---:|
+| baseline | 124094.50754200004 | 124597.88916599983 | 0 |
+| full | 87262.29954100028 | 88067.6383750001 | 0 |
+| light | 51817.59008300025 | 53873.19891700009 | 0 |
+
+First failure by mode: baseline: none; full: none; light: none.
+
+Full p95 gate: true. Light p50 gate: true.
+
+See final.json for requests, timing components, and failure selectors.

package/docs/benchmarks/interaction-latency/ios-validation.md

@@ -0,0 +1,11 @@
+# Spec 018 iOS device validation
+
+Validated on 27 September 2026 with the booted iPhone Air simulator (iOS 26.4.1).
+
+- `npm run test:device` passed all six Android and iOS screenshot, log, and UI-tree smoke checks.
+- Over a spawned stdio MCP server, `start_app` launched Settings with `light`/`compact` and returned a fresh UI observation.
+- A selector-based `tap_element` opened General with `light`/`compact`; its result included an available state delta.
+- `run_journey` used two selector taps to return to General and open About, then asserted a visible `Model Name` label. All three steps passed with `light`/`compact`.
+- A separate `full`/`debug` selector tap returned to General and reported a fresh observation. A `none`/`compact` selector tap reopened About without requesting post-action verification; a separate UI-tree read confirmed the destination.
+
+The performance benchmark and its fixture are Android-only. These iOS checks exercised the interaction contract on a system app; they did not measure latency on an iOS app with network requests.

package/docs/specs-deltas/018-interaction-latency-and-batched-journeys.md

@@ -0,0 +1,301 @@
+# RFC 018 — Interaction Latency, Verification Modes, and Batched Journeys
+
+## Status
+
+Implemented; Android and iOS device validation passed.
+
+## 1. Summary
+
+This RFC reduces mobile-debug MCP round-trip latency while retaining an explicit path to the current verification guarantees.
+
+It adds optional execution controls to action tools, compact responses, selector-based tapping, state-delta observations, and a batched `run_journey` tool. Existing callers remain compatible: omitted controls retain the current full verification and detailed response behavior.
+
+## 2. Problem
+
+The current path performs more device work and returns more data than most interaction turns need.
+
+- `start_app` captures a screen fingerprint before and after launch, in addition to the launch adapter's current-screen observation.
+- Coordinate actions capture fingerprints before and after dispatch.
+- `tap_element` captures a fingerprint, re-reads the full UI tree to revalidate a stored handle, dispatches, then captures another fingerprint.
+- `capture_debug_snapshot` always starts a screenshot collection, even when the caller needs only hierarchy or logs.
+- A normal agent journey requires separate resolve, tap, wait, expect, and snapshot calls, each with a protocol round trip and repeated device metadata.
+
+These checks are valuable when diagnosing an ambiguous result. They are unnecessary for every known-safe, immediately-followed step, and they make the server appear slow even when device execution is prompt.
+
+## 3. Goals
+
+1. Make the latency of `start_app`, `tap_element`, `wait_for_ui`, `capture_screenshot`, and `capture_debug_snapshot` measurable and reportable.
+2. Let callers choose `none`, `light`, or `full` verification per interaction without weakening the compatibility default.
+3. Make normal responses small and use `debug` only when detailed traces and device metadata are needed.
+4. Resolve and tap a selector in one server request.
+5. Run ordered interaction journeys in one device session and stop at the first failed assertion.
+6. Return changed accessibility nodes rather than repeated complete trees when a prior snapshot is available.
+7. Use UI evidence as soon as it is available; do not add a fixed settling delay after a fresh expected state is observed.
+
+## 4. Non-goals
+
+- Changing the meaning of existing `expect_*` tools or their stabilization rules.
+- Inferring app-specific expected outcomes when the caller has not supplied one.
+- Removing `capture_screenshot` or the detailed debug snapshot.
+- Bypassing target actionability checks for selector-based actions.
+- Making `none` appropriate for actions with an asserted business outcome.
+- Reporting benchmark figures without a controlled device run.
+
+## 5. Compatibility and defaults
+
+All new inputs are optional. For every existing action request:
+
+```text
+verificationMode = "full"
+responseMode = "debug"
+```
+
+is the effective behavior. Existing response fields, including action envelopes, fingerprints, device metadata, and traces, remain available in that mode. No existing tool is removed or renamed.
+
+New clients SHOULD use `responseMode: "compact"` and select the least costly verification mode that satisfies their next decision. `full` remains required for a final assertion unless that assertion is separately expressed as an `expect_*` or journey assertion.
+
+## 6. Common interaction controls
+
+The following optional properties are added to `start_app`, `restart_app`, `tap`, `tap_element`, `swipe`, `scroll_to_element`, `type_text`, and `press_back`. A tool that cannot apply a field MUST reject it with `INVALID_ARGUMENT`, rather than silently changing its semantics.
+
+```ts
+verificationMode?: "none" | "light" | "full" // default: "full"
+responseMode?: "compact" | "debug"            // default: "debug"
+actionTimeoutMs?: number                         // adapter dispatch limit
+verificationTimeoutMs?: number                   // verification budget
+```
+
+Values are finite integers. `actionTimeoutMs` is clamped to 100–30,000 ms, and `verificationTimeoutMs` to 100–60,000 ms. The server reports effective values in debug responses. Platform adapters MUST receive and enforce `actionTimeoutMs`; it must not be a cosmetic field.
+
+Unless supplied, the effective dispatch budget is 5,000 ms and the effective light-verification budget is 3,000 ms. `start_app` has a 10,000 ms dispatch default because launch includes an adapter-owned foreground check. `restart_app` has a 15,000 ms total dispatch budget: at most 5,000 ms for termination and the remaining budget, capped at 10,000 ms, for launch. `scroll_to_element` has a 10,000 ms total dispatch budget; the budget covers every scroll and resolve attempt, so no attempt may start after its deadline. Existing `full` calls retain their present adapter and stabilization behavior when the fields are omitted; a supplied budget bounds only work introduced by this RFC.
+
+The server creates one deadline at dispatch start and passes its remaining duration to each nested operation. It cancels local polling and child processes when the deadline expires. A timeout before the command is issued is `ACTION_TIMEOUT`, `success: false`, `delivery_status: "not_delivered"`, and `retryable: true`. A timeout after dispatch begins is `ACTION_TIMEOUT`, `success: false`, `delivery_status: "unknown"`, and `retryable: false`: the device may have acted, so the client MUST observe state before retrying. The response includes `dispatch_started: boolean` and elapsed timing in both response modes.
+
+### 6.1 Verification modes
+
+| Mode | Required work | Result meaning |
+|---|---|---|
+| `none` | Resolve only when required to dispatch; issue the command. No post-dispatch UI read. | `success` means the adapter accepted the action. |
+| `light` | Dispatch, then take one fresh accessibility-tree observation. | `success` means delivery succeeded; `state_delta` is best-effort evidence, not a passed assertion. |
+| `full` | Preserve the current tool's pre/post evidence, stale-reference checks, stabilization, and trace behavior. | Existing meaning is unchanged. |
+
+`light` MUST not wait for an arbitrary settling interval. It MAY poll only until the first fresh observation arrives, bounded by `verificationTimeoutMs`. If the observation cannot be obtained in time, return `success: true`, `verification.status: "unavailable"`, and a retryable verification diagnostic; do not recast a delivered action as a dispatch failure. `verification.status: "unavailable"` is therefore a successful action with incomplete evidence, never a failed action.
+
+`none` and `light` return `lifecycle_state: "pending_verification"` after delivery. `full` retains current lifecycle-state behavior.
+
+### 6.2 Response modes
+
+`debug` preserves the existing response shape. It includes device metadata, full action trace, fingerprints, resolved target detail, timing breakdown, and adapter diagnostics when present.
+
+`compact` returns only:
+
+```ts
+{
+  action_id: string,
+  action_type: string,
+  success: boolean,
+  lifecycle_state: "pending_verification" | "verified" | "failed",
+  failure_code?: string,
+  retryable?: boolean,
+  delivery_status?: "delivered" | "not_delivered" | "unknown",
+  dispatch_started: boolean,
+  verification?: { mode: "none" | "light" | "full", status: "not_requested" | "observed" | "verified" | "unavailable" | "failed" },
+  state_delta?: StateDelta,
+  timing: { total_ms: number }
+}
+```
+
+The compact form MUST retain failure and retry information. It MUST NOT emit device identifiers, full tree contents, raw adapter output, screenshots, fingerprints, or trace steps. A client can repeat an action only with a new `responseMode: "debug"`; it cannot expand a past compact result.
+
+## 7. Selector-based tap
+
+`tap_element` accepts exactly one of the existing `elementId` or a new `selector`; this does not change the `elementId` path.
+
+```ts
+selector?: {
+  text?: string,
+  resource_id?: string,
+  accessibility_id?: string,
+  contains?: boolean
+}
+waitFor?: {
+  condition?: "exists" | "visible" | "clickable",
+  timeoutMs?: number,       // default 1,500; range 100–10,000
+  pollIntervalMs?: number   // default 100; range 50–1,000
+  match?: { index?: number }
+}
+```
+
+With a selector, the server resolves a fresh UI tree, verifies that the selected node is actionable, and taps it in the same request. `waitFor` means wait for the target before resolving; it is not post-tap verification. Without `waitFor`, resolution occurs once and fails with `ELEMENT_NOT_FOUND`, `AMBIGUOUS_TARGET`, or `ELEMENT_NOT_INTERACTABLE` as applicable. A selector match must use the same matching and index semantics as `wait_for_ui`.
+
+When `verificationMode: "full"`, selector resolution replaces the separate client resolve call but does not skip required post-action verification. The resolved element returned in debug mode includes only the selected node, never the complete tree.
+
+## 8. State-delta snapshots and screenshot policy
+
+The snapshot metadata layer already tracks stable node identities and aggregate change counts. It is extended to retain the prior normalized node record by device and expose changed nodes.
+
+```ts
+type StateDelta = {
+  base_snapshot_revision: number | null,
+  snapshot_revision: number,
+  added: UIElement[],
+  removed: Array<{ stable_id?: string, resourceId?: string, contentDescription?: string, text?: string, index: number }>,
+  changed: UIElement[],
+  truncated: boolean
+  omitted_changes?: number
+}
+```
+
+For each `(platform, deviceId)`, the server retains normalized node records for the latest eight revisions or ten minutes, whichever expires first. It retains at most 500 nodes per revision. A delta contains at most 200 changed nodes; additional changes set `truncated: true` and report `omitted_changes`. Revision state is process-local and is discarded on server restart, device disconnect, or retention expiry.
+
+`get_ui_tree` adds `responseMode` and `sinceSnapshotRevision`:
+
+- `responseMode: "debug"` remains the complete tree response.
+- `responseMode: "compact"` with a matching `sinceSnapshotRevision` returns `state_delta`; it does not include unchanged nodes.
+- `responseMode: "compact"` without `sinceSnapshotRevision` returns `{ delta_available: false, snapshot_revision }` and no nodes. The caller obtains a base with a debug tree.
+- If the requested base revision is unavailable, return `{ delta_available: false, snapshot_revision, reason: "BASE_REVISION_UNAVAILABLE" }` and no nodes. The client recovers by requesting a debug tree, then resumes from its returned revision.
+- A truncated delta is valid only as a change signal. The client MUST request a debug tree before using it to make a target-selection decision.
+
+For a `light` action, the server takes exactly one post-dispatch tree. It computes `state_delta` against the most recent retained revision for the same device, if one exists; otherwise it returns `state_delta: undefined` and `state_delta_available: false`. It never takes a pre-dispatch tree merely to create a delta. The returned `base_snapshot_revision` tells the client whether its local state can safely apply that delta.
+
+Within `run_journey`, an immediately following `element_visible` assertion may supply that fresh post-dispatch tree for a `light` tap. The tap reports `verification.status: "observed"` when the assertion returns a matched count, even if the assertion fails; it reports `"unavailable"` with a retryable diagnostic if the observation cannot be confirmed.
+
+`capture_debug_snapshot` adds `includeScreenshot?: boolean`, defaulting to `true` for compatibility. New compact callers MUST set it explicitly; `includeScreenshot: false` collects no screenshot. `capture_screenshot` remains the explicit screenshot tool. No interaction tool captures a screenshot unless its request explicitly asks for one through its observation configuration.
+
+## 9. `run_journey`
+
+`run_journey` runs a bounded ordered sequence against one resolved platform and device. It avoids MCP round trips; it does not hold a cross-request device lock or conceal failures.
+
+```ts
+{
+  platform: "android" | "ios",
+  deviceId?: string,
+  responseMode?: "compact" | "debug", // default compact
+  captureOnFailure?: boolean,            // default false
+  defaults?: { verificationMode?: "none" | "light" | "full", actionTimeoutMs?: number, verificationTimeoutMs?: number },
+  steps: Array<
+    { id: string, type: "start_app", appId: string, verificationMode?: VerificationMode } |
+    { id: string, type: "wait", selector: Selector, condition?: "exists" | "not_exists" | "visible" | "clickable", timeoutMs?: number, pollIntervalMs?: number } |
+    { id: string, type: "tap", selector?: Selector, elementId?: string, waitFor?: WaitFor, verificationMode?: VerificationMode } |
+    { id: string, type: "assert", assertion: ElementVisibleAssertion | ElementAbsentAssertion | ScreenFingerprintAssertion | StateEqualsAssertion, timeoutMs?: number, pollIntervalMs?: number }
+  >
+}
+```
+
+```ts
+type ElementVisibleAssertion = { kind: "element_visible", selector: Selector }
+type ElementAbsentAssertion = { kind: "element_absent", selector: Selector }
+type ScreenFingerprintAssertion = { kind: "screen_fingerprint", fingerprint: string }
+type StateEqualsAssertion = {
+  kind: "state_equals",
+  property: string,
+  expected: boolean | number | string,
+  selector?: Selector,
+  elementId?: string
+}
+```
+
+Step IDs must be unique. Maximum steps: 50. A journey validates its complete schema before device work begins. `tap` requires exactly one target. `state_equals` requires exactly one of `selector` or `elementId`; every other assertion rejects irrelevant fields. `start_app` is the journey equivalent of `start_app`.
+
+Journey assertions use a 5,000 ms timeout and 100 ms poll interval unless supplied; their ranges are 100–60,000 ms and 50–1,000 ms. `element_visible` is the existing `expect_element_visible` predicate. `element_absent` is a new journey-only predicate: zero matching nodes must be seen in two fresh trees at least 100 ms apart, within the assertion timeout. `screen_fingerprint` matches the current fingerprint on two consecutive reads 100 ms apart. `state_equals` uses the existing `expect_state` stabilization contract, with the journey's supplied or default timeout and poll interval applied as that contract's budget and observation interval; it does not add a second two-read rule. Assertions return `ASSERTION_TIMEOUT` on budget expiry and include the last observed value or matched count. A `wait` remains synchronization, not proof of a business outcome.
+
+The journey resolves the platform and device once before the first step and passes that exact context to every step. An `elementId` is valid only when its stored platform and device ID equal the journey context. A mismatch is `STALE_REFERENCE`, no tap is dispatched, and the journey stops.
+
+The response is:
+
+```ts
+{
+  success: boolean,
+  stopped_at_step_id: string | null,
+  steps: Array<ExecutedJourneyStep | NotRunJourneyStep>,
+  timing: TimingBreakdown
+}
+
+type ExecutedJourneyStep = {
+  id: string, type: string, status: "passed" | "failed",
+  success: boolean, result: CompactOrDebugStepResult, timing: TimingBreakdown
+}
+type NotRunJourneyStep = {
+  id: string, type: string, status: "not_run", success: false,
+  reason: "PREVIOUS_STEP_FAILED", result: null, timing: { total_ms: 0 }
+}
+```
+
+It stops immediately after a failed assertion, failed wait, target-resolution failure, or dispatch failure. Later steps have the `NotRunJourneyStep` shape and are never attempted. An action with `verification.status: "unavailable"` has `status: "passed"` because dispatch succeeded; a following assertion decides whether its intended outcome occurred. Debug mode includes the failing step's trace and a snapshot only when `captureOnFailure: true` is passed at journey level. The failure snapshot has `includeScreenshot: false` unless the caller separately requests a screenshot.
+
+## 10. Timing measurement and reporting
+
+Every specified tool records monotonic durations; wall clock is used only for timestamps. The timing schema is additive:
+
+```ts
+type TimingBreakdown = {
+  total_ms: number,
+  device_resolution_ms?: number,
+  pre_observation_ms?: number,
+  target_resolution_ms?: number,
+  dispatch_ms?: number,
+  post_observation_ms?: number,
+  verification_ms?: number,
+  screenshot_ms?: number,
+  serialization_ms?: number
+}
+```
+
+`debug` returns the full applicable breakdown. `compact` returns `total_ms`; `run_journey` additionally returns each step's `total_ms`. Errors still return elapsed timing.
+
+A checked-in benchmark script must emit JSON and Markdown to `docs/benchmarks/interaction-latency/`; reports contain commit SHA, server version, platform, device model/OS, app build identifier, sample count, warm-up policy, verification/response modes, and p50/p95/min/max for each timing component. Raw data must contain no screenshots, UI text, app package identifiers, or device serials.
+
+## 11. Benchmark protocol and acceptance criteria
+
+The repository owns an Android benchmark fixture at `test/fixtures/latency-journey-app/`. It is self-contained, has no network dependency, and exposes stable accessibility IDs for a repeatable navigation-and-control flow. The benchmark harness builds and installs the fixture.
+
+Before every run, the harness must force-stop the fixture, clear its data, install the known fixture APK, and verify the reset-state `saved-session` selector. The benchmark runs against one named Android emulator image, records its API level and fixture APK SHA-256, and performs no work in parallel. It executes this exact flow:
+
+```text
+launch app → open detail → activate control → return Home → reopen detail → change control state → stop
+```
+
+The fixture and benchmark harness MUST define and use their own stable selector map, and assert each resulting state. It performs 3 warm-up runs and 10 measured runs per mode.
+
+The baseline invokes the current public tools over a spawned `dist/server.js` stdio MCP connection: `start_app`, `wait_for_ui`, `tap_element` with a handle returned by `wait_for_ui`, and the existing `expect_*` tools. The candidate invokes the new selector `tap_element` and `run_journey`. Both variants include the same waits/assertions and use `full`/`debug` for the compatibility comparison; the candidate additionally runs `light`/`compact`. End-to-end latency is measured by the benchmark client from immediately before its JSON-RPC request write until the corresponding response is fully read, excluding server process startup. The harness separately times explicit `capture_screenshot` and `capture_debug_snapshot` calls outside the journey.
+
+A failed run is reported separately and excluded from percentile calculation; the report must state its failure count, first failed fixture selector, and failed tool/step. The harness emits the exact request sequence and fixture version in the report, with selectors but no UI text or serial number.
+
+Acceptance gates:
+
+- `full` has no regression greater than 10% at p95 for the representative journey and preserves all existing contract tests.
+- `light` compact reduces p50 end-to-end MCP journey latency by at least 30% versus full/debug on the same benchmark environment.
+- `none` compact has no post-dispatch accessibility read, proven by a focused unit test and its timing breakdown.
+- A compact state delta never includes an unchanged node; a requested screenshot is the only path that captures screenshot bytes.
+- The benchmark report is checked in for baseline and final runs. If environmental variance prevents the 30% threshold, implementation stops for diagnosis rather than relaxing verification semantics or claiming success.
+
+## 12. Test plan
+
+Add focused unit coverage for:
+
+1. Every verification mode on `tap_element` and `start_app`, including the exact adapter/observation calls and lifecycle/result semantics.
+2. Compact/debug compatibility: omitted fields produce the legacy debug envelope; compact retains actionable failure fields and omits verbose metadata.
+3. Selector tap success, wait-until-present success, ambiguity, stale/non-actionable rejection, and timeout.
+4. `run_journey` ordering, single platform/device resolution, first failed assertion, unexecuted remaining steps, and per-step timing.
+5. Delta added/removed/changed identity behavior, revision miss, truncation, and no screenshot when disabled.
+6. Action and verification timeout bounds and the absence of a fixed delay once a fresh expected state is observed.
+7. Deterministic benchmark report aggregation from fixture timing data.
+
+Existing unit, lint, build, and device checks remain required. Run the real benchmark only after the automated contract gates pass.
+
+## 13. Implementation ownership
+
+- `src/server/tool-definitions.ts`: additive schemas and documentation.
+- `src/server/tool-handlers.ts` and `src/server/common.ts`: mode parsing, response shaping, timing envelope, and journey orchestration.
+- `src/interact/index.ts`: selector resolve-and-tap, verification policy, deadline propagation, and step execution primitives.
+- `src/observe/snapshot-metadata.ts` and `src/observe/index.ts`: changed-node deltas and explicit screenshot collection.
+- `src/manage/*`: action-timeout propagation and truthful launch timing.
+- `test/unit/{server,interact,observe}/`: contract coverage; `test/device/automated/`: benchmark journey harness.
+
+## 14. Rollout
+
+1. Land timing instrumentation and baseline benchmark without changing defaults.
+2. Land verification/response modes with `full`/`debug` defaults and compatibility tests.
+3. Land selector tap and state deltas.
+4. Land `run_journey` and run before/after device benchmarks.
+5. Publish the report and recommend `light`/`compact` for routine agent navigation only after the acceptance gates pass.

package/docs/tools/interact.md

@@ -616,3 +616,33 @@ Guidance:
 - `uiChanged=true` or `expectedElementVisible=true` means the action outcome is already verified
 - local-state actions should prefer refreshed snapshots, `expect_state`, or `expect_element_visible` over default network inspection
 - network activity is auxiliary evidence, not mandatory proof
+
+## Interaction execution controls (RFC 018)
+
+`start_app`, `restart_app`, `tap`, `tap_element`, `swipe`, `scroll_to_element`,
+`type_text`, and `press_back` accept `verificationMode` (`none`, `light`, `full`)
+and `responseMode` (`compact`, `debug`). Omitted controls retain `full`/`debug`.
+`none` reports dispatch without post-action accessibility reads. `light` requests
+one fresh tree; an unavailable observation does not turn successful delivery into
+a failed action. Use an explicit assertion to establish the intended outcome.
+Compact responses retain delivery, retry, lifecycle, and timing fields.
+
+`actionTimeoutMs` and `verificationTimeoutMs` are integer budgets, clamped to
+100–30,000 and 100–60,000 ms respectively. An action timeout after dispatch starts
+has unknown delivery and must be observed before retrying.
+
+`tap_element` accepts exactly one `elementId` or `selector`. A selector supports
+`text`, `resource_id`, `accessibility_id`, and `contains`. Optional `waitFor` waits
+for the target before tapping; without it, resolution uses one observation.
+Ambiguous targets require an explicit `waitFor.match.index`.
+
+`run_journey` accepts 1–50 steps with unique IDs, on one platform/device. Supported
+step types are `start_app`, `tap`, `wait`, and `assert`. Assertions support
+`element_visible`, `element_absent`, `screen_fingerprint`, and `state_equals`.
+The complete request is validated before device work. The first failed step stops
+the journey; remaining steps report `not_run`. Journey responses default to compact.
+`defaults` supplies execution controls; step verification overrides the default.
+`captureOnFailure: true` includes a screenshot-free failure snapshot in debug mode.
+
+See [RFC 018](../specs-deltas/018-interaction-latency-and-batched-journeys.md) for
+contracts and the device acceptance criteria.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "mobile-debug-mcp",
-  "version": "0.31.0",
+  "version": "0.32.0",
   "description": "MCP server for mobile app debugging (Android + iOS), with focus on security and reliability",
   "type": "module",
   "bin": {

package/src/interact/android.ts

@@ -1,3 +1,4 @@
+import { markDispatchStarted } from '../utils/operation-budget.js'
 import { TapResponse, SwipeResponse, TypeTextResponse, PressBackResponse } from "../types.js"
 import { execAdb, getAndroidDeviceMetadata, getDeviceInfo } from "../utils/android/utils.js"
 import { AndroidObserve } from "../observe/index.js"
@@ -7,31 +8,33 @@ import { scrollToElementShared } from "../utils/ui/index.js"
 export class AndroidInteract {
   private observe = new AndroidObserve();
 
-  async tap(x: number, y: number, deviceId?: string): Promise<TapResponse> {
+  async tap(x: number, y: number, deviceId?: string, timeoutMs?: number): Promise<TapResponse> {
     const metadata = await getAndroidDeviceMetadata("", deviceId)
     const deviceInfo = getDeviceInfo(deviceId || 'default', metadata)
 
     try {
-      await execAdb(['shell', 'input', 'tap', x.toString(), y.toString()], deviceId)
+      markDispatchStarted()
+      await execAdb(['shell', 'input', 'tap', x.toString(), y.toString()], deviceId, { timeout: timeoutMs })
       return { device: deviceInfo, success: true, x, y }
     } catch (e) {
       return { device: deviceInfo, success: false, x, y, error: e instanceof Error ? e.message : String(e) }
     }
   }
 
-  async swipe(x1: number, y1: number, x2: number, y2: number, duration: number, deviceId?: string): Promise<SwipeResponse> {
+  async swipe(x1: number, y1: number, x2: number, y2: number, duration: number, deviceId?: string, timeoutMs?: number): Promise<SwipeResponse> {
     const metadata = await getAndroidDeviceMetadata("", deviceId)
     const deviceInfo = getDeviceInfo(deviceId || 'default', metadata)
 
     try {
-      await execAdb(['shell', 'input', 'swipe', x1.toString(), y1.toString(), x2.toString(), y2.toString(), duration.toString()], deviceId)
+      markDispatchStarted()
+      await execAdb(['shell', 'input', 'swipe', x1.toString(), y1.toString(), x2.toString(), y2.toString(), duration.toString()], deviceId, { timeout: timeoutMs })
       return { device: deviceInfo, success: true, start: [x1, y1], end: [x2, y2], duration }
     } catch (e) {
       return { device: deviceInfo, success: false, start: [x1, y1], end: [x2, y2], duration, error: e instanceof Error ? e.message : String(e) }
     }
   }
 
-  async typeText(text: string, deviceId?: string): Promise<TypeTextResponse> {
+  async typeText(text: string, deviceId?: string, timeoutMs?: number): Promise<TypeTextResponse> {
     const metadata = await getAndroidDeviceMetadata("", deviceId)
     const deviceInfo = getDeviceInfo(deviceId || 'default', metadata)
 
@@ -39,19 +42,21 @@ export class AndroidInteract {
       // Encode spaces as %s to ensure proper input handling by adb shell input text
       const encodedText = text.replace(/\s/g, '%s')
       // Note: 'input text' might fail with some characters or if keyboard isn't ready, but it's the standard ADB way.
-      await execAdb(['shell', 'input', 'text', encodedText], deviceId)
+      markDispatchStarted()
+      await execAdb(['shell', 'input', 'text', encodedText], deviceId, { timeout: timeoutMs })
       return { device: deviceInfo, success: true, text }
     } catch (e) {
       return { device: deviceInfo, success: false, text, error: e instanceof Error ? e.message : String(e) }
     }
   }
 
-  async pressBack(deviceId?: string): Promise<PressBackResponse> {
+  async pressBack(deviceId?: string, timeoutMs?: number): Promise<PressBackResponse> {
     const metadata = await getAndroidDeviceMetadata("", deviceId)
     const deviceInfo = getDeviceInfo(deviceId || 'default', metadata)
 
     try {
-      await execAdb(['shell', 'input', 'keyevent', '4'], deviceId)
+      markDispatchStarted()
+      await execAdb(['shell', 'input', 'keyevent', '4'], deviceId, { timeout: timeoutMs })
       return { device: deviceInfo, success: true }
     } catch (e) {
       return { device: deviceInfo, success: false, error: e instanceof Error ? e.message : String(e) }

package/src/interact/index.ts

@@ -1,3 +1,4 @@
+import { remainingBudget } from '../utils/operation-budget.js'
 import { createHash } from 'crypto'
 import { AndroidInteract } from './android.js';
 import { iOSInteract } from './ios.js';
@@ -1226,22 +1227,22 @@ export class ToolsInteract {
     return { interact: interact as any, resolved, platform: effectivePlatform }
   }
 
-  static async tapHandler({ platform, x, y, deviceId }: { platform?: 'android' | 'ios', x: number, y: number, deviceId?: string }) {
+  static async tapHandler({ platform, x, y, deviceId, timeoutMs }: { platform?: 'android' | 'ios', x: number, y: number, deviceId?: string, timeoutMs?: number }) {
     const { interact, resolved } = await ToolsInteract.getInteractionService(platform, deviceId)
-    return await interact.tap(x, y, resolved.id)
+    return await interact.tap(x, y, resolved.id, timeoutMs)
   }
 
-  static async tapElementHandler({ elementId }: { elementId: string }): Promise<TapElementResponse> {
+  static async tapElementHandler({ elementId, verificationMode = 'full', timeoutMs, platform, deviceId, freshTree }: { elementId: string, verificationMode?: 'none' | 'light' | 'full', timeoutMs?: number, platform?: 'android' | 'ios', deviceId?: string, freshTree?: any }): Promise<TapElementResponse> {
     const actionType = 'tap_element'
     const selector = { elementId }
     const resolved = ToolsInteract._resolvedUiElements.get(elementId)
-    if (!resolved) {
+    if (!resolved || (platform && resolved.platform !== platform) || (deviceId && resolved.deviceId !== deviceId)) {
       return ToolsInteract._actionFailure(actionType, selector, null, 'STALE_REFERENCE', true, null)
     }
 
-    const fingerprintBefore = await ToolsInteract._captureFingerprint(resolved.platform, resolved.deviceId)
+    const fingerprintBefore = verificationMode === 'full' ? await ToolsInteract._captureFingerprint(resolved.platform, resolved.deviceId) : null
 
-    const tree = await ToolsObserve.getUITreeHandler({ platform: resolved.platform, deviceId: resolved.deviceId }) as any
+    const tree = freshTree ?? await ToolsObserve.getUITreeHandler({ platform: resolved.platform, deviceId: resolved.deviceId }) as any
     const treePlatform = tree?.device?.platform === 'ios' ? 'ios' : resolved.platform
     const treeDeviceId = tree?.device?.id || resolved.deviceId
     const elements = Array.isArray(tree?.elements) ? tree.elements as UiElement[] : []
@@ -1272,14 +1273,14 @@ export class ToolsInteract {
 
     const x = Math.floor((bounds[0] + bounds[2]) / 2)
     const y = Math.floor((bounds[1] + bounds[3]) / 2)
-    const tapResult = await ToolsInteract.tapHandler({ platform: resolved.platform, x, y, deviceId: resolved.deviceId })
+    const tapResult = await ToolsInteract.tapHandler({ platform: resolved.platform, x, y, deviceId: resolved.deviceId, timeoutMs })
 
     if (!tapResult.success) {
-      const fingerprintAfterFailure = await ToolsInteract._captureFingerprint(resolved.platform, resolved.deviceId)
-      return ToolsInteract._actionFailure(actionType, selector, resolvedTarget, 'UNKNOWN', false, fingerprintBefore, fingerprintAfterFailure)
+      const fingerprintAfterFailure = verificationMode === 'full' ? await ToolsInteract._captureFingerprint(resolved.platform, resolved.deviceId) : null
+      return { ...ToolsInteract._actionFailure(actionType, selector, resolvedTarget, 'UNKNOWN', false, fingerprintBefore, fingerprintAfterFailure), error: tapResult.error }
     }
 
-    const fingerprintAfter = await ToolsInteract._captureFingerprint(resolved.platform, resolved.deviceId)
+    const fingerprintAfter = verificationMode === 'full' ? await ToolsInteract._captureFingerprint(resolved.platform, resolved.deviceId) : null
     return buildActionExecutionResult({
       actionType,
       device: tree?.device,
@@ -2204,19 +2205,19 @@ export class ToolsInteract {
     return buildFailure('CONTROL_CONVERGENCE_FAILED', 'control did not converge within the allotted attempts', resolvedTarget, currentDevice, lastObservedState, attemptCount, lastAdjustmentMode, true, uiFingerprintAfter)
   }
 
-  static async swipeHandler({ platform = 'android', x1, y1, x2, y2, duration, deviceId }: { platform?: 'android' | 'ios', x1: number, y1: number, x2: number, y2: number, duration: number, deviceId?: string }) {
+  static async swipeHandler({ platform = 'android', x1, y1, x2, y2, duration, deviceId, timeoutMs }: { platform?: 'android' | 'ios', x1: number, y1: number, x2: number, y2: number, duration: number, deviceId?: string, timeoutMs?: number }) {
     const { interact, resolved } = await ToolsInteract.getInteractionService(platform, deviceId)
-    return await interact.swipe(x1, y1, x2, y2, duration, resolved.id)
+    return await interact.swipe(x1, y1, x2, y2, duration, resolved.id, timeoutMs)
   }
 
-  static async typeTextHandler({ text, deviceId }: { text: string, deviceId?: string }) {
+  static async typeTextHandler({ text, deviceId, timeoutMs }: { text: string, deviceId?: string, timeoutMs?: number }) {
     const resolved = await resolveTargetDevice({ platform: 'android', deviceId })
-    return await new AndroidInteract().typeText(text, resolved.id)
+    return await new AndroidInteract().typeText(text, resolved.id, timeoutMs)
   }
 
-  static async pressBackHandler({ deviceId }: { deviceId?: string }) {
+  static async pressBackHandler({ deviceId, timeoutMs }: { deviceId?: string, timeoutMs?: number }) {
     const resolved = await resolveTargetDevice({ platform: 'android', deviceId })
-    return await new AndroidInteract().pressBack(resolved.id)
+    return await new AndroidInteract().pressBack(resolved.id, timeoutMs)
   }
 
   static async scrollToElementHandler({ platform, selector, direction = 'down', maxScrolls = 10, scrollAmount = 0.7, deviceId }: { platform: 'android' | 'ios', selector: { text?: string, resourceId?: string, contentDesc?: string, className?: string }, direction?: 'down' | 'up', maxScrolls?: number, scrollAmount?: number, deviceId?: string }) {
@@ -2490,7 +2491,7 @@ export class ToolsInteract {
     return { found: true, element: outEl, score: scoreVal, confidence: scoreVal, resolution }
   }
 
-  static async waitForUIHandler({ selector, condition = 'exists', timeout_ms = 60000, poll_interval_ms = 300, match, retry = { max_attempts: 1, backoff_ms: 0 }, platform, deviceId }: { selector?: { text?: string, resource_id?: string, accessibility_id?: string, contains?: boolean }, condition?: 'exists'|'not_exists'|'visible'|'clickable', timeout_ms?: number, poll_interval_ms?: number, match?: { index?: number }, retry?: { max_attempts?: number, backoff_ms?: number }, platform?: 'android'|'ios', deviceId?: string }) {
+  static async waitForUIHandler({ selector, condition = 'exists', timeout_ms = 60000, poll_interval_ms = 300, match, retry = { max_attempts: 1, backoff_ms: 0 }, platform, deviceId, singleObservation = false, singleAttempt = false, rejectAmbiguous = false }: { selector?: { text?: string, resource_id?: string, accessibility_id?: string, contains?: boolean }, condition?: 'exists'|'not_exists'|'visible'|'clickable', timeout_ms?: number, poll_interval_ms?: number, match?: { index?: number }, retry?: { max_attempts?: number, backoff_ms?: number }, platform?: 'android'|'ios', deviceId?: string, singleObservation?: boolean, singleAttempt?: boolean, rejectAmbiguous?: boolean }) {
     const overallStart = Date.now()
     const requestedIndex = typeof match?.index === 'number' ? match.index : null
     const requested = {
@@ -2538,7 +2539,7 @@ export class ToolsInteract {
     let lastConditionSatisfied = false
     let matchedAt: number | null = null
     let stableMatchCount = 0
-    const stableObservationCount = 2
+    const stableObservationCount = singleObservation ? 1 : 2
     const snapshotStaleThresholdMs = 500
 
     // Precompute normalized selector values and helpers (constant across polls)
@@ -2554,7 +2555,8 @@ export class ToolsInteract {
         const attemptStart = Date.now()
         const deadline = attemptStart + (timeout_ms || 0)
 
-        while (Date.now() <= deadline) {
+        while (Date.now() <= deadline || (singleAttempt && totalPollCount === 0)) {
+          remainingBudget()
           totalPollCount++
           try {
             const tree = await ToolsObserve.getUITreeHandler({ platform, deviceId }) as any
@@ -2601,6 +2603,7 @@ export class ToolsInteract {
 
             // Evaluate condition
             const matchedCount = matches.length
+            if (rejectAmbiguous && matchedCount > 1 && match?.index === undefined) return { status: 'timeout', error: { code: 'AMBIGUOUS_TARGET' } }
             const pickIndex = (typeof match?.index === 'number') ? match!.index as number : undefined
             let chosen: { el: any, idx: number } | null = null
             if (matches.length > 0) {
@@ -2659,6 +2662,7 @@ export class ToolsInteract {
 
                 return {
                   status: 'success',
+                  ...(singleObservation ? { _tree: tree } : {}),
                   matched: matchedCount,
                   element: outEl,
                   metrics: { latency_ms, poll_count: totalPollCount, attempts },
@@ -2677,10 +2681,12 @@ export class ToolsInteract {
             }
 
           } catch (e) {
+            if (e instanceof Error && e.message === 'ACTION_TIMEOUT') throw e
             // Non-fatal per-poll error; record and continue
             console.warn('waitForUI: poll error (non-fatal):', e instanceof Error ? e.message : String(e))
           }
 
+          if (singleAttempt) break
           // Sleep until next poll
           await new Promise(r => setTimeout(r, effectivePoll || 50))
         }
@@ -2712,6 +2718,7 @@ export class ToolsInteract {
       }
 
     } catch (err) {
+      if (err instanceof Error && err.message === 'ACTION_TIMEOUT') throw err
       const elapsed = Date.now() - overallStart
       return {
         status: 'timeout',
@@ -3139,7 +3146,8 @@ export class ToolsInteract {
     stabilization_window_ms = 1000,
     stable_observation_count = 2,
     snapshot_stale_threshold_ms = 500,
-    poll_interval_ms = 150
+    poll_interval_ms = 150,
+    exact_budget = false
   }: {
     selector?: { text?: string, resource_id?: string, accessibility_id?: string, contains?: boolean },
     element_id?: string,
@@ -3150,16 +3158,17 @@ export class ToolsInteract {
     stabilization_window_ms?: number,
     stable_observation_count?: number,
     snapshot_stale_threshold_ms?: number,
-    poll_interval_ms?: number
+    poll_interval_ms?: number,
+    exact_budget?: boolean
   }): Promise<ExpectStateResponse> {
     const compareBoolean = (value: unknown) => typeof value === 'boolean' ? value : null
     const compareString = (value: unknown) => typeof value === 'string' ? value : null
     const compareNumber = (value: unknown) => typeof value === 'number' && Number.isFinite(value) ? value : null
     const sleep = (ms: number) => new Promise(resolve => setTimeout(resolve, ms))
     const start = Date.now()
-    const deadline = start + Math.max(500, stabilization_window_ms)
+    const deadline = start + (exact_budget ? stabilization_window_ms : Math.max(500, stabilization_window_ms))
     const stableTarget = Math.max(1, Math.floor(stable_observation_count || 2))
-    const pollDelay = Math.max(100, Math.min(poll_interval_ms || 150, 200))
+    const pollDelay = exact_budget ? Math.max(50, Math.min(poll_interval_ms || 150, 1000)) : Math.max(100, Math.min(poll_interval_ms || 150, 200))
     const staleThreshold = Math.max(300, Math.min(snapshot_stale_threshold_ms || 500, 800))
 
     let attempts = 0