@mobileaidev/ai-app-bridge 0.2.15 → 0.3.0-rc.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +201 -0
- package/NOTICE +7 -0
- package/README.md +306 -53
- package/bin/ai-app-bridge.js +51 -4504
- package/bin/android-permissions.js +152 -0
- package/bin/android-uia-xml.js +74 -0
- package/bin/artifact-paths.js +127 -1
- package/bin/bridge-forward.js +56 -0
- package/bin/command-discovery.js +86 -0
- package/bin/command-errors.js +60 -0
- package/bin/command-registry.js +504 -0
- package/bin/command-request.js +14 -0
- package/bin/command-router.js +80 -0
- package/bin/connection-cache.js +119 -0
- package/bin/device-provider.js +2934 -0
- package/bin/execution-host.js +596 -0
- package/bin/execution-runtime.js +101 -0
- package/bin/fact-codec.js +321 -0
- package/bin/fact-recorder.js +691 -0
- package/bin/fact-store.js +226 -0
- package/bin/feedback-probe.js +285 -0
- package/bin/intent/install-intent.js +267 -0
- package/bin/intent/intent-action-executor.js +129 -0
- package/bin/intent/intent-autonomous-adapter.js +46 -0
- package/bin/intent/intent-capture-port.js +64 -0
- package/bin/intent/intent-entry.js +226 -0
- package/bin/intent/intent-errors.js +28 -0
- package/bin/intent/intent-evidence-store.js +121 -0
- package/bin/intent/intent-lifetime.js +62 -0
- package/bin/intent/intent-observation-target.js +33 -0
- package/bin/intent/intent-observer.js +194 -0
- package/bin/intent/intent-production-adapter.js +334 -0
- package/bin/intent/intent-provider.js +27 -0
- package/bin/intent/intent-runtime.js +63 -0
- package/bin/intent/intent-worker.js +432 -0
- package/bin/intent/ios-intent-adapter.js +90 -0
- package/bin/intent/permission-intent.js +239 -0
- package/bin/intent/web-intent-adapter.js +31 -0
- package/bin/ios-device-outcome.js +47 -0
- package/bin/ios-execution.js +108 -0
- package/bin/ios-provider.js +497 -583
- package/bin/ios-runtime-binding.js +54 -0
- package/bin/ios-wda-execution.js +70 -0
- package/bin/ios-wda-port.js +98 -0
- package/bin/ios-wda-project.js +79 -0
- package/bin/mcp-server.js +83 -1040
- package/bin/mmap-scan-index.js +329 -0
- package/bin/observation-collector.js +862 -0
- package/bin/runtime-client.js +154 -0
- package/bin/runtime-directory.js +107 -0
- package/bin/runtime-protocol.js +36 -0
- package/bin/script/bounded-script-registry.js +103 -0
- package/bin/script/node-runtime-adapter.js +233 -0
- package/bin/script/progress-projector.js +63 -0
- package/bin/script/python-runtime-adapter.js +111 -0
- package/bin/script/rolling-summary.js +134 -0
- package/bin/script/script-agent-port.js +40 -0
- package/bin/script/script-assert.js +95 -0
- package/bin/script/script-capture-port.js +76 -0
- package/bin/script/script-catalog.js +85 -0
- package/bin/script/script-durable-restore.js +195 -0
- package/bin/script/script-entry-code.js +26 -0
- package/bin/script/script-entry-route.js +38 -0
- package/bin/script/script-entry.js +3 -0
- package/bin/script/script-errors.js +29 -0
- package/bin/script/script-evidence-store.js +22 -0
- package/bin/script/script-format-removed.js +26 -0
- package/bin/script/script-host-port.js +397 -0
- package/bin/script/script-ledger.js +64 -0
- package/bin/script/script-result.js +57 -0
- package/bin/script/script-sdk.js +152 -0
- package/bin/script/script-sdk.py +153 -0
- package/bin/script/script-session-channel.js +127 -0
- package/bin/script/script-spec.js +86 -0
- package/bin/script/script-supervisor.js +919 -0
- package/bin/script/templates/checkpoint-reentry.js +13 -0
- package/bin/segment-index.js +481 -0
- package/bin/segmented-fact-store.js +1571 -0
- package/bin/shared-kernel/android-h5-target.js +10 -0
- package/bin/shared-kernel/android-install-execution.js +176 -0
- package/bin/shared-kernel/android-sdk-endpoint.js +42 -0
- package/bin/shared-kernel/android-shell-execution.js +195 -0
- package/bin/shared-kernel/argument-schema.js +117 -0
- package/bin/shared-kernel/canonical-path.js +17 -0
- package/bin/shared-kernel/device-acknowledgements.js +53 -0
- package/bin/shared-kernel/device-completion-history.js +52 -0
- package/bin/shared-kernel/device-mutation-lease.js +219 -0
- package/bin/shared-kernel/device-ownership-recovery.js +95 -0
- package/bin/shared-kernel/device-ownership-store.js +94 -0
- package/bin/shared-kernel/evidence-adapters.js +251 -0
- package/bin/shared-kernel/evidence-archive.js +329 -0
- package/bin/shared-kernel/evidence-recording.js +131 -0
- package/bin/shared-kernel/evidence-schema.js +194 -0
- package/bin/shared-kernel/evidence-store.js +190 -0
- package/bin/shared-kernel/execution-admission.js +22 -0
- package/bin/shared-kernel/execution-contracts.js +171 -0
- package/bin/shared-kernel/execution-io.js +106 -0
- package/bin/shared-kernel/execution-ledger.js +125 -0
- package/bin/shared-kernel/execution-scope.js +87 -0
- package/bin/shared-kernel/execution-target.js +115 -0
- package/bin/shared-kernel/flutter-execution.js +13 -0
- package/bin/shared-kernel/flutter-h5-port.js +60 -0
- package/bin/shared-kernel/flutter-h5-target.js +9 -0
- package/bin/shared-kernel/flutter-target.js +75 -0
- package/bin/shared-kernel/h5-execution.js +11 -0
- package/bin/shared-kernel/h5-target.js +31 -0
- package/bin/shared-kernel/host-fact-store.js +49 -0
- package/bin/shared-kernel/ios-h5-target.js +9 -0
- package/bin/shared-kernel/ios-native-target.js +71 -0
- package/bin/shared-kernel/live-capture-query.js +115 -0
- package/bin/shared-kernel/managed-sdk-execution.js +78 -0
- package/bin/shared-kernel/native-execution.js +13 -0
- package/bin/shared-kernel/native-target.js +156 -0
- package/bin/shared-kernel/provider-command-contracts.js +55 -0
- package/bin/shared-kernel/recorded-payload-archive.js +195 -0
- package/bin/shared-kernel/request-context.js +35 -0
- package/bin/shared-kernel/semantic-node.js +55 -0
- package/bin/shared-kernel/summary-transformer.js +352 -0
- package/bin/shared-kernel/target-lease-protocol.js +47 -0
- package/bin/shared-kernel/text-wait.js +111 -0
- package/bin/shared-kernel/uia-execution.js +96 -0
- package/bin/shared-kernel/uia-protocol.js +214 -0
- package/bin/shared-kernel/uia-runtime-port.js +377 -0
- package/bin/shared-kernel/uia-target.js +39 -0
- package/bin/shared-kernel/web-dom-target.js +44 -0
- package/bin/shared-kernel/xml-attributes.js +25 -0
- package/bin/target-execution.js +275 -0
- package/bin/web/command-schema.js +60 -0
- package/bin/web/session-store.js +157 -0
- package/bin/web-provider.js +334 -553
- package/docs/COMMAND_CONTRACT.md +1563 -0
- package/docs/EVIDENCE_ARCHIVE.md +214 -0
- package/docs/INTENT_FOREGROUND.md +71 -0
- package/docs/INTENT_NATIVE_EDITING.md +79 -0
- package/docs/RELEASE.md +59 -0
- package/docs/SCRIPT_AUTHORING.md +489 -0
- package/node_modules/@mobileaidev/segmented-fact-store-native/LICENSE +201 -0
- package/node_modules/@mobileaidev/segmented-fact-store-native/NOTICE +7 -0
- package/node_modules/@mobileaidev/segmented-fact-store-native/binding.gyp +36 -0
- package/node_modules/@mobileaidev/segmented-fact-store-native/bindings/node/sfs_node.c +597 -0
- package/node_modules/@mobileaidev/segmented-fact-store-native/include/sfs.h +178 -0
- package/node_modules/@mobileaidev/segmented-fact-store-native/index.js +5 -0
- package/node_modules/@mobileaidev/segmented-fact-store-native/package.json +24 -0
- package/node_modules/@mobileaidev/segmented-fact-store-native/src/sfs.c +2349 -0
- package/package.json +60 -5
- package/runtime/ios-wda/AABWDABinding.h +19 -0
- package/runtime/ios-wda/AABWDABinding.m +97 -0
- package/runtime/ios-wda/AABWDAExecution.h +26 -0
- package/runtime/ios-wda/AABWDAExecution.m +172 -0
- package/runtime/ios-wda/AABWDAIntegration.h +71 -0
- package/runtime/ios-wda/AABWDAManagedRoutes.h +392 -0
- package/runtime/ios-wda/AABWDAReceiptStore.h +10 -0
- package/runtime/ios-wda/AABWDAReceiptStore.m +116 -0
- package/runtime/uia/ai-app-bridge-uia.jar +0 -0
- package/runtime/uia/manifest.json +22 -0
- package/skills/ai-app-bridge-use/SKILL.md +23 -360
|
@@ -0,0 +1,1563 @@
|
|
|
1
|
+
# Command contract: aab.command/v1
|
|
2
|
+
|
|
3
|
+
Intent and Script are first-class execution interfaces. Intent observes a target,
|
|
4
|
+
accepts a decision tied to that observation, dispatches and records its receipt.
|
|
5
|
+
Script runs repeatable JavaScript/Python with progress, assertions and control.
|
|
6
|
+
Both use the device capabilities below and retained evidence. A single command
|
|
7
|
+
remains useful for observation, interaction, fixture setup and diagnosis.
|
|
8
|
+
|
|
9
|
+
## Discovery and entrypoints
|
|
10
|
+
|
|
11
|
+
MCP exposes exactly `capabilities` and `run`. Use `capabilities {"command":"tap-text"}`
|
|
12
|
+
for its current `inputSchema`, platform, role and supported entrypoints. Domain
|
|
13
|
+
`execution` contains Intent, Script, runtime lifecycle and device ownership;
|
|
14
|
+
`evidence` contains archive operations.
|
|
15
|
+
`capabilities {"includeOptions":true}` returns every current command schema.
|
|
16
|
+
|
|
17
|
+
All parameters are under `run.arguments`:
|
|
18
|
+
|
|
19
|
+
```json
|
|
20
|
+
{"command":"tap-text","arguments":{"serial":"DEVICE","packageName":"com.example.app","targetText":"设置","provider":"auto"}}
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
Names are canonical and case-sensitive. Unknown arguments, aliases, invalid
|
|
24
|
+
JSON types, null numeric values and out-of-range coordinates fail before storage
|
|
25
|
+
or device access. Validation includes nested control objects and the selected
|
|
26
|
+
operation's fields. CLI flags convert textual numbers/booleans and parse typed
|
|
27
|
+
object flags as JSON; MCP accepts the actual JSON type. CLI flags use canonical
|
|
28
|
+
`--kebab-case` names followed by separate value tokens. Extra positional tokens
|
|
29
|
+
fail with `unexpected_argument`; repeated single-value flags fail with
|
|
30
|
+
`duplicate_argument` before device access, so a later value cannot replace the
|
|
31
|
+
original target. Only `--category` and `--extra` accept repeated values. Parse
|
|
32
|
+
failures return the same JSON error envelope and exit code 1 as validation errors.
|
|
33
|
+
The CLI's `--help` and `--help COMMAND` come from the same registry. All
|
|
34
|
+
registered commands are available through CLI and MCP, including Intent, Script,
|
|
35
|
+
installation, permission dialogs, evidence and Web sessions. Both adapters call
|
|
36
|
+
`runtime-client.js`; one independent `execution-runtime.js` owns the protocol-neutral
|
|
37
|
+
`execution-host.js`. MCP does not spawn the CLI for each request, and neither
|
|
38
|
+
adapter has a separate execution state machine.
|
|
39
|
+
Within the shared Host, `target-execution.js` still owns ordinary-command
|
|
40
|
+
queueing, deadlines and request-ID idempotency. Intent and Script use the same
|
|
41
|
+
provider access and device ownership through their execution-specific contracts.
|
|
42
|
+
|
|
43
|
+
### Shared runtime lifecycle
|
|
44
|
+
|
|
45
|
+
The first executing command starts a local runtime. `runtime --operation start`
|
|
46
|
+
starts it explicitly; `runtime --operation status` inspects it without starting
|
|
47
|
+
one; `runtime --operation stop` cancels and drains active work before releasing
|
|
48
|
+
its ownership. CLI exit, MCP EOF and client SIGINT/SIGTERM close only that client.
|
|
49
|
+
Use `intent`/`script --operation cancel --operation-id ID` to cancel one task.
|
|
50
|
+
The same operation ID can be queried and controlled from either entrypoint.
|
|
51
|
+
|
|
52
|
+
A runtime namespace is keyed by the canonical FactStore directory. Configure it
|
|
53
|
+
with `AI_APP_BRIDGE_FACT_STORE_DIR`; symlinks to the same directory identify the
|
|
54
|
+
same store. Control files default to `~/.ai-app-bridge/runtimes/v1`; the explicit
|
|
55
|
+
`AI_APP_BRIDGE_RUNTIME_HOME` override must match in clients that share a runtime.
|
|
56
|
+
A private endpoint record and token authenticate loopback requests. An OS-held
|
|
57
|
+
SQLite lock controls ownership; a stale PID or failed health probe never permits
|
|
58
|
+
replacing a live owner. Startup candidates can repeat the ownership election
|
|
59
|
+
only after proving the lock is free and before dispatching any command.
|
|
60
|
+
|
|
61
|
+
Code, bundled runtime artifacts, native-store binary, Node version, FactStore
|
|
62
|
+
profile and provider configuration must match the owner. Mismatch rejects
|
|
63
|
+
execution with `runtime_code_mismatch` or `runtime_configuration_mismatch`.
|
|
64
|
+
`runtime status` reports the owner and compatibility; explicit `runtime stop`
|
|
65
|
+
remains available from a different build/configuration. An unresponsive owner
|
|
66
|
+
returns `runtime_unresponsive` without killing or replacing it. Lost execution
|
|
67
|
+
connections report an unknown outcome after request submission; execution is
|
|
68
|
+
never automatically retried. Runtime crashes retain the existing durable
|
|
69
|
+
`runtime_lost` and original-device-receipt recovery rules.
|
|
70
|
+
|
|
71
|
+
The runtime uses the environment, including PATH, that started it. PATH is not
|
|
72
|
+
silently replaced by later clients. Use explicit command/target executable paths
|
|
73
|
+
or stop and start the runtime to change its environment. Each request separately
|
|
74
|
+
carries the caller's working directory. Contract-defined local paths resolve
|
|
75
|
+
there; Script freezes its `cwd`, source and target before starting. Subsequent
|
|
76
|
+
clients cannot change those paths. `evidence verify` is offline in both adapters
|
|
77
|
+
and requires neither a running runtime nor an available FactStore.
|
|
78
|
+
|
|
79
|
+
CLI results are one JSON envelope: `{kind: "json"|"text"|"bytes", value, history?}`.
|
|
80
|
+
JSON false, zero and null remain values; text is a string and bytes are base64.
|
|
81
|
+
`history` contains the same recording outcome that MCP exposes as `_history`.
|
|
82
|
+
Command failures have `value.ok:false` and exit code 1. MCP uses its normal
|
|
83
|
+
content and `isError` representation. These are wire-format differences;
|
|
84
|
+
validation, dispatch, task control and evidence semantics are shared.
|
|
85
|
+
|
|
86
|
+
`batch`, `smoke`, `launch-native-test` and `launch-flutter` were removed. Use a
|
|
87
|
+
code Script for sequences, `launch-app` for a Flutter app's actual launcher, and
|
|
88
|
+
`launch-activity` with a supplied component/extra for app-specific test routes.
|
|
89
|
+
Full MCP surface, underscore tool aliases and duplicate outer parameters were
|
|
90
|
+
removed. There is no automatic migration or hidden command substitution.
|
|
91
|
+
|
|
92
|
+
## Android SDK transport
|
|
93
|
+
|
|
94
|
+
App commands require `packageName`. `port` selects only an optional Host TCP port;
|
|
95
|
+
it cannot identify an App or skip endpoint discovery. Device capabilities such as
|
|
96
|
+
screenshot and keyevent still work without an SDK package.
|
|
97
|
+
|
|
98
|
+
The SDK listens on `localabstract:aab-sdk-<runtimeEpoch>` and atomically publishes
|
|
99
|
+
`files/ai_app_bridge_endpoint.json` in its App-private directory. The
|
|
100
|
+
`ai-app-bridge.android-endpoint.v1` record contains `ok`, `packageName`,
|
|
101
|
+
`runtimeEpoch`, `transport:"localabstract"`, `socketName`, `version`, `updatedAtMs`
|
|
102
|
+
and `error` (null when ready). Each Host request reads that file through `run-as`
|
|
103
|
+
on the resolved serial, validates its package and runtime socket, then creates or
|
|
104
|
+
reuses the exact ADB mapping and checks it before HTTP dispatch. Reads and actions
|
|
105
|
+
are each attempted once. No cached device port, port scan, TCP listener or old
|
|
106
|
+
port-file fallback remains in the Android SDK path.
|
|
107
|
+
|
|
108
|
+
An older SDK without this endpoint must be rebuilt. Discovery failure is
|
|
109
|
+
`bridge_endpoint_discovery_failed`; malformed identity is `bridge_endpoint_invalid`
|
|
110
|
+
or `bridge_package_mismatch`; an unready endpoint is `bridge_not_ready`.
|
|
111
|
+
`remove-forward` requires `serial` and the Host `port` and verifies the mapping's
|
|
112
|
+
serial before removal.
|
|
113
|
+
|
|
114
|
+
## iOS SDK transport
|
|
115
|
+
|
|
116
|
+
SDK commands (`ios-status/tree/logs/network/state/events/h5-dom/h5-eval/h5-click/h5-input/h5-scroll` and
|
|
117
|
+
`ios-flutter-tree/nodes/action`, plus typed Flutter controls) require explicit `deviceId` and `bundleId`.
|
|
118
|
+
|
|
119
|
+
iOS Intent supports explicit `provider:"native"`, `provider:"h5"` and `provider:"flutter"` with an iOS target.
|
|
120
|
+
Native observation uses the bound WDA tree; tap and inputText require the native
|
|
121
|
+
selector below and an explicit WDA session in the target. Android native selectors,
|
|
122
|
+
keyevent, back and native gestures are not accepted on the iOS branch.
|
|
123
|
+
Native `setOrientation` requires `orientation` and no element selector; it binds
|
|
124
|
+
the observed Runner epoch, App process and WDA session before rotating.
|
|
125
|
+
An App transition currently requires an explicit session close/create and a new
|
|
126
|
+
Intent target; a system dialog disappearing does not silently retarget the old
|
|
127
|
+
Intent. The original action receipt remains recorded if its next observation fails.
|
|
128
|
+
|
|
129
|
+
For Flutter,
|
|
130
|
+
Observation, tap, text replacement, scrollBy, back and hideKeyboard use the production SDK
|
|
131
|
+
adapter and the existing managed execution port. Before a selected action,
|
|
132
|
+
the adapter reobserves and verifies the original Element identity; the SDK
|
|
133
|
+
validates the Element reference again at execution. H5 observation and typed actions
|
|
134
|
+
use the document/element contract below; the providers do not switch implicitly.
|
|
135
|
+
|
|
136
|
+
WKWebView DOM observations include `viewport` (`width`, `height`, `scrollX`,
|
|
137
|
+
`scrollY`), each control's standard text-editing eligibility in `editable`, and
|
|
138
|
+
DOM `interaction.status`: `ready`, `outside-viewport`, `obscured`, `hidden` or
|
|
139
|
+
`disabled`. Intent summaries preserve these facts, including empty editors.
|
|
140
|
+
`visible` describes CSS rendering; it does not mean a control is inside the
|
|
141
|
+
viewport or passes UIKit hit testing. Native hit testing still runs before an
|
|
142
|
+
input or click. A scroll receipt includes the resulting DOM interaction state;
|
|
143
|
+
requesting a scroll does not guarantee the control moved into view. `text` is
|
|
144
|
+
rendered DOM text and can contain an editor's own placeholder decorations.
|
|
145
|
+
Text input accepts text-like input types, textarea and editable content; it
|
|
146
|
+
does not treat date, color, checkbox or file controls as text fields.
|
|
147
|
+
|
|
148
|
+
JS/Python Script can call the catalog's iOS observations, capture queries,
|
|
149
|
+
launch, WDA session/actions and typed Flutter/H5 controls. `ios-tap-flutter`,
|
|
150
|
+
`ios-input-flutter-text` and `ios-scroll-flutter` require an exact `selector`
|
|
151
|
+
(`text` or `nodeId`); input also requires `text`, and scroll requires `delta`
|
|
152
|
+
in logical pixels. `ios-flutter-back` requests one navigation back action.
|
|
153
|
+
`ios-flutter-hide-keyboard` requests explicit focus/keyboard dismissal; observe
|
|
154
|
+
`viewport.viewInsets.bottom === 0` before acting on newly exposed controls.
|
|
155
|
+
These share the ordinary iOS provider's physical-UDID admission and original
|
|
156
|
+
completion receipts. A Script's public `requestId`, or Intent's internal
|
|
157
|
+
`runtimeActionId`, is preserved as the SDK/WDA action ID. Catalog availability
|
|
158
|
+
states implementation support, not completed complex-App or platform acceptance.
|
|
159
|
+
`deviceId` is the selected CoreDevice identifier or UDID. The device must expose
|
|
160
|
+
a connected developer tunnel with developer mode and DDI services ready.
|
|
161
|
+
|
|
162
|
+
Each command copies `Documents/ai_app_bridge_port.json` through devicectl from
|
|
163
|
+
that device's App data container. The atomically published descriptor declares
|
|
164
|
+
`schemaVersion:"aab.ios-runtime/v1"`, `bundleId`, `runtimeEpoch`, `processId`, `port`
|
|
165
|
+
and `ok:true`. The listener asks the OS for an available port and publishes the
|
|
166
|
+
actual bound port only when ready. It does not scan a fixed port range. A
|
|
167
|
+
starting, waiting or failed descriptor has `ok:false`, `port:0` and its reason;
|
|
168
|
+
it cannot authorize HTTP access. Ready descriptors and responses require a
|
|
169
|
+
port from 1 through 65535. An old, absent or invalid descriptor fails before HTTP access.
|
|
170
|
+
There is no port scan or hostname fallback. An optional `runtimeUrl` or
|
|
171
|
+
`iosHost`/`iosPort` selects the transport endpoint; it never replaces container
|
|
172
|
+
verification. A forwarded Host port may differ from the SDK's bound native port.
|
|
173
|
+
|
|
174
|
+
Host requests carry `X-AAB-Runtime-Schema`, `X-AAB-Bundle-Id`,
|
|
175
|
+
`X-AAB-Runtime-Epoch`, `X-AAB-Process-Id` and `X-AAB-Runtime-Port`. Every response
|
|
176
|
+
carries the matching `runtimeBinding`. Control requests first check bound status
|
|
177
|
+
so an old SDK cannot ignore headers and execute a write; the SDK rechecks all
|
|
178
|
+
five fields before dispatch. Missing, duplicate or mismatched identity headers
|
|
179
|
+
are rejected. These are process-routing credentials, not authentication.
|
|
180
|
+
Flutter snapshot ingress retains its existing contract.
|
|
181
|
+
|
|
182
|
+
Capture writes made immediately after SDK `start()` wait for persistent-store
|
|
183
|
+
attachment. The startup queue holds at most 256 records and 1 MiB of serialized
|
|
184
|
+
payload plus identity strings; it never supplies query results. `ios-status`
|
|
185
|
+
exposes `capture.pendingRecords` and `capture.pendingBytes`. SDK, HTTP capture
|
|
186
|
+
POSTs and Flutter capture ingress use the same append path. HTTP/Flutter replies
|
|
187
|
+
wait until the backend returns an actual receipt with its original `mobileFactId`;
|
|
188
|
+
`accepted:true, committed:false` still means queued, not durable proof. Queries
|
|
189
|
+
flush and read the original store before returning committed coverage.
|
|
190
|
+
|
|
191
|
+
Opening failure, disabled persistence, startup overflow and stop reject pending
|
|
192
|
+
writes explicitly (`capture_store_open_failed`, `capture_store_disabled`,
|
|
193
|
+
`capture_startup_queue_full`, `capture_store_unavailable`). A cancelled attachment
|
|
194
|
+
cannot replay its pending captures after restart. Dropped captures retain a gap;
|
|
195
|
+
successful startup does not erase earlier losses. A bounded decision window or
|
|
196
|
+
an exact original reference must carry its own coverage.
|
|
197
|
+
|
|
198
|
+
iOS capture queries scan the requested storage partition: logs use app-log,
|
|
199
|
+
network uses network, and state/events share state-event. Sequence-only seeks
|
|
200
|
+
exclude earlier records in that partition; returned page cursors keep the
|
|
201
|
+
existing global sequence and fixed upper watermark. Global and partition seeks
|
|
202
|
+
retain only bounded read positions, and still verify the original frame bytes.
|
|
203
|
+
Exact-reference lookup within state-event remains a paged scan; the current
|
|
204
|
+
implementation does not provide an indexed constant-time lookup for that case.
|
|
205
|
+
|
|
206
|
+
The default SDK command deadline is 30000 ms across device discovery, container
|
|
207
|
+
copy, preflight and response; `timeoutMs` replaces that total budget. Individual
|
|
208
|
+
HTTP requests default to 5000 ms within it. Descriptor reads are bounded to
|
|
209
|
+
4096 bytes and HTTP responses to 8 MiB. HTTPS verifies certificates. Invalid or
|
|
210
|
+
non-object JSON and responses without a boolean `ok` fail explicitly. Cancellation
|
|
211
|
+
waits for local child/socket close. A lost control response remains
|
|
212
|
+
`dispatched:null, ambiguous:true, settled:false` until the original SDK completion
|
|
213
|
+
is recovered; local cancellation does not prove that the App stopped executing.
|
|
214
|
+
|
|
215
|
+
### iOS WKWebView targets and DOM controls
|
|
216
|
+
|
|
217
|
+
`ios-h5-dom` requires one visible WebView, or an explicit `webViewId` previously
|
|
218
|
+
returned in `webViews` when selection is ambiguous. It returns `pageRef` with
|
|
219
|
+
`aab.ios-h5-target/v1`, SDK epoch, bundle/PID, WebView ID, document ID and URL.
|
|
220
|
+
`dom.controls` exposes stable element IDs, visible text, ariaLabel, values and bounds.
|
|
221
|
+
Password values are redacted. The first 1000 controls and first 20000 body characters
|
|
222
|
+
are retained with explicit truncation flags. Text selection rejects a truncated
|
|
223
|
+
snapshot; an explicit observed element ID remains usable within the renderer's
|
|
224
|
+
5000-control scan bound.
|
|
225
|
+
|
|
226
|
+
`ios-h5-click`, `ios-h5-input` and `ios-h5-scroll` require `selector` with exactly
|
|
227
|
+
one of `elementId`, exact `text`, or exact `ariaLabel`; optional `tag` narrows it.
|
|
228
|
+
Input requires `text` (including empty text to clear). Scroll brings that element
|
|
229
|
+
into view. Click/input require the element to be in the viewport and unobscured.
|
|
230
|
+
These are DOM operations, not trusted physical touch or keyboard events. Native
|
|
231
|
+
hit testing checks the DOM action point before click/input, then the renderer
|
|
232
|
+
checks the original identity and unchanged geometry, including DOM scroll offsets,
|
|
233
|
+
in its final action turn. The native point is mapped from document coordinates
|
|
234
|
+
using UIScrollView zoom and coordinate conversion; CSS innerHeight need not equal
|
|
235
|
+
the native area remaining between toolbars. `nativeHit` records the mapped point,
|
|
236
|
+
scroll offset, scale and actual hit view. Active scrolling/zooming requires a new
|
|
237
|
+
observation. Non-default WKWebView pageZoom returns `ios_h5_page_zoom_unsupported`;
|
|
238
|
+
invalid geometry, a point outside the native viewport and native occlusion have
|
|
239
|
+
distinct errors.
|
|
240
|
+
|
|
241
|
+
Script commands select from a fresh snapshot. Optional `expectedTarget` binds the
|
|
242
|
+
original `{pageRef,element}` as well. H5 Intent always carries that original target
|
|
243
|
+
from its committed observation, and offers `tap`, `inputText`, and `scroll`.
|
|
244
|
+
A replaced node, document navigation, history route change or BFCache restore
|
|
245
|
+
requires a new observation. Multiple simultaneously visible WebViews require explicit selection; an
|
|
246
|
+
App must be active and its selected view attached and visible. Nested frames,
|
|
247
|
+
shadow DOM, zoom/transformed viewport coverage and simultaneous multi-WebView
|
|
248
|
+
selection are not yet accepted as production coverage. Kiwix real-device tab
|
|
249
|
+
creation/closure has verified cached WebView identity, same-URL editor isolation,
|
|
250
|
+
and rejection of hidden, closed and stale targets; see the
|
|
251
|
+
[business evidence](../../../docs/IOS_H5_EDITOR_BUSINESS_2026-09-11.md#真实双标签隔离).
|
|
252
|
+
Intent selects a WebView through `start.observationTarget` or
|
|
253
|
+
`observe.observationTarget`, independently of its frozen device/App target.
|
|
254
|
+
Use `{"webViewId":"observed ID"}` for an explicit selection, or `null` to require
|
|
255
|
+
one visible WebView. `start.target.webViewId` is no longer accepted for Intent;
|
|
256
|
+
Script target defaults retain their separate contract. See Intent operations below
|
|
257
|
+
for selection, failure evidence and revision semantics.
|
|
258
|
+
|
|
259
|
+
Expert `ios-h5-eval` now requires `expectedPage` from `ios-h5-dom`; unbound scripts
|
|
260
|
+
are rejected. All iOS H5 actions use `/v1/h5/action` with a typed payload and the
|
|
261
|
+
existing managed action ID/deadline/receipt. Old SDKs return
|
|
262
|
+
`ios_h5_target_schema_required` before dispatch. Script uses the typed commands;
|
|
263
|
+
expert eval is not exposed as a Script capability.
|
|
264
|
+
|
|
265
|
+
### iOS SDK execution and recovery
|
|
266
|
+
|
|
267
|
+
`ios-h5-eval`, typed iOS H5 controls and `ios-flutter-action` require a matching SDK that advertises
|
|
268
|
+
managed execution. Host supplies the action ID, original runtime epoch and remaining
|
|
269
|
+
deadline. H5 uses the SDK process epoch; Flutter uses its Dart engine epoch.
|
|
270
|
+
The two kinds share one SDK admission slot. Queued work must obtain permission
|
|
271
|
+
immediately before a mutation. A queued cancellation revokes that permission;
|
|
272
|
+
after permission is issued, only the original execution callback can settle it.
|
|
273
|
+
The iOS Flutter MethodChannel is asynchronous and rejects unmanaged `runAction`.
|
|
274
|
+
|
|
275
|
+
Before reporting a settled result, the SDK synchronously persists an
|
|
276
|
+
`aab.ios-completion/v1` record in its internal action partition. A write failure
|
|
277
|
+
keeps admission closed. Retrying cancellation can persist that same completed
|
|
278
|
+
result; it cannot execute the action again. Queries read the segmented disk store,
|
|
279
|
+
including after an App restart. Opaque cursors retain a fixed committed upper
|
|
280
|
+
bound. Missing, evicted or invalid records cannot release unknown ownership.
|
|
281
|
+
|
|
282
|
+
All public iOS mutation commands share Host ownership keyed by the physical
|
|
283
|
+
UDID (`ios:<UDID>`), including different Apps and CoreDevice/UDID aliases.
|
|
284
|
+
The pending action is durable before dispatch. Host termination or response loss
|
|
285
|
+
therefore blocks subsequent mutations until the original completion is proven.
|
|
286
|
+
This coordination covers one OS user sharing the same ownership directory.
|
|
287
|
+
|
|
288
|
+
Use `ios-execution` through CLI or MCP:
|
|
289
|
+
|
|
290
|
+
- `status`: requires `deviceId` and `bundleId`; returns Host ownership and SDK status.
|
|
291
|
+
- `result`: additionally requires `kind:"h5"|"flutter"`, `actionId` and
|
|
292
|
+
`runtimeEpoch`; queries that original completion through bounded disk pages.
|
|
293
|
+
- `cancel`: has the same identity fields as `result`; requests cancellation and
|
|
294
|
+
reports the original completion or an unresolved result.
|
|
295
|
+
- `reconcile`: requires `deviceId` and the original `bundleId`; reads the saved
|
|
296
|
+
pending identity and releases Host ownership only with its durable SDK receipt.
|
|
297
|
+
|
|
298
|
+
An explicit `runtimeUrl` or `iosHost`/`iosPort` can select the new connection used
|
|
299
|
+
for recovery. Container verification still binds the same physical device and
|
|
300
|
+
App. A new serving process does not replace the original action epoch. `status`
|
|
301
|
+
showing an idle runtime and `cancel` receiving an acknowledgement are insufficient;
|
|
302
|
+
run `reconcile` to resolve a retained Host reservation.
|
|
303
|
+
|
|
304
|
+
These receipts prove the execution callback ended, not business success or all
|
|
305
|
+
asynchronous work triggered by arbitrary App code. WDA has its separate Runner
|
|
306
|
+
execution namespace below. Generic lost install/launch replies, simultaneous multi-WebView
|
|
307
|
+
acceptance and complete complex iOS business coverage remain open. Native, H5 and
|
|
308
|
+
Flutter Intent/Script are available within their implemented command contracts;
|
|
309
|
+
availability does not establish production acceptance.
|
|
310
|
+
|
|
311
|
+
### iOS WDA target and session
|
|
312
|
+
|
|
313
|
+
WDA commands require explicit `deviceId` and `wdaRunnerBundleId`. The latter is
|
|
314
|
+
the Runner application that owns the data container, not the App under test or
|
|
315
|
+
the test bundle. Every connection copies `Documents/ai_app_bridge_wda.json` from
|
|
316
|
+
that exact device/Runner container. It declares `schemaVersion:"aab.ios-wda/v1"`,
|
|
317
|
+
`bundleId`, `runtimeEpoch`, `processId`, `port` and `ok:true`. Status preflight,
|
|
318
|
+
request headers and every response must match all five identity fields. A WDA
|
|
319
|
+
URL, hostname, `/status` response or vendor UUID alone cannot establish a
|
|
320
|
+
physical-device binding. These fields protect routing; they are not authentication.
|
|
321
|
+
|
|
322
|
+
`ios-setup --start-wda` builds a prepared copy of pinned `appium-webdriveragent`
|
|
323
|
+
14.1.1; the dependency's installed source is not modified. Signing requires
|
|
324
|
+
`teamId` or the configured development team. `wdaTestBundleId` defaults to
|
|
325
|
+
`io.github.mobileaidev.aiappbridge.wda`; its Runner ID is the test bundle ID plus
|
|
326
|
+
`.xctrunner`. Setup returns `wdaRunnerBundleId`. Unmodified WDA is rejected.
|
|
327
|
+
Setup first runs `build-for-testing` on the Host, then starts the device with
|
|
328
|
+
`test-without-building`. A local compilation failure reports `ios_wda_build_failed`
|
|
329
|
+
without marking a device dispatch. Device-test interruption still requires original
|
|
330
|
+
device completion proof. After WDA is ready, setup launches the target App and
|
|
331
|
+
checks a fresh SDK response because starting the Runner changes the foreground App.
|
|
332
|
+
`wdaProjectPath`, `wdaBundleId`, unbound URL discovery and log/port scans are
|
|
333
|
+
removed. An optional `wdaUrl` can select an explicitly forwarded HTTP(S) endpoint;
|
|
334
|
+
without it, use only the selected developer tunnel IP and descriptor port.
|
|
335
|
+
|
|
336
|
+
`ios-wda-session` operations are explicit:
|
|
337
|
+
|
|
338
|
+
- `status`: requires the device and Runner IDs; returns foreground App and current
|
|
339
|
+
session. It does not accept `bundleId` or `wdaSessionId`.
|
|
340
|
+
- `create`: additionally requires `bundleId`. The App must already be foreground;
|
|
341
|
+
its actual bundle ID and process ID are frozen into the returned session.
|
|
342
|
+
It cannot launch an App or replace an existing session.
|
|
343
|
+
- `close`: requires `bundleId` and `wdaSessionId`. It closes exactly that session
|
|
344
|
+
without terminating the App, including after the foreground App changes.
|
|
345
|
+
|
|
346
|
+
`ios-uia-tree`, `ios-tap`, `ios-input` and `ios-swipe` require all four IDs:
|
|
347
|
+
`deviceId`, `wdaRunnerBundleId`, `bundleId`, `wdaSessionId`. The prepared Runner
|
|
348
|
+
checks its own identity and the session's foreground App/PID on its main route
|
|
349
|
+
queue before dispatch. Old Runner epochs, switched Apps and restarted processes
|
|
350
|
+
fail explicitly. A tree read never creates a session. Tap and swipe remain
|
|
351
|
+
coordinate operations scoped to this foreground App; they are not semantic
|
|
352
|
+
element actions.
|
|
353
|
+
|
|
354
|
+
`ios-set-orientation` requires the same four IDs plus `orientation`: `portrait`,
|
|
355
|
+
`landscapeLeft`, `landscapeRight` or `portraitUpsideDown`. These are App interface
|
|
356
|
+
directions; the Runner explicitly converts the opposite landscape device direction.
|
|
357
|
+
The Runner must advertise `orientationSchema:"aab.ios-orientation/v1"`.
|
|
358
|
+
The command uses the managed `set-orientation` action and the original XCTest
|
|
359
|
+
orientation callback. Its result reports `requestedOrientation` and
|
|
360
|
+
`observedInterfaceOrientation` (null if unknown); a different observed direction
|
|
361
|
+
returns `ios_wda_orientation_not_observed`, even when XCTest accepted the request.
|
|
362
|
+
Reobserve the tree/DOM and verify the intended controls after rotation.
|
|
363
|
+
|
|
364
|
+
Optional `expectedSession` has `{schemaVersion:"aab.ios-native-session/v1",
|
|
365
|
+
runnerEpoch,bundleId,processId,sessionId}` from the preceding `ios-uia-tree`.
|
|
366
|
+
Intent supplies it automatically for
|
|
367
|
+
`{action:"setOrientation",orientation:"landscapeLeft"}`. A changed observation
|
|
368
|
+
binding returns `reobserve_required` before dispatch; the Runner also checks
|
|
369
|
+
foreground identity immediately before submission. CLI/MCP use this same command,
|
|
370
|
+
and Script uses `ctx.call('ios-set-orientation',{orientation:'portrait'})` with
|
|
371
|
+
`app.interact`. No session is implicitly created or replaced.
|
|
372
|
+
|
|
373
|
+
`ios-tap-native` and `ios-input-native-text` require those same four target IDs
|
|
374
|
+
and a `selector` with exactly one of `accessibilityId`, `label`, or `elementId`;
|
|
375
|
+
optional `type` distinguishes repeated labels (for example a Button from a
|
|
376
|
+
StaticText). Input also requires `text` and replaces the selected editor's value.
|
|
377
|
+
The Runner must advertise `aab.ios-native-target/v1`. Missing, ambiguous, hidden,
|
|
378
|
+
disabled or changed controls fail explicitly; there is no coordinate fallback.
|
|
379
|
+
Intent passes its observed `expectedTarget`, binding the original element UID,
|
|
380
|
+
raw identifier, label, type, App process, session and Runner epoch. Script can
|
|
381
|
+
pass the same expectedTarget, or resolve a fresh exact selector on each call.
|
|
382
|
+
An unlabeled icon can use an elementId read from the current tree, not a UID
|
|
383
|
+
retained across App relaunches. The Runner revalidates immediately before the
|
|
384
|
+
original XCTest event submission. Managed action HTTP waits use the admitted
|
|
385
|
+
action deadline; ordinary reads retain their short request budget.
|
|
386
|
+
|
|
387
|
+
When an iOS native Intent summary exceeds its byte budget, visible buttons and
|
|
388
|
+
inputs take priority over long WKWebView text, followed by other visible nodes.
|
|
389
|
+
The returned nodes retain the original preorder, sourceIndex, element IDs and
|
|
390
|
+
visibility facts. This preserves native toolbar/sheet controls without inferring
|
|
391
|
+
that they are clickable or unoccluded. A truncated summary cannot prove absence;
|
|
392
|
+
the original observation tree remains available through the evidence export.
|
|
393
|
+
|
|
394
|
+
`ios-input` requires exactly one `elementId` or `accessibilityId`, plus `text`
|
|
395
|
+
(0–16384 characters). An accessibility ID must resolve to exactly one W3C element;
|
|
396
|
+
zero or multiple matches fail. Host submits one managed input action. Runner
|
|
397
|
+
clicks that editor, optionally sends one keyboard-clear HID event for
|
|
398
|
+
`clearFirst:true`, observes the empty value, and sends the text. Each event checks
|
|
399
|
+
App/PID/session and the selected editor; keyboard events also require its focus
|
|
400
|
+
immediately before permission. Missing, stale or unfocused editors fail; a
|
|
401
|
+
nonempty or unsupported raw value after clearing fails with `ios_wda_clear_not_observed`.
|
|
402
|
+
An empty XCTest value can be `nil` or an empty string. The displayed placeholder
|
|
403
|
+
is not used as proof of an empty editor. The clear event settles only from the
|
|
404
|
+
original XCTest daemon callback, followed by a fresh editor/focus check.
|
|
405
|
+
There are no coordinate/global-key endpoint replacements or clearing retries.
|
|
406
|
+
XCTest synthesizes keyboard events using the current keyboard focus: these checks
|
|
407
|
+
cannot atomically prevent an App from switching focus during an already submitted
|
|
408
|
+
event. Input, replacement and clearing were verified on the iPhone 17 Pro Max
|
|
409
|
+
with iOS 27.0 in the 2026-09-10 device checkpoint. Reentrant focus behavior and
|
|
410
|
+
other editor/keyboard/system combinations remain separate acceptance gaps.
|
|
411
|
+
|
|
412
|
+
`value.error` is a command failure even on HTTP 200. Missing or mismatched
|
|
413
|
+
response identity cannot pass. Normal WDA command deadlines default to 30000 ms
|
|
414
|
+
across discovery and HTTP. Cancellation closes local requests, and failed setup
|
|
415
|
+
waits for its own xcodebuild process to close; it never kills another Runner's
|
|
416
|
+
process group. A lost WDA write response retains physical-device ownership and
|
|
417
|
+
is never replayed through another route. `doctor.ready` requires a connected device,
|
|
418
|
+
developer services, the App SDK and bound WDA; it is a connectivity result, not
|
|
419
|
+
production acceptance.
|
|
420
|
+
|
|
421
|
+
### iOS WDA execution and recovery
|
|
422
|
+
|
|
423
|
+
The prepared Runner advertises `aab.wda-execution/v1` at `/aab/status`. Session
|
|
424
|
+
create/close, tap, swipe and input all use one managed admission slot. Control
|
|
425
|
+
requests run outside the UI queue, so queued work can be cancelled while main is
|
|
426
|
+
busy. Every event requires permission immediately before submission. Queued
|
|
427
|
+
cancellation ends without an event; once an event is submitted, cancellation
|
|
428
|
+
prevents later steps but retains occupancy until the original XCTest completion
|
|
429
|
+
callback. It does not actively abort an in-flight XCTest event. A missing callback
|
|
430
|
+
or an exception during submission remains unresolved.
|
|
431
|
+
|
|
432
|
+
Runner synchronously commits `aab.wda-completion/v1` into the real segmented
|
|
433
|
+
action store before reporting settlement. Records bind Runner, operation,
|
|
434
|
+
App/PID, session, action ID and original Runner epoch. Persistence failure keeps
|
|
435
|
+
admission closed; cancellation may retry storing the same finished result, never
|
|
436
|
+
replay its event. Disk reads are paged at 64 records/2 MiB with a fixed committed
|
|
437
|
+
upper bound and scope-bound cursors. Missing or evicted records cannot release
|
|
438
|
+
ownership, even when a new Runner is idle.
|
|
439
|
+
|
|
440
|
+
Use the existing `ios-execution` command with `kind:"wda"`:
|
|
441
|
+
|
|
442
|
+
- `status`: requires `deviceId` and `wdaRunnerBundleId`; returns Host ownership
|
|
443
|
+
and Runner execution status.
|
|
444
|
+
- `result`/`cancel`: also require the original `actionId` and `runtimeEpoch`.
|
|
445
|
+
- `reconcile`: reads the durable Host pending identity and queries only that
|
|
446
|
+
original Runner/action/epoch/target. A committed matching receipt releases the
|
|
447
|
+
physical UDID reservation without replay. A new serving Runner epoch does not
|
|
448
|
+
replace the original completion epoch.
|
|
449
|
+
|
|
450
|
+
The WDA branch rejects SDK-only `bundleId`, `runtimeUrl`, `iosHost` and `iosPort`.
|
|
451
|
+
An explicit `wdaUrl` can select the current forwarded connection; the exact
|
|
452
|
+
device/Runner container must still match. `status` showing idle or `cancel`
|
|
453
|
+
acknowledging the request is insufficient to release retained Host ownership.
|
|
454
|
+
Use `reconcile`. Software fault checks and unsigned arm64 compilation qualify
|
|
455
|
+
this contract; physical-device execution remains a separate gate.
|
|
456
|
+
|
|
457
|
+
## Execution operation contracts
|
|
458
|
+
|
|
459
|
+
`operation` is required. Each operation accepts only its own fields. For Intent,
|
|
460
|
+
use `start/status/observe/decide/pause/resume/cancel/intervene`; intervention
|
|
461
|
+
requires a reason. For Script, use `start/status/wait/result/pause/resume/decide/cancel/runtime-status`.
|
|
462
|
+
Read progress through `status` or event pages from `wait`; `ctx.progress()` remains
|
|
463
|
+
available inside the program. `progress`, Script `intervene`, Intent `reobserve`,
|
|
464
|
+
and `start.spec` are removed aliases.
|
|
465
|
+
|
|
466
|
+
Intent start requires `goal` and an explicit Android, iOS or Web target. Android uses
|
|
467
|
+
`platform:"android"`, `serial`, and `packageName`; optional fields include `adb`,
|
|
468
|
+
`port` and a `foregroundPackages` allowlist. iOS uses `platform:"ios"`, `deviceId`
|
|
469
|
+
and `bundleId`, plus the SDK/WDA binding described above. Web uses `platform:"web"`,
|
|
470
|
+
`sessionId`, `runtimeEpoch` and the observed `targetId`, with provider `h5`. Supervised
|
|
471
|
+
mode is the default. Autonomous mode requires `mode:"autonomous"` and `agentModule`;
|
|
472
|
+
its `budget` defaults to 30 actions, 30 Agent calls, 120000 ms and `allowlist:["tap"]`.
|
|
473
|
+
The allowlist uses the public action vocabulary, including Web `pressKey` and iOS
|
|
474
|
+
`setOrientation`; it never bypasses the selected provider/platform action schema.
|
|
475
|
+
The last permitted Agent reply is validated and may act or finish. A spent action
|
|
476
|
+
or call budget prevents another Agent call. Malformed replies remain
|
|
477
|
+
`waiting_for_decision` with a field error and require explicit correction; they
|
|
478
|
+
neither dispatch nor trigger another Agent request automatically.
|
|
479
|
+
|
|
480
|
+
Every decision requires `decisionId`, positive integer `basedOnRevision`, and
|
|
481
|
+
`agentDecision`. `act` requires a provider-specific `action`; terminal decisions
|
|
482
|
+
`complete/fail/inconclusive` forbid one. Tap uses an exact `selector` object, with
|
|
483
|
+
one identity: Native/UIA `text`, `resourceName` or `contentDescription`; Flutter
|
|
484
|
+
`text` or `nodeId`. Only Native supports a scoped `within` selector. Actions inherit
|
|
485
|
+
the provider of the committed observation; an explicit provider must match it.
|
|
486
|
+
To change providers within a supervised Intent, call
|
|
487
|
+
`{"operation":"observe","operationId":"…","provider":"h5","basedOnRevision":3}`.
|
|
488
|
+
`provider` and `basedOnRevision` are optional. Android accepts `native/uia/flutter/h5`;
|
|
489
|
+
iOS accepts `native/h5/flutter`; Web accepts `h5`. The device/App target and any WDA session stay
|
|
490
|
+
frozen. Switching to iOS native requires the original target to contain its WDA
|
|
491
|
+
Runner and session. Installation and permission workflows retain their required
|
|
492
|
+
`uia` provider and reject another provider or any explicit `observationTarget`.
|
|
493
|
+
|
|
494
|
+
For Android and iOS H5, `start` and `observe` accept `observationTarget:{webViewId:"observed ID"}`
|
|
495
|
+
or `null`. Start defaults to `null`; an explicit selection is configured at start.
|
|
496
|
+
On `observe`, omitting it retains the committed selection when the provider is
|
|
497
|
+
unchanged. Changing provider without a selection clears it to `null`. An explicit
|
|
498
|
+
`null` requires one visible WebView; ambiguity never selects the first candidate.
|
|
499
|
+
An explicit ID never selects a replacement if its view becomes hidden or closes.
|
|
500
|
+
The object accepts only `webViewId` and is supported by Android and iOS H5. Invalid
|
|
501
|
+
selections return a field error before provider I/O or revision changes.
|
|
502
|
+
|
|
503
|
+
Subsequent provider and WebView selections change only after their observation and
|
|
504
|
+
summary are committed.
|
|
505
|
+
Use the returned revision for the next decision; a decision from before the switch
|
|
506
|
+
returns `reobserve_required` without dispatch. Provider acquisition failure enters
|
|
507
|
+
`waiting_for_observation` and blocks decisions on the retained old summary. Retrying
|
|
508
|
+
`observe` with neither selection reads the last successfully selected provider and
|
|
509
|
+
WebView; specify the desired selection again to retry a failed switch. A rejected
|
|
510
|
+
provider observation is retained as an `observation-failed` checkpoint before
|
|
511
|
+
`observationFailure` exposes its evidence ID, attempted selection and original
|
|
512
|
+
structured response (including available `webViews`). A successful observation
|
|
513
|
+
clears the active failure; its checkpoint remains in history and exported evidence.
|
|
514
|
+
Failure details are not exposed if their evidence cannot be persisted. A pending
|
|
515
|
+
observation cannot accept another observation or action, and cancellation drains
|
|
516
|
+
it before finalization. H5 actions still bind the page and element from the
|
|
517
|
+
committed observation, including after an explicit WebView selection.
|
|
518
|
+
|
|
519
|
+
Native/UIA scrolling is `scroll` with required `direction:"up"|"down"`; Native also
|
|
520
|
+
requires the exact scroll-container `selector`. Flutter uses
|
|
521
|
+
`scrollBy` with required container `selector` and `delta`. Native and Flutter
|
|
522
|
+
`inputText` require `selector` and `value`, including an empty value to clear.
|
|
523
|
+
`keyevent` requires `keyCode` and accepts 0;
|
|
524
|
+
`back` has no key parameter. See the discovered schema for native editing gestures.
|
|
525
|
+
|
|
526
|
+
Intent capture requirements use `require.streams`, a nonempty unique array of
|
|
527
|
+
`logs/network/state/events`, plus the documented query window/limit options.
|
|
528
|
+
These options cannot replace the operation's target. Business JSON is open only
|
|
529
|
+
where declared, such as `script.inputs`, Script Agent answers and registered Web
|
|
530
|
+
App action data; control objects reject unknown fields and invalid values.
|
|
531
|
+
|
|
532
|
+
Intent expands `require.streams` into individual capture reads; the streams list
|
|
533
|
+
and foreground-routing options never become command arguments. A historical
|
|
534
|
+
partial page retains its gap. Its committed SDK watermark can bound the next
|
|
535
|
+
action window; only that later read can establish complete coverage for new
|
|
536
|
+
facts. Capture failures retain their machine code, field and message in the
|
|
537
|
+
observation and portable recording.
|
|
538
|
+
|
|
539
|
+
Script start accepts `script`, optional operation ID, `recordingDir`, and
|
|
540
|
+
`pythonPath` for Python only. Supply `script.target` for device work. Language is
|
|
541
|
+
exactly `javascript` or `python`; source is exactly one of `source` or `sourcePath`.
|
|
542
|
+
Policy supports `timeoutMs`, `restartPolicy`, `maxOutputBytes`, `maxProgressBytes`.
|
|
543
|
+
`policy.onFailure` was unused and has been removed: source checks call results
|
|
544
|
+
and assertion verdicts, uncaught exceptions fail execution, and pause is explicit.
|
|
545
|
+
Recording currently requires `restartPolicy:"none"`. Frozen checkpoint recovery
|
|
546
|
+
uses its saved program and never blends missing caller fields with stored fields.
|
|
547
|
+
|
|
548
|
+
Script completion persists its returned JSON before exposing `completed`. Status,
|
|
549
|
+
wait responses and terminal events carry only
|
|
550
|
+
`resultRef:{evidenceId,bytes,sha256,originalSha256,representation}`; the full value
|
|
551
|
+
is read with `script {operation:"result",operationId:"…"}`. This reads and checks
|
|
552
|
+
the retained FactStore record, including after runtime restart, independently of
|
|
553
|
+
the bounded progress ring. `maxOutputBytes` defaults to 1 MiB and accepts at most
|
|
554
|
+
64 MiB; the store must retain the whole result or report failure. A failed result
|
|
555
|
+
write ends the operation as `failed` with `result_not_persisted`.
|
|
556
|
+
|
|
557
|
+
Results use the same password/token redaction as other persistent evidence.
|
|
558
|
+
`representation` is `original-json` or `redacted-json`; `sha256` and `bytes`
|
|
559
|
+
describe the persisted canonical JSON, and `originalSha256` describes the original
|
|
560
|
+
canonical return value. Successful reads return `result`, `resultRef` and
|
|
561
|
+
`persisted:true`. An unfinished operation returns `result_not_ready`; failed or
|
|
562
|
+
cancelled work returns `result_unavailable`. Missing result persistence, retained
|
|
563
|
+
checkpoint with evicted result, and inconsistent content return
|
|
564
|
+
`result_not_persisted`, `result_not_retained` and `result_checksum_mismatch`
|
|
565
|
+
respectively. An evicted or unknown operation may return `unknown_operation`;
|
|
566
|
+
store/read errors remain errors. No missing result is reconstructed from events.
|
|
567
|
+
|
|
568
|
+
`flutter-action` and `ios-flutter-action` accept a typed `payload` object for the
|
|
569
|
+
documented SDK actions. Payload `actionId` is owned by the Host. These remain expert
|
|
570
|
+
commands; raw SDK actions do not acquire Intent's observation contract. `web-command`
|
|
571
|
+
uses a builtin name and its typed `arguments`; registered App actions use
|
|
572
|
+
`name:"action", arguments:{name:"registered.name", arguments:{...}}`. Web target
|
|
573
|
+
commands require `sessionId`.
|
|
574
|
+
|
|
575
|
+
Flutter semantic taps resolve against the complete operable tree before DOWN.
|
|
576
|
+
While the pointer is held, the SDK revalidates the original Element, semantics,
|
|
577
|
+
action ancestors and the actual touch position. A removed, changed, covered or
|
|
578
|
+
moved-away target receives CANCEL instead of UP. Automatic whole-App snapshots
|
|
579
|
+
wait until this short pointer terminates, then resume; they cannot add tree
|
|
580
|
+
inspection work to the held gesture. App callbacks can still block the Flutter
|
|
581
|
+
isolate, and a mechanical tap receipt still requires an observed business outcome.
|
|
582
|
+
|
|
583
|
+
## Target and dispatch
|
|
584
|
+
|
|
585
|
+
Every Android mutation requires explicit `serial`. App-specific commands require
|
|
586
|
+
`packageName` or, for compatible SDK reads, an explicit bridge port. Discovery
|
|
587
|
+
currently exposes common transport options; command-specific semantic and nested
|
|
588
|
+
runtime schemas are still being tightened. The current contract does not claim
|
|
589
|
+
that every advertised optional parameter is already equally meaningful on all
|
|
590
|
+
platforms. Read `entrypoints.script` independently of direct MCP availability.
|
|
591
|
+
|
|
592
|
+
`tap` uses physical pixels. An explicit package must match the foreground even
|
|
593
|
+
with `feedback:"off"`. App scope uses the SDK; `scope:"device"` chooses physical
|
|
594
|
+
ADB input. Unknown/mismatched foreground is a non-dispatched failure.
|
|
595
|
+
`tap-flutter` uses Flutter logical pixels, with no DPR multiplication.
|
|
596
|
+
|
|
597
|
+
`tree` and `uia-tree` compact reads accept `maxDepth` from 0 to 200 and
|
|
598
|
+
`maxNodes` from 1 to 1000. These parameters constrain actual returned nodes;
|
|
599
|
+
values outside the implemented limits fail validation instead of being clamped.
|
|
600
|
+
|
|
601
|
+
`tap-text` retains automatic provider selection. `provider:"auto"` inspects
|
|
602
|
+
Native, Flutter, then UIAutomator until one exact operable match is selected.
|
|
603
|
+
Read-only discovery failures are recorded and another provider may be inspected.
|
|
604
|
+
Ambiguous matches stop selection. The returned `provider`, `coordinateSpace`,
|
|
605
|
+
`selected` and `observations` explain the choice. Specify `native`, `flutter`
|
|
606
|
+
or `uia` to pin replay. Foreground changes invalidate the choice. After dispatch,
|
|
607
|
+
an error or unknown result is returned without another provider attempt.
|
|
608
|
+
This convenience command does not provide Intent's complete revision and page
|
|
609
|
+
identity contract; flows needing that contract should use Intent decisions.
|
|
610
|
+
|
|
611
|
+
`input-text` is SDK native Unicode input and accepts an empty string to clear
|
|
612
|
+
an editor. An optional exact Native `selector`, including `within`, binds a
|
|
613
|
+
unique editable View using the same target protocol as Intent. A selector and
|
|
614
|
+
coordinates are mutually exclusive. Without either, it requires a visible
|
|
615
|
+
focused EditText in the focused foreground window; it never chooses an arbitrary
|
|
616
|
+
first editor. An SDK failure is returned without ASCII ADB retry.
|
|
617
|
+
`clear-app-data` selects `method:"pm-clear"` (default) or
|
|
618
|
+
`"runtime"` before one attempt. WDA tap/input/swipe use their selected endpoint
|
|
619
|
+
once; a response timeout is ambiguous and is not retried through another endpoint.
|
|
620
|
+
|
|
621
|
+
Android mutations share physical-serial ownership across cooperating Host
|
|
622
|
+
processes and package installations under the same OS user. Different packages
|
|
623
|
+
on one serial contend; different serials remain independent. The shared directory
|
|
624
|
+
is `~/.ai-app-bridge/device-ownership/v1`, independent of FactStore and working
|
|
625
|
+
directory. `AI_APP_BRIDGE_DEVICE_OWNERSHIP_DIR` explicitly configures a shared
|
|
626
|
+
namespace; all cooperating processes must use the same directory. This does not
|
|
627
|
+
arbitrate unrelated ADB clients, other OS users, remote hosts, or two serial
|
|
628
|
+
aliases that address one phone.
|
|
629
|
+
|
|
630
|
+
An OS-managed exclusive lock has no heartbeat expiry. A separately synced journal
|
|
631
|
+
records pending work before dispatch. An idle dead owner can be replaced; an
|
|
632
|
+
unresolved action survives Host death, timeout and restart and blocks new writes
|
|
633
|
+
with `device_ownership_unresolved`. Long-running installation retains its own
|
|
634
|
+
record while observed installer UI actions execute under the same owner. Ending
|
|
635
|
+
the local ADB process does not by itself confirm a cancelled install has settled.
|
|
636
|
+
|
|
637
|
+
Use `device-ownership {operation:"status",serial:"DEVICE"}` to inspect ownership,
|
|
638
|
+
or `operation:"reconcile"` to query the recorded action through its original
|
|
639
|
+
package and transport configuration. Reconciliation exclusively owns the device
|
|
640
|
+
while checking; there is no force-release, caller-supplied replacement target,
|
|
641
|
+
expiry, or action replay. Android Native, Flutter, H5 and managed shell actions can
|
|
642
|
+
recover from an identity-matching `aab.native-execution/v1`,
|
|
643
|
+
`aab.flutter-execution/v1`, `aab.h5-execution/v1` or
|
|
644
|
+
`aab.android-shell-execution/v1` terminal receipt. UIA node actions use
|
|
645
|
+
`aab.uia.execution.v1`, additionally matching the boot ID, original request
|
|
646
|
+
bytes/hash, snapshot and node binding. Recovery queries the original runtime or
|
|
647
|
+
reads that same action's durable phone record; it never starts a replacement action.
|
|
648
|
+
Successful reconciliation returns `executionReceipt` and records it
|
|
649
|
+
in its own execution history; the original unknown result remains unchanged.
|
|
650
|
+
A missing action, idle SDK, changed runtime or failed connection is insufficient.
|
|
651
|
+
Other transports without a matching completion protocol remain unresolved.
|
|
652
|
+
Installation uses the original phone job and PackageInstaller session contract
|
|
653
|
+
described below.
|
|
654
|
+
|
|
655
|
+
The ownership journal is `aab.device-ownership/v2`, under the **same physical
|
|
656
|
+
lock directory** above. Reading v1 preserves every unresolved identity and
|
|
657
|
+
reservation; the next commit upgrades its format. v1 had no pending-ack queue,
|
|
658
|
+
so the upgrade cannot invent cleanup obligations for already forgotten actions.
|
|
659
|
+
Older writers reject v2 instead of dropping its new records.
|
|
660
|
+
|
|
661
|
+
UIA settlement atomically saves an acknowledgement obligation with the exact
|
|
662
|
+
request, receipt and original FactStore destination. At most eight may remain;
|
|
663
|
+
`device_acknowledgements_full` rejects further UIA preparation until reconciliation.
|
|
664
|
+
Acknowledgement first synchronously commits and reads back a completion record
|
|
665
|
+
in the same segmented FactStore used by MCP/Intent/Script. Only then may the phone
|
|
666
|
+
confirm its copy and the Host remove the obligation. Failures preserve the queue
|
|
667
|
+
without changing the known effect or replaying it. A busy native FactStore reports
|
|
668
|
+
`fact_store_writer_busy`: reconcile from its owning Host, or close that Host before
|
|
669
|
+
retrying. Changing the new Host's recording directory does not redirect an old
|
|
670
|
+
obligation. Ownership journals are bounded to 4 MiB.
|
|
671
|
+
|
|
672
|
+
`device-ownership {operation:"receipt",serial:"DEVICE",runtimeEpoch:"UUID",actionId:"ORIGINAL-ID"}`
|
|
673
|
+
reads retained UIA completion history from the configured Host FactStore without
|
|
674
|
+
contacting the phone. The returned request and completion each have `json`,
|
|
675
|
+
`originalSha256`, `storedSha256`, and `representation` (`original-json` or
|
|
676
|
+
`redacted-json`). `originalCompletionAvailable:false` explicitly means redaction
|
|
677
|
+
changed the original bytes. Missing or evicted history returns
|
|
678
|
+
`device_completion_not_retained`; it is never interpreted as an action outcome.
|
|
679
|
+
This query does not reconcile or release unknown ownership. History remains subject
|
|
680
|
+
to FactStore retention quotas, independent of later overwrites of `lastSettlement`.
|
|
681
|
+
|
|
682
|
+
Reconciliation also drains pending acknowledgements when there is no unresolved
|
|
683
|
+
effect, and reports each `acknowledgements[].disposition` with its durable history
|
|
684
|
+
reference. A stopped original session is acknowledged by a maintenance process
|
|
685
|
+
holding the phone root lock, without opening UiAutomation. A newer live runtime
|
|
686
|
+
may acknowledge an older epoch. `not_retained` only says its phone copy is already
|
|
687
|
+
absent; it is accepted for cleanup solely after the Host's matching completion
|
|
688
|
+
representation has been durably stored. An active epoch uses its original engine.
|
|
689
|
+
Another unresolved action remains occupied throughout cleanup of known history.
|
|
690
|
+
|
|
691
|
+
Android shell mutations, including explicit physical input, app launch,
|
|
692
|
+
PackageManager permission changes, app-ops, process signals and `logcat` clear,
|
|
693
|
+
use a detached phone worker. The Host persists the original action ID, job ID,
|
|
694
|
+
boot ID, command hash and monotonic admission deadline before starting it. The
|
|
695
|
+
phone records completion after the shell command exits. A pre-admission cancel
|
|
696
|
+
or expired deadline prevents the command from starting. After admission, the
|
|
697
|
+
command drains naturally; cancelling or killing the Host does not stop it or
|
|
698
|
+
release unknown ownership. Recovery checks this original job without replaying it.
|
|
699
|
+
|
|
700
|
+
The command process ending does not prove that navigation, PackageInstaller or
|
|
701
|
+
another asynchronous Android service completed its business operation. A shell
|
|
702
|
+
proof does not provide SDK target binding or App event correlation. UIA semantic
|
|
703
|
+
clicks use the separate node runtime described below.
|
|
704
|
+
|
|
705
|
+
Phone jobs are stored under `/data/local/tmp/ai-app-bridge-shell/v1`. Each stdout
|
|
706
|
+
and stderr file is limited to 64 KiB on the verified Android shell; exceeding
|
|
707
|
+
the limit fails the command and retains its exit code and completion proof.
|
|
708
|
+
At most 512 unacknowledged job directories may remain. A new preparation removes
|
|
709
|
+
only acknowledged directories; capacity exhaustion returns
|
|
710
|
+
`shell_execution_store_full` before dispatch. Unknown jobs are never evicted to
|
|
711
|
+
make room. A normal Host acknowledges only after reading the output and syncing
|
|
712
|
+
the matching proof to its ownership journal. Unread output, abandoned preparation
|
|
713
|
+
and recovered jobs remain retained; automatic retirement of those records is
|
|
714
|
+
not implemented. A missing directory or changed boot cannot release ownership.
|
|
715
|
+
|
|
716
|
+
`logcat` reads retain their text result. `logcat {clear:true}` returns an object
|
|
717
|
+
with `ok`, `text` and the verified execution fields because it mutates the phone.
|
|
718
|
+
|
|
719
|
+
A common command's `requestId` reuses the result only for the same target and
|
|
720
|
+
canonical command/arguments. Different content returns `idempotency_conflict`.
|
|
721
|
+
That cache is bounded and process-local, not a durable exactly-once guarantee.
|
|
722
|
+
|
|
723
|
+
## Script access and evidence
|
|
724
|
+
|
|
725
|
+
The catalog exposes executable Android, iOS and desktop Web capabilities.
|
|
726
|
+
Script calls use the same provider and target ownership as direct CLI/MCP calls;
|
|
727
|
+
catalog support does not establish complete platform business acceptance. Unsupported Script target
|
|
728
|
+
fields are rejected instead of being silently discarded. Recovery hashes include
|
|
729
|
+
target and permissions as well as source/policy.
|
|
730
|
+
|
|
731
|
+
`script decide` answers `ctx.askAgent` with a JSON value, including structured
|
|
732
|
+
objects, arrays and primitive values such as null or false. Supply the question's
|
|
733
|
+
requestId and revision. Repeating the same answer by canonical JSON content is
|
|
734
|
+
idempotent; changing an already answered question's value is rejected. Omitting
|
|
735
|
+
the answer is invalid and is distinct from explicitly answering null.
|
|
736
|
+
|
|
737
|
+
Default permissions are `app.read`, `capture.read`, `app.interact`. The trusted
|
|
738
|
+
caller may explicitly include `app.lifecycle` for clear data and
|
|
739
|
+
`app.permissions` for permission/app-op fixture changes. The host freezes this
|
|
740
|
+
selection into the operation; Script source cannot change it through `ctx.call`.
|
|
741
|
+
These are capability declarations inside a trusted local execution model, not
|
|
742
|
+
an OS sandbox or a multi-tenant authorization system. Raw eval and provider
|
|
743
|
+
management remain explicit direct expert commands. Capture-only calls cannot
|
|
744
|
+
supply CDP eval code or clear device logs.
|
|
745
|
+
|
|
746
|
+
Android mobile capture reads the device FactStore after durable attachment; Host
|
|
747
|
+
execution/observation records use the Host FactStore. iOS public capture now uses
|
|
748
|
+
the device segmented store as described below. Web capture is committed on ingress
|
|
749
|
+
to the Host FactStore; retained history and live barrier coverage have distinct
|
|
750
|
+
contracts described in the Web section. Strong mobile evidence
|
|
751
|
+
requires actual refs, matching epoch/target, the requested window and valid
|
|
752
|
+
coverage. Archive validity, command completion and business assertions are
|
|
753
|
+
separate results. See [Script authoring](SCRIPT_AUTHORING.md) and
|
|
754
|
+
[evidence archives](EVIDENCE_ARCHIVE.md).
|
|
755
|
+
|
|
756
|
+
## iOS persistent capture
|
|
757
|
+
|
|
758
|
+
The iOS SDK has one write/read path for logs, network, events and state. Before
|
|
759
|
+
persistent attachment, capture returns `ok:false` with `capture_store_opening`
|
|
760
|
+
or `capture_store_unavailable`; no in-memory payload history substitutes for it.
|
|
761
|
+
A successful POST means its record was queued. The returned `receipt` has
|
|
762
|
+
`accepted:true`, `committed:false` and the original `mobileFactId`. A later
|
|
763
|
+
successful persistent query proves the matching record exists. Flutter's iOS
|
|
764
|
+
MethodChannel forwards the complete payload, including `actionId`, and returns
|
|
765
|
+
the same receipt. App-log/NSLog/H5 console capture also uses this store; the
|
|
766
|
+
separate device-log partition remains separate.
|
|
767
|
+
|
|
768
|
+
HTTP GET accepts `view`, `sinceId`, `sinceMs`, `limit`, `runtimeEpoch`,
|
|
769
|
+
`afterActionId`, `factCursor`, `mobileFactId` and `targetKey`. Unknown names,
|
|
770
|
+
empty identifiers and invalid numbers fail explicitly. `limit` is 1–1000,
|
|
771
|
+
default 200; `sinceId` is exclusive, `sinceMs` inclusive. The old `since-id`
|
|
772
|
+
and `since-ms` HTTP aliases and silent limit clamping are removed.
|
|
773
|
+
|
|
774
|
+
- `legacy-live` is a bounded current-runtime projection; state keeps the latest
|
|
775
|
+
value per key. It reads persistent facts and reports projection limits.
|
|
776
|
+
- `decision-window` reads the current runtime. An action filter requires a
|
|
777
|
+
pre-action cursor, capture ID or timestamp boundary.
|
|
778
|
+
- `connected-history` reads retained records across runtimes. A capture ID
|
|
779
|
+
filter also requires `runtimeEpoch`, since IDs restart in each runtime.
|
|
780
|
+
An exact `mobileFactId` can resolve its original record after restart.
|
|
781
|
+
|
|
782
|
+
Queries flush and freeze the durable upper sequence in one writer operation;
|
|
783
|
+
new writes cannot enter that committed boundary. Each response carries refs,
|
|
784
|
+
coverage, the target/runtime, `nextCursor` and `watermarkCursor`. Page cursors
|
|
785
|
+
freeze the upper boundary; a watermark starts a subsequent window. A page scans
|
|
786
|
+
at most 1024 logical records or 8 MiB, with a two-second scan budget in addition
|
|
787
|
+
to bounded writer waits. An empty filtered page can have `hasMore:true`; callers
|
|
788
|
+
must use its `nextCursor`. Each writer read batch is at most 64 logical records
|
|
789
|
+
and 2 MiB. Query projection and page memory have explicit limits; payload
|
|
790
|
+
history is not hydrated into a second cache.
|
|
791
|
+
|
|
792
|
+
Clear persists stream generations and invalidates old cursors. Retention or
|
|
793
|
+
known write loss produces partial coverage. An unknown pre-attachment loss
|
|
794
|
+
cannot be cleared by `sinceId:0`. Metadata corruption, flush errors and changed
|
|
795
|
+
store bindings never return committed success. Capture status counts are bounded
|
|
796
|
+
accepted counts since attachment, not totals for retained disk history.
|
|
797
|
+
|
|
798
|
+
This storage contract has real-disk Swift integration coverage and iOS compilation
|
|
799
|
+
proof. Storage proof and complex business acceptance are separate. iOS Intent
|
|
800
|
+
and Script now use explicit native/H5/Flutter providers and bound execution
|
|
801
|
+
receipts described above; each real-App scenario retains its own evidence limits.
|
|
802
|
+
|
|
803
|
+
## Semantic targets and text waits
|
|
804
|
+
|
|
805
|
+
`tap-uia` takes an explicit package and either a selector or the complete
|
|
806
|
+
`targetRef` described below. A selector has exactly one field:
|
|
807
|
+
`{"text":"All files"}`, `{"contentDescription":"All files"}` or
|
|
808
|
+
`{"resourceName":"com.example:id/files"}`. A text selector matches only the
|
|
809
|
+
text attribute. A parent accessibility description with the same wording does
|
|
810
|
+
not make that child text ambiguous. Two actual text matches remain an error.
|
|
811
|
+
The command observes the current UIA runtime, binds the selected node, checks
|
|
812
|
+
the foreground again and sends the same original action ID through the existing
|
|
813
|
+
UIA execution journal. CLI, MCP and Script share this implementation.
|
|
814
|
+
|
|
815
|
+
For duplicate or unlabelled controls, `uia-tree` with `compact:true` exposes
|
|
816
|
+
`node.targetRef`; Intent UIA summaries expose the same reference. It contains
|
|
817
|
+
`{schemaVersion:"aab.uia.target/v1",bootId,runtimeEpoch,snapshotId,nodeRef}`.
|
|
818
|
+
Pass that complete object as `tap-uia.targetRef`, mutually exclusive with
|
|
819
|
+
`selector`. An Intent uses `selector:{nodeRef:node.targetRef.nodeRef}` from its
|
|
820
|
+
current revision, which supplies the original observation identity.
|
|
821
|
+
|
|
822
|
+
Reference actions use the retained node from that exact snapshot. The phone
|
|
823
|
+
locates the same accessibility source in the same focused window and checks all
|
|
824
|
+
original node and clickable-ancestor attributes before requesting the click.
|
|
825
|
+
Duplicate text does not choose a different node. Expired, changed or foreign
|
|
826
|
+
references fail before dispatch; they are never replaced with a fresh text
|
|
827
|
+
match. UIA retains at most eight snapshots for 60 seconds, so Scripts observe
|
|
828
|
+
immediately before selecting and acting. This is a runtime reference, not a
|
|
829
|
+
persistent selector to save between runs.
|
|
830
|
+
|
|
831
|
+
The convenience text commands below match display text and, for Native/UIA,
|
|
832
|
+
accessibility descriptions. When this yields several candidates, use `tap-uia`
|
|
833
|
+
or an Intent with a precise selector to state which attribute is intended.
|
|
834
|
+
|
|
835
|
+
`tap-text`, `tap-uia-text` and `tap-flutter-text` share exact, unique selection,
|
|
836
|
+
fresh revalidation and foreground checks. Auto discovery happens only before a
|
|
837
|
+
provider is selected. Revalidation failure never dispatches through a different
|
|
838
|
+
provider. Native selection is limited to the top non-hidden window; a dialog,
|
|
839
|
+
unknown root or disabled foreground root cannot expose a background target.
|
|
840
|
+
Flutter text targeting also checks that the native foreground is the activity.
|
|
841
|
+
Coordinates returned by Flutter remain logical pixels.
|
|
842
|
+
|
|
843
|
+
Native Intent tap/input/longPress/swipe/scroll and Flutter tap/input/scrollBy re-read before input.
|
|
844
|
+
UIA revalidates its saved node reference on the phone before Binder admission.
|
|
845
|
+
Changed identity, ambiguity or invalid bounds reject before dispatch. Native
|
|
846
|
+
input requires the SDK's explicit `editable:true`, including standard EditText.
|
|
847
|
+
Native tree nodes expose `checked:true|false` for Android `Checkable` controls
|
|
848
|
+
and `checked:null` for other Views. The checked state participates in the SDK
|
|
849
|
+
target guard; a state change after observation rejects the old target before
|
|
850
|
+
dispatch, so a stale checkbox action cannot toggle a newly changed value.
|
|
851
|
+
Semantic summaries retain both checked and unchecked controls, including
|
|
852
|
+
unlabelled preference switches. UIAutomator's `checked` value is meaningful only
|
|
853
|
+
when its node declares `checkable:true`; other UIA nodes have `checked:null`.
|
|
854
|
+
|
|
855
|
+
`tap-native` exposes a precise Native selector to CLI, MCP and Script callers:
|
|
856
|
+
|
|
857
|
+
```js
|
|
858
|
+
await ctx.call('tap-native', { selector: {
|
|
859
|
+
resourceName: 'com.example.app:id/confirm',
|
|
860
|
+
within: { text: 'Test item', ancestor: { resourceName: 'com.example.app:id/row' } }
|
|
861
|
+
} });
|
|
862
|
+
```
|
|
863
|
+
|
|
864
|
+
It requires an explicit App target and one unique match, revalidates the same View
|
|
865
|
+
before dispatch, and retains the original Native execution receipt. Its selector
|
|
866
|
+
uses the shared Native contract, including `within`; ambiguous, replaced and
|
|
867
|
+
unbound targets fail without dispatch. Passive descendants inside a selected
|
|
868
|
+
semantic container may receive its gesture; an interactive child or unrelated
|
|
869
|
+
overlay still prevents that container tap.
|
|
870
|
+
|
|
871
|
+
`input-text` with a `selector` uses the same foreground and View revalidation,
|
|
872
|
+
requires `editable:true`, and sends the exact editor reference to the SDK.
|
|
873
|
+
CLI, MCP and Script share this entry; missing, ambiguous, replaced or noneditable
|
|
874
|
+
targets fail before input. For example:
|
|
875
|
+
|
|
876
|
+
```js
|
|
877
|
+
await ctx.call('input-text', {
|
|
878
|
+
selector: { resourceName: 'com.example.app:id/search' }, text: 'Monaco'
|
|
879
|
+
});
|
|
880
|
+
```
|
|
881
|
+
|
|
882
|
+
Native Intent `tap`/`inputText`, `tap-native`, selector-based `input-text` and the Native branch of `tap-text` additionally
|
|
883
|
+
require `targetRef.schemaVersion:"aab.native-target/v1"` from the SDK tree. The
|
|
884
|
+
Host sends the selector, reference and action ID to `/v1/action/tap-target` or
|
|
885
|
+
`/v1/action/input-target`, without coordinates. In one UI-thread task the SDK
|
|
886
|
+
checks the runtime, focused window, View instance, observed semantic ancestry,
|
|
887
|
+
uniqueness, editability and clipping, then uses the current bounds/selected editor.
|
|
888
|
+
Same-View layout movement is allowed; replacing a View with an identically named
|
|
889
|
+
View is rejected. Input rechecks after synchronous focus/connection callbacks
|
|
890
|
+
and again after the input connection sets the selection, before committing text.
|
|
891
|
+
A rejection after requesting focus reports `dispatched:true`, since that effect
|
|
892
|
+
already occurred; it does not commit text to a replacement editor.
|
|
893
|
+
|
|
894
|
+
Coordinate and focused `input-text` also retain the selected editor and window
|
|
895
|
+
across those callbacks. A detached/replaced editor returns
|
|
896
|
+
`native_target_replaced`; changed focus returns `input_focus_changed`. Neither
|
|
897
|
+
path redirects text to the new focus. This binds the SDK's editor selection and
|
|
898
|
+
commit call, without making custom App input-connection behavior transactional.
|
|
899
|
+
|
|
900
|
+
Missing references or an unsupported endpoint return
|
|
901
|
+
`native_atomic_target_unavailable`; this selected Native attempt never downgrades
|
|
902
|
+
to coordinate or device input. A queued SDK task that times out is cancelled;
|
|
903
|
+
a mutation already running at timeout returns an ambiguous outcome. These checks
|
|
904
|
+
do not make app touch handlers or business outcomes transactional.
|
|
905
|
+
Explicit coordinate commands retain their primitive role.
|
|
906
|
+
|
|
907
|
+
UIA observation and semantic clicks require Android API 33 or newer and the
|
|
908
|
+
bundled, hash-verified `runtime/uia` module. `uia-tree` opens or reuses one
|
|
909
|
+
UiAutomation connection on the explicit device. XML snapshots carry boot,
|
|
910
|
+
runtime and snapshot IDs; each node has an opaque reference. Ordinary UIA text
|
|
911
|
+
commands, Intent (including installer and permission choices), and JavaScript /
|
|
912
|
+
Python Script calls send that reference and their original action ID to the
|
|
913
|
+
same runtime. They do not convert text matches into physical coordinates.
|
|
914
|
+
|
|
915
|
+
The runtime checks the focused default-display window, unique selector match,
|
|
916
|
+
node attributes and clickable ancestor before one node action. Its guarantee
|
|
917
|
+
is `same_connection_node_and_reobserved_attributes`, not a transaction across
|
|
918
|
+
the App's content changes and Android window management. Original Binder
|
|
919
|
+
callbacks distinguish handled/rejected outcomes; an admitted action without a
|
|
920
|
+
matching callback remains unknown. Host timeout, cancellation or process exit
|
|
921
|
+
does not release that action's device ownership. Pre-admission cancellation
|
|
922
|
+
persists a tombstone, so a delayed start cannot execute afterward.
|
|
923
|
+
|
|
924
|
+
For a dead UIA owner, `device-ownership --operation reconcile` can recover a
|
|
925
|
+
committed `prepared` or `queued` record. One-shot phone maintenance must acquire
|
|
926
|
+
the original root's exclusive OS lock and validate the original request, epoch,
|
|
927
|
+
executable hash, zero interaction ID and empty receipt. It writes a durable
|
|
928
|
+
`recovered_before_admission` receipt with `dispatched:false` and
|
|
929
|
+
`uia_owner_exited_before_admission`; it never starts UiAutomation. The receipt
|
|
930
|
+
separately identifies the recovery boot/elapsed time, original preparation time,
|
|
931
|
+
prior record SHA-256 and executable hashes. It has no original callback or
|
|
932
|
+
`completedAtElapsedMs`. Original request bytes and deadline stay unchanged.
|
|
933
|
+
Already admitted/unknown actions and absent original records remain unresolved.
|
|
934
|
+
The first valid recovery receipt is retained on retry, persisted to FactStore,
|
|
935
|
+
and acknowledged through the same durable cleanup queue as other UIA receipts.
|
|
936
|
+
|
|
937
|
+
`uia-runtime --serial DEVICE --operation status|start|stop` exposes expert
|
|
938
|
+
lifecycle control. `status` is read-only; `start` and `stop` share device
|
|
939
|
+
ownership with actions. Stop requires committed terminal actions. Explicit start
|
|
940
|
+
checks the phone's OS-managed process lock. A dead process can be reopened only
|
|
941
|
+
after the new process acquires that lock and audits all original action records;
|
|
942
|
+
any nonterminal or corrupt record blocks reopening, including after a reboot.
|
|
943
|
+
An HTTP timeout or a stale `running` descriptor does not authorize replacement.
|
|
944
|
+
Authentication tokens
|
|
945
|
+
stay in private phone descriptors and are absent from public status and Host
|
|
946
|
+
ownership records. Each runtime retains at most 256 actions; the root retains
|
|
947
|
+
at most 64 sessions. A fresh observation automatically rotates a full session
|
|
948
|
+
only when every action is durably terminal and acknowledged. An already bound
|
|
949
|
+
action never triggers rotation; a full journal rejects it before Host preparation.
|
|
950
|
+
Old epoch references become invalid and require a new observation.
|
|
951
|
+
|
|
952
|
+
At startup, fully acknowledged sessions move atomically into `retired/`; both
|
|
953
|
+
parent directories are synced before any contents are deleted. Interrupted
|
|
954
|
+
deletion resumes there. Unacknowledged original receipts remain byte-exact in
|
|
955
|
+
`sessions/`. Reclamation never evicts unknown actions or trusts temporary files
|
|
956
|
+
as completion proof. Host acknowledgement follows durable ownership settlement,
|
|
957
|
+
so long-term evidence must be read from Host storage/archives after phone-side
|
|
958
|
+
retirement. A stopped session's unacknowledged receipt is retained and can be
|
|
959
|
+
acknowledged through exclusive-lock maintenance. Current real-device evidence
|
|
960
|
+
covers API 36 on OPPO PGFM10 and OnePlus PKR110 with distinct checkpoint scopes.
|
|
961
|
+
|
|
962
|
+
Nonsecret JSON captured as text keeps its original bytes during evidence
|
|
963
|
+
persistence, so Android's escaped solidus characters do not invalidate receipt
|
|
964
|
+
hashes. Credential redaction still applies, including duplicate/escaped JSON
|
|
965
|
+
keys. A redacted representation is not the original cryptographic credential.
|
|
966
|
+
|
|
967
|
+
Flutter semantic actions require `aab.flutter-target/v1` references from the
|
|
968
|
+
operable tree. IDs identify live Elements across snapshots; the reference also
|
|
969
|
+
binds the runtime, observed semantics, action ancestor, editor controller/focus
|
|
970
|
+
node and scroll container. The SDK rebuilds its current targets before dispatch,
|
|
971
|
+
rejects changed/replaced/covered targets, and uses current logical geometry.
|
|
972
|
+
A missing reference returns `flutter_atomic_target_unavailable` without a
|
|
973
|
+
coordinate fallback. Truncated trees cannot establish a unique target.
|
|
974
|
+
|
|
975
|
+
Flutter Intent compares target identities structurally; JSON object key order
|
|
976
|
+
does not identify an Element. Runtime, Element, semantic or guard changes still
|
|
977
|
+
require a fresh observation. The operable viewport reports logical-pixel
|
|
978
|
+
`viewInsets`; visible geometry intersects the actual View viewport excluding the
|
|
979
|
+
native keyboard. A partially visible scroll container keeps its exposed bounds;
|
|
980
|
+
a fully covered control cannot receive a semantic action. This does not detect
|
|
981
|
+
every native system overlay outside Flutter's hit-test tree.
|
|
982
|
+
|
|
983
|
+
Flutter editor observations and Intent summaries preserve the standard field's
|
|
984
|
+
optional `label`, `hint` and `errorText`. Material declarations come from that
|
|
985
|
+
editor's InputDecorator; a Cupertino placeholder is a hint, not a label. Custom
|
|
986
|
+
label/error widgets are not inferred from nearby text. Select the observed node
|
|
987
|
+
by its metadata, then submit its exact `nodeId`; no label selector is implied.
|
|
988
|
+
Label or hint changes invalidate the observed editor identity, while a validation
|
|
989
|
+
message alone does not. Transparent TextSpan content and controls under
|
|
990
|
+
zero-opacity Opacity/FadeTransition are excluded from the operable tree;
|
|
991
|
+
diagnostic strings are not used as visible text. Arbitrary canvas/shader paint
|
|
992
|
+
still needs App semantics.
|
|
993
|
+
|
|
994
|
+
The diagnostic widget-inspector JSON is separately bounded to 64 levels and
|
|
995
|
+
4000 entries and reports `truncated`. Its truncation does not imply that the
|
|
996
|
+
operable tree is truncated or that unreported diagnostic widgets are absent.
|
|
997
|
+
|
|
998
|
+
`tap-flutter` accepts either `selector:{text|nodeId}` or the explicit logical
|
|
999
|
+
coordinate pair `tapX/tapY`. `input-flutter-text` accepts an optional selector or
|
|
1000
|
+
coordinate pair; when both are omitted it selects one focused editor, or one
|
|
1001
|
+
unique visible editor. Several unfocused editors return
|
|
1002
|
+
`flutter_selector_not_unique`. After the tap and awaited frames, the SDK verifies
|
|
1003
|
+
the original EditableText, controller and focus before addressing that exact
|
|
1004
|
+
TextInputClient. `flutter_input_focus_changed` never redirects text to the new
|
|
1005
|
+
focus. App input formatters and callbacks retain their normal behavior.
|
|
1006
|
+
|
|
1007
|
+
`scroll-flutter` accepts a container selector with `delta` or `targetText`.
|
|
1008
|
+
Omitting the selector requires one visible Scrollable. Scrolling until text
|
|
1009
|
+
uses exact unique matching and retains that container across frames; there is
|
|
1010
|
+
no last-container choice, hidden keyboard action or fallback physical swipe.
|
|
1011
|
+
`flutter_scroll_boundary` reports no movement. These commands retain the
|
|
1012
|
+
existing Script `app.interact` permission; raw `flutter-action` remains expert
|
|
1013
|
+
only. Raw target references/action IDs are not public payload parameters.
|
|
1014
|
+
|
|
1015
|
+
The Dart runtime rejects overlapping action requests with `flutter_action_busy`.
|
|
1016
|
+
Target removal/change during a bound tap sends CANCEL before UP; restarting the
|
|
1017
|
+
runtime invalidates old references. Android additionally advertises
|
|
1018
|
+
`executionSchema:aab.flutter-execution/v1`. Host freezes its runtime epoch and
|
|
1019
|
+
action ID and sends the remaining execution budget, without restarting it.
|
|
1020
|
+
Missing capability fails with `flutter_execution_unavailable`; there is no
|
|
1021
|
+
old-protocol retry. Intent, Script and direct actions share this transport.
|
|
1022
|
+
|
|
1023
|
+
Android receives Flutter actions asynchronously, leaving its HTTP listener free
|
|
1024
|
+
for cancellation. Dart obtains native admission before starting a pointer
|
|
1025
|
+
sequence, writing the bound editor, or making another scroll jump. An admitted
|
|
1026
|
+
pointer sequence handles its timing and terminal events locally; waiting on a
|
|
1027
|
+
channel while holding DOWN must not turn a short tap into a long press. Local
|
|
1028
|
+
stop signals and a monotonic deadline terminate Bridge-owned waits with CANCEL.
|
|
1029
|
+
Opaque App futures remain owned until they actually finish.
|
|
1030
|
+
|
|
1031
|
+
On a lost response or Host cancellation, Host sends one bounded cancellation
|
|
1032
|
+
request to the same connection and runtime, without replaying the action. Only
|
|
1033
|
+
the original action's matching terminal receipt proves `settled:true`.
|
|
1034
|
+
Queued requests whose admission was revoked prove `dispatched:false`. A stopped
|
|
1035
|
+
request with previously granted admission remains SDK-busy until its original
|
|
1036
|
+
completion. After the 1500 ms cleanup grace period, missing confirmation returns
|
|
1037
|
+
`dispatched:null, ambiguous:true, settled:false`; it does not free SDK admission.
|
|
1038
|
+
A repeated cancel can retrieve the single retained terminal receipt. This is
|
|
1039
|
+
bounded cleanup evidence, not a persistent idempotency or rollback service.
|
|
1040
|
+
|
|
1041
|
+
Cancellation does not undo delivered input or App callbacks; an action may
|
|
1042
|
+
finish before its stop signal is observed. SDK settlement means no further
|
|
1043
|
+
Bridge-controlled continuation for that action. Cross-process device ownership
|
|
1044
|
+
retains unresolved work across Host death and can reconcile the original SDK
|
|
1045
|
+
receipt as described above. iOS
|
|
1046
|
+
still uses its distinct transport and does not advertise this Android contract.
|
|
1047
|
+
|
|
1048
|
+
Native Android tap, text input and gestures share one SDK execution coordinator.
|
|
1049
|
+
`status.debugBridge.nativeExecutionSchema` advertises `aab.native-execution/v1`;
|
|
1050
|
+
`nativeAction` identifies the active operation or is null. Host sends its action
|
|
1051
|
+
ID, the observed runtime epoch and remaining timeout in `execution`. Missing
|
|
1052
|
+
capability is `native_execution_unavailable`; there is no old-protocol retry.
|
|
1053
|
+
The shared `/v1/action/cancel` requires the original action ID and runtime epoch.
|
|
1054
|
+
This replaces the former gesture-only cancellation endpoint and status field.
|
|
1055
|
+
|
|
1056
|
+
Cancellation before main-thread dispatch prevents a late tap or input. Once an
|
|
1057
|
+
App callback has begun, the SDK retains occupation until it actually returns;
|
|
1058
|
+
the 1500 ms cleanup grace only bounds the reply, not callback lifetime. A blocked
|
|
1059
|
+
callback returns `settled:false, dispatched:null, ambiguous:true`. Tap stops
|
|
1060
|
+
with CANCEL before a late UP; input checks cancellation before subsequent writes.
|
|
1061
|
+
Input already committed and App side effects are not rolled back.
|
|
1062
|
+
|
|
1063
|
+
The coordinator retains one immutable terminal receipt in memory. Repeated
|
|
1064
|
+
cancellation can retrieve that receipt; reusing its action ID for another action
|
|
1065
|
+
is rejected. This is not durable deduplication: SDK restart or a missing cached
|
|
1066
|
+
receipt cannot prove an unresolved prior action settled. Terminal
|
|
1067
|
+
`native.action.settled` events carry the original action identity without input
|
|
1068
|
+
text. Native, Flutter, H5 evaluation and runtime clear-data cannot overlap an
|
|
1069
|
+
active Native operation through this SDK.
|
|
1070
|
+
|
|
1071
|
+
Native/Flutter/H5/managed-shell action results expose a compact `executionReceipt` when verified
|
|
1072
|
+
settlement or explicit non-dispatch is known, and null when it remains unknown.
|
|
1073
|
+
Ordinary execution history, Intent action receipts and Script action receipts
|
|
1074
|
+
preserve it. Commands containing several physical operations can return
|
|
1075
|
+
`executionReceipts`; ordinary history and Script action receipts preserve that
|
|
1076
|
+
array. Each response hash covers the validated protocol result before Host
|
|
1077
|
+
metadata, not the raw transport bytes. Settlement proves the recorded execution ended;
|
|
1078
|
+
it does not turn an ambiguous action or unverified business assertion into a pass.
|
|
1079
|
+
|
|
1080
|
+
Native Intent `longPress`, `swipe` and `scroll` use `/v1/action/gesture-target`.
|
|
1081
|
+
The top visible SDK window owns pointer selection. A touchable, non-focusable
|
|
1082
|
+
popup can use its focused owner with the same application window token; the SDK
|
|
1083
|
+
does not select a background window when the popup blocks an action. Native/H5
|
|
1084
|
+
window selection shares this rule. Native window observation carries
|
|
1085
|
+
`focused`, `focusable`, `touchable` and nullable
|
|
1086
|
+
`focusOwnerWindowId`. Native input requires actual focus in the selected window.
|
|
1087
|
+
Native touch events keep screen-space `rawX/rawY` and window-local `x/y` distinct,
|
|
1088
|
+
so App outside-touch interceptors receive the same coordinate spaces as normal input.
|
|
1089
|
+
Script and direct callers use the shared `native-gesture` command, requiring
|
|
1090
|
+
`serial`, `packageName` and a typed `payload` with the same selector/action fields:
|
|
1091
|
+
|
|
1092
|
+
```javascript
|
|
1093
|
+
await ctx.call('native-gesture', {
|
|
1094
|
+
payload: { action: 'longPress', selector: { text: 'Observed note' }, durationMs: 700 }
|
|
1095
|
+
});
|
|
1096
|
+
await ctx.call('native-gesture', {
|
|
1097
|
+
payload: { action: 'scroll', selector: { resourceName: 'example.app:id/list' }, direction: 'down' }
|
|
1098
|
+
});
|
|
1099
|
+
```
|
|
1100
|
+
|
|
1101
|
+
The shared command observes a fresh target; Intent binds its decision to the
|
|
1102
|
+
committed observation. Both send an SDK reference and Host-owned action ID.
|
|
1103
|
+
Callers cannot supply `targetRef` or `actionId` in the public payload. Long press
|
|
1104
|
+
requires `durationMs:500..10000`; swipe requires `durationMs:1..10000` and numeric
|
|
1105
|
+
`deltaX/deltaY` relative to the selected node's current center. The endpoint must
|
|
1106
|
+
remain in the current window. Scroll uses the selected container's visible area,
|
|
1107
|
+
with `durationMs` defaulting to 400; a container that cannot move in the requested
|
|
1108
|
+
direction returns `native_scroll_boundary` before DOWN. Rounded zero movement
|
|
1109
|
+
and out-of-window endpoints also reject before DOWN. Public `swipe` remains the
|
|
1110
|
+
separate ADB device-coordinate primitive.
|
|
1111
|
+
|
|
1112
|
+
The SDK checks the reference immediately before DOWN, then runs a timed stream
|
|
1113
|
+
on the UI handler so actual long-click callbacks and frames can run. Android
|
|
1114
|
+
owns routing inside the original window until UP/CANCEL; moving/recycling list
|
|
1115
|
+
children does not cause semantic retargeting during the stream. Window/focus loss
|
|
1116
|
+
sends CANCEL to that original window and reports `native_gesture_window_changed`.
|
|
1117
|
+
This can occur after a long-click has already opened a dialog; inspect the result.
|
|
1118
|
+
|
|
1119
|
+
After Host cancellation or response loss, Host sends one bounded cancellation to
|
|
1120
|
+
the same forwarded endpoint with the exact action ID and runtime epoch, outside
|
|
1121
|
+
the aborted execution scope. It waits for that reply, never retries the gesture,
|
|
1122
|
+
and preserves uncertainty if acknowledgement is missing or invalid. The SDK
|
|
1123
|
+
keeps a blocked active touch reserved until CANCEL actually runs. Native tap/input,
|
|
1124
|
+
Flutter action, H5 eval and runtime clear-data reject `native_action_busy` during
|
|
1125
|
+
that interval. This SDK reservation does not arbitrate external device inputs.
|
|
1126
|
+
Queued cancellation proves `dispatched:false`; blocked delivery may return
|
|
1127
|
+
`dispatched:null, ambiguous:true`. CANCEL stops the stream, not prior App effects.
|
|
1128
|
+
|
|
1129
|
+
Gesture receipts include actual duration, event count, start/end coordinates and
|
|
1130
|
+
completion. Started/terminal `ui.interaction` events carry the action ID. A capture
|
|
1131
|
+
callback failure is exposed as `evidenceError` separately from touch delivery;
|
|
1132
|
+
it cannot trigger another terminal touch. Arbitrary asynchronous App callbacks
|
|
1133
|
+
are not all causally tagged. Verify fresh phone events and independent App state.
|
|
1134
|
+
|
|
1135
|
+
`wait-text` accepts `timeoutMs` (default 10000), `intervalMs` (default 500),
|
|
1136
|
+
`provider:auto|native|flutter|uia`, optional `targetText`, `requireText` and
|
|
1137
|
+
`absentText` arrays, and `requireActivity` (full class name). All text matches are
|
|
1138
|
+
exact and come from one fresh visible provider tree; metadata, hidden windows
|
|
1139
|
+
and text from another provider cannot complete a condition. For example:
|
|
1140
|
+
|
|
1141
|
+
```json
|
|
1142
|
+
{"command":"wait-text","arguments":{"serial":"DEVICE","packageName":"com.example.app","provider":"native","targetText":"Save, draft","requireText":["Editor"],"absentText":["Loading"],"timeoutMs":5000}}
|
|
1143
|
+
```
|
|
1144
|
+
|
|
1145
|
+
At least one condition is required. Pure absence or Activity-only waits require
|
|
1146
|
+
an explicit provider. An unreadable/invalid tree returns `observation_unavailable`,
|
|
1147
|
+
never successful absence. A deadline returns `deadline_exceeded`; the last
|
|
1148
|
+
available check reports missing/unexpected labels. Cancellation remains
|
|
1149
|
+
`cancelled`. `timeoutSec` and CSV strings are rejected for this command. In the
|
|
1150
|
+
CLI, pass JSON arrays to `--require-text`/`--absent-text`; commas remain part of the label.
|
|
1151
|
+
The H5 wait contract is described below; other platform waits remain separate.
|
|
1152
|
+
|
|
1153
|
+
## Android H5 execution and DOM operations
|
|
1154
|
+
|
|
1155
|
+
Android Intent supports `provider: "h5"` through the same SDK and typed commands
|
|
1156
|
+
used by CLI, MCP and Script. `intent start/observe` accepts
|
|
1157
|
+
`observationTarget: { webViewId }`; `null` requires exactly one visible WebView.
|
|
1158
|
+
Ambiguity returns candidate IDs. An explicit ID never selects a replacement.
|
|
1159
|
+
|
|
1160
|
+
`h5-dom` returns `h5TargetSchema: "aab.android-h5-target/v1"`, `pageRef` and
|
|
1161
|
+
`dom.controls`. The page reference binds the runtime epoch, package, process,
|
|
1162
|
+
Activity, foreground window, WebView, document generation and URL. Each control
|
|
1163
|
+
has an `elementId` tied to that DOM object. Navigation, including history return
|
|
1164
|
+
to the same URL, and replacing an element invalidate the original action.
|
|
1165
|
+
|
|
1166
|
+
`h5-click`, `h5-input`, `h5-scroll` and `h5-wait` use an object `selector` with
|
|
1167
|
+
exactly one of `elementId`, exact `text` or exact `ariaLabel`, optionally qualified
|
|
1168
|
+
by `tag`. CSS strings and the old `targetText`/`exact` aliases are removed from
|
|
1169
|
+
these Android commands. A truncated snapshot requires an explicit observed
|
|
1170
|
+
`elementId`; duplicate visible matches are rejected. `expectedTarget` on typed
|
|
1171
|
+
mutations binds the original `{ pageRef, element }`. Intent supplies this binding
|
|
1172
|
+
automatically from its committed observation.
|
|
1173
|
+
|
|
1174
|
+
`h5-input` uses string `text`, including `""` to clear an editor. It validates
|
|
1175
|
+
editability and rechecks the same element and value after focus callbacks.
|
|
1176
|
+
Controls retain separate label, text and value; sensitive values are redacted.
|
|
1177
|
+
Snapshots contain at most 1000 controls, with the actual count and truncation
|
|
1178
|
+
flag, and at most 20000 body characters. Display text and values are capped at
|
|
1179
|
+
500 characters. These semantic DOM operations do not promise trusted browser
|
|
1180
|
+
input events or business acceptance.
|
|
1181
|
+
|
|
1182
|
+
Click/input first obtain a DOM action point, then check Android clipping and
|
|
1183
|
+
native occlusion. The final renderer turn must still match the original element,
|
|
1184
|
+
document and geometry. Standard Android WebView coordinates account for scale
|
|
1185
|
+
and scroll offsets. Non-standard WebView geometry and transformed native views
|
|
1186
|
+
return explicit unsupported errors. Snapshot adapters may still observe them;
|
|
1187
|
+
there is no automatic provider fallback. Frame selection remains unimplemented.
|
|
1188
|
+
|
|
1189
|
+
`h5-scroll` accepts either a selector to bring an element into view, or both
|
|
1190
|
+
`deltaX` and `deltaY` to scroll the page in CSS pixels. Mixing these forms or
|
|
1191
|
+
supplying two zero deltas is rejected. Page scrolling can include `expectedPage`.
|
|
1192
|
+
Expert `h5-eval` always requires `expectedPage`; it executes synchronous JavaScript
|
|
1193
|
+
in that observed page. Script does not permit this expert escape hatch.
|
|
1194
|
+
|
|
1195
|
+
Mutations use `/v1/h5/action` and `aab.h5-execution/v1`. The SDK advertises
|
|
1196
|
+
`h5ExecutionSchema`, `h5TargetSchema` and the active `h5Action` in
|
|
1197
|
+
`status.debugBridge`. `/v1/h5/cancel` accepts only the original action ID and
|
|
1198
|
+
runtime epoch. Cancellation before admission prevents submission; cancellation
|
|
1199
|
+
between the read-only geometry probe and final action prevents the mutation.
|
|
1200
|
+
After submission, WebView offers no cancellation acknowledgement: occupancy is
|
|
1201
|
+
retained until the original callback and Java invocation have both completed.
|
|
1202
|
+
Uncertain effects continue to block other device mutations. Native, Flutter and
|
|
1203
|
+
H5 share coordination while retaining their own identities and receipts.
|
|
1204
|
+
|
|
1205
|
+
`h5-wait` only polls observations. It keeps the original WebView ID and retries
|
|
1206
|
+
only a readable snapshot with an absent selector. It uses `timeoutMs` (default
|
|
1207
|
+
10000) and `intervalMs` (default 500), returns `h5_wait_timeout` on local expiry,
|
|
1208
|
+
and respects the encompassing Host deadline and cancellation. It has no mutation
|
|
1209
|
+
action ID. Missing WebViews, ambiguity and protocol errors return immediately.
|
|
1210
|
+
|
|
1211
|
+
Flutter H5 uses an explicit adapter contract. Registration requires
|
|
1212
|
+
`isVisible: bool Function()` backed by the App's actual route/widget state;
|
|
1213
|
+
it does not infer native occlusion. `flutter-h5-dom` may omit `adapterId` only
|
|
1214
|
+
when exactly one registered adapter is visible. Multiple visible adapters require
|
|
1215
|
+
an explicit ID. Duplicate registration is rejected; unregister before replacing
|
|
1216
|
+
an ID, and discard its old page references.
|
|
1217
|
+
|
|
1218
|
+
`pageRef` uses `aab.flutter-h5-target/v1` and binds `runtimeEpoch`, `adapterId`,
|
|
1219
|
+
`adapterGeneration`, `documentId` and `url`. `flutter-h5-click/input/wait` use
|
|
1220
|
+
`selector:{elementId|text|ariaLabel,tag?}`; CSS strings, `targetText` and `exact`
|
|
1221
|
+
are removed. Input uses string `text`, including an empty string. Typed mutations
|
|
1222
|
+
may carry `expectedTarget:{pageRef,element}` from the observation. Scroll accepts
|
|
1223
|
+
either a selector or both `deltaX`/`deltaY`; page scrolling may bind `expectedPage`.
|
|
1224
|
+
Expert eval requires `expectedPage` and retains managed original completion and
|
|
1225
|
+
cancellation. Wait only observes the same adapter and document; navigation or
|
|
1226
|
+
replacement fails instead of selecting another document. The shared renderer
|
|
1227
|
+
checks DOM occlusion; platform/App visibility and business acceptance remain
|
|
1228
|
+
separate evidence requirements.
|
|
1229
|
+
|
|
1230
|
+
Android and iOS share the generated DOM identity renderer
|
|
1231
|
+
in `shared/h5/renderer.js`; native window/geometry checks remain platform-owned.
|
|
1232
|
+
Regenerate with `python3 shared/h5/generate.py` and check mirrors with `--check`.
|
|
1233
|
+
|
|
1234
|
+
## Web DOM, Intent and Script
|
|
1235
|
+
|
|
1236
|
+
Web commands run on the shared runtime that owns the browser's SDK
|
|
1237
|
+
connection, accessible from both CLI and MCP. Select a live `sessionId`, `runtimeEpoch` and `targetId: "main"`
|
|
1238
|
+
from `web-sessions`. A Web Intent target adds `platform: "web"` and uses
|
|
1239
|
+
`provider: "h5"`; other providers are rejected. Script supports the catalog's
|
|
1240
|
+
typed Web commands with the same target. Neither entrypoint creates a second
|
|
1241
|
+
provider or reconnects a different page behind the original target.
|
|
1242
|
+
|
|
1243
|
+
`web-dom` returns `webTargetSchema: "aab.web-dom-target/v1"`, `pageRef` and
|
|
1244
|
+
`dom.controls`. The page reference contains the schema, session, runtime, target,
|
|
1245
|
+
navigation ID and URL. Navigation IDs change even when a page returns to the same
|
|
1246
|
+
URL. Each control has an `elementId` tied to the actual DOM element in that
|
|
1247
|
+
document; replacing a node with identical HTML creates a different identity.
|
|
1248
|
+
|
|
1249
|
+
Every control includes `checked: true | false | "mixed" | null` and the raw
|
|
1250
|
+
`ariaChecked` attribute (`string | null`). Native checkbox/radio state comes
|
|
1251
|
+
from its live property, with an indeterminate checkbox reported as `"mixed"`.
|
|
1252
|
+
Supported ARIA controls use their declared state; missing, invalid or unsupported
|
|
1253
|
+
state remains `null`. For `radio`, `menuitemradio` and `switch`, an ARIA `mixed`
|
|
1254
|
+
value means false under [WAI-ARIA 1.2](https://www.w3.org/TR/wai-aria-1.2/#aria-checked);
|
|
1255
|
+
the raw attribute is still preserved. Checkbox/menuitemcheckbox retain `"mixed"`.
|
|
1256
|
+
Intent summaries preserve these checkable roles and state, including textless
|
|
1257
|
+
controls. A hidden companion input and its visible custom checkbox remain
|
|
1258
|
+
separate nodes with separate interaction readiness.
|
|
1259
|
+
|
|
1260
|
+
Use the matching current Web SDK: a snapshot without `checked`, or with a value
|
|
1261
|
+
outside this contract, fails as `invalid_web_dom_control`. The Host does not
|
|
1262
|
+
substitute false for missing state. Checked state is observation data, not part
|
|
1263
|
+
of element identity. `web-click` remains one click, not an idempotent set-state
|
|
1264
|
+
operation; reobserve and verify the business result after it.
|
|
1265
|
+
|
|
1266
|
+
Typed actions take an object `selector` with exactly one identity field:
|
|
1267
|
+
`elementId`, `text`, `ariaLabel` or `css`, optionally restricted by `tag`/`role`.
|
|
1268
|
+
They require one visible match. String selectors, mixed identity fields and
|
|
1269
|
+
ambiguous text matches are rejected. `web-dom.selector` is a CSS string for a
|
|
1270
|
+
bounded read; it is a different argument contract from an action selector.
|
|
1271
|
+
Intent decisions select observed controls by `elementId`, `text` or `ariaLabel`;
|
|
1272
|
+
they do not run an unobserved CSS query. A truncated Intent snapshot is rejected
|
|
1273
|
+
for action selection, rather than assuming the missing controls cannot match.
|
|
1274
|
+
|
|
1275
|
+
An action may bind its original observation using
|
|
1276
|
+
`expectedTarget: { pageRef, element }`. Copy the element fields `elementId`,
|
|
1277
|
+
`tag`, `id`, `name`, `type`, `role`, `ariaLabel`, `placeholder`, `href` and `text`
|
|
1278
|
+
from that observation. Intent supplies this binding automatically. The SDK
|
|
1279
|
+
rechecks the original page and element before dispatch, including after focus
|
|
1280
|
+
callbacks. A changed page or element requires a new observation; it never selects
|
|
1281
|
+
a replacement using the old text or CSS selector.
|
|
1282
|
+
|
|
1283
|
+
| Command | Behavior |
|
|
1284
|
+
| --- | --- |
|
|
1285
|
+
| `web-click` | Invoke one semantic DOM click on the unique visible, enabled, unobscured control. |
|
|
1286
|
+
| `web-input` | Replace `value` (a string, including empty) in a supported text input, textarea or contenteditable. Native form setters and browser input events reach framework handlers; rich-text insertion uses the browser editing path. |
|
|
1287
|
+
| `web-key` | Deliver one bubbling, cancellable `keydown` for `Enter` or `Escape` to the selected control. The receipt reports `trusted: false` and `defaultPrevented`; keyup, trusted keyboard defaults and general shortcuts are not promised. |
|
|
1288
|
+
| `web-scroll` | `mode: "into-view"` requires a selector and no deltas. `mode: "by"` requires at least one explicit `deltaX`/`deltaY`, with an optional element selector. |
|
|
1289
|
+
| `web-wait` | Wait for exactly one of a typed selector or body `targetText`, within the command deadline. |
|
|
1290
|
+
|
|
1291
|
+
DOM observations include `visible`, `disabled`, `editable` and
|
|
1292
|
+
`interaction.status`: `ready`, `hidden`, `disabled`, `outside-viewport` or
|
|
1293
|
+
`obscured`. A ready/obscured observation includes the checked point; an obscured
|
|
1294
|
+
one includes the hit element's diagnostic identity. Interaction state explains
|
|
1295
|
+
the observation and is checked again at execution. Errors retain their original
|
|
1296
|
+
receipt and dispatch status, including `web_element_ambiguous`,
|
|
1297
|
+
`web_element_changed`, `web_element_obscured`, `web_element_outside_viewport`
|
|
1298
|
+
and `reobserve_required`. Successful dispatch does not establish asynchronous
|
|
1299
|
+
business completion; observe and assert the actual resulting state.
|
|
1300
|
+
|
|
1301
|
+
Script admits `web-status`/`web-dom` under `app.read` and these typed controls
|
|
1302
|
+
under `app.interact`. Raw `web-command` remains outside its catalog. DOM reads
|
|
1303
|
+
produce Host tree references for device assertions. Ordinary
|
|
1304
|
+
`web-logs/network/state/events` query committed Host FactStore records. They are
|
|
1305
|
+
available under Script `capture.read` and Intent `require.streams`.
|
|
1306
|
+
The default `view: "decision-window"` requires the exact connected document.
|
|
1307
|
+
Each query obtains an SDK capture barrier; its upper FactStore cursor is frozen
|
|
1308
|
+
when that response reaches the Host, before later frames can enter the window.
|
|
1309
|
+
The `wc1:` watermark and its SDK/Host counters are persisted in FactStore.
|
|
1310
|
+
|
|
1311
|
+
Observe before acting and pass the returned `watermarkCursor` as `factCursor`
|
|
1312
|
+
after the action. `afterActionId` additionally filters explicitly associated
|
|
1313
|
+
records and requires that lower watermark. In Intent requirements, omitting
|
|
1314
|
+
`afterActionId` selects the current action; explicit `null` retains all records
|
|
1315
|
+
in the temporal window. Repeated observations of one action keep its original
|
|
1316
|
+
lower watermark so a delayed response remains observable. This temporal window
|
|
1317
|
+
does not assert that every included request was caused by the action.
|
|
1318
|
+
|
|
1319
|
+
`coverage.scope: "sdk-captures-through-barrier"` describes recorded SDK captures.
|
|
1320
|
+
Queue loss, sequence gaps, rejected captures and connection changes retain partial
|
|
1321
|
+
coverage; pending requests also prevent complete coverage. The window does not
|
|
1322
|
+
cover uncaptured traffic or future asynchronous work. Each record identifies
|
|
1323
|
+
`association: "explicit" | "synchronous" | "unattributed"`; unknown origins keep
|
|
1324
|
+
`actionId: null`. Request identity is captured when a request starts, never taken
|
|
1325
|
+
from the action active when its response arrives. Script dispatch now supplies
|
|
1326
|
+
the same original action ID to the provider, receipt and synchronous UI event.
|
|
1327
|
+
|
|
1328
|
+
Pages scan at most 16 records. Keep the original `factCursor`, filters and returned
|
|
1329
|
+
`throughCursor` with `nextCursor` for continuation. The upper bound stays fixed.
|
|
1330
|
+
A continuation page alone cannot pass a whole-window Script assertion; automatic
|
|
1331
|
+
aggregation of evidence across pages remains open. `history: true` or
|
|
1332
|
+
`view: "connected-history"` reads retained Host records, including while offline,
|
|
1333
|
+
with partial `retained-host-history` coverage. History forbids `factCursor` and
|
|
1334
|
+
`afterActionId`; explicit `history: true` conflicts with `view: "decision-window"`.
|
|
1335
|
+
|
|
1336
|
+
Optional request/response bodies carry `BodyState` and `BodyEncoding` fields.
|
|
1337
|
+
`utf8` bodies contain text; `base64` bodies preserve binary protocol bytes. Decode
|
|
1338
|
+
only complete bodies using the declared encoding and the App's protocol schema.
|
|
1339
|
+
Missing, disabled, oversized or timed-out bodies cannot prove submitted content.
|
|
1340
|
+
Full payload exports preserve Web Host references, document identity, action
|
|
1341
|
+
association and barrier diagnostics, and can be checked offline.
|
|
1342
|
+
|
|
1343
|
+
A browser harness screenshot is a separate artifact, not a Script-owned
|
|
1344
|
+
screenshot reference or an archive attachment by default. See [Script authoring](SCRIPT_AUTHORING.md) and the
|
|
1345
|
+
[Memos real-App result](../../../docs/WEB_MEMOS_BUSINESS_2026-09-11.md) for the
|
|
1346
|
+
implemented scope and remaining evidence gaps.
|
|
1347
|
+
|
|
1348
|
+
## Errors and remaining work
|
|
1349
|
+
|
|
1350
|
+
Ordinary Android commands share one Host deadline from queue entry through
|
|
1351
|
+
provider calls and optional feedback. `timeoutMs` defaults to 30000; a timed
|
|
1352
|
+
logcat collection defaults to its requested duration plus 1000 ms. An explicit
|
|
1353
|
+
`adbTimeoutMs` caps each subprocess within that same deadline. Nested calls cannot
|
|
1354
|
+
extend the deadline. ADB/HTTP work and local polling stop cooperatively, and local
|
|
1355
|
+
subprocesses/sockets close before the call settles. `deadline_exceeded` before an
|
|
1356
|
+
action has `dispatched:false`; after submission it retains `ambiguous:true`.
|
|
1357
|
+
This does not undo work already accepted by Android or the App SDK. Install and
|
|
1358
|
+
permission Intent workflows retain the separate lifecycle rules below; iOS/Web
|
|
1359
|
+
provider cancellation still needs its own redesign.
|
|
1360
|
+
|
|
1361
|
+
Common command failures use `ok:false`, a stable string `error`, human `message`,
|
|
1362
|
+
optional `field`/`details`, and `dispatched`/`ambiguous`. `dispatched:null` means
|
|
1363
|
+
unknown. Argument rejections have `dispatched:false, ambiguous:false`. MCP
|
|
1364
|
+
`isError` follows command failure; CLI returns exit code 1. Runtime operations
|
|
1365
|
+
retain their own status/control and evidence envelopes; a completed Script may
|
|
1366
|
+
contain failed or inconclusive assertions.
|
|
1367
|
+
|
|
1368
|
+
The implemented contracts above define the supported command scope. Frame
|
|
1369
|
+
selection and merged multi-page strong assertions remain unsupported. Platform
|
|
1370
|
+
capability, execution completion and acceptance of a particular App workflow
|
|
1371
|
+
must be assessed separately from its retained evidence.
|
|
1372
|
+
|
|
1373
|
+
## Ordinary Intent lifetime and recovery
|
|
1374
|
+
|
|
1375
|
+
`intent start.timeoutMs` is a total lifetime budget (default 300000 ms). It starts
|
|
1376
|
+
before the first observation and includes decision waits, paused time, Agent
|
|
1377
|
+
replies, subsequent observations and actions. An autonomous budget can shorten
|
|
1378
|
+
this deadline. Nested provider work cannot extend it. Pause invalidates the
|
|
1379
|
+
pending Agent reply; resume obtains a new decision. Agent adapters receive an
|
|
1380
|
+
`AbortSignal`; late replies cannot dispatch through the stopped Intent. The
|
|
1381
|
+
adapter must cooperate to stop any external computation it owns.
|
|
1382
|
+
|
|
1383
|
+
Cancel first enters `cancelling`, aborts owned provider work and waits for that
|
|
1384
|
+
work and its evidence writes. Timeout similarly stops new dispatch with
|
|
1385
|
+
`deadline_exceeded`. Other terminal outcomes pass through `finishing`. A terminal
|
|
1386
|
+
checkpoint is committed after the pending work settles and before the terminal
|
|
1387
|
+
status is exposed. Final persistence can extend beyond the execution deadline.
|
|
1388
|
+
`blocked_evidence_store` means this commit failed; `terminalEvidenceId` is absent.
|
|
1389
|
+
`pendingOperations` counts owned execution calls, not the final checkpoint write.
|
|
1390
|
+
Cancel cannot rewrite an already settled outcome. An accepted but unconfirmed
|
|
1391
|
+
action retains `lastAction.dispatched` and `lastAction.ambiguous`; cancellation
|
|
1392
|
+
does not imply rollback. Complete/fail/inconclusive decisions also require the
|
|
1393
|
+
current `basedOnRevision`.
|
|
1394
|
+
|
|
1395
|
+
After execution-runtime restart, `intent status` reads checksum-verified Host evidence and
|
|
1396
|
+
returns `live:false, recovered:true, restartPolicy:"none"`. A durable terminal
|
|
1397
|
+
checkpoint restores the outcome. Retained evidence without a terminal checkpoint
|
|
1398
|
+
returns `interrupted/runtime_restarted`, including unmatched action markers.
|
|
1399
|
+
There is no automatic replay, and a retained operation ID cannot be reused.
|
|
1400
|
+
Store read failures are errors, not an empty history. Explicit runtime stop or a
|
|
1401
|
+
signal to the runtime closes Intent intake, drains pending start/decide work and
|
|
1402
|
+
persists terminal evidence before closing Host storage. CLI exit and MCP EOF
|
|
1403
|
+
only disconnect the client. Mobile capture queries continue to use the live mobile
|
|
1404
|
+
store; this recovery contract applies to Host execution evidence.
|
|
1405
|
+
|
|
1406
|
+
## Installation is an Intent operation
|
|
1407
|
+
|
|
1408
|
+
`install-apk` starts an ordinary supervised Intent operation through CLI or MCP. It
|
|
1409
|
+
returns `command:"intent"`, an `operationId` and `installation` status. Supply
|
|
1410
|
+
`serial` and `apkPath`; optional `packageName` must match the APK manifest.
|
|
1411
|
+
Android SDK `aapt` and `apksigner` are required, found under `ANDROID_SDK_ROOT`
|
|
1412
|
+
or supplied as `aaptPath`/`apksignerPath`.
|
|
1413
|
+
|
|
1414
|
+
The Host freezes and inspects the APK, stages those exact bytes on the phone,
|
|
1415
|
+
and commits the original action/job/boot identity before launching one detached
|
|
1416
|
+
installation worker. The worker rechecks the APK SHA-256, creates one non-staged
|
|
1417
|
+
PackageInstaller session, writes the base APK, and commits that session. The
|
|
1418
|
+
session ID and original PackageManager CLI response are retained on the phone.
|
|
1419
|
+
`allowDowngrade` is opt-in. `streaming` has been removed and is rejected.
|
|
1420
|
+
|
|
1421
|
+
`timeoutMs` bounds device staging, admission and Host waiting (default 180000).
|
|
1422
|
+
APK inspection, initial/final installed-identity reads and UI observation still
|
|
1423
|
+
have separate bounded calls; this is not yet one end-to-end deadline. Inspectors
|
|
1424
|
+
accept verified signer output from `Signer #N` and SDK `V1`, `V2`, `V3`, `V3.0`,
|
|
1425
|
+
`V3.1`, `V4` signer formats; unsupported formats reject without submitting an APK.
|
|
1426
|
+
|
|
1427
|
+
When system interaction is needed, the calling Agent uses `intent observe`, reads
|
|
1428
|
+
the actual foreground tree, then submits `intent decide` with its revision and
|
|
1429
|
+
one exact selector. The adapter has no ROM package table, positive-button labels
|
|
1430
|
+
or automatic click loop. Observation and decision may need to repeat as the
|
|
1431
|
+
system page changes. Without an Agent decision, no system button is clicked.
|
|
1432
|
+
The operation holds the phone's mutation lease through package verification.
|
|
1433
|
+
|
|
1434
|
+
For example, after receiving a fresh observation:
|
|
1435
|
+
|
|
1436
|
+
```json
|
|
1437
|
+
{"command":"intent","arguments":{"operation":"decide","operationId":"FROM_START","decision":{"decisionId":"choice-1","basedOnRevision":2,"agentDecision":"act","action":{"action":"tap","selector":{"resourceName":"ID_FROM_ACTUAL_OBSERVATION"}}}}}
|
|
1438
|
+
```
|
|
1439
|
+
|
|
1440
|
+
Completion requires the matching original shell-job receipt, a successful
|
|
1441
|
+
PackageInstaller commit response, and an independent `pm path`/`sha256sum` check
|
|
1442
|
+
of the installed bytes. The proof binds actionId, jobId, Android boot, command
|
|
1443
|
+
hash, admission deadline, installId, package and APK hash, and identifies the
|
|
1444
|
+
PackageInstaller session. The install receipt response hashes use canonical JSON
|
|
1445
|
+
with sorted object keys, so archive serialization preserves their verification.
|
|
1446
|
+
A matching old APK, a local ADB exit, a missing session
|
|
1447
|
+
or a mismatched receipt cannot settle this request. Caller `complete` decisions
|
|
1448
|
+
cannot override verification. Split APK layouts are explicitly unsupported.
|
|
1449
|
+
|
|
1450
|
+
The supported PackageManager CLI protocol recognizes exact `Success` and final
|
|
1451
|
+
`INSTALL_FAILED_*` / `INSTALL_PARSE_FAILED_*` failure responses. This is parsing
|
|
1452
|
+
machine command output, not recognizing installer button text. Pending-user-action,
|
|
1453
|
+
warnings, unfamiliar OEM output, truncated output, process death and changed boot
|
|
1454
|
+
remain unresolved. A terminal install rejection releases ownership but fails the
|
|
1455
|
+
workflow, even when the old APK still matches.
|
|
1456
|
+
|
|
1457
|
+
`intent cancel` stops Host waiting and UI decisions. It can prevent admission;
|
|
1458
|
+
once the phone worker is admitted it drains the original request, without a
|
|
1459
|
+
second install or a rollback promise. If its original result is unavailable,
|
|
1460
|
+
physical-device ownership remains blocked after cancellation, timeout, EOF or
|
|
1461
|
+
Host death. `device-ownership reconcile` reads that same job and response; it
|
|
1462
|
+
never repeats the install. It does not resume the lost Intent decision loop.
|
|
1463
|
+
A still-waiting PackageInstaller interaction must finish before it can provide
|
|
1464
|
+
completion proof. `pause` only suspends UI decisions.
|
|
1465
|
+
|
|
1466
|
+
The phone staging area admits at most 64 retained installations. APK staging is
|
|
1467
|
+
removed and the shell receipt acknowledged only after a known result is settled;
|
|
1468
|
+
normal completion and cross-Host recovery persist the exact ownership proof first.
|
|
1469
|
+
Cleanup failure is reported separately and cannot erase a known execution result.
|
|
1470
|
+
Unknown/orphaned staging, eviction/reboot recovery, warning/OEM variants, active
|
|
1471
|
+
PackageInstaller cancellation and multi-ROM verification remain separate gates.
|
|
1472
|
+
|
|
1473
|
+
## Ordinary MCP command history
|
|
1474
|
+
|
|
1475
|
+
Ordinary command execution and auxiliary history persistence have separate
|
|
1476
|
+
outcomes. `ok`, dispatch state and `executionReceipt` describe the original
|
|
1477
|
+
provider operation. A history failure never retries or rewrites that operation.
|
|
1478
|
+
Intent/Script required evidence commits retain their existing strict admission
|
|
1479
|
+
and terminal-evidence contracts.
|
|
1480
|
+
|
|
1481
|
+
Object results carry `_history` with schema `aab.command-history/v1`, including
|
|
1482
|
+
when `feedback:"off"`. Every ordinary CLI/MCP result also carries the same value in
|
|
1483
|
+
`_meta["ai-app-bridge/history"]`, so raw text/image-result consumers can inspect
|
|
1484
|
+
history without rewriting the original content. This is MCP history metadata;
|
|
1485
|
+
CLI and MCP use the same history store; direct internal provider calls do not acquire an auxiliary one.
|
|
1486
|
+
|
|
1487
|
+
- `stored`: this invocation's action and returned Host evidence references were
|
|
1488
|
+
committed. It is not a claim of complete mobile history or business correctness.
|
|
1489
|
+
- `partial`: one or more writes failed. `action` and `evidence` identify which
|
|
1490
|
+
references were stored; `errors` preserves their failure codes.
|
|
1491
|
+
- `unavailable`: history initialization or recording setup failed.
|
|
1492
|
+
- `disabled`: auxiliary history was explicitly disabled or no recorder was supplied.
|
|
1493
|
+
|
|
1494
|
+
A failed action reference has `stored:false`; unavailable `globalSeq` is `null`.
|
|
1495
|
+
`replayed:false` makes the no-retry rule explicit. Use read-only verification of
|
|
1496
|
+
the original action when history is partial/unavailable. Do not repeat a mutation
|
|
1497
|
+
just to obtain a stored reference. A failed evidence append is not added to the
|
|
1498
|
+
in-process dedupe set, so a later observation may persist the same evidence.
|
|
1499
|
+
Repeated request IDs reuse execution without appending references to the cached
|
|
1500
|
+
feedback object; each delivery reports its own history-write attempt.
|
|
1501
|
+
Background observer health remains separate from this foreground write report.
|
|
1502
|
+
|
|
1503
|
+
## Runtime permission requests use Intent
|
|
1504
|
+
|
|
1505
|
+
Trigger the runtime permission request through the App's existing flow, then call:
|
|
1506
|
+
|
|
1507
|
+
```json
|
|
1508
|
+
{"command":"permission-dialog","arguments":{"serial":"DEVICE","packageName":"com.example.app","permission":"android.permission.RECORD_AUDIO","outcome":"allow-once"}}
|
|
1509
|
+
```
|
|
1510
|
+
|
|
1511
|
+
This MCP entry returns an ordinary Intent ID, the actual UI summary, and
|
|
1512
|
+
`permissionDialog`. Choose an exact selector from that summary through `intent
|
|
1513
|
+
decide`. The Host has no permission-button label or resource-ID table. The
|
|
1514
|
+
request's App, Android user, UID and Activity instance must still match at input.
|
|
1515
|
+
The UIA node runtime revalidates the saved reference and current selector;
|
|
1516
|
+
changed text, window identity or duplicate matches stop dispatch. Android ActivityManager currently exposes the requesting App,
|
|
1517
|
+
not the requested permission list. The Agent must match the dialog's meaning to
|
|
1518
|
+
`permission`; the independent PackageManager query verifies that named permission.
|
|
1519
|
+
|
|
1520
|
+
| outcome | Required result after an owned action and original Activity closure |
|
|
1521
|
+
| --- | --- |
|
|
1522
|
+
| `allow` | `granted:true`, `ONE_TIME` absent |
|
|
1523
|
+
| `allow-once` | `granted:true`, `ONE_TIME` present |
|
|
1524
|
+
| `deny` | `granted:false`, `USER_SET` present |
|
|
1525
|
+
| `dismiss` | Grant and all permission flags unchanged from the initial state |
|
|
1526
|
+
|
|
1527
|
+
`deny` accepts only an observed tap. Its state check does not distinguish first
|
|
1528
|
+
refusal from "don't ask again"; the selected visible control is retained in the
|
|
1529
|
+
receipt. `dismiss` accepts an observed tap or `{ "action":"back" }`. A named
|
|
1530
|
+
permission alone does not verify every permission in a grouped request, an app-op,
|
|
1531
|
+
or background access. These require their own observations and state assertions.
|
|
1532
|
+
|
|
1533
|
+
When Android has closed the window but is still retiring its Activity record,
|
|
1534
|
+
`status:waiting_for_observation`, `permissionDialog.pending:activity_closure`
|
|
1535
|
+
means verification is pending. Use `intent observe` to refresh the proof; no
|
|
1536
|
+
second button click is required. Missing/changed evidence never becomes success.
|
|
1537
|
+
Caller `complete` cannot override verification. Export the Intent's plan, UI,
|
|
1538
|
+
decision, receipt and final permission checkpoint with `evidence export`.
|
|
1539
|
+
|
|
1540
|
+
`intent cancel` stops further dispatch, waits for any submitted input to settle,
|
|
1541
|
+
then records the actual state. It does not close the dialog or undo an Android
|
|
1542
|
+
grant. `timeoutMs` defaults to 60000: after that deadline no new input is sent;
|
|
1543
|
+
in-flight bounded ADB calls and final verification can finish later.
|
|
1544
|
+
`adbTimeoutMs` bounds individual ADB calls (default 15000). Pause suspends Agent
|
|
1545
|
+
input while the deadline continues. EOF and signal shutdown finalize active
|
|
1546
|
+
workflows before closing FactStore. Ownership is still within one Host process;
|
|
1547
|
+
external input and abrupt process death are not an exactly-once guarantee.
|
|
1548
|
+
|
|
1549
|
+
The requester probe currently requires one top-resumed Activity and its detailed
|
|
1550
|
+
ActivityManager record. Unsupported/multi-display layouts return explicit errors;
|
|
1551
|
+
there is no ROM-name fallback. These Android contracts do not imply iOS parity.
|
|
1552
|
+
|
|
1553
|
+
`permission-state`, `permission-grant` and `permission-revoke` remain direct
|
|
1554
|
+
CLI/MCP capabilities and Script calls. All require `serial`, `packageName` and
|
|
1555
|
+
`permission`; optional `userId` selects an Android user, otherwise the actual
|
|
1556
|
+
current user is resolved and frozen. They read the runtime-permission section of
|
|
1557
|
+
that exact package/user, never a Host cache or SDK port. `grant`/`revoke` verify the
|
|
1558
|
+
result after one `pm` submission and retain the before state. Failure distinguishes
|
|
1559
|
+
missing runtime permission, unavailable query, rejected change and unverified
|
|
1560
|
+
result. OEM restrictions may reject the shell identity with `permission_change_denied`; the result includes the platform reason and the independent before/after states. No automatic root or UI fallback is attempted. Script requires `app.permissions` for these fixture mutations and
|
|
1561
|
+
`app.read` for queries. `permission-dialog` is a supervised operation and is not
|
|
1562
|
+
a synchronous Script primitive; fixed regression can replay known selectors with
|
|
1563
|
+
explicit state assertions, or request Agent help through `ctx.askAgent`.
|