@mobileaidev/ai-app-bridge 0.2.15 → 0.3.0-rc.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +201 -0
- package/NOTICE +7 -0
- package/README.md +306 -53
- package/bin/ai-app-bridge.js +51 -4504
- package/bin/android-permissions.js +152 -0
- package/bin/android-uia-xml.js +74 -0
- package/bin/artifact-paths.js +127 -1
- package/bin/bridge-forward.js +56 -0
- package/bin/command-discovery.js +86 -0
- package/bin/command-errors.js +60 -0
- package/bin/command-registry.js +504 -0
- package/bin/command-request.js +14 -0
- package/bin/command-router.js +80 -0
- package/bin/connection-cache.js +119 -0
- package/bin/device-provider.js +2934 -0
- package/bin/execution-host.js +596 -0
- package/bin/execution-runtime.js +101 -0
- package/bin/fact-codec.js +321 -0
- package/bin/fact-recorder.js +691 -0
- package/bin/fact-store.js +226 -0
- package/bin/feedback-probe.js +285 -0
- package/bin/intent/install-intent.js +267 -0
- package/bin/intent/intent-action-executor.js +129 -0
- package/bin/intent/intent-autonomous-adapter.js +46 -0
- package/bin/intent/intent-capture-port.js +64 -0
- package/bin/intent/intent-entry.js +226 -0
- package/bin/intent/intent-errors.js +28 -0
- package/bin/intent/intent-evidence-store.js +121 -0
- package/bin/intent/intent-lifetime.js +62 -0
- package/bin/intent/intent-observation-target.js +33 -0
- package/bin/intent/intent-observer.js +194 -0
- package/bin/intent/intent-production-adapter.js +334 -0
- package/bin/intent/intent-provider.js +27 -0
- package/bin/intent/intent-runtime.js +63 -0
- package/bin/intent/intent-worker.js +432 -0
- package/bin/intent/ios-intent-adapter.js +90 -0
- package/bin/intent/permission-intent.js +239 -0
- package/bin/intent/web-intent-adapter.js +31 -0
- package/bin/ios-device-outcome.js +47 -0
- package/bin/ios-execution.js +108 -0
- package/bin/ios-provider.js +497 -583
- package/bin/ios-runtime-binding.js +54 -0
- package/bin/ios-wda-execution.js +70 -0
- package/bin/ios-wda-port.js +98 -0
- package/bin/ios-wda-project.js +79 -0
- package/bin/mcp-server.js +83 -1040
- package/bin/mmap-scan-index.js +329 -0
- package/bin/observation-collector.js +862 -0
- package/bin/runtime-client.js +154 -0
- package/bin/runtime-directory.js +107 -0
- package/bin/runtime-protocol.js +36 -0
- package/bin/script/bounded-script-registry.js +103 -0
- package/bin/script/node-runtime-adapter.js +233 -0
- package/bin/script/progress-projector.js +63 -0
- package/bin/script/python-runtime-adapter.js +111 -0
- package/bin/script/rolling-summary.js +134 -0
- package/bin/script/script-agent-port.js +40 -0
- package/bin/script/script-assert.js +95 -0
- package/bin/script/script-capture-port.js +76 -0
- package/bin/script/script-catalog.js +85 -0
- package/bin/script/script-durable-restore.js +195 -0
- package/bin/script/script-entry-code.js +26 -0
- package/bin/script/script-entry-route.js +38 -0
- package/bin/script/script-entry.js +3 -0
- package/bin/script/script-errors.js +29 -0
- package/bin/script/script-evidence-store.js +22 -0
- package/bin/script/script-format-removed.js +26 -0
- package/bin/script/script-host-port.js +397 -0
- package/bin/script/script-ledger.js +64 -0
- package/bin/script/script-result.js +57 -0
- package/bin/script/script-sdk.js +152 -0
- package/bin/script/script-sdk.py +153 -0
- package/bin/script/script-session-channel.js +127 -0
- package/bin/script/script-spec.js +86 -0
- package/bin/script/script-supervisor.js +919 -0
- package/bin/script/templates/checkpoint-reentry.js +13 -0
- package/bin/segment-index.js +481 -0
- package/bin/segmented-fact-store.js +1571 -0
- package/bin/shared-kernel/android-h5-target.js +10 -0
- package/bin/shared-kernel/android-install-execution.js +176 -0
- package/bin/shared-kernel/android-sdk-endpoint.js +42 -0
- package/bin/shared-kernel/android-shell-execution.js +195 -0
- package/bin/shared-kernel/argument-schema.js +117 -0
- package/bin/shared-kernel/canonical-path.js +17 -0
- package/bin/shared-kernel/device-acknowledgements.js +53 -0
- package/bin/shared-kernel/device-completion-history.js +52 -0
- package/bin/shared-kernel/device-mutation-lease.js +219 -0
- package/bin/shared-kernel/device-ownership-recovery.js +95 -0
- package/bin/shared-kernel/device-ownership-store.js +94 -0
- package/bin/shared-kernel/evidence-adapters.js +251 -0
- package/bin/shared-kernel/evidence-archive.js +329 -0
- package/bin/shared-kernel/evidence-recording.js +131 -0
- package/bin/shared-kernel/evidence-schema.js +194 -0
- package/bin/shared-kernel/evidence-store.js +190 -0
- package/bin/shared-kernel/execution-admission.js +22 -0
- package/bin/shared-kernel/execution-contracts.js +171 -0
- package/bin/shared-kernel/execution-io.js +106 -0
- package/bin/shared-kernel/execution-ledger.js +125 -0
- package/bin/shared-kernel/execution-scope.js +87 -0
- package/bin/shared-kernel/execution-target.js +115 -0
- package/bin/shared-kernel/flutter-execution.js +13 -0
- package/bin/shared-kernel/flutter-h5-port.js +60 -0
- package/bin/shared-kernel/flutter-h5-target.js +9 -0
- package/bin/shared-kernel/flutter-target.js +75 -0
- package/bin/shared-kernel/h5-execution.js +11 -0
- package/bin/shared-kernel/h5-target.js +31 -0
- package/bin/shared-kernel/host-fact-store.js +49 -0
- package/bin/shared-kernel/ios-h5-target.js +9 -0
- package/bin/shared-kernel/ios-native-target.js +71 -0
- package/bin/shared-kernel/live-capture-query.js +115 -0
- package/bin/shared-kernel/managed-sdk-execution.js +78 -0
- package/bin/shared-kernel/native-execution.js +13 -0
- package/bin/shared-kernel/native-target.js +156 -0
- package/bin/shared-kernel/provider-command-contracts.js +55 -0
- package/bin/shared-kernel/recorded-payload-archive.js +195 -0
- package/bin/shared-kernel/request-context.js +35 -0
- package/bin/shared-kernel/semantic-node.js +55 -0
- package/bin/shared-kernel/summary-transformer.js +352 -0
- package/bin/shared-kernel/target-lease-protocol.js +47 -0
- package/bin/shared-kernel/text-wait.js +111 -0
- package/bin/shared-kernel/uia-execution.js +96 -0
- package/bin/shared-kernel/uia-protocol.js +214 -0
- package/bin/shared-kernel/uia-runtime-port.js +377 -0
- package/bin/shared-kernel/uia-target.js +39 -0
- package/bin/shared-kernel/web-dom-target.js +44 -0
- package/bin/shared-kernel/xml-attributes.js +25 -0
- package/bin/target-execution.js +275 -0
- package/bin/web/command-schema.js +60 -0
- package/bin/web/session-store.js +157 -0
- package/bin/web-provider.js +334 -553
- package/docs/COMMAND_CONTRACT.md +1563 -0
- package/docs/EVIDENCE_ARCHIVE.md +214 -0
- package/docs/INTENT_FOREGROUND.md +71 -0
- package/docs/INTENT_NATIVE_EDITING.md +79 -0
- package/docs/RELEASE.md +59 -0
- package/docs/SCRIPT_AUTHORING.md +489 -0
- package/node_modules/@mobileaidev/segmented-fact-store-native/LICENSE +201 -0
- package/node_modules/@mobileaidev/segmented-fact-store-native/NOTICE +7 -0
- package/node_modules/@mobileaidev/segmented-fact-store-native/binding.gyp +36 -0
- package/node_modules/@mobileaidev/segmented-fact-store-native/bindings/node/sfs_node.c +597 -0
- package/node_modules/@mobileaidev/segmented-fact-store-native/include/sfs.h +178 -0
- package/node_modules/@mobileaidev/segmented-fact-store-native/index.js +5 -0
- package/node_modules/@mobileaidev/segmented-fact-store-native/package.json +24 -0
- package/node_modules/@mobileaidev/segmented-fact-store-native/src/sfs.c +2349 -0
- package/package.json +60 -5
- package/runtime/ios-wda/AABWDABinding.h +19 -0
- package/runtime/ios-wda/AABWDABinding.m +97 -0
- package/runtime/ios-wda/AABWDAExecution.h +26 -0
- package/runtime/ios-wda/AABWDAExecution.m +172 -0
- package/runtime/ios-wda/AABWDAIntegration.h +71 -0
- package/runtime/ios-wda/AABWDAManagedRoutes.h +392 -0
- package/runtime/ios-wda/AABWDAReceiptStore.h +10 -0
- package/runtime/ios-wda/AABWDAReceiptStore.m +116 -0
- package/runtime/uia/ai-app-bridge-uia.jar +0 -0
- package/runtime/uia/manifest.json +22 -0
- package/skills/ai-app-bridge-use/SKILL.md +23 -360
|
@@ -0,0 +1,214 @@
|
|
|
1
|
+
# Record, export and verify execution evidence
|
|
2
|
+
|
|
3
|
+
Discover the public MCP command with `capabilities {"command":"evidence"}`.
|
|
4
|
+
It works for both Intent operation IDs and Script execution IDs, including
|
|
5
|
+
records retained after the execution runtime has restarted. CLI and MCP use the
|
|
6
|
+
same operation IDs and export command. Export does not require a
|
|
7
|
+
connected phone or a live worker.
|
|
8
|
+
|
|
9
|
+
```json
|
|
10
|
+
{
|
|
11
|
+
"command": "evidence",
|
|
12
|
+
"arguments": {
|
|
13
|
+
"operation": "export",
|
|
14
|
+
"namespace": "intent",
|
|
15
|
+
"operationId": "intent-from-your-response",
|
|
16
|
+
"outputDir": "/absolute/existing-parent/new-archive"
|
|
17
|
+
}
|
|
18
|
+
}
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
Use `namespace: "script"` for a Script operation. Export reads the current
|
|
22
|
+
execution runtime's configured FactStore (`AI_APP_BRIDGE_FACT_STORE_DIR`), drains its
|
|
23
|
+
queued writes, and freezes the retained records for exactly that namespace
|
|
24
|
+
and operation. The parent directory must exist; the output directory must
|
|
25
|
+
not exist. No existing directory is replaced.
|
|
26
|
+
|
|
27
|
+
The response includes `archiveDir`, `manifestPath`, `manifestSha256`,
|
|
28
|
+
`recordCount`, `targets`, and `coverage`. Save the returned manifest SHA256
|
|
29
|
+
separately when handing the archive to an author or reviewer.
|
|
30
|
+
|
|
31
|
+
Without `includeRecordedPayloads`, the directory contains two files:
|
|
32
|
+
|
|
33
|
+
- `records.json`: the original ordered Facts, including their globalSeq,
|
|
34
|
+
outer source binding, full evidence envelopes, raw trees where stored,
|
|
35
|
+
checksums, and original device/app identities.
|
|
36
|
+
- `manifest.json`: schema `aab.evidence-archive/v1`, namespace/operation,
|
|
37
|
+
source store ID and sequence watermark, partition retention metadata,
|
|
38
|
+
file size/SHA256, record counts, target inventory, and coverage.
|
|
39
|
+
|
|
40
|
+
The sequence can have gaps because unrelated operations share the store.
|
|
41
|
+
An expired cursor, unreadable page, checksum error, mismatched binding, or
|
|
42
|
+
snapshot change fails export. An unknown operation returns
|
|
43
|
+
`operation_not_found`. Limits are 10,000 records and 128 MiB of record JSON;
|
|
44
|
+
the manifest is limited to 2 MiB. Exceeding a limit fails rather than
|
|
45
|
+
truncating. The manifest is written last; a directory left by an interrupted
|
|
46
|
+
write is not a successfully verified archive.
|
|
47
|
+
|
|
48
|
+
## Verify without the source store or phone
|
|
49
|
+
|
|
50
|
+
Move or copy the directory as a unit, then call:
|
|
51
|
+
|
|
52
|
+
```json
|
|
53
|
+
{
|
|
54
|
+
"command": "evidence",
|
|
55
|
+
"arguments": {
|
|
56
|
+
"operation": "verify",
|
|
57
|
+
"archiveDir": "/absolute/moved-archive",
|
|
58
|
+
"manifestSha256": "<the 64 lowercase hex characters saved from export>"
|
|
59
|
+
}
|
|
60
|
+
}
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
Verification opens only the archive's ordinary files. It does not open a
|
|
64
|
+
FactStore, initialize an operation, or contact a phone. It verifies the
|
|
65
|
+
externally supplied manifest hash, records hash/size, each evidence checksum,
|
|
66
|
+
namespace/operation/source binding, IDs, sequence, internal reference order
|
|
67
|
+
and binding, and recalculated coverage and target inventory. Archive files cannot be symlinks, and the manifest
|
|
68
|
+
cannot redirect the verifier to another file. A missing or changed file
|
|
69
|
+
fails verification. Source paths embedded in evidence are data and are never
|
|
70
|
+
opened or executed.
|
|
71
|
+
|
|
72
|
+
Success returns `ok: true` and `integrity: "verified"`. The hash establishes
|
|
73
|
+
that the supplied archive matches the frozen export; it is not a signature
|
|
74
|
+
authenticating the producer. Obtain the expected hash from the export
|
|
75
|
+
response, not by hashing an untrusted archive and treating that as approval.
|
|
76
|
+
|
|
77
|
+
## What this archive establishes
|
|
78
|
+
|
|
79
|
+
`coverage.scope` is `retained-host-records`. `priorHistoryComplete` remains
|
|
80
|
+
`unknown`: a store can already have evicted older records before export.
|
|
81
|
+
Partition retention metadata describes the entire partition, not proof that
|
|
82
|
+
this particular operation lost records.
|
|
83
|
+
|
|
84
|
+
`referenceClosure` is `complete` when the included records' checked internal
|
|
85
|
+
references are present, or `partial` with `missingReferences`. This is
|
|
86
|
+
separate from whole-history completeness. Exporting a partial retained set
|
|
87
|
+
can succeed as a diagnostic archive; it does not fill its missing evidence.
|
|
88
|
+
Contradictory references fail: a reference to a future record, a summary with
|
|
89
|
+
a different observation revision, a receipt preceding its dispatch marker,
|
|
90
|
+
or duplicate observation/decision/action identities cannot become a complete
|
|
91
|
+
reference set merely because the IDs exist.
|
|
92
|
+
|
|
93
|
+
The archive preserves already persisted, sanitized values byte-for-value as
|
|
94
|
+
JSON values. It does not change recorded serials or infer missing targets
|
|
95
|
+
from the currently connected phone. `owner` retains the declared context,
|
|
96
|
+
`observed` retains each observation's declared device/App identity, and `foreground`
|
|
97
|
+
retains explicit Intent route data. Script marker targets use `dispatch`
|
|
98
|
+
and can contain explicit target overrides, including another serial when
|
|
99
|
+
the Script supplies it. These roles preserve the current execution contract;
|
|
100
|
+
the archive does not apply a new device allowlist or infer an absent target.
|
|
101
|
+
|
|
102
|
+
New execution envelopes declare `schemaVersion: "aab.execution-evidence/v1"`.
|
|
103
|
+
Their targets use an explicit platform discriminator; observations carry both
|
|
104
|
+
`target` and `observedTarget` when the latter was actually observed. A pure
|
|
105
|
+
Script call can have a null target. Archive verification also understands the
|
|
106
|
+
original unversioned record format already frozen in archive v1/v2. It checks
|
|
107
|
+
those original fields and hashes without adding a platform or rewriting bytes.
|
|
108
|
+
New execution writes do not accept an old observation in place of a platform target.
|
|
109
|
+
|
|
110
|
+
Currently persisted Intent content includes observations with raw trees,
|
|
111
|
+
summaries, decisions, dispatch markers, and action receipts. Script content
|
|
112
|
+
includes durable checkpoints, dispatch markers, and action receipts.
|
|
113
|
+
Completed Scripts also persist their final JSON as a `result` record, with a
|
|
114
|
+
terminal checkpoint `resultRef` binding its evidence ID, canonical byte count,
|
|
115
|
+
persisted SHA256, original SHA256 and `original-json`/`redacted-json`
|
|
116
|
+
representation. Export includes this retained record without requiring
|
|
117
|
+
`includeRecordedPayloads`; it is independent of progress-event retention.
|
|
118
|
+
Verification checks the checkpoint's binding to that result. An evicted result
|
|
119
|
+
leaves partial reference closure rather than a reconstructed value. For a direct
|
|
120
|
+
checked read, use `script {operation:"result",operationId:"…"}`.
|
|
121
|
+
|
|
122
|
+
Without recorded payload inclusion, `externalPayloads` is `not-included`. External screenshot files, phone
|
|
123
|
+
logs/network capture items, individual Script call results/assertions, and in-memory
|
|
124
|
+
events are outside this archive. Existing external references are listed
|
|
125
|
+
but not downloaded or dereferenced. Preserve those artifacts separately
|
|
126
|
+
when the intended check requires them.
|
|
127
|
+
|
|
128
|
+
`executionStatus` is `not-inferred` and `businessVerdict` is `not-evaluated`.
|
|
129
|
+
The caller can inspect recorded terminal decisions/checkpoints, including a
|
|
130
|
+
persisted cancellation. Export verification does not infer missing lifecycle
|
|
131
|
+
state or reconstruct evicted records.
|
|
132
|
+
Archive integrity alone does not establish a passed business assertion.
|
|
133
|
+
|
|
134
|
+
## Explicit recording for one execution
|
|
135
|
+
|
|
136
|
+
Add `recordingDir: "/absolute/existing-parent/new-recording"` to the
|
|
137
|
+
**arguments of `script start` or `intent start`**, alongside `script` or
|
|
138
|
+
`goal`/`target`. The parent must exist and the directory must be new.
|
|
139
|
+
This is opt-in file output for this execution; ordinary live calls do not
|
|
140
|
+
copy mobile payloads to a Host history database.
|
|
141
|
+
|
|
142
|
+
Script records each returned Host call envelope (command, arguments, actual
|
|
143
|
+
result, call/action/observation IDs, source hash, window, coverage, refs and
|
|
144
|
+
capture metadata), plus each assertion input and Host verdict. The Host
|
|
145
|
+
copies referenced screenshot PNG bytes immediately, checking the issued
|
|
146
|
+
SHA256 before an original path can be reused. It awaits the attachment's
|
|
147
|
+
FactStore receipt before replying to the child. This does not depend on the
|
|
148
|
+
child writing files or on retaining the bounded event log.
|
|
149
|
+
|
|
150
|
+
Intent records the complete capture pages already fetched by its `require`
|
|
151
|
+
streams, including items. Each attachment references its persisted raw-tree
|
|
152
|
+
observation. Recording itself does not query extra streams or take Intent
|
|
153
|
+
screenshots. With no `require` streams, Intent has no mobile payload to record.
|
|
154
|
+
|
|
155
|
+
The existing Host FactStore stores small `attachment` records containing
|
|
156
|
+
file hashes and execution bindings; the payload files remain in the explicit
|
|
157
|
+
output directory. The live mobile query path never consults those files.
|
|
158
|
+
Script recording currently requires `restartPolicy: "none"`; a checkpoint
|
|
159
|
+
restart request with recording is rejected as `recording_restart_unsupported`.
|
|
160
|
+
Live pause/resume can continue the same recording. No recording is silently
|
|
161
|
+
reopened after Host or child loss.
|
|
162
|
+
|
|
163
|
+
Limits are 10,000 attachments, 16 MiB per JSON/PNG file and 256 MiB per
|
|
164
|
+
recording. A write, checksum or persistence failure stops further successful
|
|
165
|
+
recording: Script receives an explicit error and blocks further calls;
|
|
166
|
+
Intent stops exposing the new observation. Previously committed material
|
|
167
|
+
can still be exported. Unreferenced files left by a failed write are not
|
|
168
|
+
included. Existing output directories are never replaced or cleaned up.
|
|
169
|
+
|
|
170
|
+
JSON uses the existing FactStore credential redaction. `representation`
|
|
171
|
+
distinguishes `original-json` from `redacted-json`. Original and archived
|
|
172
|
+
data hashes are stored separately; a redacted payload is never presented as
|
|
173
|
+
the original `source.payloadSha256` bytes. PNGs retain their original bytes.
|
|
174
|
+
|
|
175
|
+
## Include recorded files in the portable archive
|
|
176
|
+
|
|
177
|
+
Use the same public export call with `includeRecordedPayloads: true`:
|
|
178
|
+
|
|
179
|
+
```json
|
|
180
|
+
{
|
|
181
|
+
"command": "evidence",
|
|
182
|
+
"arguments": {
|
|
183
|
+
"operation": "export",
|
|
184
|
+
"namespace": "script",
|
|
185
|
+
"operationId": "script-from-your-response",
|
|
186
|
+
"outputDir": "/absolute/existing-parent/new-archive",
|
|
187
|
+
"includeRecordedPayloads": true
|
|
188
|
+
}
|
|
189
|
+
}
|
|
190
|
+
```
|
|
191
|
+
|
|
192
|
+
This produces `aab.evidence-archive/v2`: records, manifest and flat
|
|
193
|
+
`payload-<sha256>.json/png` files. All retained attachment references must
|
|
194
|
+
resolve with the expected size/hash. Export never re-fetches mobile facts
|
|
195
|
+
or substitutes another screenshot. Keep the recording directory in place
|
|
196
|
+
until export; after export, move the archive as a unit and use the same
|
|
197
|
+
`verify` operation and separately saved manifest hash. Offline verification
|
|
198
|
+
never opens the old recording directory or screenshot paths.
|
|
199
|
+
|
|
200
|
+
`recordedPayloads` reports included calls, Host assertion verdict counts,
|
|
201
|
+
screenshots, mobile pages/items and checked item/ref associations. Item IDs,
|
|
202
|
+
timestamps, stream, target and explicitly filtered epoch must agree. A
|
|
203
|
+
connected-history page's current epoch is not substituted for an older
|
|
204
|
+
fact's own epoch. Coverage, query window, pagination and store generation
|
|
205
|
+
remain the values returned by the source. Uncommitted items remain visible
|
|
206
|
+
without being counted as bound mobile facts.
|
|
207
|
+
|
|
208
|
+
This remains an archive of **retained recorded payloads**, not proof of a
|
|
209
|
+
complete run. Unqueried facts, calls made without recording, evicted
|
|
210
|
+
attachment references, in-memory progress and late/in-flight calls outside
|
|
211
|
+
the export watermark are not reconstructed. Assertion verdicts are retained
|
|
212
|
+
Host judgments of the supplied condition/evidence, not an independent
|
|
213
|
+
recalculation of business expectations. Integrity verification does not
|
|
214
|
+
turn a failed or inconclusive assertion into a pass.
|
|
@@ -0,0 +1,71 @@
|
|
|
1
|
+
# Android Intent foreground routing
|
|
2
|
+
|
|
3
|
+
An Intent can keep its original business target while navigating through explicitly listed Android system apps. Supply `target.foregroundPackages` to enable this mode. Without that field, the existing explicit provider contract is unchanged.
|
|
4
|
+
|
|
5
|
+
```json
|
|
6
|
+
{
|
|
7
|
+
"command": "intent",
|
|
8
|
+
"arguments": {
|
|
9
|
+
"operation": "start",
|
|
10
|
+
"goal": "Export a backup, choose its file in the system picker, and return to the notes app",
|
|
11
|
+
"provider": "native",
|
|
12
|
+
"target": {
|
|
13
|
+
"serial": "DEVICE_SERIAL",
|
|
14
|
+
"packageName": "io.github.mobileaidev.notallyx.sample",
|
|
15
|
+
"foregroundPackages": ["com.coloros.filemanager"]
|
|
16
|
+
}
|
|
17
|
+
}
|
|
18
|
+
}
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
Use packages actually resolved on the device for the requested flow. `foregroundPackages` is an explicit array of additional package names; an empty array restricts the operation to its original app. It does not discover or grant access to unrelated apps.
|
|
22
|
+
|
|
23
|
+
Each observation checks the foreground component before and after acquiring a fresh tree. In the original app it uses the selected primary provider (`native`, `flutter`, or `uia`); in a listed external package it uses UIA. Provider failures remain errors, and never trigger a substitute tree. A UIA dump must identify the same foreground package.
|
|
24
|
+
|
|
25
|
+
`uia_tree_changed` means the tree changed during traversal; that read supplies no
|
|
26
|
+
complete tree. A caller may make another explicit read within its observation
|
|
27
|
+
budget, retaining the failed read. It must obtain a complete current tree before
|
|
28
|
+
choosing any action. The Host does not retry the read or substitute a provider.
|
|
29
|
+
|
|
30
|
+
The summary contains `provider` and `foreground` (`packageName`, `activity`, `component`, probe source and timestamps). Observation, decision, dispatch marker and receipt records preserve that route. The original `target` continues to identify the business operation and its app capture streams. System UIA observations do not imply that system-app network/state/event capture is available.
|
|
31
|
+
|
|
32
|
+
Actions inherit the provider of their committed observation. An explicit conflicting provider is rejected. Before dispatch, the adapter checks the foreground component again; a changed component returns `reobserve_required` without dispatching a tap. Exact UIA taps use the API 33+ phone node runtime. Its `uia-node` execution receipt binds the Intent action ID, original request hash, runtime epoch, node/window attributes and original callback. The selected node is revalidated on the same automation connection before dispatch. A callback still requires a fresh observation and business checks; it does not create system-app SDK events.
|
|
33
|
+
|
|
34
|
+
Device-scoped physical taps remain managed Android shell input. Cancellation
|
|
35
|
+
and recovery follow the selected executor's admission and completion contract.
|
|
36
|
+
If a dispatched action's terminal receipt is unavailable, durable device
|
|
37
|
+
ownership blocks later mutations until `device-ownership reconcile` verifies
|
|
38
|
+
that original receipt.
|
|
39
|
+
Intent records retain both the original unknown result and any separate recovery
|
|
40
|
+
history. A completed tap still requires a fresh observation and business checks.
|
|
41
|
+
|
|
42
|
+
For exact UIA taps, use one of these selectors:
|
|
43
|
+
|
|
44
|
+
```json
|
|
45
|
+
{ "action": "tap", "selector": { "text": "NotallyX Backup 2026-09-07 \n15-44.zip" } }
|
|
46
|
+
{ "action": "tap", "selector": { "resourceName": "com.coloros.filemanager:id/action_file_operate" } }
|
|
47
|
+
{ "action": "tap", "selector": { "contentDescription": "返回" } }
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
The selector must identify exactly one enabled node in the observed package. Duplicate matches, unsupported selector fields and unavailable bounds fail before input. UIA lookup, compact trees and Intent summaries share the same XML attribute decoder, including decimal/hexadecimal character references. Decoding happens once: `&#10;` stays literal ` `, while ` ` becomes a newline. Invalid XML character references fail explicitly.
|
|
51
|
+
|
|
52
|
+
If the foreground changes during observation, the operation enters `waiting_for_observation`. Call `operation: "observe"` with the same `operationId`; the operation returns to `waiting_for_decision` only after a new observation commits. Acting or completing from `waiting_for_observation` is rejected. Existing evidence-store failures retain their separate blocked state.
|
|
53
|
+
|
|
54
|
+
Flutter taps accept one exact selector: `{ "text": "Settings" }` or
|
|
55
|
+
`{ "nodeId": "15" }`. Put the identity inside `action.selector`.
|
|
56
|
+
Text must identify one actionable node. Repeated
|
|
57
|
+
labels such as three settings rows displaying "System" return
|
|
58
|
+
`flutter_selector_not_unique` without dispatch. Select the intended row's
|
|
59
|
+
`nodeId` from the current committed observation instead. IDs are local to that
|
|
60
|
+
observation; a new tree requires a new lookup. A node must supply valid
|
|
61
|
+
`tap.bounds`; the Host does not invent tap bounds for text-only nodes. Flutter
|
|
62
|
+
coordinates remain logical pixels. Material `NavigationDestination` nodes
|
|
63
|
+
expose their public labels and individual destination bounds.
|
|
64
|
+
|
|
65
|
+
The operable tree traverses live Elements, including framework-owned pages
|
|
66
|
+
such as `LicensePage`. It is separate from the diagnostic inspector summary.
|
|
67
|
+
Repeated text for the same action region is emitted once; distinct settings
|
|
68
|
+
rows keep distinct targets. Each emitted label needs its own visible bounds.
|
|
69
|
+
The traversal reports truncation when its 512-level depth limit is reached.
|
|
70
|
+
|
|
71
|
+
This brackets foreground identity; it is not an atomic OS screenshot/action transaction. Animations and layout changes within the same Activity still require fresh stable observations and post-action verification. This change does not add arbitrary UIA Unicode input or attachment/backup business assertions.
|
|
@@ -0,0 +1,79 @@
|
|
|
1
|
+
# Native editing through Intent
|
|
2
|
+
|
|
3
|
+
Use a current supervised Intent observation. To acquire a newer screen without dispatching an action, call:
|
|
4
|
+
|
|
5
|
+
```json
|
|
6
|
+
{"operation":"observe","operationId":"your-intent","basedOnRevision":1}
|
|
7
|
+
```
|
|
8
|
+
|
|
9
|
+
Use the canonical `observe` operation. It reads the provider and persists the observation and summary before exposing revision + 1 while waiting for a decision or observation. Old decisions then require the new revision. It does not sleep, tap, or restart a terminal execution.
|
|
10
|
+
|
|
11
|
+
Submit an editing decision using the returned revision:
|
|
12
|
+
|
|
13
|
+
```json
|
|
14
|
+
{"operation":"decide","operationId":"your-intent","decision":{"decisionId":"edit-title-1","agentDecision":"act","basedOnRevision":2,"action":{"action":"inputText","selector":{"resourceName":"io.github.mobileaidev.notallyx.sample:id/EnterTitle"},"value":"Example title"}}}
|
|
15
|
+
```
|
|
16
|
+
|
|
17
|
+
`tap` uses the same selector shape. Select one identity through `resourceName`, exact `text`, or `contentDescription`; an optional `within` scopes the native match. Native selection requires exactly one enabled, effectively visible node in the topmost observed window and a center inside its viewport. Input additionally requires SDK `editable: true`. Missing or false `editable` always rejects; class-name guesses are not supported. Duplicate, invisible, obscured and noneditable targets fail without dispatch. Before dispatch, the adapter reads the current tree again and checks the window and semantic identity. A replaced, ambiguous or ineligible target rejects without input. Native `scroll` requires the selected container and `direction:"down"` or `"up"`. Flutter scrolling uses the separate `scrollBy` action.
|
|
18
|
+
|
|
19
|
+
Native summaries preserve `resourceName`, `editable`, `visible`, and `effectiveVisible` when supplied by the SDK, including blank input fields. Missing metadata remains unknown. The top-level `activity` is copied from the current raw tree. Summary visibility does not establish that a node belongs to the foreground window; the adapter separately enforces the topmost observed window. Each summary node retains its `rawTreeId` and `sourceIndex` for evidence lookup.
|
|
20
|
+
|
|
21
|
+
An exact native accessibility name can use `selector: {"contentDescription":"置于顶部"}`. It is a separate selector from `text`; no substring match or text/description fallback occurs. A native summary `label` derived from the raw `contentDescription` corresponds to this explicit selector. The same unique visible foreground target rules apply.
|
|
22
|
+
|
|
23
|
+
Native tap and input pass the selector, SDK `targetRef` and Host action ID to the dedicated `/v1/action/tap-target` and `/v1/action/input-target` endpoints. They do not send coordinates. The SDK validates the runtime, original operable window, View instance, observed semantic ancestry, uniqueness and eligibility on the UI thread immediately before the action. Geometry changes on the same View use its current position. Input binds to that exact editor and rechecks after synchronous focus/connection callbacks; a rejection after requesting focus reports that dispatch already began. It never inserts text into a replacement editor.
|
|
24
|
+
|
|
25
|
+
Script and direct CLI/MCP callers can use the same editor binding through
|
|
26
|
+
`input-text` with `{selector, text}`. Empty text clears the editor. The command
|
|
27
|
+
reobserves and revalidates the selector; an Intent decision additionally names
|
|
28
|
+
the Agent-reviewed revision. Neither entry substitutes coordinates when the
|
|
29
|
+
SDK target reference is unavailable.
|
|
30
|
+
|
|
31
|
+
Window snapshots distinguish `focused`, `focusable`, `touchable` and nullable `focusOwnerWindowId`. A touchable, non-focusable popup accepts pointer actions only while a window with the same application window token retains focus. The top visible window still owns selection: an overlay never grants access to a background target. Missing metadata, a non-touchable window, or loss of the owning focus rejects explicitly. Native text input still requires the selected window itself to have focus. Tap and gesture events preserve screen-space `rawX/rawY` as well as window-local `x/y`, including cancellation events.
|
|
32
|
+
|
|
33
|
+
These actions require `targetRef.schemaVersion:"aab.native-target/v1"`. Missing metadata or an unsupported endpoint returns `native_atomic_target_unavailable` without a coordinate/ADB fallback. SDK tasks still queued when their UI wait times out cannot execute later; already-started mutations can have an ambiguous timeout outcome. Native gestures use this reference contract too, with the timed stream described below. Flutter and system UIA do not yet have this SDK execution binding. Host receipts and SDK target validation still require runtime evidence and independent business-state verification.
|
|
34
|
+
|
|
35
|
+
Unsupported actions/providers fail closed. UIA and Flutter inputText are outside this addition. Unknown actions never become taps. Intent execution completion is separate from business acceptance.
|
|
36
|
+
|
|
37
|
+
Native `longPress` requires an exact `selector` and integer `durationMs:500..10000`.
|
|
38
|
+
Native `swipe` requires an exact `selector`, numeric `deltaX`/`deltaY`, and integer
|
|
39
|
+
`durationMs:1..10000`. The SDK computes the start from the selected node's current
|
|
40
|
+
center; the endpoint adds those deltas and must remain inside the current window.
|
|
41
|
+
A downward finger gesture has positive `deltaY`.
|
|
42
|
+
|
|
43
|
+
Native `scroll` uses an explicit container, for example
|
|
44
|
+
`{action:"scroll", selector:{resourceName:"example.app:id/list"}, direction:"down"}`.
|
|
45
|
+
It swipes within that container's visible area (`durationMs` defaults to 400).
|
|
46
|
+
If it cannot scroll in that direction, `native_scroll_boundary` rejects before
|
|
47
|
+
touch. UIA `scroll` remains a separate viewport action with an explicit provider.
|
|
48
|
+
|
|
49
|
+
The SDK binds the window/View before DOWN and runs real-time touch events through
|
|
50
|
+
the UI handler. Window/focus loss or cancellation sends CANCEL to the original
|
|
51
|
+
window. It does not undo a long-click callback or other App effects. Host waits
|
|
52
|
+
for a bounded cancellation acknowledgement after response loss; missing or
|
|
53
|
+
invalid confirmation remains ambiguous and never triggers a gesture replay.
|
|
54
|
+
Started/terminal phone `ui.interaction` events retain the action ID. Arbitrary
|
|
55
|
+
delayed callbacks are not all causally tagged. Reobserve and verify actual App
|
|
56
|
+
state; a completed touch stream alone does not establish business acceptance.
|
|
57
|
+
|
|
58
|
+
The same implementation is available to Script and direct commands as
|
|
59
|
+
`native-gesture`, with the action object under `payload`. Script uses
|
|
60
|
+
`ctx.call('native-gesture', {payload:{action:'longPress', selector:{text:'Observed note'}, durationMs:700}})`;
|
|
61
|
+
the Script target supplies serial/packageName, and Host owns action identity.
|
|
62
|
+
See [the command contract](COMMAND_CONTRACT.md) for cancellation and receipt fields.
|
|
63
|
+
|
|
64
|
+
`action: "back"` sends Android Back. `action: "keyevent", keyCode: 4` expresses
|
|
65
|
+
that key explicitly. Observe the keyboard/page state before deciding what Back
|
|
66
|
+
should accomplish; dismissing a keyboard and leaving a page are different outcomes.
|
|
67
|
+
|
|
68
|
+
Preserve per-step Intent responses and action receipts. Native observations
|
|
69
|
+
currently do not take a screenshot automatically: `summary.screenshotId` can
|
|
70
|
+
be null. A separate `screenshot` call should retain its artifact hash, timestamp
|
|
71
|
+
and the adjacent observation ID. This association is sequential, not an atomic
|
|
72
|
+
tree/screenshot capture. `status.history` is a bounded execution ledger and
|
|
73
|
+
does not expose the full stored raw tree for each observation.
|
|
74
|
+
|
|
75
|
+
Use [the public evidence export](EVIDENCE_ARCHIVE.md) with this Intent's
|
|
76
|
+
operationId to freeze retained raw observations, summaries, decisions and
|
|
77
|
+
receipts. A returned manifest hash supports verification after moving the
|
|
78
|
+
archive or restarting MCP. External screenshot files and phone capture bodies
|
|
79
|
+
remain separate; the export reports this coverage explicitly.
|
package/docs/RELEASE.md
ADDED
|
@@ -0,0 +1,59 @@
|
|
|
1
|
+
# 0.3.0-rc.2 发行与接入交接
|
|
2
|
+
|
|
3
|
+
本文件记录候选版的依赖关系和出仓库交付入口。封版要求是同一提交的源码、发行包与公开接入合同一致;单个样本的测试进度不改变包版本或发布状态。推送 Git、创建远端标签及发布 npm/pub 包由维护者执行。
|
|
4
|
+
|
|
5
|
+
## 版本与消费方式
|
|
6
|
+
|
|
7
|
+
| 交付物 | 候选版本 | 独立消费入口 | 发布依赖 |
|
|
8
|
+
| --- | --- | --- | --- |
|
|
9
|
+
| Android SDK | `0.3.0-rc.2` | JitPack `com.github.mobileAiDev.ai-app-bridge:ai-app-bridge-android:0.3.0-rc.2` | 同名 Git tag,JitPack 对该提交成功构建 |
|
|
10
|
+
| Android Gradle 插件 | `0.3.0-rc.2` | JitPack `ai-app-bridge-gradle-plugin` 模块及插件 ID `io.github.mobileaidev.aiappbridge.android` | 与 SDK 相同的 Git tag;不再使用旧默认 `0.2.8` |
|
|
11
|
+
| 原生 iOS SDK | Git tag `0.3.0-rc.2` | Git URL 的仓库根 `Package.swift`,产品 `AiAppBridgeIOS` | 根清单包含 Swift runtime、C adapter 和 segmented C store,无外部 C 包路径 |
|
|
12
|
+
| Flutter 插件 | `0.3.0-rc.2` | pub `ai_app_bridge_flutter` | Android 固定依赖上述 SDK;iOS Swift/C 源码随插件分发 |
|
|
13
|
+
| Desktop CLI/MCP | `0.3.0-rc.2` | npm `@mobileaidev/ai-app-bridge` | 包含 UIA bundle、WDA 模板和 native store 源码;WDA 上游固定 `14.1.1` |
|
|
14
|
+
| Web SDK | `0.3.0-rc.2` | npm `@mobileaidev/ai-app-bridge-web` | 独立浏览器源码包,无 npm 对 CLI 的安装依赖 |
|
|
15
|
+
| Native store | `0.1.0` | 随 CLI 的 bundled dependency 安装 | 不要求另行发布到 npm;`file:../../native/segmented-fact-store` 是工作区构建入口,最终 tarball 必须包含该依赖源码 |
|
|
16
|
+
|
|
17
|
+
Flutter 的 podspec 是随 pub 插件消费的本地 podspec,不是独立 CocoaPods trunk 发布包;原生 iOS 使用根 Swift package。Flutter SwiftPM 的 `../FlutterFramework` 由 Flutter 的集成生成,不能当作本仓库的外部私有依赖,也不应将本机 Flutter framework 打包进插件。
|
|
18
|
+
|
|
19
|
+
Host 支持范围声明为 Node `>=26.3.0 <27`,本轮实际验证基线是 **26.3.0**。共享 runtime 与查询索引依赖 `node:sqlite`,native store 安装需要 node-gyp 所需的 Python 和 C/C++ 编译工具。未对其他 Node 版本或跨主版本兼容作实测声明。Python Script 另需可用的 `python3`,从实际 `script runtime-status` 读取环境能力。
|
|
20
|
+
|
|
21
|
+
## 发布顺序
|
|
22
|
+
|
|
23
|
+
1. 完成源码审阅并冻结一个提交,核对以下命令的产物确实来自它;包含当前 untracked 的实际源码、测试和文档,排除本机生成目录。所有候选对外版本使用同一个 `0.3.0-rc.2`,若需要改版本,先同时更新上表涉及的 manifest 与固定依赖。
|
|
24
|
+
2. 维护者推送提交与 `0.3.0-rc.2` 标签,让 JitPack 构建 Android SDK/插件。确认两条公开坐标可解析后,再发布依赖它们的 Flutter 包。本地 Gradle project/path/AAR 替换不能证明 JitPack 坐标可消费。
|
|
25
|
+
3. 原生 iOS 消费相同 Git tag 的根 package;完成根 package 的 iOS 构建,不仅构建 `ios/ai-app-bridge-ios/Package.swift`。Flutter iOS 则检查实际 pub 包内 Swift/C 源码与声明相符。
|
|
26
|
+
4. CLI 与 Web SDK 可分别发布到 npm 的候选 dist-tag。CLI 的 native store 已打包随行,不等待一个不存在的单独 registry 依赖。Flutter 包发布以第 2 步完成为前提。
|
|
27
|
+
5. 从 registry/tag 安装刚发布的确切版本,读取 `capabilities` 和版本,核对来源及支持范围,再按发布策略提升正式 dist-tag。候选发布不自动等于全平台生产验收完成。
|
|
28
|
+
|
|
29
|
+
候选发布命令需在对应目录由维护者执行,例如 npm 使用 `npm publish --tag next`;pub 使用 `flutter pub publish`。这些命令属于发布动作,不能混入本地验证脚本。
|
|
30
|
+
|
|
31
|
+
## 本地检查与最终包验证
|
|
32
|
+
|
|
33
|
+
以下检查不发布版本。路径相对仓库根;输出使用新的、Git 忽略的目录。
|
|
34
|
+
|
|
35
|
+
```sh
|
|
36
|
+
swift package --package-path . dump-package
|
|
37
|
+
swift package --package-path ios/ai-app-bridge-ios dump-package
|
|
38
|
+
|
|
39
|
+
cd desktop/ai-app-bridge-cli
|
|
40
|
+
npm pack --dry-run --json --ignore-scripts
|
|
41
|
+
|
|
42
|
+
cd ../../web/ai-app-bridge-web
|
|
43
|
+
npm pack --dry-run --json --ignore-scripts
|
|
44
|
+
|
|
45
|
+
cd ../../flutter/ai_app_bridge_flutter
|
|
46
|
+
flutter pub publish --dry-run
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
`dump-package` 只证明 manifest 可解析及目标声明,不能替代 iOS 编译;`npm pack --dry-run` 只证明拟打包文件清单,不能替代安装;pub dry-run 中的分析/网络检查结果应原样记录。干净安装与 Host 协议验证必须在核心源码冻结、没有运行中修改时执行:
|
|
50
|
+
|
|
51
|
+
```sh
|
|
52
|
+
cd desktop/ai-app-bridge-cli
|
|
53
|
+
npm ci
|
|
54
|
+
npm run verify:package -- ../../build/ai_app_bridge_artifacts/release-package-NEW
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
`verify:package` 在仓库外安装实际 tarball,检查 native 安装编译、CLI/MCP 共享运行时、控制接口与随包运行时身份。它使用受控 ADB,不声称完成新真机业务验收。报告、tgz 哈希、安装日志和源码提交身份一起交接;已运行的旧包验证不能代替后来修改过的包。
|
|
58
|
+
|
|
59
|
+
CLI 的 `files` 已排除旧 `fact-cache.js` 发布载荷及 fake/P9/旧设备 adapter;旧 fact-cache 实现仅保留为 `test-support` 测试夹具,无生产引用。各 npm 包和 Flutter 目录的 `LICENSE`/`NOTICE` 均来自仓库根原文,发行时核对内容一致,不生成替代版权说明。
|