@mobileaidev/ai-app-bridge 0.2.15 → 0.3.0-rc.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (156) hide show
  1. package/LICENSE +201 -0
  2. package/NOTICE +7 -0
  3. package/README.md +306 -53
  4. package/bin/ai-app-bridge.js +51 -4504
  5. package/bin/android-permissions.js +152 -0
  6. package/bin/android-uia-xml.js +74 -0
  7. package/bin/artifact-paths.js +127 -1
  8. package/bin/bridge-forward.js +56 -0
  9. package/bin/command-discovery.js +86 -0
  10. package/bin/command-errors.js +60 -0
  11. package/bin/command-registry.js +504 -0
  12. package/bin/command-request.js +14 -0
  13. package/bin/command-router.js +80 -0
  14. package/bin/connection-cache.js +119 -0
  15. package/bin/device-provider.js +2934 -0
  16. package/bin/execution-host.js +596 -0
  17. package/bin/execution-runtime.js +101 -0
  18. package/bin/fact-codec.js +321 -0
  19. package/bin/fact-recorder.js +691 -0
  20. package/bin/fact-store.js +226 -0
  21. package/bin/feedback-probe.js +285 -0
  22. package/bin/intent/install-intent.js +267 -0
  23. package/bin/intent/intent-action-executor.js +129 -0
  24. package/bin/intent/intent-autonomous-adapter.js +46 -0
  25. package/bin/intent/intent-capture-port.js +64 -0
  26. package/bin/intent/intent-entry.js +226 -0
  27. package/bin/intent/intent-errors.js +28 -0
  28. package/bin/intent/intent-evidence-store.js +121 -0
  29. package/bin/intent/intent-lifetime.js +62 -0
  30. package/bin/intent/intent-observation-target.js +33 -0
  31. package/bin/intent/intent-observer.js +194 -0
  32. package/bin/intent/intent-production-adapter.js +334 -0
  33. package/bin/intent/intent-provider.js +27 -0
  34. package/bin/intent/intent-runtime.js +63 -0
  35. package/bin/intent/intent-worker.js +432 -0
  36. package/bin/intent/ios-intent-adapter.js +90 -0
  37. package/bin/intent/permission-intent.js +239 -0
  38. package/bin/intent/web-intent-adapter.js +31 -0
  39. package/bin/ios-device-outcome.js +47 -0
  40. package/bin/ios-execution.js +108 -0
  41. package/bin/ios-provider.js +497 -583
  42. package/bin/ios-runtime-binding.js +54 -0
  43. package/bin/ios-wda-execution.js +70 -0
  44. package/bin/ios-wda-port.js +98 -0
  45. package/bin/ios-wda-project.js +79 -0
  46. package/bin/mcp-server.js +83 -1040
  47. package/bin/mmap-scan-index.js +329 -0
  48. package/bin/observation-collector.js +862 -0
  49. package/bin/runtime-client.js +154 -0
  50. package/bin/runtime-directory.js +107 -0
  51. package/bin/runtime-protocol.js +36 -0
  52. package/bin/script/bounded-script-registry.js +103 -0
  53. package/bin/script/node-runtime-adapter.js +233 -0
  54. package/bin/script/progress-projector.js +63 -0
  55. package/bin/script/python-runtime-adapter.js +111 -0
  56. package/bin/script/rolling-summary.js +134 -0
  57. package/bin/script/script-agent-port.js +40 -0
  58. package/bin/script/script-assert.js +95 -0
  59. package/bin/script/script-capture-port.js +76 -0
  60. package/bin/script/script-catalog.js +85 -0
  61. package/bin/script/script-durable-restore.js +195 -0
  62. package/bin/script/script-entry-code.js +26 -0
  63. package/bin/script/script-entry-route.js +38 -0
  64. package/bin/script/script-entry.js +3 -0
  65. package/bin/script/script-errors.js +29 -0
  66. package/bin/script/script-evidence-store.js +22 -0
  67. package/bin/script/script-format-removed.js +26 -0
  68. package/bin/script/script-host-port.js +397 -0
  69. package/bin/script/script-ledger.js +64 -0
  70. package/bin/script/script-result.js +57 -0
  71. package/bin/script/script-sdk.js +152 -0
  72. package/bin/script/script-sdk.py +153 -0
  73. package/bin/script/script-session-channel.js +127 -0
  74. package/bin/script/script-spec.js +86 -0
  75. package/bin/script/script-supervisor.js +919 -0
  76. package/bin/script/templates/checkpoint-reentry.js +13 -0
  77. package/bin/segment-index.js +481 -0
  78. package/bin/segmented-fact-store.js +1571 -0
  79. package/bin/shared-kernel/android-h5-target.js +10 -0
  80. package/bin/shared-kernel/android-install-execution.js +176 -0
  81. package/bin/shared-kernel/android-sdk-endpoint.js +42 -0
  82. package/bin/shared-kernel/android-shell-execution.js +195 -0
  83. package/bin/shared-kernel/argument-schema.js +117 -0
  84. package/bin/shared-kernel/canonical-path.js +17 -0
  85. package/bin/shared-kernel/device-acknowledgements.js +53 -0
  86. package/bin/shared-kernel/device-completion-history.js +52 -0
  87. package/bin/shared-kernel/device-mutation-lease.js +219 -0
  88. package/bin/shared-kernel/device-ownership-recovery.js +95 -0
  89. package/bin/shared-kernel/device-ownership-store.js +94 -0
  90. package/bin/shared-kernel/evidence-adapters.js +251 -0
  91. package/bin/shared-kernel/evidence-archive.js +329 -0
  92. package/bin/shared-kernel/evidence-recording.js +131 -0
  93. package/bin/shared-kernel/evidence-schema.js +194 -0
  94. package/bin/shared-kernel/evidence-store.js +190 -0
  95. package/bin/shared-kernel/execution-admission.js +22 -0
  96. package/bin/shared-kernel/execution-contracts.js +171 -0
  97. package/bin/shared-kernel/execution-io.js +106 -0
  98. package/bin/shared-kernel/execution-ledger.js +125 -0
  99. package/bin/shared-kernel/execution-scope.js +87 -0
  100. package/bin/shared-kernel/execution-target.js +115 -0
  101. package/bin/shared-kernel/flutter-execution.js +13 -0
  102. package/bin/shared-kernel/flutter-h5-port.js +60 -0
  103. package/bin/shared-kernel/flutter-h5-target.js +9 -0
  104. package/bin/shared-kernel/flutter-target.js +75 -0
  105. package/bin/shared-kernel/h5-execution.js +11 -0
  106. package/bin/shared-kernel/h5-target.js +31 -0
  107. package/bin/shared-kernel/host-fact-store.js +49 -0
  108. package/bin/shared-kernel/ios-h5-target.js +9 -0
  109. package/bin/shared-kernel/ios-native-target.js +71 -0
  110. package/bin/shared-kernel/live-capture-query.js +115 -0
  111. package/bin/shared-kernel/managed-sdk-execution.js +78 -0
  112. package/bin/shared-kernel/native-execution.js +13 -0
  113. package/bin/shared-kernel/native-target.js +156 -0
  114. package/bin/shared-kernel/provider-command-contracts.js +55 -0
  115. package/bin/shared-kernel/recorded-payload-archive.js +195 -0
  116. package/bin/shared-kernel/request-context.js +35 -0
  117. package/bin/shared-kernel/semantic-node.js +55 -0
  118. package/bin/shared-kernel/summary-transformer.js +352 -0
  119. package/bin/shared-kernel/target-lease-protocol.js +47 -0
  120. package/bin/shared-kernel/text-wait.js +111 -0
  121. package/bin/shared-kernel/uia-execution.js +96 -0
  122. package/bin/shared-kernel/uia-protocol.js +214 -0
  123. package/bin/shared-kernel/uia-runtime-port.js +377 -0
  124. package/bin/shared-kernel/uia-target.js +39 -0
  125. package/bin/shared-kernel/web-dom-target.js +44 -0
  126. package/bin/shared-kernel/xml-attributes.js +25 -0
  127. package/bin/target-execution.js +275 -0
  128. package/bin/web/command-schema.js +60 -0
  129. package/bin/web/session-store.js +157 -0
  130. package/bin/web-provider.js +334 -553
  131. package/docs/COMMAND_CONTRACT.md +1563 -0
  132. package/docs/EVIDENCE_ARCHIVE.md +214 -0
  133. package/docs/INTENT_FOREGROUND.md +71 -0
  134. package/docs/INTENT_NATIVE_EDITING.md +79 -0
  135. package/docs/RELEASE.md +59 -0
  136. package/docs/SCRIPT_AUTHORING.md +489 -0
  137. package/node_modules/@mobileaidev/segmented-fact-store-native/LICENSE +201 -0
  138. package/node_modules/@mobileaidev/segmented-fact-store-native/NOTICE +7 -0
  139. package/node_modules/@mobileaidev/segmented-fact-store-native/binding.gyp +36 -0
  140. package/node_modules/@mobileaidev/segmented-fact-store-native/bindings/node/sfs_node.c +597 -0
  141. package/node_modules/@mobileaidev/segmented-fact-store-native/include/sfs.h +178 -0
  142. package/node_modules/@mobileaidev/segmented-fact-store-native/index.js +5 -0
  143. package/node_modules/@mobileaidev/segmented-fact-store-native/package.json +24 -0
  144. package/node_modules/@mobileaidev/segmented-fact-store-native/src/sfs.c +2349 -0
  145. package/package.json +60 -5
  146. package/runtime/ios-wda/AABWDABinding.h +19 -0
  147. package/runtime/ios-wda/AABWDABinding.m +97 -0
  148. package/runtime/ios-wda/AABWDAExecution.h +26 -0
  149. package/runtime/ios-wda/AABWDAExecution.m +172 -0
  150. package/runtime/ios-wda/AABWDAIntegration.h +71 -0
  151. package/runtime/ios-wda/AABWDAManagedRoutes.h +392 -0
  152. package/runtime/ios-wda/AABWDAReceiptStore.h +10 -0
  153. package/runtime/ios-wda/AABWDAReceiptStore.m +116 -0
  154. package/runtime/uia/ai-app-bridge-uia.jar +0 -0
  155. package/runtime/uia/manifest.json +22 -0
  156. package/skills/ai-app-bridge-use/SKILL.md +23 -360
@@ -0,0 +1,214 @@
1
+ # Record, export and verify execution evidence
2
+
3
+ Discover the public MCP command with `capabilities {"command":"evidence"}`.
4
+ It works for both Intent operation IDs and Script execution IDs, including
5
+ records retained after the execution runtime has restarted. CLI and MCP use the
6
+ same operation IDs and export command. Export does not require a
7
+ connected phone or a live worker.
8
+
9
+ ```json
10
+ {
11
+ "command": "evidence",
12
+ "arguments": {
13
+ "operation": "export",
14
+ "namespace": "intent",
15
+ "operationId": "intent-from-your-response",
16
+ "outputDir": "/absolute/existing-parent/new-archive"
17
+ }
18
+ }
19
+ ```
20
+
21
+ Use `namespace: "script"` for a Script operation. Export reads the current
22
+ execution runtime's configured FactStore (`AI_APP_BRIDGE_FACT_STORE_DIR`), drains its
23
+ queued writes, and freezes the retained records for exactly that namespace
24
+ and operation. The parent directory must exist; the output directory must
25
+ not exist. No existing directory is replaced.
26
+
27
+ The response includes `archiveDir`, `manifestPath`, `manifestSha256`,
28
+ `recordCount`, `targets`, and `coverage`. Save the returned manifest SHA256
29
+ separately when handing the archive to an author or reviewer.
30
+
31
+ Without `includeRecordedPayloads`, the directory contains two files:
32
+
33
+ - `records.json`: the original ordered Facts, including their globalSeq,
34
+ outer source binding, full evidence envelopes, raw trees where stored,
35
+ checksums, and original device/app identities.
36
+ - `manifest.json`: schema `aab.evidence-archive/v1`, namespace/operation,
37
+ source store ID and sequence watermark, partition retention metadata,
38
+ file size/SHA256, record counts, target inventory, and coverage.
39
+
40
+ The sequence can have gaps because unrelated operations share the store.
41
+ An expired cursor, unreadable page, checksum error, mismatched binding, or
42
+ snapshot change fails export. An unknown operation returns
43
+ `operation_not_found`. Limits are 10,000 records and 128 MiB of record JSON;
44
+ the manifest is limited to 2 MiB. Exceeding a limit fails rather than
45
+ truncating. The manifest is written last; a directory left by an interrupted
46
+ write is not a successfully verified archive.
47
+
48
+ ## Verify without the source store or phone
49
+
50
+ Move or copy the directory as a unit, then call:
51
+
52
+ ```json
53
+ {
54
+ "command": "evidence",
55
+ "arguments": {
56
+ "operation": "verify",
57
+ "archiveDir": "/absolute/moved-archive",
58
+ "manifestSha256": "<the 64 lowercase hex characters saved from export>"
59
+ }
60
+ }
61
+ ```
62
+
63
+ Verification opens only the archive's ordinary files. It does not open a
64
+ FactStore, initialize an operation, or contact a phone. It verifies the
65
+ externally supplied manifest hash, records hash/size, each evidence checksum,
66
+ namespace/operation/source binding, IDs, sequence, internal reference order
67
+ and binding, and recalculated coverage and target inventory. Archive files cannot be symlinks, and the manifest
68
+ cannot redirect the verifier to another file. A missing or changed file
69
+ fails verification. Source paths embedded in evidence are data and are never
70
+ opened or executed.
71
+
72
+ Success returns `ok: true` and `integrity: "verified"`. The hash establishes
73
+ that the supplied archive matches the frozen export; it is not a signature
74
+ authenticating the producer. Obtain the expected hash from the export
75
+ response, not by hashing an untrusted archive and treating that as approval.
76
+
77
+ ## What this archive establishes
78
+
79
+ `coverage.scope` is `retained-host-records`. `priorHistoryComplete` remains
80
+ `unknown`: a store can already have evicted older records before export.
81
+ Partition retention metadata describes the entire partition, not proof that
82
+ this particular operation lost records.
83
+
84
+ `referenceClosure` is `complete` when the included records' checked internal
85
+ references are present, or `partial` with `missingReferences`. This is
86
+ separate from whole-history completeness. Exporting a partial retained set
87
+ can succeed as a diagnostic archive; it does not fill its missing evidence.
88
+ Contradictory references fail: a reference to a future record, a summary with
89
+ a different observation revision, a receipt preceding its dispatch marker,
90
+ or duplicate observation/decision/action identities cannot become a complete
91
+ reference set merely because the IDs exist.
92
+
93
+ The archive preserves already persisted, sanitized values byte-for-value as
94
+ JSON values. It does not change recorded serials or infer missing targets
95
+ from the currently connected phone. `owner` retains the declared context,
96
+ `observed` retains each observation's declared device/App identity, and `foreground`
97
+ retains explicit Intent route data. Script marker targets use `dispatch`
98
+ and can contain explicit target overrides, including another serial when
99
+ the Script supplies it. These roles preserve the current execution contract;
100
+ the archive does not apply a new device allowlist or infer an absent target.
101
+
102
+ New execution envelopes declare `schemaVersion: "aab.execution-evidence/v1"`.
103
+ Their targets use an explicit platform discriminator; observations carry both
104
+ `target` and `observedTarget` when the latter was actually observed. A pure
105
+ Script call can have a null target. Archive verification also understands the
106
+ original unversioned record format already frozen in archive v1/v2. It checks
107
+ those original fields and hashes without adding a platform or rewriting bytes.
108
+ New execution writes do not accept an old observation in place of a platform target.
109
+
110
+ Currently persisted Intent content includes observations with raw trees,
111
+ summaries, decisions, dispatch markers, and action receipts. Script content
112
+ includes durable checkpoints, dispatch markers, and action receipts.
113
+ Completed Scripts also persist their final JSON as a `result` record, with a
114
+ terminal checkpoint `resultRef` binding its evidence ID, canonical byte count,
115
+ persisted SHA256, original SHA256 and `original-json`/`redacted-json`
116
+ representation. Export includes this retained record without requiring
117
+ `includeRecordedPayloads`; it is independent of progress-event retention.
118
+ Verification checks the checkpoint's binding to that result. An evicted result
119
+ leaves partial reference closure rather than a reconstructed value. For a direct
120
+ checked read, use `script {operation:"result",operationId:"…"}`.
121
+
122
+ Without recorded payload inclusion, `externalPayloads` is `not-included`. External screenshot files, phone
123
+ logs/network capture items, individual Script call results/assertions, and in-memory
124
+ events are outside this archive. Existing external references are listed
125
+ but not downloaded or dereferenced. Preserve those artifacts separately
126
+ when the intended check requires them.
127
+
128
+ `executionStatus` is `not-inferred` and `businessVerdict` is `not-evaluated`.
129
+ The caller can inspect recorded terminal decisions/checkpoints, including a
130
+ persisted cancellation. Export verification does not infer missing lifecycle
131
+ state or reconstruct evicted records.
132
+ Archive integrity alone does not establish a passed business assertion.
133
+
134
+ ## Explicit recording for one execution
135
+
136
+ Add `recordingDir: "/absolute/existing-parent/new-recording"` to the
137
+ **arguments of `script start` or `intent start`**, alongside `script` or
138
+ `goal`/`target`. The parent must exist and the directory must be new.
139
+ This is opt-in file output for this execution; ordinary live calls do not
140
+ copy mobile payloads to a Host history database.
141
+
142
+ Script records each returned Host call envelope (command, arguments, actual
143
+ result, call/action/observation IDs, source hash, window, coverage, refs and
144
+ capture metadata), plus each assertion input and Host verdict. The Host
145
+ copies referenced screenshot PNG bytes immediately, checking the issued
146
+ SHA256 before an original path can be reused. It awaits the attachment's
147
+ FactStore receipt before replying to the child. This does not depend on the
148
+ child writing files or on retaining the bounded event log.
149
+
150
+ Intent records the complete capture pages already fetched by its `require`
151
+ streams, including items. Each attachment references its persisted raw-tree
152
+ observation. Recording itself does not query extra streams or take Intent
153
+ screenshots. With no `require` streams, Intent has no mobile payload to record.
154
+
155
+ The existing Host FactStore stores small `attachment` records containing
156
+ file hashes and execution bindings; the payload files remain in the explicit
157
+ output directory. The live mobile query path never consults those files.
158
+ Script recording currently requires `restartPolicy: "none"`; a checkpoint
159
+ restart request with recording is rejected as `recording_restart_unsupported`.
160
+ Live pause/resume can continue the same recording. No recording is silently
161
+ reopened after Host or child loss.
162
+
163
+ Limits are 10,000 attachments, 16 MiB per JSON/PNG file and 256 MiB per
164
+ recording. A write, checksum or persistence failure stops further successful
165
+ recording: Script receives an explicit error and blocks further calls;
166
+ Intent stops exposing the new observation. Previously committed material
167
+ can still be exported. Unreferenced files left by a failed write are not
168
+ included. Existing output directories are never replaced or cleaned up.
169
+
170
+ JSON uses the existing FactStore credential redaction. `representation`
171
+ distinguishes `original-json` from `redacted-json`. Original and archived
172
+ data hashes are stored separately; a redacted payload is never presented as
173
+ the original `source.payloadSha256` bytes. PNGs retain their original bytes.
174
+
175
+ ## Include recorded files in the portable archive
176
+
177
+ Use the same public export call with `includeRecordedPayloads: true`:
178
+
179
+ ```json
180
+ {
181
+ "command": "evidence",
182
+ "arguments": {
183
+ "operation": "export",
184
+ "namespace": "script",
185
+ "operationId": "script-from-your-response",
186
+ "outputDir": "/absolute/existing-parent/new-archive",
187
+ "includeRecordedPayloads": true
188
+ }
189
+ }
190
+ ```
191
+
192
+ This produces `aab.evidence-archive/v2`: records, manifest and flat
193
+ `payload-<sha256>.json/png` files. All retained attachment references must
194
+ resolve with the expected size/hash. Export never re-fetches mobile facts
195
+ or substitutes another screenshot. Keep the recording directory in place
196
+ until export; after export, move the archive as a unit and use the same
197
+ `verify` operation and separately saved manifest hash. Offline verification
198
+ never opens the old recording directory or screenshot paths.
199
+
200
+ `recordedPayloads` reports included calls, Host assertion verdict counts,
201
+ screenshots, mobile pages/items and checked item/ref associations. Item IDs,
202
+ timestamps, stream, target and explicitly filtered epoch must agree. A
203
+ connected-history page's current epoch is not substituted for an older
204
+ fact's own epoch. Coverage, query window, pagination and store generation
205
+ remain the values returned by the source. Uncommitted items remain visible
206
+ without being counted as bound mobile facts.
207
+
208
+ This remains an archive of **retained recorded payloads**, not proof of a
209
+ complete run. Unqueried facts, calls made without recording, evicted
210
+ attachment references, in-memory progress and late/in-flight calls outside
211
+ the export watermark are not reconstructed. Assertion verdicts are retained
212
+ Host judgments of the supplied condition/evidence, not an independent
213
+ recalculation of business expectations. Integrity verification does not
214
+ turn a failed or inconclusive assertion into a pass.
@@ -0,0 +1,71 @@
1
+ # Android Intent foreground routing
2
+
3
+ An Intent can keep its original business target while navigating through explicitly listed Android system apps. Supply `target.foregroundPackages` to enable this mode. Without that field, the existing explicit provider contract is unchanged.
4
+
5
+ ```json
6
+ {
7
+ "command": "intent",
8
+ "arguments": {
9
+ "operation": "start",
10
+ "goal": "Export a backup, choose its file in the system picker, and return to the notes app",
11
+ "provider": "native",
12
+ "target": {
13
+ "serial": "DEVICE_SERIAL",
14
+ "packageName": "io.github.mobileaidev.notallyx.sample",
15
+ "foregroundPackages": ["com.coloros.filemanager"]
16
+ }
17
+ }
18
+ }
19
+ ```
20
+
21
+ Use packages actually resolved on the device for the requested flow. `foregroundPackages` is an explicit array of additional package names; an empty array restricts the operation to its original app. It does not discover or grant access to unrelated apps.
22
+
23
+ Each observation checks the foreground component before and after acquiring a fresh tree. In the original app it uses the selected primary provider (`native`, `flutter`, or `uia`); in a listed external package it uses UIA. Provider failures remain errors, and never trigger a substitute tree. A UIA dump must identify the same foreground package.
24
+
25
+ `uia_tree_changed` means the tree changed during traversal; that read supplies no
26
+ complete tree. A caller may make another explicit read within its observation
27
+ budget, retaining the failed read. It must obtain a complete current tree before
28
+ choosing any action. The Host does not retry the read or substitute a provider.
29
+
30
+ The summary contains `provider` and `foreground` (`packageName`, `activity`, `component`, probe source and timestamps). Observation, decision, dispatch marker and receipt records preserve that route. The original `target` continues to identify the business operation and its app capture streams. System UIA observations do not imply that system-app network/state/event capture is available.
31
+
32
+ Actions inherit the provider of their committed observation. An explicit conflicting provider is rejected. Before dispatch, the adapter checks the foreground component again; a changed component returns `reobserve_required` without dispatching a tap. Exact UIA taps use the API 33+ phone node runtime. Its `uia-node` execution receipt binds the Intent action ID, original request hash, runtime epoch, node/window attributes and original callback. The selected node is revalidated on the same automation connection before dispatch. A callback still requires a fresh observation and business checks; it does not create system-app SDK events.
33
+
34
+ Device-scoped physical taps remain managed Android shell input. Cancellation
35
+ and recovery follow the selected executor's admission and completion contract.
36
+ If a dispatched action's terminal receipt is unavailable, durable device
37
+ ownership blocks later mutations until `device-ownership reconcile` verifies
38
+ that original receipt.
39
+ Intent records retain both the original unknown result and any separate recovery
40
+ history. A completed tap still requires a fresh observation and business checks.
41
+
42
+ For exact UIA taps, use one of these selectors:
43
+
44
+ ```json
45
+ { "action": "tap", "selector": { "text": "NotallyX Backup 2026-09-07 \n15-44.zip" } }
46
+ { "action": "tap", "selector": { "resourceName": "com.coloros.filemanager:id/action_file_operate" } }
47
+ { "action": "tap", "selector": { "contentDescription": "返回" } }
48
+ ```
49
+
50
+ The selector must identify exactly one enabled node in the observed package. Duplicate matches, unsupported selector fields and unavailable bounds fail before input. UIA lookup, compact trees and Intent summaries share the same XML attribute decoder, including decimal/hexadecimal character references. Decoding happens once: `&amp;#10;` stays literal `&#10;`, while `&#10;` becomes a newline. Invalid XML character references fail explicitly.
51
+
52
+ If the foreground changes during observation, the operation enters `waiting_for_observation`. Call `operation: "observe"` with the same `operationId`; the operation returns to `waiting_for_decision` only after a new observation commits. Acting or completing from `waiting_for_observation` is rejected. Existing evidence-store failures retain their separate blocked state.
53
+
54
+ Flutter taps accept one exact selector: `{ "text": "Settings" }` or
55
+ `{ "nodeId": "15" }`. Put the identity inside `action.selector`.
56
+ Text must identify one actionable node. Repeated
57
+ labels such as three settings rows displaying "System" return
58
+ `flutter_selector_not_unique` without dispatch. Select the intended row's
59
+ `nodeId` from the current committed observation instead. IDs are local to that
60
+ observation; a new tree requires a new lookup. A node must supply valid
61
+ `tap.bounds`; the Host does not invent tap bounds for text-only nodes. Flutter
62
+ coordinates remain logical pixels. Material `NavigationDestination` nodes
63
+ expose their public labels and individual destination bounds.
64
+
65
+ The operable tree traverses live Elements, including framework-owned pages
66
+ such as `LicensePage`. It is separate from the diagnostic inspector summary.
67
+ Repeated text for the same action region is emitted once; distinct settings
68
+ rows keep distinct targets. Each emitted label needs its own visible bounds.
69
+ The traversal reports truncation when its 512-level depth limit is reached.
70
+
71
+ This brackets foreground identity; it is not an atomic OS screenshot/action transaction. Animations and layout changes within the same Activity still require fresh stable observations and post-action verification. This change does not add arbitrary UIA Unicode input or attachment/backup business assertions.
@@ -0,0 +1,79 @@
1
+ # Native editing through Intent
2
+
3
+ Use a current supervised Intent observation. To acquire a newer screen without dispatching an action, call:
4
+
5
+ ```json
6
+ {"operation":"observe","operationId":"your-intent","basedOnRevision":1}
7
+ ```
8
+
9
+ Use the canonical `observe` operation. It reads the provider and persists the observation and summary before exposing revision + 1 while waiting for a decision or observation. Old decisions then require the new revision. It does not sleep, tap, or restart a terminal execution.
10
+
11
+ Submit an editing decision using the returned revision:
12
+
13
+ ```json
14
+ {"operation":"decide","operationId":"your-intent","decision":{"decisionId":"edit-title-1","agentDecision":"act","basedOnRevision":2,"action":{"action":"inputText","selector":{"resourceName":"io.github.mobileaidev.notallyx.sample:id/EnterTitle"},"value":"Example title"}}}
15
+ ```
16
+
17
+ `tap` uses the same selector shape. Select one identity through `resourceName`, exact `text`, or `contentDescription`; an optional `within` scopes the native match. Native selection requires exactly one enabled, effectively visible node in the topmost observed window and a center inside its viewport. Input additionally requires SDK `editable: true`. Missing or false `editable` always rejects; class-name guesses are not supported. Duplicate, invisible, obscured and noneditable targets fail without dispatch. Before dispatch, the adapter reads the current tree again and checks the window and semantic identity. A replaced, ambiguous or ineligible target rejects without input. Native `scroll` requires the selected container and `direction:"down"` or `"up"`. Flutter scrolling uses the separate `scrollBy` action.
18
+
19
+ Native summaries preserve `resourceName`, `editable`, `visible`, and `effectiveVisible` when supplied by the SDK, including blank input fields. Missing metadata remains unknown. The top-level `activity` is copied from the current raw tree. Summary visibility does not establish that a node belongs to the foreground window; the adapter separately enforces the topmost observed window. Each summary node retains its `rawTreeId` and `sourceIndex` for evidence lookup.
20
+
21
+ An exact native accessibility name can use `selector: {"contentDescription":"置于顶部"}`. It is a separate selector from `text`; no substring match or text/description fallback occurs. A native summary `label` derived from the raw `contentDescription` corresponds to this explicit selector. The same unique visible foreground target rules apply.
22
+
23
+ Native tap and input pass the selector, SDK `targetRef` and Host action ID to the dedicated `/v1/action/tap-target` and `/v1/action/input-target` endpoints. They do not send coordinates. The SDK validates the runtime, original operable window, View instance, observed semantic ancestry, uniqueness and eligibility on the UI thread immediately before the action. Geometry changes on the same View use its current position. Input binds to that exact editor and rechecks after synchronous focus/connection callbacks; a rejection after requesting focus reports that dispatch already began. It never inserts text into a replacement editor.
24
+
25
+ Script and direct CLI/MCP callers can use the same editor binding through
26
+ `input-text` with `{selector, text}`. Empty text clears the editor. The command
27
+ reobserves and revalidates the selector; an Intent decision additionally names
28
+ the Agent-reviewed revision. Neither entry substitutes coordinates when the
29
+ SDK target reference is unavailable.
30
+
31
+ Window snapshots distinguish `focused`, `focusable`, `touchable` and nullable `focusOwnerWindowId`. A touchable, non-focusable popup accepts pointer actions only while a window with the same application window token retains focus. The top visible window still owns selection: an overlay never grants access to a background target. Missing metadata, a non-touchable window, or loss of the owning focus rejects explicitly. Native text input still requires the selected window itself to have focus. Tap and gesture events preserve screen-space `rawX/rawY` as well as window-local `x/y`, including cancellation events.
32
+
33
+ These actions require `targetRef.schemaVersion:"aab.native-target/v1"`. Missing metadata or an unsupported endpoint returns `native_atomic_target_unavailable` without a coordinate/ADB fallback. SDK tasks still queued when their UI wait times out cannot execute later; already-started mutations can have an ambiguous timeout outcome. Native gestures use this reference contract too, with the timed stream described below. Flutter and system UIA do not yet have this SDK execution binding. Host receipts and SDK target validation still require runtime evidence and independent business-state verification.
34
+
35
+ Unsupported actions/providers fail closed. UIA and Flutter inputText are outside this addition. Unknown actions never become taps. Intent execution completion is separate from business acceptance.
36
+
37
+ Native `longPress` requires an exact `selector` and integer `durationMs:500..10000`.
38
+ Native `swipe` requires an exact `selector`, numeric `deltaX`/`deltaY`, and integer
39
+ `durationMs:1..10000`. The SDK computes the start from the selected node's current
40
+ center; the endpoint adds those deltas and must remain inside the current window.
41
+ A downward finger gesture has positive `deltaY`.
42
+
43
+ Native `scroll` uses an explicit container, for example
44
+ `{action:"scroll", selector:{resourceName:"example.app:id/list"}, direction:"down"}`.
45
+ It swipes within that container's visible area (`durationMs` defaults to 400).
46
+ If it cannot scroll in that direction, `native_scroll_boundary` rejects before
47
+ touch. UIA `scroll` remains a separate viewport action with an explicit provider.
48
+
49
+ The SDK binds the window/View before DOWN and runs real-time touch events through
50
+ the UI handler. Window/focus loss or cancellation sends CANCEL to the original
51
+ window. It does not undo a long-click callback or other App effects. Host waits
52
+ for a bounded cancellation acknowledgement after response loss; missing or
53
+ invalid confirmation remains ambiguous and never triggers a gesture replay.
54
+ Started/terminal phone `ui.interaction` events retain the action ID. Arbitrary
55
+ delayed callbacks are not all causally tagged. Reobserve and verify actual App
56
+ state; a completed touch stream alone does not establish business acceptance.
57
+
58
+ The same implementation is available to Script and direct commands as
59
+ `native-gesture`, with the action object under `payload`. Script uses
60
+ `ctx.call('native-gesture', {payload:{action:'longPress', selector:{text:'Observed note'}, durationMs:700}})`;
61
+ the Script target supplies serial/packageName, and Host owns action identity.
62
+ See [the command contract](COMMAND_CONTRACT.md) for cancellation and receipt fields.
63
+
64
+ `action: "back"` sends Android Back. `action: "keyevent", keyCode: 4` expresses
65
+ that key explicitly. Observe the keyboard/page state before deciding what Back
66
+ should accomplish; dismissing a keyboard and leaving a page are different outcomes.
67
+
68
+ Preserve per-step Intent responses and action receipts. Native observations
69
+ currently do not take a screenshot automatically: `summary.screenshotId` can
70
+ be null. A separate `screenshot` call should retain its artifact hash, timestamp
71
+ and the adjacent observation ID. This association is sequential, not an atomic
72
+ tree/screenshot capture. `status.history` is a bounded execution ledger and
73
+ does not expose the full stored raw tree for each observation.
74
+
75
+ Use [the public evidence export](EVIDENCE_ARCHIVE.md) with this Intent's
76
+ operationId to freeze retained raw observations, summaries, decisions and
77
+ receipts. A returned manifest hash supports verification after moving the
78
+ archive or restarting MCP. External screenshot files and phone capture bodies
79
+ remain separate; the export reports this coverage explicitly.
@@ -0,0 +1,59 @@
1
+ # 0.3.0-rc.2 发行与接入交接
2
+
3
+ 本文件记录候选版的依赖关系和出仓库交付入口。封版要求是同一提交的源码、发行包与公开接入合同一致;单个样本的测试进度不改变包版本或发布状态。推送 Git、创建远端标签及发布 npm/pub 包由维护者执行。
4
+
5
+ ## 版本与消费方式
6
+
7
+ | 交付物 | 候选版本 | 独立消费入口 | 发布依赖 |
8
+ | --- | --- | --- | --- |
9
+ | Android SDK | `0.3.0-rc.2` | JitPack `com.github.mobileAiDev.ai-app-bridge:ai-app-bridge-android:0.3.0-rc.2` | 同名 Git tag,JitPack 对该提交成功构建 |
10
+ | Android Gradle 插件 | `0.3.0-rc.2` | JitPack `ai-app-bridge-gradle-plugin` 模块及插件 ID `io.github.mobileaidev.aiappbridge.android` | 与 SDK 相同的 Git tag;不再使用旧默认 `0.2.8` |
11
+ | 原生 iOS SDK | Git tag `0.3.0-rc.2` | Git URL 的仓库根 `Package.swift`,产品 `AiAppBridgeIOS` | 根清单包含 Swift runtime、C adapter 和 segmented C store,无外部 C 包路径 |
12
+ | Flutter 插件 | `0.3.0-rc.2` | pub `ai_app_bridge_flutter` | Android 固定依赖上述 SDK;iOS Swift/C 源码随插件分发 |
13
+ | Desktop CLI/MCP | `0.3.0-rc.2` | npm `@mobileaidev/ai-app-bridge` | 包含 UIA bundle、WDA 模板和 native store 源码;WDA 上游固定 `14.1.1` |
14
+ | Web SDK | `0.3.0-rc.2` | npm `@mobileaidev/ai-app-bridge-web` | 独立浏览器源码包,无 npm 对 CLI 的安装依赖 |
15
+ | Native store | `0.1.0` | 随 CLI 的 bundled dependency 安装 | 不要求另行发布到 npm;`file:../../native/segmented-fact-store` 是工作区构建入口,最终 tarball 必须包含该依赖源码 |
16
+
17
+ Flutter 的 podspec 是随 pub 插件消费的本地 podspec,不是独立 CocoaPods trunk 发布包;原生 iOS 使用根 Swift package。Flutter SwiftPM 的 `../FlutterFramework` 由 Flutter 的集成生成,不能当作本仓库的外部私有依赖,也不应将本机 Flutter framework 打包进插件。
18
+
19
+ Host 支持范围声明为 Node `>=26.3.0 <27`,本轮实际验证基线是 **26.3.0**。共享 runtime 与查询索引依赖 `node:sqlite`,native store 安装需要 node-gyp 所需的 Python 和 C/C++ 编译工具。未对其他 Node 版本或跨主版本兼容作实测声明。Python Script 另需可用的 `python3`,从实际 `script runtime-status` 读取环境能力。
20
+
21
+ ## 发布顺序
22
+
23
+ 1. 完成源码审阅并冻结一个提交,核对以下命令的产物确实来自它;包含当前 untracked 的实际源码、测试和文档,排除本机生成目录。所有候选对外版本使用同一个 `0.3.0-rc.2`,若需要改版本,先同时更新上表涉及的 manifest 与固定依赖。
24
+ 2. 维护者推送提交与 `0.3.0-rc.2` 标签,让 JitPack 构建 Android SDK/插件。确认两条公开坐标可解析后,再发布依赖它们的 Flutter 包。本地 Gradle project/path/AAR 替换不能证明 JitPack 坐标可消费。
25
+ 3. 原生 iOS 消费相同 Git tag 的根 package;完成根 package 的 iOS 构建,不仅构建 `ios/ai-app-bridge-ios/Package.swift`。Flutter iOS 则检查实际 pub 包内 Swift/C 源码与声明相符。
26
+ 4. CLI 与 Web SDK 可分别发布到 npm 的候选 dist-tag。CLI 的 native store 已打包随行,不等待一个不存在的单独 registry 依赖。Flutter 包发布以第 2 步完成为前提。
27
+ 5. 从 registry/tag 安装刚发布的确切版本,读取 `capabilities` 和版本,核对来源及支持范围,再按发布策略提升正式 dist-tag。候选发布不自动等于全平台生产验收完成。
28
+
29
+ 候选发布命令需在对应目录由维护者执行,例如 npm 使用 `npm publish --tag next`;pub 使用 `flutter pub publish`。这些命令属于发布动作,不能混入本地验证脚本。
30
+
31
+ ## 本地检查与最终包验证
32
+
33
+ 以下检查不发布版本。路径相对仓库根;输出使用新的、Git 忽略的目录。
34
+
35
+ ```sh
36
+ swift package --package-path . dump-package
37
+ swift package --package-path ios/ai-app-bridge-ios dump-package
38
+
39
+ cd desktop/ai-app-bridge-cli
40
+ npm pack --dry-run --json --ignore-scripts
41
+
42
+ cd ../../web/ai-app-bridge-web
43
+ npm pack --dry-run --json --ignore-scripts
44
+
45
+ cd ../../flutter/ai_app_bridge_flutter
46
+ flutter pub publish --dry-run
47
+ ```
48
+
49
+ `dump-package` 只证明 manifest 可解析及目标声明,不能替代 iOS 编译;`npm pack --dry-run` 只证明拟打包文件清单,不能替代安装;pub dry-run 中的分析/网络检查结果应原样记录。干净安装与 Host 协议验证必须在核心源码冻结、没有运行中修改时执行:
50
+
51
+ ```sh
52
+ cd desktop/ai-app-bridge-cli
53
+ npm ci
54
+ npm run verify:package -- ../../build/ai_app_bridge_artifacts/release-package-NEW
55
+ ```
56
+
57
+ `verify:package` 在仓库外安装实际 tarball,检查 native 安装编译、CLI/MCP 共享运行时、控制接口与随包运行时身份。它使用受控 ADB,不声称完成新真机业务验收。报告、tgz 哈希、安装日志和源码提交身份一起交接;已运行的旧包验证不能代替后来修改过的包。
58
+
59
+ CLI 的 `files` 已排除旧 `fact-cache.js` 发布载荷及 fake/P9/旧设备 adapter;旧 fact-cache 实现仅保留为 `test-support` 测试夹具,无生产引用。各 npm 包和 Flutter 目录的 `LICENSE`/`NOTICE` 均来自仓库根原文,发行时核对内容一致,不生成替代版权说明。