machine-bridge-mcp 2.0.0 → 3.0.0-beta.102

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (473) hide show
  1. package/CHANGELOG.md +1142 -0
  2. package/CONTRIBUTING.md +23 -25
  3. package/GOVERNANCE.md +2 -2
  4. package/README.md +58 -20
  5. package/SECURITY.md +161 -120
  6. package/SUPPORT.md +1 -1
  7. package/browser-extension/broker-auth.js +83 -0
  8. package/browser-extension/broker-liveness.js +35 -0
  9. package/browser-extension/browser-error-boundary.js +96 -0
  10. package/browser-extension/browser-operations.js +1456 -101
  11. package/browser-extension/devtools-input.js +195 -34
  12. package/browser-extension/devtools-observation.js +429 -0
  13. package/browser-extension/devtools-session.js +48 -0
  14. package/browser-extension/manifest.json +4 -3
  15. package/browser-extension/page-automation.js +578 -118
  16. package/browser-extension/pairing-bootstrap.js +42 -0
  17. package/browser-extension/pairing.js +28 -12
  18. package/browser-extension/service-worker.js +126 -106
  19. package/docs/AGENT_CONTEXT.md +13 -10
  20. package/docs/ARCHITECTURE.md +159 -67
  21. package/docs/AUDIT.md +1302 -4
  22. package/docs/CLIENTS.md +19 -6
  23. package/docs/COMPUTER_USE.md +297 -0
  24. package/docs/ENGINEERING.md +72 -13
  25. package/docs/GETTING_STARTED.md +18 -20
  26. package/docs/LOCAL_AUTHORIZATION.md +115 -72
  27. package/docs/LOCAL_AUTOMATION.md +27 -21
  28. package/docs/LOGGING.md +40 -19
  29. package/docs/MANAGED_JOBS.md +37 -34
  30. package/docs/MULTI_ACCOUNT.md +98 -37
  31. package/docs/OPERATIONS.md +145 -51
  32. package/docs/OVERVIEW.md +14 -11
  33. package/docs/POLICY_REFERENCE.md +71 -3
  34. package/docs/PRIVACY.md +41 -2
  35. package/docs/PROJECT_STANDARDS.md +29 -6
  36. package/docs/RELEASING.md +238 -64
  37. package/docs/TESTING.md +150 -39
  38. package/docs/THREAT_MODEL.md +164 -67
  39. package/docs/TOOL_REFERENCE.md +653 -60
  40. package/docs/UPGRADING.md +78 -18
  41. package/native/macos/MachineBridgeBackgroundInput.swift +1051 -0
  42. package/native/macos/MachineBridgeBackgroundInputSmokeFixture.swift +295 -0
  43. package/native/macos/MachineBridgeTrustBroker.swift +233 -0
  44. package/package.json +94 -16
  45. package/scripts/accepted-candidate-tarball.mjs +77 -0
  46. package/scripts/candidate-runtime-store.mjs +93 -0
  47. package/scripts/check-plan.mjs +73 -3
  48. package/scripts/check-runner.mjs +129 -17
  49. package/scripts/commit-message-check.mjs +5 -2
  50. package/scripts/consumer-package-security.mjs +293 -0
  51. package/scripts/coverage-check.mjs +251 -15
  52. package/scripts/coverage-generation.mjs +41 -0
  53. package/scripts/coverage-range-merge.mjs +41 -0
  54. package/scripts/foreground-daemon-recovery.mjs +88 -0
  55. package/scripts/generate-policy-reference.mjs +28 -5
  56. package/scripts/generate-tool-reference.mjs +1 -1
  57. package/scripts/generate-worker-types.mjs +21 -11
  58. package/scripts/github-backlog.mjs +11 -10
  59. package/scripts/github-push.mjs +62 -16
  60. package/scripts/github-release-asset.mjs +58 -0
  61. package/scripts/github-release.mjs +241 -132
  62. package/scripts/global-package-installation.mjs +68 -0
  63. package/scripts/hardened-npm-session.mjs +42 -0
  64. package/scripts/install-published-prerelease.mjs +198 -0
  65. package/scripts/local-release-acceptance.mjs +103 -45
  66. package/scripts/macos-background-input-smoke.mjs +515 -0
  67. package/scripts/network-retry.mjs +8 -1
  68. package/scripts/npm-global-prefix.mjs +27 -0
  69. package/scripts/npm-publication-policy.mjs +17 -0
  70. package/scripts/official-mcp-conformance.mjs +267 -0
  71. package/scripts/persistent-activation-process.mjs +101 -0
  72. package/scripts/prepare-pinned-npm.mjs +12 -69
  73. package/scripts/prerelease-activation.mjs +160 -0
  74. package/scripts/privacy-check.mjs +14 -6
  75. package/scripts/promotion-digest.mjs +128 -0
  76. package/scripts/publish-npm.mjs +260 -0
  77. package/scripts/published-release.mjs +132 -0
  78. package/scripts/release-acceptance.mjs +62 -54
  79. package/scripts/release-browser-extension-store.mjs +172 -0
  80. package/scripts/release-candidate-manifest.mjs +52 -0
  81. package/scripts/release-channel.mjs +106 -0
  82. package/scripts/release-diagnostic.mjs +41 -0
  83. package/scripts/release-impact-check.mjs +12 -22
  84. package/scripts/release-oauth-canary-core.mjs +325 -0
  85. package/scripts/release-oauth-canary-evidence.mjs +100 -0
  86. package/scripts/release-oauth-canary.mjs +127 -0
  87. package/scripts/release-publication-guard.mjs +67 -0
  88. package/scripts/release-soak.mjs +284 -0
  89. package/scripts/release-state.mjs +1 -1
  90. package/scripts/run-checks.mjs +44 -7
  91. package/scripts/run-worker-dry-run.mjs +21 -0
  92. package/scripts/sarif-security-gate.mjs +2 -1
  93. package/scripts/sbom-check.mjs +106 -0
  94. package/scripts/start-release-candidate.mjs +267 -44
  95. package/scripts/syntax-check.mjs +36 -14
  96. package/scripts/verification-generation-guard.mjs +15 -0
  97. package/scripts/verification-idle-sleep-guard.mjs +37 -0
  98. package/scripts/verification-state.mjs +108 -0
  99. package/scripts/wrangler-command-lifecycle.mjs +115 -0
  100. package/src/local/account-access.mjs +13 -1
  101. package/src/local/account-admin-response.mjs +92 -0
  102. package/src/local/account-admin.mjs +52 -19
  103. package/src/local/agent-context-projection.mjs +26 -7
  104. package/src/local/agent-context.mjs +88 -49
  105. package/src/local/agent-skill-discovery.mjs +16 -5
  106. package/src/local/app-automation-macos-jxa.mjs +350 -0
  107. package/src/local/app-automation.mjs +873 -163
  108. package/src/local/application-capability-projection.mjs +24 -0
  109. package/src/local/authority-context.mjs +106 -0
  110. package/src/local/autostart-log-maintenance.mjs +36 -0
  111. package/src/local/browser-bridge-http.mjs +5 -3
  112. package/src/local/browser-bridge.mjs +109 -31
  113. package/src/local/browser-broker-auth-http.mjs +67 -0
  114. package/src/local/browser-broker-auth.mjs +98 -0
  115. package/src/local/browser-broker-routes.mjs +63 -20
  116. package/src/local/browser-broker-server.mjs +17 -7
  117. package/src/local/browser-command.mjs +79 -30
  118. package/src/local/browser-computer-observation-service.mjs +266 -0
  119. package/src/local/browser-extension-identity.mjs +50 -0
  120. package/src/local/browser-extension-path.mjs +22 -0
  121. package/src/local/browser-extension-protocol.mjs +50 -11
  122. package/src/local/browser-operation-service.mjs +147 -117
  123. package/src/local/browser-pairing-grant.mjs +109 -0
  124. package/src/local/browser-pairing-http.mjs +36 -0
  125. package/src/local/browser-pairing-launch.mjs +58 -0
  126. package/src/local/browser-pairing-store.mjs +85 -48
  127. package/src/local/browser-request-registry.mjs +40 -14
  128. package/src/local/browser-request-settlement.mjs +78 -0
  129. package/src/local/browser-resource-input.mjs +74 -0
  130. package/src/local/browser-trusted-input-health.mjs +63 -0
  131. package/src/local/call-authority.mjs +21 -0
  132. package/src/local/call-capacity.mjs +45 -0
  133. package/src/local/call-registry-drain.mjs +54 -0
  134. package/src/local/call-registry.mjs +30 -15
  135. package/src/local/capability-observer.mjs +5 -0
  136. package/src/local/capability-ranking.mjs +3 -2
  137. package/src/local/child-process-settlement.mjs +132 -0
  138. package/src/local/cli-account-admin.mjs +40 -6
  139. package/src/local/cli-activate.mjs +162 -0
  140. package/src/local/cli-local-admin.mjs +20 -45
  141. package/src/local/cli-options.mjs +12 -13
  142. package/src/local/cli-ready-output.mjs +52 -0
  143. package/src/local/cli-service.mjs +91 -11
  144. package/src/local/cli.mjs +232 -150
  145. package/src/local/computer-use-application-observation.mjs +224 -0
  146. package/src/local/computer-use-arguments.mjs +270 -0
  147. package/src/local/computer-use-deadline.mjs +50 -0
  148. package/src/local/computer-use-dispatch-settlement.mjs +40 -0
  149. package/src/local/computer-use-expectation.mjs +171 -0
  150. package/src/local/computer-use-observation.mjs +769 -0
  151. package/src/local/computer-use-recovery.mjs +82 -0
  152. package/src/local/computer-use-result-budget.mjs +145 -0
  153. package/src/local/computer-use-snapshot-store.mjs +67 -0
  154. package/src/local/computer-use.mjs +2255 -0
  155. package/src/local/daemon-http-relay-auth.mjs +26 -0
  156. package/src/local/daemon-http-relay-connection.mjs +298 -0
  157. package/src/local/daemon-http-relay-request.mjs +64 -0
  158. package/src/local/daemon-http-relay-sequence.mjs +49 -0
  159. package/src/local/daemon-process.mjs +40 -7
  160. package/src/local/delegated-process-sandbox.mjs +148 -0
  161. package/src/local/device-identity.mjs +156 -49
  162. package/src/local/device-root-provider.mjs +81 -0
  163. package/src/local/directory-metadata.mjs +47 -0
  164. package/src/local/doctor-reporting.mjs +23 -0
  165. package/src/local/durable-process-spec.mjs +80 -0
  166. package/src/local/errors.mjs +25 -15
  167. package/src/local/exclusive-file.mjs +74 -26
  168. package/src/local/execution-limits.mjs +9 -0
  169. package/src/local/execution-routing.mjs +276 -0
  170. package/src/local/execution-surface.mjs +23 -0
  171. package/src/local/file-mutation-coordinator.mjs +47 -0
  172. package/src/local/file-snapshot-preservation.mjs +24 -0
  173. package/src/local/filesystem-identity.mjs +54 -0
  174. package/src/local/fixed-process-environment.mjs +21 -0
  175. package/src/local/full-access-test.mjs +31 -13
  176. package/src/local/git-commit.mjs +69 -0
  177. package/src/local/git-config-safety.mjs +55 -0
  178. package/src/local/git-log-parser.mjs +41 -0
  179. package/src/local/git-metadata-boundary.mjs +77 -0
  180. package/src/local/git-metadata-tree-safety.mjs +59 -0
  181. package/src/local/git-operation-state.mjs +19 -0
  182. package/src/local/git-service.mjs +165 -30
  183. package/src/local/hardened-npm-download-timeout.mjs +46 -0
  184. package/src/local/hardened-npm-download.mjs +109 -0
  185. package/src/local/hardened-npm-extract.mjs +20 -0
  186. package/src/local/hardened-npm-verification.mjs +104 -0
  187. package/src/local/hardened-npm.mjs +199 -0
  188. package/src/local/job-runner.mjs +244 -89
  189. package/src/local/lifecycle.mjs +6 -0
  190. package/src/local/log.mjs +19 -45
  191. package/src/local/macos-background-input.mjs +398 -0
  192. package/src/local/macos-idle-sleep-assertion.mjs +65 -0
  193. package/src/local/macos-trust-broker.mjs +404 -0
  194. package/src/local/managed-job-cancellation.mjs +42 -0
  195. package/src/local/managed-job-capacity.mjs +16 -0
  196. package/src/local/managed-job-directory-generation.mjs +60 -0
  197. package/src/local/managed-job-directory.mjs +75 -0
  198. package/src/local/managed-job-durable-process.mjs +49 -0
  199. package/src/local/managed-job-lock.mjs +46 -14
  200. package/src/local/managed-job-plan.mjs +38 -14
  201. package/src/local/managed-job-projection.mjs +23 -15
  202. package/src/local/managed-job-retention.mjs +128 -0
  203. package/src/local/managed-job-runner-claim.mjs +70 -0
  204. package/src/local/managed-job-runner.mjs +35 -10
  205. package/src/local/managed-job-storage.mjs +22 -6
  206. package/src/local/managed-job-terminal-maintenance.mjs +40 -0
  207. package/src/local/managed-job-terminal.mjs +150 -0
  208. package/src/local/managed-jobs.mjs +424 -287
  209. package/src/local/npm-cli.mjs +42 -0
  210. package/src/local/npm-environment.mjs +37 -0
  211. package/src/local/operation-authorization.mjs +68 -324
  212. package/src/local/operation-risk.mjs +51 -4
  213. package/src/local/owner-state-lock.mjs +118 -0
  214. package/src/local/package-identity.mjs +9 -0
  215. package/src/local/patch.mjs +20 -18
  216. package/src/local/path-inspection.mjs +23 -0
  217. package/src/local/policy.mjs +19 -0
  218. package/src/local/private-toolchain-integrity.mjs +54 -0
  219. package/src/local/process-execution.mjs +87 -75
  220. package/src/local/process-foreground-timeout.mjs +50 -0
  221. package/src/local/process-identity.mjs +41 -7
  222. package/src/local/process-nonreplayable-settlement.mjs +50 -0
  223. package/src/local/process-result-projection.mjs +22 -0
  224. package/src/local/process-session-events.mjs +47 -0
  225. package/src/local/process-session-remote-activity.mjs +10 -0
  226. package/src/local/process-session-termination.mjs +73 -0
  227. package/src/local/process-sessions.mjs +144 -112
  228. package/src/local/process-tracker.mjs +140 -13
  229. package/src/local/process-tree-ownership-types.d.ts +37 -0
  230. package/src/local/process-tree-ownership.mjs +57 -0
  231. package/src/local/process-tree-signal.mjs +34 -0
  232. package/src/local/process-tree-snapshot.mjs +77 -0
  233. package/src/local/process-tree-supervisor.mjs +69 -0
  234. package/src/local/process-tree.mjs +11 -53
  235. package/src/local/project-metadata.mjs +19 -6
  236. package/src/local/relay-call-recovery.mjs +58 -38
  237. package/src/local/relay-connection-classification.mjs +131 -0
  238. package/src/local/relay-connection-support.mjs +91 -0
  239. package/src/local/relay-connection.mjs +179 -272
  240. package/src/local/relay-diagnostics.mjs +91 -0
  241. package/src/local/relay-heartbeat.mjs +119 -0
  242. package/src/local/relay-liveness.mjs +109 -0
  243. package/src/local/relay-peer-diagnostics.mjs +29 -0
  244. package/src/local/release-runtime-lock.mjs +18 -0
  245. package/src/local/remote-activity-idle-sleep-guard.mjs +77 -0
  246. package/src/local/remote-configuration.mjs +48 -0
  247. package/src/local/resilient-relay-connection.mjs +178 -0
  248. package/src/local/resource-admission-diagnostics.mjs +52 -0
  249. package/src/local/resource-admission-policy.mjs +115 -0
  250. package/src/local/resource-admission.mjs +339 -0
  251. package/src/local/resource-build-root.mjs +83 -0
  252. package/src/local/resource-cargo-concurrency.mjs +38 -0
  253. package/src/local/resource-cmake-concurrency.mjs +42 -0
  254. package/src/local/resource-command-concurrency.mjs +37 -0
  255. package/src/local/resource-command-profile.mjs +176 -0
  256. package/src/local/resource-coordinator-accounting.mjs +21 -0
  257. package/src/local/resource-cpu-window.mjs +20 -0
  258. package/src/local/resource-disk-headroom.mjs +13 -0
  259. package/src/local/resource-elastic-memory.mjs +19 -0
  260. package/src/local/resource-elastic-request.mjs +36 -0
  261. package/src/local/resource-foreground-wait.mjs +29 -0
  262. package/src/local/resource-go-concurrency.mjs +43 -0
  263. package/src/local/resource-gradle-concurrency.mjs +57 -0
  264. package/src/local/resource-host-cache.mjs +39 -0
  265. package/src/local/resource-host-darwin.mjs +54 -0
  266. package/src/local/resource-host-linux.mjs +35 -0
  267. package/src/local/resource-host-sample-file.mjs +23 -0
  268. package/src/local/resource-host-snapshot.mjs +99 -0
  269. package/src/local/resource-lease-accounting.mjs +119 -0
  270. package/src/local/resource-light-command.mjs +6 -0
  271. package/src/local/resource-make-concurrency.mjs +42 -0
  272. package/src/local/resource-maven-concurrency.mjs +38 -0
  273. package/src/local/resource-ninja-command-concurrency.mjs +20 -0
  274. package/src/local/resource-ninja-concurrency.mjs +28 -0
  275. package/src/local/resource-operations.mjs +26 -10
  276. package/src/local/resource-pressure.mjs +67 -0
  277. package/src/local/resource-probe-command.mjs +35 -0
  278. package/src/local/resource-process-admission.mjs +59 -0
  279. package/src/local/resource-process-ancestry-cache.mjs +43 -0
  280. package/src/local/resource-process-ancestry.mjs +32 -0
  281. package/src/local/resource-process-priority.mjs +15 -0
  282. package/src/local/resource-project-key.mjs +53 -0
  283. package/src/local/resource-pytest-concurrency.mjs +33 -0
  284. package/src/local/resource-release-control-classification.mjs +14 -0
  285. package/src/local/resource-release-control-executable.mjs +70 -0
  286. package/src/local/resource-release-control-workspace.mjs +41 -0
  287. package/src/local/resource-request-contract.mjs +30 -0
  288. package/src/local/resource-script-classification.mjs +44 -0
  289. package/src/local/resource-shell-analysis.mjs +58 -0
  290. package/src/local/resource-staging-recovery.mjs +97 -0
  291. package/src/local/resource-swift-concurrency.mjs +31 -0
  292. package/src/local/resource-transaction-lock.mjs +189 -0
  293. package/src/local/resource-wait.mjs +48 -0
  294. package/src/local/resource-waiters.mjs +139 -0
  295. package/src/local/resource-xcode-command.mjs +7 -0
  296. package/src/local/resource-xcode-concurrency.mjs +34 -0
  297. package/src/local/resource-xcode-non-build.mjs +19 -0
  298. package/src/local/runtime-activation.mjs +700 -0
  299. package/src/local/runtime-activity-projection.mjs +48 -0
  300. package/src/local/runtime-capabilities.mjs +48 -15
  301. package/src/local/runtime-diagnostic-state.mjs +14 -0
  302. package/src/local/runtime-diagnostics.mjs +40 -13
  303. package/src/local/runtime-info-projection.mjs +60 -0
  304. package/src/local/runtime-info-relay-projection.mjs +21 -0
  305. package/src/local/runtime-paths.mjs +2 -1
  306. package/src/local/runtime-process-routing.mjs +33 -0
  307. package/src/local/runtime-relay-connection-options.mjs +44 -0
  308. package/src/local/runtime-relay-control.mjs +98 -0
  309. package/src/local/runtime-relay.mjs +26 -50
  310. package/src/local/runtime-reporting.mjs +37 -21
  311. package/src/local/runtime-resource-service.mjs +77 -0
  312. package/src/local/runtime-tool-handlers.mjs +13 -10
  313. package/src/local/runtime.mjs +244 -208
  314. package/src/local/secure-file.mjs +73 -20
  315. package/src/local/security-audit-dispatch.mjs +35 -0
  316. package/src/local/security-audit-log.mjs +215 -0
  317. package/src/local/security-audit-state.mjs +139 -0
  318. package/src/local/security-audit-storage.mjs +135 -0
  319. package/src/local/security-audit-warning.mjs +31 -0
  320. package/src/local/security-audit-worker.mjs +87 -0
  321. package/src/local/service-convergence.mjs +42 -1
  322. package/src/local/service-definition.mjs +24 -0
  323. package/src/local/service-environment.mjs +18 -10
  324. package/src/local/service-owner.mjs +146 -0
  325. package/src/local/service-ownership.mjs +17 -0
  326. package/src/local/service-restart-handoff.mjs +64 -0
  327. package/src/local/service-restart-scheduler.mjs +49 -0
  328. package/src/local/service-runtime-convergence.mjs +52 -0
  329. package/src/local/service-runtime.mjs +105 -0
  330. package/src/local/service-status.mjs +60 -0
  331. package/src/local/service.mjs +385 -92
  332. package/src/local/shell.mjs +39 -19
  333. package/src/local/ssh-key.mjs +158 -42
  334. package/src/local/state-inventory.mjs +50 -24
  335. package/src/local/state-owner-lock-inventory.mjs +23 -0
  336. package/src/local/state-root-owned-namespaces.mjs +83 -0
  337. package/src/local/state-root-retirement.mjs +89 -0
  338. package/src/local/state.mjs +386 -96
  339. package/src/local/stdio.mjs +96 -66
  340. package/src/local/support-state-projection.mjs +30 -0
  341. package/src/local/system-network-route.mjs +76 -0
  342. package/src/local/systemd-removal.mjs +20 -0
  343. package/src/local/tool-executor.mjs +85 -20
  344. package/src/local/tool-result-boundary.mjs +51 -0
  345. package/src/local/toolchain-operation-lock.mjs +41 -0
  346. package/src/local/tools.mjs +7 -5
  347. package/src/local/trusted-executable.mjs +53 -0
  348. package/src/local/trusted-git-executable.mjs +34 -0
  349. package/src/local/trusted-github-cli.mjs +24 -0
  350. package/src/local/windows-service-convergence.mjs +49 -0
  351. package/src/local/windows-service.mjs +52 -45
  352. package/src/local/worker-deployment-fingerprint.mjs +102 -0
  353. package/src/local/worker-deployment.mjs +48 -60
  354. package/src/local/worker-health.mjs +12 -4
  355. package/src/local/worker-secret-file.mjs +37 -17
  356. package/src/local/workspace-file-service.mjs +156 -239
  357. package/src/local/workspace-file-transaction.mjs +189 -0
  358. package/src/local/workspace-search.mjs +55 -0
  359. package/src/local/wrangler-toolchain/package-lock.json +1505 -0
  360. package/src/local/wrangler-toolchain/package.json +23 -0
  361. package/src/local/wrangler-toolchain-verification.mjs +135 -0
  362. package/src/local/wrangler-toolchain.mjs +188 -0
  363. package/src/shared/access-contract.json +7 -1
  364. package/src/shared/activation-recovery.mjs +49 -0
  365. package/src/shared/admin-auth.d.mts +2 -1
  366. package/src/shared/admin-auth.mjs +3 -3
  367. package/src/shared/authority-revocation.d.mts +9 -0
  368. package/src/shared/authority-revocation.mjs +28 -0
  369. package/src/shared/daemon-auth.d.mts +11 -0
  370. package/src/shared/daemon-auth.mjs +17 -0
  371. package/src/shared/device-session-auth.d.mts +20 -0
  372. package/src/shared/device-session-auth.mjs +51 -0
  373. package/src/shared/foreground-timeout.d.mts +15 -0
  374. package/src/shared/foreground-timeout.mjs +70 -0
  375. package/src/shared/log-redaction.d.mts +9 -0
  376. package/src/shared/log-redaction.mjs +51 -0
  377. package/src/shared/mcp-protocol.d.mts +27 -0
  378. package/src/shared/mcp-protocol.mjs +275 -0
  379. package/src/shared/project-overview-projection.d.mts +4 -0
  380. package/src/shared/project-overview-projection.mjs +96 -0
  381. package/src/shared/relay-contract.json +28 -0
  382. package/src/shared/result-projection.d.mts +2 -1
  383. package/src/shared/result-projection.mjs +13 -2
  384. package/src/shared/server-metadata.json +15 -9
  385. package/src/shared/tool-argument-validation.d.mts +17 -0
  386. package/src/shared/tool-argument-validation.mjs +325 -0
  387. package/src/shared/tool-call-capacity.d.mts +41 -0
  388. package/src/shared/tool-call-capacity.mjs +59 -0
  389. package/src/shared/tool-catalog.json +626 -57
  390. package/src/worker/access.ts +12 -0
  391. package/src/worker/account-admin.ts +55 -33
  392. package/src/worker/authority-revocations.ts +174 -0
  393. package/src/worker/authority.ts +52 -6
  394. package/src/worker/daemon-auth.ts +49 -68
  395. package/src/worker/daemon-channel.ts +36 -0
  396. package/src/worker/daemon-http-auth.ts +63 -0
  397. package/src/worker/daemon-http-channel.ts +120 -0
  398. package/src/worker/daemon-http-controller.ts +163 -0
  399. package/src/worker/daemon-http-protocol.ts +62 -0
  400. package/src/worker/daemon-http-registry.ts +82 -0
  401. package/src/worker/daemon-last-observation.ts +30 -0
  402. package/src/worker/daemon-liveness.ts +8 -6
  403. package/src/worker/daemon-ready-dispatch.ts +17 -0
  404. package/src/worker/daemon-ready-messages.ts +73 -0
  405. package/src/worker/daemon-ready-waiters.ts +79 -0
  406. package/src/worker/daemon-recovery-budget.ts +13 -0
  407. package/src/worker/daemon-registry.ts +58 -0
  408. package/src/worker/daemon-relay-diagnostics.ts +127 -0
  409. package/src/worker/daemon-socket-attachment.ts +55 -0
  410. package/src/worker/daemon-sockets.ts +72 -62
  411. package/src/worker/daemon-status.ts +38 -0
  412. package/src/worker/device-session-verifier.ts +129 -0
  413. package/src/worker/dpop.ts +157 -0
  414. package/src/worker/durable-process-timeout.ts +29 -0
  415. package/src/worker/errors.ts +57 -0
  416. package/src/worker/http.ts +112 -26
  417. package/src/worker/index.ts +446 -440
  418. package/src/worker/mcp-access.ts +57 -0
  419. package/src/worker/mcp-controller.ts +188 -0
  420. package/src/worker/mcp-http-accept.ts +11 -0
  421. package/src/worker/mcp-http-contract.ts +322 -0
  422. package/src/worker/mcp-initialization-compat.ts +141 -0
  423. package/src/worker/mcp-jsonrpc.ts +11 -29
  424. package/src/worker/mcp-removed-protocol.ts +17 -0
  425. package/src/worker/mcp-response-proxy.ts +135 -0
  426. package/src/worker/mcp-response-stream.ts +65 -0
  427. package/src/worker/mcp-stale-schema-compat.ts +57 -0
  428. package/src/worker/mcp-stream-proxy-contract.ts +48 -0
  429. package/src/worker/mcp-tool-call-input.ts +23 -0
  430. package/src/worker/nonce-store.ts +33 -35
  431. package/src/worker/oauth-authorization-page.ts +21 -1
  432. package/src/worker/oauth-client-admin.ts +52 -0
  433. package/src/worker/oauth-client-contract.ts +38 -0
  434. package/src/worker/oauth-controller.ts +107 -47
  435. package/src/worker/oauth-field-contract.ts +31 -0
  436. package/src/worker/oauth-record-contract.ts +7 -0
  437. package/src/worker/oauth-refresh-authority.ts +27 -0
  438. package/src/worker/oauth-refresh-exchange.ts +162 -0
  439. package/src/worker/oauth-refresh-families.ts +130 -27
  440. package/src/worker/oauth-refresh-persistence.ts +116 -0
  441. package/src/worker/oauth-state.ts +33 -18
  442. package/src/worker/oauth-store-validation.ts +131 -0
  443. package/src/worker/oauth-token-derivation.ts +33 -0
  444. package/src/worker/oauth-token-issuance.ts +119 -0
  445. package/src/worker/oauth-tokens.ts +56 -190
  446. package/src/worker/observability.ts +38 -5
  447. package/src/worker/pending-admission.ts +15 -0
  448. package/src/worker/pending-call-capacity.ts +108 -0
  449. package/src/worker/pending-call-contract.ts +42 -5
  450. package/src/worker/pending-call-deadlines.ts +75 -0
  451. package/src/worker/pending-calls.ts +131 -125
  452. package/src/worker/runtime-alarm-storage.ts +37 -0
  453. package/src/worker/runtime-alarm.ts +136 -0
  454. package/src/worker/server-info-activity.ts +36 -0
  455. package/src/worker/server-info-tool-delivery.ts +15 -0
  456. package/src/worker/server-info.ts +168 -0
  457. package/src/worker/tool-catalog.ts +47 -1
  458. package/src/worker/tool-timeout.ts +75 -14
  459. package/src/worker/websocket-protocol.ts +26 -2
  460. package/src/worker/worker-edge-guard.ts +90 -0
  461. package/src/worker/worker-edge-log.ts +65 -0
  462. package/src/worker/worker-entry.ts +52 -0
  463. package/src/worker/worker-mcp-config.ts +23 -0
  464. package/src/worker/worker-metadata.ts +48 -0
  465. package/src/worker/worker-rate-limit-key.ts +30 -0
  466. package/src/worker/worker-runtime-config.ts +20 -0
  467. package/src/worker/worker-static-routes.ts +54 -0
  468. package/src/worker/worker-task-lifetime.ts +18 -0
  469. package/tsconfig.local.json +17 -1
  470. package/wrangler.jsonc +13 -0
  471. package/src/local/cli-approval.mjs +0 -117
  472. package/src/local/operation-state-lock.mjs +0 -92
  473. package/src/worker/mcp-session.ts +0 -72
package/docs/CLIENTS.md CHANGED
@@ -52,9 +52,13 @@ It is therefore an optional compatibility and reuse surface, not a replacement f
52
52
 
53
53
  The MCP specification defines stdio and Streamable HTTP as standard transports. In stdio, the host launches the server as a subprocess and exchanges newline-delimited JSON-RPC through stdin/stdout. Streamable HTTP runs the server independently behind an HTTP endpoint.
54
54
 
55
+ Machine Bridge uses MCP `2026-07-28` as its native protocol and stdio is current-only. Native requests carry version and capabilities in `_meta`; `server/discover` replaces initialization, there is no MCP protocol session, and an HTTP response stream belongs to exactly one request. Remote Streamable HTTP additionally accepts the bounded stateless initialization-era dates `2025-06-18` and `2025-11-25` for `initialize`, `notifications/initialized`, `ping`, `tools/list`, and `tools/call`; those tool calls still execute through the current controller and never create `Mcp-Session-Id`, recovery GET, `Last-Event-ID`, replay state, or an initialization-owned authorization/session model. Any other removed-session behavior receives bounded upgrade guidance. Closing a native request stream cancels that request, and a connection, process, OAuth token, or stdio lifetime is never treated as conversation identity.
56
+
55
57
  ## Automatic capability selection
56
58
 
57
- MCP initialization and `resolve_task_capabilities` give the host a current view of conservative built-in working agreements, bounded project facts, explicit global/project instructions, skills, explicit/automatic package commands, applications, and browser capability. The resolver rescans rather than relying on a stale dynamic tool list and can return the best matching skill instructions in one call. No instruction file is required for the default layers, and no repository file is written automatically.
59
+ `server/discover` supplies conservative static server guidance, while `session_bootstrap` and `resolve_task_capabilities` explicitly refresh bounded project facts, authority-permitted instructions, skills, explicit/automatic package commands, applications, browser capability, and task-specific execution routes. Use the resolver when that refreshed context or route selection is material; straightforward file, Git, and shell work should use the already exposed tools directly rather than paying an extra discovery round trip. The resolver scores route bundles instead of forcing a single tool: registered commands, Bash/direct argv, process sessions, managed jobs, files/Git, browser, applications, resources, and diagnostics may appear together with ambiguity and fallback metadata. This advice never removes a policy-visible tool; direct Bash remains available under shell-capable authority.
60
+
61
+ The resolver always rescans task-specific metadata. A client that already holds `refresh.fingerprint` may send it as `known_refresh_fingerprint`; if the static instruction/skill/command identity is unchanged, the response omits that repeated material but still returns fresh skill/command matches, application results, ranked tools, and route advice. Capability results are filtered by effective account authority, so a restricted role does not receive hidden application/browser/shell metadata from a fuller daemon. No instruction file is required for the default layers, and no repository file is written automatically.
58
62
 
59
63
  The host still owns the agent loop. A hosted client may use the recommendation automatically, ask for confirmation, expose only part of the catalog, or ignore server instructions. No MCP implementation can guarantee automatic invocation from the server side. Machine Bridge models that limitation explicitly instead of treating a recommendation as execution.
60
64
 
@@ -126,7 +130,7 @@ Enter the printed `/mcp` URL in the remote MCP connector. During OAuth authoriza
126
130
 
127
131
  Supported configuration paths are:
128
132
 
129
- - **ChatGPT and Grok:** enter the `/mcp` URL in the corresponding remote connector. The Worker has exact built-in CORS response support for `https://chatgpt.com`, legacy `https://chat.openai.com`, `https://grok.com`, and the X-hosted Grok surface at `https://x.com`.
133
+ - **ChatGPT and Grok:** enter the `/mcp` URL in the corresponding remote connector. The Worker has exact built-in CORS response support for `https://chatgpt.com`, `https://grok.com`, and the X-hosted Grok surface at `https://x.com`.
130
134
  - **Claude custom connectors:** add the exact `/mcp` URL under Claude's connector settings and leave static OAuth client credentials empty so Claude can use DCR. Hosted Claude surfaces use `https://claude.ai/api/mcp/auth_callback`; the Worker accepts that exact registered redirect and rotates refresh tokens for the public client. See Anthropic's [remote custom connector guide](https://support.claude.com/en/articles/11175166-get-started-with-custom-connectors-using-remote-mcp) and [authentication contract](https://claude.com/docs/connectors/building/authentication).
131
135
  - **Microsoft Copilot Studio:** from the agent's **Tools** page, select **Add a tool > New tool > Model Context Protocol**, enter the `/mcp` URL, then select **OAuth 2.0 > Dynamic discovery**. Create the connection and add it to the agent. See Microsoft's [existing MCP server guide](https://learn.microsoft.com/en-us/microsoft-copilot-studio/mcp-add-existing-server-to-agent).
132
136
 
@@ -136,7 +140,7 @@ Several OAuth clients and named accounts can coexist. Accounts have independent
136
140
 
137
141
  ## Profile guidance
138
142
 
139
- - `full` is the default capability ceiling and prioritizes immediate usability. It is a canonical contract exposing every catalog tool, shell execution, unrestricted direct filesystem paths, absolute path output, and the full parent environment. Any individual narrowing is represented as `custom`. Remote high-impact transaction leases are a separate final authorization layer and do not change the profile identity.
143
+ - `full` is the default capability ceiling and prioritizes immediate usability. It is a canonical contract exposing every catalog tool, shell execution, unrestricted direct filesystem paths, absolute path output, and the full parent environment. Any individual narrowing is represented as `custom`. Remote accounts remain further constrained by immutable role, trusted-client, token-family, operation-invariant, and object-ownership checks.
140
144
  - `agent` retains file mutation and direct process execution but removes shell parsing, confines direct filesystem tools to the workspace, and isolates the process environment.
141
145
  - `edit` permits deterministic file mutation without process execution.
142
146
  - `review` is read-only and workspace-confined.
@@ -157,12 +161,21 @@ For SSH automation, prefer `generate_ssh_key_resource` under canonical full, or
157
161
 
158
162
  The local `full` profile controls Machine Bridge's own tool catalog, path resolver, path display, process environment, and shell availability. For remote transport, the local transaction gate additionally controls when high-impact effects may run. It does not control the MCP host's model policy, approval UI, connector gateway, or platform execution filters. See [LOCAL_AUTHORIZATION.md](LOCAL_AUTHORIZATION.md).
159
163
 
160
- Machine Bridge itself does not block files because their names look sensitive. In remote mode, first inspect `server_info.authorization.effective_policy` and `effective_tools`; `daemon.policy` is only the local ceiling. If the effective profile is `full` and the effective tool is present but a direct call is still rejected before a structured result, the host/connector may have blocked delivery. If `diagnose_runtime` responds but its fixed process or shell probe fails, the likely source is local OS policy, endpoint-security software, permissions, or shell configuration. Changing `--profile`, `--unrestricted-paths`, or `--absolute-paths` cannot override either layer.
164
+ Host-rendered tool invocation chips, status lines, or labels such as “called tool” are presentation owned by the MCP host, not Machine Bridge log records. Machine Bridge can suppress its own routine per-tool logs outside debug and can reduce unnecessary calls, but it cannot hide or restyle host-owned tool indicators. A host-side UI setting, if one exists, is the only layer that can change that presentation. Prefer fewer, coarser calls and avoid unnecessary capability-resolution calls when direct tools already suffice.
165
+
166
+ Hosts may also cache MCP discovery metadata and input schemas independently of the running server, and may keep conversation/surface app-routing state separately from the workspace-level app definition. For routine health checks, call `server_info` with `detail: "summary"`; it keeps the effective policy/count, account role without account ID, daemon readiness/relay state, pending/socket capacity, and foreground/settlement limits while omitting OAuth metadata, exact tool arrays, and full observability. The empty/default call returns the current `full` projection and is the cold path when an exact effective-tool list, account identity, or detailed counters are required. `server_info.tool_delivery` reports the current remote foreground maximum, process one-shot maximum, ordinary daemon-tool ceiling, process-poll wait ceiling, and Worker settlement overhead. If an already-open host still advertises older/larger values, refresh/review the workspace app or reconnect/recreate the connector so the host refreshes discovery; the Worker handles known maximum-only version skew as a structured no-side-effect tool error, but Machine Bridge cannot invalidate a host-owned schema/action cache from inside an already cached tool definition.
167
+
168
+ For routine workspace inventory, prefer `project_overview` with `detail: "summary"`. It keeps workspace/Git identity, effective and daemon policy/tool counts, compact capability-routing status, and up to 40 top-level names/types, while omitting account ID, exact tool arrays, routing fingerprints, and repeated entry paths/sizes. Request full detail only when those exact diagnostic fields are needed. As with `server_info`, summary is a presentation projection after authorization, not a lower-authority mode.
169
+
170
+
171
+ Machine Bridge itself does not block files because their names look sensitive. In remote mode, use the compact projection to inspect `server_info.authorization.effective_policy`; request full detail when exact `effective_tools` membership is required. `daemon.policy` is only the local ceiling. If the effective profile is `full` and the effective tool is present but a direct call is rejected before a structured result, Machine Bridge has no observation of that rejected request and cannot identify the cause from server-side evidence alone; conversation/surface app-routing state, a stale host action/tool snapshot, host tool filtering, connector-gateway policy, client routing, and platform policy are possibilities, not a diagnosis. Conversely, a successful `diagnose_runtime` response proves that request reached the local daemon through the currently used connector path and therefore does not support a blanket claim that Machine Bridge is currently disabled by the platform. A successful same-tool invocation in a fresh supported conversation is additional host-side evidence against a blanket outage when an older conversation still fails before dispatch; treat the difference as a host conversation/surface/snapshot issue until stronger evidence identifies the exact layer. If `diagnose_runtime` arrives but its fixed process or shell probe fails, the likely source moves downstream to local OS policy, endpoint-security software, permissions, shell configuration, or Machine Bridge policy. If an execution tool instead returns a child exit code plus bounded stdout/stderr, the child did start; for nested commands such as `ssh`, a remote forced-command usage screen or allowlist refusal in that output is target-side authorization evidence, not a local spawn or Machine Bridge policy denial. Changing `--profile`, `--unrestricted-paths`, or `--absolute-paths` cannot override host-side, operating-system, or remote-target enforcement.
172
+
173
+ Expected file-operation failures arrive as ordinary MCP tool-error results, not JSON-RPC transport failures. Clients should branch first on `structuredContent.error.code`, then optionally on the bounded `details.reason`. For example, `conflict/already_exists`, `conflict/hash_mismatch`, `conflict/text_ambiguous`, and `conflict/context_not_found` require a fresh read and reconciliation; `not_found/text_not_found` means the requested edit fragment is absent; `invalid_request` means the request or patch syntax must change. Do not log or display tool arguments to reconstruct diagnostics: public error details intentionally omit paths, file content, edit fragments, and compared hashes.
161
174
 
162
- Do not attempt to evade a host refusal by renaming, encoding, or switching to another arbitrary execution tool. Instead:
175
+ Remote configurable foreground tools advertise a 60-second maximum while preserving each tool’s 30- or 60-second default. That value bounds daemon execution; the Worker records its settlement deadline five seconds later. Admission and transport latency may consume part of that interval, and it is not a guarantee of host receipt. Missing or role-hidden tools, non-object arguments, and requests above that limit fail at the shared Worker schema boundary before daemon dispatch; schema failures include `side_effects_started=false`. SSE-capable current requests receive the same pre-dispatch validation as JSON responses and never allocate a recovery stream. Do not treat this as a retry invitation for the same oversized mutation, and do not attempt to evade a host refusal by renaming, encoding, or switching to another arbitrary execution tool. Instead:
163
176
 
164
177
  1. register credentials locally as resource aliases so their values never enter MCP arguments;
165
- 2. submit a complete `start_job` plan before the workflow depends on later cleanup calls, or use `stage_job` plus local `job approve` when execution-class tools are unavailable;
178
+ 2. submit a complete owner-authorized `start_job` plan before the workflow depends on later cleanup calls; `stage_job` is only a non-running draft, while an explicit local operator may use `machine-mcp job submit PLAN.json`;
166
179
  3. use job-scoped temporary files or remote stdin scripts;
167
180
  4. put idempotent cleanup in `finally_steps`;
168
181
  5. inspect/cancel through `machine-mcp job ...` if the host later denies tools.
@@ -0,0 +1,297 @@
1
+ # Computer Use
2
+
3
+ Machine Bridge Computer Use is a snapshot-bound control layer above the existing browser and desktop automation backends. It does not replace the browser broker, extension, macOS Accessibility implementation, or their low-level tools.
4
+
5
+ The design goal is to make GUI actions stateful and verifiable:
6
+
7
+ ```text
8
+ observe -> bind target to snapshot -> preflight -> dispatch -> verify -> observe post-state
9
+ ```
10
+
11
+ A successful input dispatch is not treated as proof that the requested user-visible or business effect occurred.
12
+
13
+ ## Tools
14
+
15
+ ### `computer_observe`
16
+
17
+ Creates one bounded observation and returns an opaque `snapshot_id`. `timeout_seconds` defaults to 30 seconds locally and on the hosted Worker and is one end-to-end observation budget: an application screenshot, Accessibility inspection, and fallback window revalidation consume the same monotonic deadline rather than each receiving a fresh timeout. Snapshot publication is the final observation settlement point: after the asynchronous browser/application capture finishes, cancellation is checked again before the snapshot enters the bounded store or consumes an id. A cancelled observe therefore cannot leave an actionable snapshot behind for a later `computer_act` call.
18
+
19
+ Browser observations contain:
20
+
21
+ - the existing executable DOM/actionability refs used by the trusted-input backend;
22
+ - a real Chromium Accessibility tree when the paired extension supports `computer_observation_v1`;
23
+ - DOMSnapshot-derived layout boxes, clickability, and paint order joined through `backendDOMNodeId`;
24
+ - conservative `action_ref` links from Accessibility nodes to executable DOM refs only when role/name/geometry matching is unambiguous;
25
+ - private snapshot bindings from those same **high-confidence** fused refs to Chromium `backendDOMNodeId` values. Correlation score and executable evidence are separate: `high` requires strong box overlap (IoU at least 0.7), no conflict between non-empty role/name evidence, and at least one exact role or name agreement. Center-point proximity or moderate overlap may still contribute to a useful `medium` observational correlation, but cannot authorize input; likewise, perfect geometry with a conflicting accessible name remains non-executable. A medium-confidence AX↔DOM correlation may remain visible as observational `action_ref` evidence, but it never receives an executable backend-node binding. The daemon independently requires `action_ref_confidence=high` before accepting any private backend id, so a stale or regressed extension cannot promote a medium heuristic match into trusted input. The raw backend ids are stripped from the public Computer Use observation; for `click`, `double_click`, `hover`, ref-to-ref `drag`, and ref-anchored `scroll`, the internal path can resolve `DOM.getContentQuads` to viewport coordinates and send trusted DevTools input even when a semantic endpoint is inside an accessible child frame;
26
+ - global cross-frame salience selection for executable DOM controls: each accessible frame contributes a bounded probe, then the aggregate `max_elements` budget is assigned to the highest-value controls across the whole page instead of being divided by frame traversal order;
27
+ - the selected tab identity, URL, document epoch, viewport, and frame tree;
28
+ - an optional native MCP screenshot, enabled by default and normally captured through `Page.captureScreenshot` in the same serialized DevTools session;
29
+ - explicit capture metadata describing navigation coherence and any legacy fallback;
30
+ - capability flags describing which grounding modes are actually implemented.
31
+
32
+ The Accessibility tree is used for richer semantic understanding; action dispatch still flows through constrained snapshot refs/trusted-input paths. For a fused semantic ref, Computer Use may privately use its snapshot-bound backend-node binding to obtain content quads; the public observation strips `backend_dom_node_id`, and callers cannot supply a backend id or raw CDP method. The low-level public `browser_action` contract is unchanged.
33
+
34
+ Application observations contain:
35
+
36
+ - one bounded macOS Accessibility inspection;
37
+ - snapshot-scoped refs (`a0`, `a1`, ...);
38
+ - process/frontmost state;
39
+ - optional window-local Accessibility geometry;
40
+ - a best-effort target-window PNG by default when Screen Recording/window capture is available;
41
+ - explicit native visual-point backend status (`visual_point_configured`, `visual_point_probed`, `snapshot_bound_visual_points`, and a bounded `visual_point_error_class`). A screenshot by itself never implies that native pixel dispatch is available.
42
+
43
+ Surface-specific arguments are strict. Browser-only arguments are rejected for application observations and application-only arguments are rejected for browser observations.
44
+
45
+ Snapshots are process-local, bounded to 64 recent entries, and expire after ten minutes measured by the daemon process monotonic clock. Wall-clock adjustment cannot extend or prematurely expire snapshot authority. They are intentionally not durable job state.
46
+
47
+ ### `computer_act`
48
+
49
+ Requires the exact `snapshot_id` produced by `computer_observe` plus the same `surface`. `timeout_seconds` defaults to 30 seconds locally and on the hosted Worker and is one end-to-end action budget spanning read-only preflight, the single mutation dispatch attempt, verification, and post-observation. Every internal stage receives only the remaining monotonic budget; a preflight deadline exhaustion remains a definite no-side-effect timeout and is checked before the snapshot is claimed for mutation, so that definite failure does not consume one-shot mutation authority. Exhaustion after mutation handoff cannot authorize replay and instead preserves dispatch/effect ambiguity plus re-observation guidance.
50
+
51
+ Page/application actions normally target a semantic snapshot ref. Browser `click`, `double_click`, `hover`, and `scroll` can additionally target a `normalized_viewport` point from the exact screenshot carried by the referenced snapshot, but only when that screenshot came from the navigation-coherent CDP capture path. Browser `drag` adds a required `destination` from the **same** snapshot. Its source and destination must use the same evidence class: ref-to-ref or point-to-point. Mixed ref/point drag is rejected instead of silently combining different grounding guarantees. Browser `scroll` takes bounded CSS-pixel `delta_x`/`delta_y` wheel deltas and requires at least one non-zero component; the target is the wheel anchor, so a ref can select a nested/virtualized scroll container while a point can target canvas or custom-drawn content. Raw CSS, arbitrary JavaScript, raw CDP methods, free-form AX selectors, device-pixel coordinates, and coordinates not bound to a Computer Use snapshot are not accepted.
52
+
53
+ The unified action schema is interpreted strictly by action, not merely by surface. Browser `value`/`value_resource` is accepted only for `fill`, `type_text`, and `select`; `key` is accepted only for `press`; `url` is accepted only for `navigate`; `destination` is accepted only for browser `drag`; `delta_x`/`delta_y` are accepted only for browser `scroll`; targetless navigation rejects target/input-only parameters instead of silently ignoring them. Application values are accepted only for `set_value` and `keystroke`, while `activate` rejects a target. The internal backend-node delegate repeats the sensitive-field checks so an accidental pointer action cannot resolve and forward a secret value even if an upper layer regresses.
54
+
55
+ Before dispatch:
56
+
57
+ - browser actions verify that the original tab still exists and has the same URL. Targetless `navigate`, `reload`, `back`, and `forward` also carry that observed URL privately through the daemon and re-check it in the extension immediately before the navigation mutation API, so a concurrent user navigation after daemon preflight wins instead of being overwritten;
58
+ - the top-frame per-document epoch must match whenever it is available, so a same-URL reload/replacement is rejected before input dispatch. `reload`, `back`, and `forward` additionally require both a successful epoch read and the private observed Navigation API history-entry key. The key binds the snapshot to the exact current history-list slot, closing same-document/same-URL traversal races that URL+document epoch cannot distinguish. Both authorities are carried privately through `BrowserOperationService`; for snapshot-bound history actions the fixed renderer module revalidates URL + document epoch + current Navigation API entry key, requires `navigation.canGoBack`/`canGoForward` for directional traversal, and synchronously invokes `navigation.reload()`, `navigation.back()`, or `navigation.forward()` in that same JS turn. This removes the prior service-worker await gap between history evidence collection and mutation dispatch. Missing/changed evidence, an unavailable requested history direction, or an unavailable Navigation API fails closed before dispatch, while loss of the renderer mutation response after invocation is classified as an unknown outcome and is never replayed through `chrome.tabs.*`. Ordinary non-Computer-Use browser history actions retain the generic `chrome.tabs` path. Neither the key nor other private authority fields are projected in the public Computer Use observation. Explicit `navigate(URL)` remains bound to the observed tab identity/URL but does not require the old document/history entry, because the requested destination is itself explicit;
59
+ - browser refs must still be attached;
60
+ - visual point actions require the same viewport dimensions/scale, an unoffset/unpinched visual viewport, and the same document epoch as the screenshot snapshot;
61
+ - browser drag preflights both endpoints. Point drag revalidates the exact screenshot once inside the trusted-input debugger session before any `Input.*` command. Ref drag independently revalidates both frame-document/ref identities and proves both backend-node content quads are already inside the current viewport before dispatch;
62
+ - browser scroll preflights its anchor with the same evidence strength. Point scroll revalidates the exact screenshot immediately before the wheel event. Ref scroll revalidates frame-document/ref identity and requires the backend-node content quad to already be inside the current viewport. It does **not** call `DOM.scrollIntoViewIfNeeded`: silently repositioning the anchor would be a separate mutation before the requested wheel effect;
63
+ - application refs are resolved back to their observed Accessibility identity and re-checked against the live UI;
64
+ - duplicate Accessibility identities preserve the observed occurrence index.
65
+
66
+ A Computer Use snapshot is one-shot mutation authority. Synchronous argument validation and read-only preflight happen before consumption, but immediately after preflight and before entering any browser/application mutation backend, `computer_act` atomically claims and removes that exact snapshot-store generation. Two concurrent acts can therefore inspect the same snapshot during read-only preflight, but only one can acquire dispatch authority; the other fails before mutation. Once the lower mutation handoff is attempted, the original snapshot is never restored, even if the backend reports a definite last-hop pre-dispatch failure or the later post-observation is cancelled. Continue only from the returned `post_snapshot_id`, or observe again. The existing `snapshot_missing_or_expired` conflict reason is retained for compatibility and also covers already-consumed snapshot authority.
67
+
68
+ After dispatch, Computer Use attempts a post-observation. To keep long trajectories bounded, browser post-observations capture 180 executable DOM elements and 180 Chromium Accessibility nodes by default, then return `post_observation_detail=summary` plus a bounded semantic delta. Application post-observations also default to a compact summary, retaining up to 24 salience-ranked snapshot refs plus the bounded semantic delta. The complete post observation is still retained behind `post_snapshot_id`; callers that genuinely need the complete tree can request `post_observation_detail=full`.
69
+
70
+ Post-action images use a separate `post_screenshot` policy. The default `auto` omits the image for an ordinary semantically verifiable action, but includes it for a snapshot-bound visual-point action, explicit `expect.visual_change`, or an ambiguous/unknown dispatch outcome where visual evidence can help recovery. `always` and `never` provide explicit control. The legacy `include_post_screenshot` boolean remains as a mutually exclusive compatibility override (`true` = `always`, `false` = `never`). This keeps image history from growing on every action while preserving screenshots exactly where they carry additional evidence. `post_screenshot_included` reports whether native image content is actually present in the returned result, not merely whether the selected policy requested an image.
71
+
72
+ Computer Use also applies the ordinary Machine Bridge tool-result byte ceiling **before publishing new snapshot authority**. If an otherwise valid observation would exceed that ceiling only because of its screenshot, the image is omitted, screenshot-bound point authority is revoked, and the still-bounded semantic snapshot is published. If a post-action image would overflow the result, the image is likewise omitted from both the returned post observation and the stored continuation snapshot while dispatch/effect settlement and `post_snapshot_id` are preserved. A still-oversized action result is compacted to bounded settlement/continuation/retry fields rather than discarding a completed mutation outcome or encouraging replay. `result_budget_compacted=true` and `post_screenshot_omitted_reason=tool_result_budget` identify this downgrade.
73
+
74
+ `observed_diff.semantic_delta` reports bounded added, removed, and changed controls. Browser delta identity is scoped by **top document epoch + extension frame id + that frame's own document epoch/URL + ref**. A same-URL child-frame reload therefore removes the old frame-document ref and adds the replacement ref even if both page worlds happen to assign the same string such as `e1`. Within an unchanged frame document, stable refs report mutable changes such as enabled/checked/focus/geometry and semantic-identity changes such as role/name/id separately. `observed_diff.frame_epoch_changed` reports whether the bounded extension-frame epoch map changed between snapshots (or `null` when frame coverage is insufficient). If the top document epoch changes, old refs are likewise treated as removed and new refs as added rather than pretending identity survived navigation. Application deltas use private selector-occurrence identity and also compare private owner-window bounds; an owner-window migration is surfaced only as `changed_fields:["owner_window"]`, never as global coordinates. Application `semantic_changed` is derived from this identity-aware delta rather than from public JSON alone, so a sibling-window migration cannot disappear merely because the visible role/name/state stayed the same. Sample lists are bounded even when counts are larger.
75
+
76
+ Every successful post-capture also returns a `continuation` record. Browser refs are **frame-document scoped**, not globally unique strings. `previous_target_ref_reusable=true` now requires the same top document epoch, the same target extension frame id with the same frame document epoch+URL, the same ref in that frame, and continued semantic identity. A sibling frame containing the same `e1`, a replacement child document that generates a new `e1`, or a same-object SPA repurpose such as `Save`→`Delete` cannot inherit the old target. `stable_ref_namespace` is true only when the top epoch and the complete bounded frame-epoch map both remain stable; target-specific `target_frame_epoch_same` and `target_identity_continues` explain the narrower reuse decision. The record also includes bounded added/changed/removed refs, delta truncation, and whether a fresh observe is recommended. If post-state capture fails, continuation is explicitly unavailable and re-observation is recommended. Application refs remain snapshot-scoped, so continuation never claims that old `aN` strings are reusable. Instead, when the acted target has a private semantic identity, continuation attempts a conservative mapping into the **new** post snapshot. `mapped_post_target_ref` is emitted only when application/process identity, private selector, original occurrence, and owner-window evidence continue; if either snapshot has owner-window evidence, both sides must have matching bounds. The same owner-window rule is used by post-condition lookup and private value readback, so a same-selector control in a sibling window cannot confirm the original target's focus/checked/value effect. Screen position is deliberately not part of cross-snapshot continuity so legitimate layout movement within the same owner window does not break identity. A salience reorder can therefore map old `a1` to new `a0`, while the old `a1` remains invalid. Target removal, sibling-window migration, or missing identity evidence yields no mapping and callers must replan from the post observation.
77
+
78
+ Browser `expect.target_state` uses the live wait only as a bounded convergence wait; the wait result alone is never final proof. The post snapshot must still show the target in the same frame document with the same semantic identity, and the requested visible/enabled/editable/checked state is read again from that post element. If the target is absent from an incomplete/truncated post projection, verification is `inconclusive` and the effect stays `unknown`. A changed frame epoch or semantic identity cannot confirm a positive target state. Explicit `detached`/`hidden` remains compatible with the old target actually disappearing. This prevents a recycled same-ref control from satisfying the original action's effect contract merely because it happens to expose the requested state.
79
+
80
+ Action results also return `retry_guidance`. It is deliberately conservative: `same_action_retry_allowed` is always false. A confirmed effect produces `disposition=do_not_retry`; a completed dispatch with no explicit effect contract also produces `do_not_retry` rather than treating missing verification as a reason to replay. An ambiguous dispatch or an unobserved expected effect uses `disposition=use_post_snapshot` only when a usable post snapshot exists; otherwise it returns `reobserve_before_retry`. When application continuation has a conservative target mapping, retry guidance repeats the **new** `mapped_post_target_ref` with `mapped_ref_for_replanning_only=true`; this is a pointer for inspecting/replanning from `post_snapshot_id`, never permission to replay the prior action. The `next_step` is either `continue_from_post_snapshot` or `computer_observe`, so callers do not have to infer recovery policy from free-form prose.
81
+
82
+ ## Dispatch status and effect status
83
+
84
+ `computer_act` reports two independent states.
85
+
86
+ `dispatch_status`:
87
+
88
+ - `completed`: the backend reported that its dispatch sequence completed;
89
+ - `unknown`: the backend may have started dispatching before an error or timeout made the final input outcome ambiguous.
90
+
91
+ `effect_status`:
92
+
93
+ - `confirmed`: the requested or intrinsic post-condition was observed;
94
+ - `not_observed`: dispatch completed, but the requested effect was not observed within the verification window;
95
+ - `unknown`: there was no verifiable post-condition, dispatch itself was ambiguous without confirming evidence, or post-state could not be observed.
96
+
97
+ An `unknown` result must not be automatically replayed. Re-observe before deciding whether another action is safe. When `dispatch_status=unknown`, the public `dispatch_error` is deliberately coarse: it identifies the surface and a bounded `error_class` but does not echo lower-layer exception text, paths, account-shaped data, or other backend diagnostics. Detailed causes remain local debugging evidence rather than part of a successful Computer Use result.
98
+
99
+ Post-observation verifier checks expose their deterministic evidence channel as `verification.post_checks[].evidence_source`. Stable values are `browser_tab_state`, `browser_semantic`, `application_process`, `application_workspace`, `application_accessibility`, and `screenshot`. The field identifies where the check was grounded; it does not increase confidence by itself, and screenshot evidence is not allowed to substitute for available semantic/process state. This keeps state-based verification distinguishable from visual fallback evidence when an agent decides how much trust to place in an observed effect.
100
+
101
+ Examples of intrinsic verification in v1:
102
+
103
+ - browser `check` expects the target to become checked;
104
+ - browser `uncheck` expects the target to become unchecked;
105
+ - application `activate` expects the process to become frontmost;
106
+ - application `focus` expects the target to become focused.
107
+
108
+ Callers can also provide explicit expectations.
109
+
110
+ Browser expectations:
111
+
112
+ - `url_contains`;
113
+ - `url_changed`;
114
+ - page `text`;
115
+ - `load_state`;
116
+ - `target_state` such as visible, enabled, checked, or detached;
117
+ - explicit `semantic_change`, which compares the semantic observation fingerprint;
118
+ - explicit `visual_change`, which compares SHA-256 fingerprints of the baseline and post-action screenshots.
119
+
120
+ Browser verification is deliberately two-stage. `url_contains`, page `text`, `load_state`, and `target_state` can be checked by the bounded browser wait backend; `url_changed`, `semantic_change`, and `visual_change` require the actual post observation/diff. A successful wait never substitutes for a pending post-observation condition. Until every requested post condition has actually been computed, verification remains `post_check_pending` and cannot produce `effect_status=confirmed`. If post capture fails after the wait succeeded, the effect is `unknown` and recovery requires a fresh observation rather than treating the missing diff/screenshot as success.
121
+
122
+ `visual_change` is intentionally opt-in. It requires screenshots in both the referenced and post-action observations and is useful for canvas/WebGL effects that do not produce a reliable DOM state change. Animation, video, cursors, clocks, or other incidental rendering can also change a screenshot, so visual change alone should only be requested when rendering change is the intended effect. The screenshot digest is computed locally from the actual returned image bytes and is included in capture metadata; no screenshot is synthesized when image capture failed.
123
+
124
+ Application expectations:
125
+
126
+ - `frontmost`;
127
+ - `target_exists`;
128
+ - `target_enabled`;
129
+ - `target_focused`;
130
+ - `target_checked`;
131
+ - `target_selected`;
132
+ - `target_expanded`;
133
+ - `target_value_matches` for exact private readback of a non-sensitive `set_value` target;
134
+ - explicit `semantic_change` and `visual_change`.
135
+
136
+ Application verification uses `verify_timeout_seconds` as a bounded read-only settlement window measured by a monotonic deadline; wall-clock adjustment cannot enlarge or collapse the verification budget. After the mutation has been dispatched exactly once, Computer Use may repeat only the post-action application observation while the requested condition is still unsettled or a post capture transiently fails. The verifier performs at most nine post-action captures with exponential 100/200/400/800 ms backoff (800 ms cap). Every post-action capture, including the first mandatory one, is capped by both the remaining `verify_timeout_seconds` window and the remaining overall `computer_act.timeout_seconds` deadline; no verification stage receives a fresh full action timeout after preflight or mutation work has already consumed time. A newer successful verification snapshot replaces and discards the prior internal retry snapshot, so only the final continuation snapshot remains addressable. The mutation is never replayed by this loop. Cancellation or a later verification/capture failure after dispatch stops further read-only verification and makes an incomplete result `unknown`; it never overwrites the already-known dispatch settlement or turns the issued mutation into a retryable cancellation. If an earlier unmatched snapshot exists but later captures become unavailable before the bounded verifier settles, that older snapshot is retained as continuation evidence but is not strong enough to claim `not_observed`. For `target_value_matches`, the retained exact-value handle remains one-shot: a missing post-target binding can be re-observed before comparison, but once exact AX value comparison is attempted the handle is not consumed again. Exact AX value readback is also capped to both remaining budgets, so a short `verify_timeout_seconds` or nearly exhausted overall action deadline cannot be followed by an unrelated full-length value-readback timeout.
137
+
138
+ ## Snapshot identity
139
+
140
+ Browser refs are stable backend refs, but Computer Use still binds them to a particular Computer Use snapshot. The wrapper recovers the correct `frame_id` from that snapshot rather than asking the model to carry frame coordinates independently.
141
+
142
+ Application refs are intentionally snapshot-scoped. The macOS Accessibility backend does not currently expose durable element handles, so Computer Use derives a selector from the observed Accessibility identity and uses an occurrence index only when duplicate identities exist.
143
+
144
+ Application process identity is also snapshot-scoped and is stronger than a PID. The private snapshot binds both the observed PID and an opaque process-generation token derived from the public macOS `proc_pidinfo(PROC_PIDTBSDINFO)` process-birth timestamp (`pbi_start_tvsec/pbi_start_tvusec`). The fixed JXA helper and fixed native input helper compute the same token from that ABI; there is no `launchDate`/`/bin/ps` coarse fallback. If the exact birth identity cannot be read, process-bound authorization fails closed. The token is never exposed in the public observation. Accessibility inspection, mutation, exact value readback, window recapture, and native visual input all require the generation to remain equal. The native pointer helper additionally retains the validated `NSRunningApplication` instance and compares it with the current same-PID instance before every gesture boundary, so a PID recycled between snapshot validation and a later down/up/drag/scroll event cannot inherit the old snapshot's authority.
145
+
146
+ Snapshot IDs are opaque and have no authorization meaning. Existing policy and operation-authorization checks still apply to every Computer Use call.
147
+
148
+ ## Security and authority
149
+
150
+ The Computer Use layer does not introduce arbitrary code execution.
151
+
152
+ It deliberately does not expose:
153
+
154
+ - caller-supplied browser JavaScript;
155
+ - caller-selected CDP methods;
156
+ - caller-supplied AppleScript/JXA;
157
+ - raw/unbound pixel clicks; snapshot-bound visual points use normalized viewport coordinates and only the fixed trusted-input path;
158
+ - a new secret transport.
159
+
160
+ `value_resource` is passed through to the existing browser/application protected-resource paths. Operation classification adds the existing `data-export` scope when a Computer Use action injects a registered local resource.
161
+
162
+ Browser Computer Use is classified under `browser-session`. Application Computer Use is classified under `application-control`.
163
+
164
+ ## Current limitations
165
+
166
+ ### Browser observation coherence
167
+
168
+ A current extension captures Chromium Accessibility, DOMSnapshot layout data, and the viewport screenshot inside one serialized `chrome.debugger` session. It also compares the top Page frame/loader before and after the CDP observation and revalidates the per-document extension epoch+URL for **every extension frame actually inspected** before accepting the complete capture. The frame recheck is one bounded batch over the observed frame ids rather than a fresh all-frame scan. A child iframe reload/replacement therefore invalidates the observation even when that frame returns to the same URL, preventing an old extension ref from being fused to a backend node from the replacement document.
169
+
170
+ A successful enhanced capture reports:
171
+
172
+ ```text
173
+ capture.cdp = true
174
+ capture.navigation_coherent = true
175
+ capture.frame_epochs_coherent = true
176
+ capture.screenshot_source = cdp_surface
177
+ capture.atomic = false
178
+ ```
179
+
180
+ `navigation_coherent=true` means no detected top-document replacement/navigation crossed the observation window. `atomic=false` remains deliberate: Accessibility, DOMSnapshot, and screenshot are sequential CDP commands, so ordinary in-document DOM/animation mutation can still occur between them. Computer Use retries one observation when a document replacement is detected and refuses the snapshot if coherence cannot be established.
181
+
182
+ The extension also performs a final `tabs.get` boundary check after semantic/CDP capture and the post-document epoch check. If the target tab has closed or become unreadable during observation, the entire snapshot is rejected instead of falling back to stale tab metadata captured at the beginning of the request. A returned browser snapshot therefore always refers to a tab that still existed at the final observation boundary; action-time preflight independently repeats tab existence, URL, document-epoch, and target checks.
183
+
184
+ The CDP observation is component-aware rather than all-or-nothing. `capture.cdp_components` records whether layout metrics, DOMSnapshot geometry, Accessibility, and the requested screenshot succeeded. If DOMSnapshot fails, Accessibility nodes are retained but may lack bounding boxes/action-ref fusion. If the CDP screenshot alone fails, the Accessibility tree is retained and the image can fall back to `captureVisibleTab` **only when the target tab is already the active tab in its window**. Computer Use never activates an inactive target merely to obtain this legacy screenshot: the fallback calls the screenshot helper in an internal non-disruptive mode that forbids tab switching. The target is checked as active again inside that helper; if the user switches before/during capture or the optional fallback capture otherwise fails, Computer Use discards the image, records a bounded `capture.screenshot_fallback_reason`, and keeps the already-valid semantic/CDP components subject to the normal final document/tab coherence checks. This avoids pairing a pre-activation semantic tree with rendering/page state changed by `visibilitychange` or focus handlers. Because any visible-tab fallback image is not the screenshot from the serialized CDP capture, `screenshot_source` is not `cdp_surface` and the snapshot is deliberately ineligible for visual point dispatch.
185
+
186
+ If the debugger cannot be attached at all (for example another debugger owns the target) or an older paired extension lacks `computer_observation_v1`, Computer Use falls back to the existing DOM semantic inspection. A visible-tab screenshot is added only under the same already-active rule above. The fallback is explicitly marked in `capture.fallback_reason`, an intentionally skipped image in `capture.screenshot_fallback_reason`, and the observation never pretends to contain a Chromium Accessibility tree. Because `captureVisibleTab` identifies only a window rather than a tab, the extension also proves that the requested tab is active immediately before **and** immediately after the capture; if the user switches tabs across that boundary, the image is discarded instead of being mislabeled as the requested tab. The separate explicit `browser_screenshot` tool may temporarily activate its requested tab because screenshot capture is the caller's direct goal, but it first requires a previous active-tab restore baseline and revalidates immediately before activation that the same baseline tab is still active. If the user switched tabs during preparation, the screenshot fails definite pre-dispatch and does not activate anything. Once the activation API has actually been invoked, a rejected activation response is ambiguous: capture stops, no rollback is attempted, and the caller must inspect the tab inventory before retrying because the activation may already have happened. After activation resolves successfully, the pre-capture active-tab check, capture, and post-capture active-tab check share one `try/finally` restoration lifecycle. A successful explicit screenshot takes its returned title/URL from that verified post-capture active-tab observation and reports `tab_metadata_verified=true`; it never reuses provenance resolved before capture. Restoration first re-reads the active tab: if the target is still active it restores the original baseline and verifies that restoration; if the user has switched to a third tab it preserves that user choice; if active-tab state cannot be verified or restoration fails, the request fails closed. Failure or uncertainty about capture identity or restoration is returned as a fixed safe error rather than silently returning pixels from the wrong tab or overwriting the user's browser state.
187
+
188
+ The new extension capabilities are feature-detected rather than made mandatory in the extension handshake. A daemon upgrade can therefore continue to operate with an older paired extension; reloading the newer packaged extension enables the enhanced CDP path.
189
+
190
+ ### Visual grounding and screenshot-bound points
191
+
192
+ Computer Use still reports `visual_grounding: false`: it does not ship or silently call a dedicated UI detector/model. A vision-capable caller can, however, use the screenshot returned by `computer_observe` and issue a snapshot-bound visual point for browser `click`, `double_click`, `hover`, point-to-point `drag`, or anchored `scroll`. The capability is advertised separately as `snapshot_bound_visual_points`.
193
+
194
+ Visual points deliberately use normalized viewport coordinates:
195
+
196
+ ```text
197
+ target.point = { x: 0.0..1.0, y: 0.0..1.0, space: normalized_viewport }
198
+ ```
199
+
200
+ They are accepted only when all of the following hold:
201
+
202
+ - the referenced observation actually returned a CDP surface screenshot;
203
+ - the observation detected no top-document navigation/replacement during capture;
204
+ - the top-frame document epoch is still identical immediately before dispatch;
205
+ - viewport width, height, and scale still match the snapshot;
206
+ - the visual viewport is not currently pinched/offset;
207
+ - the normalized point remains inside the current viewport;
208
+ - the exact screenshot SHA-256 still matches: inside the same serialized debugger attachment that will issue input, the trusted-input module recaptures the viewport with the original format/quality and rejects the action before any `Input.*` command when the bytes differ;
209
+ - trusted DevTools input is available. There is no DOM-event fallback for a point action.
210
+
211
+ Computer Use does not treat JavaScript scalar coercion as an alternate encoding of any of this authority. `surface`, `snapshot_id`, action, ref/resource names, input mode, waits, expectations, point coordinates/space, screenshot source/hash/quality, viewport dimensions, live tab URL/document epoch, and the private semantic-ref identity projection must arrive with the native JSON type produced by their source. For example `["click"]`, `["button-ref"]`, `[digest]`, `[0.5]`, `["complete"]`, or a string viewport width are invalid rather than equivalent to their scalar contents. The last-hop browser ref precondition separately type-checks string identity fields and Boolean sensitivity/shadow fields; malformed snapshot or live identity becomes stale before any page/CDP mutation.
212
+
213
+ Immediately before trusted input, the extension performs a fixed `elementFromPoint` probe and returns a redacted semantic description of the hit surface when available. The probe is diagnostic rather than a claim that the visual target has a semantic DOM equivalent; canvas/WebGL remains a valid reason to use the point path.
214
+
215
+ The preferred routing remains:
216
+
217
+ ```text
218
+ snapshot ref -> fused backend-node trusted input when available -> existing ref action path -> screenshot-bound visual point
219
+ ```
220
+
221
+ Drag is deliberately implemented as one trusted-input settlement, not as a sequence of public mouse calls. The browser extension emits a bounded `move -> press -> interpolated moves -> release` CDP sequence with the left-button bit held across intermediate moves. If any command after input starts fails or loses its response, the whole drag is an unknown outcome and is never replayed through another path. The trusted-input layer attempts a best-effort release after an interrupted drag so a failed request does not intentionally leave a virtual button held, but that cleanup does not make the business outcome definite. Ref drag also refuses to auto-scroll either endpoint: scrolling one endpoint can invalidate the other's snapshot geometry, so callers must first establish a snapshot where both endpoints are visible. This keeps the observe-to-act proof stronger than convenience-oriented drag helpers that silently mutate scroll state before the gesture.
222
+
223
+ Scroll follows the same settlement rule but is intentionally a **single** CDP wheel mutation. The caller supplies bounded CSS-pixel `delta_x`/`delta_y` and a same-snapshot anchor; `input_mode=dom` and `wait_for` are rejected before dispatch. The extension sends exactly one `Input.dispatchMouseEvent(type="mouseWheel")` at that anchor. There is no fallback to `Input.synthesizeScrollGesture`, JavaScript `scrollBy`, DOM scrolling, or a second wheel attempt after that command has started: a lost response means the scroll may already have happened, so the outcome is unknown and must be inspected before another wheel action. A semantic ref anchor revalidates the extension-frame epoch, stable ref identity, and backend-node content quad and must already be visible; Computer Use refuses to call `DOM.scrollIntoViewIfNeeded` because that would be a separate scroll mutation and could change which nested container receives the subsequent event. A point anchor instead revalidates document, viewport, and the exact screenshot in the same trusted-input debugger session immediately before the one wheel command. This makes positioned scroll useful for virtual lists, nested panes, canvas, and custom-drawn surfaces without weakening snapshot provenance or replay safety.
224
+
225
+ For accessible child frames, a fused backend-node binding removes the old top-frame-only trusted-click limitation: CDP `DOM.getContentQuads` supplies viewport-relative quads directly, so Computer Use does not guess iframe offsets or equate extension numeric frame ids with CDP frame ids. Each private binding also retains the observed extension frame's document epoch+URL. Immediately before any backend-node trusted action, the extension re-reads `documentState` in that exact extension `frameId`; a same-URL iframe reload/replacement therefore becomes stale **before opening the DevTools input session**. CDP frame ids remain provenance only and are never assumed equivalent to extension frame ids. The same private binding also supports `press`, `type_text`, and `fill`: text actions use fixed `DOM.focus` on the backend node and then the existing trusted DevTools Input path. `fill` runs select-all, Backspace, and `Input.insertText` inside one dispatch-tracking scope, so a failure after any Input command is an unknown outcome and cannot silently fall back/replay. Registered `value_resource` data is still resolved only inside the daemon's browser operation service; the Computer Use kernel carries the alias and the extension receives only the transient resolved value, exactly as with the existing browser secret-input path.
226
+
227
+ A stable extension ref is not treated as sufficient semantic identity by itself. An SPA or virtualized UI can keep the same DOM element object while repurposing it from, for example, a `Save` button into a `Delete` button without replacing the document. Computer Use therefore carries a private bounded identity projection containing only non-value target identity fields (`tag`, `type`, `role`, accessible `name`, `id`, field name, label, placeholder, sanitized `href`, sensitivity classification, and open-shadow-root membership). The page-world ref route compares that identity after actionability lookup but **before** any focus, scroll, click, or value mutation. A fused backend route first revalidates its extension-frame epoch and then uses the same persistent extension ref to run a mutation-free identity check **before opening the DevTools input session**. A mismatch is stale-snapshot evidence and is never replayed through another input path. Mutable state such as `value`, `checked`, `enabled`, focus, and `bounding_box` is deliberately excluded so legitimate state and layout changes do not redefine the target's identity.
228
+
229
+ The older top-frame ref route follows the same replay rule even though its preparation happens in the page world. `prepareAction` can focus keyboard targets or scroll pointer targets, so it is itself a side-effect boundary. In `auto` mode the extension checks that the fixed trusted-input module exists **before** calling `prepareAction`; only module absence at that point may fall back directly to one DOM action. Once page preparation has completed, debugger attach failure, trusted-input setup failure, cancellation, or any later trusted-input error is reported as an unknown/possibly partial dispatch even when the lower layer labels the Input portion safe to fall back. This prevents focus/scroll handlers from running once during preparation and then being followed by a replayed DOM action.
230
+
231
+ `wait_for` is strictly post-dispatch evidence, not part of deciding whether an action happened. Once navigation, a page action, a snapshot-bound backend action, or `fill_form` has returned from its mutating primitive, a later load/navigation wait timeout, cancellation, or tab closure is reported as `action outcome is unknown` and is never converted into trusted-input fallback or another DOM action. Computer Use therefore captures post-state and requires replanning instead of replaying the mutation merely because the requested wait condition was not observed.
232
+
233
+ The extension-to-page mutation boundary is tracked independently from the broker transport. Loading the fixed page module remains a mutation-free setup step, and cancellation is re-checked after that setup completes before the mutating page invocation is allowed to start. The page module carries an explicit internal version: reinjecting the same version preserves its document epoch and ref registry, while an older or unversioned isolated-world instance is replaced before use; the mutation wrapper also verifies the expected version before calling any page mutation API. The subsequent `scripting.executeScript` call that invokes `action`, trusted `prepareAction`, `fillForm`, or `uploadFiles` returns a small settlement envelope when the page function itself completes or throws. That preserves definite errors such as stale refs or invalid fields. If the scripting invocation itself rejects or returns no settlement after it has been attempted—for example because navigation destroys the execution context after a click—the page mutation may already have run, so the extension emits the fixed unknown-outcome signal and never automatically re-invokes the same page mutation.
234
+
235
+ Cancellation is also checked at each browser mutation-stage boundary, not only when the request first enters the extension. If cancellation arrives while an asynchronous target lookup or mutation-free preparation read is pending, the next mutation primitive is not invoked. In a multi-stage operation, already-completed stages are preserved rather than rolled back: cancellation after a tab has been activated stops a later window-focus mutation; cancellation after ref `prepareAction` can stop DevTools Input even though focus/scroll preparation may already have produced page effects. Snapshot-bound backend actions apply the same rule around asynchronous AX/geometry reads: cancellation during a pre-focus checked-state or off-screen geometry read prevents the later focus/scroll stage, while cancellation during the post-focus checked-state recheck suppresses the Space key but preserves the earlier focus as an unknown action outcome. Trusted DevTools sessions also gate cancellation after any same-tab session queue wait and before `chrome.debugger.attach`, so a cancelled queued request does not attach the debugger merely because an earlier request held the tab session. Top-frame trusted page actions and snapshot-bound visual point input carry the same gate through to the first `Input.*` command; visual point input additionally rechecks after screenshot-digest revalidation, so cancellation during that read-only verification cannot start trusted input. Explicit screenshot capture uses the transactional variant of the same rule: cancellation observed after confirmed temporary tab activation but before capture skips capture and still enters the restoration `finally`, restoring the prior active tab when safely verifiable. These checks minimize new side effects after cancellation without pretending that an earlier stage did not happen.
236
+
237
+ The same rule applies one transport layer lower. For mutating browser RPCs, once the owner request registry or a proxy broker route has attempted the WebSocket send to the extension, a missing response no longer means “not executed”: timeout, socket loss, broker/extension replacement, or response-delivery failure is surfaced as a fixed unknown mutation outcome. The extension applies the same rule after a successful mutation has returned internally: if that mutating success result cannot be JSON-serialized or fit within the bounded extension response budget, the action may already have completed, so the extension returns a fixed unknown-outcome settlement instead of a definite serialization/size error. Sanitized extension/broker mutation settlements are converted to public, non-retryable `BridgeError`s before the direct MCP error boundary, while arbitrary read-only transport details remain non-exposed. Computer Use does not infer ambiguity from generic text such as `timeout` or even an arbitrary sentence containing `outcome is unknown`; it accepts only the fixed browser/application mutation-settlement prefixes, captures post-state when those markers occur, and keeps same-action retry disabled. Read-only browser RPCs keep ordinary transport and result-limit failure semantics, and an explicit extension preflight/error settlement remains authoritative.
238
+
239
+ The Chrome navigation mutation APIs themselves use the same conservative boundary. Request validation and target-tab resolution happen before mutation, but once `tabs.update({url})`, `tabs.reload`, `tabs.goBack`, or `tabs.goForward` has been invoked, a rejected Promise is treated as an ambiguous dispatch rather than proof that navigation never started. The extension emits the same fixed unknown-outcome guidance, Computer Use captures post-state, and `same_action_retry_allowed` remains false. Obvious pre-invocation errors such as a missing navigation URL still fail before the mutation API is called.
240
+
241
+ Tab metadata read after a settled browser action is deliberately weaker evidence than the action settlement itself. Navigation, legacy ref actions, screenshot-bound point actions, fused backend-node actions, multi-field form fills, and file uploads all perform one final `tabs.get` only to attach current tab provenance. If that read succeeds, the result carries `tab_metadata_verified=true` with current tab metadata. If it fails after the mutating primitive and any requested post-dispatch wait have already succeeded, the action remains successful, the result retains the target `tab_id`, reports `tab_metadata_verified=false`, and omits title/URL/window/status rather than falling back to metadata captured before dispatch. Computer Use preserves this evidence; an older paired extension that does not provide the field remains distinguishable as `null` rather than being claimed verified or unverified. A metadata-only read failure therefore cannot retroactively convert a successful mutation into a same-action retry opportunity.
242
+
243
+ The DOM fallback path uses the same side-effect accounting instead of treating page-world JavaScript as inherently replay-safe. A direct DOM action starts with mutation-free actionability lookup; desired-state no-ops, select-option lookup, and keyboard-shortcut validation happen before scroll, focus, click, or value mutation. Once a side-effecting step begins, any later exception is converted to the fixed unknown-outcome contract. Trusted `prepareAction` also marks failures after its focus/scroll preparation as unknown. Multi-field form operations preserve that boundary across fields, and file-input assignment/event failures do the same, so an agent never replays a DOM mutation whose page handlers may already have run. Packaged actionability polling does not depend on background-page renderer timers: its bounded 50/100 ms waits are serviced through a same-extension runtime message whose service-worker endpoint accepts only the current extension identity and integer delays from 1 through 250 ms. Failure of that timing service does not bypass stable-box or pointer-hit checks; it remains definite before mutation and becomes unknown after any side-effecting preparation step. The page-local `setTimeout` fallback exists only when no extension runtime messaging API is present, such as the pure unit fixture.
244
+
245
+ If backend geometry is unavailable before any side-effecting trusted step, `auto` mode may fall back to the existing ref action path; explicit `trusted` mode fails closed. Focus-based actions no longer perform a preliminary scroll: `DOM.focus` itself is the no-fallback boundary because focus handlers can mutate application/page state even before a keyboard `Input.*` command is sent. Pointer actions take the complementary route: they first read `DOM.getContentQuads` plus `Page.getLayoutMetrics` without scrolling. A missing/invalid viewport metric is treated as unavailable geometry, **not** as evidence that the target is off-screen, so it cannot trigger a speculative scroll. If the bound node is proven inside the visual viewport, trusted pointer Input proceeds without changing scroll state. Only a point proven outside a valid viewport enters `DOM.scrollIntoViewIfNeeded`; that scroll is also treated as a no-fallback boundary because scroll handlers, sticky UI, virtualization, and lazy loading may mutate the page. After a successful or ambiguous scroll, geometry is re-read before Input, but any scroll ambiguity, cancellation, later geometry failure, or trusted-input failure is reported as an unknown/possibly partial dispatch and is never replayed through the DOM path. Before either focus or scroll has been attempted, missing geometry/trusted-input support remains a safe `auto` fallback; a known missing/detached backend node remains stale-before-dispatch. Fused `check/uncheck` is a desired-state action at both layers. The daemon deliberately does **not** short-circuit an already-satisfied trusted binding from its earlier page-state evidence: even a no-op must pass the extension's current frame-epoch and semantic-ref identity gates. The extension then starts its fixed DevTools session with a mutation-free `Accessibility.getPartialAXTree` read on the snapshot-bound backend node. If the desired state already holds, it returns `coordinate_source=cdp_ax_state_noop` / `no_input_required=true` **without calling `DOM.focus` or sending any trusted Input command**. Otherwise it focuses the node, which becomes the no-fallback boundary, and immediately repeats the checked-state read before Space. Only if the state is still opposite does it dispatch Space exactly once. If the state converges while focus is being applied, no key is sent but focus may already have fired page handlers; if the post-focus state is indeterminate, the outcome is unknown and the DOM path is never replayed. Post-state verification remains mandatory, so neither no-key path is itself treated as proof of the final effect. Fused `submit` uses focus + Enter. `select` continues to use the existing frame-scoped DOM selection path rather than adding a less deterministic keyboard-selection protocol.
246
+
247
+ A future dedicated visual grounder should emit the same snapshot-bound normalized-point representation rather than introducing a separate unversioned coordinate API.
248
+
249
+ ### Application screenshots and background input
250
+
251
+ macOS application observation now combines the existing Accessibility snapshot with a best-effort target-window PNG. The capture path is fixed and local: JXA gets the front Accessibility-window bounds/title and a bounded list of ordinary same-process CoreGraphics windows; the daemon matches those bounds so a front modal/dialog wins over a larger background main window. Window binding is fail-closed: the closest bounds must identify one unique CG window, or an exact front-window title must uniquely break an equal-bounds tie. If multiple windows remain indistinguishable, or several CG candidates exist while Accessibility cannot provide front-window bounds, Machine Bridge refuses screenshot binding rather than trusting CoreGraphics list order. `/usr/sbin/screencapture -x -o -l <window-id>` then captures that exact window without activating or raising the application. The temporary PNG lives in a fresh system temp directory, is size/magic-validated, read into memory, and deleted in a `finally` block.
252
+
253
+ The application screenshot/Accessibility pair is still reported as `atomic:false`; Machine Bridge does not claim a native atomic snapshot primitive that macOS does not provide. Instead, the first fixed JXA call binds the target window, the exact window screenshot is captured, and the second fixed JXA call performs Accessibility inspection **then** re-runs the same front-window selection algorithm before that JXA process returns. This preserves screenshot → AX → post-AX window ordering while avoiding a third `osascript` launch. A stable id/bounds pair yields `window_coherent=true` and `coherence=window_screenshot_then_accessibility_window_stable`. Backends that do not support inline post-AX window state retain the separate `inspectApplicationWindow` fallback; production macOS AppAutomation uses the inline path. Geometry collection also propagates each AX element's owning `AXWindow` bounds privately; window-local boxes are emitted only when that owner matches the captured window. This prevents a same-process sibling window that overlaps the screenshot in global coordinates from becoming a visual-point candidate. The owner-window screen bounds are stripped from the public projection. If the window changes during capture, the screenshot and AX text remain useful evidence but window-local AX geometry is removed, `window_coherent=false`, and both semantic-point and native pixel-point capabilities are disabled for that snapshot. If window revalidation itself is unavailable, the observation says `window_coherent=null` / `window_screenshot_then_accessibility_unverified`; semantic point delivery stays disabled rather than treating unverified geometry as authoritative. Screenshot failure degrades to the single AX inspection with `atomic:true` and `coherence=accessibility_only_after_screenshot_failure`.
254
+
255
+ Application snapshots bind a private macOS **process instance** rather than treating PID as a durable identity. The fixed JXA helper resolves the selected `System Events` `unixId` and binds that PID to the public libproc `PROC_PIDTBSDINFO` birth timestamp; its seconds/microseconds fields are encoded into the same opaque `proc:` generation token used by the fixed native input helper. A missing/failed libproc identity read is an authorization failure, not permission to fall back to second-resolution process metadata. Window selection, Accessibility inspection, post-AX window state, value verification, visual recapture, and native input must all match both PID and generation. These fields remain snapshot-private and are never projected in `computer_observe`, post observations, or ordinary `inspect_local_application` output. The native helper additionally retains the corresponding `NSRunningApplication`; transport classification consumes that retained instance instead of re-resolving by PID, SkyLight focus preparation rechecks the same instance after private PSN resolution and again before focus dispatch, and pointer-event boundaries revalidate it throughout the gesture. Even same-PID reuse therefore cannot inherit an old snapshot, transport choice, focus authorization, or in-flight gesture. If the process instance changes only after dispatch, post-state reports `process_changed=true`; effect verification refuses to let the replacement process confirm the old action, continuation does not map the old target, and an old-process exact-value handle is discarded without reading the replacement process.
256
+
257
+ The fixed macOS application helper requires both Accessibility authorization and macOS Automation permission to control `System Events`. Apple Events denial `-1743` is detected while resolving the target PID, before any application mutation is attempted, and is returned as a definite non-retryable `permission_denied` result rather than being misreported as a missing PID or ambiguous input outcome. Screenshot capture has the separate Screen Recording permission described below.
258
+
259
+ Application screenshot capture is a separate observation component. `computer_observe` attempts it by default when the application backend supports it, but failure never discards Accessibility state. Permission or capture failures are projected as fixed, path-free messages; Screen Recording permission may be required on macOS. Minimized/off-screen apps may have no capturable on-screen window. Application post-action observations intentionally disable screenshots by default so long trajectories do not accumulate redundant images.
260
+
261
+ Native visual point dispatch is **disabled by default**, even when a screenshot exists. The default macOS path remains Accessibility-first. An explicit experimental backend can be enabled with `MBM_MACOS_BACKGROUND_VISUAL_BACKEND=skylight-experimental`; ordinary runtime startup does not load or compile the helper or resolve private SkyLight symbols when that setting is absent. On first window capture after visual opt-in, the fixed helper is compiled into an owner-only runtime cache and probed once. Only a successful probe changes `snapshot_bound_visual_points` to true. The same fixed native binary is also used lazily for public PID-scoped keyboard delivery, but its `unicode_keystroke`/`key_press` operations bypass `SkyLightSymbols` construction entirely; using keyboard actions therefore does not opt into or probe the private visual backend. Probe failure degrades to screenshot + Accessibility observation and is surfaced through `visual_point_probed=true`, `snapshot_bound_visual_points=false`, and `visual_point_error_class`; it does not make observation fail. The native boundary itself accepts only an absolute runtime cache path, native finite coordinates/window bounds, native integer timeout/PID/window id, native string process generation/text/special key, and a native configured backend. Application screenshot point authority likewise requires an exact `macos_window` source, native PNG payload, native SHA-256/window id/bounds, and native normalized coordinates. A coercible array/object never enables the backend or reaches JXA/Swift input merely because `String(...)` or `Number(...)` would resemble the intended value.
262
+
263
+ Application screenshot points have a semantic-first delivery ladder. Computer Use claims `snapshot_bound_semantic_points=true` only when the snapshot's Accessibility candidate coverage is complete (neither the underlying AX traversal nor the salience projection was truncated) and the point lies inside exactly one enabled, addressable interactive Accessibility element. Before dispatch it re-captures and SHA-256-checks the exact window screenshot, performs a bounded full live AX reinspection with geometry, requires that coverage to remain complete, and recomputes all actionable same-window elements containing the point. The live point must still resolve uniquely to the exact selector/occurrence target from the snapshot; a newly overlapping Accessibility control therefore makes the snapshot stale even when the screenshot pixels did not change. Only then does Computer Use dispatch through the existing Accessibility action path (`coordinate_source=accessibility_point_resolution`). This works without enabling SkyLight and keeps visual targeting orthogonal to delivery. If coverage is incomplete, or zero/multiple actionable Accessibility elements contain the point, Computer Use does not guess: it uses the opt-in native pixel backend only when the original snapshot did not claim a semantic target and `snapshot_bound_visual_points=true`; an already semantic-bound point that becomes ambiguous at preflight is rejected as stale before input.
264
+
265
+ Snapshot-scoped application refs also preserve the source Accessibility traversal contract. The private snapshot records the observe-time `max_depth` and `include_menus`; ref preflight, semantic-point preflight, and AX dispatch reuse those exact settings so a duplicate selector's occurrence cannot be reinterpreted by a deeper/different traversal. `post_max_depth` and action-time `include_menus` affect only the newly captured post-action observation. Menu targets therefore must be observed with `computer_observe(include_menus=true)` before they can be addressed reliably.
266
+
267
+ When Accessibility geometry is available, application refs also retain the observed element's global `screen_box` and owning `AXWindow` bounds privately. Live preflight re-inspects geometry and rejects the snapshot if the nth selector has moved or changed owner window. The same expected owner/element bounds are then passed into the fixed JXA action helper; immediately before `AXPress`, focus, or value mutation, the helper recomputes the chosen element geometry and refuses dispatch if either bound changed. This turns occurrence/index into a snapshot-scoped identity aid rather than trusting it alone across sibling windows or reordered duplicate controls. The global bounds remain private and are never returned in the Computer Use element projection.
268
+
269
+ Discrete Accessibility state is read independently from optional content values. Computer Use always carries `checked`, `selected`, and `expanded` when the corresponding AX attribute has a definite Boolean state; unsupported state is `null`. Checkbox/radio `checked` is derived only from those roles' Boolean AX value, so `include_values=false` does not cause text fields or secure controls to be read for verification. Application expectations accept `target_checked`, `target_selected`, and `target_expanded`. A checkbox click with a known pre-state automatically expects the inverse checked state; a radio-button click expects checked=true; a disclosure-triangle click with a known expanded state expects the inverse. AX dispatch completion alone is insufficient: if the post observation does not show the expected state, the effect is `not_observed` rather than confirmed.
270
+
271
+ Application `check` and `uncheck` are stronger desired-state operations than `click`. `check` accepts an `AXCheckBox` or `AXRadioButton`; `uncheck` accepts an `AXCheckBox`. The snapshot must contain a definite checked state, and the fixed JXA helper re-reads that state at the last dispatch boundary after selector/window/element-geometry validation. If the desired state already holds, it returns `no_input_required=true` and sends no AXPress. Otherwise it sends exactly one AXPress and reports bounded `checked_before/checked_after` provenance. Computer Use always requires the corresponding post-state (`target_checked=true` for `check`, false for `uncheck`), even when the caller supplies additional explicit expectations; a contradictory expectation is rejected before input. Browser `check/uncheck` now applies the same non-weakenable desired-state expectation rule. Action matching itself runs with content-value collection disabled; discrete state readback is separate from optional text/value inspection.
272
+
273
+ The low-level Accessibility dispatcher validates text arguments before touching the application. Only `set_value` and `keystroke` accept `value`/`value_resource`; pointer, state, focus, and activation actions reject those fields before a registered resource is resolved, text actions reject a missing value before JXA runs, and `keystroke` additionally rejects an empty resolved string before mutation (while `set_value("")` remains a valid field-clear operation). Cancellation is re-checked after asynchronous registered-resource resolution and immediately before the fixed JXA mutation helper is invoked. Once a mutating Accessibility primitive starts, exceptions from activation, AXPress/click, focus, or AXValue assignment are reported as potentially partial dispatches. Keyboard actions deliberately use a two-stage boundary rather than posting CoreGraphics events from JXA. For `keystroke` and `key_press`, the fixed JXA helper performs only snapshot-bound selector/occurrence, owner-window/element-geometry, process-generation, and AX-focus preparation; the helper receives neither the Unicode text nor the special key. It returns the prepared PID/generation privately, and only after the target reports `focused=true` does the fixed native helper retain that exact `NSRunningApplication` and use public `CGEvent.postToPid`. Unicode text is split into at most 20 UTF-16 units without splitting a surrogate pair, and the native helper revalidates the retained process instance immediately before every key-down and again between each down/up boundary. Each down/up boundary uses a fixed 5 ms settlement gap; this is pacing, not replay, and the release fixture requires all three mixed CJK/emoji chunks to be consumed exactly once before the action is accepted as OS-level evidence. `key_press` maps only the documented layout-independent fixed key set plus optional Shift; command/control/option shortcut synthesis remains unsupported rather than guessed. Real inactive AppKit probes accepted mixed CJK/emoji text and fixed special keys while remaining non-key/non-active and keeping another process frontmost; multi-chunk delivery preserved order with zero activation. `input_transport=public-cgevent-pid` and `focus_prepared=true` are returned as dispatch provenance. Computer Use automatically requires the post-state `target_focused=true` for every ref-targeted `keystroke`, and a caller cannot weaken that intrinsic contract with `target_focused=false`; this verifies the deterministic focus preparation without pretending that arbitrary typed content itself was semantically confirmed. `key_press` does **not** inherit that postcondition because keys such as Tab/Shift+Tab intentionally transfer focus; its target is still required to be focused successfully before native dispatch, but any post-action effect must come from an explicit expectation or other evidence. A native helper response loss or failure after the first posted keyboard boundary remains an unknown, non-replayable overall keyboard outcome; a process-generation mismatch proven before the first native post never authorizes keyboard delivery. The same settlement boundary now extends across the local `osascript` transport: an explicit JXA preflight error remains definite, but after the mutating helper process invocation is attempted, process timeout/cancellation, missing output, or malformed JSON is an unknown action outcome because the Accessibility mutation may already have run. Fixed launch/Accessibility/visual unknown settlements are emitted as public, non-retryable `BridgeError`s so direct application-tool callers keep the inspect-before-retry instruction across the MCP error boundary; arbitrary JXA/preflight errors remain non-exposed. The Computer Use layer recognizes ambiguity only from an explicit allowlist of lower-layer mutation-settlement prefixes; generic words such as `timeout`, `osascript`, `accessibility helper`, or an unrelated sentence containing `outcome is unknown` are not independently treated as mutation evidence, so a resource or preflight failure before helper invocation remains definite. Computer Use therefore performs post-observation and effect verification instead of interpreting an exception after a mutation attempt as proof that nothing happened.
274
+
275
+ Non-sensitive application `set_value` also gains an intrinsic effect contract without returning the control value. Direct MCP `value` input is already immutable within the request, so Computer Use reuses that exact request value only inside the fixed read-only verification helper and does **not** duplicate it into retained-value storage; transient read-only verification transport failure may therefore be retried within `verify_timeout_seconds` without replaying the mutation. `value_resource` is different because re-resolving an alias could yield a different secret. For a resource-backed write, AppAutomation retains the exact value that was actually dispatched behind a bounded opaque `av_*` handle (maximum 32 live handles, 60-second monotonic crash/abandon fallback lifetime) **only when the live element returned by that same JXA dispatch explicitly reports `sensitive=false`**. The older snapshot's sensitivity classification is not sufficient: if the target becomes secure/sensitive between preflight and dispatch, no exact-value handle is retained. Computer Use never exposes the handle in MCP output. After the normal value-free post snapshot resolves the target again through its new private selector occurrence and geometry binding, a fixed internal `verify_value` JXA operation consumes that one-shot resource handle, compares the live AXValue locally, and returns only `supported/matched`; a second use of the handle fails. `computer_act` also runs idempotent best-effort cleanup in `finally`, so successful consumption, post-capture failure, target disappearance, and unsupported readback all release the retained resource value immediately during normal execution instead of waiting for TTL expiry. If a resource-backed write has an ambiguous dispatch and therefore no retained exact-value handle was returned, verification is deliberately inconclusive rather than resolving the resource alias again. A direct MCP `value` can still be safely compared after an ambiguous dispatch because the request value itself is the exact immutable input. The fixed JXA setter also classifies an exception from the AXValue assignment itself as a potentially partial dispatch rather than a definite pre-input failure. `target_value_matches=true` is mandatory for non-sensitive `set_value` even when additional expectations are supplied. If the private readback is supported and differs, the effect is `not_observed`; if readback is unavailable, effect remains `unknown`. Targets classified sensitive (including `AXSecureTextField`) do not retain or enable this intrinsic readback and reject explicit `target_value_matches`, so secret-field writes still require another safe external post-condition when effect confirmation is needed.
276
+
277
+ - macOS application observations can include a best-effort target-window PNG without activating the application; Screen Recording failure degrades to Accessibility-only observation.
278
+ - macOS application Accessibility elements include window-local geometry when a window capture is available. `focus_query` scans a bounded larger AX candidate set, ranks by task relevance/focus/actionability/visibility, and returns only the requested budget; sorted `aN` refs retain private bindings to the original selector occurrence so ranking cannot retarget duplicate controls.
279
+ - `focus_query` is evidence-bearing rather than merely a ranking hint: application and browser DOM observations report match count and whether the bounded search was exhaustive. Chromium AX reports matches too, but its depth-bounded tree conservatively reports `query_search_exhaustive=false`; a query miss therefore never proves absence unless the relevant observation channel explicitly says the search was exhaustive.
280
+ - opt-in macOS application visual click uses snapshot-bound normalized window coordinates. The snapshot privately binds process generation, CG window id/bounds, and screenshot digest. `AppAutomationManager.pointApplication` immediately recaptures and compares generation/SHA-256/id/bounds, rechecks cancellation after that read-only validation, and only then enters the fixed native helper. The helper revalidates the same process instance and front window, stamps window/pid routing fields, and selects **one** PID event transport for the whole click: native AppKit uses `CGEvent.postToPid` with subtype 0 and deliberately skips focus preparation, while Chromium/Electron/Catalyst families use `SLEventPostToPid` with subtype 3 and perform bounded focus-without-raise preparation only when the exact target application instance is not already frontmost. A skipped focus is reported as `focus_without_raise=false`; once any focus-preparation mutation is attempted, including a partial focus failure, the provenance becomes `focus_without_raise=true`. If event-source creation then fails without any prior focus mutation, settlement remains definite pre-dispatch rather than being promoted to an unknown mutation. Once focus or pointer dispatch actually starts, later failure is unknown and non-replayable. The single-transport rule is evidence-driven: broadcasting one logical event through both routes duplicates delivery. The selected route is exposed as `input_transport` (`public-cgevent-pid` or `skylight-pid`). Raw global screen coordinates and global `CGEventPost` are not exposed.
281
+ - opt-in macOS application `double_click` is deliberately point-only and uses the exact same screenshot/window/process binding as visual click. It never approximates a double click as repeated `AXPress`, even when the chosen pixel lies inside one uniquely addressable Accessibility control. `ComputerUseManager` calls `AppAutomationManager.pointApplication` once with fixed `click_count=2`; the native helper keeps one transport and one click group, sends one move followed by down/up pairs with click state 1 then 2, and revalidates the target front window before every button boundary. A failure after either button-down is an unknown overall double-click outcome; the helper attempts one matching best-effort mouse-up when needed, but cleanup is not treated as evidence that either click succeeded or failed. The live AppKit release smoke requires exactly two additional `mouseDown` deliveries whose final click-count sequence is `[1,2]`, while another application remains frontmost and the hardware cursor remains unchanged.
282
+ - opt-in macOS application visual drag uses the same private process-generation/window-id/bounds/screenshot binding as visual click, but requires both source and destination to be normalized points from that one window snapshot. It never guesses a semantic Accessibility drag fallback. `AppAutomationManager.dragApplication` recaptures and hashes the window, rechecks cancellation, projects both endpoints into one screen/window coordinate system, and invokes exactly one native drag helper. Native AppKit drag now uses the same public PID route as AppKit click and deliberately performs no focus preparation; live testing with correct out-of-event-stream `NSEvent.mouseLocation` cursor evidence delivers exactly one down, eight drags, and one up without changing the frontmost application or hardware cursor when the target view accepts first-mouse delivery. AppKit may intentionally withhold a first mouse from an inactive-window view whose own policy rejects it; Machine Bridge does not transiently activate the app to bypass that policy and does not claim view-level effect from native dispatch alone. Chromium/Electron/Catalyst retain the SkyLight route and focus preparation they require. The helper revalidates the same process instance and front window at every event boundary and never changes transport mid-sequence. Any failure after focus or pointer dispatch starts is unknown and non-replayable; after an interrupted post-down sequence the helper attempts one best-effort mouse-up cleanup without treating cleanup as effect evidence.
283
+ - opt-in macOS application visual scroll uses one normalized point from the same exact window screenshot plus bounded `delta_x`/`delta_y` values. Positive deltas keep the public Computer Use convention used by browser scroll (right/down); the native helper reverses those values only at the final macOS wheel mapping because CoreGraphics uses the opposite reveal-direction convention. `AppAutomationManager.scrollApplication` revalidates process generation, window id/bounds, screenshot SHA-256, and cancellation immediately before dispatch. Scroll keeps one fixed `skylight-pid` event transport because that route has the strongest live wheel evidence, but native AppKit targets skip focus preparation; Chromium/Electron/Catalyst retain it. The helper primes the same window-local point once, revalidates the same process instance/window, and emits one precise pixel-unit wheel event with no AX-center, keyboard, second-route, or foreground fallback. The release smoke uses a real nested `NSScrollView` and requires one precise wheel event, exact signed delta, visible post-state change, and the expected content-offset movement while the fixture never becomes active and the hardware cursor remains unchanged. Any uncertainty after actual focus/input begins is non-replayable and must be inspected from the post snapshot before another scroll.
284
+ - native visual focus/cleanup is concurrency-aware and non-disruptive best effort. SkyLight focus handling is decided from one captured `priorFront` application instance, and that exact frontmost instance is revalidated before the SkyLight path can continue. If the target was already frontmost, no focus mutation is sent but a concurrent user switch still causes a definite pre-dispatch failure. A background target is allowed to enter focus preparation only when there is a live, stable non-target `priorFront` restore baseline; a nil/terminated baseline fails before mutation. Before private focus preparation, `_SLPSGetFrontProcess` must resolve to the same PSN as `GetProcessForPID(priorFront.processIdentifier)`, while the retained target process instance and frontmost baseline are rechecked, so disagreement between AppKit/Workspace identity and SkyLight front-process state cannot authorize input. The helper then performs a last-hop front-process reread immediately before the first private defocus event. If that first mutation reports failure, it stops without attempting the second target-focus event; otherwise it re-resolves the target PID→PSN and retained process instance immediately before target-focus. At that second boundary, any concrete `NSWorkspace` frontmost application must still be either the captured `priorFront` or the target; a third application means the user won the race and target-focus is not sent. A temporarily nil frontmost value is tolerated during the helper's own defocus transition. The 50 ms focus-settlement delay runs only when focus mutation actually started, so AppKit/no-focus and already-frontmost paths do not pay it. After that conditional focus-settlement delay and before any pointer event source is used, SkyLight-framework paths revalidate the frontmost application without defeating focus-without-raise semantics: if no helper focus mutation ran, the already-frontmost target must still be frontmost; after background focus preparation, the current frontmost instance may be either the captured `priorFront` or the target, but never a third application. The same frontmost authorization is then combined with process-instance/window revalidation at every pointer boundary, including double-click button boundaries and every drag step, so a later user switch to a third application stops the remaining gesture rather than being ignored. Native AppKit uses the public framework policy and therefore keeps its proven non-frontmost background-input path. Settlement tracks focus dispatch and pointer dispatch independently: a boundary or event-construction failure that occurs before either mutation remains definite pre-dispatch, while any prior focus or pointer post makes the overall action unknown and non-replayable. If click/double-click/drag/scroll later becomes ambiguous after focus or pointer dispatch starts, the helper keeps the public result `dispatch_outcome_unknown`. Any required matching mouse-up is sent only while the original retained target `NSRunningApplication` is still the process behind that PID, so PID reuse cannot redirect release cleanup into a replacement process. The helper restores the previously frontmost application only when **this helper actually dispatched focus preparation** and the target application is still frontmost. Native AppKit click/drag/scroll skip focus preparation, so they never restore foreground state and cannot overwrite a concurrent user switch to the target app. Cleanup never converts an unknown mutation into success evidence, and a switch to any third application is never overwritten.
285
+ - macOS application `hover` is intentionally unsupported rather than emulated through an unreliable pointer fallback. A real AppKit tracking-area investigation showed that a PID-scoped synthetic `mouseMoved` can reach the exact target window and have `NSWindow.hitTest` resolve the intended view while `NSTrackingArea` still receives neither `mouseMoved` nor `mouseEntered`; the remaining AppKit tracking bookkeeping depends on state that per-process posting does not reliably reproduce. Computer Use therefore rejects application hover fail-closed instead of moving the user's hardware cursor, stealing foreground state, guessing an AX-center fallback, or reporting dispatch as semantic hover success. Browser hover remains supported through the browser trusted-input path.
286
+ - the SkyLight route is deliberately experimental because it depends on private macOS SPI and may break across OS releases. The helper source is fixed in the package, hashed before compilation, compiled lazily into an owner-only cache, and exposes no caller-selected symbol/event surface. Concurrent callers share an in-flight compiler setup, but cancellation is owned by the call that started that build: an uncancelled waiter does not inherit another call's cancelled `swiftc`, and instead rebuilds under its own context before probing. Capability probe results are cached when they settle as real availability/unavailability evidence, but a cancelled probe is propagated without updating that cache so one transient caller cancellation cannot make the backend appear persistently unavailable. Application launch, mutating Accessibility JXA, and native input helpers use the shared process runner's explicit non-replayable settlement boundary: cancellation, policy denial, or spawn failure proven before a child process starts remains definite and is never mislabeled as input ambiguity. After spawn, timeout/cancellation, process-error response loss, or a nonzero launcher/JXA exit is non-replayable unknown because side effects may already have occurred. The native pixel helper intentionally runs with `allowFailure=true`, so a nonzero helper exit still returns its structured `dispatch_started` settlement for more precise caller-side classification instead of being flattened into generic unknown. A manual disposable AppKit smoke harness is provided for release/OS validation and is not part of normal CI or startup.
287
+
288
+ The manual smoke requires two independent opt-ins and uses only a disposable accessory AppKit fixture:
289
+
290
+ ```sh
291
+ MBM_MACOS_BACKGROUND_VISUAL_BACKEND=skylight-experimental npm run macos-background-input:smoke -- --run
292
+ ```
293
+
294
+ The smoke compiles the fixed fixture into a fresh disposable `LSUIElement` `.app` and launches it through LaunchServices with `open -g`, because raw spawned AppKit binaries can own a WindowServer window before System Events exposes a usable Accessibility tree. To keep release validation aligned with the production surface, it tests **ComputerUseManager end to end only** instead of maintaining parallel release harnesses: coherent observation/process-generation binding, idempotent `uncheck`, `check` through the mapped post-snapshot ref, non-sensitive direct `set_value` with retry-safe private value readback, three-chunk mixed CJK/emoji `keystroke`, one exact `Shift+Tab` `key_press`, semantic-ref focus, raw visual click, exact double-click click counts `[1,2]`, exact 1/8/1 drag, and one verified scroll effect. The fixture records its own `applicationDidBecomeActive` count; it must never increase, the fixture must not be frontmost at completion, and every native visual gesture must preserve the hardware cursor. Unrelated user switches between other applications are allowed. Cleanup restores the original app only if the fixture itself is frontmost, terminates only the fixture-published PID, and removes the whole disposable directory. This single smoke is a deliberate OS/release check rather than normal CI.
295
+ - Application visual action results also report up to four snapshot-local `semantic_point_candidates` whose window-local AX boxes contain the chosen point, smallest region first. This is provenance only: an empty/ambiguous candidate set remains valid for canvas/custom UI and never substitutes for screenshot-bound dispatch or effect verification.
296
+ - Post-action screenshot policy is shared across browser and application surfaces: `auto` emits visual evidence only for visual-point actions, explicit `expect.visual_change`, or ambiguous dispatch; ordinary semantic/Accessibility actions stay image-free. Application `expect.visual_change` compares window-screenshot SHA-256 fingerprints and never treats visual change as success unless explicitly requested.
297
+ - Application post-state now defaults to a compact summary too. It retains up to 24 salience-ranked post-snapshot refs plus the bounded semantic delta, so long trajectories can continue from `post_snapshot_id` without replaying the full AX tree; `post_observation_detail=full` remains available explicitly.