pi-agent-browser-native 0.9.3 → 0.9.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +106 -5
- package/README.md +58 -791
- package/dist/extensions/agent-browser/index.js +43101 -2031
- package/dist/extensions/agent-browser/lib/argv-descriptor.js +49 -26
- package/dist/extensions/agent-browser/lib/argv-grammar.js +82 -93
- package/dist/extensions/agent-browser/lib/bash-guard.js +129 -126
- package/dist/extensions/agent-browser/lib/batch-lifecycle.js +112 -51
- package/dist/extensions/agent-browser/lib/browser-execution-claim-codec.js +61 -0
- package/dist/extensions/agent-browser/lib/browser-execution-claim-storage.js +232 -0
- package/dist/extensions/agent-browser/lib/browser-execution-claims.js +202 -0
- package/dist/extensions/agent-browser/lib/browser-execution-identity.js +77 -0
- package/dist/extensions/agent-browser/lib/browser-journal-branch.js +98 -0
- package/dist/extensions/agent-browser/lib/browser-journal-contracts.js +1 -0
- package/dist/extensions/agent-browser/lib/browser-journal-projection.js +221 -0
- package/dist/extensions/agent-browser/lib/browser-journal-reader.js +89 -0
- package/dist/extensions/agent-browser/lib/browser-journal-replay.js +246 -0
- package/dist/extensions/agent-browser/lib/browser-journal.js +103 -394
- package/dist/extensions/agent-browser/lib/browser-legacy-pages.js +220 -0
- package/dist/extensions/agent-browser/lib/browser-legacy-projection.js +282 -0
- package/dist/extensions/agent-browser/lib/browser-legacy-types.js +1 -0
- package/dist/extensions/agent-browser/lib/browser-session-conversion-stream.js +217 -0
- package/dist/extensions/agent-browser/lib/browser-session-conversion.js +80 -335
- package/dist/extensions/agent-browser/lib/browser-transcript-contracts.js +35 -0
- package/dist/extensions/agent-browser/lib/browser-transcript-validation.js +166 -0
- package/dist/extensions/agent-browser/lib/browser-transcript.js +137 -82
- package/dist/extensions/agent-browser/lib/browser-value-parsing.js +15 -0
- package/dist/extensions/agent-browser/lib/command-capabilities.js +340 -0
- package/dist/extensions/agent-browser/lib/command-policy.js +107 -88
- package/dist/extensions/agent-browser/lib/command-taxonomy.js +49 -322
- package/dist/extensions/agent-browser/lib/config-policy.js +138 -343
- package/dist/extensions/agent-browser/lib/config-providers.js +71 -0
- package/dist/extensions/agent-browser/lib/config-validation.js +231 -0
- package/dist/extensions/agent-browser/lib/config.js +20 -8
- package/dist/extensions/agent-browser/lib/electron/cdp.js +20 -8
- package/dist/extensions/agent-browser/lib/electron/cleanup.js +128 -83
- package/dist/extensions/agent-browser/lib/electron/discovery-darwin.js +109 -0
- package/dist/extensions/agent-browser/lib/electron/discovery-desktop.js +136 -0
- package/dist/extensions/agent-browser/lib/electron/discovery-evidence.js +90 -0
- package/dist/extensions/agent-browser/lib/electron/discovery-linux.js +93 -0
- package/dist/extensions/agent-browser/lib/electron/discovery-packages.js +162 -0
- package/dist/extensions/agent-browser/lib/electron/discovery-types.js +2 -0
- package/dist/extensions/agent-browser/lib/electron/discovery.js +140 -549
- package/dist/extensions/agent-browser/lib/electron/launch-monitor.js +102 -0
- package/dist/extensions/agent-browser/lib/electron/launch-process.js +209 -0
- package/dist/extensions/agent-browser/lib/electron/launch-types.js +2 -0
- package/dist/extensions/agent-browser/lib/electron/launch.js +261 -355
- package/dist/extensions/agent-browser/lib/executable-path.js +21 -12
- package/dist/extensions/agent-browser/lib/execution-cwd.js +35 -18
- package/dist/extensions/agent-browser/lib/fs-utils.js +1 -9
- package/dist/extensions/agent-browser/lib/input-modes/electron.js +162 -121
- package/dist/extensions/agent-browser/lib/input-modes/job.js +123 -317
- package/dist/extensions/agent-browser/lib/input-modes/lookup-network-evidence.js +116 -0
- package/dist/extensions/agent-browser/lib/input-modes/lookup-plan.js +132 -0
- package/dist/extensions/agent-browser/lib/input-modes/lookup-redaction.js +57 -0
- package/dist/extensions/agent-browser/lib/input-modes/lookup-source-evidence.js +151 -0
- package/dist/extensions/agent-browser/lib/input-modes/lookup-workspace.js +142 -0
- package/dist/extensions/agent-browser/lib/input-modes/lookups.js +58 -412
- package/dist/extensions/agent-browser/lib/input-modes/params.js +86 -40
- package/dist/extensions/agent-browser/lib/input-modes/qa-analysis.js +268 -0
- package/dist/extensions/agent-browser/lib/input-modes/qa-presentation.js +117 -0
- package/dist/extensions/agent-browser/lib/input-modes/script-child.js +396 -0
- package/dist/extensions/agent-browser/lib/input-modes/script-protocol.js +86 -0
- package/dist/extensions/agent-browser/lib/input-modes/script-types.js +1 -0
- package/dist/extensions/agent-browser/lib/input-modes/script-validation.js +60 -0
- package/dist/extensions/agent-browser/lib/input-modes/script.js +60 -409
- package/dist/extensions/agent-browser/lib/input-modes/semantic-action.js +218 -128
- package/dist/extensions/agent-browser/lib/input-modes/shared.js +15 -4
- package/dist/extensions/agent-browser/lib/input-modes/types.js +56 -7
- package/dist/extensions/agent-browser/lib/json-schema.js +22 -10
- package/dist/extensions/agent-browser/lib/launch-scoped-flags.js +17 -5
- package/dist/extensions/agent-browser/lib/managed-session-policy-lock.js +92 -425
- package/dist/extensions/agent-browser/lib/managed-session-posix-storage.js +58 -0
- package/dist/extensions/agent-browser/lib/managed-session-project-identity.js +244 -0
- package/dist/extensions/agent-browser/lib/managed-session-restore-context.js +84 -0
- package/dist/extensions/agent-browser/lib/managed-session-restore-policy.js +119 -0
- package/dist/extensions/agent-browser/lib/managed-session-restore-state.js +83 -0
- package/dist/extensions/agent-browser/lib/managed-session-restore.js +168 -300
- package/dist/extensions/agent-browser/lib/managed-session-snapshot-records.js +277 -0
- package/dist/extensions/agent-browser/lib/managed-session-snapshots.js +128 -305
- package/dist/extensions/agent-browser/lib/managed-session-storage.js +56 -253
- package/dist/extensions/agent-browser/lib/native-confirmation.js +140 -0
- package/dist/extensions/agent-browser/lib/native-output.js +43 -9
- package/dist/extensions/agent-browser/lib/orchestration/batch-stdin.js +106 -59
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/artifact-merge.js +14 -6
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/artifact-paths.js +109 -57
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/browser-observation-result.js +82 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/click-dispatch.js +164 -60
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/diagnostic-values.js +31 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/diagnostics.js +152 -781
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/electron-observation.js +203 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/electron-text-diagnostics.js +109 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/fill-diagnostics.js +119 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/final-result-action-recovery.js +147 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/final-result-actions.js +144 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/final-result-content.js +123 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/final-result-contracts.js +1 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/final-result-details.js +156 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/final-result-electron.js +187 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/final-result-evidence.js +39 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/final-result-missing-binary.js +89 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/final-result-recovery.js +97 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/final-result-redaction.js +106 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/final-result-timeout.js +110 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/final-result.js +41 -538
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/index.js +199 -154
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/interaction-diagnostics.js +290 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/managed-session-close.js +167 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/managed-session-daemon-inspection.js +108 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/managed-session-daemon-policy.js +127 -205
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/native-helper-observation.js +65 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/observation-types.js +1 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/overlay-diagnostics.js +120 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/prepare/network-page-filter.js +113 -30
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/prepare/scroll-shims.js +109 -28
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/prepare/snapshot-filter-data.js +150 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/prepare/snapshot-filter.js +147 -253
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/prepare/snapshot-rendered-text.js +147 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/prepare/wait-timeouts.js +59 -24
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/prepare-artifacts.js +159 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/prepare-command.js +131 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/prepare-contracts.js +1 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/prepare-daemon.js +93 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/prepare-failure.js +42 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/prepare-guards.js +242 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/prepare-input.js +75 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/prepare-observations.js +92 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/prepare-page-selection.js +254 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/prepare-page-state.js +121 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/prepare-refs.js +130 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/prepare-resources.js +114 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/prepare-semantic.js +121 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/prepare-session-plan.js +222 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/prepare-timeouts.js +68 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/prepare.js +176 -1123
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/process-output-analysis.js +191 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/process-output-confirmation.js +152 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/process-output-diagnostics.js +182 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/process-output-electron.js +150 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/process-output-lifecycle-contracts.js +1 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/process-output-lifecycle-phase-contracts.js +1 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/process-output-lifecycle-phase.js +115 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/process-output-lifecycle.js +256 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/process-output-managed-contracts.js +1 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/process-output-managed.js +256 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/process-output-native-phase-contracts.js +1 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/process-output-native-phase.js +93 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/process-output-native.js +113 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/process-output-navigation-contracts.js +1 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/process-output-navigation.js +316 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/process-output-ownership.js +28 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/process-output-page-phase-contracts.js +1 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/process-output-page-phase.js +142 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/process-output-presentation-contracts.js +1 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/process-output-presentation.js +333 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/process-output-publication-phase-contracts.js +1 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/process-output-publication-phase.js +71 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/process-output-publication.js +201 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/process-output-receipts.js +183 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/process-output-refs.js +200 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/process-output-tabs.js +142 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/process-output.js +18 -968
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/prompt-guards.js +23 -13
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/recording-receipt-reconciliation.js +198 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/recording-recovery-contracts.js +1 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/recording-recovery-presentation.js +154 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/recording-recovery.js +68 -151
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/screenshot-capture.js +14 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/screenshot-observation.js +169 -46
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/session-state-commands.js +130 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/session-state-electron.js +212 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/session-state-observations.js +61 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/session-state-outcomes.js +187 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/session-state-refs.js +169 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/session-state-tabs.js +228 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/session-state.js +77 -765
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/text-visibility-diagnostics.js +182 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/timeout-diagnostics.js +256 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/timeout-progress.js +108 -0
- package/dist/extensions/agent-browser/lib/orchestration/code-artifact-observation.js +39 -0
- package/dist/extensions/agent-browser/lib/orchestration/code-image-observation.js +119 -0
- package/dist/extensions/agent-browser/lib/orchestration/electron-host/branch.js +93 -0
- package/dist/extensions/agent-browser/lib/orchestration/electron-host/cleanup-result.js +58 -0
- package/dist/extensions/agent-browser/lib/orchestration/electron-host/cleanup.js +78 -0
- package/dist/extensions/agent-browser/lib/orchestration/electron-host/contracts.js +1 -0
- package/dist/extensions/agent-browser/lib/orchestration/electron-host/discovery.js +106 -0
- package/dist/extensions/agent-browser/lib/orchestration/electron-host/index.js +48 -821
- package/dist/extensions/agent-browser/lib/orchestration/electron-host/policy.js +86 -0
- package/dist/extensions/agent-browser/lib/orchestration/electron-host/probe-collection.js +68 -0
- package/dist/extensions/agent-browser/lib/orchestration/electron-host/probe-data.js +169 -0
- package/dist/extensions/agent-browser/lib/orchestration/electron-host/probe-presentation.js +182 -0
- package/dist/extensions/agent-browser/lib/orchestration/electron-host/probe.js +161 -0
- package/dist/extensions/agent-browser/lib/orchestration/electron-host/status.js +167 -0
- package/dist/extensions/agent-browser/lib/orchestration/execution-queue.js +22 -10
- package/dist/extensions/agent-browser/lib/orchestration/extension-artifact-preflight.js +211 -0
- package/dist/extensions/agent-browser/lib/orchestration/extension-artifacts.js +23 -0
- package/dist/extensions/agent-browser/lib/orchestration/extension-attachment-replay.js +99 -0
- package/dist/extensions/agent-browser/lib/orchestration/extension-branch-restore.js +275 -0
- package/dist/extensions/agent-browser/lib/orchestration/extension-code.js +280 -0
- package/dist/extensions/agent-browser/lib/orchestration/extension-command-admission.js +303 -0
- package/dist/extensions/agent-browser/lib/orchestration/extension-command-output.js +36 -0
- package/dist/extensions/agent-browser/lib/orchestration/extension-command-result.js +335 -0
- package/dist/extensions/agent-browser/lib/orchestration/extension-command-run.js +227 -0
- package/dist/extensions/agent-browser/lib/orchestration/extension-command-state.js +1 -0
- package/dist/extensions/agent-browser/lib/orchestration/extension-electron-ownership.js +204 -0
- package/dist/extensions/agent-browser/lib/orchestration/extension-execution-locks.js +139 -0
- package/dist/extensions/agent-browser/lib/orchestration/extension-execution-queue.js +24 -0
- package/dist/extensions/agent-browser/lib/orchestration/extension-host-coordination.js +98 -0
- package/dist/extensions/agent-browser/lib/orchestration/extension-host-run.js +295 -0
- package/dist/extensions/agent-browser/lib/orchestration/extension-intent-replay.js +41 -0
- package/dist/extensions/agent-browser/lib/orchestration/extension-invocation.js +7 -0
- package/dist/extensions/agent-browser/lib/orchestration/extension-lifecycle.js +102 -0
- package/dist/extensions/agent-browser/lib/orchestration/extension-managed-ownership.js +133 -0
- package/dist/extensions/agent-browser/lib/orchestration/extension-observation.js +120 -0
- package/dist/extensions/agent-browser/lib/orchestration/extension-prompt.js +180 -0
- package/dist/extensions/agent-browser/lib/orchestration/extension-recording.js +290 -0
- package/dist/extensions/agent-browser/lib/orchestration/extension-resource-contracts.js +1 -0
- package/dist/extensions/agent-browser/lib/orchestration/extension-resource-replay.js +269 -0
- package/dist/extensions/agent-browser/lib/orchestration/extension-result-state.js +113 -0
- package/dist/extensions/agent-browser/lib/orchestration/extension-runtime.js +146 -0
- package/dist/extensions/agent-browser/lib/orchestration/extension-script-leases.js +58 -0
- package/dist/extensions/agent-browser/lib/orchestration/extension-tool-boundary.js +62 -0
- package/dist/extensions/agent-browser/lib/orchestration/extension-version-check.js +54 -0
- package/dist/extensions/agent-browser/lib/orchestration/input-plan-types.js +1 -0
- package/dist/extensions/agent-browser/lib/orchestration/input-plan.js +261 -185
- package/dist/extensions/agent-browser/lib/orchestration/input-redaction.js +62 -0
- package/dist/extensions/agent-browser/lib/orchestration/native-config.js +144 -0
- package/dist/extensions/agent-browser/lib/orchestration/native-root-launch.js +77 -0
- package/dist/extensions/agent-browser/lib/orchestration/native-session-defaults.js +210 -157
- package/dist/extensions/agent-browser/lib/orchestration/operation-paths.js +124 -58
- package/dist/extensions/agent-browser/lib/orchestration/output-file.js +197 -99
- package/dist/extensions/agent-browser/lib/orchestration/script-mode.js +199 -96
- package/dist/extensions/agent-browser/lib/owned-managed-session-context.js +55 -0
- package/dist/extensions/agent-browser/lib/page-target-batch.js +118 -0
- package/dist/extensions/agent-browser/lib/page-target-guards.js +50 -0
- package/dist/extensions/agent-browser/lib/page-target-navigation.js +104 -0
- package/dist/extensions/agent-browser/lib/page-target-validation.js +55 -246
- package/dist/extensions/agent-browser/lib/parsing.js +13 -4
- package/dist/extensions/agent-browser/lib/pi-tool-rendering.js +133 -68
- package/dist/extensions/agent-browser/lib/playbook.js +16 -12
- package/dist/extensions/agent-browser/lib/process-child.js +284 -0
- package/dist/extensions/agent-browser/lib/process-environment.js +13 -6
- package/dist/extensions/agent-browser/lib/process-errors.js +23 -0
- package/dist/extensions/agent-browser/lib/process-identity.js +47 -15
- package/dist/extensions/agent-browser/lib/process-socket-storage.js +256 -0
- package/dist/extensions/agent-browser/lib/process-stdout.js +74 -0
- package/dist/extensions/agent-browser/lib/process.js +145 -536
- package/dist/extensions/agent-browser/lib/prompt-artifacts.js +287 -0
- package/dist/extensions/agent-browser/lib/prompt-policy.js +15 -194
- package/dist/extensions/agent-browser/lib/read-command.js +41 -0
- package/dist/extensions/agent-browser/lib/read-confirmation.js +131 -68
- package/dist/extensions/agent-browser/lib/recording-reservations.js +118 -45
- package/dist/extensions/agent-browser/lib/redaction-fields.js +75 -0
- package/dist/extensions/agent-browser/lib/redaction-json.js +149 -0
- package/dist/extensions/agent-browser/lib/redaction-url.js +88 -0
- package/dist/extensions/agent-browser/lib/results/action-contracts.js +1 -0
- package/dist/extensions/agent-browser/lib/results/action-recommendations.js +28 -271
- package/dist/extensions/agent-browser/lib/results/artifact-contracts.js +1 -0
- package/dist/extensions/agent-browser/lib/results/artifact-manifest-validation.js +133 -0
- package/dist/extensions/agent-browser/lib/results/artifact-manifest.js +96 -94
- package/dist/extensions/agent-browser/lib/results/artifact-recommendations.js +73 -0
- package/dist/extensions/agent-browser/lib/results/categories.js +111 -54
- package/dist/extensions/agent-browser/lib/results/confirmation.js +29 -13
- package/dist/extensions/agent-browser/lib/results/editable-ref-evidence.js +62 -27
- package/dist/extensions/agent-browser/lib/results/electron-recommendations.js +83 -0
- package/dist/extensions/agent-browser/lib/results/envelope-decoding.js +48 -0
- package/dist/extensions/agent-browser/lib/results/envelope.js +90 -67
- package/dist/extensions/agent-browser/lib/results/evidence-contracts.js +1 -0
- package/dist/extensions/agent-browser/lib/results/failure-recommendations.js +180 -0
- package/dist/extensions/agent-browser/lib/results/network-routes.js +86 -45
- package/dist/extensions/agent-browser/lib/results/network.js +35 -12
- package/dist/extensions/agent-browser/lib/results/next-actions.js +49 -19
- package/dist/extensions/agent-browser/lib/results/presentation/artifact-files.js +320 -0
- package/dist/extensions/agent-browser/lib/results/presentation/artifact-format.js +138 -0
- package/dist/extensions/agent-browser/lib/results/presentation/artifact-images.js +113 -0
- package/dist/extensions/agent-browser/lib/results/presentation/artifact-verification.js +145 -0
- package/dist/extensions/agent-browser/lib/results/presentation/artifacts.js +99 -555
- package/dist/extensions/agent-browser/lib/results/presentation/batch-contracts.js +1 -0
- package/dist/extensions/agent-browser/lib/results/presentation/batch-recording.js +118 -0
- package/dist/extensions/agent-browser/lib/results/presentation/batch-redaction.js +68 -0
- package/dist/extensions/agent-browser/lib/results/presentation/batch-rendering.js +135 -0
- package/dist/extensions/agent-browser/lib/results/presentation/batch-step.js +227 -0
- package/dist/extensions/agent-browser/lib/results/presentation/batch-summary.js +91 -0
- package/dist/extensions/agent-browser/lib/results/presentation/batch.js +91 -460
- package/dist/extensions/agent-browser/lib/results/presentation/browser-profile-recovery.js +9 -5
- package/dist/extensions/agent-browser/lib/results/presentation/command-presenters.js +216 -0
- package/dist/extensions/agent-browser/lib/results/presentation/common.js +26 -9
- package/dist/extensions/agent-browser/lib/results/presentation/content.js +86 -29
- package/dist/extensions/agent-browser/lib/results/presentation/diagnostic-a11y.js +67 -0
- package/dist/extensions/agent-browser/lib/results/presentation/diagnostic-health.js +79 -0
- package/dist/extensions/agent-browser/lib/results/presentation/diagnostic-identity.js +199 -0
- package/dist/extensions/agent-browser/lib/results/presentation/diagnostic-logs.js +75 -0
- package/dist/extensions/agent-browser/lib/results/presentation/diagnostic-network-actions.js +216 -0
- package/dist/extensions/agent-browser/lib/results/presentation/diagnostic-network.js +124 -0
- package/dist/extensions/agent-browser/lib/results/presentation/diagnostic-redaction.js +64 -0
- package/dist/extensions/agent-browser/lib/results/presentation/diagnostic-storage.js +209 -0
- package/dist/extensions/agent-browser/lib/results/presentation/diagnostic-stream.js +69 -0
- package/dist/extensions/agent-browser/lib/results/presentation/diagnostic-summary.js +222 -0
- package/dist/extensions/agent-browser/lib/results/presentation/diagnostics.js +103 -1052
- package/dist/extensions/agent-browser/lib/results/presentation/early-result.js +70 -0
- package/dist/extensions/agent-browser/lib/results/presentation/errors.js +130 -60
- package/dist/extensions/agent-browser/lib/results/presentation/input-contracts.js +1 -0
- package/dist/extensions/agent-browser/lib/results/presentation/large-output.js +60 -145
- package/dist/extensions/agent-browser/lib/results/presentation/navigation.js +147 -88
- package/dist/extensions/agent-browser/lib/results/presentation/notices.js +75 -0
- package/dist/extensions/agent-browser/lib/results/presentation/observation-contracts.js +1 -0
- package/dist/extensions/agent-browser/lib/results/presentation/observation-rendering.js +127 -0
- package/dist/extensions/agent-browser/lib/results/presentation/outcomes.js +120 -0
- package/dist/extensions/agent-browser/lib/results/presentation/output-completion.js +77 -0
- package/dist/extensions/agent-browser/lib/results/presentation/output-spill.js +51 -0
- package/dist/extensions/agent-browser/lib/results/presentation/registry.js +116 -240
- package/dist/extensions/agent-browser/lib/results/presentation/semantic-action.js +35 -21
- package/dist/extensions/agent-browser/lib/results/presentation/skills.js +109 -73
- package/dist/extensions/agent-browser/lib/results/presentation/source.js +109 -0
- package/dist/extensions/agent-browser/lib/results/presentation.js +17 -261
- package/dist/extensions/agent-browser/lib/results/recommendation-contracts.js +1 -0
- package/dist/extensions/agent-browser/lib/results/recording.js +52 -14
- package/dist/extensions/agent-browser/lib/results/recovery-actions.js +14 -5
- package/dist/extensions/agent-browser/lib/results/recovery-next-actions.js +30 -22
- package/dist/extensions/agent-browser/lib/results/selector-recovery-target.js +83 -0
- package/dist/extensions/agent-browser/lib/results/selector-recovery.js +119 -119
- package/dist/extensions/agent-browser/lib/results/snapshot-compact.js +266 -0
- package/dist/extensions/agent-browser/lib/results/snapshot-content.js +32 -0
- package/dist/extensions/agent-browser/lib/results/snapshot-high-value-controls.js +104 -71
- package/dist/extensions/agent-browser/lib/results/snapshot-refs.js +31 -11
- package/dist/extensions/agent-browser/lib/results/snapshot-segments.js +125 -66
- package/dist/extensions/agent-browser/lib/results/snapshot-spill.js +26 -7
- package/dist/extensions/agent-browser/lib/results/snapshot.js +28 -248
- package/dist/extensions/agent-browser/lib/results/text.js +16 -5
- package/dist/extensions/agent-browser/lib/runtime-args-validation.js +133 -0
- package/dist/extensions/agent-browser/lib/runtime-compatibility.js +81 -0
- package/dist/extensions/agent-browser/lib/runtime-contracts.js +1 -0
- package/dist/extensions/agent-browser/lib/runtime-redaction.js +118 -0
- package/dist/extensions/agent-browser/lib/runtime-replay-events.js +144 -0
- package/dist/extensions/agent-browser/lib/runtime-session-identity.js +76 -0
- package/dist/extensions/agent-browser/lib/runtime-session-plan.js +114 -0
- package/dist/extensions/agent-browser/lib/runtime-session-replay.js +123 -0
- package/dist/extensions/agent-browser/lib/runtime-tab-correction.js +65 -0
- package/dist/extensions/agent-browser/lib/runtime-timeouts.js +28 -0
- package/dist/extensions/agent-browser/lib/runtime.js +137 -1041
- package/dist/extensions/agent-browser/lib/session-identity.js +51 -0
- package/dist/extensions/agent-browser/lib/session-page-observation.js +357 -0
- package/dist/extensions/agent-browser/lib/session-page-state.js +178 -333
- package/dist/extensions/agent-browser/lib/session-page-types.js +1 -0
- package/dist/extensions/agent-browser/lib/temp-artifact-budget.js +81 -0
- package/dist/extensions/agent-browser/lib/temp-mutation-queue.js +7 -0
- package/dist/extensions/agent-browser/lib/temp-root-ownership.js +234 -0
- package/dist/extensions/agent-browser/lib/temp-root.js +211 -0
- package/dist/extensions/agent-browser/lib/temp.js +33 -446
- package/dist/extensions/agent-browser/lib/tool-surface.js +94 -24
- package/dist/extensions/agent-browser/lib/upstream-version.js +3 -2
- package/dist/extensions/agent-browser/lib/web-search-providers.js +193 -0
- package/dist/extensions/agent-browser/lib/web-search-request.js +102 -0
- package/dist/extensions/agent-browser/lib/web-search-results.js +281 -0
- package/dist/extensions/agent-browser/lib/web-search-schema.js +82 -0
- package/dist/extensions/agent-browser/lib/web-search.js +124 -614
- package/dist/extensions/agent-browser/lib/webmcp-observation.js +13 -6
- package/dist/extensions/agent-browser/lib/windows-stock-launcher.js +63 -37
- package/dist/extensions/agent-browser/script-worker.js +80 -43
- package/dist/scripts/agent-browser-target.mjs +4 -2
- package/docs/ARCHITECTURE.md +19 -6
- package/docs/CODE_QUALITY.md +267 -0
- package/docs/COMMAND_REFERENCE.md +272 -241
- package/docs/ELECTRON.md +77 -46
- package/docs/REFERENCE.md +869 -0
- package/docs/RELEASE.md +25 -22
- package/docs/REQUIREMENTS.md +2 -1
- package/docs/SUPPORT_MATRIX.md +81 -79
- package/docs/TOOL_CONTRACT.md +158 -125
- package/docs/platform-smoke.md +8 -8
- package/package.json +144 -120
- package/scripts/agent-browser-capability-baseline.mjs +361 -957
- package/scripts/agent-browser-target.mjs +6 -2
- package/scripts/build.mjs +37 -8
- package/scripts/capability-core-commands.mjs +193 -0
- package/scripts/capability-options.mjs +283 -0
- package/scripts/capability-samples.mjs +133 -0
- package/scripts/capability-section.mjs +11 -0
- package/scripts/config.mjs +235 -132
- package/scripts/convert-browser-session.mjs +30 -9
- package/scripts/doctor-sources.mjs +195 -0
- package/scripts/doctor.mjs +94 -249
- package/scripts/platform-smoke/artifacts.mjs +38 -11
- package/scripts/platform-smoke/commands.mjs +104 -0
- package/scripts/platform-smoke/crabbox-runner.mjs +167 -80
- package/scripts/platform-smoke/doctor-support.mjs +157 -0
- package/scripts/platform-smoke/doctor-windows.mjs +230 -0
- package/scripts/platform-smoke/doctor.mjs +212 -320
- package/scripts/platform-smoke/lease-evidence.mjs +132 -0
- package/scripts/platform-smoke/linux-image/Dockerfile +8 -2
- package/scripts/platform-smoke/suite-checks.mjs +98 -0
- package/scripts/platform-smoke/suite-evidence.mjs +281 -0
- package/scripts/platform-smoke/targets.mjs +154 -472
- package/scripts/platform-smoke.mjs +55 -40
- package/scripts/prepare.mjs +16 -7
package/docs/TOOL_CONTRACT.md
CHANGED
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
# Tool contract
|
|
2
2
|
|
|
3
3
|
Related docs:
|
|
4
|
+
|
|
4
5
|
- [`../README.md`](../README.md)
|
|
5
6
|
- [`REQUIREMENTS.md`](REQUIREMENTS.md)
|
|
6
7
|
- [`ARCHITECTURE.md`](ARCHITECTURE.md)
|
|
@@ -23,6 +24,7 @@ Official Pi 1.0's initial SDK resume supplies an initial tool selection that byp
|
|
|
23
24
|
## Why this tool shape
|
|
24
25
|
|
|
25
26
|
This keeps the integration:
|
|
27
|
+
|
|
26
28
|
- thin
|
|
27
29
|
- powerful
|
|
28
30
|
- low-drift
|
|
@@ -43,9 +45,9 @@ An SDK host can import the package's compiled `dist/extensions/agent-browser/ind
|
|
|
43
45
|
|
|
44
46
|
```ts
|
|
45
47
|
agentBrowserExtension(pi, {
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
48
|
+
async beforeExecute(toolCallId, ctx) {
|
|
49
|
+
await saveHostState(toolCallId, ctx.signal);
|
|
50
|
+
},
|
|
49
51
|
});
|
|
50
52
|
```
|
|
51
53
|
|
|
@@ -65,13 +67,13 @@ Config shape:
|
|
|
65
67
|
|
|
66
68
|
```json
|
|
67
69
|
{
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
70
|
+
"webSearch": {
|
|
71
|
+
"enabled": true,
|
|
72
|
+
"preferredProvider": "exa",
|
|
73
|
+
"defaultSearchType": "deep-lite",
|
|
74
|
+
"exaApiKey": "$EXA_API_KEY",
|
|
75
|
+
"braveApiKey": "$BRAVE_API_KEY"
|
|
76
|
+
}
|
|
75
77
|
}
|
|
76
78
|
```
|
|
77
79
|
|
|
@@ -79,24 +81,25 @@ Schema:
|
|
|
79
81
|
|
|
80
82
|
```json
|
|
81
83
|
{
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
84
|
+
"query": "search text",
|
|
85
|
+
"provider": "auto",
|
|
86
|
+
"searchType": "deep-lite",
|
|
87
|
+
"includeDomains": ["docs.exa.ai"],
|
|
88
|
+
"excludeDomains": ["archive.example"],
|
|
89
|
+
"category": "publication",
|
|
90
|
+
"additionalQueries": ["Exa search API defaults"],
|
|
91
|
+
"highlightsDynamic": false,
|
|
92
|
+
"count": 5,
|
|
93
|
+
"offset": 0,
|
|
94
|
+
"country": "US",
|
|
95
|
+
"searchLang": "en-US",
|
|
96
|
+
"safesearch": "moderate",
|
|
97
|
+
"freshness": "pw"
|
|
96
98
|
}
|
|
97
99
|
```
|
|
98
100
|
|
|
99
101
|
Provider notes:
|
|
102
|
+
|
|
100
103
|
- `provider` is optional; `auto` uses available keys plus `webSearch.preferredProvider`.
|
|
101
104
|
- `offset` skips results, not pages, for both providers. Each fetches the first `count + offset` results and slices locally; the maximum request is 19 results (`count: 10`, `offset: 9`), within Brave's 20-result page limit.
|
|
102
105
|
- `searchType` applies to Exa only and supports `auto`, `fast`, `instant`, `deep-lite`, `deep`, and `deep-reasoning`. Effective precedence is the per-call field, then `webSearch.defaultSearchType`, then `auto`. Use `deep-lite` for implementation research, `deep` for hard multi-source work, and `deep-reasoning` only for the hardest or exhaustive work.
|
|
@@ -111,44 +114,44 @@ Result details:
|
|
|
111
114
|
|
|
112
115
|
```json
|
|
113
116
|
{
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
117
|
+
"provider": "exa",
|
|
118
|
+
"query": "search text",
|
|
119
|
+
"returnedQuery": "search text",
|
|
120
|
+
"count": 5,
|
|
121
|
+
"offset": 0,
|
|
122
|
+
"searchType": "deep-lite",
|
|
123
|
+
"requestId": "request-id-when-provider-returns-one",
|
|
124
|
+
"duplicatesRemoved": 1,
|
|
125
|
+
"fetchedAt": "2026-06-02T00:00:00.000Z",
|
|
126
|
+
"results": [
|
|
127
|
+
{
|
|
128
|
+
"title": "Result title",
|
|
129
|
+
"url": "https://example.com/",
|
|
130
|
+
"description": "Compact summary or first highlight",
|
|
131
|
+
"highlights": ["Relevant excerpt"],
|
|
132
|
+
"source": "Example",
|
|
133
|
+
"pageDate": "2026-06-02",
|
|
134
|
+
"language": "en"
|
|
135
|
+
}
|
|
136
|
+
]
|
|
134
137
|
}
|
|
135
138
|
```
|
|
136
139
|
|
|
137
|
-
For Exa, `details.searchType` is the effective requested type even when the provider response does not echo it. `requestId` is included when Exa returns one. Brave omits both fields. `pageDate` comes from Exa `publishedDate` (an estimated page creation date) or Brave `page_age`; Brave may separately return `age`. These provider fields are not crawl/retrieval age or proof that a result matches a requested version. For version-sensitive work, inspect the result, constrain one follow-up to the primary domain (`includeDomains` for Exa or `site:` in a Brave query), and read the primary page. Failures return structured results: local config/validation problems (invalid config, disabled tool, missing credential, blank query, Exa-only filters resolved to Brave) are `validation-error`; provider/network failures are `upstream-error`; both keep `isError` and redacted messages.
|
|
140
|
+
For Exa, `details.searchType` is the effective requested type even when the provider response does not echo it. `requestId` is included when Exa returns one. Brave omits both fields. `pageDate` comes from Exa `publishedDate` (an estimated page creation date) or Brave `page_age`; Brave may separately return `age`. These provider fields are not crawl/retrieval age or proof that a result matches a requested version. For version-sensitive work, inspect the result, constrain one follow-up to the primary domain (`includeDomains` for Exa or `site:` in a Brave query), and read the primary page. Failures return structured results: local config/validation problems (invalid config, disabled tool, missing credential, blank query, Exa-only filters resolved to Brave) are `validation-error`; provider/network failures are `upstream-error`; both keep `isError` and redacted messages. Malformed provider JSON reports the provider-specific invalid-JSON error without parser previews or parser causes. Malformed config files, in extension loading and the config CLI alike, report their scope/path and `Invalid JSON` plus the parser's numeric position/line/column when available, never source fragments; either payload may contain credentials. Provider HTTP failures never echo the response body or status text: they report the provider, `HTTP <status>`, and a fixed hint (credential/permission check for 401/403, retry later for 5xx, check the search parameters otherwise); HTTP 429 keeps the rate-limit guidance. Text content, `details`, and structured output are rendered from one redacted result object.
|
|
138
141
|
|
|
139
142
|
## Input mode chooser
|
|
140
143
|
|
|
141
|
-
| Need
|
|
142
|
-
|
|
|
143
|
-
| One native command
|
|
144
|
-
| Fixed sequence
|
|
145
|
-
| Loops, branches, aggregation
|
|
146
|
-
| Discover/enable advanced tools
|
|
147
|
-
| Stable locator or native select
|
|
148
|
-
| Diagnostic verdict
|
|
149
|
-
| Desktop app lifecycle
|
|
150
|
-
| UI/source candidates
|
|
151
|
-
| Failed-request/source candidates | `agent_browser_network_source` | Flat request/filter fields
|
|
144
|
+
| Need | Tool | Input |
|
|
145
|
+
| -------------------------------- | ------------------------------ | ---------------------------------------------------------------- |
|
|
146
|
+
| One native command | `agent_browser` | `{ args, stdin?, outputPath?, timeoutMs?, sessionMode? }` |
|
|
147
|
+
| Fixed sequence | `agent_browser` | `args: ["batch", "--bail"]` and native JSON-array `stdin` |
|
|
148
|
+
| Loops, branches, aggregation | `agent_browser_code` | `{ code, session?, namespace?, timeoutMs?, outputPath? }` |
|
|
149
|
+
| Discover/enable advanced tools | `agent_browser_tools` | `{ enable?: ["action", "qa", "electron", "source", "network"] }` |
|
|
150
|
+
| Stable locator or native select | `agent_browser_action` | Flat action/locator fields |
|
|
151
|
+
| Diagnostic verdict | `agent_browser_qa` | Flat URL or attached-check fields |
|
|
152
|
+
| Desktop app lifecycle | `agent_browser_electron` | Flat action/target fields |
|
|
153
|
+
| UI/source candidates | `agent_browser_source` | Flat selector/fiber/component fields |
|
|
154
|
+
| Failed-request/source candidates | `agent_browser_network_source` | Flat request/filter fields |
|
|
152
155
|
|
|
153
156
|
Run `agent_browser_tools {}` for inventory; `enable` adds selected capabilities without removing unrelated active tools. Pi's native selected-tool history handles reload/resume rather than a package-owned activation registry. If an exact recovery action names an inactive advanced tool, enable its capability before using that payload.
|
|
154
157
|
|
|
@@ -200,7 +203,7 @@ Upstream 0.35.2 adds `dashboard start --allowed-origins <origins>` and `AGENT_BR
|
|
|
200
203
|
|
|
201
204
|
## Headed and local fixture limits
|
|
202
205
|
|
|
203
|
-
Local Chrome uses stock `--args --no-startup-window` at bootstrap
|
|
206
|
+
Local Chrome uses stock `--args --no-startup-window` at bootstrap only when native environment/config does not own local launch defaults; caller CLI argument values then follow that default. With native environment/config launch defaults, no wrapper Chrome startup arguments are injected, keeping browser, local auth, and helper invocations consistent. Include `--no-startup-window` in native `args` when a single headed window is required. CLI still overrides environment, which overrides native config. Plain active-session follow-ups do not resend wrapper bootstrap settings. Changed launch settings can trigger native's normal session-identity restart; there is no migration or launch-settings journal. Headless remains the default. CDP/auto-connect, providers, Electron, and Lightpanda do not receive this Chrome default. The wrapper does not delete tabs or edit profiles.
|
|
204
207
|
|
|
205
208
|
URL-less `open` is executed as native `get url`, including effective raw/stdin batch rows and code calls. This lazily opens a missing browser without navigating an existing page or issuing a second incomplete launch. `details.args` retains the request; `effectiveArgs`, `command`, native URL data and lifecycle show what ran. No launch receipt is synthesized. Explicit URLs, required-URL errors for `goto`/`navigate`, and top-level help/version argv keep their native meanings. Ignored batch stdin remains unchanged.
|
|
206
209
|
|
|
@@ -218,6 +221,7 @@ URL-less `open` is executed as native `get url`, including effective raw/stdin b
|
|
|
218
221
|
|
|
219
222
|
<!-- agent-browser-playbook:start shared-guidelines -->
|
|
220
223
|
<!-- Generated from extensions/agent-browser/lib/playbook.ts. Run `npm run docs -- playbook write` to update. -->
|
|
224
|
+
|
|
221
225
|
- Choose agent_browser for one native command, native batch --bail with JSON-array stdin for a known fixed sequence, or agent_browser_code for loops, branches, and aggregation. Return to the model when a fresh observation needs judgment. Neither tool is a named recipe registry.
|
|
222
226
|
- agent_browser_code takes { code, session?, namespace?, timeoutMs?, outputPath? }. JavaScript globals are fresh for every call; the selected browser persists and is shared with agent_browser. Use await browser({ args, stdin?, timeoutMs? }), inspect result.success, and emit selected JSON instead of whole envelopes. Use emitImage(result.imageObservations[0]) only when image inspection is needed; image handles belong to the current code call. No imports or host filesystem/network/process APIs are exposed.
|
|
223
227
|
- Standard workflow: open the page, snapshot -i, interact using current @refs from that snapshot, and re-snapshot after navigation, scrolling, rerendering, or other major DOM changes because refs are page-scoped; the wrapper fails mutation-prone stale/recycled refs before upstream can silently target a different current-page element. On dense pages, use wrapper-side snapshot -i --search <text> or snapshot -i --filter role=<role> to render matching refs while preserving the full ref map in details.refSnapshot, add snapshot --viewport when scroll position or above/below-fold context matters, and add snapshot --diff when a quick before/after ref-map delta would prevent reading a full spill file.
|
|
@@ -260,6 +264,7 @@ URL-less `open` is executed as native `get url`, including effective raw/stdin b
|
|
|
260
264
|
- Respect explicit user stop boundaries yourself. When the surrounding authenticated employee or automation context is explicitly unattended/auto-approved, ordinary non-destructive form submissions within the requested flow may proceed without separate confirmation. Still require explicit authorization for purchases, production-control actions, destructive or irreversible actions, and account, security, or privacy changes. The wrapper does not infer broad business intent from prompt text; details.promptGuard is reserved for concrete artifact-before-close checks.
|
|
261
265
|
- Recording needs ffmpeg on PATH before start. Current upstream checks it at startup; older natives may defer failure. A pending recording is not verified output.
|
|
262
266
|
- Do not call --help or other exploratory inspection commands unless the user explicitly asks for them or debugging the browser integration is necessary.
|
|
267
|
+
|
|
263
268
|
<!-- agent-browser-playbook:end shared-guidelines -->
|
|
264
269
|
|
|
265
270
|
## Parameters
|
|
@@ -295,7 +300,7 @@ Examples:
|
|
|
295
300
|
|
|
296
301
|
Relative file operands, `outputPath`, and source/network-source scans use one `pi-change-working-dir` owner reply captured before browser policy and queues; code calls inherit it. The integration uses the synchronous `pi-change-working-dir:resolve-execution-cwd` event with `{ sessionManager, result?: { cwd, error? } }`. Owner errors or invalid replies fail validation. No owner means native `ctx.cwd`; a known older owner identified through public tool/command package provenance requires an update and Pi restart. No private directory journal is read.
|
|
297
302
|
|
|
298
|
-
Browser/session/config ownership stays separate. Ordinary calls retain the native project or managed launch root, while an unnamed fresh launch and explicit native config resolve against the execution directory. `details.managedSessionCwd`, when present, records the managed launch root for transcript replay, including ordered code transitions. Pi package config/trust and artifact-store identity remain unchanged. A deleted launch directory returns `validation-error` before dispatch; restore it or explicitly choose fresh/config from the new directory. See [working-directory changes](
|
|
303
|
+
Browser/session/config ownership stays separate. Ordinary calls retain the native project or managed launch root, while an unnamed fresh launch and explicit native config resolve against the execution directory. `details.managedSessionCwd`, when present, records the managed launch root for transcript replay, including ordered code transitions. Pi package config/trust and artifact-store identity remain unchanged. A deleted launch directory returns `validation-error` before dispatch; restore it or explicitly choose fresh/config from the new directory. See [working-directory changes](REFERENCE.md#working-directory-changes).
|
|
299
304
|
|
|
300
305
|
### `agent_browser_code`
|
|
301
306
|
|
|
@@ -316,13 +321,13 @@ The browser persists between code and direct calls. JavaScript variables do not.
|
|
|
316
321
|
|
|
317
322
|
```json
|
|
318
323
|
{
|
|
319
|
-
|
|
324
|
+
"code": "const page = await browser({ args: ['get', 'title'] }); if (!page.success) throw new Error(page.error); emit({ title: page.data.title ?? page.data.result });"
|
|
320
325
|
}
|
|
321
326
|
```
|
|
322
327
|
|
|
323
328
|
```json
|
|
324
329
|
{
|
|
325
|
-
|
|
330
|
+
"code": "const shot = await browser({ args: ['screenshot', '/tmp/page.png'] }); if (!shot.success) throw new Error(shot.error); emit(shot.artifactVerification); if (shot.imageObservations?.length) emitImage(shot.imageObservations[0]);"
|
|
326
331
|
}
|
|
327
332
|
```
|
|
328
333
|
|
|
@@ -342,16 +347,16 @@ Abort, timeout, branch change, reload, and shutdown stop/reap the child, settle
|
|
|
342
347
|
|
|
343
348
|
Enable `action` through `agent_browser_tools`, then call the tool with flat fields. Optional `outputPath` and `timeoutMs` apply to the ordinary executor; this tool has no `sessionMode` field.
|
|
344
349
|
|
|
345
|
-
| Field
|
|
346
|
-
|
|
|
347
|
-
| `action`
|
|
348
|
-
| `selector`
|
|
349
|
-
| `locator`
|
|
350
|
-
| `value`
|
|
351
|
-
| `values`
|
|
352
|
-
| `text`
|
|
353
|
-
| `role`, `name` | Accessible role and optional name; role may replace locator value and must agree if both are given
|
|
354
|
-
| `session`
|
|
350
|
+
| Field | Contract |
|
|
351
|
+
| -------------- | ---------------------------------------------------------------------------------------------------- |
|
|
352
|
+
| `action` | Required: `click`, `check`, `fill`, or `select` |
|
|
353
|
+
| `selector` | Direct CSS/selector/current-ref target; cannot combine with locator fields for click/check/fill |
|
|
354
|
+
| `locator` | `role`, `text`, `label`, `placeholder`, `alt`, `title`, or `testid`; select supports only role/label |
|
|
355
|
+
| `value` | Locator value or one select option; for label select, this is the label |
|
|
356
|
+
| `values` | One or more select options; required for label select; rejected on other actions |
|
|
357
|
+
| `text` | Fill text; required for fill, rejected on select |
|
|
358
|
+
| `role`, `name` | Accessible role and optional name; role may replace locator value and must agree if both are given |
|
|
359
|
+
| `session` | Explicit native session; preserved in compiled commands and recovery |
|
|
355
360
|
|
|
356
361
|
Locator click/check/fill compiles to native `find`; direct targets compile to their ordinary command. Select uses native `select <selector> <value...>`. Accessible select resolves exactly one current visible combobox/listbox through a fresh snapshot or the eligible one-use confirmed capture; missing or ambiguous targets fail before action. Use `role: "combobox"` or `"listbox"` plus `name`, or label plus option `values`. No native `find ... select` capability is assumed.
|
|
357
362
|
|
|
@@ -372,7 +377,7 @@ Enable `qa`. Flat input is either `{ url, ...checks }` or `{ attached: true, ...
|
|
|
372
377
|
{ "attached": true, "expectedText": "Explorer", "checkErrors": true }
|
|
373
378
|
```
|
|
374
379
|
|
|
375
|
-
QA compiles a fixed `batch --bail` diagnostic plan. URL QA clears enabled network/console buffers, captures page-error residue after native clear, opens the requested URL, waits for readiness, allows a bounded 150 ms diagnostic settle when needed, then performs visible-text/selector assertions, enabled diagnostic reads, and optional screenshot. Expected text uses bounded visible-element predicates with a 5-second native timeout. Missing assertions stop before later diagnostics. URL QA does not require the previous page target.
|
|
380
|
+
QA compiles a fixed `batch --bail` diagnostic plan. URL QA clears enabled network/console buffers, captures page-error residue after native clear, opens the requested URL, waits for readiness, allows a bounded 150 ms diagnostic settle when needed, then performs visible-text/selector assertions, enabled diagnostic reads, and optional screenshot. Expected text uses bounded visible-element predicates with a 5-second native timeout. Text nodes and control values under `display: none` or `opacity: 0` ancestors, or whose computed visibility is hidden, do not satisfy the assertion; `display: contents` wrappers and `visibility: visible` overrides still count. Missing assertions stop before later diagnostics. URL QA does not require the previous page target.
|
|
376
381
|
|
|
377
382
|
Attached QA checks the current browser without navigation or buffer clears. It first requires a live nonempty URL, including file/custom app URLs. Failed target verification returns exact tab/URL recovery instead of running the batch. Diagnostic reads default off because preserved buffers may predate this check; URL diagnostics default on. Explicit `checkErrors: false` skips the error-baseline check.
|
|
378
383
|
|
|
@@ -384,29 +389,33 @@ The verdict is `qaPreset: { passed, failedChecks, notRunChecks, warnings, summar
|
|
|
384
389
|
|
|
385
390
|
Enable `electron`. Inputs are flat `{ action, ...fields, outputPath? }`; caller `stdin` and `sessionMode` are not accepted. The [Electron guide](ELECTRON.md) covers workflows and ownership.
|
|
386
391
|
|
|
387
|
-
| Action
|
|
388
|
-
|
|
|
389
|
-
| `list`
|
|
390
|
-
| `launch`
|
|
391
|
-
| `status`
|
|
392
|
-
| `probe`
|
|
393
|
-
| `cleanup` | `launchId?` or `all: true`, `timeoutMs?`
|
|
392
|
+
| Action | Fields | Behavior |
|
|
393
|
+
| --------- | ----------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
394
|
+
| `list` | `query?`, `maxResults?` | Host-only bounded discovery; default 50, cap 200. No timeout field. macOS/Linux scanning only; Windows returns unsupported discovery. |
|
|
395
|
+
| `launch` | Exactly one of `appPath`, `appName`, `bundleId`, `executablePath`; `appArgs?`, `handoff?`, `targetType?`, `timeoutMs?`, `allow?`, `deny?` | Verify Electron evidence, create a temporary isolated profile/OS-chosen debug port, attach through native connect, hand off, and record ownership. |
|
|
396
|
+
| `status` | `launchId?` or `all: true`, `timeoutMs?` | Inspect tracked PID/port/profile and targets; explicit cleaned IDs remain inspectable as historical records. |
|
|
397
|
+
| `probe` | `launchId?`, `timeoutMs?` | Verify URL then read title/focus/tabs/snapshot with owned session settings. Without an ID, inspect the current attached managed session. |
|
|
398
|
+
| `cleanup` | `launchId?` or `all: true`, `timeoutMs?` | Close the tracked native session, stop only the owned app, verify port shutdown, remove its temporary profile. Partial cleanup fails with exact retry guidance. |
|
|
394
399
|
|
|
395
400
|
```json
|
|
396
401
|
{ "action": "launch", "appName": "Visual Studio Code", "handoff": "snapshot" }
|
|
397
402
|
{ "action": "cleanup", "launchId": "electron-…" }
|
|
398
403
|
```
|
|
399
404
|
|
|
400
|
-
`handoff` defaults to `snapshot`; `tabs` verifies/discovers targets without an interactive snapshot, and `connect` stops after attach. `targetType` is `page` by default, or `webview`/`any`. Launch cannot reuse the normal signed-in app profile or an existing app; use host launch plus direct native `connect` for that. Windows launch requires a verifiable path rather than scan-based name resolution; this is not Windows qualification evidence.
|
|
405
|
+
`handoff` defaults to `snapshot`; `tabs` verifies/discovers targets without an interactive snapshot, and `connect` stops after attach. `targetType` is `page` by default, or `webview`/`any`. CDP readiness and debug-port liveness require `/json/version` to contain a nonempty, non-whitespace `Browser` or `browser` string; other version metadata is optional. Launch cannot reuse the normal signed-in app profile or an existing app; use host launch plus direct native `connect` for that. Windows launch requires a verifiable path rather than scan-based name resolution; this is not Windows qualification evidence.
|
|
401
406
|
|
|
402
407
|
`appArgs` cannot override user-data-dir/debug lifecycle flags or include bare `--`. Caller `allow`/`deny` substring policies match app names/IDs/paths, with deny winning. Likely-sensitive discovery annotations are advisory. Non-Electron executables fail validation.
|
|
403
408
|
|
|
409
|
+
Electron payload evidence accepts `app.asar`, `app/`, or the stock runtime's `default_app.asar`, alongside platform runtime resources (including Electron Framework on macOS). Stock npm Electron can launch an unpackaged app by passing its binary as `executablePath` and the app directory in `appArgs`; discovery and explicit paths share the same evidence check.
|
|
410
|
+
|
|
404
411
|
Timeouts are per action: launch readiness defaults to 15 seconds, capped at 120 seconds after discovery; status/probe use the normal 35-second subprocess budget per read (or configured override); cleanup applies its default 35-second managed-close budget separately to close and initial process-exit wait. Local CDP HTTP probes use one second. There is no combined end-to-end teardown deadline. Cancellation before spawn opens nothing; cancellation during launch cleans the new process/profile.
|
|
405
412
|
|
|
406
|
-
`details.electron` retains action-specific `apps`, `launch`, `statuses`, `targets`, `probe`, `cleanup`, and failure diagnostics; `compiledElectron` retains the action plan. `launchId` identifies host lifecycle; `sessionName` identifies browser commands. `userDataDirState` measures the tracked path as present, absent (ENOENT), or unknown; it does not audit all app residue. Failed startup output includes redacted last-4096-byte stdout/stderr tails; private profile-local files follow profile ownership, not a lifetime byte cap. Failed cleanup preserves live profiles. Reload preserves current branch-visible launches and cleans off-branch owned launches; quit cleans owned launches. Restored attachments must match live endpoint/PID/profile evidence before reuse.
|
|
413
|
+
`details.electron` retains action-specific `apps`, `launch`, `statuses`, `targets`, `probe`, `cleanup`, and failure diagnostics; `compiledElectron` retains the action plan. `launchId` identifies host lifecycle; `sessionName` identifies browser commands. `userDataDirState` measures the tracked path as present, absent (ENOENT), or unknown; it does not audit all app residue. Failed startup output includes redacted last-4096-byte stdout/stderr tails; private profile-local files follow profile ownership, not a lifetime byte cap. Failed cleanup preserves live profiles. Launch records the app process start identity. Before signaling a live restored PID without a tracked child handle, cleanup requires that recorded identity to match the live process; a missing (including records from older versions), unreadable, or mismatched identity leaves the process/profile intact. Reload preserves current branch-visible launches and cleans off-branch owned launches; quit cleans owned launches. Restored attachments must match live endpoint/PID/profile evidence before reuse.
|
|
407
414
|
|
|
408
415
|
Post-command process/target loss fails as `tab-drift`; fill verification and broad-shell-text diagnostics remain available. Recovery actions use `agent_browser_electron` with flat params for status/probe/cleanup and `agent_browser` for native tab/URL/ref operations. If inactive, enable `electron` first. Manually launched apps and explicit saved artifacts remain host-owned.
|
|
409
416
|
|
|
417
|
+
Status/probe mismatch checks and probe page-target state use complete URLs, not display previews. Managed-session/probe URL summaries remain capped at 300 characters and probe tab URLs at 240; shortened output does not itself imply drift. Probe `sessionTabTarget` and `refSnapshot` retain full target identity for follow-up commands and replay.
|
|
418
|
+
|
|
410
419
|
### `agent_browser_source`
|
|
411
420
|
|
|
412
421
|
Enable `source`. Flat fields: at least one of `selector`, `reactFiberId`, or `componentName`; optional `includeDomHints` (default true), `maxWorkspaceFiles` (default 2,000, maximum 5,000), `outputPath`, `timeoutMs`, and `sessionMode`.
|
|
@@ -427,7 +436,7 @@ Enable `network`. Flat fields: at least one of `requestId`, `filter`, or `url`;
|
|
|
427
436
|
{ "requestId": "req-1", "url": "/api/fail" }
|
|
428
437
|
```
|
|
429
438
|
|
|
430
|
-
A request ID runs native `network request`; filter/URL adds `network requests --filter`, with explicit filter winning. Failed HTTP/failed/error rows are correlated with initiator metadata and a bounded URL-literal search under the captured execution directory (up to eight needles and ten workspace matches). `networkSourceLookup` retains `{ status, failedRequests, candidates, limitations, summary }`; statuses are `failed-requests-found`, `no-failed-requests`, or `no-candidates`. URLs and credentials are redacted. Request URLs are diagnostic evidence and never replace the active page target. No candidates alone does not fail a successful batch or assign blame. HAR and full native diagnostics remain available through `args`.
|
|
439
|
+
A request ID runs native `network request`; filter/URL adds `network requests --filter`, with explicit filter winning for both the native query and failed-request/initiator analysis. Initiators of failed requests outside that filter/URL scope are ignored. Repeated request IDs count once (first matching failed row wins); anonymous rows and different IDs at the same URL remain distinct. Failed HTTP/failed/error rows are correlated with initiator metadata and a bounded URL-literal search under the captured execution directory (up to eight needles and ten workspace matches). `networkSourceLookup` retains `{ status, failedRequests, candidates, limitations, summary }`; statuses are `failed-requests-found`, `no-failed-requests`, or `no-candidates`. URLs and credentials are redacted. Request URLs are diagnostic evidence and never replace the active page target. No candidates alone does not fail a successful batch or assign blame. HAR and full native diagnostics remain available through `args`.
|
|
431
440
|
|
|
432
441
|
### `stdin`
|
|
433
442
|
|
|
@@ -450,7 +459,10 @@ For `eval --stdin`, put the script in the top-level `stdin` field. The wrapper n
|
|
|
450
459
|
```
|
|
451
460
|
|
|
452
461
|
```json
|
|
453
|
-
{
|
|
462
|
+
{
|
|
463
|
+
"args": ["auth", "save", "my-login", "--password-stdin"],
|
|
464
|
+
"stdin": "password from the user-approved secret source"
|
|
465
|
+
}
|
|
454
466
|
```
|
|
455
467
|
|
|
456
468
|
### `outputPath`
|
|
@@ -466,7 +478,11 @@ For `eval --stdin`, put the script in the top-level `stdin` field. The wrapper n
|
|
|
466
478
|
Example:
|
|
467
479
|
|
|
468
480
|
```json
|
|
469
|
-
{
|
|
481
|
+
{
|
|
482
|
+
"args": ["eval", "--stdin"],
|
|
483
|
+
"stdin": "({ title: document.title, url: location.href })",
|
|
484
|
+
"outputPath": "logs/page-state.json"
|
|
485
|
+
}
|
|
470
486
|
```
|
|
471
487
|
|
|
472
488
|
### `timeoutMs`
|
|
@@ -487,13 +503,14 @@ Example:
|
|
|
487
503
|
|
|
488
504
|
### `sessionMode`
|
|
489
505
|
|
|
490
|
-
A native `session` default from user/project JSON or `AGENT_BROWSER_SESSION` selects a caller-owned browser for direct, code, action, QA, and source tools. Per-call flags still win. Like literal `--session`, this selection takes precedence over `fresh`; it reports `usedImplicitSession: false` and is not closed when Pi quits. Without a selected native session, attachment, or explicit fresh launch, auto calls select a stable root-Pi named session and root-specific native restore key. `PI_SUBAGENT_ROOT_SESSION_ID` carries the parent identity to descendants; ordinary new/fork/clone roots differ, while resume/cwd changes keep identity. Root sessions report `usedImplicitSession: false` and survive parent/child quit. Global/override package profile names with `policy: "always"` and executable defaults bootstrap only inactive automatic roots whose effective engine is Chrome. Lightpanda receives no wrapper-supplied Chrome profile/executable defaults; caller native args, environment and config remain authoritative. Active roots preserve native launch settings/current restore key, including matching root-name follow-ups; unrelated explicit browsers are unchanged. Profile paths and project-only profile config remain advisory. Native restore covers cookies and web storage, not newly created IndexedDB or memory-only authentication. See the
|
|
506
|
+
A native `session` default from user/project JSON or `AGENT_BROWSER_SESSION` selects a caller-owned browser for direct, code, action, QA, and source tools. Per-call flags still win. Like literal `--session`, this selection takes precedence over `fresh`; it reports `usedImplicitSession: false` and is not closed when Pi quits. Without a selected native session, attachment, or explicit fresh launch, auto calls select a stable root-Pi named session and root-specific native restore key. `PI_SUBAGENT_ROOT_SESSION_ID` carries the parent identity to descendants; ordinary new/fork/clone roots differ, while resume/cwd changes keep identity. Root sessions report `usedImplicitSession: false` and survive parent/child quit. Global/override package profile names with `policy: "always"` and executable defaults bootstrap only inactive automatic roots whose effective engine is Chrome. Lightpanda receives no wrapper-supplied Chrome profile/executable defaults; caller native args, environment and config remain authoritative. Active roots preserve native launch settings/current restore key, including matching root-name follow-ups; unrelated explicit browsers are unchanged. Profile paths and project-only profile config remain advisory. Native restore covers cookies and web storage, not newly created IndexedDB or memory-only authentication. See the detailed reference for communal bootstrap and persistent-profile fallback. Active older/fresh managed sessions retain the lifecycle below. `agent_browser_qa` with `attached: true` can inspect this shared current session using the same live target checks. See [shared browser defaults](REFERENCE.md#shared-browser-defaults).
|
|
491
507
|
|
|
492
508
|
- type: `"auto" | "fresh"`
|
|
493
509
|
- optional
|
|
494
510
|
- default: `"auto"`
|
|
495
511
|
|
|
496
512
|
Behavior:
|
|
513
|
+
|
|
497
514
|
- if `args` already include `--session` (including argv compiled from `agent_browser_action.session`), upstream session choice wins
|
|
498
515
|
- `"auto"` prepends the current extension-managed active session when appropriate
|
|
499
516
|
- after resume, a confirmed inactive wrapper-owned daemon with automatic managed restore enabled reopens its complete recorded URL, including its fragment, before the first current-page operation, including `get url` and `reload`. Non-page calls such as `tab list` or explicit HTTP `read <url>` can start a daemon without consuming that pending reopen, even across branch/reload replay. The wrapper verifies the observed tab and invalidates old refs; native `open` resets frame scope. This reloads the page with restored cookies/storage, not unsaved forms, JavaScript memory, or history. Live wrong-tab recovery does not navigate. Explicit URL reads, URL `a11y`/`vitals`, `diff url`, `window new`, URL-bearing recording commands, and explicit navigation/context changes do not require the old tab. Caller-owned/attached and restore-disabled sessions are not auto-reopened
|
|
@@ -504,6 +521,7 @@ Behavior:
|
|
|
504
521
|
- sessionless paths skip that injection even under `"auto"`: plain-text help/version, read-only skills, local auth/profile/setup commands, `session list`, and syntactically local state lifecycle operations keep `effectiveArgs` free of the implicit managed `--session` unless the caller supplied one. All upstream rows and supported targets remain available. Browser-backed or context-dependent commands such as `auth login` and `state save/load` keep normal managed-session injection when the caller did not choose an explicit session (`extensions/agent-browser/lib/command-policy.ts`, `needsManagedSession`; `extensions/agent-browser/lib/runtime.ts`, `buildExecutionPlan`)
|
|
505
522
|
|
|
506
523
|
Recommended use:
|
|
524
|
+
|
|
507
525
|
- use `"auto"` for the common browse/snapshot/click flow inside one `pi` session
|
|
508
526
|
- use `"fresh"` when switching from an already-active implicit session to domain containment or a new profile/browser executable/debug/auth/provider launch without inventing a fixed explicit session name
|
|
509
527
|
- when a fresh launch fails or times out before becoming current, check `details.managedSessionOutcome`: it states whether the prior managed session was preserved or whether the attempted fresh session was abandoned because no prior managed session existed; when `sessionMode` is `"fresh"` and the tool ultimately fails, the model-visible result also appends `Managed session outcome: …` (see `#details` below). Failures under `sessionMode: "auto"` still expose the struct on `details` when the extension injects a managed `--session`, but they do not add that extra prose line.
|
|
@@ -513,6 +531,7 @@ Recommended use:
|
|
|
513
531
|
Caller `args` may omit `--json` for structured details and prose; explicit `--json false` selects opaque native text. See [Wrapper `--json`](#wrapper-json).
|
|
514
532
|
|
|
515
533
|
The extension should:
|
|
534
|
+
|
|
516
535
|
- inject `--json` when no caller output mode is selected
|
|
517
536
|
- invoke `agent-browser` directly, not through a shell
|
|
518
537
|
- parse JSON-mode output into tool details
|
|
@@ -522,6 +541,7 @@ The extension should:
|
|
|
522
541
|
|
|
523
542
|
<!-- agent-browser-playbook:start inspection -->
|
|
524
543
|
<!-- Generated from extensions/agent-browser/lib/playbook.ts. Run `npm run docs -- playbook write` to update. -->
|
|
544
|
+
|
|
525
545
|
Native inspection calls use the `agent_browser` tool shape, not shell-like direct-binary commands:
|
|
526
546
|
|
|
527
547
|
- { "args": ["--help"] }
|
|
@@ -529,6 +549,7 @@ Native inspection calls use the `agent_browser` tool shape, not shell-like direc
|
|
|
529
549
|
|
|
530
550
|
These calls return plain text and stay stateless: the extension does not inject its implicit session and does not let inspection consume the managed-session slot needed for later profile, session, CDP, state, auto-connect, or provider-backed launches.
|
|
531
551
|
<!-- agent-browser-playbook:end inspection -->
|
|
552
|
+
|
|
532
553
|
- still describe normal browser workflows in guidance so models do not overuse inspection for routine tasks
|
|
533
554
|
- surface stderr and non-zero exits clearly
|
|
534
555
|
- attach images when the result points to a screenshot-like artifact
|
|
@@ -560,6 +581,7 @@ Conditional `changed: false` captures have no new image/artifact attachment. A p
|
|
|
560
581
|
### Content
|
|
561
582
|
|
|
562
583
|
Primary content should be:
|
|
584
|
+
|
|
563
585
|
- useful result text for the model, not just a status line
|
|
564
586
|
- an image attachment when relevant
|
|
565
587
|
- browser-aware compacting for oversized snapshots so the model gets a concise actionable view before raw page noise
|
|
@@ -568,6 +590,7 @@ Primary content should be:
|
|
|
568
590
|
- wrapper-side `snapshot -i --search <text>` and `snapshot -i --filter role=<role>` filters should strip those wrapper-only flags before upstream spawn, preserve the full latest ref map in `details.refSnapshot`, and render matching direct refs plus surrounding snapshot context with `details.snapshotFilter` counts so dense-page agents can find controls without opening raw spill files; search should also run one bounded read-only rendered-DOM probe across the full document so visible below-fold warnings and accessible labels omitted from the accessibility snapshot remain discoverable while hidden nodes stay excluded; the visible summary should distinguish direct ref matches from rendered/contextual matches to avoid apparent count mismatches; wrapper-side `--viewport` should also strip before upstream spawn, run one read-only viewport/scroll probe, and report `details.snapshotViewport`; wrapper-side `--diff` should strip before upstream spawn and report `details.snapshotDiff` against the previous wrapper-tracked ref map for that session
|
|
569
591
|
|
|
570
592
|
Examples:
|
|
593
|
+
|
|
571
594
|
- small `snapshot` results should include the actual snapshot text
|
|
572
595
|
- oversized `snapshot` results should switch to a compact view that preserves the primary content, nearby sections, a trimmed set of high-value refs, and a separate bounded list of omitted high-value controls when dense pages or desktop host screens would otherwise hide editable inputs, named surfaces/tabs, or primary action buttons, while exposing the full redacted snapshot path directly in the rendered tool text and via `details.fullOutputPath`
|
|
573
596
|
- successful navigation actions like `click`, `back`, `forward`, and `reload` should include a lightweight post-action title/url summary when the wrapper can address the active session
|
|
@@ -580,22 +603,22 @@ Recommended details:
|
|
|
580
603
|
|
|
581
604
|
```json
|
|
582
605
|
{
|
|
583
|
-
|
|
584
|
-
|
|
585
|
-
|
|
586
|
-
|
|
587
|
-
|
|
588
|
-
|
|
589
|
-
|
|
590
|
-
|
|
591
|
-
|
|
592
|
-
|
|
593
|
-
|
|
594
|
-
|
|
595
|
-
|
|
596
|
-
|
|
597
|
-
|
|
598
|
-
|
|
606
|
+
"args": ["snapshot", "-i"],
|
|
607
|
+
"effectiveArgs": ["--json", "--session", "pi-abc123", "snapshot", "-i"],
|
|
608
|
+
"command": "snapshot",
|
|
609
|
+
"sessionMode": "auto",
|
|
610
|
+
"sessionName": "pi-abc123",
|
|
611
|
+
"usedImplicitSession": true,
|
|
612
|
+
"resultCategory": "success",
|
|
613
|
+
"successCategory": "completed",
|
|
614
|
+
"data": {
|
|
615
|
+
"origin": "https://example.com/",
|
|
616
|
+
"refs": {
|
|
617
|
+
"e1": { "name": "Example Domain", "role": "heading" }
|
|
618
|
+
},
|
|
619
|
+
"snapshot": "- heading \"Example Domain\" [level=1, ref=e1]"
|
|
620
|
+
},
|
|
621
|
+
"summary": "Snapshot: 1 refs on https://example.com/"
|
|
599
622
|
}
|
|
600
623
|
```
|
|
601
624
|
|
|
@@ -625,7 +648,7 @@ Ref preflight details (command taxonomy in `extensions/agent-browser/lib/command
|
|
|
625
648
|
|
|
626
649
|
- **Spelling and operands:** `@eN`, `eN`, and `ref=eN` share the same stale-ref checks, including inside raw and stdin batches. Only upstream ref-resolving selector slots are considered; `get count` and `diff snapshot --selector` keep their CSS/XPath or CSS semantics, while literal text and key/mouse data are not refs.
|
|
627
650
|
- **URL alignment:** `refSnapshot.target.url` and the session’s current tab URL are compared via `targetsMatch` / `normalizeComparableUrl` in `extensions/agent-browser/index.ts`: values are trimmed, parsed as URLs when possible, compared **after dropping the `#fragment`**, and the query string remains significant. If either side lacks a `url`, `targetsMatch` treats the pair as matching so early-session calls are not blocked.
|
|
628
|
-
- **Batch
|
|
651
|
+
- **Batch ordering:** effective raw-argument or stdin `batch` steps are scanned in order. Any step whose first token satisfies `isRefInvalidatingBatchCommand` sets a latch that blocks later steps whose first token satisfies `isRefGuardedCommand` and that mention `@e…` refs, except for same-snapshot native form-control steps whose current snapshot role metadata identifies all refs as safe controls (`check`/`uncheck` or direct `click`/`tap` on checkbox or radio refs, and `select` on combobox refs). A step whose first token is `snapshot` clears that latch for subsequent steps (pre-spawn intent only; it does not wait for upstream success). The scan continues after every snapshot, even when no refs precede it: `snapshot -i`, a non-form `click @e1`, then `get value @e2` is rejected before spawn unless another snapshot separates the click from the getter. These predicates read explicit command capability flags from `command-taxonomy.ts`: navigation/mutation verbs such as `open` / `goto`, `reload`, non-form `click`, and related upstream commands have `invalidatesBatchRefs`, and `record start` steps (any outcome), `record restart` steps with a URL operand, plus WebMCP `invoke` / `result` / `cancel` steps also set the latch because upstream swaps or navigates the active page; same-snapshot `fill` rows and the role-checked native form-control rows stay guarded against missing/stale refs but do not set the latch, allowing ordinary form batches before a final click/submit step. Direct `click`/`tap @e…` is only treated as a safe form-control row when every ref in that step is a latest-snapshot checkbox or radio; other click/tap refs remain invalidating. Ref-guarded commands accept page-scoped refs for interaction (`click`, `fill`, `download`, `scrollintoview` / `scrollinto`, and others centralized in the command taxonomy). Changing either capability requires updating this contract, [`docs/SUPPORT_MATRIX.md`](SUPPORT_MATRIX.md) `RQ-0072`/`RQ-0087` notes, README and command-reference pitfalls, and `test/agent-browser.extension-ref-guards.test.ts`.
|
|
629
652
|
|
|
630
653
|
**Presentation redaction (implementation map):** Successful non-`batch` tool calls and each successful `batchSteps[]` row run upstream `data` through `redactPresentationData` in `extensions/agent-browser/lib/results/presentation/diagnostics.ts`: `cookies` still walk objects/arrays and replace case-insensitive `value` keys with `"[REDACTED]"`; `storage` redacts values when the key or value looks credential-like (token, cookie, auth, secret, JWT, bearer/basic credential, high-entropy token-like string, or nested sensitive JSON) but keeps low-risk primitive QA values such as booleans, numbers, and short strings visible. Redacted storage entries add `valueRedacted` plus `valueRedactionReason` in `details.data`; diagnostic formatters mirror the same decision. Every other command’s payload is recursively scrubbed with the shared `redactSensitiveValue`, which redacts known sensitive key names and applies string-level sensitivity heuristics so network, diff, trace/profiler, stream, dashboard, chat, and other structured results do not echo bearer tokens, proxy credentials, or similar fields verbatim into `details.data`. Echoed `command` arrays in `details` and in batch roll-ups use `redactInvocationArgs` from `extensions/agent-browser/lib/runtime.ts` to mask trailing values for sensitive global flags (including `--body`, `--headers`, `--password`, and `--proxy`), preserve the special positional rules for `cookies set`, `storage local|session set`, and `set credentials` (native `cookies set --curl <file>` imports are file operands, so ordinary import paths remain visible in echoes and failed batch diagnostics), and scrub other argv tokens for URLs and inline secrets. Raw batch row strings are parsed and command-redacted for presentation and ordered intent/completion journals; execution argv remains unchanged. Failed batch steps additionally run `redactExactValues` on structured step errors so literals taken from that step’s argv (cookie value, storage set value, `--password` / `--password=` tokens) cannot reappear inside formatted error blobs. When the full batch is large enough to need its own aggregate spill, that spill reapplies these per-command data and argv redactors before persistence rather than using generic batch redaction.
|
|
631
654
|
|
|
@@ -703,6 +726,7 @@ Implementation and precedence:
|
|
|
703
726
|
- The main tool implementation merges these fields into Pi-facing `details` from `extensions/agent-browser/index.ts` and from `extensions/agent-browser/lib/results/presentation.ts` for presentation-time failures.
|
|
704
727
|
|
|
705
728
|
Additional structured fields can appear when relevant:
|
|
729
|
+
|
|
706
730
|
- `sessionTabReopenPending: boolean` persists a confirmed-cold managed session's outstanding URL reopen through non-page commands and branch/reload replay. `true` means a daemon may have started but the remembered page has not been reopened; `false` means a reopen attempt or an executed explicit context/navigation command consumed the obligation, not that navigation succeeded. Cancellation after the reopen CLI starts returns `failureCategory: "aborted"`, the exact `sessionName` / `namespace`, the consumed `false` marker and ref invalidation through the normal result path, so replay cannot repeat the navigation. Cancellation before the CLI starts leaves the obligation pending. Successful close clears it with the rest of that session's page state. Internal remembered URLs retain their complete fragment; comparison remains fragment-insensitive and presentation redaction is unchanged. Old persisted targets without a fragment cannot reconstruct it.
|
|
707
731
|
- `closeAllApplied: true` when a successful direct or nested `close` / `quit` / `exit --all` reached upstream. The marker makes live state and transcript replay clear every managed/attached/page/ref/route/trace/recording identity in the effective canonical namespace; a later batch row that proves browser reactivation may rebuild only the effective session.
|
|
708
732
|
- `attachedBrowserSession: true` on successful calls that establish or reuse a wrapper-tracked CDP/auto-connect/Electron attachment, and on a failed fresh attachment only when `managedSessionOutcome.activeAfter` proves its daemon remained active for cleanup. The marker restores attachment continuity from the active transcript branch, including that active-after-failure case; live state and transcript replay remove it after a terminal successful close/cleanup even when aggregate verification failed; a close followed by a later step whose lifecycle reports a browser launch preserves it, while a successful non-launching diagnostic leaves the close terminal. Caller config, environment, paths, and file-access settings remain upstream-owned; the marker only adds live-URL verification and lifecycle continuity.
|
|
@@ -735,7 +759,7 @@ Additional structured fields can appear when relevant:
|
|
|
735
759
|
- `networkRequestsPageFilter` after wrapper-side `network requests --current-page`, `--current-origin`, or `--current-url`. Shape: `{ cleanArgs, currentUrl, mode, matchedRows, totalRows }`; the visible rows and `details.data.requests` / `items` / `entries` are filtered while the active session page target is read with `get url`.
|
|
736
760
|
- `richInputRecovery` after a raw `find` or compiled `agent_browser_action` `fill` fails with `selector-not-found` and the same current-ref diagnostic finds exact editable `searchbox` / `textbox` candidates. Shape follows `RichInputRecoveryDiagnostic` in `extensions/agent-browser/lib/results/selector-recovery.ts`: `{ candidates, inputMethodHint, nextActionIds, summary, target }`, where each candidate has `ref`, `role`, `name`, `focusArgs`, `clickArgs`, and `reason`. Visible text appends `Rich input recovery`, and `details.nextActions` gains ids from `getAgentBrowserRichInputRecoveryNextActionIds`: `focus-current-editable-ref` / `click-current-editable-ref` (or numbered variants). These actions are bounded to focus/click/inspect-style recovery: they do not include the fill text, do not press `Enter`, and do not submit. After the right current editable ref is focused, use `keyboard type` for framework-controlled editors that require real key events. Use paste-like `keyboard inserttext` only with separate application-state verification, and submit only when explicitly required by the flow.
|
|
737
761
|
- unsupported `scrollintoview text=<label>` / `scrollinto text=<label>` fails before upstream dispatch, directly or inside an effective raw/stdin batch row, because current upstream can report success without moving the page; help forms pass through unchanged. Visible failure text and `details.nextActions` both return the session-scoped `scroll-semantic-text-target` (`find text <label> hover`) when hover side effects are acceptable, plus `refresh-refs-for-scroll-target` (`snapshot -i`) before `scrollintoview <@ref>`. CSS, `xpath=...`, and current `@e…` targets remain native pass-through.
|
|
738
|
-
- `scrollPage` when the wrapper moves `document.scrollingElement` directly for `scroll <up|down|left|right> [px|percent]` or `scroll to end|top`; it temporarily disables smooth scrolling so immediate before/after offsets are reliable, returns `{ request, result }`, and includes `exitCode: 0` on success. Directional document no-movement falls through to upstream wheel behavior so nested panes still work. Explicit CSS-container calls `scroll <selector> <up|down|left|right> [px|percent]` remain wrapper-handled, use `Element.scrollBy` with `behavior: "instant"` before measuring movement so smooth-scroll CSS cannot cause a premature no-movement failure, and report `details.scrollContainer`. All scroll helper shims are skipped when startup-scoped flags are present so the requested browser/profile launches before any helper command.
|
|
762
|
+
- `scrollPage` when the wrapper moves `document.scrollingElement` directly for `scroll <up|down|left|right> [px|percent]` or `scroll to end|top`; it temporarily disables smooth scrolling so immediate before/after offsets are reliable, returns `{ request, result }`, and includes `exitCode: 0` on success. Directional document no-movement falls through to upstream wheel behavior so nested panes still work. Explicit CSS-container calls `scroll <selector> <up|down|left|right> [px|percent]` remain wrapper-handled, use `Element.scrollBy` with `behavior: "instant"` before measuring movement so smooth-scroll CSS cannot cause a premature no-movement failure, and report `details.scrollContainer`. Shims recognize only complete command shapes: exactly three tokens for `scroll to end|top`, or three/four for CSS-container scroll. Surplus operands, such as `scroll to end --selector #panel`, pass through to native agent-browser without a shim discarding them. All scroll helper shims are skipped when startup-scoped flags are present so the requested browser/profile launches before any helper command.
|
|
739
763
|
- `scrollNoop` after a nominally successful large **top-level** upstream scroll fallback on an existing or fresh managed session when wrapper-side read-only probes before and after the command show no change in `window.scrollX` / `window.scrollY` and no change in the sampled prominent scrollable containers. The wrapper reclassifies this outcome as `failureCategory: "upstream-error"` rather than claiming the page scrolled. To avoid pre-launching a session without caller startup state, this probe is skipped for small pixel scrolls, calls that would create a managed session only for the probe, and invocations with startup-scoped flags such as `--profile`, `--state`, `--restore`, `--namespace`, `--session-name`, `--cdp`, providers, init scripts, or similar launch settings. Shape: `{ reason: "no-observed-scroll-position-change", message, before, after, recommendations }`; `before` / `after` include viewport dimensions, document scroll dimensions, and up to ten sampled container descriptors plus scroll offsets. Container descriptors use only sample index, tag name, and ARIA role; DOM ids/classes are intentionally not stored. This diagnostic is conservative evidence that the page-level scroll likely missed a nested pane, not proof that every app-specific region is unchanged. Visible text starts with `Scroll completed with no observed movement`, appends `Scroll diagnostic: no observed scroll movement`, sets `details.data.scrolled` to `false` / `details.data.noMovement` to `true`, and `details.nextActions` gains `inspect-after-noop-scroll` (`snapshot -i`) plus `verify-noop-scroll-visually` (`screenshot`), session-prefixed when applicable.
|
|
740
764
|
- `comboboxFocus` after a successful explicit combobox-targeted `click` / `fill` / `find … click|fill` (for example `agent_browser_action` with role `combobox`, including when that semantic action resolves through a current visible `@ref` before execution) when a read-only probe sees the active element is combobox-like, `aria-expanded` is explicitly present (`false` or `true`), and no visible `listbox` / `option` / menu option elements are open. Shape: `{ reason: "focused-combobox-without-visible-options", message, activeElement, visibleListboxCount, visibleOptionCount, recommendations }`; `activeElement` includes bounded role/tag/expanded/hasPopup/name metadata with normal text redaction. Visible text appends `Combobox diagnostic: focused combobox did not expose visible options`, and `details.nextActions` gains `inspect-focused-combobox` (`snapshot -i`), `try-open-combobox-with-arrow` (`press ArrowDown`), and `try-open-combobox-with-enter` (`press Enter`), session-prefixed when applicable. The diagnostic is deliberately gated to explicit combobox-targeted calls to avoid extra probes or false positives on ordinary clicks/textboxes.
|
|
741
765
|
- `recordingDependencyWarning` after a successful `record start` or `record restart` when the wrapper cannot find an executable `ffmpeg` on the Pi process `PATH`. Shape: `{ reason: "ffmpeg-missing-for-recording", dependency: "ffmpeg", command, message, recommendations }`. Visible text appends `Recording dependency warning: ffmpeg not found on PATH`. This is a non-blocking warning after native success, not a preflight or encoding check. Native 0.37 validates ffmpeg at startup; older supported natives may defer failure. Treat the pending output as unverified, stop and check its result, then install ffmpeg before starting a new recording.
|
|
@@ -750,8 +774,8 @@ Additional structured fields can appear when relevant:
|
|
|
750
774
|
- `imagePath` / `imagePaths` for Pi inline image attachments from the **`screenshot`** command (including batched screenshot steps). **`diff screenshot`** still records the diff output as an `image`-kind entry in `details.artifacts`, but it does **not** populate `imagePath` / `imagePaths` or attach an inline image: only plain `screenshot` is treated as a trusted live-capture path for automatic inlining (`isTrustedScreenshotOutput` in `extensions/agent-browser/lib/results/presentation/artifacts.ts`).
|
|
751
775
|
- `artifacts` for saved files such as screenshots, `state save` outputs, `diff screenshot` diff images, PDFs, downloads, `wait --download` / `wait -d` files, traces, CPU profiles, completed video recordings, path-bearing HAR captures, and future recording output paths reported by `record start` / `record restart`. Non-file URL payloads such as `data:` / `blob:` / `http(s):` values are not treated as verified local artifacts. For direct artifact commands and batch artifact steps, the wrapper creates parent directories for requested paths before spawning upstream. Filesystem `mkdir` failures at this shared preparation boundary return `validation-error`, `agentBrowserStarted: false`, the attempted directory and `verify-artifact-path` guidance. When execution cwd differs from the launch/project root, relative file operands in raw batch strings are bound to the captured invocation root; otherwise raw strings stay unchanged and absolute artifact paths are preferred. Each artifact includes the original saved or requested `path`, resolved `absolutePath`, `kind`/`artifactType`, optional `mediaType`, optional `extension`, best-effort disk metadata such as `exists`, `sizeBytes`, and `updatedAtMs`, plus `requestedPath`, `status`, `cwd`, `session`, `namespace`, and `tempPath` when applicable. `requestedPath` is retained only when known from the caller, separately from reported/resolved locations; a differing screenshot report remains in `tempPath` and is displayed as `Reported path`, whether it is a temporary file or a canonical path alias. Ordinary file `mediaType` values come from bounded PNG/JPEG/GIF/WebP header recognition, not suffixes; unknown, missing, unreadable or truncated headers leave it undefined. Header recognition is not full-file format validation. Inline screenshot attachments use the same byte classifier and existing size limit. For commands that create/update artifacts, a path that existed but was not updated during this command uses `status: "stale"`; observational `wait --download` may accept a file completed just before the wait began. Pending `record start` / `record restart` artifacts use `status: "pending"`, omit `exists` rather than reporting false, and include `recordingState: "openRecording"` / `willExistOnStop: true`. Within one Pi extension process, the wrapper keeps an unbounded transcript-backed active-recording reservation index separate from the bounded artifact manifest, keyed by canonical namespace plus session; still-live process-owned reservations survive branch switches, while known closures are appended after tree navigation and during shutdown/reload so a close on one branch cannot be resurrected after returning to an older branch. Persisted active reservations require absolute storage paths and cwd; their display paths may remain relative. If a journal append fails, the next serialized browser boundary, tree navigation, or shutdown retries all current reservations and known closures. `recordingPersistenceWarning` and visible warning text remain present while restart protection is not durable; successful recovery is quiet and cleanup still runs. Artifact lifecycle calls, explicit `wait --download <path>` / `wait -d <path>` destinations, and result `outputPath` writes serialize around the global destination check/update, every successful direct, ordered nested-batch, fresh-replacement, code, Electron, or shutdown close retires only its exact identity at that lifecycle point, and destination reuse is rejected through lexical, existing or dangling symlink, hardlink, full Unicode-fold, or macOS/Windows case aliases. Batch preflight rejects `record start` / `record restart` after a close row because upstream can report a recording that did not start; split those operations into separate calls. A `No recording in progress` stop failure, direct or nested, first checks the matching native receipt once. Retirement preserves the receipt and freshly checked file metadata; it never turns an existing file into a missing file merely because no recording is active. A later successful batch recording row opens its new pending path normally. Batch preflight applies the same distinct-destination rule to the steps upstream will execute: raw argument command strings exclusively when any exist, stdin arrays only otherwise; upstream-ignored stdin rows cannot fail artifact preflight, add pending recordings, or create parent directories. Parent directories are prepared for the effective steps in both modes; outside execution-directory binding, raw argument strings stay unchanged, so screenshot normalization and tracked path requests apply to stdin rows only. Outer CLI globals are removed before artifact parsing, but native batch row operands stay literal: `pdf --quick ignored.pdf` targets `--quick`, not `ignored.pdf`. Reservation checks, preparation, and requested-path presentation follow that same distinction. Recording path/URL consumers skip complete numeric `--fps` pairs without rewriting argv; native still validates rate, format and extra arguments. FPS-only calls keep the intended pinned tab.
|
|
752
776
|
|
|
753
|
-
|
|
754
777
|
Recording destinations are reserved within one Pi process, not across processes. Use unique paths for concurrent Pi processes: different explicit sessions can overwrite one file even when both `record stop` results are verified. Upstream’s same-session `record start` guard does not reserve the filename across other sessions.
|
|
778
|
+
|
|
755
779
|
- `savedFilePath` / `savedFile` for direct `download`, `pdf`, and `wait --download` / `wait -d` saved-file workflows when a host file path is reported or wrapper-verified. Batch results preserve the same fields on the relevant `batchSteps` entry. These fields are metadata only until `artifactVerification` verifies the file. Native `download <selector> <path>` always owns the click and download, including loopback links, generated Blob exports, and redirects; the wrapper does not substitute a fetch of an anchor's `href`.
|
|
756
780
|
- `batchSteps[].artifacts` for per-step artifacts in `batch` output; top-level `artifacts` and `artifactManifest` coalesce an earlier pending recording into the later saved, missing, or stale terminal result for the same namespace/session identity. `record restart` includes both the previous recording it finalized (or an explicit missing/stale failure) and the new pending recording; missing/stale terminal rows retire the prior pending manifest row. A successful later `close` / `quit` / `exit` retires an earlier unfinalized pending recording as `subcommand: "close-abandoned"`, clears its stop action, and leaves its file unverified. Batch presentation checks the path: only a confirmed absent file becomes `status: "missing"`; a present or inaccessible file stays unverified. It updates aggregate verification/manifest state consistently; a later successful `record stop` replaces that intermediate abandoned row with its saved artifact. Close also resets ref/page/network-route state produced by earlier rows; later lifecycle-proven browser launches, including `record stop`, can rebuild that state without triggering stale pre-close `about:blank` recovery, explicitly non-launching diagnostics cannot, and unknown later rows stay conservatively active. Per-step history remains unchanged. When any later call on the same namespace/session fails while a recording remains pending, `nextActions` combines its normal recovery with exact `stop-pending-recording` args and visible cleanup guidance; the same applies at top level when a later batch step fails. After reload in a non-Git checkout or with managed restore disabled, a live daemon without current-instance provenance or a matching same-owner native daemon receipt cannot accept a stop. A tracked Electron attachment can rebuild that proof through the live debug-endpoint check described above; unproven restore-disabled sessions cannot rebuild provenance from a name or URL. That policy refusal includes `managedSessionCleanupOnlyReason: "restore-disabled-daemon-without-provenance"` plus the exact `sessionName`/`namespace`, including on implicit calls. It replaces the impossible stop with `close-pending-recording`, an exact close without `sessionMode: "fresh"`. Close retires the recording as `close-abandoned`; any file it leaves is unverified. Same-instance recordings and supported durable-Git reloads still use stop and normal WebM verification.
|
|
757
781
|
- `artifactVerification` for a normalized verification summary on the unified result and on each successful `batchSteps[]` row and on failed recording rows whose receipt identifies an artifact. Top-level `batch` verification rolls up all step file artifacts; each step’s summary reflects that step’s nested tool presentation (including its spill paths and manifest slice). It reports `verified`, `verifiedCount`, `missingCount`, `pendingCount`, `unverifiedCount`, and `artifacts[]` entries with `path`, optional `absolutePath`, optional `requestedPath`, `kind` (a normal file artifact kind or `"spill"` for manifest-backed rows), optional `mediaType`, optional `exists`, optional `sizeBytes`, optional `updatedAtMs`, optional `status`, optional `retentionState` / `storageScope` on manifest-derived rows, `state` (`verified`, `missing`, `pending`, or `unverified`), and optional `limitation` (human-readable lifecycle or retention context, for example pending `record start` / `record restart`, missing, stale, or otherwise unverified files, ephemeral spill files, or evicted persisted spills). The summary `verified` boolean is true only when every entry is `verified`. `record start` / `record restart` are `pending` until `record stop`; `state load` may mention a path in command output but is not a saved artifact row.
|
|
@@ -806,6 +830,7 @@ A failed inner native confirmed action is a tool failure, including inside a bat
|
|
|
806
830
|
The TUI renderer is user-facing only. It may compact or colorize what the human sees in the Pi transcript, but it must not further truncate, summarize, or remove the model-facing `content` returned by the tool. Use the existing `details.fullOutputPath` / spill-file contracts for content that is too large for the model.
|
|
807
831
|
|
|
808
832
|
Worth doing in v1:
|
|
833
|
+
|
|
809
834
|
- screenshots → saved-path summary, visible artifact metadata, `details.artifacts` metadata, and inline image attachment when safe; screenshot paths that upstream would treat ambiguously, such as `.dogfood/run/foo.png`, are normalized to absolute paths before launch and repaired from upstream temp output when possible
|
|
810
835
|
- file artifacts such as PDFs, downloads, `wait --download` / `wait -d` files, `state save` state files, diff screenshot output images, traces, CPU profiles, completed video recordings, and path-bearing HAR captures → concise saved-path summaries plus metadata in `details.artifacts` and changed/referenced invocation receipts in `details.artifactManifest`; `record start` / `record restart` report recording lifecycle state and the future output path without adding a missing manifest entry, and `record restart` can also report the previous wrapper-known recording that was finalized by the restart; native 0.37 checks `ffmpeg` before starting video capture, while older supported natives may defer failure; successful start/restart calls without ffmpeg expose `details.recordingDependencyWarning` and leave output unverified until checked after stop; direct saved-file workflows also expose `details.savedFilePath` / `details.savedFile`; large or binary artifacts are not inlined into model context; the recent manifest cap can age out explicit-file metadata but does not remove explicit saved files from disk
|
|
811
836
|
- `diff screenshot` → same file-artifact pattern as above for the **diff** image path only (summary text uses “Saved diff image” only when the diff output exists; missing output says “Diff image reported; file not verified” and fails as `artifact-missing`); baseline paths and other fields stay in the structured payload but are not echoed as separate saved artifacts in the visible artifact block, and there is no Pi inline image attachment for the diff output
|
|
@@ -825,6 +850,7 @@ Worth doing in v1:
|
|
|
825
850
|
## Missing binary behavior
|
|
826
851
|
|
|
827
852
|
If `agent-browser` is not on `PATH`, fail with a message that:
|
|
853
|
+
|
|
828
854
|
- says `agent-browser` is required
|
|
829
855
|
- says this project does not bundle it
|
|
830
856
|
- points to upstream install/docs
|
|
@@ -851,15 +877,19 @@ If `agent-browser` is not on `PATH`, fail with a message that:
|
|
|
851
877
|
- treat every explicit caller-provided `--session` as user-managed, including `piab-*` names. Wrapper-owned implicit sessions set a Pi-transcript- and Git-checkout-generation-scoped `AGENT_BROWSER_RESTORE` key automatically unless disabled with `PI_AGENT_BROWSER_MANAGED_SESSION_RESTORE=0`; explicit caller sessions do not receive that injection unless they exactly target the current wrapper-owned identity. Caller state/restore paths, profiles, upstream config, file access, launch arguments, environment variables, local file pages, `outputPath`, and close arguments pass through unchanged. `session list` and `state list` keep all upstream rows and restore identifiers visible. Automatic restore still validates and pins its own checkout/storage/namespace identity and coordinates same-daemon reuse so the wrapper cannot mix restore pools or corrupt managed lifecycle state. Ambiguous tab, attachment, history, code, or state-load transitions remain page-target correctness boundaries: content calls live-check `get url` or require explicit navigation before acting. Windows uses `cross-spawn` for native executable and `.cmd` argument transport, rather than PowerShell or wrapper-owned argument reordering. Empty operands such as `fill #field ""`, `--args ""`, and explicit default `--namespace ""`, literal doublequotes in fill text, and command/subcommand adjacency are retained. Upstream receives the empty namespace rather than a wrapper omission or environment workaround. The selected child `PATH` shim is not bypassed; POSIX keeps native Node `spawn`.
|
|
852
878
|
- before a DOM/content-bearing read or interaction against a caller-owned explicit session or established attachment, run a session-scoped `get url` probe so stale transcript state cannot target the wrong page. A failed or non-URL probe blocks the requested content command. The shared execution lock orders that probe, semantic snapshot resolution, and the main command across cooperating updated Pi processes. Nested `batch` steps remain unsupported; raw batch command strings mirror upstream's ASCII-space tokenizer, including quoting and backslash handling.
|
|
853
879
|
- pass explicit `--profile` straight through to upstream `agent-browser`; no profile-cloning or isolation layer is added in v1
|
|
880
|
+
|
|
854
881
|
<!-- agent-browser-playbook:start wrapper-tab-recovery -->
|
|
855
882
|
<!-- Generated from extensions/agent-browser/lib/playbook.ts. Run `npm run docs -- playbook write` to update. -->
|
|
883
|
+
|
|
856
884
|
- After open/goto/navigate calls with --profile, --restore, --session-name, or --state, agent_browser best-effort re-selects the tab whose URL matches the returned page when restored tabs steal focus during launch or reconnect.
|
|
857
885
|
- After confirmed shutdown of an automatically restored managed session, the wrapper retains its complete recorded URL, including the fragment, until the first current-page operation (including get url and reload). Non-page calls such as tab list may start a daemon without fulfilling that reopen; explicit URL reads leave the managed browser and pending reopen untouched. The wrapper uses native open once, verifies the observed tab, and discards old refs/frame scope; it does not restore unsaved forms, JavaScript memory, or history. Explicit navigation, caller-owned/attached sessions, and restore-disabled sessions are not auto-reopened.
|
|
858
886
|
- For a still-live browser after tab drift or resume, the wrapper verifies/selects the intended tab before ref/semantic helpers and page commands; failed selection stops the call without navigating. Local commands, read <url>, URL a11y/vitals, diff url, window new, and explicit tab/navigation/connection/state recovery do not require the prior tab. Batch checks follow effective rows past non-page prefixes and stop at explicit context changes, preserving caller argv/stdin and continue-on-error behavior. Same-tab reselection is avoided because it clears refs. Use exact batch --bail for fail-fast, not --bail=<value>. Routine same-session calls skip tab-list preflights.
|
|
859
887
|
- For sessions with observed tab-drift risk, after a successful command on a known target tab, agent_browser also best-effort restores that intended tab if a restored/background tab steals focus after the command completes. Routine same-session commands skip focus correction; observed pages without a native targetId use tab list to establish their identity.
|
|
860
888
|
- If a known session target unexpectedly reports about:blank, agent_browser best-effort re-selects the prior intended target when it still exists; if recovery fails, it records the observed about:blank target and reports exact recovery guidance instead of treating the prior page as active.
|
|
861
889
|
- If upstream reports tab_gone, the pinned bound tab is gone; use details.nextActions (tab list / tab new) instead of assuming another tab is yours.
|
|
890
|
+
|
|
862
891
|
<!-- agent-browser-playbook:end wrapper-tab-recovery -->
|
|
892
|
+
|
|
863
893
|
- caller-owned sessions honor native `AGENT_BROWSER_SOCKET_DIR` unless the wrapper-specific socket override is set, using the same integrity checks; on other local Unix launches, set a short private socket directory for wrapper-spawned `agent-browser` processes so extension-generated session names do not fail the upstream Unix socket-path length limit in longer cwd/session-name combinations; require an absolute non-symlink directory owned by the current uid with mode `0700`, otherwise fail before spawn. Socket checks trust the operating environment's actual `/`, not its reported UID; all non-root ownership, permission and alias-destination checks remain in force. This is not protection from the controller of the root filesystem; see [socket trust](ARCHITECTURE.md#ownership). Android/Termux uses a short directory under the owner-only `/data/data/<package>` app sandbox, compacts generated managed identities to one 80-bit digest so ordinary namespace plus fresh-session paths remain within the limit, stores policy-lock coordination under `os.tmpdir()`, and probes process identity with Termux's `ps` beside Node instead of unavailable `/bin/ps`
|
|
864
894
|
- keep wrapper-spawned commands bounded by clamping `AGENT_BROWSER_DEFAULT_TIMEOUT` to the upstream documented default of 25 seconds while the default wrapper child-process watchdog is 35 seconds (`PI_AGENT_BROWSER_PROCESS_TIMEOUT_MS` overrides it, and top-level `timeoutMs` overrides it per call for browser CLI subprocesses). Explicit `wait <ms>`, `wait --timeout <ms>`, and WebMCP `invoke` / `result --timeout <ms>` calls can exceed that default; when top-level `timeoutMs` is omitted, the wrapper derives a per-call subprocess watchdog from the requested command duration plus a small grace window. Dialog commands use `PI_AGENT_BROWSER_DIALOG_PROCESS_TIMEOUT_MS` (default 5000 ms), and click/tap/find refs or tokens plus `eval --stdin` snippets whose text looks like alert/confirm/prompt/dialog triggers use `PI_AGENT_BROWSER_DIALOG_TRIGGER_PROCESS_TIMEOUT_MS` (default 8000 ms). Timed-out advanced QA or native batch calls may add `details.timeoutPartialProgress` and visible `Timeout partial progress` evidence with per-step status, retry payloads, current page title/URL, and declared artifact path checks; timed-out dialog-like commands may add dialog status/dismiss/fresh-session recovery next actions
|
|
865
895
|
- interactive or long-running upstream families such as `chat` without a prompt, `dashboard start`, `stream enable`, `trace start`, `profiler start`, `record start`, `inspect`, `install`, `upgrade`, `doctor --fix`, and `confirm-interactive` are passed through thinly but remain bounded by the same wrapper timeout/session planning rules; prefer explicit arguments, single-shot `chat <message>`, non-interactive flags like `doctor --offline --quick` or `doctor --json`, and cleanup pairs such as `dashboard stop`, `stream disable`, `trace stop`, `profiler stop`, and `record stop`
|
|
@@ -877,35 +907,38 @@ This is neither a browser transaction nor rollback. It does not coordinate human
|
|
|
877
907
|
|
|
878
908
|
The 0.7 public contract removes old multimode `agent_browser` inputs. Native CLI coverage, authentication, recordings, artifacts, diagnostics, and specialized outcomes remain available.
|
|
879
909
|
|
|
880
|
-
| Previous 0.6 input
|
|
881
|
-
|
|
|
882
|
-
| `agent_browser { args, stdin?, outputPath?, timeoutMs?, sessionMode? }` | Same compact direct shape
|
|
883
|
-
| `agent_browser { script: source }`
|
|
884
|
-
| `agent_browser { job: { steps, failFast? } }`
|
|
885
|
-
| `agent_browser { semanticAction: fields }`
|
|
886
|
-
| `agent_browser { qa: fields }`
|
|
887
|
-
| `agent_browser { electron: fields }`
|
|
888
|
-
| `agent_browser { sourceLookup: fields }`
|
|
889
|
-
| `agent_browser { networkSourceLookup: fields }`
|
|
910
|
+
| Previous 0.6 input | 0.7 replacement |
|
|
911
|
+
| ----------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
|
|
912
|
+
| `agent_browser { args, stdin?, outputPath?, timeoutMs?, sessionMode? }` | Same compact direct shape |
|
|
913
|
+
| `agent_browser { script: source }` | `agent_browser_code { code: source, session?, namespace?, timeoutMs?, outputPath? }`; check `.success`, not `.ok`; browser now persists |
|
|
914
|
+
| `agent_browser { job: { steps, failFast? } }` | Native `batch --bail` with argv rows; omit `--bail` only for deliberate safe continuation, or use code for branching |
|
|
915
|
+
| `agent_browser { semanticAction: fields }` | Enable `action`, then `agent_browser_action { ...fields }` |
|
|
916
|
+
| `agent_browser { qa: fields }` | Enable `qa`, then `agent_browser_qa { ...fields }` |
|
|
917
|
+
| `agent_browser { electron: fields }` | Enable `electron`, then `agent_browser_electron { ...fields }` |
|
|
918
|
+
| `agent_browser { sourceLookup: fields }` | Enable `source`, then `agent_browser_source { ...fields }` |
|
|
919
|
+
| `agent_browser { networkSourceLookup: fields }` | Enable `network`, then `agent_browser_network_source { ...fields }` |
|
|
890
920
|
|
|
891
921
|
Move former outer `outputPath`/`timeoutMs` alongside advanced flat fields where supported. Action has optional `session` but no `sessionMode`; QA/source/network retain `sessionMode`; Electron `timeoutMs` is flat and action-specific (`list` rejects it). Code chooses session/namespace in the parent.
|
|
892
922
|
|
|
893
|
-
| Former job step
|
|
894
|
-
|
|
|
895
|
-
| `open` plus `loadState`
|
|
896
|
-
| Selector `click` / `fill`
|
|
897
|
-
| Locator click/fill
|
|
898
|
-
| `select`
|
|
899
|
-
| `wait`
|
|
900
|
-
| `assertText` / `assertUrl` | `wait --text <text>` / `wait --url <url-or-glob>`
|
|
901
|
-
| `waitForDownload`
|
|
902
|
-
| `snapshot` / `screenshot`
|
|
903
|
-
| Paced `type`
|
|
923
|
+
| Former job step | Native argv rows |
|
|
924
|
+
| -------------------------- | --------------------------------------------------------------------------------------------------------------------- |
|
|
925
|
+
| `open` plus `loadState` | `open <url>`, then `wait --load <state>` |
|
|
926
|
+
| Selector `click` / `fill` | Ordinary command with selector and optional text |
|
|
927
|
+
| Locator click/fill | `find <locator> <value> <action> ...` |
|
|
928
|
+
| `select` | `select <selector> <value...>` |
|
|
929
|
+
| `wait` | `wait <milliseconds>` |
|
|
930
|
+
| `assertText` / `assertUrl` | `wait --text <text>` / `wait --url <url-or-glob>` |
|
|
931
|
+
| `waitForDownload` | `wait --download <path>` |
|
|
932
|
+
| `snapshot` / `screenshot` | `snapshot -i` / `screenshot <path>` |
|
|
933
|
+
| Paced `type` | Optional `focus <selector>`, interleaved `keyboard type <character>` / `wait <delayMs>`, optional final `press <key>` |
|
|
904
934
|
|
|
905
935
|
For exact delayed-input behavior, build native rows instead of relying on unverified native `--delay` equivalence. Keep the former 200-character pacing bound when porting that workflow. Typing `hi` with a 20 ms gap:
|
|
906
936
|
|
|
907
937
|
```json
|
|
908
|
-
{
|
|
938
|
+
{
|
|
939
|
+
"args": ["batch", "--bail"],
|
|
940
|
+
"stdin": "[[\"focus\",\"#prompt\"],[\"keyboard\",\"type\",\"h\"],[\"wait\",\"20\"],[\"keyboard\",\"type\",\"i\"]]"
|
|
941
|
+
}
|
|
909
942
|
```
|
|
910
943
|
|
|
911
944
|
Fixed waits are elapsed time, not postconditions. Preserve explicit URL/text waits, same-snapshot fill ordering, fresh refs after mutations, and partial-progress inspection. There is no legacy public script/job runtime. Exact isolated-script cleanup leases are normalized only by offline conversion so canonical replay can retire their recorded pre-upgrade resources. Saved auth and original Pi transcripts are not rewritten or deleted; [rollback](RELEASE.md#07-upgrade-and-rollback) changes the package version, not browser side effects.
|