@vellumai/assistant 0.6.5 → 0.6.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +9 -1
- package/ARCHITECTURE.md +15 -17
- package/Dockerfile +6 -4
- package/__tests__/permissions/gateway-threshold-reader.test.ts +283 -0
- package/docs/architecture/integrations.md +32 -39
- package/docs/architecture/memory.md +25 -30
- package/docs/architecture/security.md +7 -6
- package/docs/browser-use-architecture-phase2.md +63 -20
- package/docs/plugins.md +761 -0
- package/examples/plugins/echo/README.md +132 -0
- package/examples/plugins/echo/package.json +17 -0
- package/examples/plugins/echo/register.ts +187 -0
- package/node_modules/@vellumai/egress-proxy/src/types.ts +19 -0
- package/openapi.yaml +212 -68
- package/package.json +1 -1
- package/src/__tests__/app-compiler.test.ts +57 -0
- package/src/__tests__/approval-cascade.test.ts +7 -2
- package/src/__tests__/auto-analysis-end-to-end.test.ts +1 -0
- package/src/__tests__/avatar-generator.test.ts +4 -2
- package/src/__tests__/bundled-asset.test.ts +6 -6
- package/src/__tests__/catalog-cache.test.ts +69 -0
- package/src/__tests__/checker.test.ts +459 -171
- package/src/__tests__/circuit-breaker-pipeline.test.ts +406 -0
- package/src/__tests__/compaction-events.test.ts +501 -0
- package/src/__tests__/compaction-pipeline.test.ts +210 -0
- package/src/__tests__/compaction-strip-metadata-clear.test.ts +181 -0
- package/src/__tests__/compaction-timeout-recovery.test.ts +262 -0
- package/src/__tests__/config-model-image-provider.test.ts +110 -0
- package/src/__tests__/config-schema.test.ts +22 -9
- package/src/__tests__/config-watcher-cleanup-throttle.test.ts +0 -4
- package/src/__tests__/contacts-tools.test.ts +26 -0
- package/src/__tests__/context-overflow-policy.test.ts +7 -7
- package/src/__tests__/context-window-manager.test.ts +355 -4
- package/src/__tests__/conversation-abort-tool-results.test.ts +4 -1
- package/src/__tests__/conversation-agent-loop-overflow.test.ts +26 -30
- package/src/__tests__/conversation-agent-loop.test.ts +30 -141
- package/src/__tests__/conversation-confirmation-signals.test.ts +6 -1
- package/src/__tests__/conversation-history-web-search.test.ts +1 -0
- package/src/__tests__/conversation-init.benchmark.test.ts +2 -16
- package/src/__tests__/conversation-pairing.test.ts +174 -10
- package/src/__tests__/conversation-pre-run-repair.test.ts +4 -1
- package/src/__tests__/conversation-process-callsite.test.ts +3 -0
- package/src/__tests__/conversation-provider-retry-repair.test.ts +16 -7
- package/src/__tests__/conversation-queue.test.ts +29 -14
- package/src/__tests__/conversation-routes-disk-view.test.ts +7 -6
- package/src/__tests__/conversation-runtime-assembly.test.ts +155 -110
- package/src/__tests__/conversation-runtime-workspace.test.ts +23 -38
- package/src/__tests__/conversation-seed-composer.test.ts +2 -2
- package/src/__tests__/conversation-slash-queue.test.ts +7 -2
- package/src/__tests__/conversation-slash-unknown.test.ts +25 -2
- package/src/__tests__/conversation-speed-override.test.ts +6 -1
- package/src/__tests__/conversation-title-service.test.ts +116 -0
- package/src/__tests__/conversation-tool-setup-app-refresh.test.ts +41 -2
- package/src/__tests__/conversation-usage.test.ts +1 -1
- package/src/__tests__/conversation-workspace-cache-state.test.ts +4 -1
- package/src/__tests__/conversation-workspace-injection.test.ts +3 -0
- package/src/__tests__/conversation-workspace-tool-tracking.test.ts +4 -1
- package/src/__tests__/credential-health-service.test.ts +78 -9
- package/src/__tests__/credential-security-invariants.test.ts +2 -2
- package/src/__tests__/db-schedule-syntax-migration.test.ts +1 -0
- package/src/__tests__/empty-response-pipeline.test.ts +305 -0
- package/src/__tests__/extension-id-sync-guard.test.ts +3 -3
- package/src/__tests__/first-greeting.test.ts +247 -5
- package/src/__tests__/headless-browser-mode.test.ts +57 -0
- package/src/__tests__/history-repair-pipeline.test.ts +399 -0
- package/src/__tests__/host-browser-e2e-cloud.test.ts +307 -0
- package/src/__tests__/host-browser-e2e-self-hosted.test.ts +3 -3
- package/src/__tests__/host-proxy-interface.test.ts +36 -2
- package/src/__tests__/image-credentials.test.ts +137 -0
- package/src/__tests__/image-service-dispatcher.test.ts +186 -0
- package/src/__tests__/injector-chain.test.ts +526 -0
- package/src/__tests__/intent-routing.test.ts +0 -26
- package/src/__tests__/llm-call-pipeline.test.ts +285 -0
- package/src/__tests__/llm-schema.test.ts +1 -1
- package/src/__tests__/media-generate-image.test.ts +119 -13
- package/src/__tests__/memory-retrieval-pipeline.test.ts +401 -0
- package/src/__tests__/memory-upsert-concurrency.test.ts +1 -0
- package/src/__tests__/migration-import-from-url.test.ts +5 -68
- package/src/__tests__/model-intents.test.ts +4 -2
- package/src/__tests__/notification-broadcaster.test.ts +3 -3
- package/src/__tests__/notification-decision-strategy.test.ts +0 -11
- package/src/__tests__/notification-schedule-notify-dedup.test.ts +108 -0
- package/src/__tests__/oauth-apps-routes.test.ts +1 -1
- package/src/__tests__/oauth-cli.test.ts +14 -12
- package/src/__tests__/oauth-connect-orchestrator.test.ts +4 -13
- package/src/__tests__/oauth-provider-serializer.test.ts +6 -4
- package/src/__tests__/oauth-provider-visibility.test.ts +3 -5
- package/src/__tests__/oauth-providers-routes.test.ts +3 -2
- package/src/__tests__/oauth-store.test.ts +41 -76
- package/src/__tests__/onboarding-template-contract.test.ts +16 -64
- package/src/__tests__/openai-image-service.test.ts +368 -0
- package/src/__tests__/overflow-reduce-pipeline.test.ts +676 -0
- package/src/__tests__/permission-checker-host-gate.test.ts +0 -24
- package/src/__tests__/persist-onboarding-artifacts.test.ts +266 -0
- package/src/__tests__/persistence-pipeline.test.ts +377 -0
- package/src/__tests__/pipeline-runner.test.ts +565 -0
- package/src/__tests__/platform.test.ts +5 -2
- package/src/__tests__/plugin-bootstrap.test.ts +483 -0
- package/src/__tests__/plugin-registry.test.ts +273 -0
- package/src/__tests__/plugin-route-contribution.test.ts +288 -0
- package/src/__tests__/plugin-skill-contribution.test.ts +367 -0
- package/src/__tests__/plugin-tool-contribution.test.ts +286 -0
- package/src/__tests__/plugin-types.test.ts +320 -0
- package/src/__tests__/pricing.test.ts +44 -12
- package/src/__tests__/proxy-approval-callback.test.ts +69 -8
- package/src/__tests__/reaction-persistence.test.ts +1 -0
- package/src/__tests__/regenerate-fire-and-forget-trace.test.ts +1 -0
- package/src/__tests__/registry.test.ts +0 -2
- package/src/__tests__/schedule-routes.test.ts +131 -1
- package/src/__tests__/scheduler-recurrence.test.ts +14 -70
- package/src/__tests__/scheduler-reuse-conversation.test.ts +10 -50
- package/src/__tests__/secret-detection-handler.test.ts +0 -10
- package/src/__tests__/shell-identity.test.ts +0 -134
- package/src/__tests__/suggestion-routes.test.ts +103 -4
- package/src/__tests__/task-memory-cleanup.test.ts +1 -0
- package/src/__tests__/task-scheduler.test.ts +3 -15
- package/src/__tests__/test-preload.ts +11 -0
- package/src/__tests__/title-generate-pipeline.test.ts +224 -0
- package/src/__tests__/token-estimate-pipeline.test.ts +431 -0
- package/src/__tests__/tool-error-pipeline.test.ts +244 -0
- package/src/__tests__/tool-execute-pipeline.test.ts +431 -0
- package/src/__tests__/tool-execution-pipeline.benchmark.test.ts +0 -6
- package/src/__tests__/tool-executor-shell-integration.test.ts +7 -10
- package/src/__tests__/tool-executor.test.ts +141 -0
- package/src/__tests__/tool-result-truncate-pipeline.test.ts +356 -0
- package/src/__tests__/tool-result-truncation.test.ts +0 -110
- package/src/__tests__/user-plugin-loader.test.ts +191 -0
- package/src/__tests__/workspace-migration-046-seed-conversation-starters-callsite.test.ts +185 -0
- package/src/__tests__/workspace-migration-049-release-notes-default-sonnet.test.ts +100 -0
- package/src/__tests__/workspace-migration-050-seed-main-agent-opus-callsite.test.ts +171 -0
- package/src/__tests__/workspace-migration-051-seed-conversation-summarization-callsite.test.ts +252 -0
- package/src/__tests__/workspace-migration-remove-hooks.test.ts +99 -0
- package/src/__tests__/workspace-policy.test.ts +21 -3
- package/src/agent/loop.ts +340 -102
- package/src/approvals/__tests__/guardian-feed-event.test.ts +304 -0
- package/src/approvals/guardian-request-resolvers.ts +80 -0
- package/src/backup/__tests__/backup-worker.test.ts +2 -13
- package/src/backup/backup-worker.ts +3 -15
- package/src/bundler/app-compiler.ts +84 -1
- package/src/calls/call-state.ts +2 -2
- package/src/channels/__tests__/types.test.ts +3 -3
- package/src/channels/types.ts +6 -4
- package/src/cli/__tests__/notifications.test.ts +87 -211
- package/src/cli/commands/__tests__/backup.test.ts +1 -1
- package/src/cli/commands/__tests__/image-generation.test.ts +255 -35
- package/src/cli/commands/__tests__/inference-send.test.ts +12 -0
- package/src/cli/commands/__tests__/tts-synthesize.test.ts +12 -0
- package/src/cli/commands/backup.ts +2 -2
- package/src/cli/commands/clients.ts +138 -0
- package/src/cli/commands/completions.ts +2 -9
- package/src/cli/commands/conversations.ts +55 -7
- package/src/cli/commands/image-generation.ts +33 -34
- package/src/cli/commands/notifications.ts +68 -103
- package/src/cli/commands/oauth/__tests__/providers-register.test.ts +1 -1
- package/src/cli/commands/oauth/__tests__/providers-update.test.ts +1 -1
- package/src/cli/commands/oauth/connect.ts +2 -2
- package/src/cli/commands/oauth/providers.ts +176 -8
- package/src/cli/commands/oauth/status.ts +46 -36
- package/src/cli/commands/skills.ts +3 -4
- package/src/cli/program.ts +25 -29
- package/src/config/__tests__/backup-schema.test.ts +7 -2
- package/src/config/bundled-skills/app-builder/SKILL.md +2 -2
- package/src/config/bundled-skills/app-builder/references/WIDGETS.md +10 -10
- package/src/config/bundled-skills/contacts/tools/contact-merge.ts +66 -87
- package/src/config/bundled-skills/contacts/tools/contact-search.ts +28 -51
- package/src/config/bundled-skills/contacts/tools/contact-upsert.ts +22 -40
- package/src/config/bundled-skills/image-studio/SKILL.md +2 -1
- package/src/config/bundled-skills/image-studio/TOOLS.json +2 -1
- package/src/config/bundled-skills/image-studio/tools/media-generate-image.ts +23 -39
- package/src/config/bundled-skills/messaging/SKILL.md +3 -3
- package/src/config/bundled-skills/messaging/tools/__tests__/messaging-feed-events.test.ts +207 -0
- package/src/config/bundled-skills/messaging/tools/messaging-archive-by-sender.ts +12 -0
- package/src/config/bundled-skills/messaging/tools/messaging-send.ts +58 -0
- package/src/config/bundled-skills/schedule/SKILL.md +8 -3
- package/src/config/bundled-skills/schedule/TOOLS.json +15 -7
- package/src/config/bundled-skills/schedule/references/SCRIPT_MODE_PATTERNS.md +59 -0
- package/src/config/bundled-tool-registry.ts +0 -15
- package/src/config/feature-flag-registry.json +17 -1
- package/src/config/schema.ts +19 -0
- package/src/config/schemas/backup.ts +1 -1
- package/src/config/schemas/conversations.ts +16 -0
- package/src/config/schemas/llm.ts +2 -3
- package/src/config/schemas/security.ts +6 -6
- package/src/config/schemas/tts.ts +11 -0
- package/src/config/skill-state.ts +6 -2
- package/src/config/skills.ts +94 -5
- package/src/context/__tests__/compact-prompt.test.ts +27 -9
- package/src/context/prompts/compact.md +26 -12
- package/src/context/tool-result-truncation.ts +3 -63
- package/src/context/window-manager.ts +190 -16
- package/src/credential-health/credential-health-service.ts +19 -6
- package/src/daemon/__tests__/conversation-feed-event.test.ts +317 -0
- package/src/daemon/__tests__/conversation-lifecycle-auto-analyze.test.ts +4 -12
- package/src/daemon/__tests__/conversation-tool-setup.test.ts +14 -15
- package/src/daemon/config-watcher.ts +0 -2
- package/src/daemon/context-overflow-policy.ts +4 -13
- package/src/daemon/conversation-agent-loop-handlers.ts +83 -22
- package/src/daemon/conversation-agent-loop.ts +984 -683
- package/src/daemon/conversation-history.ts +10 -19
- package/src/daemon/conversation-lifecycle.ts +37 -19
- package/src/daemon/conversation-notifiers.ts +2 -110
- package/src/daemon/conversation-process.ts +14 -7
- package/src/daemon/conversation-runtime-assembly.ts +532 -411
- package/src/daemon/conversation-tool-setup.ts +41 -4
- package/src/daemon/conversation.ts +80 -35
- package/src/daemon/external-plugins-bootstrap.ts +478 -0
- package/src/daemon/first-greeting.ts +191 -14
- package/src/daemon/handlers/config-model.ts +11 -0
- package/src/daemon/handlers/skills.ts +5 -1
- package/src/daemon/lifecycle.ts +33 -68
- package/src/daemon/message-types/computer-use.ts +2 -34
- package/src/daemon/message-types/conversations.ts +49 -0
- package/src/daemon/message-types/messages.ts +12 -0
- package/src/daemon/server.ts +5 -3
- package/src/daemon/shutdown-handlers.ts +2 -12
- package/src/daemon/tool-side-effects.ts +14 -56
- package/src/heartbeat/__tests__/heartbeat-feed-event.test.ts +160 -0
- package/src/heartbeat/heartbeat-service.ts +24 -1
- package/src/home/__tests__/feed-population-integration.test.ts +312 -0
- package/src/home/emit-feed-event.ts +7 -0
- package/src/home/feed-types.ts +41 -2
- package/src/home/rewrite-command-preview.ts +66 -0
- package/src/ipc/__tests__/socket-path.test.ts +11 -50
- package/src/ipc/cli-client.ts +1 -1
- package/src/ipc/cli-server.ts +3 -3
- package/src/ipc/gateway-client.ts +4 -1
- package/src/ipc/routes/browser-context.ts +2 -0
- package/src/ipc/routes/browser.ts +1 -0
- package/src/ipc/routes/get-contact.ts +16 -0
- package/src/ipc/routes/index.ts +14 -0
- package/src/ipc/routes/list-clients.ts +31 -0
- package/src/ipc/routes/merge-contacts.ts +17 -0
- package/src/ipc/routes/notification.ts +133 -0
- package/src/ipc/routes/rename-conversation.ts +59 -0
- package/src/ipc/routes/search-contacts.ts +19 -0
- package/src/ipc/routes/upsert-contact.ts +25 -0
- package/src/ipc/socket-path.ts +14 -38
- package/src/media/app-icon-generator.ts +23 -46
- package/src/media/avatar-router.ts +26 -41
- package/src/media/gemini-image-service.ts +8 -41
- package/src/media/image-credentials.ts +73 -0
- package/src/media/image-service.ts +85 -0
- package/src/media/openai-image-service.ts +131 -0
- package/src/media/types.ts +46 -0
- package/src/memory/conversation-crud.ts +48 -18
- package/src/memory/conversation-queries.ts +57 -4
- package/src/memory/conversation-title-service.ts +25 -0
- package/src/memory/db-init.ts +8 -0
- package/src/memory/embedding-gemini.test.ts +41 -2
- package/src/memory/embedding-gemini.ts +6 -1
- package/src/memory/graph/bootstrap.test.ts +282 -0
- package/src/memory/graph/bootstrap.ts +8 -5
- package/src/memory/graph/extraction.ts +10 -2
- package/src/memory/graph/graph-search.test.ts +1 -0
- package/src/memory/graph/inspect.ts +2 -2
- package/src/memory/graph/retriever.ts +10 -3
- package/src/memory/migrations/041-approval-prompt-ts-tracker.ts +26 -0
- package/src/memory/migrations/149-oauth-tables.ts +1 -0
- package/src/memory/migrations/223-schedule-script-column.ts +11 -0
- package/src/memory/migrations/224-oauth-providers-managed-service-is-paid.ts +24 -0
- package/src/memory/migrations/225-oauth-providers-available-scopes.ts +13 -0
- package/src/memory/migrations/index.ts +4 -0
- package/src/memory/pkb/pkb-index.test.ts +1 -0
- package/src/memory/pkb/pkb-reconcile.test.ts +1 -0
- package/src/memory/pkb/pkb-search.test.ts +65 -4
- package/src/memory/pkb/pkb-search.ts +40 -18
- package/src/memory/qdrant-client.test.ts +60 -0
- package/src/memory/qdrant-client.ts +25 -0
- package/src/memory/schema/infrastructure.ts +1 -0
- package/src/memory/schema/oauth.ts +4 -1
- package/src/messaging/providers/slack/render-transcript.test.ts +77 -29
- package/src/messaging/providers/slack/render-transcript.ts +58 -0
- package/src/notifications/conversation-pairing.ts +78 -19
- package/src/notifications/copy-composer.ts +0 -5
- package/src/notifications/emit-signal.ts +1 -1
- package/src/notifications/signal.ts +1 -2
- package/src/oauth/AGENTS.md +1 -1
- package/src/oauth/__tests__/identity-verifier.test.ts +2 -1
- package/src/oauth/connect-orchestrator.ts +8 -34
- package/src/oauth/connect-types.ts +6 -10
- package/src/oauth/manual-token-connection.ts +23 -0
- package/src/oauth/oauth-store.ts +30 -14
- package/src/oauth/provider-serializer.ts +6 -1
- package/src/oauth/seed-providers.ts +56 -108
- package/src/outbound-proxy/http-forwarder.ts +9 -0
- package/src/permissions/approval-policy.test.ts +293 -18
- package/src/permissions/approval-policy.ts +110 -58
- package/src/permissions/arg-parser.test.ts +161 -0
- package/src/permissions/arg-parser.ts +141 -0
- package/src/permissions/bash-risk-classifier.test.ts +414 -2
- package/src/permissions/bash-risk-classifier.ts +303 -60
- package/src/permissions/checker.ts +157 -29
- package/src/permissions/command-registry.test.ts +239 -0
- package/src/permissions/command-registry.ts +234 -54
- package/src/permissions/defaults.ts +5 -4
- package/src/permissions/gateway-threshold-reader.ts +196 -0
- package/src/permissions/prompter.ts +4 -0
- package/src/permissions/risk-types.ts +61 -4
- package/src/permissions/schedule-risk-classifier.test.ts +129 -0
- package/src/permissions/schedule-risk-classifier.ts +85 -0
- package/src/permissions/shell-identity.ts +2 -42
- package/src/permissions/types.ts +2 -0
- package/src/permissions/workspace-policy.ts +8 -3
- package/src/plugins/defaults/circuit-breaker.ts +146 -0
- package/src/plugins/defaults/compaction.ts +145 -0
- package/src/plugins/defaults/empty-response.ts +126 -0
- package/src/plugins/defaults/history-repair.ts +85 -0
- package/src/plugins/defaults/index.ts +116 -0
- package/src/plugins/defaults/injectors.ts +491 -0
- package/src/plugins/defaults/llm-call.ts +82 -0
- package/src/plugins/defaults/memory-retrieval.ts +226 -0
- package/src/plugins/defaults/overflow-reduce.ts +181 -0
- package/src/plugins/defaults/persistence.ts +129 -0
- package/src/plugins/defaults/title-generate.ts +95 -0
- package/src/plugins/defaults/token-estimate.ts +104 -0
- package/src/plugins/defaults/tool-error.ts +126 -0
- package/src/plugins/defaults/tool-execute.ts +89 -0
- package/src/plugins/defaults/tool-result-truncate.ts +88 -0
- package/src/plugins/pipeline.ts +316 -0
- package/src/plugins/plugin-skill-contributions.ts +292 -0
- package/src/plugins/registry.ts +241 -0
- package/src/plugins/types.ts +1134 -0
- package/src/plugins/user-loader.ts +177 -0
- package/src/prompts/templates/BOOTSTRAP.md +27 -77
- package/src/providers/model-catalog.ts +52 -29
- package/src/providers/model-intents.ts +1 -1
- package/src/providers/openrouter/client.ts +5 -1
- package/src/providers/speech-to-text/deepgram-realtime.test.ts +61 -0
- package/src/providers/speech-to-text/deepgram-realtime.ts +57 -0
- package/src/providers/speech-to-text/xai-realtime.test.ts +72 -4
- package/src/providers/speech-to-text/xai-realtime.ts +39 -14
- package/src/runtime/AGENTS.md +25 -16
- package/src/runtime/__tests__/browser-extension-pair-routes.test.ts +3 -3
- package/src/runtime/__tests__/client-registry.test.ts +293 -0
- package/src/runtime/client-registry.ts +261 -0
- package/src/runtime/http-server.ts +77 -8
- package/src/runtime/http-types.ts +0 -2
- package/src/runtime/migrations/vbundle-builder.ts +1 -22
- package/src/runtime/routes/approval-prompt-ts-tracker.ts +51 -31
- package/src/runtime/routes/approval-routes.ts +17 -0
- package/src/runtime/routes/browser-extension-pair-routes.ts +27 -8
- package/src/runtime/routes/conversation-routes.ts +223 -116
- package/src/runtime/routes/inbound-message-handler.ts +88 -13
- package/src/runtime/routes/memory-item-routes.test.ts +1 -0
- package/src/runtime/routes/migration-routes.ts +0 -3
- package/src/runtime/routes/playground/__tests__/force-compact.test.ts +284 -0
- package/src/runtime/routes/playground/__tests__/guard.test.ts +80 -0
- package/src/runtime/routes/playground/__tests__/inject-failures.test.ts +294 -0
- package/src/runtime/routes/playground/__tests__/reset-circuit.test.ts +271 -0
- package/src/runtime/routes/playground/__tests__/seed-conversation.test.ts +202 -0
- package/src/runtime/routes/playground/__tests__/seeded-conversations.test.ts +309 -0
- package/src/runtime/routes/playground/__tests__/state.test.ts +224 -0
- package/src/runtime/routes/playground/conversation-not-found.ts +29 -0
- package/src/runtime/routes/playground/deps.ts +56 -0
- package/src/runtime/routes/playground/force-compact.ts +73 -0
- package/src/runtime/routes/playground/guard.ts +37 -0
- package/src/runtime/routes/playground/index.ts +28 -0
- package/src/runtime/routes/playground/inject-failures.ts +159 -0
- package/src/runtime/routes/playground/reset-circuit.ts +115 -0
- package/src/runtime/routes/playground/seed-conversation.ts +139 -0
- package/src/runtime/routes/playground/seeded-conversations.ts +78 -0
- package/src/runtime/routes/playground/state.ts +78 -0
- package/src/runtime/routes/schedule-routes.ts +89 -8
- package/src/runtime/skill-route-registry.ts +75 -15
- package/src/schedule/run-script.ts +68 -0
- package/src/schedule/schedule-store.ts +7 -1
- package/src/schedule/scheduler.ts +48 -8
- package/src/skills/catalog-cache.ts +12 -5
- package/src/tools/browser/__tests__/browser-status.test.ts +189 -0
- package/src/tools/browser/browser-execution.ts +88 -19
- package/src/tools/browser/cdp-client/__tests__/extension-cdp-client.test.ts +230 -0
- package/src/tools/browser/cdp-client/__tests__/factory.test.ts +146 -3
- package/src/tools/browser/cdp-client/extension-cdp-client.ts +54 -3
- package/src/tools/browser/cdp-client/factory.ts +15 -4
- package/src/tools/executor.ts +126 -74
- package/src/tools/network/script-proxy/session-manager.ts +37 -1
- package/src/tools/permission-checker.ts +98 -49
- package/src/tools/policy-context.ts +4 -0
- package/src/tools/registry.ts +140 -3
- package/src/tools/schedule/create.ts +23 -8
- package/src/tools/schedule/update.ts +3 -1
- package/src/tools/secret-detection-handler.ts +0 -51
- package/src/tools/system/avatar-generator.ts +6 -2
- package/src/tools/types.ts +28 -2
- package/src/util/platform.ts +7 -2
- package/src/util/pricing.ts +26 -3
- package/src/workspace/migrations/006-services-config.ts +2 -4
- package/src/workspace/migrations/022-move-hooks-to-workspace.ts +2 -3
- package/src/workspace/migrations/041-backfill-google-gmail-settings-scope.ts +3 -4
- package/src/workspace/migrations/046-seed-conversation-starters-callsite.ts +108 -0
- package/src/workspace/migrations/047-remove-watch-callsites.ts +54 -0
- package/src/workspace/migrations/048-remove-workspace-hooks.ts +81 -0
- package/src/workspace/migrations/049-release-notes-default-sonnet.ts +80 -0
- package/src/workspace/migrations/050-seed-main-agent-opus-callsite.ts +86 -0
- package/src/workspace/migrations/051-seed-conversation-summarization-callsite.ts +128 -0
- package/src/workspace/migrations/registry.ts +12 -0
- package/tsconfig.json +1 -1
- package/hook-templates/debug-prompt-logger/hook.json +0 -7
- package/hook-templates/debug-prompt-logger/run.sh +0 -66
- package/src/__tests__/compaction-circuit-breaker.test.ts +0 -336
- package/src/__tests__/context-overflow-approval.test.ts +0 -156
- package/src/__tests__/hooks-blocking.test.ts +0 -178
- package/src/__tests__/hooks-cli.test.ts +0 -182
- package/src/__tests__/hooks-config.test.ts +0 -108
- package/src/__tests__/hooks-discovery.test.ts +0 -211
- package/src/__tests__/hooks-integration.test.ts +0 -196
- package/src/__tests__/hooks-manager.test.ts +0 -226
- package/src/__tests__/hooks-runner.test.ts +0 -175
- package/src/__tests__/hooks-settings.test.ts +0 -160
- package/src/__tests__/hooks-templates.test.ts +0 -169
- package/src/__tests__/hooks-ts-runner.test.ts +0 -170
- package/src/__tests__/hooks-watch.test.ts +0 -112
- package/src/__tests__/notification-schedule-dedup.test.ts +0 -213
- package/src/__tests__/oauth-scope-policy.test.ts +0 -180
- package/src/__tests__/send-notification-tool.test.ts +0 -83
- package/src/cli/commands/shotgun.ts +0 -266
- package/src/config/bundled-skills/conversations/SKILL.md +0 -20
- package/src/config/bundled-skills/conversations/TOOLS.json +0 -23
- package/src/config/bundled-skills/conversations/tools/rename-conversation.ts +0 -88
- package/src/config/bundled-skills/heartbeat/SKILL.md +0 -43
- package/src/config/bundled-skills/notifications/SKILL.md +0 -40
- package/src/config/bundled-skills/notifications/TOOLS.json +0 -80
- package/src/config/bundled-skills/notifications/tools/send-notification.ts +0 -152
- package/src/config/bundled-skills/notifications/tools/shared.ts +0 -13
- package/src/config/bundled-skills/screen-watch/SKILL.md +0 -27
- package/src/config/bundled-skills/screen-watch/TOOLS.json +0 -35
- package/src/config/bundled-skills/screen-watch/tools/start-screen-watch.ts +0 -12
- package/src/config/bundled-skills/skills-catalog/SKILL.md +0 -84
- package/src/daemon/context-overflow-approval.ts +0 -52
- package/src/daemon/watch-handler.ts +0 -399
- package/src/hooks/cli.ts +0 -253
- package/src/hooks/config.ts +0 -100
- package/src/hooks/discovery.ts +0 -135
- package/src/hooks/manager.ts +0 -179
- package/src/hooks/runner.ts +0 -117
- package/src/hooks/templates.ts +0 -77
- package/src/hooks/types.ts +0 -75
- package/src/oauth/scope-policy.ts +0 -89
- package/src/runtime/gateway-internal-client.ts +0 -94
- package/src/runtime/routes/watch-routes.ts +0 -156
- package/src/signals/shotgun.ts +0 -203
- package/src/tools/watch/screen-watch.ts +0 -144
- package/src/tools/watch/watch-state.ts +0 -142
|
@@ -177,9 +177,7 @@ graph TB
|
|
|
177
177
|
OVF_T2["Tier 2: Tool-result truncation<br/>4,000 chars per result"]
|
|
178
178
|
OVF_T3["Tier 3: Media/file stubbing"]
|
|
179
179
|
OVF_T4["Tier 4: Injection downgrade<br/>→ minimal mode"]
|
|
180
|
-
|
|
181
|
-
OVF_APPROVE["Approval gate<br/>PermissionPrompter"]
|
|
182
|
-
OVF_DENY["Graceful deny message<br/>(not conversation_error)"]
|
|
180
|
+
OVF_LATEST["Auto-compress latest turn<br/>(force=true, minKeepRecentUserTurns=0)"]
|
|
183
181
|
end
|
|
184
182
|
|
|
185
183
|
MSG_IN --> STORE
|
|
@@ -237,11 +235,8 @@ graph TB
|
|
|
237
235
|
OVF_T1 --> OVF_T2
|
|
238
236
|
OVF_T2 --> OVF_T3
|
|
239
237
|
OVF_T3 --> OVF_T4
|
|
240
|
-
OVF_T4 -->
|
|
241
|
-
|
|
242
|
-
OVF_POLICY -->|"non-interactive"| OVF_T1
|
|
243
|
-
OVF_APPROVE -->|"denied"| OVF_DENY
|
|
244
|
-
OVF_APPROVE -->|"approved"| OVF_T1
|
|
238
|
+
OVF_T4 --> OVF_LATEST
|
|
239
|
+
OVF_LATEST -.->|"uses"| SUMMARIZE
|
|
245
240
|
OVF_T1 -.->|"uses"| SUMMARIZE
|
|
246
241
|
```
|
|
247
242
|
|
|
@@ -251,7 +246,7 @@ Normal context compaction (the "Context Window Management" subgraph above) runs
|
|
|
251
246
|
|
|
252
247
|
When compaction alone is insufficient — either because the conversation grew too fast between turns or because a single turn contains extremely large payloads — the overflow recovery pipeline takes over. The pipeline's first tier (forced compaction) reuses the same `maybeCompact()` summarization machinery but with emergency parameters: `force: true` bypasses cooldown guards, `minKeepRecentUserTurns: 0` allows summarizing even the most recent history, and `targetInputTokensOverride` sets a tighter budget. Subsequent tiers (tool-result truncation, media stubbing, injection downgrade) apply progressively more aggressive payload reduction without involving the summarizer.
|
|
253
248
|
|
|
254
|
-
If all four reducer tiers are exhausted, the overflow policy resolver determines whether to compress the latest user turn.
|
|
249
|
+
If all four reducer tiers are exhausted, the overflow policy resolver determines whether to compress the latest user turn. All sessions — interactive and non-interactive alike — auto-compress the latest turn without prompting. The only explicit opt-out is setting `contextWindow.overflowRecovery.interactiveLatestTurnCompression` (or `nonInteractiveLatestTurnCompression`) to `"drop"`, which short-circuits to a graceful failure instead. Disabling overflow recovery entirely (`contextWindow.overflowRecovery.enabled: false`) also yields a graceful failure.
|
|
255
250
|
|
|
256
251
|
The key distinction: normal compaction is a cost-optimized background process that preserves conversational quality; overflow recovery is a convergence mechanism that prioritizes session survival over context richness. Both share the same summarization infrastructure but operate under different pressure thresholds and constraints.
|
|
257
252
|
|
|
@@ -303,16 +298,16 @@ stateDiagram-v2
|
|
|
303
298
|
|
|
304
299
|
**Item extraction** uses LLM-powered extraction (with pattern-based fallback) to identify memorable information from conversation messages. Each extracted item belongs to one of eight kinds:
|
|
305
300
|
|
|
306
|
-
| Kind | Description
|
|
307
|
-
| ------------ |
|
|
308
|
-
| `identity` | Personal info, facts, relationships
|
|
309
|
-
| `preference` | Likes, dislikes, preferred approaches/tools
|
|
301
|
+
| Kind | Description | Base Lifetime |
|
|
302
|
+
| ------------ | -------------------------------------------------------------------- | ------------- |
|
|
303
|
+
| `identity` | Personal info, facts, relationships | 6 months |
|
|
304
|
+
| `preference` | Likes, dislikes, preferred approaches/tools | 3 months |
|
|
310
305
|
| `journal` | Experiential reflections, journal-style notes, forward-looking items | 3 months |
|
|
311
|
-
| `constraint` | Rules, requirements, directives
|
|
312
|
-
| `project` | Project details, repos, tech stacks, action items
|
|
313
|
-
| `decision` | Choices made, approaches selected
|
|
314
|
-
| `event` | Deadlines, milestones, meetings, dates
|
|
315
|
-
| `capability` | Skill catalog entries (system-generated, not LLM-extracted)
|
|
306
|
+
| `constraint` | Rules, requirements, directives | 1 month |
|
|
307
|
+
| `project` | Project details, repos, tech stacks, action items | 2 weeks |
|
|
308
|
+
| `decision` | Choices made, approaches selected | 2 weeks |
|
|
309
|
+
| `event` | Deadlines, milestones, meetings, dates | 3 days |
|
|
310
|
+
| `capability` | Skill catalog entries (system-generated, not LLM-extracted) | never expires |
|
|
316
311
|
|
|
317
312
|
**Supersession chains** replace the old conflict resolution system. When the LLM extracts a new item that updates an existing one, it sets `supersedes` to the old item's ID and `overrideConfidence` to one of three levels:
|
|
318
313
|
|
|
@@ -552,13 +547,13 @@ The Anthropic provider places `cache_control: { type: 'ephemeral' }` on the **la
|
|
|
552
547
|
|
|
553
548
|
### Key files
|
|
554
549
|
|
|
555
|
-
| File | Role
|
|
556
|
-
| ------------------------------------------------------- |
|
|
557
|
-
| `assistant/src/workspace/top-level-scanner.ts` | Synchronous directory scanner with `MAX_TOP_LEVEL_ENTRIES` cap
|
|
558
|
-
| `assistant/src/workspace/top-level-renderer.ts` | Renders `TopLevelSnapshot` to `<workspace>` XML block
|
|
559
|
-
| `assistant/src/daemon/conversation-runtime-assembly.ts` | Runtime injections and legacy-block strip helpers (`<workspace>`, `<turn_context>`, `<channel_onboarding_playbook>`, `<onboarding_mode>`)
|
|
560
|
-
| `assistant/src/onboarding/onboarding-orchestrator.ts` | Builds assistant-owned onboarding runtime guidance from channel playbook + transport metadata
|
|
561
|
-
| `assistant/src/daemon/conversation-agent-loop.ts` | Agent loop orchestration, runtime injection wiring, legacy-block strip chain
|
|
550
|
+
| File | Role |
|
|
551
|
+
| ------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
|
|
552
|
+
| `assistant/src/workspace/top-level-scanner.ts` | Synchronous directory scanner with `MAX_TOP_LEVEL_ENTRIES` cap |
|
|
553
|
+
| `assistant/src/workspace/top-level-renderer.ts` | Renders `TopLevelSnapshot` to `<workspace>` XML block |
|
|
554
|
+
| `assistant/src/daemon/conversation-runtime-assembly.ts` | Runtime injections and legacy-block strip helpers (`<workspace>`, `<turn_context>`, `<channel_onboarding_playbook>`, `<onboarding_mode>`) |
|
|
555
|
+
| `assistant/src/onboarding/onboarding-orchestrator.ts` | Builds assistant-owned onboarding runtime guidance from channel playbook + transport metadata |
|
|
556
|
+
| `assistant/src/daemon/conversation-agent-loop.ts` | Agent loop orchestration, runtime injection wiring, legacy-block strip chain |
|
|
562
557
|
|
|
563
558
|
---
|
|
564
559
|
|
|
@@ -593,11 +588,11 @@ graph TB
|
|
|
593
588
|
|
|
594
589
|
### Key files
|
|
595
590
|
|
|
596
|
-
| File | Role
|
|
597
|
-
| ------------------------------------------------------- |
|
|
598
|
-
| `assistant/src/daemon/date-context.ts` | `formatTurnTimestamp()` — generates the timestamp string used in `<turn_context>`
|
|
599
|
-
| `assistant/src/daemon/conversation-runtime-assembly.ts` | `buildUnifiedTurnContextBlock()` — constructs the unified `<turn_context>` block; legacy block strip helpers
|
|
600
|
-
| `assistant/src/daemon/conversation-agent-loop.ts` | Wiring: builds turn context, passes to `applyRuntimeInjections`
|
|
591
|
+
| File | Role |
|
|
592
|
+
| ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
|
|
593
|
+
| `assistant/src/daemon/date-context.ts` | `formatTurnTimestamp()` — generates the timestamp string used in `<turn_context>` |
|
|
594
|
+
| `assistant/src/daemon/conversation-runtime-assembly.ts` | `buildUnifiedTurnContextBlock()` — constructs the unified `<turn_context>` block; legacy block strip helpers |
|
|
595
|
+
| `assistant/src/daemon/conversation-agent-loop.ts` | Wiring: builds turn context, passes to `applyRuntimeInjections` |
|
|
601
596
|
|
|
602
597
|
---
|
|
603
598
|
|
|
@@ -16,13 +16,14 @@ graph TB
|
|
|
16
16
|
|
|
17
17
|
FIND_RULE -->|"Deny rule"| DENY["decision: deny<br/>Blocked by rule"]
|
|
18
18
|
FIND_RULE -->|"Ask rule"| PROMPT_ASK["decision: prompt<br/>Always ask user"]
|
|
19
|
-
FIND_RULE -->|"Allow rule"|
|
|
20
|
-
|
|
19
|
+
FIND_RULE -->|"Allow rule / No match"| SANDBOX_CHECK{"sandboxAutoApprove?<br/>(bash + allowlisted +<br/>containerized)"}
|
|
20
|
+
|
|
21
|
+
SANDBOX_CHECK -->|"yes"| AUTO_SANDBOX["decision: allow<br/>Sandbox auto-approve"]
|
|
22
|
+
SANDBOX_CHECK -->|"no, has Allow rule"| RISK_CHECK{"Risk level?"}
|
|
23
|
+
SANDBOX_CHECK -->|"no, no match"| NO_MATCH{"Fallback logic"}
|
|
21
24
|
|
|
22
25
|
RISK_CHECK -->|"Low / Medium"| AUTO_ALLOW["decision: allow<br/>Auto-allowed by rule"]
|
|
23
|
-
RISK_CHECK -->|"High"|
|
|
24
|
-
HIGH_CHECK -->|"yes"| AUTO_ALLOW
|
|
25
|
-
HIGH_CHECK -->|"no"| RISK_THRESHOLD{"Risk-based<br/>threshold fallback"}
|
|
26
|
+
RISK_CHECK -->|"High"| RISK_THRESHOLD{"Risk-based<br/>threshold fallback"}
|
|
26
27
|
|
|
27
28
|
NO_MATCH -->|"tool.origin === 'skill'"| PROMPT_SKILL["decision: prompt<br/>Skill tools always ask"]
|
|
28
29
|
NO_MATCH -->|"strict mode"| PROMPT_STRICT["decision: prompt<br/>No implicit auto-allow"]
|
|
@@ -165,7 +166,7 @@ When a permission prompt is sent to the client (via `confirmation_request` SSE e
|
|
|
165
166
|
| `allowlistOptions` | Suggested patterns for "always allow" rules |
|
|
166
167
|
| `scopeOptions` | Suggested scopes for rule persistence |
|
|
167
168
|
|
|
168
|
-
The user can respond with: `allow` (one-time), `always_allow` (create allow rule), `deny` (one-time), or `always_deny` (create deny rule).
|
|
169
|
+
The user can respond with: `allow` (one-time), `always_allow` (create allow rule), `deny` (one-time), or `always_deny` (create deny rule). In containerized environments, commands tagged with `sandboxAutoApprove` in their risk spec are auto-allowed via the approval policy's sandbox auto-approve check; non-allowlisted commands (network tools, runtimes, package managers) use the user's `autoApproveUpTo` threshold. All other risk-based decisions use the `autoApproveUpTo` threshold (default: `"low"`) -- tools at or below the threshold are auto-allowed, those above are prompted.
|
|
169
170
|
|
|
170
171
|
### Canonical Paths
|
|
171
172
|
|
|
@@ -2,23 +2,39 @@
|
|
|
2
2
|
|
|
3
3
|
## Overview
|
|
4
4
|
|
|
5
|
-
Phase 2 of browser automation introduced a three-tier backend selection chain for macOS-originated turns, enabling the assistant to prefer the user's real Chrome session
|
|
5
|
+
Phase 2 of browser automation introduced a three-tier backend selection chain for macOS-originated turns, enabling the assistant to prefer the user's real Chrome session over a sandboxed Playwright instance. Two transport paths exist for the top-priority "extension" backend:
|
|
6
6
|
|
|
7
|
-
|
|
7
|
+
1. **Chrome Extension Registry (WebSocket)**: When the user has the Vellum Chrome Extension installed and paired, `host_browser_request` frames route through the `ChromeExtensionRegistry` singleton over a dedicated `/v1/browser-relay` WebSocket.
|
|
8
|
+
2. **macOS Host Browser Proxy (SSE)**: When the macOS desktop client is connected but the Chrome extension is absent, `host_browser_request` frames travel through `assistantEventHub` (SSE). The desktop client receives the frames via its SSE connection, executes CDP commands against the local Chrome, and POSTs results back to `/v1/host-browser-result`.
|
|
9
|
+
|
|
10
|
+
When neither transport is available, the system falls back through cdp-inspect (direct Chrome DevTools Protocol attach) before resorting to the local Playwright browser.
|
|
11
|
+
|
|
12
|
+
This document describes the runtime architecture, backend precedence rules, transport matrix, and the manual QA playbook for verifying correct backend selection.
|
|
8
13
|
|
|
9
14
|
## Component Inventory
|
|
10
15
|
|
|
11
|
-
| Component | Location | Role
|
|
12
|
-
| --------------------------- | -------------------------------------------------- |
|
|
13
|
-
| **ChromeExtensionRegistry** | `runtime/chrome-extension-registry.ts` | Tracks active extension WebSocket connections keyed by `(guardianId, clientInstanceId)`. Populated on WS `open`, drained on WS `close`.
|
|
14
|
-
| **HostBrowserProxy** | `daemon/host-browser-proxy.ts` | Per-conversation proxy that dispatches `host_browser_request` frames and awaits `host_browser_result` responses.
|
|
15
|
-
| **CDP Factory** | `tools/browser/cdp-client/factory.ts` | Builds the ordered candidate list and returns a `ScopedCdpClient` with per-invocation failover.
|
|
16
|
-
| **BrowserSessionManager** | `browser-session/manager.ts` | Routes CDP commands through the selected backend with session tracking.
|
|
17
|
-
| **CdpInspectClient** | `tools/browser/cdp-client/cdp-inspect-client.ts` | Connects to a host Chrome instance via its remote-debugging WebSocket endpoint.
|
|
18
|
-
| **LocalCdpClient** | `tools/browser/cdp-client/local-cdp-client.ts` | Drives Playwright's CDPSession against the sacrificial-profile browser.
|
|
19
|
-
| **ExtensionCdpClient** | `tools/browser/cdp-client/extension-cdp-client.ts` | Routes CDP commands through the HostBrowserProxy to the user's real Chrome.
|
|
20
|
-
| **conversation-routes.ts** | `runtime/routes/conversation-routes.ts` | Wires `resolveHostBrowserSender()` to set `hostBrowserSenderOverride` when the extension is connected.
|
|
21
|
-
| **Desktop-auto config** | `config/schemas/host-browser.ts` | `desktopAuto.enabled` (default `true`) and `desktopAuto.cooldownMs` (default 30s) control automatic cdp-inspect on macOS.
|
|
16
|
+
| Component | Location | Role |
|
|
17
|
+
| --------------------------- | -------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
18
|
+
| **ChromeExtensionRegistry** | `runtime/chrome-extension-registry.ts` | Tracks active extension WebSocket connections keyed by `(guardianId, clientInstanceId)`. Populated on WS `open`, drained on WS `close`. |
|
|
19
|
+
| **HostBrowserProxy** | `daemon/host-browser-proxy.ts` | Per-conversation proxy that dispatches `host_browser_request` frames and awaits `host_browser_result` responses. Wired to either the registry sender or the SSE hub sender. |
|
|
20
|
+
| **CDP Factory** | `tools/browser/cdp-client/factory.ts` | Builds the ordered candidate list and returns a `ScopedCdpClient` with per-invocation failover. |
|
|
21
|
+
| **BrowserSessionManager** | `browser-session/manager.ts` | Routes CDP commands through the selected backend with session tracking. |
|
|
22
|
+
| **CdpInspectClient** | `tools/browser/cdp-client/cdp-inspect-client.ts` | Connects to a host Chrome instance via its remote-debugging WebSocket endpoint. |
|
|
23
|
+
| **LocalCdpClient** | `tools/browser/cdp-client/local-cdp-client.ts` | Drives Playwright's CDPSession against the sacrificial-profile browser. |
|
|
24
|
+
| **ExtensionCdpClient** | `tools/browser/cdp-client/extension-cdp-client.ts` | Routes CDP commands through the HostBrowserProxy to the user's real Chrome. |
|
|
25
|
+
| **conversation-routes.ts** | `runtime/routes/conversation-routes.ts` | Wires `resolveHostBrowserSender()` to set `hostBrowserSenderOverride` when the extension is connected. For macOS without extension, provisions the proxy with the SSE hub sender. |
|
|
26
|
+
| **Desktop-auto config** | `config/schemas/host-browser.ts` | `desktopAuto.enabled` (default `true`) and `desktopAuto.cooldownMs` (default 30s) control automatic cdp-inspect on macOS. |
|
|
27
|
+
|
|
28
|
+
## Transport Matrix
|
|
29
|
+
|
|
30
|
+
The following table shows how `host_browser_request` frames are delivered to the client based on the originating interface and extension connectivity:
|
|
31
|
+
|
|
32
|
+
| Interface | Extension Connected | Transport | Sender | Notes |
|
|
33
|
+
| ------------------ | ------------------- | ------------------------------- | ----------------------------------------------- | ---------------------------------------------------------------------- |
|
|
34
|
+
| `chrome-extension` | Yes (always) | WebSocket (`/v1/browser-relay`) | `ChromeExtensionRegistry.send(guardianId, msg)` | The only transport for chrome-extension turns; SSE fallback is invalid |
|
|
35
|
+
| `macos` | Yes | WebSocket (`/v1/browser-relay`) | `ChromeExtensionRegistry.send(guardianId, msg)` | Extension takes priority; browser tools route through user's Chrome |
|
|
36
|
+
| `macos` | No | SSE (`assistantEventHub`) | `onEvent` hub publisher | macOS host browser proxy mode; desktop client handles CDP locally |
|
|
37
|
+
| Other | Any | N/A | No browser proxy provisioned | Falls through to cdp-inspect or local Playwright |
|
|
22
38
|
|
|
23
39
|
## Wire Diagram
|
|
24
40
|
|
|
@@ -37,10 +53,12 @@ conversation-routes.ts
|
|
|
37
53
|
| |
|
|
38
54
|
| +-- entry found? --> registrySender (WS to extension)
|
|
39
55
|
| | hostBrowserSenderOverride = registrySender
|
|
40
|
-
| | provision HostBrowserProxy
|
|
56
|
+
| | provision HostBrowserProxy(registrySender)
|
|
41
57
|
| |
|
|
42
58
|
| +-- entry not found? --> SSE hub sender (default)
|
|
43
59
|
| hostBrowserSenderOverride = undefined
|
|
60
|
+
| provision HostBrowserProxy(onEvent)
|
|
61
|
+
| [macOS natively supports host_browser]
|
|
44
62
|
|
|
|
45
63
|
v
|
|
46
64
|
Agent loop invokes browser tool
|
|
@@ -50,6 +68,8 @@ getCdpClient(toolContext)
|
|
|
50
68
|
|-- toolContext.hostBrowserProxy set?
|
|
51
69
|
| AND hostBrowserProxy.isAvailable()?
|
|
52
70
|
| --> candidate: extension (priority 1)
|
|
71
|
+
| [Transport: WS via registry when extension present,
|
|
72
|
+
| SSE via hub when macOS host proxy only]
|
|
53
73
|
|
|
|
54
74
|
|-- transportInterface === "macos"
|
|
55
75
|
| AND desktopAuto.enabled?
|
|
@@ -61,7 +81,7 @@ getCdpClient(toolContext)
|
|
|
61
81
|
v
|
|
62
82
|
ScopedCdpClient.send(method, params)
|
|
63
83
|
|
|
|
64
|
-
+-- Try candidate 1 (extension)
|
|
84
|
+
+-- Try candidate 1 (extension / macOS host proxy)
|
|
65
85
|
| transport_error? --> failover to candidate 2
|
|
66
86
|
| cdp_error? --> propagate immediately (no failover)
|
|
67
87
|
| success? --> sticky for remainder of invocation
|
|
@@ -76,11 +96,11 @@ ScopedCdpClient.send(method, params)
|
|
|
76
96
|
|
|
77
97
|
## Backend Precedence (macOS)
|
|
78
98
|
|
|
79
|
-
| Priority | Backend
|
|
80
|
-
| -------- |
|
|
81
|
-
| 1 | Extension
|
|
82
|
-
| 2 | cdp-inspect
|
|
83
|
-
| 3 | Local (Playwright)
|
|
99
|
+
| Priority | Backend | When selected | Transport | Failover trigger |
|
|
100
|
+
| -------- | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------- | ---------------------------------------------------------------------------------------- |
|
|
101
|
+
| 1 | Extension / macOS host proxy | `hostBrowserProxy` present and `isAvailable()` is `true`. On macOS, the proxy is always provisioned (SSE sender when no extension, registry-routed when extension is connected) | WS (registry) or SSE (hub) | Transport error (WebSocket disconnected, SSE send failed) |
|
|
102
|
+
| 2 | cdp-inspect | Config `enabled: true`, OR macOS + `desktopAuto.enabled` (default) + cooldown not active | Direct CDP WebSocket | Transport error (endpoint unreachable, WS connect failure). Records cooldown on failure. |
|
|
103
|
+
| 3 | Local (Playwright) | Always present as final fallback | In-process CDP | Errors propagate to the tool |
|
|
84
104
|
|
|
85
105
|
After the first successful CDP command on any backend, that backend becomes **sticky** for the remainder of the tool invocation.
|
|
86
106
|
|
|
@@ -114,6 +134,29 @@ When cdp-inspect fails with a transport error during a desktop-auto attempt:
|
|
|
114
134
|
- No `browserManager` launch log (Playwright not started).
|
|
115
135
|
- Extension WebSocket receives `host_browser_request` frames.
|
|
116
136
|
|
|
137
|
+
### Scenario 1b: macOS Host Browser Proxy (No Extension)
|
|
138
|
+
|
|
139
|
+
**Setup:**
|
|
140
|
+
|
|
141
|
+
1. macOS desktop client is running and connected to the assistant via SSE.
|
|
142
|
+
2. No browser extension is installed or connected.
|
|
143
|
+
3. Chrome is running on the desktop machine.
|
|
144
|
+
|
|
145
|
+
**Test:**
|
|
146
|
+
|
|
147
|
+
1. Send a message that triggers browser automation (e.g. "navigate to example.com and take a screenshot").
|
|
148
|
+
2. Observe the assistant drives the user's Chrome session via the macOS host browser proxy (the desktop client receives `host_browser_request` frames over SSE and executes CDP commands locally).
|
|
149
|
+
|
|
150
|
+
**Expected telemetry/log signals:**
|
|
151
|
+
|
|
152
|
+
- `cdp-factory` log: `CDP factory: built candidate list` with `candidates: [{kind: "extension", ...}, {kind: "cdp-inspect", ...}, {kind: "local", ...}]`
|
|
153
|
+
- `cdp-factory` log: `CDP factory: candidate succeeded, backend is now sticky` with `candidateKind: "extension"`
|
|
154
|
+
- No `browserManager` launch log (Playwright not started).
|
|
155
|
+
- SSE event stream delivers `host_browser_request` frames (not WebSocket relay).
|
|
156
|
+
- `browser_status` output shows `transport: "macos-sse"` in the extension mode details.
|
|
157
|
+
|
|
158
|
+
**Difference from Scenario 1:** The transport is SSE-based (`assistantEventHub`), not the `/v1/browser-relay` WebSocket. The `hostBrowserSenderOverride` is `undefined` because no registry entry exists. The proxy is provisioned because macOS natively supports `host_browser`.
|
|
159
|
+
|
|
117
160
|
### Scenario 2: Extension Absent + cdp-inspect Enabled
|
|
118
161
|
|
|
119
162
|
**Setup:**
|