@vellumai/assistant 0.11.3 → 0.11.4-dev.202608190019.b94dbf2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +9 -3
- package/ARCHITECTURE.md +32 -10
- package/docs/architecture/memory.md +43 -5
- package/docs/architecture/security.md +96 -152
- package/docs/architecture/turn-actor.md +70 -0
- package/docs/browser-use-architecture-phase2.md +128 -56
- package/docs/flux-turn-detection-spike.md +248 -0
- package/docs/guardian-request-flow.md +35 -0
- package/docs/stt-provider-onboarding.md +3 -1
- package/knip.json +3 -0
- package/node_modules/@vellumai/ces-client/node_modules/@vellumai/service-contracts/package.json +1 -0
- package/node_modules/@vellumai/ces-client/node_modules/@vellumai/service-contracts/src/__tests__/url-normalization.test.ts +135 -0
- package/node_modules/@vellumai/ces-client/node_modules/@vellumai/service-contracts/src/channels.ts +11 -0
- package/node_modules/@vellumai/ces-client/node_modules/@vellumai/service-contracts/src/index.ts +1 -0
- package/node_modules/@vellumai/ces-client/node_modules/@vellumai/service-contracts/src/remote-web-pairing.ts +60 -0
- package/node_modules/@vellumai/ces-client/node_modules/@vellumai/service-contracts/src/url-normalization.ts +107 -0
- package/node_modules/@vellumai/gateway-client/node_modules/@vellumai/service-contracts/package.json +1 -0
- package/node_modules/@vellumai/gateway-client/node_modules/@vellumai/service-contracts/src/__tests__/url-normalization.test.ts +135 -0
- package/node_modules/@vellumai/gateway-client/node_modules/@vellumai/service-contracts/src/channels.ts +11 -0
- package/node_modules/@vellumai/gateway-client/node_modules/@vellumai/service-contracts/src/index.ts +1 -0
- package/node_modules/@vellumai/gateway-client/node_modules/@vellumai/service-contracts/src/remote-web-pairing.ts +60 -0
- package/node_modules/@vellumai/gateway-client/node_modules/@vellumai/service-contracts/src/url-normalization.ts +107 -0
- package/node_modules/@vellumai/gateway-client/src/admission-policy-contract.ts +34 -0
- package/node_modules/@vellumai/gateway-client/src/gateway-ipc-contracts.ts +166 -1
- package/node_modules/@vellumai/gateway-client/src/index.ts +2 -0
- package/node_modules/@vellumai/service-contracts/package.json +1 -0
- package/node_modules/@vellumai/service-contracts/src/__tests__/url-normalization.test.ts +135 -0
- package/node_modules/@vellumai/service-contracts/src/channels.ts +11 -0
- package/node_modules/@vellumai/service-contracts/src/index.ts +1 -0
- package/node_modules/@vellumai/service-contracts/src/remote-web-pairing.ts +60 -0
- package/node_modules/@vellumai/service-contracts/src/url-normalization.ts +107 -0
- package/openapi.yaml +289 -113
- package/package.json +1 -1
- package/scripts/voice-ttft-spike.ts +3 -3
- package/scripts/write-plugin-api-shim.ts +10 -0
- package/src/__tests__/app-compiler.test.ts +38 -3
- package/src/__tests__/app-control-flow.test.ts +1 -0
- package/src/__tests__/approval-routes-http.test.ts +2 -2
- package/src/__tests__/assistant-feature-flag-guard.test.ts +25 -3
- package/src/__tests__/attachments-store.test.ts +21 -12
- package/src/__tests__/byok-default-profile-ensure.test.ts +17 -0
- package/src/__tests__/call-setup-flow-name-capture.test.ts +0 -1
- package/src/__tests__/call-site-routing-provider.test.ts +1 -1
- package/src/__tests__/channel-availability-routes.test.ts +14 -1
- package/src/__tests__/channel-capabilities-dedupe.test.ts +214 -0
- package/src/__tests__/channel-delivery-store.test.ts +14 -14
- package/src/__tests__/channel-inbound-disk-pressure.test.ts +4 -4
- package/src/__tests__/channel-setup-panel-ack.test.ts +1 -1
- package/src/__tests__/checker.test.ts +20 -314
- package/src/__tests__/cli-memory-v2-reembed-skills.test.ts +6 -2
- package/src/__tests__/compaction-events.test.ts +8 -10
- package/src/__tests__/config-loader-backfill.test.ts +3 -3
- package/src/__tests__/config-schema.test.ts +25 -10
- package/src/__tests__/context-search-memory-v2-source.test.ts +0 -1
- package/src/__tests__/conversation-agent-loop-inference-profile.test.ts +8 -11
- package/src/__tests__/conversation-agent-loop-overflow.test.ts +8 -11
- package/src/__tests__/conversation-agent-loop.test.ts +32 -21
- package/src/__tests__/conversation-attachments.test.ts +0 -1
- package/src/__tests__/conversation-attention-store.test.ts +63 -0
- package/src/__tests__/conversation-confirmation-signals.test.ts +112 -0
- package/src/__tests__/conversation-delete-schedule-cleanup.test.ts +0 -4
- package/src/__tests__/conversation-fork-crud.test.ts +69 -0
- package/src/__tests__/conversation-fork-referential.test.ts +67 -0
- package/src/__tests__/conversation-fork-retrospective.test.ts +24 -0
- package/src/__tests__/conversation-load-history-repair.test.ts +209 -0
- package/src/__tests__/conversation-notifiers-provenance.test.ts +59 -0
- package/src/__tests__/conversation-queue.test.ts +154 -4
- package/src/__tests__/conversation-routes-disk-view.test.ts +1 -1
- package/src/__tests__/conversation-routes-enabled-plugins.test.ts +1 -1
- package/src/__tests__/conversation-routes-guardian-reply.test.ts +9 -9
- package/src/__tests__/conversation-routes-hidden-queue.test.ts +1 -1
- package/src/__tests__/conversation-routes-slash-commands.test.ts +1 -1
- package/src/__tests__/conversation-runtime-assembly.test.ts +187 -102
- package/src/__tests__/conversation-runtime-workspace.test.ts +14 -10
- package/src/__tests__/conversation-slash-queue.test.ts +3 -0
- package/src/__tests__/conversation-surfaces-action-delivery.test.ts +1 -0
- package/src/__tests__/conversation-surfaces-activation-emit.test.ts +1 -0
- package/src/__tests__/conversation-surfaces-app-control.test.ts +1 -0
- package/src/__tests__/conversation-surfaces-app-open.test.ts +1 -1
- package/src/__tests__/conversation-surfaces-data-persist.test.ts +1 -1
- package/src/__tests__/conversation-surfaces-history-restored-completion.test.ts +21 -14
- package/src/__tests__/conversation-surfaces-queued-emit.test.ts +1 -0
- package/src/__tests__/conversation-surfaces-standalone-payloads.test.ts +1 -0
- package/src/__tests__/conversation-surfaces-standalone.test.ts +1 -0
- package/src/__tests__/conversation-surfaces-state-update.test.ts +1 -1
- package/src/__tests__/conversation-surfaces-table-action.test.ts +1 -1
- package/src/__tests__/conversation-surfaces-task-progress.test.ts +1 -1
- package/src/__tests__/conversation-tool-setup-app-refresh.test.ts +1 -1
- package/src/__tests__/conversation-tool-setup-attribution.test.ts +1 -1
- package/src/__tests__/credential-prompt-route.test.ts +7 -10
- package/src/__tests__/credential-security-invariants.test.ts +1 -0
- package/src/__tests__/cu-unified-flow.test.ts +1 -0
- package/src/__tests__/custom-profile-ensure.test.ts +5 -1
- package/src/__tests__/default-plugin-names-guard.test.ts +12 -1
- package/src/__tests__/discord-access-request-privacy.test.ts +5 -1
- package/src/__tests__/discord-requester-notice-privacy.test.ts +3 -3
- package/src/__tests__/document-append-idempotency.test.ts +233 -0
- package/src/__tests__/document-sync-tags.test.ts +0 -75
- package/src/__tests__/dynamic-skill-background-guard.test.ts +0 -2
- package/src/__tests__/edit-propagation.test.ts +107 -11
- package/src/__tests__/file-ops-service.test.ts +163 -30
- package/src/__tests__/filesystem-tools.test.ts +23 -24
- package/src/__tests__/gateway-only-guard.test.ts +2 -5
- package/src/__tests__/helpers/gateway-classify-mock.ts +27 -6
- package/src/__tests__/helpers/mock-actor-context.ts +49 -0
- package/src/__tests__/helpers/mock-conversation.ts +13 -1
- package/src/__tests__/host-file-read-tool.test.ts +16 -19
- package/src/__tests__/http-user-message-parity.test.ts +1 -1
- package/src/__tests__/init-feature-flag-overrides.test.ts +49 -0
- package/src/__tests__/injector-chain.test.ts +63 -41
- package/src/__tests__/injector-disk-pressure.test.ts +11 -23
- package/src/__tests__/inline-skill-load-permissions.test.ts +47 -36
- package/src/__tests__/llm-context-resolution.test.ts +73 -1
- package/src/__tests__/llm-schema.test.ts +5 -2
- package/src/__tests__/managed-skill-lifecycle.test.ts +7 -0
- package/src/__tests__/mcp-list-plugin-servers.test.ts +250 -0
- package/src/__tests__/media-generate-image.test.ts +131 -21
- package/src/__tests__/memory-retrieval-hook.test.ts +100 -7
- package/src/__tests__/messages-read-boundary-guard.test.ts +134 -0
- package/src/__tests__/mtime-cache.test.ts +3 -1
- package/src/__tests__/non-member-access-request.test.ts +0 -20
- package/src/__tests__/outbound-slack-persistence.test.ts +40 -1
- package/src/__tests__/platform-bash-auto-approve.test.ts +0 -4
- package/src/__tests__/plugin-api-store-credential.test.ts +261 -0
- package/src/__tests__/plugin-api-webhook-url.test.ts +10 -7
- package/src/__tests__/plugin-disabled-state.test.ts +2 -0
- package/src/__tests__/plugin-effective-enabled-set.test.ts +55 -3
- package/src/__tests__/plugin-execution-context.test.ts +73 -0
- package/src/__tests__/plugin-import-boundary-guard.test.ts +5 -1
- package/src/__tests__/plugin-import-boundary-reverse-guard.test.ts +7 -4
- package/src/__tests__/plugin-secret-pattern-contribution.test.ts +1 -1
- package/src/__tests__/post-compaction-reinjection-idempotency.test.ts +14 -7
- package/src/__tests__/provider-commit-message-generator.test.ts +20 -0
- package/src/__tests__/proxy-approval-callback.test.ts +1 -0
- package/src/__tests__/qdrant-manager.test.ts +14 -1
- package/src/__tests__/reaction-persistence.test.ts +200 -7
- package/src/__tests__/require-fresh-approval.test.ts +0 -4
- package/src/__tests__/risk-classification-boundary-guard.test.ts +115 -0
- package/src/__tests__/run-conversation-turn-persistence.test.ts +460 -105
- package/src/__tests__/run-due-schedules.test.ts +21 -0
- package/src/__tests__/scaffold-managed-skill-tool.test.ts +187 -18
- package/src/__tests__/schedule-routes.test.ts +23 -0
- package/src/__tests__/schedule-store.test.ts +17 -0
- package/src/__tests__/scoped-approval-grants.test.ts +11 -6
- package/src/__tests__/secret-ingress-channel.test.ts +0 -1
- package/src/__tests__/secret-ingress-http.test.ts +1 -1
- package/src/__tests__/send-endpoint-busy.test.ts +3 -3
- package/src/__tests__/skills.test.ts +32 -0
- package/src/__tests__/slack-edit-ordering-characterization.test.ts +0 -1
- package/src/__tests__/starter-task-flow.test.ts +5 -4
- package/src/__tests__/subagent-call-site-routing.test.ts +31 -19
- package/src/__tests__/subagent-fork-prompt-role.test.ts +1 -1
- package/src/__tests__/subagent-spawn-and-await.test.ts +18 -17
- package/src/__tests__/subagent-tool-gate-mode.test.ts +1 -1
- package/src/__tests__/subagent-tools.test.ts +81 -101
- package/src/__tests__/surface-completion-in-flight-snapshot.test.ts +1 -0
- package/src/__tests__/tool-execution-pipeline.benchmark.test.ts +1 -1
- package/src/__tests__/tool-executor-lifecycle-events.test.ts +132 -5
- package/src/__tests__/tool-executor.test.ts +68 -39
- package/src/__tests__/turn-events-store.test.ts +43 -0
- package/src/__tests__/ui-choice-copy-surfaces.test.ts +1 -1
- package/src/__tests__/ui-shape-teaching.test.ts +33 -0
- package/src/__tests__/ui-visual-surface.test.ts +1 -1
- package/src/__tests__/ui-voice-picker-surface.test.ts +128 -0
- package/src/__tests__/ui-work-result-surface.test.ts +1 -1
- package/src/__tests__/user-plugin-loader.test.ts +3 -1
- package/src/__tests__/verification-control-plane-policy.test.ts +0 -2
- package/src/__tests__/visible-app-context.test.ts +16 -9
- package/src/__tests__/voice-config-update.test.ts +40 -0
- package/src/__tests__/voice-scoped-grant-consumer.test.ts +5 -3
- package/src/__tests__/voice-session-bridge.test.ts +85 -29
- package/src/__tests__/worker-entrypoint-guards.test.ts +54 -0
- package/src/__tests__/worker-plugin-surface.test.ts +77 -0
- package/src/__tests__/workspace-migration-142-consolidate-voice-front-door.test.ts +158 -0
- package/src/__tests__/workspace-migration-143-repair-deprecated-codex-model-id.test.ts +133 -0
- package/src/__tests__/workspace-migration-144-convert-stranded-subscription-openai-profiles.test.ts +316 -0
- package/src/__tests__/workspace-migration-145-collapse-profile-bindings-to-entries.test.ts +325 -0
- package/src/__tests__/workspace-migration-146-repair-retired-fireworks-deepseek-flash-model-id.test.ts +235 -0
- package/src/acp/__tests__/acp-claude-oauth.test.ts +10 -2
- package/src/acp/__tests__/auth-required.test.ts +161 -0
- package/src/acp/acp-claude-oauth.ts +19 -2
- package/src/acp/agent-process.test.ts +100 -0
- package/src/acp/agent-process.ts +29 -26
- package/src/acp/auth-required.ts +102 -0
- package/src/acp/session-manager.test.ts +119 -0
- package/src/acp/session-manager.ts +76 -3
- package/src/api/events/acp-auth-required.ts +55 -0
- package/src/api/events/host-file.ts +2 -2
- package/src/api/index.ts +7 -0
- package/src/api/surfaces.ts +12 -3
- package/src/apps/app-store.ts +3 -0
- package/src/bundler/package-resolver.ts +2 -30
- package/src/calls/__tests__/voice-session-bridge.test.ts +194 -11
- package/src/calls/__tests__/voice-triage-escalate.test.ts +102 -0
- package/src/calls/call-controller.ts +19 -3
- package/src/calls/call-setup-flow.ts +0 -1
- package/src/calls/media-stream-stt-session.ts +15 -0
- package/src/calls/voice-session-bridge.ts +115 -41
- package/src/calls/voice-triage-escalate.ts +105 -2
- package/src/channels/__tests__/plugin-channel-declarations.test.ts +161 -0
- package/src/channels/config.ts +13 -0
- package/src/channels/plugin-channel-declarations.ts +108 -0
- package/src/channels/types.ts +30 -0
- package/src/cli/AGENTS.md +5 -2
- package/src/cli/bundled-modules.ts +29 -0
- package/src/cli/commands/credentials.help.ts +2 -2
- package/src/cli/commands/db/repair.ts +4 -8
- package/src/cli/commands/domain.ts +6 -3
- package/src/cli/commands/email.ts +6 -3
- package/src/cli/commands/inference-providers.ts +1 -1
- package/src/cli/commands/keys.ts +8 -3
- package/src/cli/commands/mcp.help.ts +13 -4
- package/src/cli/commands/mcp.ts +9 -0
- package/src/cli/commands/memory/__tests__/memory-v2.test.ts +57 -7
- package/src/cli/commands/memory/__tests__/memory-v3.test.ts +128 -5
- package/src/cli/commands/memory/index.help.ts +84 -28
- package/src/cli/commands/memory/index.ts +2 -0
- package/src/cli/commands/memory/memory-v2.ts +58 -54
- package/src/cli/commands/memory/memory-v3.ts +64 -0
- package/src/cli/commands/memory/memory-validate.ts +18 -0
- package/src/cli/commands/plugins.ts +85 -36
- package/src/cli/commands/schedules.ts +35 -1
- package/src/cli/commands/stt.help.ts +27 -2
- package/src/cli/commands/trust.ts +3 -14
- package/src/cli/lib/__tests__/upgrade-plugin.test.ts +39 -0
- package/src/cli/lib/bundled-marketplace.json +14 -1
- package/src/cli/lib/upgrade-plugin.ts +42 -0
- package/src/config/__tests__/balanced-model-experiment.test.ts +278 -0
- package/src/config/__tests__/default-profile-catalog.test.ts +34 -2
- package/src/config/__tests__/default-provider.test.ts +6 -1
- package/src/config/__tests__/profile-materialization.test.ts +75 -19
- package/src/config/assistant-feature-flags.ts +36 -15
- package/src/config/balanced-model-experiment.ts +35 -0
- package/src/config/bundled-skills/acp/SKILL.md +6 -7
- package/src/config/bundled-skills/document-editor/SKILL.md +2 -2
- package/src/config/bundled-skills/document-editor/TOOLS.json +2 -2
- package/src/config/bundled-skills/image-studio/SKILL.md +5 -4
- package/src/config/bundled-skills/image-studio/TOOLS.json +1 -1
- package/src/config/bundled-skills/image-studio/tools/media-generate-image.ts +101 -0
- package/src/config/bundled-skills/media-processing/services/preprocess.ts +14 -4
- package/src/config/bundled-skills/settings/TOOLS.json +3 -3
- package/src/config/bundled-skills/settings/tools/navigate-settings-tab.test.ts +65 -0
- package/src/config/bundled-skills/settings/tools/navigate-settings-tab.ts +7 -1
- package/src/config/bundled-skills/settings/tools/shared.ts +16 -0
- package/src/config/bundled-skills/settings/tools/voice-config-update.ts +19 -1
- package/src/config/bundled-skills/skill-management/TOOLS.json +9 -3
- package/src/config/bundled-skills/subagent/SKILL.md +17 -12
- package/src/config/bundled-skills/subagent/TOOLS.json +4 -4
- package/src/config/bundled-skills/transcribe/tools/transcribe-media.test.ts +22 -1
- package/src/config/bundled-skills/transcribe/tools/transcribe-media.ts +9 -2
- package/src/config/call-site-defaults.ts +11 -5
- package/src/config/default-profile-catalog.ts +179 -16
- package/src/config/default-profile-names.ts +4 -1
- package/src/config/default-provider-resolution.ts +4 -0
- package/src/config/feature-flag-registry.json +12 -12
- package/src/config/llm-context-resolution.ts +11 -3
- package/src/config/llm-resolver.ts +28 -1
- package/src/config/profile-materialization.ts +70 -22
- package/src/config/schemas/__tests__/live-voice.test.ts +107 -4
- package/src/config/schemas/call-site-catalog.ts +4 -4
- package/src/config/schemas/live-voice.ts +57 -23
- package/src/config/schemas/llm.ts +59 -32
- package/src/config/schemas/mcp.ts +23 -0
- package/src/config/schemas/plugin-updates.ts +6 -2
- package/src/config/schemas/stt.ts +1 -0
- package/src/config/skills.ts +9 -2
- package/src/context/outbound-sanitize.ts +96 -1
- package/src/daemon/__tests__/conversation-surfaces-launch.test.ts +1 -1
- package/src/daemon/__tests__/plugin-mcp-reconcile.test.ts +82 -0
- package/src/daemon/conversation-agent-loop-handlers.ts +15 -10
- package/src/daemon/conversation-agent-loop.ts +31 -19
- package/src/daemon/conversation-messaging.ts +5 -1
- package/src/daemon/conversation-notifiers.ts +20 -10
- package/src/daemon/conversation-process.ts +20 -6
- package/src/daemon/conversation-runtime-assembly.ts +12 -6
- package/src/daemon/conversation-store.ts +14 -4
- package/src/daemon/conversation-surfaces.ts +75 -26
- package/src/daemon/conversation-tool-setup.ts +3 -6
- package/src/daemon/conversation.ts +154 -51
- package/src/daemon/doordash-steps.ts +2 -2
- package/src/daemon/lifecycle.ts +14 -1
- package/src/daemon/mcp-reload-service.ts +36 -6
- package/src/daemon/process-message.ts +13 -26
- package/src/daemon/providers-setup.ts +6 -3
- package/src/daemon/trust-context-types.ts +29 -0
- package/src/daemon/trust-context.ts +12 -7
- package/src/daemon/wake-conversation-ops.ts +3 -2
- package/src/daemon/windows-compiled-entry.ts +4 -0
- package/src/documents/document-store.ts +143 -240
- package/src/hooks/hook-loader.ts +3 -3
- package/src/hooks/registry.ts +103 -45
- package/src/hooks/types.ts +5 -0
- package/src/inbound/__tests__/oauth-callback-url.test.ts +83 -0
- package/src/inbound/oauth-callback-url.ts +61 -0
- package/src/ipc/gateway-client.test.ts +1 -1
- package/src/ipc/gateway-client.ts +14 -21
- package/src/ipc/gateway-flag-listener.ts +17 -3
- package/src/live-voice/__tests__/live-voice-agent-turn.test.ts +1 -104
- package/src/live-voice/__tests__/live-voice-events.test.ts +7 -8
- package/src/live-voice/__tests__/live-voice-flux-turn-end.test.ts +1050 -0
- package/src/live-voice/__tests__/live-voice-metrics.test.ts +115 -8
- package/src/live-voice/__tests__/live-voice-photo.test.ts +100 -0
- package/src/live-voice/__tests__/live-voice-progress.test.ts +60 -194
- package/src/live-voice/__tests__/live-voice-stt.test.ts +14 -0
- package/src/live-voice/__tests__/live-voice-triage-escalate.test.ts +29 -0
- package/src/live-voice/__tests__/live-voice-tts-session.test.ts +0 -483
- package/src/live-voice/__tests__/live-voice-vad.test.ts +0 -16
- package/src/live-voice/__tests__/progress-narration.test.ts +214 -0
- package/src/live-voice/live-voice-archive.ts +2 -0
- package/src/live-voice/live-voice-manager.ts +16 -3
- package/src/live-voice/live-voice-metrics.ts +57 -32
- package/src/live-voice/live-voice-photo.ts +1 -2
- package/src/live-voice/live-voice-session.ts +598 -323
- package/src/live-voice/progress-narration.ts +277 -0
- package/src/live-voice/protocol.ts +21 -1
- package/src/live-voice/windows-compiled-live-voice.ts +4 -0
- package/src/mcp/__tests__/effective-config.test.ts +238 -0
- package/src/mcp/__tests__/mcp-auth-orchestrator.test.ts +0 -1
- package/src/mcp/__tests__/mcp-oauth-client-registration.test.ts +200 -0
- package/src/mcp/__tests__/mcp-oauth-provider.test.ts +9 -9
- package/src/mcp/__tests__/plugin-server-credential-isolation.test.ts +95 -0
- package/src/mcp/client.ts +16 -11
- package/src/mcp/effective-config.ts +113 -0
- package/src/mcp/manager.ts +11 -6
- package/src/mcp/mcp-auth-orchestrator.ts +13 -22
- package/src/mcp/mcp-oauth-provider.ts +205 -240
- package/src/monitoring/__tests__/plugin-auto-update.test.ts +166 -3
- package/src/monitoring/control.ts +1 -0
- package/src/monitoring/db-integrity-sample.ts +4 -5
- package/src/monitoring/plugin-auto-update.ts +128 -24
- package/src/notifications/AGENTS.md +2 -0
- package/src/notifications/approval-card-data.ts +33 -0
- package/src/notifications/signal.ts +1 -0
- package/src/permissions/AGENTS.md +16 -0
- package/src/permissions/checker.test.ts +75 -122
- package/src/permissions/checker.ts +44 -560
- package/src/permissions/confirmation-guardian-request.test.ts +15 -11
- package/src/permissions/confirmation-guardian-request.ts +2 -2
- package/src/permissions/prompter.ts +1 -5
- package/src/permissions/question-guardian-request.test.ts +14 -6
- package/src/permissions/question-guardian-request.ts +1 -2
- package/src/persistence/attachments-store.ts +8 -1
- package/src/persistence/bookmark-crud.ts +3 -7
- package/src/persistence/conversation-attention-store.ts +16 -45
- package/src/persistence/conversation-crud.ts +33 -4
- package/src/persistence/conversation-lineage.ts +9 -0
- package/src/persistence/conversation-queries.ts +182 -52
- package/src/persistence/delivery-crud.ts +150 -36
- package/src/persistence/embeddings/qdrant-manager.ts +84 -49
- package/src/persistence/external-conversation-store.ts +32 -4
- package/src/persistence/llm-request-log-store.ts +4 -10
- package/src/persistence/llm-usage-store.ts +8 -3
- package/src/persistence/message-reads.test.ts +197 -0
- package/src/persistence/message-reads.ts +211 -0
- package/src/persistence/migrations/360-add-document-workspace-path.ts +5 -14
- package/src/persistence/migrations/366-chatgpt-subscription-row-identity.test.ts +120 -0
- package/src/persistence/migrations/366-chatgpt-subscription-row-identity.ts +62 -0
- package/src/persistence/real-user-turn-filter.ts +27 -3
- package/src/persistence/schema/documents.ts +4 -4
- package/src/persistence/steps.ts +9 -0
- package/src/plugin-api/__tests__/oauth-callback-url-export.test.ts +29 -0
- package/src/plugin-api/constants.ts +8 -0
- package/src/plugin-api/conversation-turn.ts +195 -5
- package/src/plugin-api/index.ts +38 -6
- package/src/plugin-api/resolve-credential.ts +3 -2
- package/src/plugin-api/store-credential.ts +146 -0
- package/src/plugin-api/vision-support.test.ts +1 -1
- package/src/plugin-api/webhook-url.ts +13 -11
- package/src/plugins/__tests__/mcp-servers.test.ts +371 -0
- package/src/plugins/defaults/main.ts +21 -7
- package/src/plugins/defaults/memory/AGENTS.md +15 -3
- package/src/plugins/defaults/memory/__tests__/buffer-format.test.ts +204 -0
- package/src/plugins/defaults/memory/__tests__/memory-retrospective-accounting.test.ts +72 -0
- package/src/plugins/defaults/memory/__tests__/memory-retrospective-job.test.ts +4 -1
- package/src/plugins/defaults/memory/__tests__/memory-retrospective-provider-path.test.ts +4 -1
- package/src/plugins/defaults/memory/__tests__/memory-tier-boundary-guard.test.ts +0 -1
- package/src/plugins/defaults/memory/buffer-format.ts +165 -0
- package/src/plugins/defaults/memory/context-search/sources/conversations.ts +6 -0
- package/src/plugins/defaults/memory/graph/__tests__/conversation-graph-memory-v2-routing.test.ts +77 -0
- package/src/plugins/defaults/memory/graph/conversation-graph-memory.ts +24 -6
- package/src/plugins/defaults/memory/graph/image-ref-utils.ts +3 -0
- package/src/plugins/defaults/memory/graph/tool-handlers.ts +1 -30
- package/src/plugins/defaults/memory/graph-topology/pending-buffer.test.ts +34 -0
- package/src/plugins/defaults/memory/graph-topology/pending-buffer.ts +17 -24
- package/src/plugins/defaults/memory/hooks/post-compact.ts +1 -4
- package/src/plugins/defaults/memory/hooks/user-prompt-submit.ts +53 -5
- package/src/plugins/defaults/memory/indexer.ts +3 -1
- package/src/plugins/defaults/memory/memory-retrospective-accounting.ts +19 -7
- package/src/plugins/defaults/memory/memory-retrospective-job.ts +4 -4
- package/src/plugins/defaults/memory/src/__tests__/memory-v3-gate-stats.test.ts +281 -0
- package/src/plugins/defaults/memory/src/memory-v2-routes.ts +12 -10
- package/src/plugins/defaults/memory/src/memory-v3-routes.ts +207 -0
- package/src/plugins/defaults/memory/substrate/__tests__/consolidation-job.test.ts +147 -0
- package/src/plugins/defaults/memory/substrate/__tests__/consolidation-prompt-flag-gating-guard.test.ts +8 -0
- package/src/plugins/defaults/memory/substrate/__tests__/edge-index.test.ts +0 -30
- package/src/plugins/defaults/memory/substrate/__tests__/ingest.test.ts +73 -1
- package/src/plugins/defaults/memory/substrate/__tests__/page-index.test.ts +37 -0
- package/src/plugins/defaults/memory/substrate/__tests__/page-links.test.ts +208 -0
- package/src/plugins/defaults/memory/substrate/__tests__/prompts-consolidation.test.ts +158 -1
- package/src/plugins/defaults/memory/substrate/__tests__/static-context.test.ts +195 -58
- package/src/plugins/defaults/memory/substrate/consolidation-job.ts +93 -24
- package/src/plugins/defaults/memory/substrate/edge-index.ts +0 -31
- package/src/plugins/defaults/memory/substrate/ingest.ts +59 -0
- package/src/plugins/defaults/memory/substrate/page-index.ts +35 -8
- package/src/plugins/defaults/memory/substrate/page-links.ts +133 -0
- package/src/plugins/defaults/memory/substrate/page-store.ts +10 -0
- package/src/plugins/defaults/memory/substrate/prompts/consolidation.ts +124 -34
- package/src/plugins/defaults/memory/substrate/skill-content.ts +8 -1
- package/src/plugins/defaults/memory/substrate/static-context.ts +160 -33
- package/src/plugins/defaults/memory/substrate/sweep-job.ts +2 -4
- package/src/plugins/defaults/memory/v1/graph/extraction.ts +3 -1
- package/src/plugins/defaults/memory/v3/__tests__/injection.test.ts +18 -0
- package/src/plugins/defaults/memory/v3/__tests__/shadow-plugin.test.ts +66 -1
- package/src/plugins/defaults/memory/v3/card.ts +9 -9
- package/src/plugins/defaults/memory/v3/core-set.test.ts +7 -0
- package/src/plugins/defaults/memory/v3/core-set.ts +5 -1
- package/src/plugins/defaults/memory/v3/edge.ts +13 -57
- package/src/plugins/defaults/memory/v3/injector.ts +8 -0
- package/src/plugins/defaults/memory/v3/prune.ts +2 -0
- package/src/plugins/defaults/memory/v3/selection-log-store.ts +2 -0
- package/src/plugins/defaults/memory/v3/shadow-plugin.ts +23 -9
- package/src/plugins/defaults/memory/worker-control.ts +1 -0
- package/src/plugins/defaults/memory/worker.ts +6 -3
- package/src/plugins/defaults/worker-entrypoints.ts +3 -0
- package/src/plugins/external-plugin-loader.ts +47 -0
- package/src/plugins/mcp-servers.ts +361 -0
- package/src/plugins/mtime-cache.ts +37 -46
- package/src/plugins/pipeline.ts +14 -7
- package/src/plugins/plugin-execution-context.ts +35 -11
- package/src/plugins/worker-plugin-surface.ts +33 -0
- package/src/prompts/templates/system-sections.ts +0 -7
- package/src/providers/__tests__/connection-model-compat.test.ts +1 -1
- package/src/providers/__tests__/context-overflow-error.test.ts +24 -0
- package/src/providers/__tests__/dispatch-connection-routing.test.ts +214 -2
- package/src/providers/__tests__/preflight-resolved-config.test.ts +57 -0
- package/src/providers/__tests__/retry-callsite.test.ts +25 -2
- package/src/providers/anthropic/__tests__/pause-turn-continuation.test.ts +170 -0
- package/src/providers/anthropic/client.ts +332 -241
- package/src/providers/call-site-routing.ts +30 -3
- package/src/providers/connection-resolution.ts +194 -11
- package/src/providers/inference/__tests__/adapter-factory-openai-compatible.test.ts +4 -0
- package/src/providers/inference/adapter-factory.ts +3 -0
- package/src/providers/inference/auth.ts +6 -0
- package/src/providers/inference/connection-availability.ts +24 -2
- package/src/providers/inference/connections.ts +2 -0
- package/src/providers/model-catalog.ts +3 -3
- package/src/providers/model-intents.ts +28 -8
- package/src/providers/openai/__tests__/chat-completions-provider-reasoning.test.ts +163 -40
- package/src/providers/openai/__tests__/tool-choice-mapping.test.ts +125 -2
- package/src/providers/openai/chat-completions-provider.ts +133 -36
- package/src/providers/openai/codex-models.ts +2 -1
- package/src/providers/provider-send-message.ts +32 -3
- package/src/providers/speech-to-text/__tests__/deepgram-flux-frames.test.ts +433 -0
- package/src/providers/speech-to-text/__tests__/deepgram-flux-realtime.test.ts +625 -0
- package/src/providers/speech-to-text/__tests__/provider-catalog.test.ts +34 -0
- package/src/providers/speech-to-text/__tests__/resolve.test.ts +285 -6
- package/src/providers/speech-to-text/deepgram-flux-frames.ts +395 -0
- package/src/providers/speech-to-text/deepgram-flux-realtime.ts +679 -0
- package/src/providers/speech-to-text/provider-catalog.ts +99 -8
- package/src/providers/speech-to-text/resolve.ts +25 -2
- package/src/routes/control.ts +1 -0
- package/src/routes/route-host-client.ts +1 -0
- package/src/routes/route-host-protocol.ts +7 -0
- package/src/routes/worker.ts +48 -15
- package/src/runtime/AGENTS.md +16 -17
- package/src/runtime/access-request-helper.ts +9 -12
- package/src/runtime/agent-wake.ts +18 -15
- package/src/runtime/effective-capabilities.test.ts +43 -0
- package/src/runtime/effective-capabilities.ts +27 -0
- package/src/runtime/guardian-reply-router.ts +6 -2
- package/src/runtime/pre-first-message-gate.ts +4 -0
- package/src/runtime/routes/__tests__/acp-claude-auth-routes.test.ts +12 -4
- package/src/runtime/routes/__tests__/conversation-list-routes.test.ts +522 -2
- package/src/runtime/routes/__tests__/conversation-query-routes.test.ts +52 -0
- package/src/runtime/routes/__tests__/default-provider-routes.test.ts +61 -0
- package/src/runtime/routes/__tests__/inference-profiles-routes.test.ts +44 -0
- package/src/runtime/routes/__tests__/inference-provider-connection-routes.test.ts +102 -1
- package/src/runtime/routes/__tests__/plugins-routes.test.ts +132 -35
- package/src/runtime/routes/__tests__/schedule-routes-disarm-reason.test.ts +215 -0
- package/src/runtime/routes/__tests__/stt-routes.test.ts +25 -0
- package/src/runtime/routes/__tests__/surface-action-routes.test.ts +55 -6
- package/src/runtime/routes/__tests__/user-route-dispatcher-host.test.ts +52 -1
- package/src/runtime/routes/__tests__/user-route-dispatcher.test.ts +133 -1
- package/src/runtime/routes/channel-availability-routes.ts +32 -14
- package/src/runtime/routes/channel-route-shared.ts +0 -6
- package/src/runtime/routes/chatgpt-subscription-auth-routes.ts +6 -6
- package/src/runtime/routes/conversation-list-routes.ts +181 -23
- package/src/runtime/routes/conversation-management-routes.ts +17 -12
- package/src/runtime/routes/conversation-query-routes.ts +40 -27
- package/src/runtime/routes/conversation-routes.ts +43 -29
- package/src/runtime/routes/credential-prompt-routes.ts +4 -7
- package/src/runtime/routes/credential-routes.ts +26 -82
- package/src/runtime/routes/default-provider-routes.ts +15 -0
- package/src/runtime/routes/documents-routes.ts +3 -222
- package/src/runtime/routes/inbound-message-handler.ts +21 -45
- package/src/runtime/routes/inbound-stages/acl-enforcement.test.ts +0 -1
- package/src/runtime/routes/inbound-stages/acl-enforcement.ts +0 -9
- package/src/runtime/routes/inbound-stages/admission-policy.ts +1 -17
- package/src/runtime/routes/inbound-stages/bootstrap-intercept.test.ts +0 -1
- package/src/runtime/routes/inbound-stages/bootstrap-intercept.ts +2 -3
- package/src/runtime/routes/inbound-stages/edit-intercept.ts +101 -79
- package/src/runtime/routes/inbound-stages/guardian-reply-intercept.test.ts +0 -1
- package/src/runtime/routes/inbound-stages/guardian-reply-intercept.ts +12 -7
- package/src/runtime/routes/inbound-stages/reaction-intercept.test.ts +91 -8
- package/src/runtime/routes/inbound-stages/reaction-intercept.ts +72 -79
- package/src/runtime/routes/inbound-stages/secret-ingress-check.ts +2 -3
- package/src/runtime/routes/inference-profiles-routes.ts +20 -11
- package/src/runtime/routes/inference-provider-connection-routes.ts +77 -15
- package/src/runtime/routes/log-export-routes.ts +3 -0
- package/src/runtime/routes/mcp-auth-routes.ts +148 -57
- package/src/runtime/routes/playground/__tests__/inject-failures.test.ts +2 -0
- package/src/runtime/routes/playground/__tests__/reset-circuit.test.ts +3 -0
- package/src/runtime/routes/playground/inject-failures.ts +2 -2
- package/src/runtime/routes/playground/reset-circuit.ts +1 -1
- package/src/runtime/routes/plugins-routes.ts +38 -8
- package/src/runtime/routes/schedule-routes.ts +94 -7
- package/src/runtime/routes/settings-routes.ts +6 -6
- package/src/runtime/routes/stt-routes.ts +31 -25
- package/src/runtime/routes/surface-action-routes.ts +20 -19
- package/src/runtime/routes/surface-conversation-resolver.ts +3 -0
- package/src/runtime/routes/user-route-dispatcher.ts +69 -15
- package/src/runtime/routes/user-route-import.ts +108 -0
- package/src/runtime/routes/user-route-resolution.ts +18 -1
- package/src/runtime/routes/workspace-routes.ts +0 -9
- package/src/runtime/routes/workspace-utils.ts +3 -13
- package/src/runtime/services/conversation-serializer.ts +7 -2
- package/src/schedule/__tests__/plugin-schedule-declarations.test.ts +68 -5
- package/src/schedule/__tests__/plugin-schedule-reconciler.test.ts +81 -0
- package/src/schedule/plugin-schedule-availability.ts +58 -0
- package/src/schedule/plugin-schedule-declarations.ts +23 -27
- package/src/schedule/plugin-schedule-reconciler.ts +12 -3
- package/src/schedule/schedule-store.ts +5 -1
- package/src/schedule/scheduler.ts +9 -4
- package/src/schedule/worker-control.ts +1 -0
- package/src/schedule/worker.ts +6 -0
- package/src/security/oauth2.ts +6 -22
- package/src/stt/__tests__/daemon-batch-transcriber.test.ts +22 -0
- package/src/stt/__tests__/types.test.ts +94 -0
- package/src/stt/daemon-batch-transcriber.ts +10 -0
- package/src/stt/stt-stream-session.ts +8 -4
- package/src/stt/types.ts +103 -0
- package/src/subagent/__tests__/consult-prompt.test.ts +26 -15
- package/src/subagent/consult-context.ts +11 -11
- package/src/subagent/consult-prompt.ts +26 -35
- package/src/subagent/manager.ts +21 -40
- package/src/subagent/notify.ts +7 -1
- package/src/subagent/types.ts +15 -12
- package/src/tools/__tests__/tool-input-schemas.test.ts +7 -7
- package/src/tools/acp/spawn.test.ts +97 -0
- package/src/tools/acp/spawn.ts +38 -4
- package/src/tools/credentials/ref-parse.ts +35 -0
- package/src/tools/credentials/resolve.ts +4 -10
- package/src/tools/credentials/store.ts +168 -0
- package/src/tools/document/document-tool.ts +12 -3
- package/src/tools/executor.ts +53 -15
- package/src/tools/filesystem/read.ts +27 -10
- package/src/tools/host-filesystem/read.ts +27 -15
- package/src/tools/network/url-safety.ts +9 -16
- package/src/tools/permission-checker.ts +43 -52
- package/src/tools/registry.ts +2 -1
- package/src/tools/shared/filesystem/file-ops-service.ts +63 -35
- package/src/tools/shared/filesystem/legacy-read-args.ts +22 -0
- package/src/tools/shared/filesystem/types.ts +5 -5
- package/src/tools/skills/scaffold-managed.ts +25 -7
- package/src/tools/subagent/spawn.ts +28 -88
- package/src/tools/tool-types.ts +26 -13
- package/src/tools/types.ts +6 -6
- package/src/tools/ui-surface/surface-shape-docs.ts +12 -1
- package/src/tools/workflows/run-workflow.ts +1 -2
- package/src/tts/__tests__/reasoning-tag-filter.test.ts +78 -0
- package/src/tts/reasoning-tag-filter.ts +89 -0
- package/src/util/__tests__/worker-process-command.test.ts +37 -0
- package/src/util/logger.ts +16 -0
- package/src/util/think-tag-stream.ts +95 -0
- package/src/util/worker-process.ts +37 -4
- package/src/windows-compiled-cli.ts +32 -0
- package/src/windows-compiled-entry.ts +4 -0
- package/src/windows-compiled-logger.ts +6 -0
- package/src/windows-compiled-worker-entry.ts +29 -0
- package/src/workspace/byok-default-profile-ensure.ts +76 -24
- package/src/workspace/custom-profile-ensure.ts +4 -24
- package/src/workspace/migrations/142-consolidate-voice-front-door.ts +70 -0
- package/src/workspace/migrations/143-repair-deprecated-codex-model-id.ts +134 -0
- package/src/workspace/migrations/144-convert-stranded-subscription-openai-profiles.ts +265 -0
- package/src/workspace/migrations/145-collapse-profile-bindings-to-entries.ts +328 -0
- package/src/workspace/migrations/146-repair-retired-fireworks-deepseek-flash-model-id.ts +195 -0
- package/src/workspace/migrations/__tests__/141-stt-english-default-to-multilingual.test.ts +0 -10
- package/src/workspace/migrations/registry.ts +10 -0
- package/src/workspace/provider-commit-message-generator.ts +7 -5
- package/src/__tests__/document-workspace-file.test.ts +0 -467
- package/src/live-voice/__tests__/front-decision.test.ts +0 -645
- package/src/live-voice/front-decision.ts +0 -476
- package/src/permissions/ipc-risk-types.ts +0 -143
- package/src/permissions/risk-types.ts +0 -76
- package/src/subagent/__tests__/consult-transcript.test.ts +0 -184
- package/src/subagent/consult-transcript.ts +0 -90
|
@@ -0,0 +1,70 @@
|
|
|
1
|
+
# Turn Actor
|
|
2
|
+
|
|
3
|
+
Which actor a piece of code means when it reads trust from a conversation.
|
|
4
|
+
|
|
5
|
+
## Two different questions
|
|
6
|
+
|
|
7
|
+
A `Conversation` is long-lived and several actors can send into it. Code that reads trust is asking one of two things, and they have different answers:
|
|
8
|
+
|
|
9
|
+
- **The acting actor.** Who this turn is executing for. Governs authorization, and is recorded as the provenance of anything the turn persists. Undefined between turns.
|
|
10
|
+
- **The resting actor.** Who the conversation belongs to when no turn is running. Used to hydrate a new turn, to scope history, and by routes answering questions about a conversation rather than about a turn.
|
|
11
|
+
|
|
12
|
+
They coincide most of the time, which is why one field answering both went unnoticed. They diverge whenever another actor sends while a turn is in flight, or when a turn runs on a conversation an earlier actor last touched.
|
|
13
|
+
|
|
14
|
+
## Read through the accessors
|
|
15
|
+
|
|
16
|
+
```ts
|
|
17
|
+
conversation.getTurnTrust(); // who this turn is for; undefined if unrecorded
|
|
18
|
+
conversation.getTrustContext(); // who the conversation belongs to
|
|
19
|
+
conversation.getTurnOrRestingTrust(); // the turn's actor, else the owner
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
The names follow the existing convention on `Conversation`: `getTurn*` for
|
|
23
|
+
per-turn values (`getTurnActorPrincipalId`, `getTurnChannelContext`), plain
|
|
24
|
+
`get*` for conversation-level ones (`getAuthContext`).
|
|
25
|
+
|
|
26
|
+
Do not read `trustContext` or `currentTurnTrustContext` directly. Call sites handed a conversation-shaped context rather than the class (handler deps, the messaging context) use the structural counterparts `turnOrRestingTrust(ctx)` / `restingTrust(ctx)` from `trust-context-types.ts`, which read the same fields and carry the same names. The accessors exist so that every call site states which question it is asking; a raw field read states nothing, and the wrong answer is silent.
|
|
27
|
+
|
|
28
|
+
**Use `getTurnTrust()`** for authorization decisions and for routing a reply back to the requester: cases where substituting the conversation's owner would be wrong rather than approximate, so `undefined` must surface and be handled.
|
|
29
|
+
|
|
30
|
+
**Use `getTurnOrRestingTrust()`** for provenance stamped onto persisted rows (see `provenanceFromTrustContext`). Provenance is read back by the memory indexer, which runs extraction only for guardian rows, and by the transcript assembly, which wraps non-guardian user content before it reaches the model. The turn's actor is correct when a turn is running; rows persisted with no turn in flight (wake notices) must keep the owner's class, because stamping `"unknown"` there silently stops memory extraction for the owner's own flows.
|
|
31
|
+
|
|
32
|
+
**Use `getTrustContext()`** when there is no turn: HTTP routes reporting conversation state, hydration, and persistence of conversation-level options. Also when a caller deliberately wants the conversation's owner rather than whoever is currently acting; that intent should be obvious from the call site, and if it is not, it is probably the wrong accessor.
|
|
33
|
+
|
|
34
|
+
## When the acting actor is unknown
|
|
35
|
+
|
|
36
|
+
`getTurnTrust()` returns `undefined`. It does not substitute the owner, because
|
|
37
|
+
a caller asking who is acting should not silently receive someone else.
|
|
38
|
+
|
|
39
|
+
Callers that can accept the owner as a stand-in ask for that by name:
|
|
40
|
+
|
|
41
|
+
```ts
|
|
42
|
+
conversation.getTurnOrRestingTrust();
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
That fallback is load-bearing, not transitional politeness: a deferred wake
|
|
46
|
+
fires with no inbound actor, and refusing it an answer denies every sensitive
|
|
47
|
+
tool in the resumed turn (LUM-2929). The substitution lives in one named
|
|
48
|
+
method, so it is greppable, and removable in one place once every entry point
|
|
49
|
+
records a turn actor.
|
|
50
|
+
|
|
51
|
+
Callers for which the owner would be wrong rather than approximate, such as
|
|
52
|
+
authorization, call `getTurnTrust()` alone and handle `undefined`. Provenance
|
|
53
|
+
is deliberately not in that set: its readers treat an absent class more
|
|
54
|
+
permissively than `"unknown"`, so failing closed there fails open downstream.
|
|
55
|
+
|
|
56
|
+
## Writers that stamp for a run
|
|
57
|
+
|
|
58
|
+
`agent-wake` and `voice-session-bridge` set the conversation's trust before a run
|
|
59
|
+
and restore the prior value afterwards, each guarding the restore so a turn that
|
|
60
|
+
started in between is not clobbered. They are supplying the acting actor for
|
|
61
|
+
their run, and are covered by this contract.
|
|
62
|
+
|
|
63
|
+
`call-controller` keeps its own `trustContext` on its own object and never reads
|
|
64
|
+
the conversation's. It is outside this contract.
|
|
65
|
+
|
|
66
|
+
## Why this exists
|
|
67
|
+
|
|
68
|
+
Trust was read as `currentTurnTrustContext ?? trustContext` at each consumer, so every call site independently chose which actor it got, invisibly. Between June and August 2026 that produced seven separate fixes, alternating between a turn running as the wrong actor and a turn running as nobody, depending on whether the conversation was still resident in cache. A per-caller `preferTurnSnapshot` flag was added so one consumer could opt into the turn value, which is the same choice made explicit for one call site.
|
|
69
|
+
|
|
70
|
+
Naming the two questions is what makes the wrong answer visible at the point it is chosen.
|
|
@@ -1,40 +1,42 @@
|
|
|
1
|
-
# Browser Use Architecture
|
|
1
|
+
# Browser Use Architecture
|
|
2
2
|
|
|
3
3
|
## Overview
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
macOS-originated turns choose a browser backend from a three-tier chain, so the assistant prefers the user's real Chrome session over a sandboxed Playwright instance. The top tier, the host browser proxy, has two transports, both riding the SSE event hub:
|
|
6
6
|
|
|
7
|
-
1. **Chrome Extension
|
|
8
|
-
2. **macOS
|
|
7
|
+
1. **Chrome Extension**: when the Vellum Chrome Extension is installed and paired, `HostBrowserProxy` publishes `host_browser_request` frames to `assistantEventHub` targeted at the extension's SSE subscription; the extension executes CDP commands through `chrome.debugger` and POSTs results to `/v1/host-browser-result`.
|
|
8
|
+
2. **macOS desktop bridge**: when the macOS desktop client is connected but no extension is, the same publish targets the desktop client's SSE subscription; it executes CDP commands against the local Chrome and POSTs results the same way.
|
|
9
9
|
|
|
10
|
-
When neither
|
|
10
|
+
When neither is available, the chain falls through to cdp-inspect (direct Chrome DevTools Protocol attach) before resorting to the local Playwright browser.
|
|
11
11
|
|
|
12
12
|
This document describes the runtime architecture, backend precedence rules, transport matrix, and the manual QA playbook for verifying correct backend selection.
|
|
13
13
|
|
|
14
14
|
## Component Inventory
|
|
15
15
|
|
|
16
|
-
| Component
|
|
17
|
-
|
|
|
18
|
-
| **
|
|
19
|
-
| **
|
|
20
|
-
| **
|
|
21
|
-
| **
|
|
22
|
-
| **
|
|
23
|
-
| **
|
|
24
|
-
| **
|
|
25
|
-
| **
|
|
26
|
-
| **Desktop-auto config**
|
|
16
|
+
| Component | Location | Role |
|
|
17
|
+
| ------------------------- | -------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
18
|
+
| **HostBrowserProxy** | `daemon/host-browser-proxy.ts` | Lazily-created singleton. Publishes `host_browser_request` frames to the hub with `targetCapability: "host_browser"` and an explicit target client chosen at send time (`resolveTargetClient`), then awaits the matching `host_browser_result`. `Vellum.*` pseudo-methods go only to a chrome-extension client; raw CDP prefers an extension client over the macOS bridge; `targetClientId` pins one; `sourceActorPrincipalId` restricts to that actor's clients. |
|
|
19
|
+
| **events-routes.ts** | `runtime/routes/events-routes.ts` | `GET /v1/events`: registers each SSE client on the hub with the capabilities its `X-Vellum-Interface-Id` supports (`host_browser` for `chrome-extension`; every host-proxy capability for `macos`). |
|
|
20
|
+
| **pair.ts** | `gateway/src/http/routes/pair.ts` | `POST /v1/pair`: loopback-only, rate-limited pairing that mints the extension's `actor_client_v1` JWT for self-hosted deployments. Cloud deployments issue the guardian-bound JWT through the gateway's WorkOS-backed flow. |
|
|
21
|
+
| **CDP Factory** | `tools/browser/cdp-client/factory.ts` | Builds the ordered candidate list and returns a `ScopedCdpClient` with per-invocation failover. The macOS bridge is its internal `"host-bridge"` candidate kind. |
|
|
22
|
+
| **BrowserSessionManager** | `browser-session/manager.ts` | Routes CDP commands through the selected backend with session tracking. |
|
|
23
|
+
| **CdpInspectClient** | `tools/browser/cdp-client/cdp-inspect-client.ts` | Connects to a host Chrome instance via its remote-debugging WebSocket endpoint. |
|
|
24
|
+
| **LocalCdpClient** | `tools/browser/cdp-client/local-cdp-client.ts` | Drives Playwright's CDPSession against the sacrificial-profile browser. |
|
|
25
|
+
| **ExtensionCdpClient** | `tools/browser/cdp-client/extension-cdp-client.ts` | Routes CDP commands through the HostBrowserProxy to the user's real Chrome. |
|
|
26
|
+
| **Desktop-auto config** | `config/schemas/host-browser.ts` | `desktopAuto.enabled` (default `true`) and `desktopAuto.cooldownMs` (default 30s) control automatic cdp-inspect on macOS. |
|
|
27
|
+
|
|
28
|
+
Nothing in the turn layer wires the proxy: it reads the hub's live roster on every send, and a conversation's own event sink is fixed for its life (see `assistant/AGENTS.md`, "Conversation event delivery and turn presence"), so a queued or drained turn reaches the extension exactly like a live one.
|
|
27
29
|
|
|
28
30
|
## Transport Matrix
|
|
29
31
|
|
|
30
|
-
|
|
32
|
+
How `host_browser_request` frames reach a client, by originating interface and extension connectivity:
|
|
31
33
|
|
|
32
|
-
| Interface | Extension Connected | Transport
|
|
33
|
-
| ------------------ | ------------------- |
|
|
34
|
-
| `chrome-extension` | Yes (always) |
|
|
35
|
-
| `macos` | Yes |
|
|
36
|
-
| `macos` | No | SSE (`assistantEventHub`)
|
|
37
|
-
| Other | Any |
|
|
34
|
+
| Interface | Extension Connected | Transport | Target | Notes |
|
|
35
|
+
| ------------------ | ------------------- | ------------------------- | --------------------------------------- | --------------------------------------------------------------------------- |
|
|
36
|
+
| `chrome-extension` | Yes (always) | SSE (`assistantEventHub`) | The chrome-extension client | The only transport for chrome-extension turns |
|
|
37
|
+
| `macos` | Yes | SSE (`assistantEventHub`) | The chrome-extension client (preferred) | Browser tools route through the user's real Chrome session |
|
|
38
|
+
| `macos` | No | SSE (`assistantEventHub`) | The macOS desktop client (bridge) | Desktop client executes CDP locally against the user's Chrome |
|
|
39
|
+
| Other | Any | SSE (`assistantEventHub`) | Any connected `host_browser` client | Falls through to cdp-inspect or local Playwright when no client is eligible |
|
|
38
40
|
|
|
39
41
|
## Wire Diagram
|
|
40
42
|
|
|
@@ -45,22 +47,6 @@ macOS app (user message)
|
|
|
45
47
|
POST /v1/messages { interface: "macos", ... }
|
|
46
48
|
|
|
|
47
49
|
v
|
|
48
|
-
conversation-routes.ts
|
|
49
|
-
|-- setTurnInterfaceContext({ userMessageInterface: "macos", ... })
|
|
50
|
-
|-- resolveHostBrowserSender()
|
|
51
|
-
| |
|
|
52
|
-
| +-- ChromeExtensionRegistry.get(guardianId)
|
|
53
|
-
| |
|
|
54
|
-
| +-- entry found? --> registrySender (WS to extension)
|
|
55
|
-
| | hostBrowserSenderOverride = registrySender
|
|
56
|
-
| | provision HostBrowserProxy(registrySender)
|
|
57
|
-
| |
|
|
58
|
-
| +-- entry not found? --> SSE hub sender (default)
|
|
59
|
-
| hostBrowserSenderOverride = undefined
|
|
60
|
-
| provision HostBrowserProxy(onEvent)
|
|
61
|
-
| [macOS natively supports host_browser]
|
|
62
|
-
|
|
|
63
|
-
v
|
|
64
50
|
Agent loop invokes browser tool
|
|
65
51
|
|
|
|
66
52
|
v
|
|
@@ -68,8 +54,9 @@ getCdpClient(toolContext)
|
|
|
68
54
|
|-- toolContext.hostBrowserProxy set?
|
|
69
55
|
| AND hostBrowserProxy.isAvailable()?
|
|
70
56
|
| --> candidate: extension (priority 1)
|
|
71
|
-
| [
|
|
72
|
-
|
|
|
57
|
+
| [HostBrowserProxy publishes to the hub with
|
|
58
|
+
| targetCapability "host_browser"; the target client is
|
|
59
|
+
| the extension when one is connected, else the macOS bridge]
|
|
73
60
|
|
|
|
74
61
|
|-- transportInterface === "macos"
|
|
75
62
|
| AND desktopAuto.enabled?
|
|
@@ -81,7 +68,7 @@ getCdpClient(toolContext)
|
|
|
81
68
|
v
|
|
82
69
|
ScopedCdpClient.send(method, params)
|
|
83
70
|
|
|
|
84
|
-
+-- Try candidate 1 (extension / macOS
|
|
71
|
+
+-- Try candidate 1 (extension / macOS bridge)
|
|
85
72
|
| transport_error? --> failover to candidate 2
|
|
86
73
|
| cdp_error? --> propagate immediately (no failover)
|
|
87
74
|
| success? --> sticky for remainder of invocation
|
|
@@ -96,14 +83,18 @@ ScopedCdpClient.send(method, params)
|
|
|
96
83
|
|
|
97
84
|
## Backend Precedence (macOS)
|
|
98
85
|
|
|
99
|
-
| Priority | Backend | When selected
|
|
100
|
-
| -------- | ---------------------------- |
|
|
101
|
-
| 1 | Extension / macOS host proxy | `
|
|
102
|
-
| 2 | cdp-inspect | Config `enabled: true`, OR macOS + `desktopAuto.enabled` (default) + cooldown not active
|
|
103
|
-
| 3 | Local (Playwright) | Always present as final fallback
|
|
86
|
+
| Priority | Backend | When selected | Transport | Failover trigger |
|
|
87
|
+
| -------- | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
|
|
88
|
+
| 1 | Extension / macOS host proxy | `extension` candidate when `hasExtensionClient(actor)` finds a chrome-extension client; otherwise `host-bridge` candidate when `isAvailable(actor)` finds a `host_browser` client and the actor's host-bridge cooldown is not active | SSE (hub) | Transport error (client disconnected, no eligible client, publish failed). A `host-bridge` failure records the per-actor cooldown. |
|
|
89
|
+
| 2 | cdp-inspect | Config `enabled: true`, OR macOS + `desktopAuto.enabled` (default) + cooldown not active | Direct CDP WebSocket | Transport error (endpoint unreachable, WS connect failure). Records cooldown on failure. |
|
|
90
|
+
| 3 | Local (Playwright) | Always present as final fallback | In-process CDP | Errors propagate to the tool |
|
|
104
91
|
|
|
105
92
|
After the first successful CDP command on any backend, that backend becomes **sticky** for the remainder of the tool invocation.
|
|
106
93
|
|
|
94
|
+
## Host-bridge Cooldown
|
|
95
|
+
|
|
96
|
+
When the `host-bridge` candidate fails with a transport error, the factory records a per-actor cooldown (`recordHostBridgeCooldown`, keyed by `sourceActorPrincipalId`, `__default__` when unresolved) for the same `desktopAuto.cooldownMs` window. While it is active, `buildCandidateList` skips the bridge (log `CDP factory: host-bridge skipped (cooldown active)`) and the turn goes straight to cdp-inspect/local. Per-actor rather than process-global because on a multi-actor cloud daemon the bridge reaches a different desktop per actor. The `extension` candidate is never cooled down.
|
|
97
|
+
|
|
107
98
|
## Desktop-auto cdp-inspect Cooldown
|
|
108
99
|
|
|
109
100
|
When cdp-inspect fails with a transport error during a desktop-auto attempt:
|
|
@@ -119,8 +110,8 @@ When cdp-inspect fails with a transport error during a desktop-auto attempt:
|
|
|
119
110
|
|
|
120
111
|
**Setup:**
|
|
121
112
|
|
|
122
|
-
1. Pair the browser extension (
|
|
123
|
-
2.
|
|
113
|
+
1. Pair the browser extension (self-hosted: the extension pairs itself through `POST /v1/pair`; cloud: sign in through the platform).
|
|
114
|
+
2. Verify the extension's SSE subscription is registered (runtime log `subscriber registered (client)` with `interfaceId: "chrome-extension"` and `capabilities: ["host_browser"]`).
|
|
124
115
|
|
|
125
116
|
**Test:**
|
|
126
117
|
|
|
@@ -132,7 +123,7 @@ When cdp-inspect fails with a transport error during a desktop-auto attempt:
|
|
|
132
123
|
- `cdp-factory` log: `CDP factory: built candidate list` with `candidates: [{kind: "extension", ...}, {kind: "cdp-inspect", ...}, {kind: "local", ...}]`
|
|
133
124
|
- `cdp-factory` log: `CDP factory: candidate succeeded, backend is now sticky` with `candidateKind: "extension"`
|
|
134
125
|
- No `browserManager` launch log (Playwright not started).
|
|
135
|
-
-
|
|
126
|
+
- The extension's SSE stream receives `host_browser_request` frames; results arrive as `POST /v1/host-browser-result`.
|
|
136
127
|
|
|
137
128
|
### Scenario 1b: macOS Host Browser Proxy (No Extension)
|
|
138
129
|
|
|
@@ -149,13 +140,13 @@ When cdp-inspect fails with a transport error during a desktop-auto attempt:
|
|
|
149
140
|
|
|
150
141
|
**Expected telemetry/log signals:**
|
|
151
142
|
|
|
152
|
-
- `cdp-factory` log: `CDP factory: built candidate list` with `candidates: [{kind: "
|
|
153
|
-
- `cdp-factory` log: `CDP factory: candidate succeeded, backend is now sticky` with `candidateKind: "
|
|
143
|
+
- `cdp-factory` log: `CDP factory: built candidate list` with `candidates: [{kind: "host-bridge", ...}, {kind: "cdp-inspect", ...}, {kind: "local", ...}]`
|
|
144
|
+
- `cdp-factory` log: `CDP factory: candidate succeeded, backend is now sticky` with `candidateKind: "host-bridge"`
|
|
154
145
|
- No `browserManager` launch log (Playwright not started).
|
|
155
|
-
- SSE
|
|
156
|
-
-
|
|
146
|
+
- The macOS client's SSE stream delivers the `host_browser_request` frames.
|
|
147
|
+
- No failover: the bridge is the first candidate and succeeds.
|
|
157
148
|
|
|
158
|
-
**Difference from Scenario 1:**
|
|
149
|
+
**Difference from Scenario 1:** Same transport, different target: with no chrome-extension client on the hub roster, `resolveTargetClient` selects the macOS desktop client, which registers every host-proxy capability including `host_browser`.
|
|
159
150
|
|
|
160
151
|
### Scenario 2: Extension Absent + cdp-inspect Enabled
|
|
161
152
|
|
|
@@ -205,7 +196,7 @@ In all scenarios, the definitive signal is the `cdp-factory` structured log:
|
|
|
205
196
|
|
|
206
197
|
```
|
|
207
198
|
CDP factory: candidate succeeded, backend is now sticky
|
|
208
|
-
candidateKind: "extension" | "cdp-inspect" | "local"
|
|
199
|
+
candidateKind: "extension" | "host-bridge" | "cdp-inspect" | "local"
|
|
209
200
|
conversationId: "<id>"
|
|
210
201
|
method: "<first CDP method called>"
|
|
211
202
|
```
|
|
@@ -215,3 +206,84 @@ Filter runtime logs with:
|
|
|
215
206
|
```bash
|
|
216
207
|
grep -h "cdp-factory" "$VELLUM_WORKSPACE_DIR"/data/logs/assistant-*.log
|
|
217
208
|
```
|
|
209
|
+
|
|
210
|
+
## Steady-state contract
|
|
211
|
+
|
|
212
|
+
After the first successful Connect, the extension operates as a
|
|
213
|
+
background service with no further user interaction required:
|
|
214
|
+
|
|
215
|
+
1. **Install once**: Load the extension.
|
|
216
|
+
2. **Connect once**: Click Connect in the popup. The worker
|
|
217
|
+
auto-bootstraps credentials (local pair token or cloud JWT) as part
|
|
218
|
+
of the single-click flow.
|
|
219
|
+
3. **Forget it**: The extension keeps its SSE subscription up indefinitely.
|
|
220
|
+
A `chrome.alarms` alarm (`vellum-relay-keepalive`, every 30 s) wakes the
|
|
221
|
+
MV3 service worker and reconnects the stream if it is closed; transient
|
|
222
|
+
drops reconnect with exponential backoff (`sse-connection.ts`). The
|
|
223
|
+
`autoConnect` flag persists across browser sessions so reopening Chrome
|
|
224
|
+
automatically reconnects. An authentication failure is not retried
|
|
225
|
+
silently: it surfaces as the `auth_required` health state below.
|
|
226
|
+
|
|
227
|
+
Users should only interact with the extension again when:
|
|
228
|
+
|
|
229
|
+
- They want to **Pause** (intentionally disconnect and disable
|
|
230
|
+
auto-reconnect).
|
|
231
|
+
- The popup shows **Action required** (`auth_required` or `error` health
|
|
232
|
+
state), meaning automatic recovery has been exhausted.
|
|
233
|
+
|
|
234
|
+
A transient extension disconnect does change backend selection. In auto
|
|
235
|
+
mode `buildCandidateList` reads the hub roster at the start of each
|
|
236
|
+
browser operation: with no chrome-extension client connected it skips the
|
|
237
|
+
`extension` candidate and the chain proceeds to `host-bridge` (macOS),
|
|
238
|
+
cdp-inspect (opt-in, or desktop-auto on macOS), then local; and an
|
|
239
|
+
extension that drops mid-command surfaces a `transport_error`, which
|
|
240
|
+
advances the chained client to the next candidate. Only a dispatch pinned
|
|
241
|
+
with `browser_mode: "extension"` waits through the proxy's reconnect grace
|
|
242
|
+
(`EXTENSION_RECONNECT_GRACE_MS`, 3 s) before failing. `cdp-inspect` is an
|
|
243
|
+
advanced backend for users who cannot install the extension or who need
|
|
244
|
+
broad session-level CDP access; see
|
|
245
|
+
[the `cdp-inspect` backend doc](../../docs/browser-use-cdp-inspect-backend.md).
|
|
246
|
+
|
|
247
|
+
## Known UX considerations
|
|
248
|
+
|
|
249
|
+
### `chrome.debugger` infobar
|
|
250
|
+
|
|
251
|
+
When the Chrome extension calls
|
|
252
|
+
`chrome.debugger.attach(target, requiredVersion)`, Chrome displays a
|
|
253
|
+
persistent yellow infobar at the top of the affected tab saying "Vellum
|
|
254
|
+
started debugging this browser." This is an intentional security
|
|
255
|
+
mitigation; it cannot be suppressed via the public MV3 API.
|
|
256
|
+
|
|
257
|
+
Chrome API notes:
|
|
258
|
+
|
|
259
|
+
- `chrome.debugger.attach(target, requiredVersion, callback)`: three-
|
|
260
|
+
argument form, no options parameter. Chrome 120+.
|
|
261
|
+
(https://developer.chrome.com/docs/extensions/reference/api/debugger)
|
|
262
|
+
- There is no `{ silent: true }` option on attach.
|
|
263
|
+
- The `--silent-debugger-extension-api` command-line flag exists for
|
|
264
|
+
Chromium but (a) requires the user to launch Chrome with the flag,
|
|
265
|
+
(b) is not enabled by default in stable channels, and (c) is not
|
|
266
|
+
something we can enforce on end users.
|
|
267
|
+
- Chrome 126+ added `chrome.debugger.attach` acceptance via `targetId`
|
|
268
|
+
/ `tabId` but did not add a silent-mode option.
|
|
269
|
+
- Closing the infobar does not detach the debugger; it is purely
|
|
270
|
+
informational.
|
|
271
|
+
|
|
272
|
+
Decision: accept the infobar; no public API exists to suppress it. End-user messaging in the
|
|
273
|
+
Mac app popup should explain that the banner is expected and normal
|
|
274
|
+
when Vellum is driving the browser.
|
|
275
|
+
|
|
276
|
+
Alternatives considered:
|
|
277
|
+
|
|
278
|
+
- Playwright / `chrome --remote-debugging-port` in a sacrificial profile
|
|
279
|
+
avoids the infobar but requires installing Chromium and is out-of-
|
|
280
|
+
scope.
|
|
281
|
+
- The assistant-local `cdp-inspect` backend attaches to an existing
|
|
282
|
+
Chrome instance via `chrome://inspect` / `--remote-debugging-port`
|
|
283
|
+
and avoids the per-tab debugger infobar entirely. It is implemented
|
|
284
|
+
and opt-in via `hostBrowser.cdpInspect.enabled`; see
|
|
285
|
+
[the `cdp-inspect` backend doc](../../docs/browser-use-cdp-inspect-backend.md)
|
|
286
|
+
for setup, security trade-offs, and troubleshooting. Note that in auto
|
|
287
|
+
mode it is also the next candidate after `host-bridge` when the
|
|
288
|
+
extension is disconnected at the start of an operation (see the
|
|
289
|
+
steady-state contract above).
|