@vellumai/assistant 0.6.5 → 0.6.6

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (443) hide show
  1. package/AGENTS.md +9 -1
  2. package/ARCHITECTURE.md +15 -17
  3. package/Dockerfile +6 -4
  4. package/__tests__/permissions/gateway-threshold-reader.test.ts +283 -0
  5. package/docs/architecture/integrations.md +32 -39
  6. package/docs/architecture/memory.md +25 -30
  7. package/docs/architecture/security.md +7 -6
  8. package/docs/browser-use-architecture-phase2.md +63 -20
  9. package/docs/plugins.md +761 -0
  10. package/examples/plugins/echo/README.md +132 -0
  11. package/examples/plugins/echo/package.json +17 -0
  12. package/examples/plugins/echo/register.ts +187 -0
  13. package/node_modules/@vellumai/egress-proxy/src/types.ts +19 -0
  14. package/openapi.yaml +212 -68
  15. package/package.json +1 -1
  16. package/src/__tests__/app-compiler.test.ts +57 -0
  17. package/src/__tests__/approval-cascade.test.ts +7 -2
  18. package/src/__tests__/auto-analysis-end-to-end.test.ts +1 -0
  19. package/src/__tests__/avatar-generator.test.ts +4 -2
  20. package/src/__tests__/bundled-asset.test.ts +6 -6
  21. package/src/__tests__/catalog-cache.test.ts +69 -0
  22. package/src/__tests__/checker.test.ts +459 -171
  23. package/src/__tests__/circuit-breaker-pipeline.test.ts +406 -0
  24. package/src/__tests__/compaction-events.test.ts +501 -0
  25. package/src/__tests__/compaction-pipeline.test.ts +210 -0
  26. package/src/__tests__/compaction-strip-metadata-clear.test.ts +181 -0
  27. package/src/__tests__/compaction-timeout-recovery.test.ts +262 -0
  28. package/src/__tests__/config-model-image-provider.test.ts +110 -0
  29. package/src/__tests__/config-schema.test.ts +22 -9
  30. package/src/__tests__/config-watcher-cleanup-throttle.test.ts +0 -4
  31. package/src/__tests__/contacts-tools.test.ts +26 -0
  32. package/src/__tests__/context-overflow-policy.test.ts +7 -7
  33. package/src/__tests__/context-window-manager.test.ts +355 -4
  34. package/src/__tests__/conversation-abort-tool-results.test.ts +4 -1
  35. package/src/__tests__/conversation-agent-loop-overflow.test.ts +26 -30
  36. package/src/__tests__/conversation-agent-loop.test.ts +30 -141
  37. package/src/__tests__/conversation-confirmation-signals.test.ts +6 -1
  38. package/src/__tests__/conversation-history-web-search.test.ts +1 -0
  39. package/src/__tests__/conversation-init.benchmark.test.ts +2 -16
  40. package/src/__tests__/conversation-pairing.test.ts +174 -10
  41. package/src/__tests__/conversation-pre-run-repair.test.ts +4 -1
  42. package/src/__tests__/conversation-process-callsite.test.ts +3 -0
  43. package/src/__tests__/conversation-provider-retry-repair.test.ts +16 -7
  44. package/src/__tests__/conversation-queue.test.ts +29 -14
  45. package/src/__tests__/conversation-routes-disk-view.test.ts +7 -6
  46. package/src/__tests__/conversation-runtime-assembly.test.ts +155 -110
  47. package/src/__tests__/conversation-runtime-workspace.test.ts +23 -38
  48. package/src/__tests__/conversation-seed-composer.test.ts +2 -2
  49. package/src/__tests__/conversation-slash-queue.test.ts +7 -2
  50. package/src/__tests__/conversation-slash-unknown.test.ts +25 -2
  51. package/src/__tests__/conversation-speed-override.test.ts +6 -1
  52. package/src/__tests__/conversation-title-service.test.ts +116 -0
  53. package/src/__tests__/conversation-tool-setup-app-refresh.test.ts +41 -2
  54. package/src/__tests__/conversation-usage.test.ts +1 -1
  55. package/src/__tests__/conversation-workspace-cache-state.test.ts +4 -1
  56. package/src/__tests__/conversation-workspace-injection.test.ts +3 -0
  57. package/src/__tests__/conversation-workspace-tool-tracking.test.ts +4 -1
  58. package/src/__tests__/credential-health-service.test.ts +78 -9
  59. package/src/__tests__/credential-security-invariants.test.ts +2 -2
  60. package/src/__tests__/db-schedule-syntax-migration.test.ts +1 -0
  61. package/src/__tests__/empty-response-pipeline.test.ts +305 -0
  62. package/src/__tests__/extension-id-sync-guard.test.ts +3 -3
  63. package/src/__tests__/first-greeting.test.ts +247 -5
  64. package/src/__tests__/headless-browser-mode.test.ts +57 -0
  65. package/src/__tests__/history-repair-pipeline.test.ts +399 -0
  66. package/src/__tests__/host-browser-e2e-cloud.test.ts +307 -0
  67. package/src/__tests__/host-browser-e2e-self-hosted.test.ts +3 -3
  68. package/src/__tests__/host-proxy-interface.test.ts +36 -2
  69. package/src/__tests__/image-credentials.test.ts +137 -0
  70. package/src/__tests__/image-service-dispatcher.test.ts +186 -0
  71. package/src/__tests__/injector-chain.test.ts +526 -0
  72. package/src/__tests__/intent-routing.test.ts +0 -26
  73. package/src/__tests__/llm-call-pipeline.test.ts +285 -0
  74. package/src/__tests__/llm-schema.test.ts +1 -1
  75. package/src/__tests__/media-generate-image.test.ts +119 -13
  76. package/src/__tests__/memory-retrieval-pipeline.test.ts +401 -0
  77. package/src/__tests__/memory-upsert-concurrency.test.ts +1 -0
  78. package/src/__tests__/migration-import-from-url.test.ts +5 -68
  79. package/src/__tests__/model-intents.test.ts +4 -2
  80. package/src/__tests__/notification-broadcaster.test.ts +3 -3
  81. package/src/__tests__/notification-decision-strategy.test.ts +0 -11
  82. package/src/__tests__/notification-schedule-notify-dedup.test.ts +108 -0
  83. package/src/__tests__/oauth-apps-routes.test.ts +1 -1
  84. package/src/__tests__/oauth-cli.test.ts +14 -12
  85. package/src/__tests__/oauth-connect-orchestrator.test.ts +4 -13
  86. package/src/__tests__/oauth-provider-serializer.test.ts +6 -4
  87. package/src/__tests__/oauth-provider-visibility.test.ts +3 -5
  88. package/src/__tests__/oauth-providers-routes.test.ts +3 -2
  89. package/src/__tests__/oauth-store.test.ts +41 -76
  90. package/src/__tests__/onboarding-template-contract.test.ts +16 -64
  91. package/src/__tests__/openai-image-service.test.ts +368 -0
  92. package/src/__tests__/overflow-reduce-pipeline.test.ts +676 -0
  93. package/src/__tests__/permission-checker-host-gate.test.ts +0 -24
  94. package/src/__tests__/persist-onboarding-artifacts.test.ts +266 -0
  95. package/src/__tests__/persistence-pipeline.test.ts +377 -0
  96. package/src/__tests__/pipeline-runner.test.ts +565 -0
  97. package/src/__tests__/platform.test.ts +5 -2
  98. package/src/__tests__/plugin-bootstrap.test.ts +483 -0
  99. package/src/__tests__/plugin-registry.test.ts +273 -0
  100. package/src/__tests__/plugin-route-contribution.test.ts +288 -0
  101. package/src/__tests__/plugin-skill-contribution.test.ts +367 -0
  102. package/src/__tests__/plugin-tool-contribution.test.ts +286 -0
  103. package/src/__tests__/plugin-types.test.ts +320 -0
  104. package/src/__tests__/pricing.test.ts +44 -12
  105. package/src/__tests__/proxy-approval-callback.test.ts +69 -8
  106. package/src/__tests__/reaction-persistence.test.ts +1 -0
  107. package/src/__tests__/regenerate-fire-and-forget-trace.test.ts +1 -0
  108. package/src/__tests__/registry.test.ts +0 -2
  109. package/src/__tests__/schedule-routes.test.ts +131 -1
  110. package/src/__tests__/scheduler-recurrence.test.ts +14 -70
  111. package/src/__tests__/scheduler-reuse-conversation.test.ts +10 -50
  112. package/src/__tests__/secret-detection-handler.test.ts +0 -10
  113. package/src/__tests__/shell-identity.test.ts +0 -134
  114. package/src/__tests__/suggestion-routes.test.ts +103 -4
  115. package/src/__tests__/task-memory-cleanup.test.ts +1 -0
  116. package/src/__tests__/task-scheduler.test.ts +3 -15
  117. package/src/__tests__/test-preload.ts +11 -0
  118. package/src/__tests__/title-generate-pipeline.test.ts +224 -0
  119. package/src/__tests__/token-estimate-pipeline.test.ts +431 -0
  120. package/src/__tests__/tool-error-pipeline.test.ts +244 -0
  121. package/src/__tests__/tool-execute-pipeline.test.ts +431 -0
  122. package/src/__tests__/tool-execution-pipeline.benchmark.test.ts +0 -6
  123. package/src/__tests__/tool-executor-shell-integration.test.ts +7 -10
  124. package/src/__tests__/tool-executor.test.ts +141 -0
  125. package/src/__tests__/tool-result-truncate-pipeline.test.ts +356 -0
  126. package/src/__tests__/tool-result-truncation.test.ts +0 -110
  127. package/src/__tests__/user-plugin-loader.test.ts +191 -0
  128. package/src/__tests__/workspace-migration-046-seed-conversation-starters-callsite.test.ts +185 -0
  129. package/src/__tests__/workspace-migration-049-release-notes-default-sonnet.test.ts +100 -0
  130. package/src/__tests__/workspace-migration-050-seed-main-agent-opus-callsite.test.ts +171 -0
  131. package/src/__tests__/workspace-migration-051-seed-conversation-summarization-callsite.test.ts +252 -0
  132. package/src/__tests__/workspace-migration-remove-hooks.test.ts +99 -0
  133. package/src/__tests__/workspace-policy.test.ts +21 -3
  134. package/src/agent/loop.ts +340 -102
  135. package/src/approvals/__tests__/guardian-feed-event.test.ts +304 -0
  136. package/src/approvals/guardian-request-resolvers.ts +80 -0
  137. package/src/backup/__tests__/backup-worker.test.ts +2 -13
  138. package/src/backup/backup-worker.ts +3 -15
  139. package/src/bundler/app-compiler.ts +84 -1
  140. package/src/calls/call-state.ts +2 -2
  141. package/src/channels/__tests__/types.test.ts +3 -3
  142. package/src/channels/types.ts +6 -4
  143. package/src/cli/__tests__/notifications.test.ts +87 -211
  144. package/src/cli/commands/__tests__/backup.test.ts +1 -1
  145. package/src/cli/commands/__tests__/image-generation.test.ts +255 -35
  146. package/src/cli/commands/__tests__/inference-send.test.ts +12 -0
  147. package/src/cli/commands/__tests__/tts-synthesize.test.ts +12 -0
  148. package/src/cli/commands/backup.ts +2 -2
  149. package/src/cli/commands/clients.ts +138 -0
  150. package/src/cli/commands/completions.ts +2 -9
  151. package/src/cli/commands/conversations.ts +55 -7
  152. package/src/cli/commands/image-generation.ts +33 -34
  153. package/src/cli/commands/notifications.ts +68 -103
  154. package/src/cli/commands/oauth/__tests__/providers-register.test.ts +1 -1
  155. package/src/cli/commands/oauth/__tests__/providers-update.test.ts +1 -1
  156. package/src/cli/commands/oauth/connect.ts +2 -2
  157. package/src/cli/commands/oauth/providers.ts +176 -8
  158. package/src/cli/commands/oauth/status.ts +46 -36
  159. package/src/cli/commands/skills.ts +3 -4
  160. package/src/cli/program.ts +25 -29
  161. package/src/config/__tests__/backup-schema.test.ts +7 -2
  162. package/src/config/bundled-skills/app-builder/SKILL.md +2 -2
  163. package/src/config/bundled-skills/app-builder/references/WIDGETS.md +10 -10
  164. package/src/config/bundled-skills/contacts/tools/contact-merge.ts +66 -87
  165. package/src/config/bundled-skills/contacts/tools/contact-search.ts +28 -51
  166. package/src/config/bundled-skills/contacts/tools/contact-upsert.ts +22 -40
  167. package/src/config/bundled-skills/image-studio/SKILL.md +2 -1
  168. package/src/config/bundled-skills/image-studio/TOOLS.json +2 -1
  169. package/src/config/bundled-skills/image-studio/tools/media-generate-image.ts +23 -39
  170. package/src/config/bundled-skills/messaging/SKILL.md +3 -3
  171. package/src/config/bundled-skills/messaging/tools/__tests__/messaging-feed-events.test.ts +207 -0
  172. package/src/config/bundled-skills/messaging/tools/messaging-archive-by-sender.ts +12 -0
  173. package/src/config/bundled-skills/messaging/tools/messaging-send.ts +58 -0
  174. package/src/config/bundled-skills/schedule/SKILL.md +8 -3
  175. package/src/config/bundled-skills/schedule/TOOLS.json +15 -7
  176. package/src/config/bundled-skills/schedule/references/SCRIPT_MODE_PATTERNS.md +59 -0
  177. package/src/config/bundled-tool-registry.ts +0 -15
  178. package/src/config/feature-flag-registry.json +17 -1
  179. package/src/config/schema.ts +19 -0
  180. package/src/config/schemas/backup.ts +1 -1
  181. package/src/config/schemas/conversations.ts +16 -0
  182. package/src/config/schemas/llm.ts +2 -3
  183. package/src/config/schemas/security.ts +6 -6
  184. package/src/config/schemas/tts.ts +11 -0
  185. package/src/config/skill-state.ts +6 -2
  186. package/src/config/skills.ts +94 -5
  187. package/src/context/__tests__/compact-prompt.test.ts +27 -9
  188. package/src/context/prompts/compact.md +26 -12
  189. package/src/context/tool-result-truncation.ts +3 -63
  190. package/src/context/window-manager.ts +190 -16
  191. package/src/credential-health/credential-health-service.ts +19 -6
  192. package/src/daemon/__tests__/conversation-feed-event.test.ts +317 -0
  193. package/src/daemon/__tests__/conversation-lifecycle-auto-analyze.test.ts +4 -12
  194. package/src/daemon/__tests__/conversation-tool-setup.test.ts +14 -15
  195. package/src/daemon/config-watcher.ts +0 -2
  196. package/src/daemon/context-overflow-policy.ts +4 -13
  197. package/src/daemon/conversation-agent-loop-handlers.ts +83 -22
  198. package/src/daemon/conversation-agent-loop.ts +984 -683
  199. package/src/daemon/conversation-history.ts +10 -19
  200. package/src/daemon/conversation-lifecycle.ts +37 -19
  201. package/src/daemon/conversation-notifiers.ts +2 -110
  202. package/src/daemon/conversation-process.ts +14 -7
  203. package/src/daemon/conversation-runtime-assembly.ts +532 -411
  204. package/src/daemon/conversation-tool-setup.ts +41 -4
  205. package/src/daemon/conversation.ts +80 -35
  206. package/src/daemon/external-plugins-bootstrap.ts +478 -0
  207. package/src/daemon/first-greeting.ts +191 -14
  208. package/src/daemon/handlers/config-model.ts +11 -0
  209. package/src/daemon/handlers/skills.ts +5 -1
  210. package/src/daemon/lifecycle.ts +33 -68
  211. package/src/daemon/message-types/computer-use.ts +2 -34
  212. package/src/daemon/message-types/conversations.ts +49 -0
  213. package/src/daemon/message-types/messages.ts +12 -0
  214. package/src/daemon/server.ts +5 -3
  215. package/src/daemon/shutdown-handlers.ts +2 -12
  216. package/src/daemon/tool-side-effects.ts +14 -56
  217. package/src/heartbeat/__tests__/heartbeat-feed-event.test.ts +160 -0
  218. package/src/heartbeat/heartbeat-service.ts +24 -1
  219. package/src/home/__tests__/feed-population-integration.test.ts +312 -0
  220. package/src/home/emit-feed-event.ts +7 -0
  221. package/src/home/feed-types.ts +41 -2
  222. package/src/home/rewrite-command-preview.ts +66 -0
  223. package/src/ipc/__tests__/socket-path.test.ts +11 -50
  224. package/src/ipc/cli-client.ts +1 -1
  225. package/src/ipc/cli-server.ts +3 -3
  226. package/src/ipc/gateway-client.ts +4 -1
  227. package/src/ipc/routes/browser-context.ts +2 -0
  228. package/src/ipc/routes/browser.ts +1 -0
  229. package/src/ipc/routes/get-contact.ts +16 -0
  230. package/src/ipc/routes/index.ts +14 -0
  231. package/src/ipc/routes/list-clients.ts +31 -0
  232. package/src/ipc/routes/merge-contacts.ts +17 -0
  233. package/src/ipc/routes/notification.ts +133 -0
  234. package/src/ipc/routes/rename-conversation.ts +59 -0
  235. package/src/ipc/routes/search-contacts.ts +19 -0
  236. package/src/ipc/routes/upsert-contact.ts +25 -0
  237. package/src/ipc/socket-path.ts +14 -38
  238. package/src/media/app-icon-generator.ts +23 -46
  239. package/src/media/avatar-router.ts +26 -41
  240. package/src/media/gemini-image-service.ts +8 -41
  241. package/src/media/image-credentials.ts +73 -0
  242. package/src/media/image-service.ts +85 -0
  243. package/src/media/openai-image-service.ts +131 -0
  244. package/src/media/types.ts +46 -0
  245. package/src/memory/conversation-crud.ts +48 -18
  246. package/src/memory/conversation-queries.ts +57 -4
  247. package/src/memory/conversation-title-service.ts +25 -0
  248. package/src/memory/db-init.ts +8 -0
  249. package/src/memory/embedding-gemini.test.ts +41 -2
  250. package/src/memory/embedding-gemini.ts +6 -1
  251. package/src/memory/graph/bootstrap.test.ts +282 -0
  252. package/src/memory/graph/bootstrap.ts +8 -5
  253. package/src/memory/graph/extraction.ts +10 -2
  254. package/src/memory/graph/graph-search.test.ts +1 -0
  255. package/src/memory/graph/inspect.ts +2 -2
  256. package/src/memory/graph/retriever.ts +10 -3
  257. package/src/memory/migrations/041-approval-prompt-ts-tracker.ts +26 -0
  258. package/src/memory/migrations/149-oauth-tables.ts +1 -0
  259. package/src/memory/migrations/223-schedule-script-column.ts +11 -0
  260. package/src/memory/migrations/224-oauth-providers-managed-service-is-paid.ts +24 -0
  261. package/src/memory/migrations/225-oauth-providers-available-scopes.ts +13 -0
  262. package/src/memory/migrations/index.ts +4 -0
  263. package/src/memory/pkb/pkb-index.test.ts +1 -0
  264. package/src/memory/pkb/pkb-reconcile.test.ts +1 -0
  265. package/src/memory/pkb/pkb-search.test.ts +65 -4
  266. package/src/memory/pkb/pkb-search.ts +40 -18
  267. package/src/memory/qdrant-client.test.ts +60 -0
  268. package/src/memory/qdrant-client.ts +25 -0
  269. package/src/memory/schema/infrastructure.ts +1 -0
  270. package/src/memory/schema/oauth.ts +4 -1
  271. package/src/messaging/providers/slack/render-transcript.test.ts +77 -29
  272. package/src/messaging/providers/slack/render-transcript.ts +58 -0
  273. package/src/notifications/conversation-pairing.ts +78 -19
  274. package/src/notifications/copy-composer.ts +0 -5
  275. package/src/notifications/emit-signal.ts +1 -1
  276. package/src/notifications/signal.ts +1 -2
  277. package/src/oauth/AGENTS.md +1 -1
  278. package/src/oauth/__tests__/identity-verifier.test.ts +2 -1
  279. package/src/oauth/connect-orchestrator.ts +8 -34
  280. package/src/oauth/connect-types.ts +6 -10
  281. package/src/oauth/manual-token-connection.ts +23 -0
  282. package/src/oauth/oauth-store.ts +30 -14
  283. package/src/oauth/provider-serializer.ts +6 -1
  284. package/src/oauth/seed-providers.ts +56 -108
  285. package/src/outbound-proxy/http-forwarder.ts +9 -0
  286. package/src/permissions/approval-policy.test.ts +293 -18
  287. package/src/permissions/approval-policy.ts +110 -58
  288. package/src/permissions/arg-parser.test.ts +161 -0
  289. package/src/permissions/arg-parser.ts +141 -0
  290. package/src/permissions/bash-risk-classifier.test.ts +414 -2
  291. package/src/permissions/bash-risk-classifier.ts +303 -60
  292. package/src/permissions/checker.ts +157 -29
  293. package/src/permissions/command-registry.test.ts +239 -0
  294. package/src/permissions/command-registry.ts +234 -54
  295. package/src/permissions/defaults.ts +5 -4
  296. package/src/permissions/gateway-threshold-reader.ts +196 -0
  297. package/src/permissions/prompter.ts +4 -0
  298. package/src/permissions/risk-types.ts +61 -4
  299. package/src/permissions/schedule-risk-classifier.test.ts +129 -0
  300. package/src/permissions/schedule-risk-classifier.ts +85 -0
  301. package/src/permissions/shell-identity.ts +2 -42
  302. package/src/permissions/types.ts +2 -0
  303. package/src/permissions/workspace-policy.ts +8 -3
  304. package/src/plugins/defaults/circuit-breaker.ts +146 -0
  305. package/src/plugins/defaults/compaction.ts +145 -0
  306. package/src/plugins/defaults/empty-response.ts +126 -0
  307. package/src/plugins/defaults/history-repair.ts +85 -0
  308. package/src/plugins/defaults/index.ts +116 -0
  309. package/src/plugins/defaults/injectors.ts +491 -0
  310. package/src/plugins/defaults/llm-call.ts +82 -0
  311. package/src/plugins/defaults/memory-retrieval.ts +226 -0
  312. package/src/plugins/defaults/overflow-reduce.ts +181 -0
  313. package/src/plugins/defaults/persistence.ts +129 -0
  314. package/src/plugins/defaults/title-generate.ts +95 -0
  315. package/src/plugins/defaults/token-estimate.ts +104 -0
  316. package/src/plugins/defaults/tool-error.ts +126 -0
  317. package/src/plugins/defaults/tool-execute.ts +89 -0
  318. package/src/plugins/defaults/tool-result-truncate.ts +88 -0
  319. package/src/plugins/pipeline.ts +316 -0
  320. package/src/plugins/plugin-skill-contributions.ts +292 -0
  321. package/src/plugins/registry.ts +241 -0
  322. package/src/plugins/types.ts +1134 -0
  323. package/src/plugins/user-loader.ts +177 -0
  324. package/src/prompts/templates/BOOTSTRAP.md +27 -77
  325. package/src/providers/model-catalog.ts +52 -29
  326. package/src/providers/model-intents.ts +1 -1
  327. package/src/providers/openrouter/client.ts +5 -1
  328. package/src/providers/speech-to-text/deepgram-realtime.test.ts +61 -0
  329. package/src/providers/speech-to-text/deepgram-realtime.ts +57 -0
  330. package/src/providers/speech-to-text/xai-realtime.test.ts +72 -4
  331. package/src/providers/speech-to-text/xai-realtime.ts +39 -14
  332. package/src/runtime/AGENTS.md +25 -16
  333. package/src/runtime/__tests__/browser-extension-pair-routes.test.ts +3 -3
  334. package/src/runtime/__tests__/client-registry.test.ts +293 -0
  335. package/src/runtime/client-registry.ts +261 -0
  336. package/src/runtime/http-server.ts +77 -8
  337. package/src/runtime/http-types.ts +0 -2
  338. package/src/runtime/migrations/vbundle-builder.ts +1 -22
  339. package/src/runtime/routes/approval-prompt-ts-tracker.ts +51 -31
  340. package/src/runtime/routes/approval-routes.ts +17 -0
  341. package/src/runtime/routes/browser-extension-pair-routes.ts +27 -8
  342. package/src/runtime/routes/conversation-routes.ts +223 -116
  343. package/src/runtime/routes/inbound-message-handler.ts +88 -13
  344. package/src/runtime/routes/memory-item-routes.test.ts +1 -0
  345. package/src/runtime/routes/migration-routes.ts +0 -3
  346. package/src/runtime/routes/playground/__tests__/force-compact.test.ts +284 -0
  347. package/src/runtime/routes/playground/__tests__/guard.test.ts +80 -0
  348. package/src/runtime/routes/playground/__tests__/inject-failures.test.ts +294 -0
  349. package/src/runtime/routes/playground/__tests__/reset-circuit.test.ts +271 -0
  350. package/src/runtime/routes/playground/__tests__/seed-conversation.test.ts +202 -0
  351. package/src/runtime/routes/playground/__tests__/seeded-conversations.test.ts +309 -0
  352. package/src/runtime/routes/playground/__tests__/state.test.ts +224 -0
  353. package/src/runtime/routes/playground/conversation-not-found.ts +29 -0
  354. package/src/runtime/routes/playground/deps.ts +56 -0
  355. package/src/runtime/routes/playground/force-compact.ts +73 -0
  356. package/src/runtime/routes/playground/guard.ts +37 -0
  357. package/src/runtime/routes/playground/index.ts +28 -0
  358. package/src/runtime/routes/playground/inject-failures.ts +159 -0
  359. package/src/runtime/routes/playground/reset-circuit.ts +115 -0
  360. package/src/runtime/routes/playground/seed-conversation.ts +139 -0
  361. package/src/runtime/routes/playground/seeded-conversations.ts +78 -0
  362. package/src/runtime/routes/playground/state.ts +78 -0
  363. package/src/runtime/routes/schedule-routes.ts +89 -8
  364. package/src/runtime/skill-route-registry.ts +75 -15
  365. package/src/schedule/run-script.ts +68 -0
  366. package/src/schedule/schedule-store.ts +7 -1
  367. package/src/schedule/scheduler.ts +48 -8
  368. package/src/skills/catalog-cache.ts +12 -5
  369. package/src/tools/browser/__tests__/browser-status.test.ts +189 -0
  370. package/src/tools/browser/browser-execution.ts +88 -19
  371. package/src/tools/browser/cdp-client/__tests__/extension-cdp-client.test.ts +230 -0
  372. package/src/tools/browser/cdp-client/__tests__/factory.test.ts +146 -3
  373. package/src/tools/browser/cdp-client/extension-cdp-client.ts +54 -3
  374. package/src/tools/browser/cdp-client/factory.ts +15 -4
  375. package/src/tools/executor.ts +126 -74
  376. package/src/tools/network/script-proxy/session-manager.ts +37 -1
  377. package/src/tools/permission-checker.ts +98 -49
  378. package/src/tools/policy-context.ts +4 -0
  379. package/src/tools/registry.ts +140 -3
  380. package/src/tools/schedule/create.ts +23 -8
  381. package/src/tools/schedule/update.ts +3 -1
  382. package/src/tools/secret-detection-handler.ts +0 -51
  383. package/src/tools/system/avatar-generator.ts +6 -2
  384. package/src/tools/types.ts +28 -2
  385. package/src/util/platform.ts +7 -2
  386. package/src/util/pricing.ts +26 -3
  387. package/src/workspace/migrations/006-services-config.ts +2 -4
  388. package/src/workspace/migrations/022-move-hooks-to-workspace.ts +2 -3
  389. package/src/workspace/migrations/041-backfill-google-gmail-settings-scope.ts +3 -4
  390. package/src/workspace/migrations/046-seed-conversation-starters-callsite.ts +108 -0
  391. package/src/workspace/migrations/047-remove-watch-callsites.ts +54 -0
  392. package/src/workspace/migrations/048-remove-workspace-hooks.ts +81 -0
  393. package/src/workspace/migrations/049-release-notes-default-sonnet.ts +80 -0
  394. package/src/workspace/migrations/050-seed-main-agent-opus-callsite.ts +86 -0
  395. package/src/workspace/migrations/051-seed-conversation-summarization-callsite.ts +128 -0
  396. package/src/workspace/migrations/registry.ts +12 -0
  397. package/tsconfig.json +1 -1
  398. package/hook-templates/debug-prompt-logger/hook.json +0 -7
  399. package/hook-templates/debug-prompt-logger/run.sh +0 -66
  400. package/src/__tests__/compaction-circuit-breaker.test.ts +0 -336
  401. package/src/__tests__/context-overflow-approval.test.ts +0 -156
  402. package/src/__tests__/hooks-blocking.test.ts +0 -178
  403. package/src/__tests__/hooks-cli.test.ts +0 -182
  404. package/src/__tests__/hooks-config.test.ts +0 -108
  405. package/src/__tests__/hooks-discovery.test.ts +0 -211
  406. package/src/__tests__/hooks-integration.test.ts +0 -196
  407. package/src/__tests__/hooks-manager.test.ts +0 -226
  408. package/src/__tests__/hooks-runner.test.ts +0 -175
  409. package/src/__tests__/hooks-settings.test.ts +0 -160
  410. package/src/__tests__/hooks-templates.test.ts +0 -169
  411. package/src/__tests__/hooks-ts-runner.test.ts +0 -170
  412. package/src/__tests__/hooks-watch.test.ts +0 -112
  413. package/src/__tests__/notification-schedule-dedup.test.ts +0 -213
  414. package/src/__tests__/oauth-scope-policy.test.ts +0 -180
  415. package/src/__tests__/send-notification-tool.test.ts +0 -83
  416. package/src/cli/commands/shotgun.ts +0 -266
  417. package/src/config/bundled-skills/conversations/SKILL.md +0 -20
  418. package/src/config/bundled-skills/conversations/TOOLS.json +0 -23
  419. package/src/config/bundled-skills/conversations/tools/rename-conversation.ts +0 -88
  420. package/src/config/bundled-skills/heartbeat/SKILL.md +0 -43
  421. package/src/config/bundled-skills/notifications/SKILL.md +0 -40
  422. package/src/config/bundled-skills/notifications/TOOLS.json +0 -80
  423. package/src/config/bundled-skills/notifications/tools/send-notification.ts +0 -152
  424. package/src/config/bundled-skills/notifications/tools/shared.ts +0 -13
  425. package/src/config/bundled-skills/screen-watch/SKILL.md +0 -27
  426. package/src/config/bundled-skills/screen-watch/TOOLS.json +0 -35
  427. package/src/config/bundled-skills/screen-watch/tools/start-screen-watch.ts +0 -12
  428. package/src/config/bundled-skills/skills-catalog/SKILL.md +0 -84
  429. package/src/daemon/context-overflow-approval.ts +0 -52
  430. package/src/daemon/watch-handler.ts +0 -399
  431. package/src/hooks/cli.ts +0 -253
  432. package/src/hooks/config.ts +0 -100
  433. package/src/hooks/discovery.ts +0 -135
  434. package/src/hooks/manager.ts +0 -179
  435. package/src/hooks/runner.ts +0 -117
  436. package/src/hooks/templates.ts +0 -77
  437. package/src/hooks/types.ts +0 -75
  438. package/src/oauth/scope-policy.ts +0 -89
  439. package/src/runtime/gateway-internal-client.ts +0 -94
  440. package/src/runtime/routes/watch-routes.ts +0 -156
  441. package/src/signals/shotgun.ts +0 -203
  442. package/src/tools/watch/screen-watch.ts +0 -144
  443. package/src/tools/watch/watch-state.ts +0 -142
@@ -177,9 +177,7 @@ graph TB
177
177
  OVF_T2["Tier 2: Tool-result truncation<br/>4,000 chars per result"]
178
178
  OVF_T3["Tier 3: Media/file stubbing"]
179
179
  OVF_T4["Tier 4: Injection downgrade<br/>→ minimal mode"]
180
- OVF_POLICY["Overflow policy resolver<br/>auto_compress / request_approval / fail_gracefully"]
181
- OVF_APPROVE["Approval gate<br/>PermissionPrompter"]
182
- OVF_DENY["Graceful deny message<br/>(not conversation_error)"]
180
+ OVF_LATEST["Auto-compress latest turn<br/>(force=true, minKeepRecentUserTurns=0)"]
183
181
  end
184
182
 
185
183
  MSG_IN --> STORE
@@ -237,11 +235,8 @@ graph TB
237
235
  OVF_T1 --> OVF_T2
238
236
  OVF_T2 --> OVF_T3
239
237
  OVF_T3 --> OVF_T4
240
- OVF_T4 --> OVF_POLICY
241
- OVF_POLICY -->|"interactive"| OVF_APPROVE
242
- OVF_POLICY -->|"non-interactive"| OVF_T1
243
- OVF_APPROVE -->|"denied"| OVF_DENY
244
- OVF_APPROVE -->|"approved"| OVF_T1
238
+ OVF_T4 --> OVF_LATEST
239
+ OVF_LATEST -.->|"uses"| SUMMARIZE
245
240
  OVF_T1 -.->|"uses"| SUMMARIZE
246
241
  ```
247
242
 
@@ -251,7 +246,7 @@ Normal context compaction (the "Context Window Management" subgraph above) runs
251
246
 
252
247
  When compaction alone is insufficient — either because the conversation grew too fast between turns or because a single turn contains extremely large payloads — the overflow recovery pipeline takes over. The pipeline's first tier (forced compaction) reuses the same `maybeCompact()` summarization machinery but with emergency parameters: `force: true` bypasses cooldown guards, `minKeepRecentUserTurns: 0` allows summarizing even the most recent history, and `targetInputTokensOverride` sets a tighter budget. Subsequent tiers (tool-result truncation, media stubbing, injection downgrade) apply progressively more aggressive payload reduction without involving the summarizer.
253
248
 
254
- If all four reducer tiers are exhausted, the overflow policy resolver determines whether to compress the latest user turn. In interactive sessions, the user is prompted for approval before this lossy step. If the user declines, the session emits a graceful assistant explanation message rather than a `conversation_error`, ending the turn cleanly. Non-interactive sessions auto-compress without prompting.
249
+ If all four reducer tiers are exhausted, the overflow policy resolver determines whether to compress the latest user turn. All sessions — interactive and non-interactive alike — auto-compress the latest turn without prompting. The only explicit opt-out is setting `contextWindow.overflowRecovery.interactiveLatestTurnCompression` (or `nonInteractiveLatestTurnCompression`) to `"drop"`, which short-circuits to a graceful failure instead. Disabling overflow recovery entirely (`contextWindow.overflowRecovery.enabled: false`) also yields a graceful failure.
255
250
 
256
251
  The key distinction: normal compaction is a cost-optimized background process that preserves conversational quality; overflow recovery is a convergence mechanism that prioritizes session survival over context richness. Both share the same summarization infrastructure but operate under different pressure thresholds and constraints.
257
252
 
@@ -303,16 +298,16 @@ stateDiagram-v2
303
298
 
304
299
  **Item extraction** uses LLM-powered extraction (with pattern-based fallback) to identify memorable information from conversation messages. Each extracted item belongs to one of eight kinds:
305
300
 
306
- | Kind | Description | Base Lifetime |
307
- | ------------ | ------------------------------------------------------------------ | ------------- |
308
- | `identity` | Personal info, facts, relationships | 6 months |
309
- | `preference` | Likes, dislikes, preferred approaches/tools | 3 months |
301
+ | Kind | Description | Base Lifetime |
302
+ | ------------ | -------------------------------------------------------------------- | ------------- |
303
+ | `identity` | Personal info, facts, relationships | 6 months |
304
+ | `preference` | Likes, dislikes, preferred approaches/tools | 3 months |
310
305
  | `journal` | Experiential reflections, journal-style notes, forward-looking items | 3 months |
311
- | `constraint` | Rules, requirements, directives | 1 month |
312
- | `project` | Project details, repos, tech stacks, action items | 2 weeks |
313
- | `decision` | Choices made, approaches selected | 2 weeks |
314
- | `event` | Deadlines, milestones, meetings, dates | 3 days |
315
- | `capability` | Skill catalog entries (system-generated, not LLM-extracted) | never expires |
306
+ | `constraint` | Rules, requirements, directives | 1 month |
307
+ | `project` | Project details, repos, tech stacks, action items | 2 weeks |
308
+ | `decision` | Choices made, approaches selected | 2 weeks |
309
+ | `event` | Deadlines, milestones, meetings, dates | 3 days |
310
+ | `capability` | Skill catalog entries (system-generated, not LLM-extracted) | never expires |
316
311
 
317
312
  **Supersession chains** replace the old conflict resolution system. When the LLM extracts a new item that updates an existing one, it sets `supersedes` to the old item's ID and `overrideConfidence` to one of three levels:
318
313
 
@@ -552,13 +547,13 @@ The Anthropic provider places `cache_control: { type: 'ephemeral' }` on the **la
552
547
 
553
548
  ### Key files
554
549
 
555
- | File | Role |
556
- | ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
557
- | `assistant/src/workspace/top-level-scanner.ts` | Synchronous directory scanner with `MAX_TOP_LEVEL_ENTRIES` cap |
558
- | `assistant/src/workspace/top-level-renderer.ts` | Renders `TopLevelSnapshot` to `<workspace>` XML block |
559
- | `assistant/src/daemon/conversation-runtime-assembly.ts` | Runtime injections and legacy-block strip helpers (`<workspace>`, `<turn_context>`, `<channel_onboarding_playbook>`, `<onboarding_mode>`) |
560
- | `assistant/src/onboarding/onboarding-orchestrator.ts` | Builds assistant-owned onboarding runtime guidance from channel playbook + transport metadata |
561
- | `assistant/src/daemon/conversation-agent-loop.ts` | Agent loop orchestration, runtime injection wiring, legacy-block strip chain |
550
+ | File | Role |
551
+ | ------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
552
+ | `assistant/src/workspace/top-level-scanner.ts` | Synchronous directory scanner with `MAX_TOP_LEVEL_ENTRIES` cap |
553
+ | `assistant/src/workspace/top-level-renderer.ts` | Renders `TopLevelSnapshot` to `<workspace>` XML block |
554
+ | `assistant/src/daemon/conversation-runtime-assembly.ts` | Runtime injections and legacy-block strip helpers (`<workspace>`, `<turn_context>`, `<channel_onboarding_playbook>`, `<onboarding_mode>`) |
555
+ | `assistant/src/onboarding/onboarding-orchestrator.ts` | Builds assistant-owned onboarding runtime guidance from channel playbook + transport metadata |
556
+ | `assistant/src/daemon/conversation-agent-loop.ts` | Agent loop orchestration, runtime injection wiring, legacy-block strip chain |
562
557
 
563
558
  ---
564
559
 
@@ -593,11 +588,11 @@ graph TB
593
588
 
594
589
  ### Key files
595
590
 
596
- | File | Role |
597
- | ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
598
- | `assistant/src/daemon/date-context.ts` | `formatTurnTimestamp()` — generates the timestamp string used in `<turn_context>` |
599
- | `assistant/src/daemon/conversation-runtime-assembly.ts` | `buildUnifiedTurnContextBlock()` — constructs the unified `<turn_context>` block; legacy block strip helpers |
600
- | `assistant/src/daemon/conversation-agent-loop.ts` | Wiring: builds turn context, passes to `applyRuntimeInjections` |
591
+ | File | Role |
592
+ | ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
593
+ | `assistant/src/daemon/date-context.ts` | `formatTurnTimestamp()` — generates the timestamp string used in `<turn_context>` |
594
+ | `assistant/src/daemon/conversation-runtime-assembly.ts` | `buildUnifiedTurnContextBlock()` — constructs the unified `<turn_context>` block; legacy block strip helpers |
595
+ | `assistant/src/daemon/conversation-agent-loop.ts` | Wiring: builds turn context, passes to `applyRuntimeInjections` |
601
596
 
602
597
  ---
603
598
 
@@ -16,13 +16,14 @@ graph TB
16
16
 
17
17
  FIND_RULE -->|"Deny rule"| DENY["decision: deny<br/>Blocked by rule"]
18
18
  FIND_RULE -->|"Ask rule"| PROMPT_ASK["decision: prompt<br/>Always ask user"]
19
- FIND_RULE -->|"Allow rule"| RISK_CHECK{"Risk level?"}
20
- FIND_RULE -->|"No match"| NO_MATCH{"Fallback logic"}
19
+ FIND_RULE -->|"Allow rule / No match"| SANDBOX_CHECK{"sandboxAutoApprove?<br/>(bash + allowlisted +<br/>containerized)"}
20
+
21
+ SANDBOX_CHECK -->|"yes"| AUTO_SANDBOX["decision: allow<br/>Sandbox auto-approve"]
22
+ SANDBOX_CHECK -->|"no, has Allow rule"| RISK_CHECK{"Risk level?"}
23
+ SANDBOX_CHECK -->|"no, no match"| NO_MATCH{"Fallback logic"}
21
24
 
22
25
  RISK_CHECK -->|"Low / Medium"| AUTO_ALLOW["decision: allow<br/>Auto-allowed by rule"]
23
- RISK_CHECK -->|"High"| HIGH_CHECK{"shouldAutoAllowHighRisk()<br/>(containerized bash?)"}
24
- HIGH_CHECK -->|"yes"| AUTO_ALLOW
25
- HIGH_CHECK -->|"no"| RISK_THRESHOLD{"Risk-based<br/>threshold fallback"}
26
+ RISK_CHECK -->|"High"| RISK_THRESHOLD{"Risk-based<br/>threshold fallback"}
26
27
 
27
28
  NO_MATCH -->|"tool.origin === 'skill'"| PROMPT_SKILL["decision: prompt<br/>Skill tools always ask"]
28
29
  NO_MATCH -->|"strict mode"| PROMPT_STRICT["decision: prompt<br/>No implicit auto-allow"]
@@ -165,7 +166,7 @@ When a permission prompt is sent to the client (via `confirmation_request` SSE e
165
166
  | `allowlistOptions` | Suggested patterns for "always allow" rules |
166
167
  | `scopeOptions` | Suggested scopes for rule persistence |
167
168
 
168
- The user can respond with: `allow` (one-time), `always_allow` (create allow rule), `deny` (one-time), or `always_deny` (create deny rule). High-risk operations with an allow rule in containerized environments are auto-allowed at runtime by `DefaultApprovalPolicy.shouldAutoAllowHighRisk()` without requiring persisted state. All other risk-based decisions use the `autoApproveUpTo` threshold (default: `"low"`) -- tools at or below the threshold are auto-allowed, those above are prompted.
169
+ The user can respond with: `allow` (one-time), `always_allow` (create allow rule), `deny` (one-time), or `always_deny` (create deny rule). In containerized environments, commands tagged with `sandboxAutoApprove` in their risk spec are auto-allowed via the approval policy's sandbox auto-approve check; non-allowlisted commands (network tools, runtimes, package managers) use the user's `autoApproveUpTo` threshold. All other risk-based decisions use the `autoApproveUpTo` threshold (default: `"low"`) -- tools at or below the threshold are auto-allowed, those above are prompted.
169
170
 
170
171
  ### Canonical Paths
171
172
 
@@ -2,23 +2,39 @@
2
2
 
3
3
  ## Overview
4
4
 
5
- Phase 2 of browser automation introduced a three-tier backend selection chain for macOS-originated turns, enabling the assistant to prefer the user's real Chrome session (via the paired extension) over a sandboxed Playwright instance. When the extension is unavailable, the system falls back through cdp-inspect (direct Chrome DevTools Protocol attach) before resorting to the local Playwright browser.
5
+ Phase 2 of browser automation introduced a three-tier backend selection chain for macOS-originated turns, enabling the assistant to prefer the user's real Chrome session over a sandboxed Playwright instance. Two transport paths exist for the top-priority "extension" backend:
6
6
 
7
- This document describes the runtime architecture, backend precedence rules, and the manual QA playbook for verifying correct backend selection.
7
+ 1. **Chrome Extension Registry (WebSocket)**: When the user has the Vellum Chrome Extension installed and paired, `host_browser_request` frames route through the `ChromeExtensionRegistry` singleton over a dedicated `/v1/browser-relay` WebSocket.
8
+ 2. **macOS Host Browser Proxy (SSE)**: When the macOS desktop client is connected but the Chrome extension is absent, `host_browser_request` frames travel through `assistantEventHub` (SSE). The desktop client receives the frames via its SSE connection, executes CDP commands against the local Chrome, and POSTs results back to `/v1/host-browser-result`.
9
+
10
+ When neither transport is available, the system falls back through cdp-inspect (direct Chrome DevTools Protocol attach) before resorting to the local Playwright browser.
11
+
12
+ This document describes the runtime architecture, backend precedence rules, transport matrix, and the manual QA playbook for verifying correct backend selection.
8
13
 
9
14
  ## Component Inventory
10
15
 
11
- | Component | Location | Role |
12
- | --------------------------- | -------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
13
- | **ChromeExtensionRegistry** | `runtime/chrome-extension-registry.ts` | Tracks active extension WebSocket connections keyed by `(guardianId, clientInstanceId)`. Populated on WS `open`, drained on WS `close`. |
14
- | **HostBrowserProxy** | `daemon/host-browser-proxy.ts` | Per-conversation proxy that dispatches `host_browser_request` frames and awaits `host_browser_result` responses. |
15
- | **CDP Factory** | `tools/browser/cdp-client/factory.ts` | Builds the ordered candidate list and returns a `ScopedCdpClient` with per-invocation failover. |
16
- | **BrowserSessionManager** | `browser-session/manager.ts` | Routes CDP commands through the selected backend with session tracking. |
17
- | **CdpInspectClient** | `tools/browser/cdp-client/cdp-inspect-client.ts` | Connects to a host Chrome instance via its remote-debugging WebSocket endpoint. |
18
- | **LocalCdpClient** | `tools/browser/cdp-client/local-cdp-client.ts` | Drives Playwright's CDPSession against the sacrificial-profile browser. |
19
- | **ExtensionCdpClient** | `tools/browser/cdp-client/extension-cdp-client.ts` | Routes CDP commands through the HostBrowserProxy to the user's real Chrome. |
20
- | **conversation-routes.ts** | `runtime/routes/conversation-routes.ts` | Wires `resolveHostBrowserSender()` to set `hostBrowserSenderOverride` when the extension is connected. Sets `turnInterfaceContext` from the `interface` field on the incoming message. |
21
- | **Desktop-auto config** | `config/schemas/host-browser.ts` | `desktopAuto.enabled` (default `true`) and `desktopAuto.cooldownMs` (default 30s) control automatic cdp-inspect on macOS. |
16
+ | Component | Location | Role |
17
+ | --------------------------- | -------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
18
+ | **ChromeExtensionRegistry** | `runtime/chrome-extension-registry.ts` | Tracks active extension WebSocket connections keyed by `(guardianId, clientInstanceId)`. Populated on WS `open`, drained on WS `close`. |
19
+ | **HostBrowserProxy** | `daemon/host-browser-proxy.ts` | Per-conversation proxy that dispatches `host_browser_request` frames and awaits `host_browser_result` responses. Wired to either the registry sender or the SSE hub sender. |
20
+ | **CDP Factory** | `tools/browser/cdp-client/factory.ts` | Builds the ordered candidate list and returns a `ScopedCdpClient` with per-invocation failover. |
21
+ | **BrowserSessionManager** | `browser-session/manager.ts` | Routes CDP commands through the selected backend with session tracking. |
22
+ | **CdpInspectClient** | `tools/browser/cdp-client/cdp-inspect-client.ts` | Connects to a host Chrome instance via its remote-debugging WebSocket endpoint. |
23
+ | **LocalCdpClient** | `tools/browser/cdp-client/local-cdp-client.ts` | Drives Playwright's CDPSession against the sacrificial-profile browser. |
24
+ | **ExtensionCdpClient** | `tools/browser/cdp-client/extension-cdp-client.ts` | Routes CDP commands through the HostBrowserProxy to the user's real Chrome. |
25
+ | **conversation-routes.ts** | `runtime/routes/conversation-routes.ts` | Wires `resolveHostBrowserSender()` to set `hostBrowserSenderOverride` when the extension is connected. For macOS without extension, provisions the proxy with the SSE hub sender. |
26
+ | **Desktop-auto config** | `config/schemas/host-browser.ts` | `desktopAuto.enabled` (default `true`) and `desktopAuto.cooldownMs` (default 30s) control automatic cdp-inspect on macOS. |
27
+
28
+ ## Transport Matrix
29
+
30
+ The following table shows how `host_browser_request` frames are delivered to the client based on the originating interface and extension connectivity:
31
+
32
+ | Interface | Extension Connected | Transport | Sender | Notes |
33
+ | ------------------ | ------------------- | ------------------------------- | ----------------------------------------------- | ---------------------------------------------------------------------- |
34
+ | `chrome-extension` | Yes (always) | WebSocket (`/v1/browser-relay`) | `ChromeExtensionRegistry.send(guardianId, msg)` | The only transport for chrome-extension turns; SSE fallback is invalid |
35
+ | `macos` | Yes | WebSocket (`/v1/browser-relay`) | `ChromeExtensionRegistry.send(guardianId, msg)` | Extension takes priority; browser tools route through user's Chrome |
36
+ | `macos` | No | SSE (`assistantEventHub`) | `onEvent` hub publisher | macOS host browser proxy mode; desktop client handles CDP locally |
37
+ | Other | Any | N/A | No browser proxy provisioned | Falls through to cdp-inspect or local Playwright |
22
38
 
23
39
  ## Wire Diagram
24
40
 
@@ -37,10 +53,12 @@ conversation-routes.ts
37
53
  | |
38
54
  | +-- entry found? --> registrySender (WS to extension)
39
55
  | | hostBrowserSenderOverride = registrySender
40
- | | provision HostBrowserProxy
56
+ | | provision HostBrowserProxy(registrySender)
41
57
  | |
42
58
  | +-- entry not found? --> SSE hub sender (default)
43
59
  | hostBrowserSenderOverride = undefined
60
+ | provision HostBrowserProxy(onEvent)
61
+ | [macOS natively supports host_browser]
44
62
  |
45
63
  v
46
64
  Agent loop invokes browser tool
@@ -50,6 +68,8 @@ getCdpClient(toolContext)
50
68
  |-- toolContext.hostBrowserProxy set?
51
69
  | AND hostBrowserProxy.isAvailable()?
52
70
  | --> candidate: extension (priority 1)
71
+ | [Transport: WS via registry when extension present,
72
+ | SSE via hub when macOS host proxy only]
53
73
  |
54
74
  |-- transportInterface === "macos"
55
75
  | AND desktopAuto.enabled?
@@ -61,7 +81,7 @@ getCdpClient(toolContext)
61
81
  v
62
82
  ScopedCdpClient.send(method, params)
63
83
  |
64
- +-- Try candidate 1 (extension)
84
+ +-- Try candidate 1 (extension / macOS host proxy)
65
85
  | transport_error? --> failover to candidate 2
66
86
  | cdp_error? --> propagate immediately (no failover)
67
87
  | success? --> sticky for remainder of invocation
@@ -76,11 +96,11 @@ ScopedCdpClient.send(method, params)
76
96
 
77
97
  ## Backend Precedence (macOS)
78
98
 
79
- | Priority | Backend | When selected | Failover trigger |
80
- | -------- | ------------------ | ---------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
81
- | 1 | Extension | `hostBrowserProxy` present and `isAvailable()` is `true` | Transport error (WebSocket disconnected, send failed) |
82
- | 2 | cdp-inspect | Config `enabled: true`, OR macOS + `desktopAuto.enabled` (default) + cooldown not active | Transport error (endpoint unreachable, WS connect failure). Records cooldown on failure. |
83
- | 3 | Local (Playwright) | Always present as final fallback | Errors propagate to the tool |
99
+ | Priority | Backend | When selected | Transport | Failover trigger |
100
+ | -------- | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------- | ---------------------------------------------------------------------------------------- |
101
+ | 1 | Extension / macOS host proxy | `hostBrowserProxy` present and `isAvailable()` is `true`. On macOS, the proxy is always provisioned (SSE sender when no extension, registry-routed when extension is connected) | WS (registry) or SSE (hub) | Transport error (WebSocket disconnected, SSE send failed) |
102
+ | 2 | cdp-inspect | Config `enabled: true`, OR macOS + `desktopAuto.enabled` (default) + cooldown not active | Direct CDP WebSocket | Transport error (endpoint unreachable, WS connect failure). Records cooldown on failure. |
103
+ | 3 | Local (Playwright) | Always present as final fallback | In-process CDP | Errors propagate to the tool |
84
104
 
85
105
  After the first successful CDP command on any backend, that backend becomes **sticky** for the remainder of the tool invocation.
86
106
 
@@ -114,6 +134,29 @@ When cdp-inspect fails with a transport error during a desktop-auto attempt:
114
134
  - No `browserManager` launch log (Playwright not started).
115
135
  - Extension WebSocket receives `host_browser_request` frames.
116
136
 
137
+ ### Scenario 1b: macOS Host Browser Proxy (No Extension)
138
+
139
+ **Setup:**
140
+
141
+ 1. macOS desktop client is running and connected to the assistant via SSE.
142
+ 2. No browser extension is installed or connected.
143
+ 3. Chrome is running on the desktop machine.
144
+
145
+ **Test:**
146
+
147
+ 1. Send a message that triggers browser automation (e.g. "navigate to example.com and take a screenshot").
148
+ 2. Observe the assistant drives the user's Chrome session via the macOS host browser proxy (the desktop client receives `host_browser_request` frames over SSE and executes CDP commands locally).
149
+
150
+ **Expected telemetry/log signals:**
151
+
152
+ - `cdp-factory` log: `CDP factory: built candidate list` with `candidates: [{kind: "extension", ...}, {kind: "cdp-inspect", ...}, {kind: "local", ...}]`
153
+ - `cdp-factory` log: `CDP factory: candidate succeeded, backend is now sticky` with `candidateKind: "extension"`
154
+ - No `browserManager` launch log (Playwright not started).
155
+ - SSE event stream delivers `host_browser_request` frames (not WebSocket relay).
156
+ - `browser_status` output shows `transport: "macos-sse"` in the extension mode details.
157
+
158
+ **Difference from Scenario 1:** The transport is SSE-based (`assistantEventHub`), not the `/v1/browser-relay` WebSocket. The `hostBrowserSenderOverride` is `undefined` because no registry entry exists. The proxy is provisioned because macOS natively supports `host_browser`.
159
+
117
160
  ### Scenario 2: Extension Absent + cdp-inspect Enabled
118
161
 
119
162
  **Setup:**