@vellumai/assistant 0.6.3 → 0.6.4

package/docs/browser-use-architecture-phase2.md

@@ -0,0 +1,174 @@
+# Browser Use Architecture — Phase 2
+
+## Overview
+
+Phase 2 of browser automation introduced a three-tier backend selection chain for macOS-originated turns, enabling the assistant to prefer the user's real Chrome session (via the paired extension) over a sandboxed Playwright instance. When the extension is unavailable, the system falls back through cdp-inspect (direct Chrome DevTools Protocol attach) before resorting to the local Playwright browser.
+
+This document describes the runtime architecture, backend precedence rules, and the manual QA playbook for verifying correct backend selection.
+
+## Component Inventory
+
+| Component                   | Location                                           | Role                                                                                                                                                                                   |
+| --------------------------- | -------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| **ChromeExtensionRegistry** | `runtime/chrome-extension-registry.ts`             | Tracks active extension WebSocket connections keyed by `(guardianId, clientInstanceId)`. Populated on WS `open`, drained on WS `close`.                                                |
+| **HostBrowserProxy**        | `daemon/host-browser-proxy.ts`                     | Per-conversation proxy that dispatches `host_browser_request` frames and awaits `host_browser_result` responses.                                                                       |
+| **CDP Factory**             | `tools/browser/cdp-client/factory.ts`              | Builds the ordered candidate list and returns a `ScopedCdpClient` with per-invocation failover.                                                                                        |
+| **BrowserSessionManager**   | `browser-session/manager.ts`                       | Routes CDP commands through the selected backend with session tracking.                                                                                                                |
+| **CdpInspectClient**        | `tools/browser/cdp-client/cdp-inspect-client.ts`   | Connects to a host Chrome instance via its remote-debugging WebSocket endpoint.                                                                                                        |
+| **LocalCdpClient**          | `tools/browser/cdp-client/local-cdp-client.ts`     | Drives Playwright's CDPSession against the sacrificial-profile browser.                                                                                                                |
+| **ExtensionCdpClient**      | `tools/browser/cdp-client/extension-cdp-client.ts` | Routes CDP commands through the HostBrowserProxy to the user's real Chrome.                                                                                                            |
+| **conversation-routes.ts**  | `runtime/routes/conversation-routes.ts`            | Wires `resolveHostBrowserSender()` to set `hostBrowserSenderOverride` when the extension is connected. Sets `turnInterfaceContext` from the `interface` field on the incoming message. |
+| **Desktop-auto config**     | `config/schemas/host-browser.ts`                   | `desktopAuto.enabled` (default `true`) and `desktopAuto.cooldownMs` (default 30s) control automatic cdp-inspect on macOS.                                                              |
+
+## Wire Diagram
+
+```
+macOS app (user message)
+    |
+    v
+POST /v1/messages  { interface: "macos", ... }
+    |
+    v
+conversation-routes.ts
+    |-- setTurnInterfaceContext({ userMessageInterface: "macos", ... })
+    |-- resolveHostBrowserSender()
+    |       |
+    |       +-- ChromeExtensionRegistry.get(guardianId)
+    |               |
+    |               +-- entry found? --> registrySender (WS to extension)
+    |               |                    hostBrowserSenderOverride = registrySender
+    |               |                    provision HostBrowserProxy
+    |               |
+    |               +-- entry not found? --> SSE hub sender (default)
+    |                                        hostBrowserSenderOverride = undefined
+    |
+    v
+Agent loop invokes browser tool
+    |
+    v
+getCdpClient(toolContext)
+    |-- toolContext.hostBrowserProxy set?
+    |       AND hostBrowserProxy.isAvailable()?
+    |       --> candidate: extension (priority 1)
+    |
+    |-- transportInterface === "macos"
+    |       AND desktopAuto.enabled?
+    |       AND cooldown NOT active?
+    |       --> candidate: cdp-inspect (priority 2)
+    |
+    |-- always --> candidate: local (priority 3)
+    |
+    v
+ScopedCdpClient.send(method, params)
+    |
+    +-- Try candidate 1 (extension)
+    |       transport_error? --> failover to candidate 2
+    |       cdp_error? --> propagate immediately (no failover)
+    |       success? --> sticky for remainder of invocation
+    |
+    +-- Try candidate 2 (cdp-inspect)
+    |       transport_error? --> record cooldown, failover to candidate 3
+    |       success? --> sticky
+    |
+    +-- Try candidate 3 (local)
+            last resort -- errors propagate
+```
+
+## Backend Precedence (macOS)
+
+| Priority | Backend            | When selected                                                                            | Failover trigger                                                                         |
+| -------- | ------------------ | ---------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
+| 1        | Extension          | `hostBrowserProxy` present and `isAvailable()` is `true`                                 | Transport error (WebSocket disconnected, send failed)                                    |
+| 2        | cdp-inspect        | Config `enabled: true`, OR macOS + `desktopAuto.enabled` (default) + cooldown not active | Transport error (endpoint unreachable, WS connect failure). Records cooldown on failure. |
+| 3        | Local (Playwright) | Always present as final fallback                                                         | Errors propagate to the tool                                                             |
+
+After the first successful CDP command on any backend, that backend becomes **sticky** for the remainder of the tool invocation.
+
+## Desktop-auto cdp-inspect Cooldown
+
+When cdp-inspect fails with a transport error during a desktop-auto attempt:
+
+1. The factory records `_desktopAutoCooldownSince = Date.now()`.
+2. Subsequent `buildCandidateList()` calls skip cdp-inspect while `Date.now() - cooldownSince < cooldownMs`.
+3. Default cooldown is 30 seconds (`desktopAuto.cooldownMs`).
+4. Cooldown only applies to desktop-auto candidates (reason starts with `"desktopAuto:"`). Explicitly configured cdp-inspect is never suppressed.
+
+## Manual QA Checklist
+
+### Scenario 1: Extension Connected (macOS)
+
+**Setup:**
+
+1. Pair the browser extension (chrome extension installed, `assistant pair browser-extension` completed or cloud pairing via platform).
+2. Open the macOS app and verify the extension WebSocket is connected (check runtime logs for `browser-relay: registered connection`).
+
+**Test:**
+
+1. Send a message that triggers browser automation (e.g. "navigate to example.com and take a screenshot").
+2. Observe the assistant drives the user's real Chrome session (visible browser activity in the user's Chrome, not a separate Playwright window).
+
+**Expected telemetry/log signals:**
+
+- `cdp-factory` log: `CDP factory: built candidate list` with `candidates: [{kind: "extension", ...}, {kind: "cdp-inspect", ...}, {kind: "local", ...}]`
+- `cdp-factory` log: `CDP factory: candidate succeeded, backend is now sticky` with `candidateKind: "extension"`
+- No `browserManager` launch log (Playwright not started).
+- Extension WebSocket receives `host_browser_request` frames.
+
+### Scenario 2: Extension Absent + cdp-inspect Enabled
+
+**Setup:**
+
+1. No browser extension connected (or extension not installed).
+2. Launch Chrome with `--remote-debugging-port=9222`.
+3. Optionally set `hostBrowser.cdpInspect.enabled: true` in config (or rely on `desktopAuto.enabled: true` default for macOS).
+
+**Test:**
+
+1. Send a message that triggers browser automation.
+2. Observe the assistant attaches to the existing Chrome via CDP (commands execute in the already-running Chrome, not a new Playwright window).
+
+**Expected telemetry/log signals:**
+
+- `cdp-factory` log: `CDP factory: built candidate list` with `candidates: [{kind: "cdp-inspect", ...}, {kind: "local", ...}]`
+- `cdp-factory` log: `CDP factory: candidate succeeded, backend is now sticky` with `candidateKind: "cdp-inspect"`
+- No `browserManager` launch log (Playwright not started).
+- cdp-inspect discovery log: successful WebSocket connection to `ws://localhost:9222/...`.
+
+### Scenario 3: Extension Absent + cdp-inspect Disabled/Unavailable
+
+**Setup:**
+
+1. No browser extension connected.
+2. Chrome NOT launched with `--remote-debugging-port` (the common default).
+3. `desktopAuto.enabled: true` (default).
+
+**Test:**
+
+1. Send a message that triggers browser automation.
+2. Observe the assistant opens a Playwright-managed Chromium window (sacrificial profile).
+
+**Expected telemetry/log signals:**
+
+- `cdp-factory` log: `CDP factory: built candidate list` with `candidates: [{kind: "cdp-inspect", reason: "desktopAuto: ..."}, {kind: "local", ...}]`
+- `cdp-factory` log: `CDP factory: transport-level failure, failing over to next candidate` with `candidateKind: "cdp-inspect"`
+- `cdp-factory` log: `CDP factory: recording desktop-auto cdp-inspect cooldown after transport failure`
+- `cdp-factory` log: `CDP factory: candidate succeeded, backend is now sticky` with `candidateKind: "local"`
+- `browserManager` launch log visible (Playwright starting).
+- Subsequent turns within 30 seconds: `cdp-factory` log shows `desktop-auto cdp-inspect skipped (cooldown active)` and candidates are `[{kind: "local"}]` only.
+
+### Verifying Which Backend Executed
+
+In all scenarios, the definitive signal is the `cdp-factory` structured log:
+
+```
+CDP factory: candidate succeeded, backend is now sticky
+  candidateKind: "extension" | "cdp-inspect" | "local"
+  conversationId: "<id>"
+  method: "<first CDP method called>"
+```
+
+Filter runtime logs with:
+
+```bash
+grep "cdp-factory" ~/.vellum/workspace/data/logs/vellum.log
+```

package/docs/stt-provider-onboarding.md

@@ -0,0 +1,120 @@
+# STT Provider Onboarding Checklist
+
+Step-by-step guide for adding a new speech-to-text provider to the assistant. Follow each section in order; the parity tests (step 7) will fail CI if any side is out of sync.
+
+## 1. Daemon provider catalog entry
+
+**File:** `src/providers/speech-to-text/provider-catalog.ts`
+
+Add a new entry to the `CATALOG` map with:
+
+- `id` — a unique `SttProviderId` string (e.g. `"google-gemini"`).
+- `credentialProvider` — the credential-store key name used by `getProviderKeyAsync` to retrieve the API key. If the provider shares an API key with another service (e.g. `openai-whisper` shares the `"openai"` key, or `google-gemini` shares the `"gemini"` key), reuse that name; otherwise use the provider's own name (e.g. `"deepgram"` maps to `"deepgram"`).
+- `supportedBoundaries` — the set of `SttBoundaryId` values the provider supports. Valid values are `"daemon-batch"` (post-recording transcription) and `"daemon-streaming"` (real-time streaming transcription during conversation).
+- `conversationStreamingMode` — how the provider handles streaming transcription in conversation mode: `"realtime-ws"` (provider supports real-time streaming natively via WebSocket), `"incremental-batch"` (streaming emulated via throttled polling), or `"none"` (no streaming support). Required for all providers.
+- `telephonyMode` — how the provider participates in real-time telephony STT: `"realtime-ws"`, `"batch-only"`, or `"none"`.
+- `telephonyRouting` — telephony routing metadata that drives Twilio call setup strategy selection. Declare `strategyKind` as `"conversation-relay-native"` or `"media-stream-custom"`. For native providers, include `twilioNativeMapping` with the Twilio `provider` name and `defaultSpeechModel`.
+
+## 2. Type-system registration
+
+**File:** `src/stt/types.ts`
+
+- Append the new provider ID to the `SttProviderId` union type.
+
+This ensures the exhaustive switch in `daemon-batch-transcriber.ts` produces a compile error until the adapter is wired.
+
+## 3. Config schema touchpoints
+
+**File:** `src/config/schemas/stt.ts`
+
+- Append the new provider ID string to the `VALID_STT_PROVIDERS` tuple.
+
+The `services.stt.providers` map uses a sparse `z.record(z.string(), ...)` schema, so adding a new provider does **not** require a workspace migration to seed a `services.stt.providers.<id>` entry. Users only need to set `services.stt.provider` to the new ID and supply credentials.
+
+## 4. Adapter wiring
+
+**File:** `src/stt/daemon-batch-transcriber.ts`
+
+1. Create a new `BatchTranscriber` implementation class (e.g. `GoogleGeminiBatchTranscriber`) alongside `WhisperBatchTranscriber` and `DeepgramBatchTranscriber`.
+2. Implement the `transcribe(request)` method using a lazy-imported provider module (follow the pattern in the existing adapters).
+3. Add a `case` branch in `createDaemonBatchTranscriber()` for the new `SttProviderId`. The exhaustive `never` check at the bottom of the switch ensures a compile error if this step is skipped.
+
+If the provider needs a new REST client module, add it under `src/providers/speech-to-text/` following the pattern of `openai-whisper.ts`, `deepgram.ts`, and `google-gemini.ts`.
+
+## 5. Credential plumbing
+
+**File:** `src/providers/provider-secret-catalog.ts`
+
+If the new provider introduces a credential-store key that is not already present in `LLM_AND_SEARCH_API_KEY_PROVIDERS`, it is automatically included via `sttApiKeyProviderNames()` which reads from the STT provider catalog. Verify this by checking that `API_KEY_PROVIDERS` includes the new credential name at runtime.
+
+If the new provider **shares** an existing credential name (e.g. reuses `"openai"`), the deduplication logic in `sttApiKeyProviderNames()` handles it — no changes needed.
+
+## 6. Client catalog entry
+
+**File:** `meta/stt-provider-catalog.json`
+
+Add a new entry to the `providers` array with the following fields:
+
+| Field                       | Description                                                                                                                           |
+| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
+| `id`                        | Must match the `SttProviderId` used in step 1.                                                                                        |
+| `displayName`               | Human-readable name shown in client settings UI.                                                                                      |
+| `subtitle`                  | Short description displayed below the provider selector.                                                                              |
+| `setupMode`                 | `"api-key"` (inline key field) or `"cli"` (instructions-only).                                                                        |
+| `setupHint`                 | Brief guidance shown during setup.                                                                                                    |
+| `apiKeyProviderName`        | Must match the `credentialProvider` value from the daemon catalog entry.                                                              |
+| `conversationStreamingMode` | Must match the `conversationStreamingMode` value from the daemon catalog entry (`"realtime-ws"`, `"incremental-batch"`, or `"none"`). |
+
+**Naming/mapping examples:**
+
+| Provider ID      | `credentialProvider` / `apiKeyProviderName` | Key ownership |
+| ---------------- | ------------------------------------------- | ------------- |
+| `openai-whisper` | `openai`                                    | shared        |
+| `deepgram`       | `deepgram`                                  | exclusive     |
+| `google-gemini`  | `gemini`                                    | shared        |
+
+When the provider ID differs from the credential provider name (e.g. `google-gemini` maps to `gemini`), the key is **shared** with other services that use the same credential. The `sttKeyIsExclusive` / `sttKeyIsShared` helpers in the macOS settings layer derive this automatically from the catalog.
+
+Insertion order in the JSON array must match the daemon catalog insertion order.
+
+### Client-side code
+
+**File:** `clients/shared/Utilities/STTProviderRegistry.swift`
+
+Add a matching fallback entry in `fallbackRegistry` with the same `id`, `displayName`, `subtitle`, `setupMode`, `setupHint`, `apiKeyProviderName`, and `conversationStreamingMode` as the JSON catalog entry. The fallback keeps client startup resilient when the bundled JSON is missing.
+
+### macOS settings key behavior
+
+**File:** `clients/macos/vellum-assistant/Features/Settings/SettingsStore.swift`
+
+The `sttKeyIsExclusive(for:)` / `sttKeyIsShared(for:)` helpers derive shared-vs-exclusive key behavior from the catalog automatically: if `apiKeyProviderName == id`, the key is exclusive; otherwise it is shared. No new conditionals are needed unless the provider has a non-standard key-ownership model.
+
+## 7. Parity tests
+
+**File:** `src/__tests__/stt-catalog-parity.test.ts`
+
+The existing parity test suite enforces that:
+
+- Daemon and client catalog provider IDs are identical and in the same order.
+- Each entry's `apiKeyProviderName` matches the daemon's `credentialProvider`.
+- The client catalog has all required fields populated.
+
+Run the test after completing steps 1-6:
+
+```bash
+cd assistant && bun test src/__tests__/stt-catalog-parity.test.ts
+```
+
+If any assertion fails, the error message identifies which side is out of sync and what to fix.
+
+## 8. Verify unified STT architecture
+
+`services.stt.provider` is the single source of truth for all STT routing, including telephony. There is no separate telephony STT config path.
+
+Before submitting the PR, verify that:
+
+1. **No stale config references** — grep for any references to a separate telephony transcription config. The telephony routing module (`src/calls/telephony-stt-routing.ts`) reads `services.stt.provider` and maps it to the appropriate Twilio strategy (either `conversation-relay-native` for providers Twilio supports natively, or `media-stream-custom` for server-side transcription).
+
+2. **Provider catalog `telephonyRouting` metadata** — the new provider's catalog entry (step 1) includes a `telephonyRouting` object that is the single source of truth for strategy selection in `telephony-stt-routing.ts`. This object declares the `strategyKind` (`"conversation-relay-native"` or `"media-stream-custom"`) and, for native providers, provides a `twilioNativeMapping` with the Twilio `provider` name and `defaultSpeechModel`. The routing module contains no hardcoded provider-to-Twilio maps — it reads these values directly from the catalog.
+
+3. **No duplicate wiring** — a provider should appear only once in `services.stt`. The telephony routing layer consumes the same provider ID; there is no second registration step for telephony.

package/knip.json

@@ -6,9 +6,19 @@
     "src/daemon/main.ts",
     "src/messaging/providers/*/types.ts",
     "src/messaging/providers/*/client.ts",
-    "src/memory/graph/inspect.ts"
+    "src/memory/graph/inspect.ts",
+    "../skills/meet-join/**/*.test.ts",
+    "../skills/meet-join/**/__tests__/**/*.ts",
+    "!../skills/meet-join/bot/**",
+    "../skills/meet-join/contracts/**/__tests__/**/*.ts"
+  ],
+  "project": [
+    "src/**/*.ts",
+    "src/**/*.tsx",
+    "scripts/**/*.ts",
+    "../skills/meet-join/**/*.ts",
+    "!../skills/meet-join/bot/**"
   ],
-  "project": ["src/**/*.ts", "src/**/*.tsx", "scripts/**/*.ts"],
   "ignoreDependencies": [
     "@vellumai/ces-contracts",
     "@vellumai/credential-storage",

package/node_modules/@vellumai/ces-contracts/bun.lock

@@ -5,22 +5,24 @@
     "": {
       "name": "@vellumai/ces-contracts",
       "dependencies": {
-        "zod": "^4.3.6",
+        "zod": "4.3.6",
       },
       "devDependencies": {
-        "@types/bun": "^1.2.4",
-        "typescript": "^5.7.3",
+        "@types/bun": "1.2.4",
+        "typescript": "5.7.3",
       },
     },
   },
   "packages": {
-    "@types/bun": ["@types/bun@1.3.10", "", { "dependencies": { "bun-types": "1.3.10" } }, "sha512-0+rlrUrOrTSskibryHbvQkDOWRJwJZqZlxrUs1u4oOoTln8+WIXBPmAuCF35SWB2z4Zl3E84Nl/D0P7803nigQ=="],
+    "@types/bun": ["@types/bun@1.2.4", "", { "dependencies": { "bun-types": "1.2.4" } }, "sha512-QtuV5OMR8/rdKJs213iwXDpfVvnskPXY/S0ZiFbsTjQZycuqPbMW8Gf/XhLfwE5njW8sxI2WjISURXPlHypMFA=="],
 
     "@types/node": ["@types/node@25.5.0", "", { "dependencies": { "undici-types": "~7.18.0" } }, "sha512-jp2P3tQMSxWugkCUKLRPVUpGaL5MVFwF8RDuSRztfwgN1wmqJeMSbKlnEtQqU8UrhTmzEmZdu2I6v2dpp7XIxw=="],
 
-    "bun-types": ["bun-types@1.3.10", "", { "dependencies": { "@types/node": "*" } }, "sha512-tcpfCCl6XWo6nCVnpcVrxQ+9AYN1iqMIzgrSKYMB/fjLtV2eyAVEg7AxQJuCq/26R6HpKWykQXuSOq/21RYcbg=="],
+    "@types/ws": ["@types/ws@8.5.14", "", { "dependencies": { "@types/node": "*" } }, "sha512-bd/YFLW+URhBzMXurx7lWByOu+xzU9+kb3RboOteXYDfW+tr+JZa99OyNmPINEGB/ahzKrEuc8rcv4gnpJmxTw=="],
 
-    "typescript": ["typescript@5.9.3", "", { "bin": { "tsc": "bin/tsc", "tsserver": "bin/tsserver" } }, "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw=="],
+    "bun-types": ["bun-types@1.2.4", "", { "dependencies": { "@types/node": "*", "@types/ws": "~8.5.10" } }, "sha512-nDPymR207ZZEoWD4AavvEaa/KZe/qlrbMSchqpQwovPZCKc7pwMoENjEtHgMKaAjJhy+x6vfqSBA1QU3bJgs0Q=="],
+
+    "typescript": ["typescript@5.7.3", "", { "bin": { "tsc": "bin/tsc", "tsserver": "bin/tsserver" } }, "sha512-84MVSjMEHP+FQRPy3pX9sTVV/INIex71s9TL2Gm5FG/WG1SqXeKyZ0k7/blY/4FdOzI12CBy1vGc4og/eus0fw=="],
 
     "undici-types": ["undici-types@7.18.2", "", {}, "sha512-AsuCzffGHJybSaRrmr5eHr81mwJU3kjw6M+uprWvCXiNeN9SOGwQ3Jn8jb8m3Z6izVgknn1R0FTCEAP2QrLY/w=="],
 

package/node_modules/@vellumai/ces-contracts/package.json

@@ -16,10 +16,10 @@
     "test": "bun test src/"
   },
   "dependencies": {
-    "zod": "^4.3.6"
+    "zod": "4.3.6"
   },
   "devDependencies": {
-    "@types/bun": "^1.2.4",
-    "typescript": "^5.7.3"
+    "@types/bun": "1.2.4",
+    "typescript": "5.7.3"
   }
 }