@vellumai/assistant 0.11.6 → 0.11.7-staging.1

package/ARCHITECTURE.md

@@ -84,7 +84,7 @@ All HTTP API requests use a single `Authorization: Bearer <jwt>` header for auth
 
 **Identity lifecycle:**
 
-1. **Bootstrap (loopback-only, macOS)** — On first launch, the macOS client calls `POST /v1/guardian/init` with `{ platform, deviceId }`. The endpoint is loopback-only and mints a JWT access token + refresh token pair. Returns `{ guardianPrincipalId, accessToken, accessTokenExpiresAt, refreshToken, refreshTokenExpiresAt, refreshAfter, isNew }`. The CLI obtains its bearer token during `hatch` and does not perform a separate bootstrap step.
+1. **Bootstrap (loopback-only, desktop)**: On first launch, the desktop client calls `POST /v1/guardian/init` with `{ platform, deviceId }`. The endpoint is loopback-only and mints a JWT access token + refresh token pair. Returns `{ guardianPrincipalId, accessToken, accessTokenExpiresAt, refreshToken, refreshTokenExpiresAt, refreshAfter, isNew }`. The CLI obtains its bearer token during `hatch` and does not perform a separate bootstrap step.
 
 2. **Refresh** — `POST /v1/guardian/refresh` accepts `{ refreshToken }` and returns a new access/refresh token pair. Single-use rotation with replay detection and family-based revocation.
 
@@ -577,17 +577,29 @@ All guardian decisions for voice access requests flow through:
 
 Audio-to-text conversion occurs in six distinct runtime boundaries, each with its own provider model and adapter layer. The `services.stt` config block is the single source of truth for STT provider selection across assistant, client, live voice, and telephony boundaries.
 
+**Per-consumer roles:** `services.stt.provider` (plus that provider's `services.stt.providers.<id>.model` family) is the global selection. `services.stt.roles.<role>` overrides it for one consumer, naming a `{provider, model?}` pair; an unset role falls back to the global. Roles exist because capability is per boundary while the global is not: the Flux family streams but has no batch endpoint, so a global set to it breaks file transcription and telephony at the same moment it improves live voice. A role names the consumer rather than the boundary, because one telephony call resolves a streaming transcriber and a batch fallback within the same session.
+
+| Role        | Consumers                                                                                                                       | Boundaries required           |
+| ----------- | ------------------------------------------------------------------------------------------------------------------------------- | ----------------------------- |
+| `liveVoice` | `src/live-voice/live-voice-session.ts`, `src/live-voice/live-voice-credential-preflight.ts`                                     | streaming                     |
+| `telephony` | `src/calls/media-stream-stt-session.ts` (streaming dial and batch fallback)                                                     | streaming + batch + telephony |
+| `dictation` | `/v1/stt/stream` (`src/runtime/http-server.ts`), `POST /v1/stt/transcribe`, `src/plugin-api/transcription-session.ts`           | streaming + batch             |
+| `watch`     | `/v1/watch/stream` (`src/runtime/routes/watch-routes.ts`)                                                                       | streaming                     |
+| `batch`     | inbound audio (`routes/inbound-stages/transcribe-audio.ts`), `POST /v1/stt/transcribe-file`, media-processing/transcribe skills | batch                         |
+
+Roles are defined in `src/stt/roles.ts`: `STT_ROLE_REQUIREMENTS` declares each role's boundaries, `sttSelectionForRole()`/`sttCatalogKeyForRole()` perform the lookup, and `sttRoleCapabilityGap()` states why a pair cannot serve a role. Selection is resolved inside `resolveBatchTranscriber()`/`resolveStreamingTranscriber()`, so a caller passes `role` and nothing else. Capability is checked against the **resolved catalog row** (`sttCatalogKeyFor(provider, model)`), not the bare provider id, since `deepgram` batches and `deepgram` running `flux` does not. Validation is fail-closed: `services.stt.roles` rejects an incapable pair at config parse time rather than substituting a working provider at dial time. Telephony and live-voice preflight, the dictation session label, and the telephony language pin all read their own role, so what preflight judges is what the session dials.
+
 **Provider catalog model:** The daemon's canonical provider catalog (`src/providers/speech-to-text/provider-catalog.ts`) is the single source of truth for all STT provider metadata — credential mappings, supported boundaries, telephony mode, conversation streaming mode, and client-facing display metadata (names, hints, setup mode, credentials guide). Native clients fetch provider metadata at launch via `GET /v1/stt/providers`. To add a new provider, follow the checklist in `docs/stt-provider-onboarding.md`.
 
 **Boundary overview:**
 
-| Boundary                     | Runtime                                             | Provider (current)                           | Adapter module                                                                                                                                                                                                                                             | Caller                                                  |
-| ---------------------------- | --------------------------------------------------- | -------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------- |
-| **Telephony (media-stream)** | Daemon process (Twilio Media Streams WebSocket)     | Configured STT provider (via `services.stt`) | `src/calls/media-stream-stt-session.ts`, `src/providers/speech-to-text/resolve.ts`, streaming provider adapters                                                                                                                                            | `src/calls/media-stream-server.ts`                      |
-| **Daemon batch**             | Daemon process (REST API to provider)               | Configured STT provider (via `services.stt`) | `src/stt/daemon-batch-transcriber.ts`                                                                                                                                                                                                                      | `src/runtime/routes/inbound-stages/transcribe-audio.ts` |
-| **Conversation streaming**   | Daemon process (WebSocket-based)                    | Configured STT provider (via `services.stt`) | `src/stt/stt-stream-session.ts`, `src/providers/speech-to-text/deepgram-realtime.ts`, `src/providers/speech-to-text/google-gemini-live-stream.ts`, `src/providers/speech-to-text/openai-whisper-stream.ts`, `src/providers/speech-to-text/xai-realtime.ts` | Web/Electron dictation client via gateway WS proxy      |
-| **Live voice channel**       | Assistant process (gateway-authenticated WebSocket) | Configured STT provider (via `services.stt`) | `src/runtime/http-server.ts`, `src/live-voice/live-voice-session-manager.ts`, `src/live-voice/live-voice-session.ts`, `src/providers/speech-to-text/resolve.ts`, streaming provider adapters                                                               | Web/Electron live voice client via `/v1/live-voice`     |
-| **Client service-first**     | Web/Electron via gateway → daemon                   | Configured STT provider (via `services.stt`) | `src/runtime/routes/stt-routes.ts`                                                                                                                                                                                                                         | Web/Electron dictation and voice clients                |
+| Boundary                     | Runtime                                             | Provider (current)                                                                      | Adapter module                                                                                                                                                                                                                                             | Caller                                                  |
+| ---------------------------- | --------------------------------------------------- | --------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------- |
+| **Telephony (media-stream)** | Daemon process (Twilio Media Streams WebSocket)     | `telephony` role, else `services.stt.provider`                                          | `src/calls/media-stream-stt-session.ts`, `src/providers/speech-to-text/resolve.ts`, streaming provider adapters                                                                                                                                            | `src/calls/media-stream-server.ts`                      |
+| **Daemon batch**             | Daemon process (REST API to provider)               | `batch` role, else `services.stt.provider`                                              | `src/stt/daemon-batch-transcriber.ts`                                                                                                                                                                                                                      | `src/runtime/routes/inbound-stages/transcribe-audio.ts` |
+| **Conversation streaming**   | Daemon process (WebSocket-based)                    | `dictation` role, else `services.stt.provider`                                          | `src/stt/stt-stream-session.ts`, `src/providers/speech-to-text/deepgram-realtime.ts`, `src/providers/speech-to-text/google-gemini-live-stream.ts`, `src/providers/speech-to-text/openai-whisper-stream.ts`, `src/providers/speech-to-text/xai-realtime.ts` | Web/Electron dictation client via gateway WS proxy      |
+| **Live voice channel**       | Assistant process (gateway-authenticated WebSocket) | `liveVoice` role, else `services.stt.provider`                                          | `src/runtime/http-server.ts`, `src/live-voice/live-voice-session-manager.ts`, `src/live-voice/live-voice-session.ts`, `src/providers/speech-to-text/resolve.ts`, streaming provider adapters                                                               | Web/Electron live voice client via `/v1/live-voice`     |
+| **Client service-first**     | Web/Electron via gateway → daemon                   | `dictation` (transcribe) / `batch` (transcribe-file) role, else `services.stt.provider` | `src/runtime/routes/stt-routes.ts`                                                                                                                                                                                                                         | Web/Electron dictation and voice clients                |
 
 **Telephony boundary (media-stream):**
 
@@ -595,8 +607,8 @@ Every phone call connects over Twilio Media Streams: the voice webhook emits `<C
 
 Transcription mode is selected once per session in `media-stream-stt-session.ts`:
 
-- **Streaming** (default) — when `calls.voice.telephonyStreaming` is enabled and the configured provider resolves a streaming transcriber (`resolveStreamingTranscriber()`), inbound audio is decoded (mu-law → PCM16, resampled 8 kHz → 16 kHz) and fed to the provider's realtime adapter. Replies trigger only on utterance-boundary finals (for Deepgram, `speech_final`/`UtteranceEnd`, never mid-sentence `is_final` segments), and barge-in fires from local energy VAD, never from transcriber partials.
-- **Batch fallback** — otherwise the session segments turns with the energy-based `MediaTurnDetector` and transcribes each completed turn via the provider's batch API.
+- **Streaming** (default): when `calls.voice.telephonyStreaming` is enabled and the `telephony` role resolves a streaming transcriber (`resolveStreamingTranscriber({ role: "telephony" })`), inbound audio is decoded (mu-law → PCM16, resampled 8 kHz → 16 kHz) and fed to the provider's realtime adapter. Replies trigger only on utterance-boundary finals (for Deepgram, `speech_final`/`UtteranceEnd`, never mid-sentence `is_final` segments), and barge-in fires from local energy VAD, never from transcriber partials.
+- **Batch fallback**: otherwise the session segments turns with the energy-based `MediaTurnDetector` and transcribes each completed turn via the same role's batch API. Both halves of a call read the `telephony` role, which is why a role names its consumer rather than a boundary.
 
 A credential preflight (`resolveTelephonyCredentialReadiness()` in `src/calls/telephony-credential-preflight.ts`) gates every call: it requires a credentialed, telephony-capable STT provider **and** a media-stream-playable TTS provider (the configured one or a credentialed playable fallback). Inbound calls that fail the preflight receive `<Say>` setup-required copy plus `<Hangup/>` instead of a doomed stream; outbound placement fails before dialing via `preflightVoiceIngress()` with the same user-facing message.
 
@@ -632,7 +644,7 @@ To add a new daemon batch STT provider, follow the full checklist in `docs/stt-p
 
 **Conversation streaming boundary:**
 
-Real-time conversation chat message capture on macOS uses a WebSocket-based streaming STT path. When the configured `services.stt` provider supports conversation streaming (determined by the `conversationStreamingMode` field in the provider catalog), native clients open a WebSocket session through the gateway to the daemon's `/v1/stt/stream` endpoint. The daemon resolves a `StreamingTranscriber` for the configured provider and streams partial/final transcript events back to the client in real time.
+Real-time conversation chat message capture on macOS uses a WebSocket-based streaming STT path. When the provider the `dictation` role resolves to supports conversation streaming (determined by the `conversationStreamingMode` field in the provider catalog), native clients open a WebSocket session through the gateway to the daemon's `/v1/stt/stream` endpoint. The daemon resolves a `StreamingTranscriber` for that provider and streams partial/final transcript events back to the client in real time.
 
 Each provider that advertises a conversation streaming mode in the catalog ships an adapter implementing the `StreamingTranscriber` interface from `src/stt/types.ts`. The catalog (`src/providers/speech-to-text/provider-catalog.ts`) is the source of truth for which providers are streaming-capable; `resolveStreamingTranscriber()` maps each to its adapter:
 
@@ -653,8 +665,8 @@ Each provider that advertises a conversation streaming mode in the catalog ships
 
 **Session lifecycle (daemon side):**
 
-1. Client opens a WebSocket to `/v1/stt/stream` with required query parameter `mimeType` and optional `provider` and `sampleRate`. The `provider` parameter is optional compatibility metadata — the runtime is config-authoritative and always resolves the streaming transcriber from `services.stt.provider`. When a requested provider disagrees with the configured provider, the runtime logs a mismatch warning.
-2. `SttStreamSession` (in `src/stt/stt-stream-session.ts`) resolves a `StreamingTranscriber` via `resolveStreamingTranscriber()` from `src/providers/speech-to-text/resolve.ts`, using the configured provider (not the requested one).
+1. Client opens a WebSocket to `/v1/stt/stream` with required query parameter `mimeType` and optional `provider` and `sampleRate`. The `provider` parameter is optional compatibility metadata: the runtime is config-authoritative and always resolves the streaming transcriber from the `dictation` role (`services.stt.roles.dictation`, else `services.stt.provider`). When a requested provider disagrees with the one that role resolves to, the runtime logs a mismatch warning.
+2. `SttStreamSession` (in `src/stt/stt-stream-session.ts`) resolves a `StreamingTranscriber` via `resolveStreamingTranscriber()` from `src/providers/speech-to-text/resolve.ts`, using the dictation role's provider (not the requested one).
 3. The transcriber's `start()` method opens the provider session.
 4. A `ready` event (with `provider` field) is sent to the client, signaling that audio frames are accepted.
 5. Client sends `audio` frames (binary WebSocket frames or base64-encoded JSON) and a `stop` event when recording ends.
@@ -725,7 +737,7 @@ V1 is local/gateway-scoped. Managed/cloud WebSocket proxy support, cross-region
 
 **Client service-first boundary:**
 
-All product-facing dictation and voice-streaming paths use a service-first STT strategy. Clients record audio, encode it to WAV, and POST it through the gateway to the daemon's `POST /v1/stt/transcribe` endpoint. The daemon resolves the configured STT provider through `resolveBatchTranscriber()` and returns the transcribed text.
+All product-facing dictation and voice-streaming paths use a service-first STT strategy. Clients record audio, encode it to WAV, and POST it through the gateway to the daemon's `POST /v1/stt/transcribe` endpoint. The daemon resolves the `dictation` role's provider through `resolveBatchTranscriber()` and returns the transcribed text. `POST /v1/stt/transcribe-file` resolves the `batch` role the same way.
 
 - The client receives a typed result distinguishing success from `notConfigured`, `serviceUnavailable`, and `error`, so callers can deterministically trigger their fallback path.
 - The gateway proxies the request via assistant-scoped path rewriting: `/v1/assistants/:id/stt/transcribe` is rewritten to `/v1/stt/transcribe` on the daemon.
@@ -735,7 +747,7 @@ The web client implements these flows in `clients/web/src/domains/chat/voice/`:
 
 **Cross-boundary notes:**
 
-- The `services.stt` config block is the single source of truth for STT provider selection across the daemon batch boundary, the conversation streaming boundary, the client service-first boundary, and the telephony boundary. The batch and streaming resolvers (`resolveBatchTranscriber()`, `resolveStreamingTranscriber()`) both read from `services.stt.provider` and resolve credentials through the same catalog; the telephony boundary's media-stream STT session uses these same resolvers — it selects streaming vs batch transcription via `resolveStreamingTranscriber()`/`resolveBatchTranscriber()` under the `calls.voice.telephonyStreaming` flag, gated by `resolveTelephonySttCapability()`, while call placement/TwiML is gated by `resolveTelephonyCredentialReadiness()`. The daemon provider catalog (`src/providers/speech-to-text/provider-catalog.ts`) is the authoritative registry of supported providers. Native clients fetch display metadata via `GET /v1/stt/providers`.
+- The `services.stt` config block is the single source of truth for STT provider selection across the daemon batch boundary, the conversation streaming boundary, the client service-first boundary, and the telephony boundary. The batch and streaming resolvers (`resolveBatchTranscriber()`, `resolveStreamingTranscriber()`) both read it, taking the caller's `role` override (`services.stt.roles.<role>`) when one is set and the global `services.stt.provider` otherwise, and resolve credentials through the same catalog; the telephony boundary's media-stream STT session uses these same resolvers, selecting streaming vs batch transcription via `resolveStreamingTranscriber()`/`resolveBatchTranscriber()` under the `calls.voice.telephonyStreaming` flag, gated by `resolveTelephonySttCapability()`, while call placement/TwiML is gated by `resolveTelephonyCredentialReadiness()`. The daemon provider catalog (`src/providers/speech-to-text/provider-catalog.ts`) is the authoritative registry of supported providers. Native clients fetch display metadata via `GET /v1/stt/providers`.
 - **Spoken language** is a second axis on the same block: `services.stt.language` is read by `resolveStreamingTranscriber()` and `resolveBatchTranscriber()` and forwarded to the adapters that accept one (Deepgram, xAI, and the managed relay, which passes it to Deepgram server-side). Callers may override it per streaming session, but none need to: reading it in the resolvers means every boundary inherits the setting. Gemini and Whisper take no language option and auto-detect natively, so they ignore it. Leaving it unset is **not** auto-detection on Deepgram or the managed relay: Deepgram's default is English, so non-English speech transcribes as English-sounding nonsense. `"multi"` selects nova-3 code-switching across English, Spanish, French, German, Hindi, Russian, Portuguese, Japanese, Italian, and Dutch, the mode to use when a speaker mixes languages inside one utterance. BYOK Deepgram pins `model=nova-3` for **any** explicitly configured language, not just `"multi"`: the curated roster is verified against nova-3 (the default nova-2 supports only a subset, and rejects `"multi"` outright), and `deepgramLanguageOptions()` owns the model+language pairing so every call site inherits the rule; leaving the language unset keeps the adapter default model with no language param. `"multi"` is never forwarded to xAI (it is not a BCP-47 code; xAI auto-handles multilingual audio). The managed relay pins nova-3 and allowlists `language`, so streaming needs no platform-side change; managed **batch** goes through the platform speech proxy, which accepts no language parameter, so forwarding it there is deferred pending a platform-side change.
 - Conversation streaming does not replace the client service-first batch path. When streaming is available, it runs concurrently during recording and provides real-time partials and finals. The batch path remains the fallback for providers that do not support streaming, when streaming fails mid-session, or when streaming produces no final transcript.
 - Credential mapping is catalog-driven: `provider-secret-catalog.ts` derives STT API-key provider names from the daemon catalog via `listCredentialProviderNames()`, deduplicating against the LLM/search provider list. Adding a provider to the catalog automatically includes its credential name in `API_KEY_PROVIDERS`.
@@ -1182,7 +1194,7 @@ The text_qa system prompt includes an action execution hierarchy that guides too
 | **GOOD**        | Headless browser               | `browser_*` (bundled `browser` skill)           | Web automation, form filling, scraping (background)         |
 | **LAST RESORT** | Foreground computer use        | `computer_use_*` (bundled `computer-use` skill) | Only on explicit user request ("go ahead", "take over")     |
 
-Computer-use tools are proxy tools provided by the bundled `computer-use` skill, preactivated via `preactivatedSkillIds` in desktop sessions. Each tool forwards actions to the connected macOS client via `HostCuProxy`, which handles request/resolve proxying, step counting, loop detection, and observation formatting within the unified agent loop. These tools are not core-registered at daemon startup; they exist only through skill projection.
+Computer-use tools are proxy tools provided by the bundled `computer-use` skill, preactivated via `preactivatedSkillIds` in desktop sessions. Each tool forwards actions to a compatible connected desktop client via `HostCuProxy`, which handles request/resolve proxying, step counting, loop detection, and observation formatting within the unified agent loop. Skill projection filters platform-specific actions from unsupported clients. Drag, app launching, and AppleScript remain macOS-only. These tools are not core-registered at daemon startup; they exist only through skill projection.
 
 ### Sandbox Filesystem and Host Access
 
@@ -1397,7 +1409,7 @@ The following capabilities ship as bundled skills in `assistant/src/config/bundl
 | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
 | `browser`       | `browser_navigate`, `browser_snapshot`, `browser_screenshot`, `browser_close`, `browser_click`, `browser_type`, `browser_press_key`, `browser_wait_for`, `browser_extract`, `browser_fill_credential`                                                             | Headless browser automation — web scraping, form filling, interaction (previously core-registered as `headless-browser`; now skill-provided with default allow rules)                                                                                                                                |
 | `gmail`         | Gmail search, archive, send, etc.                                                                                                                                                                                                                                 | Email management via OAuth2 integration                                                                                                                                                                                                                                                              |
-| `computer-use`  | `computer_use_observe`, `computer_use_click`, `computer_use_type_text`, `computer_use_key`, `computer_use_scroll`, `computer_use_drag`, `computer_use_wait`, `computer_use_open_app`, `computer_use_run_applescript`, `computer_use_done`, `computer_use_respond` | Computer-use proxy tools — preactivated via `preactivatedSkillIds` in desktop sessions. Each tool forwards actions to the connected macOS client via `HostCuProxy`, which handles request/resolve proxying, step counting, loop detection, and observation formatting within the unified agent loop. |
+| `computer-use`  | `computer_use_observe`, `computer_use_click`, `computer_use_type_text`, `computer_use_key`, `computer_use_scroll`, `computer_use_drag`, `computer_use_wait`, `computer_use_open_app`, `computer_use_run_applescript`, `computer_use_done`, `computer_use_respond` | Computer-use proxy tools preactivated via `preactivatedSkillIds` in desktop sessions. Each tool forwards actions to a compatible connected desktop host via `HostCuProxy`, which handles request/resolve proxying, step counting, loop detection, and observation formatting in the same agent loop. |
 | `weather`       | `get-weather`                                                                                                                                                                                                                                                     | Fetch current weather data                                                                                                                                                                                                                                                                           |
 | `app-builder`   | `app_create`, `app_delete`, `app_refresh`, `app_generate_icon`                                                                                                                                                                                                    | Dynamic app authoring — create and manage persistent apps; file editing uses generic file tools plus `app_refresh` (activated via `skill_load app-builder`; `app_open` remains a core proxy tool)                                                                                                    |
 | `self-upgrade`  | (instruction-only)                                                                                                                                                                                                                                                | Self-improvement workflow                                                                                                                                                                                                                                                                            |
@@ -1443,7 +1455,7 @@ graph TB
     RESOLVE --> PROVIDER
 ```
 
-**Internal preactivation**: Some bundled skills are preactivated programmatically rather than by model discovery. For example, desktop sessions set `preactivatedSkillIds: ['computer-use']`, causing `projectSkillTools()` to load the 11 `computer_use_*` tool definitions from the bundled skill's `TOOLS.json` on the first turn. These proxy tools forward actions to the connected macOS client via `HostCuProxy`.
+**Internal preactivation**: Some bundled skills are preactivated programmatically rather than by model discovery. For example, desktop sessions set `preactivatedSkillIds: ['computer-use']`, causing `projectSkillTools()` to load the eligible `computer_use_*` tool definitions from the bundled skill's `TOOLS.json` on the first turn. These proxy tools forward actions to a compatible connected desktop client via `HostCuProxy`.
 
 ### Skill Tool Execution
 
@@ -1647,7 +1659,7 @@ graph TB
 
 ### Auto-Approve Threshold
 
-Auto-approve thresholds are **gateway-owned** — they live in the gateway's SQLite database and are read by the assistant via IPC (`get_global_thresholds`, `get_conversation_threshold`). Users control thresholds via the **Settings UI** (Permissions & Privacy tab) or the **per-conversation risk tolerance picker**. When the gateway is unreachable, the assistant defaults to `"none"` (Strict) — fail-closed with no local fallback.
+Auto-approve thresholds are **gateway-owned**: they live in the gateway's SQLite database and are read by the assistant via IPC (`get_global_thresholds`, `get_conversation_threshold`, `get_contact_threshold`). A contact ceiling is written via `set_contact_threshold` (`gateway contacts set-risk-threshold` or `POST /v1/contacts`). Users also control thresholds via the **Settings UI** (Permissions & Privacy tab) or the **per-conversation risk tolerance picker**. When the gateway is unreachable, the assistant defaults to `"none"` (Strict): fail-closed with no local fallback.
 
 | `autoApproveUpTo` | Low-risk tools | Medium-risk tools | High-risk tools |
 | ----------------- | -------------- | ----------------- | --------------- |

package/Dockerfile

@@ -117,6 +117,7 @@ RUN ln -sf /usr/local/bin/bun /usr/local/bin/bunx
 COPY --from=builder /app /app
 
 # Copy source after installing dependencies for better layer caching.
+COPY packages/avatar-manifest /app/packages/avatar-manifest
 COPY packages/ces-client /app/packages/ces-client
 COPY packages/service-contracts /app/packages/service-contracts
 COPY packages/credential-storage /app/packages/credential-storage

package/docs/architecture/security.md

@@ -34,7 +34,7 @@ The order inside `check()`: the memory-retrospective skill-authoring grant, then
 
 ### Auto-Approve Threshold
 
-Thresholds are **gateway-owned**: stored in the gateway's SQLite database, read by the assistant over IPC (`get_global_thresholds`, `get_conversation_threshold`), and set from the Settings UI (Permissions & Privacy) or the per-conversation risk tolerance picker. When the gateway is unreachable the assistant resolves `"none"` (Strict), fail-closed with no local fallback.
+Thresholds are **gateway-owned**: stored in the gateway's SQLite database, read by the assistant over IPC (`get_global_thresholds`, `get_conversation_threshold`, `get_contact_threshold`), and written via `set_contact_threshold` (`gateway contacts set-risk-threshold` or `POST /v1/contacts`) as well as the Settings UI (Permissions & Privacy) or the per-conversation risk tolerance picker. When the gateway is unreachable the assistant resolves `"none"` (Strict), fail-closed with no local fallback.
 
 Gateway defaults per execution context (`gateway/src/ipc/threshold-handlers.ts`): `interactive` (a conversation with a client) `medium`, `autonomous` (background/scheduled) `low`, `headless` `none`. A per-conversation override wins over the global value; for non-guardian actors a channel-permission cell can only lower the effective threshold.
 
@@ -203,7 +203,7 @@ sequenceDiagram
 
 ```mermaid
 graph TB
-    TOOL["Tool (e.g. browser_fill_credential)"] --> BROKER["CredentialBroker.use(service, field, tool, domain)"]
+    TOOL["Tool (e.g. browser_fill_credential)"] --> BROKER["CredentialBroker.browserFill / serverUse"]
     BROKER --> POLICY{"Check policy:<br/>allowedTools + allowedDomains"}
     POLICY -->|denied| REJECT["PolicyDenied error"]
     POLICY -->|allowed| FETCH["getSecureKeyAsync(credential/svc/field)"]
@@ -214,7 +214,7 @@ graph TB
 
 The `allowOneTimeSend` config gate (default: `false`) enables a secondary "Send Once" button in the secret prompt UI. When used:
 
-- The secret value is handed to the `CredentialBroker`, which holds it in memory for the next `consume` or `browserFill` call
+- The secret value is handed to the `CredentialBroker`, which holds it in memory for the next `browserFill` or `serverUse` call
 - The value is **not** persisted to the credential store
 - The broker discards the value after a single use
 - The credentials prompt route output confirms delivery without including the secret value — the value is never returned to the model

package/docs/flux-turn-detection-spike.md

@@ -59,7 +59,7 @@ If it does not, set the Deepgram key the ordinary way (client Settings, Speech-t
 
 ## 2. Enable Flux
 
-> **Setting `services.stt.provider` to `deepgram-flux` turns off batch transcription for the whole workspace, not just live voice.** `services.stt.provider` is the single source of truth for every STT route, and Flux is the only provider in the catalog with no `daemon-batch` boundary. For as long as it is set, these all stop working: voice-message and inbound-attachment transcription, the `transcribe` skill, the `media-processing` skill's audio segments, `POST /v1/stt/transcribe`, and phone-call transcription. Each reports a message naming Flux as the cause rather than failing silently, but they do not fall back to `deepgram` on their own. **Set `services.stt.provider` back to `deepgram` when the spike is over**, and do not run the spike on an assistant that is also taking calls or handling voice messages.
+> **Putting a provider on the `flux` model family turns off batch transcription for the whole workspace, not just live voice.** `services.stt.provider` plus `services.stt.providers.<provider>.model` is the single source of truth for every STT route, and Flux is the only family in the catalog with no `daemon-batch` boundary. For as long as it is set, these all stop working: voice-message and inbound-attachment transcription, the `transcribe` skill, the `media-processing` skill's audio segments, `POST /v1/stt/transcribe`, and phone-call transcription. Each reports a message naming Flux as the cause rather than failing silently, but they do not fall back to `deepgram` on their own. **Set `services.stt.providers.deepgram.model` back to `nova-3` when the spike is over**, and do not run the spike on an assistant that is also taking calls or handling voice messages.
 
 Two keys in `config.json`, which lives at `$VELLUM_WORKSPACE_DIR/config.json` (default `~/.vellum/workspace/config.json`):
 
@@ -67,7 +67,8 @@ Two keys in `config.json`, which lives at `$VELLUM_WORKSPACE_DIR/config.json` (d
 {
   "services": {
     "stt": {
-      "provider": "deepgram-flux"
+      "provider": "deepgram",
+      "providers": { "deepgram": { "model": "flux" } }
     }
   },
   "liveVoice": {
@@ -93,12 +94,12 @@ The rest of `liveVoice.flux` is optional and defaulted (`config/schemas/live-voi
 
 ## 3. Run the A/B
 
-**Flip `turnEnd.enabled` between runs and leave `services.stt.provider` on `deepgram-flux` in both arms.** That holds the STT engine, the model, the socket, and the transcriber lifecycle constant, so the only thing that changes is which signal commits the turn.
+**Flip `turnEnd.enabled` between runs and leave the `flux` model family selected in both arms.** That holds the STT engine, the model, the socket, and the transcriber lifecycle constant, so the only thing that changes is which signal commits the turn.
 
-- **Arm A (control):** `provider: "deepgram-flux"`, `turnEnd.enabled: false`. Flux transcribes; the four turn-detection events are ignored; the local silence boundary and the front-door hold path run exactly as they do today.
-- **Arm B (treatment):** `provider: "deepgram-flux"`, `turnEnd.enabled: true`.
+- **Arm A (control):** `providers.deepgram.model: "flux"`, `turnEnd.enabled: false`. Flux transcribes; the four turn-detection events are ignored; the local silence boundary and the front-door hold path run exactly as they do today.
+- **Arm B (treatment):** `providers.deepgram.model: "flux"`, `turnEnd.enabled: true`.
 
-**Do not A/B by switching the provider between `deepgram` and `deepgram-flux`.** That confounds two changes at once:
+**Do not A/B by switching the model family between `nova-3` and `flux`.** That confounds two changes at once:
 
 1. It swaps the STT model. `deepgram` runs `nova-2` (`DEFAULT_MODEL`, `deepgram-realtime.ts`), so any transcript-quality or first-partial difference lands in your latency numbers.
 2. It swaps the transcriber lifecycle. `deepgram` implements `finalizeUtterance`, so the session adopts it as a persistent stream shared across the whole session (`sharedTranscriber`) and never tears it down between turns. Flux implements no `finalizeUtterance`, so every utterance owns its own `/v2/listen` socket and every release closes it. That per-turn socket churn is inside `roundTripMs` and `totalMs` on the Flux side and absent on the `deepgram` side, so a naive provider-swap A/B attributes it to turn detection. The next section is what it costs and why it cannot be removed.
@@ -230,7 +231,7 @@ Check these if you port the spike anywhere else, because a run that violates eit
 
 ## 8. Falling back
 
-Set `liveVoice.flux.turnEnd.enabled` back to `false` (or delete the key) and restart. That is the whole rollback: the latch goes down, the turn-detection events become no-ops, and the silence-boundary path runs unchanged. You can leave `services.stt.provider` on `deepgram-flux` or move it back to `deepgram`; either is a working configuration.
+Set `liveVoice.flux.turnEnd.enabled` back to `false` (or delete the key) and restart. That is the whole rollback: the latch goes down, the turn-detection events become no-ops, and the silence-boundary path runs unchanged. You can leave the `flux` family selected or move back to `nova-3`; either is a working configuration.
 
 Runtime fallback needs no action. A Flux stream that never emits `turn-end` is caught by the fail-open deadline and the utterance replays onto the silence path, so an outage degrades to today's behavior rather than to a hung turn.
 

package/docs/runbook-trusted-contacts.md

@@ -37,6 +37,8 @@ assistant contacts list --role contact
 assistant contacts list --role guardian
 assistant contacts list --channel-type telegram
 assistant contacts get <contactId>
+gateway contacts list
+gateway contacts get <contactId>
 ```
 
 ### Via HTTP API
@@ -142,6 +144,18 @@ sqlite3 "file:$GW_DB?mode=ro" \
 
 `verification_purpose` is `guardian` for guardian binding flows and `trusted_contact` for the access-request handshake.
 
+## 3b. Contact writes
+
+All write operations on contacts are gateway-owned. Use the gateway CLI, not
+`assistant contacts`. From the host, run the same command via
+`vellum exec --service gateway --`.
+
+```bash
+gateway contacts set-risk-threshold <contactId> --threshold high
+```
+
+`inherit` clears the risk ceiling so the contact follows room and trust-class settings. Stored vocabulary is `none`, `low`, `medium`, or `high`.
+
 ## 4. Force-Revoke a Trusted Contact
 
 ### Via CLI (preferred)

package/docs/stt-provider-onboarding.md

@@ -29,7 +29,7 @@ This ensures the exhaustive switch in `daemon-batch-transcriber.ts` produces a c
 
 - Append the new provider ID string to the `VALID_STT_PROVIDERS` tuple.
 
-The `services.stt.providers` map uses a sparse `z.record(z.string(), ...)` schema, so adding a new provider does **not** require a workspace migration to seed a `services.stt.providers.<id>` entry. Users only need to set `services.stt.provider` to the new ID and supply credentials.
+The `services.stt.providers` map uses a sparse `z.record(z.string(), ...)` schema, so adding a new provider does **not** require a workspace migration to seed a `services.stt.providers.<id>` entry. Users only need to set `services.stt.provider` (or a single consumer's `services.stt.roles.<role>`, see step 7) to the new ID and supply credentials.
 
 **Language handling.** `services.stt.language` is resolved centrally in `resolveStreamingTranscriber()` (and in `resolveBatchTranscriber()` for the daemon-batch boundary), so a new adapter inherits it for free: accept a `language` option in the adapter's constructor and forward it to the provider. If the provider auto-detects natively and has no language parameter (as Gemini and Whisper do), accept nothing and let the resolver's value be ignored; document that choice in the adapter, because "no language param" means auto-detect for some providers and _English_ for others (Deepgram), and that difference is an easy source of silent wrong-language transcription. A provider that supports both batch and streaming must forward the language on **both** paths, not just the streaming one.
 
@@ -84,12 +84,16 @@ Clients derive shared-vs-exclusive key behavior from the catalog automatically:
 
 ## 7. Verify unified STT architecture
 
-`services.stt.provider` is the single source of truth for all STT routing, including telephony. There is no separate telephony STT config path.
+The `services.stt` block is the single source of truth for all STT routing, including telephony. There is no separate telephony STT config path.
+
+Routing has two levels. `services.stt.provider`, plus that provider's `services.stt.providers.<id>.model` family, is the global selection every consumer falls back to. `services.stt.roles.<role>` overrides it for one consumer, naming a `{provider, model?}` pair. The roles are `liveVoice`, `telephony`, `dictation`, `watch` and `batch`; `src/stt/roles.ts` maps each to the boundaries its call sites resolve on, and `ARCHITECTURE.md` lists the consumers per role. A new provider needs no per-role registration: a role may name any provider in the catalog whose row covers the boundaries that role requires.
 
 Before submitting the PR, verify that:
 
-1. **No stale config references** — grep for any references to a separate telephony transcription config. Telephony transcription runs daemon-side over the Twilio media-stream transport (`src/calls/media-stream-stt-session.ts`), which reads `services.stt.provider` like every other boundary.
+1. **No stale config references**: grep for any references to a separate telephony transcription config. Telephony transcription runs daemon-side over the Twilio media-stream transport (`src/calls/media-stream-stt-session.ts`), which resolves through `resolveStreamingTranscriber()`/`resolveBatchTranscriber()` under the `telephony` role like every other boundary resolves under its own.
 
 2. **Provider catalog telephony metadata** — the new provider's catalog entry (step 1) declares `telephonyMode` and `supportedBoundaries`; these are the single source of truth for the telephony capability check (`resolveTelephonySttCapability()`) and for streaming-vs-batch mode selection in the media-stream STT session. No per-provider routing maps exist.
 
-3. **No duplicate wiring** — a provider should appear only once in `services.stt`. The telephony layer consumes the same provider ID; there is no second registration step for telephony.
+3. **Role capability follows the catalog row**: `sttRoleCapabilityGap()` reads `supportedBoundaries` and `telephonyMode` from the row the pair resolves to (`sttCatalogKeyFor(provider, model)`), so a model family that drops a boundary must be its own catalog row. Get the row right and config validation rejects incapable role pairs on its own; nothing per-role needs writing.
+
+4. **No duplicate wiring**: a provider should appear only once in `services.stt`. The telephony layer consumes the same catalog, and a role selects from it; there is no second registration step for telephony or for any role.

package/knip.json

@@ -15,6 +15,7 @@
   "ignore": ["src/config/preloaded-apps/**"],
   "ignoreDependencies": [
     "@microsoft/api-extractor",
+    "@vellumai/avatar-manifest",
     "@vellumai/ces-client",
     "@vellumai/credential-storage",
     "@vellumai/egress-proxy",

package/node_modules/@vellumai/avatar-manifest/package.json

@@ -0,0 +1,18 @@
+{
+  "name": "@vellumai/avatar-manifest",
+  "version": "0.0.1",
+  "private": true,
+  "license": "MIT",
+  "type": "module",
+  "exports": {
+    ".": "./src/index.ts"
+  },
+  "scripts": {
+    "typecheck": "bunx tsc --noEmit",
+    "test": "bun test src/"
+  },
+  "devDependencies": {
+    "@types/bun": "1.3.11",
+    "typescript": "5.9.3"
+  }
+}

package/node_modules/@vellumai/avatar-manifest/src/__tests__/manifest.test.ts

@@ -0,0 +1,173 @@
+import { describe, expect, test } from "bun:test";
+
+import {
+  AVATAR_FIELD_MAX_LENGTH,
+  deriveAvatarFromLegacyFiles,
+  parseAvatarManifest,
+  resolveAvatarFromFiles,
+} from "../manifest.js";
+
+const overlong = "x".repeat(AVATAR_FIELD_MAX_LENGTH + 1);
+
+const traits = { bodyShape: "round", eyeStyle: "dot", color: "#abc" };
+const image = { updatedAt: "2026-01-01T00:00:00.000Z", etag: "abc" };
+
+describe("parseAvatarManifest", () => {
+  test("accepts a character manifest", () => {
+    expect(
+      parseAvatarManifest({ kind: "character", traits, source: "builder" }),
+    ).toEqual({ kind: "character", traits, source: "builder", image: null });
+  });
+
+  test("accepts an image manifest", () => {
+    expect(parseAvatarManifest({ kind: "image", image })).toEqual({
+      kind: "image",
+      traits: null,
+      source: null,
+      image,
+    });
+  });
+
+  test("accepts kind none without a payload", () => {
+    expect(parseAvatarManifest({ kind: "none" })).toEqual({
+      kind: "none",
+      traits: null,
+      source: null,
+      image: null,
+    });
+  });
+
+  test("rejects an overlong trait or image field", () => {
+    expect(
+      parseAvatarManifest({
+        kind: "character",
+        traits: { ...traits, color: overlong },
+      }),
+    ).toBeNull();
+    expect(
+      parseAvatarManifest({
+        kind: "image",
+        image: { ...image, etag: overlong },
+      }),
+    ).toBeNull();
+    expect(
+      parseAvatarManifest({
+        kind: "character",
+        traits: { ...traits, color: "x".repeat(AVATAR_FIELD_MAX_LENGTH) },
+      }),
+    ).not.toBeNull();
+  });
+
+  test("normalizes an unknown source to null", () => {
+    expect(
+      parseAvatarManifest({ kind: "image", image, source: "unknown" }),
+    ).toEqual({ kind: "image", traits: null, source: null, image });
+    expect(parseAvatarManifest({ kind: "none", source: 7 })).toEqual({
+      kind: "none",
+      traits: null,
+      source: null,
+      image: null,
+    });
+  });
+
+  test("drops payload irrelevant to the kind", () => {
+    expect(
+      parseAvatarManifest({ kind: "image", image, traits: { bodyShape: 1 } }),
+    ).toEqual({ kind: "image", traits: null, source: null, image });
+    expect(
+      parseAvatarManifest({ kind: "character", traits, image: "stale" }),
+    ).toEqual({ kind: "character", traits, source: null, image: null });
+    expect(parseAvatarManifest({ kind: "none", traits, image })).toEqual({
+      kind: "none",
+      traits: null,
+      source: null,
+      image: null,
+    });
+  });
+
+  test.each([
+    ["non-object", "nope"],
+    ["missing kind", {}],
+    ["invalid kind", { kind: "sprite", traits }],
+    ["character without traits", { kind: "character" }],
+    [
+      "character with incomplete traits",
+      { kind: "character", traits: { bodyShape: "round" } },
+    ],
+    ["image without meta", { kind: "image" }],
+    ["image with empty etag", { kind: "image", image: { ...image, etag: "" } }],
+  ])("rejects %s", (_label, value) => {
+    expect(parseAvatarManifest(value)).toBeNull();
+  });
+});
+
+describe("deriveAvatarFromLegacyFiles", () => {
+  test("overlong traits are invalid", () => {
+    expect(
+      deriveAvatarFromLegacyFiles({
+        traitsJson: { ...traits, bodyShape: overlong },
+        hasImage: true,
+      }),
+    ).toEqual({ kind: "image" });
+  });
+
+  test("valid traits win over a present image", () => {
+    expect(
+      deriveAvatarFromLegacyFiles({ traitsJson: traits, hasImage: true }),
+    ).toEqual({ kind: "character", traits });
+  });
+
+  test("invalid traits fall back to the image", () => {
+    expect(
+      deriveAvatarFromLegacyFiles({
+        traitsJson: { bodyShape: 1 },
+        hasImage: true,
+      }),
+    ).toEqual({ kind: "image" });
+  });
+
+  test("nothing usable yields none", () => {
+    expect(
+      deriveAvatarFromLegacyFiles({ traitsJson: undefined, hasImage: false }),
+    ).toEqual({ kind: "none" });
+  });
+});
+
+describe("resolveAvatarFromFiles", () => {
+  test("a valid manifest decides regardless of sidecars", () => {
+    expect(
+      resolveAvatarFromFiles({
+        manifestJson: { kind: "none" },
+        traitsJson: traits,
+        hasImage: true,
+      }),
+    ).toEqual({ kind: "none" });
+    expect(
+      resolveAvatarFromFiles({
+        manifestJson: { kind: "image", image },
+        traitsJson: traits,
+        hasImage: true,
+      }),
+    ).toEqual({ kind: "image", image });
+  });
+
+  test("a partial image manifest falls back to traits-first derivation", () => {
+    expect(
+      resolveAvatarFromFiles({
+        manifestJson: { kind: "image" },
+        traitsJson: traits,
+        hasImage: true,
+      }),
+    ).toEqual({ kind: "character", traits });
+  });
+
+  test("a legacy image carries no manifest metadata", () => {
+    expect(
+      resolveAvatarFromFiles({
+        manifestJson: undefined,
+        traitsJson: undefined,
+        hasImage: true,
+      }),
+    ).toEqual({ kind: "image", image: null });
+  });
+});

package/node_modules/@vellumai/avatar-manifest/src/__tests__/read.test.ts

@@ -0,0 +1,114 @@
+import { afterEach, beforeEach, describe, expect, test } from "bun:test";
+import {
+  mkdirSync,
+  mkdtempSync,
+  rmSync,
+  symlinkSync,
+  writeFileSync,
+} from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import { AVATAR_SIDECAR_MAX_BYTES, readWorkspaceAvatar } from "../read.js";
+
+const traits = { bodyShape: "round", eyeStyle: "dot", color: "#abc" };
+
+describe("readWorkspaceAvatar", () => {
+  let workspaceDir: string;
+  let avatarDir: string;
+
+  const write = (name: string, contents: string): void => {
+    mkdirSync(avatarDir, { recursive: true });
+    writeFileSync(join(avatarDir, name), contents);
+  };
+
+  beforeEach(() => {
+    workspaceDir = mkdtempSync(join(tmpdir(), "avatar-manifest-"));
+    avatarDir = join(workspaceDir, "data", "avatar");
+  });
+
+  afterEach(() => {
+    rmSync(workspaceDir, { recursive: true, force: true });
+  });
+
+  test("missing avatar dir yields none", () => {
+    expect(readWorkspaceAvatar(workspaceDir)).toEqual({ kind: "none" });
+  });
+
+  test("manifest character returns its traits", () => {
+    write("avatar.json", JSON.stringify({ kind: "character", traits }));
+    write("avatar-image.png", "png");
+    expect(readWorkspaceAvatar(workspaceDir)).toEqual({
+      kind: "character",
+      traits,
+    });
+  });
+
+  test("image resolves to the PNG path", () => {
+    write("avatar-image.png", "png");
+    expect(readWorkspaceAvatar(workspaceDir)).toEqual({
+      kind: "image",
+      imagePath: join(avatarDir, "avatar-image.png"),
+    });
+  });
+
+  test("corrupt manifest falls back to the traits sidecar", () => {
+    write("avatar.json", "{ not json");
+    write("character-traits.json", JSON.stringify(traits));
+    expect(readWorkspaceAvatar(workspaceDir)).toEqual({
+      kind: "character",
+      traits,
+    });
+  });
+
+  test("oversized sidecars are treated as absent", () => {
+    const padded = JSON.stringify({
+      kind: "character",
+      traits,
+      pad: "x".repeat(AVATAR_SIDECAR_MAX_BYTES),
+    });
+    write("avatar.json", padded);
+    write("character-traits.json", JSON.stringify({ ...traits, pad: padded }));
+    expect(readWorkspaceAvatar(workspaceDir)).toEqual({ kind: "none" });
+  });
+
+  test("a sidecar at the cap still reads", () => {
+    const body = JSON.stringify({ kind: "character", traits, pad: "" });
+    write(
+      "avatar.json",
+      body.replace(
+        '"pad":""',
+        `"pad":"${"x".repeat(AVATAR_SIDECAR_MAX_BYTES - body.length)}"`,
+      ),
+    );
+    expect(readWorkspaceAvatar(workspaceDir)).toEqual({
+      kind: "character",
+      traits,
+    });
+  });
+
+  test("a symlinked avatar dir is treated as absent", () => {
+    const foreign = mkdtempSync(join(tmpdir(), "foreign-avatar-"));
+    writeFileSync(
+      join(foreign, "character-traits.json"),
+      JSON.stringify({
+        bodyShape: "blob",
+        eyeStyle: "curious",
+        color: "green",
+      }),
+    );
+    mkdirSync(join(workspaceDir, "data"), { recursive: true });
+    symlinkSync(foreign, avatarDir);
+    expect(readWorkspaceAvatar(workspaceDir)).toEqual({ kind: "none" });
+    rmSync(foreign, { recursive: true, force: true });
+  });
+
+  test("symlinked sidecars are treated as absent", () => {
+    const outside = join(workspaceDir, "outside.json");
+    writeFileSync(outside, JSON.stringify({ kind: "character", traits }));
+    mkdirSync(avatarDir, { recursive: true });
+    symlinkSync(outside, join(avatarDir, "avatar.json"));
+    symlinkSync(outside, join(avatarDir, "character-traits.json"));
+    expect(readWorkspaceAvatar(workspaceDir)).toEqual({ kind: "none" });
+  });
+});