@happyvertical/smrt-chat 0.38.23 → 0.38.25

package/AGENTS.md

@@ -2,11 +2,26 @@
 
 Chat rooms, threads, and agent sessions with app-controlled tool whitelisting.
 
+## Dev Server
+
+`pnpm --dir packages/chat dev` runs a package-local SvelteKit workbench. The root
+route is an interactive chat surface with a dev-only `/api/dev-chat` endpoint:
+it uses `@happyvertical/ai` when local provider credentials are present and
+falls back to a deterministic local assistant otherwise. `/previews` hosts the
+shared component playground entries from `src/svelte/playground.ts`.
+
+The root workbench also has a dev-only voice conversation mode. It reads voice
+gateway connection details through `/api/dev-voice/config`, streams browser mic
+audio to `WS /ws/voice` as PCM16 mono, appends gateway transcripts/responses to
+the chat, and plays returned TTS audio. Exposing
+`SMRT_CHAT_DEV_VOICE_GATEWAY_TOKEN` to the browser requires
+`SMRT_CHAT_DEV_VOICE_GATEWAY_EXPOSE_TOKEN=true`; keep that local-only.
+
 ## Models
 
-Internal models — all mutations go through the membership/owner-checked `ChatService` (S5 #1392). EVERY `@smrt()` model in this package (ChatRoom, ChatMessage, ChatParticipant, ChatThread, ChatReaction, AgentSession) has a READ-ONLY generated REST/MCP surface (`list`/`get` only); `create`/`update`/`delete` are intentionally NOT generated so the raw collection routes cannot skip the service-layer authorization. A structural regression test enumerates the registry to assert no chat model exposes a mutating op.
+Internal models — all mutations go through the membership/owner-checked `ChatService` (S5 #1392) or the voice adapter's binding-checked flow. EVERY `@smrt()` model in this package (ChatRoom, ChatMessage, ChatParticipant, ChatThread, ChatReaction, AgentSession, VoiceSession) has a READ-ONLY generated REST/MCP surface (`list`/`get` only); `create`/`update`/`delete` are intentionally NOT generated so the raw collection routes cannot skip the service-layer authorization. A structural regression test enumerates the registry to assert no chat model exposes a mutating op.
 
-`ChatService` is a CLOSED FACADE (S5 #1392). The raw collections (`rooms`, `messages`, `participants`, `threads`, `agentSessions`, `reactions`) are ES `#private` fields — they are NOT on the public `ChatService` type and the package index does NOT export the collection classes, so a consumer cannot do `chat.messages.create({senderProfileId, role})` / `new ChatParticipantCollection(...)` to mutate around the authorization. The security-sensitive internals (`#writeMessage`, `#emitAgentReply`, `#enrollParticipant`, `#loadActiveSession`, `#requireActiveMembership`, `#requireRoomAdmin`, `#extractToolName`) are ES `#private` too, so they are unreachable at runtime — TypeScript `private` alone is erased and would leave them callable on the prototype. The agent-reply bridge is a `Symbol`-keyed static (not the old enumerable `_runAgentReply`), reachable only by the module-local `sendAgentReply` that holds the non-exported symbol.
+`ChatService` is a CLOSED FACADE (S5 #1392). The raw collections (`rooms`, `messages`, `participants`, `threads`, `agentSessions`, `reactions`, `voiceSessions`) are ES `#private` fields — they are NOT on the public `ChatService` type and the package index does NOT export the collection classes, so a consumer cannot do `chat.messages.create({senderProfileId, role})` / `new ChatParticipantCollection(...)` to mutate around the authorization. The security-sensitive internals (`#writeMessage`, `#emitAgentReply`, `#enrollParticipant`, `#loadActiveSession`, `#requireActiveMembership`, `#requireRoomAdmin`, `#extractToolName`) are ES `#private` too, so they are unreachable at runtime — TypeScript `private` alone is erased and would leave them callable on the prototype. The agent-reply bridge is a `Symbol`-keyed static (not the old enumerable `_runAgentReply`), reachable only by the module-local `sendAgentReply` that holds the non-exported symbol.
 
 - **ChatRoom**: `roomType` (public/private/dm/agent), `status`, `topic`, `maxParticipants`, `lastMessageAt`. Tenant-scoped (required).
 - **ChatMessage**: shared by users + agents. `role` (user/assistant/system/tool), `messageType` (text/system/action/file/tool_call/tool_result), `toolCallData` JSON. Unified model — no separate agent message type. Tenant-scoped (required).
@@ -14,6 +29,7 @@ Internal models — all mutations go through the membership/owner-checked `ChatS
 - **ChatThread**: `rootMessageId`, `isResolved`, `messageCount`. Created via `ChatService.startThread()` (member-checked). Tenant-scoped (required).
 - **ChatReaction**: `messageId`, `profileId`, `emoji`. Added/removed via `ChatService.addReaction()`/`removeReaction()` (member-checked, self-keyed). Tenant-scoped (required).
 - **AgentSession**: `agentId` (string ref, not FK), `allowedTools` (JSON string array), `sessionContext` (JSON), `systemPrompt`, limits (`maxTokens`/`maxMessages`/`expiresAt`). Optional tenancy.
+- **VoiceSession**: short-lived voice-gateway binding over `(tenant, actorProfileId, personaId, room/thread/agentSession)` plus a persona snapshot, gateway `session_id`, expiry, replay tracking, and metadata. Tenant-scoped (required). Generated surface is read-only; creation and turns go through `createVoiceChatSession()` / `handleVoiceGatewayTurn()`.
 
 ## ChatService
 
@@ -34,6 +50,12 @@ The "chat with your learning agent" surface — the real agentic runtime for `Ag
 - **Agent orchestration** (L3, #1892) — the loop accepts non-manifest **`extraTools`** (`PrincipalTool[]`, from `@happyvertical/smrt-agents`), gated by the *same* fail-closed allow-list. The standard **`invoke-agent`** tool (`createInvokeAgentTool`, slug `agents.invoke`) lets a conversational agent delegate to a **worker agent under its own principal** — the worker runs via `executeAsPrincipal` as the originating user (never its own authority), the principal is immutable along the chain, and its completion is surfaced back into the conversation. `runPersonaConversationTurn` filters `extraTools` by the persona's `allowedTools` (offer gate); the tool's `execute` re-asserts `assertToolAllowed` (execution gate). See `@happyvertical/smrt-agents` for the delegation envelope + transports.
 - **Chat feedback capture** (`chat-feedback.ts`) — `captureChatFeedback()` + `acceptAppliedChange`/`rejectAppliedChange`/`correctResponse`/`rateResponse`/`thumbsUp`/`thumbsDown` write a `Feedback` row (personas) carrying the conversation's **correlation-id**, and (by default) reinforce the persona's learning memory (`reinforceFromFeedback`). So an in-chat reject decays a strategy below the reuse floor and it stops being recalled; a correction supersedes its stored value.
 
+## Voice Gateway Turns (#1910)
+
+Voice is an input mode for the existing persona chat harness, not a separate chat runtime. `createVoiceChatSession()` creates a short-lived `VoiceSession` for an authenticated actor/profile, binding tenant, persona, agent session, room, and optional thread. `handleVoiceGatewayTurn()` resolves that binding from `metadata.voiceSessionId`, checks the gateway's `session_id` and any supplied tenant/profile/persona/session/thread metadata against the server-side binding, persists the transcript through `ChatService.sendAgentUserMessage()`, runs `runPersonaConversationTurn()`, stamps voice/correlation metadata onto the persisted user/assistant/tool messages, records the gateway `turn_id`, and returns the gateway response contract. The Fetch-compatible `createVoiceGatewayTurnHandler()` adds the coarse gateway bearer-token check.
+
+The gateway bearer token proves only "this request came from the gateway"; it never authorizes the end user. The short-lived `VoiceSession` binding is the user/session proof, and untrusted gateway metadata must be validated against that binding before any chat write or tool loop. Tool execution remains fail-closed through the persona allow-list mirrored onto `AgentSession` by `bindPersonaToSession()`.
+
 ## Gotchas
 
 - **sessionContext, not context**: `context` is reserved for slug scoping. Use `getSessionContext()`/`updateSessionContext()` for agent memory.

package/README.md

@@ -10,6 +10,30 @@ pnpm add @happyvertical/smrt-chat
 
 ## Usage
 
+### Local dev server
+
+Run the chat package workbench directly:
+
+```bash
+pnpm --dir packages/chat dev
+```
+
+The root route opens an interactive chat surface backed by
+`src/routes/api/dev-chat/+server.ts`. If `SMRT_CHAT_DEV_PROVIDER` /
+`SMRT_CHAT_DEV_API_KEY` or standard provider credentials such as
+`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, or `GEMINI_API_KEY` are available, the
+assistant turn goes through `@happyvertical/ai`; otherwise it uses a
+deterministic local fallback. Component previews are available at `/previews`.
+
+The workbench also has a local-only voice conversation mode. It reads
+`SMRT_CHAT_DEV_VOICE_GATEWAY_HTTP_URL` and
+`SMRT_CHAT_DEV_VOICE_GATEWAY_WS_URL`, fetches gateway targets from
+`/api/dev-voice/config`, streams browser microphone audio to `WS /ws/voice` as
+PCM16 mono at 16 kHz, appends gateway transcripts/responses to the chat, and
+plays returned TTS audio. If the gateway requires a bearer token, set
+`SMRT_CHAT_DEV_VOICE_GATEWAY_TOKEN`; browser WebSocket testing also requires
+`SMRT_CHAT_DEV_VOICE_GATEWAY_EXPOSE_TOKEN=true`, so use this only for local dev.
+
 ### Rooms and messages
 
 ```typescript
@@ -104,6 +128,93 @@ if (session.isActive()) {
 }
 ```
 
+### Voice gateway turns
+
+`smrt-chat` can expose a gateway-facing turn target for browser voice input.
+The browser receives only a short-lived SMRT voice session binding; the
+gateway-to-SMRT service bearer token stays server-side.
+
+```typescript
+import {
+  createVoiceChatSession,
+  createVoiceGatewayTurnHandler,
+} from '@happyvertical/smrt-chat';
+
+// Authenticated app route: bind the speaking profile to a persona and an
+// existing or newly-created agent session.
+const voice = await createVoiceChatSession({
+  chatService: chat,
+  db,
+  tenantId: locals.tenantId,
+  actorProfileId: locals.profileId,
+  actorUserId: locals.userId,
+  persona: resolvedPersona,
+  agentSessionId: session.id, // omit to create/reuse an agent session
+  ttlSeconds: 600,
+});
+
+// Return voice.gatewaySessionId + voice.metadata to the browser/gateway. Do not
+// return the gateway service token.
+
+// Gateway target route: configure the deployed voice gateway HTTP target to
+// POST here with Authorization: Bearer <SMRT_VOICE_GATEWAY_TOKEN>.
+export const POST = ({ request }) =>
+  createVoiceGatewayTurnHandler({
+    chatService: chat,
+    db,
+    ai,
+    gatewayToken: process.env.SMRT_VOICE_GATEWAY_TOKEN ?? '',
+  })(request);
+```
+
+The gateway sends the transcribed text turn:
+
+```json
+{
+  "session_id": "gateway-or-smrt-stable-session-id",
+  "turn_id": "gateway-generated-turn-id",
+  "target": "smrt:chat",
+  "actor": "optional-display-name",
+  "text": "transcribed user utterance",
+  "metadata": {
+    "tenantId": "tenant uuid",
+    "actorProfileId": "profile id",
+    "chatRoomId": "room id",
+    "threadId": "optional thread id",
+    "agentSessionId": "agent session id",
+    "personaId": "persona id",
+    "voiceSessionId": "SMRT-issued voice session id",
+    "source": "voice-gateway"
+  }
+}
+```
+
+SMRT validates `metadata.voiceSessionId` against its server-side binding, then
+checks any supplied tenant/profile/persona/session/thread ids against that
+binding instead of trusting the gateway metadata. A valid turn is persisted as a
+normal user chat message, routed through `runPersonaConversationTurn()`, and the
+assistant reply is persisted as a normal assistant message. The response is:
+
+```json
+{
+  "session_id": "same session_id",
+  "turn_id": "same turn_id",
+  "text": "assistant response text",
+  "metadata": {
+    "tenantId": "tenant id",
+    "chatRoomId": "room id",
+    "threadId": null,
+    "agentSessionId": "agent session id",
+    "userMessageId": "persisted transcript message id",
+    "assistantMessageId": "persisted assistant message id",
+    "personaId": "persona id",
+    "correlationId": "feedback correlation id",
+    "voiceSessionId": "SMRT-issued voice session id",
+    "source": "voice-gateway"
+  }
+}
+```
+
 ### Direct messages
 
 ```typescript
@@ -129,10 +240,15 @@ const dmRoom = await chat.getOrCreateDM({
 | `ChatThread` | Threaded conversation linked to a root message, with resolve/reopen lifecycle |
 | `ChatReaction` | Emoji reaction on a message |
 | `AgentSession` | AI agent session with `allowedTools` (JSON string array), `sessionContext` for multi-turn memory, `systemPrompt`, and usage limits (`maxTokens`/`maxMessages`/`expiresAt`) |
+| `VoiceSession` | Short-lived voice gateway binding over a tenant, actor profile, persona, and agent session |
 
-### Collections
+### Internal collections
 
-| Export | Description |
+Raw collection classes are not exported from the package index. They are listed
+here as implementation details; application code should use `ChatService` and
+the voice helpers.
+
+| Internal class | Description |
 |--------|------------|
 | `ChatRoomCollection` | Room queries, `findOrCreateDM()` |
 | `ChatMessageCollection` | Message queries and search filters |
@@ -140,16 +256,20 @@ const dmRoom = await chat.getOrCreateDM({
 | `ChatThreadCollection` | Thread queries |
 | `ChatReactionCollection` | Reaction queries |
 | `AgentSessionCollection` | Session queries, `findActiveSession()`, `findOrCreate()` |
+| `VoiceSessionCollection` | Voice session binding queries and stale-expiry helper |
 
 ### Services
 
 | Export | Description |
 |--------|------------|
 | `ChatService` | Facade: `createRoom()`, `sendMessage()`, `startThread()`, `addParticipant()`, `removeParticipant()`, `updateRoom()`, `addReaction()`, `removeReaction()`, `getOrCreateDM()`, `createAgentSession()`, `sendAgentUserMessage()`, `getRoomMessages()`, `getRoomForMember()`, `updateAgentSessionConfig()`. Every write takes a server-supplied `actorProfileId`. The agent-authored reply path (`sendAgentReply`) is intentionally NOT on this facade or the package index — it is an internal function in `services/ChatService.ts` for the trusted in-process agent runtime only (S5 #1392). |
+| `createVoiceChatSession()` | Creates a short-lived voice binding for an authenticated actor and persona, optionally against an existing agent session |
+| `handleVoiceGatewayTurn()` | Lower-level gateway turn adapter for non-Fetch hosts |
+| `createVoiceGatewayTurnHandler()` | Fetch-compatible HTTP handler for the gateway target endpoint |
 
 ### Types
 
-`ChatRoomType`, `ChatRoomStatus`, `ChatRoomOptions`, `ChatMessageType`, `ChatMessageRole`, `ChatMessageOptions`, `ChatMessageSearchFilters`, `ChatParticipantRole`, `ChatParticipantStatus`, `ChatParticipantOptions`, `OnlineStatus`, `ChatThreadOptions`, `ChatReactionOptions`, `AgentSessionStatus`, `AgentSessionOptions`
+`ChatRoomType`, `ChatRoomStatus`, `ChatRoomOptions`, `ChatMessageType`, `ChatMessageRole`, `ChatMessageOptions`, `ChatMessageSearchFilters`, `ChatParticipantRole`, `ChatParticipantStatus`, `ChatParticipantOptions`, `OnlineStatus`, `ChatThreadOptions`, `ChatReactionOptions`, `AgentSessionStatus`, `AgentSessionOptions`, `VoiceSessionStatus`, `VoiceSessionOptions`, `VoiceGatewayTurnPayload`, `VoiceGatewayTurnResponse`
 
 ### Constants