@breadstone/archipel-mcp 0.0.54 → 0.0.56

package/data/guides/ai-realtime-voice.md

@@ -1,506 +1,118 @@
 ---
 title: AI Realtime Voice
-description: Build server-owned, live audio and text sessions with Vercel AI SDK providers, durable application gateways, metering, and explicit recovery.
-order: 27
+description: Build server-owned bidirectional audio and text sessions with provider-neutral events, limits, cleanup, and metering.
+order: 17
 ---
 
 # AI Realtime Voice
 
-This guide covers long-lived, bidirectional AI sessions with
-`platform-intelligence`: provider registration, live audio forwarding,
-normalized events, silent and spoken interaction modes, tools, metering, and
-failure recovery.
+`IntelligenceRealtimeSessionFactory` creates provider-neutral, server-owned
+audio/text sessions. Provider credentials and upstream sockets remain in the
+backend.
 
-Realtime sessions are not hour-long uploads. The client and application move
-small audio fragments continuously while each layer persists only the data it
-owns. This preserves live interaction without making the complete recording a
-single point of failure.
+Realtime is separate from `IntelligenceTranscriptionGenerator.stream()`:
 
-For the reusable ownership and recovery rules behind this integration, see the
-[Realtime Session Pattern](../patterns/realtime-session-pattern).
+- streaming transcription accepts raw audio and emits transcript updates;
+- realtime is a bidirectional conversational session with input/output audio,
+  text, tools, turn detection, interruption, and session state.
 
----
-
-## Architecture and SDK Boundary
-
-Archipel owns the upstream provider connection. A consuming application puts
-its authenticated client transport and domain workflow in front of the stable
-Archipel port.
-
-```mermaid
-flowchart LR
-  Client[Browser or native client\nMicrophone, playback, local chunk queue]
-  Gateway[Application realtime gateway\nAuth, tenant, sequencing, acknowledgements]
-  Persistence[Application persistence\nAudio, transcript, session state]
-  Factory[IntelligenceRealtimeSessionFactory]
-  Port[IntelligenceRealtimeSessionPort]
-  AiSdk[Vercel AI SDK 7\nRealtime provider model]
-  Socket[Generic server WebSocket transport]
-  Provider[OpenAI, Google, or xAI]
-
-  Client <--> Gateway
-  Gateway <--> Persistence
-  Gateway <--> Port
-  Factory --> Port
-  Port --> AiSdk
-  AiSdk --> Socket
-  Socket <--> Provider
-```
-
-Provider-specific token creation, event serialization, event parsing, and
-model selection use the Vercel AI SDK provider packages. Archipel uses the
-generic `ws` transport to open the WebSocket configuration returned by the AI
-SDK model adapter. It does not install the official OpenAI SDK, expose an
-OpenAI client, or make provider-specific calls outside the AI SDK adapter.
-
-The AI SDK realtime model contract is experimental. Archipel isolates that
-contract behind its own `/realtime` entry point so application code depends on
-the normalized ports and events. The `raw` field on server events remains an
-escape hatch for audited provider-specific diagnostics.
-
-### Responsibility boundary
-
-| Layer | Owns |
-| --- | --- |
-| Client | Microphone permission, capture, playback, local durable chunk queue, UI state |
-| Application | Authentication, authorization, tenant context, client WebSocket, durable audio/transcript state, tool execution, reconnect policy |
-| Archipel | Provider configuration, short-lived provider token, upstream WebSocket, event normalization, bounded queues, metering emission, shutdown |
-| Provider | Model conversation state and provider-native usage reporting |
-
-Archipel never exposes provider credentials to the client and does not own the
-application's audio recording or consultation record.
-
----
-
-## Installation
-
-Install Archipel, the AI SDK core, and only the provider packages used by the
-application:
-
-```bash
-yarn add @breadstone/archipel-platform-intelligence ai
-
-# Add one or more Vercel AI SDK providers
-yarn add @ai-sdk/openai
-yarn add @ai-sdk/google
-yarn add @ai-sdk/xai
-```
-
-| Provider | Vercel AI SDK package | Archipel loader |
-| --- | --- | --- |
-| OpenAI | `@ai-sdk/openai` | `loadOpenAIRealtimeModel` |
-| Google Gemini Live | `@ai-sdk/google` | `loadGoogleRealtimeModel` |
-| xAI / Grok Voice | `@ai-sdk/xai` | `loadGrokRealtimeModel` |
-
-Do not add the official `openai` package for this integration.
-
----
-
-## Register Realtime Providers
-
-Register realtime loaders under the existing `IntelligenceModule`. The root
-package stays provider-neutral; each loader is imported from a tree-shakable
-provider subpath.
+## Registration
 
 ```typescript
 import { Module } from '@nestjs/common';
-import {
-  IntelligenceModule,
-  IntelligenceProviderNames,
-} from '@breadstone/archipel-platform-intelligence';
-import { loadGoogleRealtimeModel } from '@breadstone/archipel-platform-intelligence/providers/google';
-import { loadGrokRealtimeModel } from '@breadstone/archipel-platform-intelligence/providers/grok';
-import { loadOpenAIRealtimeModel } from '@breadstone/archipel-platform-intelligence/providers/openai';
-import { ApplicationRealtimeMeteringAdapter } from './ApplicationRealtimeMeteringAdapter';
+import { IntelligenceModule } from '@breadstone/archipel-platform-intelligence';
+import { AI_GATEWAY_CONFIG_ENTRIES, GatewayIntelligenceProviderAdapter } from '@breadstone/archipel-platform-intelligence/provider/gateway';
+import { GOOGLE_CONFIG_ENTRIES, GoogleIntelligenceProviderAdapter } from '@breadstone/archipel-platform-intelligence/provider/google';
+import { GROK_CONFIG_ENTRIES, GrokIntelligenceProviderAdapter } from '@breadstone/archipel-platform-intelligence/provider/grok';
+import { OPENAI_CONFIG_ENTRIES, OpenAIIntelligenceProviderAdapter } from '@breadstone/archipel-platform-intelligence/provider/openai';
 
 @Module({
-  imports: [
-    IntelligenceModule.register({
-      // Use false when this module is realtime-only. Applications that also
-      // register text-generation loaders can retain the default eager check.
-      validateOnModuleInit: false,
-      realtime: {
-        providerLoaders: {
-          [IntelligenceProviderNames.OpenAI]: loadOpenAIRealtimeModel,
-          [IntelligenceProviderNames.Google]: loadGoogleRealtimeModel,
-          [IntelligenceProviderNames.Grok]: loadGrokRealtimeModel,
-        },
-        meteringAdapter: ApplicationRealtimeMeteringAdapter,
-        meteringIntervalMs: 30_000,
-      },
-    }),
-  ],
+    imports: [
+        IntelligenceModule.register({
+            configEntries: [...OPENAI_CONFIG_ENTRIES, ...GOOGLE_CONFIG_ENTRIES, ...GROK_CONFIG_ENTRIES, ...AI_GATEWAY_CONFIG_ENTRIES],
+            providerAdapters: [
+                OpenAIIntelligenceProviderAdapter,
+                GoogleIntelligenceProviderAdapter,
+                GrokIntelligenceProviderAdapter,
+                GatewayIntelligenceProviderAdapter,
+            ],
+            realtime: {
+                connectionTimeoutMs: 5_000,
+                idleTimeoutMs: 120_000,
+                absoluteSessionTimeoutMs: 1_800_000,
+                maxActiveSessions: 100,
+                maxBufferedAmountBytes: 1_048_576,
+                maxMessageBytes: 1_048_576,
+                meteringIntervalMs: 30_000,
+                meteringAdapter: ApplicationRealtimeMeteringAdapter,
+            },
+        }),
+    ],
 })
-export class RealtimeIntelligenceModule {}
-```
-
-Configure the default provider and its credentials through the normal Archipel
-configuration registry:
-
-```dotenv
-INTELLIGENCE_PROVIDER=openai
-OPENAI_API_KEY=sk-...
-
-# Optional when every createSession() call supplies its model
-INTELLIGENCE_MODEL=gpt-realtime
+export class ApplicationModule {}
 ```
 
-`INTELLIGENCE_API_KEY` can provide a global override. Google accepts
-`GOOGLE_API_KEY` or `GEMINI_API_KEY`; xAI accepts `GROK_API_KEY` or
-`XAI_API_KEY`.
+The package owns all provider runtime dependencies. Applications do not install
+or import provider SDKs.
 
-`validateOnModuleInit` controls eager text-model initialization. It does not
-disable realtime configuration validation when a session is created. If the
-same module also uses text generation, register the normal language-model
-loaders and leave validation enabled.
-
-The provider and API key still come from the normal intelligence
-configuration. `createSession()` may override the provider and model per
-logical session.
-
----
-
-## Create and Connect a Session
-
-`createSession()` validates the application-owned session identifier, resolves
-the configured provider, loads the AI SDK realtime model, and returns an idle
-port. `connect()` then creates the short-lived provider token, opens the
-upstream WebSocket, and applies the session configuration.
-
-```typescript
-import { Injectable } from '@nestjs/common';
-import { IntelligenceProviderNames } from '@breadstone/archipel-platform-intelligence';
-import {
-  IntelligenceRealtimeSessionFactory,
-  type IntelligenceRealtimeServerEvent,
-  type IntelligenceRealtimeSessionPort,
-} from '@breadstone/archipel-platform-intelligence/realtime';
-
-@Injectable()
-export class RealtimeConsultationCoordinator {
-  public constructor(
-    private readonly _sessions: IntelligenceRealtimeSessionFactory,
-    private readonly _events: ConsultationEventStore,
-  ) {}
-
-  public async start(consultationId: string): Promise<IntelligenceRealtimeSessionPort> {
-    const session = await this._sessions.createSession({
-      sessionId: consultationId,
-      provider: IntelligenceProviderNames.OpenAI,
-      model: 'gpt-realtime',
-      metadata: {
-        feature: 'consultation',
-        consultationId,
-      },
-      sessionConfiguration: {
-        instructions: 'Transcribe the discussion and suggest concise next questions.',
-        inputAudioFormat: {
-          type: 'audio/pcm',
-          rate: 24_000,
-        },
-        inputAudioTranscription: {},
-        outputModalities: ['text', 'audio'],
-        turnDetection: {
-          type: 'server-vad',
-        },
-      },
-      onEvent: async (event: IntelligenceRealtimeServerEvent): Promise<void> => {
-        await this._events.persistThenPublish(consultationId, event);
-      },
-      onError: async (error: Error): Promise<void> => {
-        await this._events.persistError(consultationId, error);
-      },
-      onClose: async (event): Promise<void> => {
-        await this._events.persistClose(consultationId, event);
-      },
-    });
-
-    await session.connect();
-
-    return session;
-  }
-}
-```
-
-The application must retain the returned port for the lifetime of its client
-gateway connection. The factory tracks ports internally for resource limits
-and Nest shutdown, but deliberately does not act as an application session
-repository.
-
-### Two identifiers
-
-| Identifier | Lifetime | Purpose |
-| --- | --- | --- |
-| `sessionId` | Application-defined logical conversation | Persistence, authorization, billing account, recovery |
-| `connectionId` | One Archipel/provider WebSocket | Idempotency, audit correlation, distinguishing reconnects |
-
-A reconnect keeps the logical `sessionId` and creates a new `connectionId`.
-
----
-
-## Forward Audio Live and Durably
-
-`appendAudio()` accepts one base64-encoded fragment and forwards it immediately
-to the active provider input buffer. A fragment may contain at most 1,500,000
-base64 characters. The configured sample rate and format must match the bytes
-produced by the client.
-
-Use a small, ordered application envelope around the Archipel payload:
+## Create and Connect
 
 ```typescript
-interface IClientAudioChunk {
-  readonly chunkId: string;
-  readonly sequence: number;
-  readonly audio: string;
-}
-```
-
-For each fragment:
-
-1. The client writes the fragment to its local durable queue before sending.
-2. The application authenticates the socket and validates the logical session.
-3. The application persists `(sessionId, chunkId, sequence, audio)`
-   idempotently.
-4. The application calls `session.appendAudio(audio)` immediately after that
-   small persistence operation.
-5. The application acknowledges the chunk only after durable persistence and
-   successful forwarding.
-6. The client removes the locally queued fragment after acknowledgement.
-
-This is persist-per-fragment, not record-for-an-hour-then-upload. If the client
-or network fails, only unacknowledged fragments remain to be retried.
-
-Archipel intentionally does not define `chunkId`, acknowledgements, IndexedDB,
-or the application WebSocket protocol. Those belong to the consuming domain;
-the upstream provider contract only needs the ordered base64 audio.
-
-### Turn completion
-
-With server voice-activity detection, the provider determines turn boundaries.
-For push-to-talk, disable automatic turn detection and commit explicitly:
-
-```typescript
-const session = await this._sessions.createSession({
-  sessionId: consultationId,
-  model: 'gpt-realtime',
-  sessionConfiguration: {
-    inputAudioFormat: { type: 'audio/pcm', rate: 24_000 },
-    turnDetection: null,
-  },
+const session = await sessions.createSession({
+    sessionId: applicationSessionId,
+    provider: 'openai',
+    model: 'gpt-realtime',
+    metadata: {
+        tenantId,
+        userId,
+    },
+    activeToolNames: ['customer_search'],
+    sessionConfiguration: {
+        instructions: 'Help with the authenticated tenant context only.',
+        modalities: ['text', 'audio'],
+    },
+    onEvent: (event) => forwardToApplicationSocket(event),
 });
 
 await session.connect();
-await session.appendAudio(audioChunk);
-await session.commitAudio();
-await session.requestResponse();
-```
-
-Use `clearAudio()` to discard the uncommitted provider input buffer.
-
----
-
-## Choose the Interaction Mode
-
-The application controls whether the model speaks. There is no separate
-provider strategy or SDK for a silent assistant.
-
-| Mode | Session behavior | Application behavior |
-| --- | --- | --- |
-| Spoken assistant | `outputModalities: ['text', 'audio']` | Stream `audio-delta` to playback and show transcript |
-| Silent copilot | `outputModalities: ['text']` | Show suggestions, questions, or tasks; do not request/play audio |
-| Push-to-talk | `turnDetection: null` | Commit audio and request a response explicitly |
-| Passive capture | Enable input transcription and avoid response requests | Persist transcript events; verify provider-specific automatic-response behavior |
-
-Providers do not all support identical voices, formats, transcription models,
-or automatic-response controls. Keep the common workflow on Archipel contracts
-and place unavoidable provider-specific values in `providerOptions`.
-
-Text turns use the same session:
-
-```typescript
-await session.sendText('Summarize the unanswered requirements.');
-await session.cancelResponse();
-```
-
----
-
-## Handle Normalized Events
-
-`onEvent` receives events in provider wire order for the current connection.
-The most relevant event families are:
-
-| Family | Examples | Typical application action |
-| --- | --- | --- |
-| Speech | `speech-started`, `speech-stopped`, `audio-committed` | Update listening state and turn boundaries |
-| Input transcript | `input-transcription-completed` | Persist the final customer/seller transcript item |
-| Text output | `text-delta`, `text-done` | Render suggestions or assistant replies |
-| Audio output | `audio-delta`, `audio-done` | Stream base64 audio to the client player |
-| Audio transcript | `audio-transcript-delta`, `audio-transcript-done` | Persist spoken assistant text |
-| Tools | `function-call-arguments-delta`, `function-call-arguments-done` | Validate, authorize, execute, and return output |
-| Lifecycle | `response-created`, `response-done`, `error` | Track response state and diagnostics |
-
-Persist final events and enough ordered deltas to recover the UI. Treat `raw`
-as sensitive provider data: redact it according to the application's AI audit
-and privacy policy before storing it.
-
-### Tool calls
-
-Tools registered on `IntelligenceModule` are converted to realtime function
-definitions through the AI SDK when `sessionConfiguration.tools` is omitted.
-Use `activeToolNames` as an allow-list per session.
-
-When `function-call-arguments-done` arrives, the application remains
-responsible for parsing and validating the arguments, checking tenant and user
-permissions, invoking the domain operation, and returning the result:
-
-```typescript
-await session.addToolOutput(event.callId, toolResult, event.name);
-```
-
-Never execute provider-supplied arguments without the normal application
-authorization and validation path.
-
----
-
-## Metering, Tokens, and AI Audit
-
-Realtime traffic does not pass through a request-scoped HTTP token
-interceptor. `IntelligenceRealtimeMeteringPort` is the dedicated accounting
-boundary for long-lived sessions.
-
-```typescript
-import { Injectable } from '@nestjs/common';
-import {
-  IntelligenceRealtimeMeteringPort,
-  type IntelligenceRealtimeMeteringEvent,
-} from '@breadstone/archipel-platform-intelligence/realtime';
-
-@Injectable()
-export class ApplicationRealtimeMeteringAdapter extends IntelligenceRealtimeMeteringPort {
-  public constructor(private readonly _ledger: AiUsageLedger) {
-    super();
-  }
-
-  public override async report(event: IntelligenceRealtimeMeteringEvent): Promise<void> {
-    await this._ledger.recordRealtimeEventAtomically({
-      idempotencyKey: event.eventId,
-      event,
-    });
-  }
-}
+await session.send({
+    type: 'input_audio_buffer.append',
+    audio: base64AudioChunk,
+});
 ```
 
-The application transaction should:
-
-1. insert the metering event with a unique constraint on `eventId`;
-2. debit the application's token or credit ledger when the event is billable;
-3. append the normalized AI audit entry;
-4. commit all three changes together.
-
-| Event | Recommended use |
-| --- | --- |
-| `session-started` | Open one auditable provider connection |
-| `billing-interval` | Provider-independent commercial charging every 30 seconds and for the final partial interval |
-| `provider-usage` | Exact input/output, text/audio, cached, and total token reconciliation when supplied |
-| `session-ended` | Final duration, close code, and close reason |
-
-Choose one commercial charging basis. A common strategy debits credits from
-`billing-interval` and stores `provider-usage` only for provider-cost
-reconciliation. Do not debit both event types unless the pricing model
-intentionally contains both dimensions.
-
-The default metering adapter is a no-op. Production applications that charge
-users or require AI audit completeness must register a durable adapter. Keep
-that adapter fast and local—normally a database transaction or transactional
-outbox. Archipel retries a failed adapter write twice and then reports a
-realtime error; it cannot make a remote application datastore durable on the
-application's behalf.
-
----
+The application WebSocket authenticates the user, resolves tenant context,
+checks permissions and AI availability, and forwards only normalized events.
+Never expose provider credentials or an upstream provider socket to the browser.
 
-## Limits and Backpressure
+## Tools and Metering
 
-Module-level limits are normalized when `IntelligenceModule.register()` runs:
+When explicit realtime tools are absent, Archipel converts registered
+application tools to provider-neutral realtime definitions. `activeToolNames`
+restricts the available set.
 
-| Option | Default | Behavior |
-| --- | --- | --- |
-| `maxActiveSessions` | `1_000` | Maximum upstream sessions per Nest module instance |
-| `maxPendingEvents` | `512` | Maximum inbound, outbound, or metering queue depth per session |
-| `maxBufferedBytes` | `4 MiB` | Maximum provider WebSocket backpressure buffer |
-| `connectionTimeoutMs` | `5_000` | Provider token/setup and socket-open timeout; capped at 5 seconds |
-| `closeTimeoutMs` | `5_000` | Graceful close and metering-flush timeout; capped at 5 seconds |
-| `meteringIntervalMs` | `30_000` | Provider-independent billing interval |
-| `tokenRequestMaxRetries` | `2` | Additional transient token-creation attempts; capped at 2 |
+Implement `IntelligenceRealtimeMeteringPort` to persist usage events
+idempotently. Session-start, interval, provider-usage, and session-end events
+carry stable event IDs for deduplication. Business-critical charging should be
+transactional with the application usage ledger.
 
-Limits apply per process, not across a cluster. Enforce tenant and account
-quotas in the application before creating a session. When the client sends
-faster than `appendAudio()` can drain, apply backpressure at the client gateway
-instead of building another unbounded queue.
+Provider-usage events expose the common `IIntelligenceUsage` dimensions for
+text, image, audio, cache reads/writes, reasoning, input, and output. Missing
+provider dimensions remain `null`; raw provider usage is retained only as
+diagnostic event data. The application owns provider/model pricing and billing
+units so lifecycle intervals and token-based provider cost cannot accidentally
+charge the same product entitlement twice.
 
----
-
-## Failure and Recovery
-
-Archipel retries transient provider-token creation, but it never silently
-reconnects a dropped conversation. A new provider WebSocket has new
-conversation state and a new `connectionId`; automatic reconnect would hide
-that semantic break.
-
-Handle failures by phase through `IntelligenceRealtimeError`:
-
-| Phase | Meaning |
-| --- | --- |
-| `configuration` | Invalid provider or session configuration |
-| `token` | Short-lived provider credential creation failed |
-| `connect` | Upstream WebSocket could not open |
-| `send` | Outbound serialization, state, queue, or socket failure |
-| `receive` | Provider event parsing or inbound queue failure |
-| `metering` | Application metering adapter failed |
-| `close` | Graceful shutdown failed |
-
-The error also contains `provider`, `model`, `sessionId`, and `isRetryable`.
-Connect and send methods reject their promises; asynchronous receive and
-metering failures arrive through `onError`. `onClose` is the terminal upstream
-state.
-
-Recovery belongs to the application:
-
-1. stop acknowledging new client chunks for the failed connection;
-2. keep unacknowledged chunks in the client's durable queue;
-3. persist the terminal connection state and close reason;
-4. create a new Archipel session with the same logical `sessionId`;
-5. reconstruct safe context from the durable transcript or a compact summary;
-6. resume with a new `connectionId` and explicit audit entry.
-
-The normalized realtime contract does not promise lossless replay of arbitrary
-provider-native conversation history. Rebuild only the context supported by
-the selected provider and application policy.
-
-For horizontal deployments, route all frames for one active upstream
-connection to its owning process, for example through sticky routing or a
-distributed ownership record. Never depend on the factory's in-memory map as a
-cluster-wide registry.
-
-Nest shutdown calls `close()` for all owned sessions and waits for queued
-metering within the configured close timeout.
-
----
+## Runtime Guarantees
 
-## Production Checklist
+The factory enforces:
 
-- Provider loaders come only from `@ai-sdk/*`-backed Archipel subpaths.
-- Provider credentials remain in backend configuration.
-- The application authenticates and authorizes every client gateway message.
-- Each audio fragment has a logical chunk ID and sequence outside Archipel.
-- Client and backend persistence happen per fragment, not after the complete recording.
-- Transcript and final response events are persisted before client acknowledgement.
-- A durable `IntelligenceRealtimeMeteringPort` is registered.
-- `eventId` has a database uniqueness constraint.
-- Commercial interval charging and provider token reconciliation are not double-counted.
-- Backpressure and tenant quotas are bounded outside the per-process Archipel limits.
-- Reconnect creates a new connection and is visible in audit history.
-- Shutdown hooks are enabled in the Nest application.
+- active-session, metadata, instruction, tool, message, and buffer bounds;
+- connection, idle, absolute-session, and close timeouts;
+- upstream error normalization through `IntelligenceRealtimeError`;
+- backpressure limits and deterministic cleanup;
+- shutdown cleanup via `OnModuleDestroy`;
+- server-owned provider/model resolution.
 
-See the [platform-intelligence package page](../packages/platform-intelligence/)
-and the [realtime API reference](../packages/platform-intelligence/api/Class.IntelligenceRealtimeSessionFactory)
-for the complete contracts.
+OpenAI, Google, xAI/Grok, and Vercel AI Gateway currently expose realtime model
+adapters. Unsupported provider/capability pairs fail before a socket is opened.