@breadstone/archipel-mcp 0.0.52 → 0.0.54

package/data/guides/ai-realtime-voice.md

@@ -0,0 +1,506 @@
+---
+title: AI Realtime Voice
+description: Build server-owned, live audio and text sessions with Vercel AI SDK providers, durable application gateways, metering, and explicit recovery.
+order: 27
+---
+
+# AI Realtime Voice
+
+This guide covers long-lived, bidirectional AI sessions with
+`platform-intelligence`: provider registration, live audio forwarding,
+normalized events, silent and spoken interaction modes, tools, metering, and
+failure recovery.
+
+Realtime sessions are not hour-long uploads. The client and application move
+small audio fragments continuously while each layer persists only the data it
+owns. This preserves live interaction without making the complete recording a
+single point of failure.
+
+For the reusable ownership and recovery rules behind this integration, see the
+[Realtime Session Pattern](../patterns/realtime-session-pattern).
+
+---
+
+## Architecture and SDK Boundary
+
+Archipel owns the upstream provider connection. A consuming application puts
+its authenticated client transport and domain workflow in front of the stable
+Archipel port.
+
+```mermaid
+flowchart LR
+  Client[Browser or native client\nMicrophone, playback, local chunk queue]
+  Gateway[Application realtime gateway\nAuth, tenant, sequencing, acknowledgements]
+  Persistence[Application persistence\nAudio, transcript, session state]
+  Factory[IntelligenceRealtimeSessionFactory]
+  Port[IntelligenceRealtimeSessionPort]
+  AiSdk[Vercel AI SDK 7\nRealtime provider model]
+  Socket[Generic server WebSocket transport]
+  Provider[OpenAI, Google, or xAI]
+
+  Client <--> Gateway
+  Gateway <--> Persistence
+  Gateway <--> Port
+  Factory --> Port
+  Port --> AiSdk
+  AiSdk --> Socket
+  Socket <--> Provider
+```
+
+Provider-specific token creation, event serialization, event parsing, and
+model selection use the Vercel AI SDK provider packages. Archipel uses the
+generic `ws` transport to open the WebSocket configuration returned by the AI
+SDK model adapter. It does not install the official OpenAI SDK, expose an
+OpenAI client, or make provider-specific calls outside the AI SDK adapter.
+
+The AI SDK realtime model contract is experimental. Archipel isolates that
+contract behind its own `/realtime` entry point so application code depends on
+the normalized ports and events. The `raw` field on server events remains an
+escape hatch for audited provider-specific diagnostics.
+
+### Responsibility boundary
+
+| Layer | Owns |
+| --- | --- |
+| Client | Microphone permission, capture, playback, local durable chunk queue, UI state |
+| Application | Authentication, authorization, tenant context, client WebSocket, durable audio/transcript state, tool execution, reconnect policy |
+| Archipel | Provider configuration, short-lived provider token, upstream WebSocket, event normalization, bounded queues, metering emission, shutdown |
+| Provider | Model conversation state and provider-native usage reporting |
+
+Archipel never exposes provider credentials to the client and does not own the
+application's audio recording or consultation record.
+
+---
+
+## Installation
+
+Install Archipel, the AI SDK core, and only the provider packages used by the
+application:
+
+```bash
+yarn add @breadstone/archipel-platform-intelligence ai
+
+# Add one or more Vercel AI SDK providers
+yarn add @ai-sdk/openai
+yarn add @ai-sdk/google
+yarn add @ai-sdk/xai
+```
+
+| Provider | Vercel AI SDK package | Archipel loader |
+| --- | --- | --- |
+| OpenAI | `@ai-sdk/openai` | `loadOpenAIRealtimeModel` |
+| Google Gemini Live | `@ai-sdk/google` | `loadGoogleRealtimeModel` |
+| xAI / Grok Voice | `@ai-sdk/xai` | `loadGrokRealtimeModel` |
+
+Do not add the official `openai` package for this integration.
+
+---
+
+## Register Realtime Providers
+
+Register realtime loaders under the existing `IntelligenceModule`. The root
+package stays provider-neutral; each loader is imported from a tree-shakable
+provider subpath.
+
+```typescript
+import { Module } from '@nestjs/common';
+import {
+  IntelligenceModule,
+  IntelligenceProviderNames,
+} from '@breadstone/archipel-platform-intelligence';
+import { loadGoogleRealtimeModel } from '@breadstone/archipel-platform-intelligence/providers/google';
+import { loadGrokRealtimeModel } from '@breadstone/archipel-platform-intelligence/providers/grok';
+import { loadOpenAIRealtimeModel } from '@breadstone/archipel-platform-intelligence/providers/openai';
+import { ApplicationRealtimeMeteringAdapter } from './ApplicationRealtimeMeteringAdapter';
+
+@Module({
+  imports: [
+    IntelligenceModule.register({
+      // Use false when this module is realtime-only. Applications that also
+      // register text-generation loaders can retain the default eager check.
+      validateOnModuleInit: false,
+      realtime: {
+        providerLoaders: {
+          [IntelligenceProviderNames.OpenAI]: loadOpenAIRealtimeModel,
+          [IntelligenceProviderNames.Google]: loadGoogleRealtimeModel,
+          [IntelligenceProviderNames.Grok]: loadGrokRealtimeModel,
+        },
+        meteringAdapter: ApplicationRealtimeMeteringAdapter,
+        meteringIntervalMs: 30_000,
+      },
+    }),
+  ],
+})
+export class RealtimeIntelligenceModule {}
+```
+
+Configure the default provider and its credentials through the normal Archipel
+configuration registry:
+
+```dotenv
+INTELLIGENCE_PROVIDER=openai
+OPENAI_API_KEY=sk-...
+
+# Optional when every createSession() call supplies its model
+INTELLIGENCE_MODEL=gpt-realtime
+```
+
+`INTELLIGENCE_API_KEY` can provide a global override. Google accepts
+`GOOGLE_API_KEY` or `GEMINI_API_KEY`; xAI accepts `GROK_API_KEY` or
+`XAI_API_KEY`.
+
+`validateOnModuleInit` controls eager text-model initialization. It does not
+disable realtime configuration validation when a session is created. If the
+same module also uses text generation, register the normal language-model
+loaders and leave validation enabled.
+
+The provider and API key still come from the normal intelligence
+configuration. `createSession()` may override the provider and model per
+logical session.
+
+---
+
+## Create and Connect a Session
+
+`createSession()` validates the application-owned session identifier, resolves
+the configured provider, loads the AI SDK realtime model, and returns an idle
+port. `connect()` then creates the short-lived provider token, opens the
+upstream WebSocket, and applies the session configuration.
+
+```typescript
+import { Injectable } from '@nestjs/common';
+import { IntelligenceProviderNames } from '@breadstone/archipel-platform-intelligence';
+import {
+  IntelligenceRealtimeSessionFactory,
+  type IntelligenceRealtimeServerEvent,
+  type IntelligenceRealtimeSessionPort,
+} from '@breadstone/archipel-platform-intelligence/realtime';
+
+@Injectable()
+export class RealtimeConsultationCoordinator {
+  public constructor(
+    private readonly _sessions: IntelligenceRealtimeSessionFactory,
+    private readonly _events: ConsultationEventStore,
+  ) {}
+
+  public async start(consultationId: string): Promise<IntelligenceRealtimeSessionPort> {
+    const session = await this._sessions.createSession({
+      sessionId: consultationId,
+      provider: IntelligenceProviderNames.OpenAI,
+      model: 'gpt-realtime',
+      metadata: {
+        feature: 'consultation',
+        consultationId,
+      },
+      sessionConfiguration: {
+        instructions: 'Transcribe the discussion and suggest concise next questions.',
+        inputAudioFormat: {
+          type: 'audio/pcm',
+          rate: 24_000,
+        },
+        inputAudioTranscription: {},
+        outputModalities: ['text', 'audio'],
+        turnDetection: {
+          type: 'server-vad',
+        },
+      },
+      onEvent: async (event: IntelligenceRealtimeServerEvent): Promise<void> => {
+        await this._events.persistThenPublish(consultationId, event);
+      },
+      onError: async (error: Error): Promise<void> => {
+        await this._events.persistError(consultationId, error);
+      },
+      onClose: async (event): Promise<void> => {
+        await this._events.persistClose(consultationId, event);
+      },
+    });
+
+    await session.connect();
+
+    return session;
+  }
+}
+```
+
+The application must retain the returned port for the lifetime of its client
+gateway connection. The factory tracks ports internally for resource limits
+and Nest shutdown, but deliberately does not act as an application session
+repository.
+
+### Two identifiers
+
+| Identifier | Lifetime | Purpose |
+| --- | --- | --- |
+| `sessionId` | Application-defined logical conversation | Persistence, authorization, billing account, recovery |
+| `connectionId` | One Archipel/provider WebSocket | Idempotency, audit correlation, distinguishing reconnects |
+
+A reconnect keeps the logical `sessionId` and creates a new `connectionId`.
+
+---
+
+## Forward Audio Live and Durably
+
+`appendAudio()` accepts one base64-encoded fragment and forwards it immediately
+to the active provider input buffer. A fragment may contain at most 1,500,000
+base64 characters. The configured sample rate and format must match the bytes
+produced by the client.
+
+Use a small, ordered application envelope around the Archipel payload:
+
+```typescript
+interface IClientAudioChunk {
+  readonly chunkId: string;
+  readonly sequence: number;
+  readonly audio: string;
+}
+```
+
+For each fragment:
+
+1. The client writes the fragment to its local durable queue before sending.
+2. The application authenticates the socket and validates the logical session.
+3. The application persists `(sessionId, chunkId, sequence, audio)`
+   idempotently.
+4. The application calls `session.appendAudio(audio)` immediately after that
+   small persistence operation.
+5. The application acknowledges the chunk only after durable persistence and
+   successful forwarding.
+6. The client removes the locally queued fragment after acknowledgement.
+
+This is persist-per-fragment, not record-for-an-hour-then-upload. If the client
+or network fails, only unacknowledged fragments remain to be retried.
+
+Archipel intentionally does not define `chunkId`, acknowledgements, IndexedDB,
+or the application WebSocket protocol. Those belong to the consuming domain;
+the upstream provider contract only needs the ordered base64 audio.
+
+### Turn completion
+
+With server voice-activity detection, the provider determines turn boundaries.
+For push-to-talk, disable automatic turn detection and commit explicitly:
+
+```typescript
+const session = await this._sessions.createSession({
+  sessionId: consultationId,
+  model: 'gpt-realtime',
+  sessionConfiguration: {
+    inputAudioFormat: { type: 'audio/pcm', rate: 24_000 },
+    turnDetection: null,
+  },
+});
+
+await session.connect();
+await session.appendAudio(audioChunk);
+await session.commitAudio();
+await session.requestResponse();
+```
+
+Use `clearAudio()` to discard the uncommitted provider input buffer.
+
+---
+
+## Choose the Interaction Mode
+
+The application controls whether the model speaks. There is no separate
+provider strategy or SDK for a silent assistant.
+
+| Mode | Session behavior | Application behavior |
+| --- | --- | --- |
+| Spoken assistant | `outputModalities: ['text', 'audio']` | Stream `audio-delta` to playback and show transcript |
+| Silent copilot | `outputModalities: ['text']` | Show suggestions, questions, or tasks; do not request/play audio |
+| Push-to-talk | `turnDetection: null` | Commit audio and request a response explicitly |
+| Passive capture | Enable input transcription and avoid response requests | Persist transcript events; verify provider-specific automatic-response behavior |
+
+Providers do not all support identical voices, formats, transcription models,
+or automatic-response controls. Keep the common workflow on Archipel contracts
+and place unavoidable provider-specific values in `providerOptions`.
+
+Text turns use the same session:
+
+```typescript
+await session.sendText('Summarize the unanswered requirements.');
+await session.cancelResponse();
+```
+
+---
+
+## Handle Normalized Events
+
+`onEvent` receives events in provider wire order for the current connection.
+The most relevant event families are:
+
+| Family | Examples | Typical application action |
+| --- | --- | --- |
+| Speech | `speech-started`, `speech-stopped`, `audio-committed` | Update listening state and turn boundaries |
+| Input transcript | `input-transcription-completed` | Persist the final customer/seller transcript item |
+| Text output | `text-delta`, `text-done` | Render suggestions or assistant replies |
+| Audio output | `audio-delta`, `audio-done` | Stream base64 audio to the client player |
+| Audio transcript | `audio-transcript-delta`, `audio-transcript-done` | Persist spoken assistant text |
+| Tools | `function-call-arguments-delta`, `function-call-arguments-done` | Validate, authorize, execute, and return output |
+| Lifecycle | `response-created`, `response-done`, `error` | Track response state and diagnostics |
+
+Persist final events and enough ordered deltas to recover the UI. Treat `raw`
+as sensitive provider data: redact it according to the application's AI audit
+and privacy policy before storing it.
+
+### Tool calls
+
+Tools registered on `IntelligenceModule` are converted to realtime function
+definitions through the AI SDK when `sessionConfiguration.tools` is omitted.
+Use `activeToolNames` as an allow-list per session.
+
+When `function-call-arguments-done` arrives, the application remains
+responsible for parsing and validating the arguments, checking tenant and user
+permissions, invoking the domain operation, and returning the result:
+
+```typescript
+await session.addToolOutput(event.callId, toolResult, event.name);
+```
+
+Never execute provider-supplied arguments without the normal application
+authorization and validation path.
+
+---
+
+## Metering, Tokens, and AI Audit
+
+Realtime traffic does not pass through a request-scoped HTTP token
+interceptor. `IntelligenceRealtimeMeteringPort` is the dedicated accounting
+boundary for long-lived sessions.
+
+```typescript
+import { Injectable } from '@nestjs/common';
+import {
+  IntelligenceRealtimeMeteringPort,
+  type IntelligenceRealtimeMeteringEvent,
+} from '@breadstone/archipel-platform-intelligence/realtime';
+
+@Injectable()
+export class ApplicationRealtimeMeteringAdapter extends IntelligenceRealtimeMeteringPort {
+  public constructor(private readonly _ledger: AiUsageLedger) {
+    super();
+  }
+
+  public override async report(event: IntelligenceRealtimeMeteringEvent): Promise<void> {
+    await this._ledger.recordRealtimeEventAtomically({
+      idempotencyKey: event.eventId,
+      event,
+    });
+  }
+}
+```
+
+The application transaction should:
+
+1. insert the metering event with a unique constraint on `eventId`;
+2. debit the application's token or credit ledger when the event is billable;
+3. append the normalized AI audit entry;
+4. commit all three changes together.
+
+| Event | Recommended use |
+| --- | --- |
+| `session-started` | Open one auditable provider connection |
+| `billing-interval` | Provider-independent commercial charging every 30 seconds and for the final partial interval |
+| `provider-usage` | Exact input/output, text/audio, cached, and total token reconciliation when supplied |
+| `session-ended` | Final duration, close code, and close reason |
+
+Choose one commercial charging basis. A common strategy debits credits from
+`billing-interval` and stores `provider-usage` only for provider-cost
+reconciliation. Do not debit both event types unless the pricing model
+intentionally contains both dimensions.
+
+The default metering adapter is a no-op. Production applications that charge
+users or require AI audit completeness must register a durable adapter. Keep
+that adapter fast and local—normally a database transaction or transactional
+outbox. Archipel retries a failed adapter write twice and then reports a
+realtime error; it cannot make a remote application datastore durable on the
+application's behalf.
+
+---
+
+## Limits and Backpressure
+
+Module-level limits are normalized when `IntelligenceModule.register()` runs:
+
+| Option | Default | Behavior |
+| --- | --- | --- |
+| `maxActiveSessions` | `1_000` | Maximum upstream sessions per Nest module instance |
+| `maxPendingEvents` | `512` | Maximum inbound, outbound, or metering queue depth per session |
+| `maxBufferedBytes` | `4 MiB` | Maximum provider WebSocket backpressure buffer |
+| `connectionTimeoutMs` | `5_000` | Provider token/setup and socket-open timeout; capped at 5 seconds |
+| `closeTimeoutMs` | `5_000` | Graceful close and metering-flush timeout; capped at 5 seconds |
+| `meteringIntervalMs` | `30_000` | Provider-independent billing interval |
+| `tokenRequestMaxRetries` | `2` | Additional transient token-creation attempts; capped at 2 |
+
+Limits apply per process, not across a cluster. Enforce tenant and account
+quotas in the application before creating a session. When the client sends
+faster than `appendAudio()` can drain, apply backpressure at the client gateway
+instead of building another unbounded queue.
+
+---
+
+## Failure and Recovery
+
+Archipel retries transient provider-token creation, but it never silently
+reconnects a dropped conversation. A new provider WebSocket has new
+conversation state and a new `connectionId`; automatic reconnect would hide
+that semantic break.
+
+Handle failures by phase through `IntelligenceRealtimeError`:
+
+| Phase | Meaning |
+| --- | --- |
+| `configuration` | Invalid provider or session configuration |
+| `token` | Short-lived provider credential creation failed |
+| `connect` | Upstream WebSocket could not open |
+| `send` | Outbound serialization, state, queue, or socket failure |
+| `receive` | Provider event parsing or inbound queue failure |
+| `metering` | Application metering adapter failed |
+| `close` | Graceful shutdown failed |
+
+The error also contains `provider`, `model`, `sessionId`, and `isRetryable`.
+Connect and send methods reject their promises; asynchronous receive and
+metering failures arrive through `onError`. `onClose` is the terminal upstream
+state.
+
+Recovery belongs to the application:
+
+1. stop acknowledging new client chunks for the failed connection;
+2. keep unacknowledged chunks in the client's durable queue;
+3. persist the terminal connection state and close reason;
+4. create a new Archipel session with the same logical `sessionId`;
+5. reconstruct safe context from the durable transcript or a compact summary;
+6. resume with a new `connectionId` and explicit audit entry.
+
+The normalized realtime contract does not promise lossless replay of arbitrary
+provider-native conversation history. Rebuild only the context supported by
+the selected provider and application policy.
+
+For horizontal deployments, route all frames for one active upstream
+connection to its owning process, for example through sticky routing or a
+distributed ownership record. Never depend on the factory's in-memory map as a
+cluster-wide registry.
+
+Nest shutdown calls `close()` for all owned sessions and waits for queued
+metering within the configured close timeout.
+
+---
+
+## Production Checklist
+
+- Provider loaders come only from `@ai-sdk/*`-backed Archipel subpaths.
+- Provider credentials remain in backend configuration.
+- The application authenticates and authorizes every client gateway message.
+- Each audio fragment has a logical chunk ID and sequence outside Archipel.
+- Client and backend persistence happen per fragment, not after the complete recording.
+- Transcript and final response events are persisted before client acknowledgement.
+- A durable `IntelligenceRealtimeMeteringPort` is registered.
+- `eventId` has a database uniqueness constraint.
+- Commercial interval charging and provider token reconciliation are not double-counted.
+- Backpressure and tenant quotas are bounded outside the per-process Archipel limits.
+- Reconnect creates a new connection and is visible in audit history.
+- Shutdown hooks are enabled in the Nest application.
+
+See the [platform-intelligence package page](../packages/platform-intelligence/)
+and the [realtime API reference](../packages/platform-intelligence/api/Class.IntelligenceRealtimeSessionFactory)
+for the complete contracts.

package/data/guides/ai-text-generation.md

@@ -8,6 +8,15 @@ order: 15
 
 This guide covers text generation with `platform-intelligence`: choosing an AI provider, configuring the module, generating completions, streaming responses, and registering class-based or provider-native AI SDK tools.
 
+For versioned feature prompts with typed inputs, structured output schemas,
+execution policy, and a NestJS executor, start with
+[Intelligence Prompts Warm-up](./intelligence-prompts-warm-up).
+
+For server-owned, bidirectional audio and text connections, use the separate
+[AI Realtime Voice](./ai-realtime-voice) guide. Realtime sessions have a
+different transport, lifecycle, persistence, and metering model from request-
+or response-scoped text generation.
+
 ---
 
 ## Installation

package/data/guides/health-indicators.md

@@ -72,7 +72,7 @@ All 12 platform libraries provide a health indicator via their `/health` subpath
 | `platform-queue` | `@breadstone/archipel-platform-queue/health` | `QueueHealthIndicator` | `queue` | Always `up` |
 | `platform-payments` | `@breadstone/archipel-platform-payments/health` | `PaymentHealthIndicator` | `payment` | Calls `PaymentClientPort.ping()` to verify provider connectivity |
 | `platform-esigning` | `@breadstone/archipel-platform-esigning/health` | `EsigningHealthIndicator` | `esigning` | Calls `EsigningClientPort.ping()` to verify provider connectivity, reports `providerId` |
-| `platform-intelligence` | `@breadstone/archipel-platform-intelligence/health` | `IntelligenceHealthIndicator` | `intelligence` | Checks `list().length > 0` on the capability registry, reports count |
+| `platform-intelligence` | `@breadstone/archipel-platform-intelligence/health` | `IntelligenceHealthIndicator` | `intelligence` | Resolves text configuration and reports registered realtime providers plus active session count |
 | `platform-analytics` | `@breadstone/archipel-platform-analytics/health` | `AnalyticsHealthIndicator` | `analytics` | Calls `AnalyticsClientPort.ping()` to verify provider readiness |
 | `platform-telemetry` | `@breadstone/archipel-platform-telemetry/health` | `TelemetryHealthIndicator` | `telemetry` | Checks `OtelSdkHolder.isInitialized` for SDK presence |
 | `platform-authentication` | `@breadstone/archipel-platform-authentication/health` | `AuthenticationHealthIndicator` | `authentication` | Always `up` |
@@ -89,7 +89,7 @@ All 12 platform libraries provide a health indicator via their `/health` subpath
 - `AnalyticsHealthIndicator` — calls `AnalyticsClientPort.ping()` (adapters can override for SDK readiness checks)
 
 **State checks** — inspect in-memory state (zero-cost, no I/O):
-- `IntelligenceHealthIndicator` — verifies at least one capability is registered
+- `IntelligenceHealthIndicator` — resolves configured provider/model data and reports process-local realtime provider and session state
 - `TelemetryHealthIndicator` — checks whether the OpenTelemetry SDK has been initialized
 - `McpHealthIndicator` — verifies at least one tool/resource/prompt handler is registered
 
@@ -153,7 +153,16 @@ export class HealthController {
     "mail": { "status": "up" },
     "payment": { "status": "up" },
     "caching": { "status": "up" },
-    "intelligence": { "status": "up", "registeredCapabilities": 3 },
+    "intelligence": {
+      "status": "up",
+      "provider": "openai",
+      "model": "gpt-4o-mini",
+      "realtime": {
+        "enabled": true,
+        "providers": ["openai", "google"],
+        "activeSessions": 2
+      }
+    },
     "mcp": { "status": "up", "tools": 5, "resources": 2, "prompts": 1 },
     "telemetry": { "status": "up", "sdkInitialized": true }
   }

package/data/guides/index.md

@@ -58,10 +58,12 @@ Practical guides for working with Archipel packages in your NestJS application.
 
 ## AI & Extensibility
 
-| Guide                                      | What you'll learn                                                               |
-| ------------------------------------------ | ------------------------------------------------------------------------------- |
-| [AI Text Generation](./ai-text-generation) | Multi-provider text completions, capability registry, and intent orchestration. |
-| [MCP Server](./mcp-server)                 | Build MCP servers with tools, resources, and prompts for AI agent integration.  |
+| Guide                                    | What you'll learn                                                                                               |
+| ---------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
+| [Intelligence Prompts Warm-up](./intelligence-prompts-warm-up) | Define, bind, execute, validate, and test reusable text and structured prompts.                              |
+| [AI Text Generation](./ai-text-generation) | Multi-provider text completions, streaming, native AI SDK tools, and per-request model options.                   |
+| [AI Realtime Voice](./ai-realtime-voice) | Server-owned live audio/text sessions, interaction modes, durable streaming, metering, and explicit recovery. |
+| [MCP Server](./mcp-server)               | Build MCP servers with tools, resources, and prompts for AI agent integration.                                |
 
 ## Platform Internals