@breadstone/archipel-mcp 0.0.52 → 0.0.53
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/data/guides/ai-realtime-voice.md +506 -0
- package/data/guides/ai-text-generation.md +5 -0
- package/data/guides/health-indicators.md +12 -3
- package/data/guides/index.md +5 -4
- package/data/packages/platform-health/index.md +1 -1
- package/data/packages/platform-intelligence/api/Class.IntelligenceHealthIndicator.md +6 -5
- package/data/packages/platform-intelligence/api/Class.IntelligenceModule.md +2 -2
- package/data/packages/platform-intelligence/api/Class.IntelligenceRealtimeError.md +110 -0
- package/data/packages/platform-intelligence/api/Class.IntelligenceRealtimeMeteringPort.md +47 -0
- package/data/packages/platform-intelligence/api/Class.IntelligenceRealtimeSessionFactory.md +129 -0
- package/data/packages/platform-intelligence/api/Class.IntelligenceRealtimeSessionPort.md +308 -0
- package/data/packages/platform-intelligence/api/Class.IntelligenceTextGenerator.md +6 -6
- package/data/packages/platform-intelligence/api/Function.loadGoogleRealtimeModel.md +25 -0
- package/data/packages/platform-intelligence/api/Function.loadGrokRealtimeModel.md +25 -0
- package/data/packages/platform-intelligence/api/Function.loadOpenAIRealtimeModel.md +25 -0
- package/data/packages/platform-intelligence/api/Interface.IIntelligenceModuleOptions.md +18 -6
- package/data/packages/platform-intelligence/api/Interface.IIntelligenceRealtimeAudioFormat.md +34 -0
- package/data/packages/platform-intelligence/api/Interface.IIntelligenceRealtimeCloseEvent.md +40 -0
- package/data/packages/platform-intelligence/api/Interface.IIntelligenceRealtimeModuleOptions.md +120 -0
- package/data/packages/platform-intelligence/api/Interface.IIntelligenceRealtimeSessionConfiguration.md +135 -0
- package/data/packages/platform-intelligence/api/Interface.IIntelligenceRealtimeSessionOptions.md +148 -0
- package/data/packages/platform-intelligence/api/Interface.IIntelligenceRealtimeToolDefinition.md +58 -0
- package/data/packages/platform-intelligence/api/Interface.IIntelligenceRealtimeTranscriptionConfiguration.md +46 -0
- package/data/packages/platform-intelligence/api/Interface.IIntelligenceRealtimeTurnDetectionConfiguration.md +58 -0
- package/data/packages/platform-intelligence/api/Interface.IIntelligenceRealtimeUsage.md +103 -0
- package/data/packages/platform-intelligence/api/TypeAlias.IntelligenceRealtimeClientEvent.md +64 -0
- package/data/packages/platform-intelligence/api/TypeAlias.IntelligenceRealtimeMetadataValue.md +14 -0
- package/data/packages/platform-intelligence/api/TypeAlias.IntelligenceRealtimeMeteringEvent.md +44 -0
- package/data/packages/platform-intelligence/api/TypeAlias.IntelligenceRealtimeModelLoader.md +25 -0
- package/data/packages/platform-intelligence/api/TypeAlias.IntelligenceRealtimeModelLoaders.md +14 -0
- package/data/packages/platform-intelligence/api/TypeAlias.IntelligenceRealtimeServerEvent.md +159 -0
- package/data/packages/platform-intelligence/api/TypeAlias.IntelligenceRealtimeStatus.md +14 -0
- package/data/packages/platform-intelligence/api/index.md +23 -0
- package/data/packages/platform-intelligence/index.md +133 -7
- package/data/patterns/index.md +1 -0
- package/data/patterns/realtime-session-pattern.md +309 -0
- package/package.json +1 -1
|
@@ -0,0 +1,506 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: AI Realtime Voice
|
|
3
|
+
description: Build server-owned, live audio and text sessions with Vercel AI SDK providers, durable application gateways, metering, and explicit recovery.
|
|
4
|
+
order: 27
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
# AI Realtime Voice
|
|
8
|
+
|
|
9
|
+
This guide covers long-lived, bidirectional AI sessions with
|
|
10
|
+
`platform-intelligence`: provider registration, live audio forwarding,
|
|
11
|
+
normalized events, silent and spoken interaction modes, tools, metering, and
|
|
12
|
+
failure recovery.
|
|
13
|
+
|
|
14
|
+
Realtime sessions are not hour-long uploads. The client and application move
|
|
15
|
+
small audio fragments continuously while each layer persists only the data it
|
|
16
|
+
owns. This preserves live interaction without making the complete recording a
|
|
17
|
+
single point of failure.
|
|
18
|
+
|
|
19
|
+
For the reusable ownership and recovery rules behind this integration, see the
|
|
20
|
+
[Realtime Session Pattern](../patterns/realtime-session-pattern).
|
|
21
|
+
|
|
22
|
+
---
|
|
23
|
+
|
|
24
|
+
## Architecture and SDK Boundary
|
|
25
|
+
|
|
26
|
+
Archipel owns the upstream provider connection. A consuming application puts
|
|
27
|
+
its authenticated client transport and domain workflow in front of the stable
|
|
28
|
+
Archipel port.
|
|
29
|
+
|
|
30
|
+
```mermaid
|
|
31
|
+
flowchart LR
|
|
32
|
+
Client[Browser or native client\nMicrophone, playback, local chunk queue]
|
|
33
|
+
Gateway[Application realtime gateway\nAuth, tenant, sequencing, acknowledgements]
|
|
34
|
+
Persistence[Application persistence\nAudio, transcript, session state]
|
|
35
|
+
Factory[IntelligenceRealtimeSessionFactory]
|
|
36
|
+
Port[IntelligenceRealtimeSessionPort]
|
|
37
|
+
AiSdk[Vercel AI SDK 7\nRealtime provider model]
|
|
38
|
+
Socket[Generic server WebSocket transport]
|
|
39
|
+
Provider[OpenAI, Google, or xAI]
|
|
40
|
+
|
|
41
|
+
Client <--> Gateway
|
|
42
|
+
Gateway <--> Persistence
|
|
43
|
+
Gateway <--> Port
|
|
44
|
+
Factory --> Port
|
|
45
|
+
Port --> AiSdk
|
|
46
|
+
AiSdk --> Socket
|
|
47
|
+
Socket <--> Provider
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
Provider-specific token creation, event serialization, event parsing, and
|
|
51
|
+
model selection use the Vercel AI SDK provider packages. Archipel uses the
|
|
52
|
+
generic `ws` transport to open the WebSocket configuration returned by the AI
|
|
53
|
+
SDK model adapter. It does not install the official OpenAI SDK, expose an
|
|
54
|
+
OpenAI client, or make provider-specific calls outside the AI SDK adapter.
|
|
55
|
+
|
|
56
|
+
The AI SDK realtime model contract is experimental. Archipel isolates that
|
|
57
|
+
contract behind its own `/realtime` entry point so application code depends on
|
|
58
|
+
the normalized ports and events. The `raw` field on server events remains an
|
|
59
|
+
escape hatch for audited provider-specific diagnostics.
|
|
60
|
+
|
|
61
|
+
### Responsibility boundary
|
|
62
|
+
|
|
63
|
+
| Layer | Owns |
|
|
64
|
+
| --- | --- |
|
|
65
|
+
| Client | Microphone permission, capture, playback, local durable chunk queue, UI state |
|
|
66
|
+
| Application | Authentication, authorization, tenant context, client WebSocket, durable audio/transcript state, tool execution, reconnect policy |
|
|
67
|
+
| Archipel | Provider configuration, short-lived provider token, upstream WebSocket, event normalization, bounded queues, metering emission, shutdown |
|
|
68
|
+
| Provider | Model conversation state and provider-native usage reporting |
|
|
69
|
+
|
|
70
|
+
Archipel never exposes provider credentials to the client and does not own the
|
|
71
|
+
application's audio recording or consultation record.
|
|
72
|
+
|
|
73
|
+
---
|
|
74
|
+
|
|
75
|
+
## Installation
|
|
76
|
+
|
|
77
|
+
Install Archipel, the AI SDK core, and only the provider packages used by the
|
|
78
|
+
application:
|
|
79
|
+
|
|
80
|
+
```bash
|
|
81
|
+
yarn add @breadstone/archipel-platform-intelligence ai
|
|
82
|
+
|
|
83
|
+
# Add one or more Vercel AI SDK providers
|
|
84
|
+
yarn add @ai-sdk/openai
|
|
85
|
+
yarn add @ai-sdk/google
|
|
86
|
+
yarn add @ai-sdk/xai
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
| Provider | Vercel AI SDK package | Archipel loader |
|
|
90
|
+
| --- | --- | --- |
|
|
91
|
+
| OpenAI | `@ai-sdk/openai` | `loadOpenAIRealtimeModel` |
|
|
92
|
+
| Google Gemini Live | `@ai-sdk/google` | `loadGoogleRealtimeModel` |
|
|
93
|
+
| xAI / Grok Voice | `@ai-sdk/xai` | `loadGrokRealtimeModel` |
|
|
94
|
+
|
|
95
|
+
Do not add the official `openai` package for this integration.
|
|
96
|
+
|
|
97
|
+
---
|
|
98
|
+
|
|
99
|
+
## Register Realtime Providers
|
|
100
|
+
|
|
101
|
+
Register realtime loaders under the existing `IntelligenceModule`. The root
|
|
102
|
+
package stays provider-neutral; each loader is imported from a tree-shakable
|
|
103
|
+
provider subpath.
|
|
104
|
+
|
|
105
|
+
```typescript
|
|
106
|
+
import { Module } from '@nestjs/common';
|
|
107
|
+
import {
|
|
108
|
+
IntelligenceModule,
|
|
109
|
+
IntelligenceProviderNames,
|
|
110
|
+
} from '@breadstone/archipel-platform-intelligence';
|
|
111
|
+
import { loadGoogleRealtimeModel } from '@breadstone/archipel-platform-intelligence/providers/google';
|
|
112
|
+
import { loadGrokRealtimeModel } from '@breadstone/archipel-platform-intelligence/providers/grok';
|
|
113
|
+
import { loadOpenAIRealtimeModel } from '@breadstone/archipel-platform-intelligence/providers/openai';
|
|
114
|
+
import { ApplicationRealtimeMeteringAdapter } from './ApplicationRealtimeMeteringAdapter';
|
|
115
|
+
|
|
116
|
+
@Module({
|
|
117
|
+
imports: [
|
|
118
|
+
IntelligenceModule.register({
|
|
119
|
+
// Use false when this module is realtime-only. Applications that also
|
|
120
|
+
// register text-generation loaders can retain the default eager check.
|
|
121
|
+
validateOnModuleInit: false,
|
|
122
|
+
realtime: {
|
|
123
|
+
providerLoaders: {
|
|
124
|
+
[IntelligenceProviderNames.OpenAI]: loadOpenAIRealtimeModel,
|
|
125
|
+
[IntelligenceProviderNames.Google]: loadGoogleRealtimeModel,
|
|
126
|
+
[IntelligenceProviderNames.Grok]: loadGrokRealtimeModel,
|
|
127
|
+
},
|
|
128
|
+
meteringAdapter: ApplicationRealtimeMeteringAdapter,
|
|
129
|
+
meteringIntervalMs: 30_000,
|
|
130
|
+
},
|
|
131
|
+
}),
|
|
132
|
+
],
|
|
133
|
+
})
|
|
134
|
+
export class RealtimeIntelligenceModule {}
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
Configure the default provider and its credentials through the normal Archipel
|
|
138
|
+
configuration registry:
|
|
139
|
+
|
|
140
|
+
```dotenv
|
|
141
|
+
INTELLIGENCE_PROVIDER=openai
|
|
142
|
+
OPENAI_API_KEY=sk-...
|
|
143
|
+
|
|
144
|
+
# Optional when every createSession() call supplies its model
|
|
145
|
+
INTELLIGENCE_MODEL=gpt-realtime
|
|
146
|
+
```
|
|
147
|
+
|
|
148
|
+
`INTELLIGENCE_API_KEY` can provide a global override. Google accepts
|
|
149
|
+
`GOOGLE_API_KEY` or `GEMINI_API_KEY`; xAI accepts `GROK_API_KEY` or
|
|
150
|
+
`XAI_API_KEY`.
|
|
151
|
+
|
|
152
|
+
`validateOnModuleInit` controls eager text-model initialization. It does not
|
|
153
|
+
disable realtime configuration validation when a session is created. If the
|
|
154
|
+
same module also uses text generation, register the normal language-model
|
|
155
|
+
loaders and leave validation enabled.
|
|
156
|
+
|
|
157
|
+
The provider and API key still come from the normal intelligence
|
|
158
|
+
configuration. `createSession()` may override the provider and model per
|
|
159
|
+
logical session.
|
|
160
|
+
|
|
161
|
+
---
|
|
162
|
+
|
|
163
|
+
## Create and Connect a Session
|
|
164
|
+
|
|
165
|
+
`createSession()` validates the application-owned session identifier, resolves
|
|
166
|
+
the configured provider, loads the AI SDK realtime model, and returns an idle
|
|
167
|
+
port. `connect()` then creates the short-lived provider token, opens the
|
|
168
|
+
upstream WebSocket, and applies the session configuration.
|
|
169
|
+
|
|
170
|
+
```typescript
|
|
171
|
+
import { Injectable } from '@nestjs/common';
|
|
172
|
+
import { IntelligenceProviderNames } from '@breadstone/archipel-platform-intelligence';
|
|
173
|
+
import {
|
|
174
|
+
IntelligenceRealtimeSessionFactory,
|
|
175
|
+
type IntelligenceRealtimeServerEvent,
|
|
176
|
+
type IntelligenceRealtimeSessionPort,
|
|
177
|
+
} from '@breadstone/archipel-platform-intelligence/realtime';
|
|
178
|
+
|
|
179
|
+
@Injectable()
|
|
180
|
+
export class RealtimeConsultationCoordinator {
|
|
181
|
+
public constructor(
|
|
182
|
+
private readonly _sessions: IntelligenceRealtimeSessionFactory,
|
|
183
|
+
private readonly _events: ConsultationEventStore,
|
|
184
|
+
) {}
|
|
185
|
+
|
|
186
|
+
public async start(consultationId: string): Promise<IntelligenceRealtimeSessionPort> {
|
|
187
|
+
const session = await this._sessions.createSession({
|
|
188
|
+
sessionId: consultationId,
|
|
189
|
+
provider: IntelligenceProviderNames.OpenAI,
|
|
190
|
+
model: 'gpt-realtime',
|
|
191
|
+
metadata: {
|
|
192
|
+
feature: 'consultation',
|
|
193
|
+
consultationId,
|
|
194
|
+
},
|
|
195
|
+
sessionConfiguration: {
|
|
196
|
+
instructions: 'Transcribe the discussion and suggest concise next questions.',
|
|
197
|
+
inputAudioFormat: {
|
|
198
|
+
type: 'audio/pcm',
|
|
199
|
+
rate: 24_000,
|
|
200
|
+
},
|
|
201
|
+
inputAudioTranscription: {},
|
|
202
|
+
outputModalities: ['text', 'audio'],
|
|
203
|
+
turnDetection: {
|
|
204
|
+
type: 'server-vad',
|
|
205
|
+
},
|
|
206
|
+
},
|
|
207
|
+
onEvent: async (event: IntelligenceRealtimeServerEvent): Promise<void> => {
|
|
208
|
+
await this._events.persistThenPublish(consultationId, event);
|
|
209
|
+
},
|
|
210
|
+
onError: async (error: Error): Promise<void> => {
|
|
211
|
+
await this._events.persistError(consultationId, error);
|
|
212
|
+
},
|
|
213
|
+
onClose: async (event): Promise<void> => {
|
|
214
|
+
await this._events.persistClose(consultationId, event);
|
|
215
|
+
},
|
|
216
|
+
});
|
|
217
|
+
|
|
218
|
+
await session.connect();
|
|
219
|
+
|
|
220
|
+
return session;
|
|
221
|
+
}
|
|
222
|
+
}
|
|
223
|
+
```
|
|
224
|
+
|
|
225
|
+
The application must retain the returned port for the lifetime of its client
|
|
226
|
+
gateway connection. The factory tracks ports internally for resource limits
|
|
227
|
+
and Nest shutdown, but deliberately does not act as an application session
|
|
228
|
+
repository.
|
|
229
|
+
|
|
230
|
+
### Two identifiers
|
|
231
|
+
|
|
232
|
+
| Identifier | Lifetime | Purpose |
|
|
233
|
+
| --- | --- | --- |
|
|
234
|
+
| `sessionId` | Application-defined logical conversation | Persistence, authorization, billing account, recovery |
|
|
235
|
+
| `connectionId` | One Archipel/provider WebSocket | Idempotency, audit correlation, distinguishing reconnects |
|
|
236
|
+
|
|
237
|
+
A reconnect keeps the logical `sessionId` and creates a new `connectionId`.
|
|
238
|
+
|
|
239
|
+
---
|
|
240
|
+
|
|
241
|
+
## Forward Audio Live and Durably
|
|
242
|
+
|
|
243
|
+
`appendAudio()` accepts one base64-encoded fragment and forwards it immediately
|
|
244
|
+
to the active provider input buffer. A fragment may contain at most 1,500,000
|
|
245
|
+
base64 characters. The configured sample rate and format must match the bytes
|
|
246
|
+
produced by the client.
|
|
247
|
+
|
|
248
|
+
Use a small, ordered application envelope around the Archipel payload:
|
|
249
|
+
|
|
250
|
+
```typescript
|
|
251
|
+
interface IClientAudioChunk {
|
|
252
|
+
readonly chunkId: string;
|
|
253
|
+
readonly sequence: number;
|
|
254
|
+
readonly audio: string;
|
|
255
|
+
}
|
|
256
|
+
```
|
|
257
|
+
|
|
258
|
+
For each fragment:
|
|
259
|
+
|
|
260
|
+
1. The client writes the fragment to its local durable queue before sending.
|
|
261
|
+
2. The application authenticates the socket and validates the logical session.
|
|
262
|
+
3. The application persists `(sessionId, chunkId, sequence, audio)`
|
|
263
|
+
idempotently.
|
|
264
|
+
4. The application calls `session.appendAudio(audio)` immediately after that
|
|
265
|
+
small persistence operation.
|
|
266
|
+
5. The application acknowledges the chunk only after durable persistence and
|
|
267
|
+
successful forwarding.
|
|
268
|
+
6. The client removes the locally queued fragment after acknowledgement.
|
|
269
|
+
|
|
270
|
+
This is persist-per-fragment, not record-for-an-hour-then-upload. If the client
|
|
271
|
+
or network fails, only unacknowledged fragments remain to be retried.
|
|
272
|
+
|
|
273
|
+
Archipel intentionally does not define `chunkId`, acknowledgements, IndexedDB,
|
|
274
|
+
or the application WebSocket protocol. Those belong to the consuming domain;
|
|
275
|
+
the upstream provider contract only needs the ordered base64 audio.
|
|
276
|
+
|
|
277
|
+
### Turn completion
|
|
278
|
+
|
|
279
|
+
With server voice-activity detection, the provider determines turn boundaries.
|
|
280
|
+
For push-to-talk, disable automatic turn detection and commit explicitly:
|
|
281
|
+
|
|
282
|
+
```typescript
|
|
283
|
+
const session = await this._sessions.createSession({
|
|
284
|
+
sessionId: consultationId,
|
|
285
|
+
model: 'gpt-realtime',
|
|
286
|
+
sessionConfiguration: {
|
|
287
|
+
inputAudioFormat: { type: 'audio/pcm', rate: 24_000 },
|
|
288
|
+
turnDetection: null,
|
|
289
|
+
},
|
|
290
|
+
});
|
|
291
|
+
|
|
292
|
+
await session.connect();
|
|
293
|
+
await session.appendAudio(audioChunk);
|
|
294
|
+
await session.commitAudio();
|
|
295
|
+
await session.requestResponse();
|
|
296
|
+
```
|
|
297
|
+
|
|
298
|
+
Use `clearAudio()` to discard the uncommitted provider input buffer.
|
|
299
|
+
|
|
300
|
+
---
|
|
301
|
+
|
|
302
|
+
## Choose the Interaction Mode
|
|
303
|
+
|
|
304
|
+
The application controls whether the model speaks. There is no separate
|
|
305
|
+
provider strategy or SDK for a silent assistant.
|
|
306
|
+
|
|
307
|
+
| Mode | Session behavior | Application behavior |
|
|
308
|
+
| --- | --- | --- |
|
|
309
|
+
| Spoken assistant | `outputModalities: ['text', 'audio']` | Stream `audio-delta` to playback and show transcript |
|
|
310
|
+
| Silent copilot | `outputModalities: ['text']` | Show suggestions, questions, or tasks; do not request/play audio |
|
|
311
|
+
| Push-to-talk | `turnDetection: null` | Commit audio and request a response explicitly |
|
|
312
|
+
| Passive capture | Enable input transcription and avoid response requests | Persist transcript events; verify provider-specific automatic-response behavior |
|
|
313
|
+
|
|
314
|
+
Providers do not all support identical voices, formats, transcription models,
|
|
315
|
+
or automatic-response controls. Keep the common workflow on Archipel contracts
|
|
316
|
+
and place unavoidable provider-specific values in `providerOptions`.
|
|
317
|
+
|
|
318
|
+
Text turns use the same session:
|
|
319
|
+
|
|
320
|
+
```typescript
|
|
321
|
+
await session.sendText('Summarize the unanswered requirements.');
|
|
322
|
+
await session.cancelResponse();
|
|
323
|
+
```
|
|
324
|
+
|
|
325
|
+
---
|
|
326
|
+
|
|
327
|
+
## Handle Normalized Events
|
|
328
|
+
|
|
329
|
+
`onEvent` receives events in provider wire order for the current connection.
|
|
330
|
+
The most relevant event families are:
|
|
331
|
+
|
|
332
|
+
| Family | Examples | Typical application action |
|
|
333
|
+
| --- | --- | --- |
|
|
334
|
+
| Speech | `speech-started`, `speech-stopped`, `audio-committed` | Update listening state and turn boundaries |
|
|
335
|
+
| Input transcript | `input-transcription-completed` | Persist the final customer/seller transcript item |
|
|
336
|
+
| Text output | `text-delta`, `text-done` | Render suggestions or assistant replies |
|
|
337
|
+
| Audio output | `audio-delta`, `audio-done` | Stream base64 audio to the client player |
|
|
338
|
+
| Audio transcript | `audio-transcript-delta`, `audio-transcript-done` | Persist spoken assistant text |
|
|
339
|
+
| Tools | `function-call-arguments-delta`, `function-call-arguments-done` | Validate, authorize, execute, and return output |
|
|
340
|
+
| Lifecycle | `response-created`, `response-done`, `error` | Track response state and diagnostics |
|
|
341
|
+
|
|
342
|
+
Persist final events and enough ordered deltas to recover the UI. Treat `raw`
|
|
343
|
+
as sensitive provider data: redact it according to the application's AI audit
|
|
344
|
+
and privacy policy before storing it.
|
|
345
|
+
|
|
346
|
+
### Tool calls
|
|
347
|
+
|
|
348
|
+
Tools registered on `IntelligenceModule` are converted to realtime function
|
|
349
|
+
definitions through the AI SDK when `sessionConfiguration.tools` is omitted.
|
|
350
|
+
Use `activeToolNames` as an allow-list per session.
|
|
351
|
+
|
|
352
|
+
When `function-call-arguments-done` arrives, the application remains
|
|
353
|
+
responsible for parsing and validating the arguments, checking tenant and user
|
|
354
|
+
permissions, invoking the domain operation, and returning the result:
|
|
355
|
+
|
|
356
|
+
```typescript
|
|
357
|
+
await session.addToolOutput(event.callId, toolResult, event.name);
|
|
358
|
+
```
|
|
359
|
+
|
|
360
|
+
Never execute provider-supplied arguments without the normal application
|
|
361
|
+
authorization and validation path.
|
|
362
|
+
|
|
363
|
+
---
|
|
364
|
+
|
|
365
|
+
## Metering, Tokens, and AI Audit
|
|
366
|
+
|
|
367
|
+
Realtime traffic does not pass through a request-scoped HTTP token
|
|
368
|
+
interceptor. `IntelligenceRealtimeMeteringPort` is the dedicated accounting
|
|
369
|
+
boundary for long-lived sessions.
|
|
370
|
+
|
|
371
|
+
```typescript
|
|
372
|
+
import { Injectable } from '@nestjs/common';
|
|
373
|
+
import {
|
|
374
|
+
IntelligenceRealtimeMeteringPort,
|
|
375
|
+
type IntelligenceRealtimeMeteringEvent,
|
|
376
|
+
} from '@breadstone/archipel-platform-intelligence/realtime';
|
|
377
|
+
|
|
378
|
+
@Injectable()
|
|
379
|
+
export class ApplicationRealtimeMeteringAdapter extends IntelligenceRealtimeMeteringPort {
|
|
380
|
+
public constructor(private readonly _ledger: AiUsageLedger) {
|
|
381
|
+
super();
|
|
382
|
+
}
|
|
383
|
+
|
|
384
|
+
public override async report(event: IntelligenceRealtimeMeteringEvent): Promise<void> {
|
|
385
|
+
await this._ledger.recordRealtimeEventAtomically({
|
|
386
|
+
idempotencyKey: event.eventId,
|
|
387
|
+
event,
|
|
388
|
+
});
|
|
389
|
+
}
|
|
390
|
+
}
|
|
391
|
+
```
|
|
392
|
+
|
|
393
|
+
The application transaction should:
|
|
394
|
+
|
|
395
|
+
1. insert the metering event with a unique constraint on `eventId`;
|
|
396
|
+
2. debit the application's token or credit ledger when the event is billable;
|
|
397
|
+
3. append the normalized AI audit entry;
|
|
398
|
+
4. commit all three changes together.
|
|
399
|
+
|
|
400
|
+
| Event | Recommended use |
|
|
401
|
+
| --- | --- |
|
|
402
|
+
| `session-started` | Open one auditable provider connection |
|
|
403
|
+
| `billing-interval` | Provider-independent commercial charging every 30 seconds and for the final partial interval |
|
|
404
|
+
| `provider-usage` | Exact input/output, text/audio, cached, and total token reconciliation when supplied |
|
|
405
|
+
| `session-ended` | Final duration, close code, and close reason |
|
|
406
|
+
|
|
407
|
+
Choose one commercial charging basis. A common strategy debits credits from
|
|
408
|
+
`billing-interval` and stores `provider-usage` only for provider-cost
|
|
409
|
+
reconciliation. Do not debit both event types unless the pricing model
|
|
410
|
+
intentionally contains both dimensions.
|
|
411
|
+
|
|
412
|
+
The default metering adapter is a no-op. Production applications that charge
|
|
413
|
+
users or require AI audit completeness must register a durable adapter. Keep
|
|
414
|
+
that adapter fast and local—normally a database transaction or transactional
|
|
415
|
+
outbox. Archipel retries a failed adapter write twice and then reports a
|
|
416
|
+
realtime error; it cannot make a remote application datastore durable on the
|
|
417
|
+
application's behalf.
|
|
418
|
+
|
|
419
|
+
---
|
|
420
|
+
|
|
421
|
+
## Limits and Backpressure
|
|
422
|
+
|
|
423
|
+
Module-level limits are normalized when `IntelligenceModule.register()` runs:
|
|
424
|
+
|
|
425
|
+
| Option | Default | Behavior |
|
|
426
|
+
| --- | --- | --- |
|
|
427
|
+
| `maxActiveSessions` | `1_000` | Maximum upstream sessions per Nest module instance |
|
|
428
|
+
| `maxPendingEvents` | `512` | Maximum inbound, outbound, or metering queue depth per session |
|
|
429
|
+
| `maxBufferedBytes` | `4 MiB` | Maximum provider WebSocket backpressure buffer |
|
|
430
|
+
| `connectionTimeoutMs` | `5_000` | Provider token/setup and socket-open timeout; capped at 5 seconds |
|
|
431
|
+
| `closeTimeoutMs` | `5_000` | Graceful close and metering-flush timeout; capped at 5 seconds |
|
|
432
|
+
| `meteringIntervalMs` | `30_000` | Provider-independent billing interval |
|
|
433
|
+
| `tokenRequestMaxRetries` | `2` | Additional transient token-creation attempts; capped at 2 |
|
|
434
|
+
|
|
435
|
+
Limits apply per process, not across a cluster. Enforce tenant and account
|
|
436
|
+
quotas in the application before creating a session. When the client sends
|
|
437
|
+
faster than `appendAudio()` can drain, apply backpressure at the client gateway
|
|
438
|
+
instead of building another unbounded queue.
|
|
439
|
+
|
|
440
|
+
---
|
|
441
|
+
|
|
442
|
+
## Failure and Recovery
|
|
443
|
+
|
|
444
|
+
Archipel retries transient provider-token creation, but it never silently
|
|
445
|
+
reconnects a dropped conversation. A new provider WebSocket has new
|
|
446
|
+
conversation state and a new `connectionId`; automatic reconnect would hide
|
|
447
|
+
that semantic break.
|
|
448
|
+
|
|
449
|
+
Handle failures by phase through `IntelligenceRealtimeError`:
|
|
450
|
+
|
|
451
|
+
| Phase | Meaning |
|
|
452
|
+
| --- | --- |
|
|
453
|
+
| `configuration` | Invalid provider or session configuration |
|
|
454
|
+
| `token` | Short-lived provider credential creation failed |
|
|
455
|
+
| `connect` | Upstream WebSocket could not open |
|
|
456
|
+
| `send` | Outbound serialization, state, queue, or socket failure |
|
|
457
|
+
| `receive` | Provider event parsing or inbound queue failure |
|
|
458
|
+
| `metering` | Application metering adapter failed |
|
|
459
|
+
| `close` | Graceful shutdown failed |
|
|
460
|
+
|
|
461
|
+
The error also contains `provider`, `model`, `sessionId`, and `isRetryable`.
|
|
462
|
+
Connect and send methods reject their promises; asynchronous receive and
|
|
463
|
+
metering failures arrive through `onError`. `onClose` is the terminal upstream
|
|
464
|
+
state.
|
|
465
|
+
|
|
466
|
+
Recovery belongs to the application:
|
|
467
|
+
|
|
468
|
+
1. stop acknowledging new client chunks for the failed connection;
|
|
469
|
+
2. keep unacknowledged chunks in the client's durable queue;
|
|
470
|
+
3. persist the terminal connection state and close reason;
|
|
471
|
+
4. create a new Archipel session with the same logical `sessionId`;
|
|
472
|
+
5. reconstruct safe context from the durable transcript or a compact summary;
|
|
473
|
+
6. resume with a new `connectionId` and explicit audit entry.
|
|
474
|
+
|
|
475
|
+
The normalized realtime contract does not promise lossless replay of arbitrary
|
|
476
|
+
provider-native conversation history. Rebuild only the context supported by
|
|
477
|
+
the selected provider and application policy.
|
|
478
|
+
|
|
479
|
+
For horizontal deployments, route all frames for one active upstream
|
|
480
|
+
connection to its owning process, for example through sticky routing or a
|
|
481
|
+
distributed ownership record. Never depend on the factory's in-memory map as a
|
|
482
|
+
cluster-wide registry.
|
|
483
|
+
|
|
484
|
+
Nest shutdown calls `close()` for all owned sessions and waits for queued
|
|
485
|
+
metering within the configured close timeout.
|
|
486
|
+
|
|
487
|
+
---
|
|
488
|
+
|
|
489
|
+
## Production Checklist
|
|
490
|
+
|
|
491
|
+
- Provider loaders come only from `@ai-sdk/*`-backed Archipel subpaths.
|
|
492
|
+
- Provider credentials remain in backend configuration.
|
|
493
|
+
- The application authenticates and authorizes every client gateway message.
|
|
494
|
+
- Each audio fragment has a logical chunk ID and sequence outside Archipel.
|
|
495
|
+
- Client and backend persistence happen per fragment, not after the complete recording.
|
|
496
|
+
- Transcript and final response events are persisted before client acknowledgement.
|
|
497
|
+
- A durable `IntelligenceRealtimeMeteringPort` is registered.
|
|
498
|
+
- `eventId` has a database uniqueness constraint.
|
|
499
|
+
- Commercial interval charging and provider token reconciliation are not double-counted.
|
|
500
|
+
- Backpressure and tenant quotas are bounded outside the per-process Archipel limits.
|
|
501
|
+
- Reconnect creates a new connection and is visible in audit history.
|
|
502
|
+
- Shutdown hooks are enabled in the Nest application.
|
|
503
|
+
|
|
504
|
+
See the [platform-intelligence package page](../packages/platform-intelligence/)
|
|
505
|
+
and the [realtime API reference](../packages/platform-intelligence/api/Class.IntelligenceRealtimeSessionFactory)
|
|
506
|
+
for the complete contracts.
|
|
@@ -8,6 +8,11 @@ order: 15
|
|
|
8
8
|
|
|
9
9
|
This guide covers text generation with `platform-intelligence`: choosing an AI provider, configuring the module, generating completions, streaming responses, and registering class-based or provider-native AI SDK tools.
|
|
10
10
|
|
|
11
|
+
For server-owned, bidirectional audio and text connections, use the separate
|
|
12
|
+
[AI Realtime Voice](./ai-realtime-voice) guide. Realtime sessions have a
|
|
13
|
+
different transport, lifecycle, persistence, and metering model from request-
|
|
14
|
+
or response-scoped text generation.
|
|
15
|
+
|
|
11
16
|
---
|
|
12
17
|
|
|
13
18
|
## Installation
|
|
@@ -72,7 +72,7 @@ All 12 platform libraries provide a health indicator via their `/health` subpath
|
|
|
72
72
|
| `platform-queue` | `@breadstone/archipel-platform-queue/health` | `QueueHealthIndicator` | `queue` | Always `up` |
|
|
73
73
|
| `platform-payments` | `@breadstone/archipel-platform-payments/health` | `PaymentHealthIndicator` | `payment` | Calls `PaymentClientPort.ping()` to verify provider connectivity |
|
|
74
74
|
| `platform-esigning` | `@breadstone/archipel-platform-esigning/health` | `EsigningHealthIndicator` | `esigning` | Calls `EsigningClientPort.ping()` to verify provider connectivity, reports `providerId` |
|
|
75
|
-
| `platform-intelligence` | `@breadstone/archipel-platform-intelligence/health` | `IntelligenceHealthIndicator` | `intelligence` |
|
|
75
|
+
| `platform-intelligence` | `@breadstone/archipel-platform-intelligence/health` | `IntelligenceHealthIndicator` | `intelligence` | Resolves text configuration and reports registered realtime providers plus active session count |
|
|
76
76
|
| `platform-analytics` | `@breadstone/archipel-platform-analytics/health` | `AnalyticsHealthIndicator` | `analytics` | Calls `AnalyticsClientPort.ping()` to verify provider readiness |
|
|
77
77
|
| `platform-telemetry` | `@breadstone/archipel-platform-telemetry/health` | `TelemetryHealthIndicator` | `telemetry` | Checks `OtelSdkHolder.isInitialized` for SDK presence |
|
|
78
78
|
| `platform-authentication` | `@breadstone/archipel-platform-authentication/health` | `AuthenticationHealthIndicator` | `authentication` | Always `up` |
|
|
@@ -89,7 +89,7 @@ All 12 platform libraries provide a health indicator via their `/health` subpath
|
|
|
89
89
|
- `AnalyticsHealthIndicator` — calls `AnalyticsClientPort.ping()` (adapters can override for SDK readiness checks)
|
|
90
90
|
|
|
91
91
|
**State checks** — inspect in-memory state (zero-cost, no I/O):
|
|
92
|
-
- `IntelligenceHealthIndicator` —
|
|
92
|
+
- `IntelligenceHealthIndicator` — resolves configured provider/model data and reports process-local realtime provider and session state
|
|
93
93
|
- `TelemetryHealthIndicator` — checks whether the OpenTelemetry SDK has been initialized
|
|
94
94
|
- `McpHealthIndicator` — verifies at least one tool/resource/prompt handler is registered
|
|
95
95
|
|
|
@@ -153,7 +153,16 @@ export class HealthController {
|
|
|
153
153
|
"mail": { "status": "up" },
|
|
154
154
|
"payment": { "status": "up" },
|
|
155
155
|
"caching": { "status": "up" },
|
|
156
|
-
"intelligence": {
|
|
156
|
+
"intelligence": {
|
|
157
|
+
"status": "up",
|
|
158
|
+
"provider": "openai",
|
|
159
|
+
"model": "gpt-4o-mini",
|
|
160
|
+
"realtime": {
|
|
161
|
+
"enabled": true,
|
|
162
|
+
"providers": ["openai", "google"],
|
|
163
|
+
"activeSessions": 2
|
|
164
|
+
}
|
|
165
|
+
},
|
|
157
166
|
"mcp": { "status": "up", "tools": 5, "resources": 2, "prompts": 1 },
|
|
158
167
|
"telemetry": { "status": "up", "sdkInitialized": true }
|
|
159
168
|
}
|
package/data/guides/index.md
CHANGED
|
@@ -58,10 +58,11 @@ Practical guides for working with Archipel packages in your NestJS application.
|
|
|
58
58
|
|
|
59
59
|
## AI & Extensibility
|
|
60
60
|
|
|
61
|
-
| Guide
|
|
62
|
-
|
|
|
63
|
-
| [AI Text Generation](./ai-text-generation) | Multi-provider text completions,
|
|
64
|
-
| [
|
|
61
|
+
| Guide | What you'll learn |
|
|
62
|
+
| ---------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
|
|
63
|
+
| [AI Text Generation](./ai-text-generation) | Multi-provider text completions, streaming, native AI SDK tools, and per-request model options. |
|
|
64
|
+
| [AI Realtime Voice](./ai-realtime-voice) | Server-owned live audio/text sessions, interaction modes, durable streaming, metering, and explicit recovery. |
|
|
65
|
+
| [MCP Server](./mcp-server) | Build MCP servers with tools, resources, and prompts for AI agent integration. |
|
|
65
66
|
|
|
66
67
|
## Platform Internals
|
|
67
68
|
|
|
@@ -103,7 +103,7 @@ All Archipel platform libraries provide health indicators via their `/health` su
|
|
|
103
103
|
| `platform-queue` | `QueueHealthIndicator` | `queue` | Always `up` |
|
|
104
104
|
| `platform-payments` | `PaymentHealthIndicator` | `payment` | `up` if `PaymentClientPort` injected, else `disabled` |
|
|
105
105
|
| `platform-esigning` | `EsigningHealthIndicator` | `esigning` | `up` if `EsigningClientPort` injected, else `disabled` |
|
|
106
|
-
| `platform-intelligence` | `IntelligenceHealthIndicator` | `intelligence` |
|
|
106
|
+
| `platform-intelligence` | `IntelligenceHealthIndicator` | `intelligence` | Resolves text configuration and reports realtime providers plus active sessions; `disabled` when neither service is present |
|
|
107
107
|
| `platform-analytics` | `AnalyticsHealthIndicator` | `analytics` | `up` if `AnalyticsClientPort` injected, else `disabled` |
|
|
108
108
|
| `platform-telemetry` | `TelemetryHealthIndicator` | `telemetry` | `up` if `OtelSdkHolder` injected, else `disabled` |
|
|
109
109
|
| `platform-authentication`| `AuthenticationHealthIndicator`| `authentication` | Always `up` |
|
|
@@ -5,7 +5,7 @@ editUrl: false
|
|
|
5
5
|
---
|
|
6
6
|
# Class: IntelligenceHealthIndicator
|
|
7
7
|
|
|
8
|
-
Defined in: [health/IntelligenceHealthIndicator.ts:
|
|
8
|
+
Defined in: [health/IntelligenceHealthIndicator.ts:18](https://github.com/RueDeRennes/archipel/blob/main/libs/platform-intelligence/src/health/IntelligenceHealthIndicator.ts#L18)
|
|
9
9
|
|
|
10
10
|
Health indicator for the intelligence infrastructure.
|
|
11
11
|
Verifies that the intelligence provider configuration can be resolved.
|
|
@@ -19,16 +19,17 @@ Verifies that the intelligence provider configuration can be resolved.
|
|
|
19
19
|
### Constructor
|
|
20
20
|
|
|
21
21
|
```ts
|
|
22
|
-
new IntelligenceHealthIndicator(textGenerator?): IntelligenceHealthIndicator;
|
|
22
|
+
new IntelligenceHealthIndicator(textGenerator?, realtimeSessionFactory?): IntelligenceHealthIndicator;
|
|
23
23
|
```
|
|
24
24
|
|
|
25
|
-
Defined in: [health/IntelligenceHealthIndicator.ts:
|
|
25
|
+
Defined in: [health/IntelligenceHealthIndicator.ts:28](https://github.com/RueDeRennes/archipel/blob/main/libs/platform-intelligence/src/health/IntelligenceHealthIndicator.ts#L28)
|
|
26
26
|
|
|
27
27
|
#### Parameters
|
|
28
28
|
|
|
29
29
|
| Parameter | Type |
|
|
30
30
|
| ------ | ------ |
|
|
31
31
|
| `textGenerator?` | [`IntelligenceTextGenerator`](Class.IntelligenceTextGenerator) |
|
|
32
|
+
| `realtimeSessionFactory?` | [`IntelligenceRealtimeSessionFactory`](Class.IntelligenceRealtimeSessionFactory) |
|
|
32
33
|
|
|
33
34
|
#### Returns
|
|
34
35
|
|
|
@@ -42,7 +43,7 @@ Defined in: [health/IntelligenceHealthIndicator.ts:26](https://github.com/RueDeR
|
|
|
42
43
|
readonly key: string = 'intelligence';
|
|
43
44
|
```
|
|
44
45
|
|
|
45
|
-
Defined in: [health/IntelligenceHealthIndicator.ts:
|
|
46
|
+
Defined in: [health/IntelligenceHealthIndicator.ts:37](https://github.com/RueDeRennes/archipel/blob/main/libs/platform-intelligence/src/health/IntelligenceHealthIndicator.ts#L37)
|
|
46
47
|
|
|
47
48
|
Unique key identifying the indicator
|
|
48
49
|
|
|
@@ -60,7 +61,7 @@ IHealthIndicator.key
|
|
|
60
61
|
check(): HealthIndicatorResult<string>;
|
|
61
62
|
```
|
|
62
63
|
|
|
63
|
-
Defined in: [health/IntelligenceHealthIndicator.ts:
|
|
64
|
+
Defined in: [health/IntelligenceHealthIndicator.ts:50](https://github.com/RueDeRennes/archipel/blob/main/libs/platform-intelligence/src/health/IntelligenceHealthIndicator.ts#L50)
|
|
64
65
|
|
|
65
66
|
Verifies that the provider configuration can be resolved.
|
|
66
67
|
Returns disabled status when no text generator is configured.
|
|
@@ -5,7 +5,7 @@ editUrl: false
|
|
|
5
5
|
---
|
|
6
6
|
# Class: IntelligenceModule
|
|
7
7
|
|
|
8
|
-
Defined in: [IntelligenceModule.ts:
|
|
8
|
+
Defined in: [IntelligenceModule.ts:59](https://github.com/RueDeRennes/archipel/blob/main/libs/platform-intelligence/src/IntelligenceModule.ts#L59)
|
|
9
9
|
|
|
10
10
|
Core module exposing the shared intelligence infrastructure.
|
|
11
11
|
|
|
@@ -29,7 +29,7 @@ new IntelligenceModule(): IntelligenceModule;
|
|
|
29
29
|
static register(options?): DynamicModule;
|
|
30
30
|
```
|
|
31
31
|
|
|
32
|
-
Defined in: [IntelligenceModule.ts:
|
|
32
|
+
Defined in: [IntelligenceModule.ts:70](https://github.com/RueDeRennes/archipel/blob/main/libs/platform-intelligence/src/IntelligenceModule.ts#L70)
|
|
33
33
|
|
|
34
34
|
Registers the intelligence module and optional class-based tools.
|
|
35
35
|
|