@memberjunction/ai-realtime-client 6.1.0-edge.6 → 6.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +16 -4
- package/dist/drivers/openAILiveClient.d.ts +137 -0
- package/dist/drivers/openAILiveClient.d.ts.map +1 -0
- package/dist/drivers/openAILiveClient.js +731 -0
- package/dist/drivers/openAILiveClient.js.map +1 -0
- package/dist/generic/baseRealtimeClient.d.ts +7 -2
- package/dist/generic/baseRealtimeClient.d.ts.map +1 -1
- package/dist/generic/baseRealtimeClient.js.map +1 -1
- package/dist/index.d.ts +1 -0
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +1 -0
- package/dist/index.js.map +1 -1
- package/package.json +3 -3
package/README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# @memberjunction/ai-realtime-client
|
|
2
2
|
|
|
3
|
-
Framework-agnostic **browser-side** abstraction for provider-direct realtime (voice) sessions: the `BaseRealtimeClient` contract plus the
|
|
3
|
+
Framework-agnostic **browser-side** abstraction for provider-direct realtime (voice) sessions: the `BaseRealtimeClient` contract plus the provider drivers (`OpenAIRealtimeClient`, `OpenAILiveClient`, `GeminiRealtimeClient`, `ElevenLabsRealtimeClient`, `AssemblyAIRealtimeClient`, `xAIRealtimeClient`, `HuggingFaceRealtimeClient`) and the shared PCM audio plane (`src/audio/`) the websocket drivers build on.
|
|
4
4
|
|
|
5
5
|
This package is the **client-side mirror** of the server's `BaseRealtimeModel` pattern (`@memberjunction/ai`). In the **client-direct topology**, the MJ server mints an ephemeral credential + provider-native session config (`ClientRealtimeSessionConfig`) through its server driver, and the browser resolves the matching *client* driver through the MemberJunction `ClassFactory` using the config's `Provider` string as the registration key. The browser owns the provider socket (lowest audio latency — frames never transit the MJ server), while **prompt and tool authority stay server-side**: the client applies the server-built `SessionConfig` verbatim.
|
|
6
6
|
|
|
@@ -25,7 +25,7 @@ BaseRealtimeModel driver BaseRealtimeClient driver
|
|
|
25
25
|
│ ClientRealtimeSessionConfig │ ClassFactory.CreateInstance(
|
|
26
26
|
│ { Provider, Model, │ BaseRealtimeClient,
|
|
27
27
|
│ EphemeralToken, ExpiresAt, ────────► │ config.Provider)
|
|
28
|
-
│ SessionConfig (opaque) } │ // 'openai' | 'gemini' | 'elevenlabs' | 'assemblyai'
|
|
28
|
+
│ SessionConfig (opaque) } │ // 'openai' | 'openAILive' | 'gemini' | 'elevenlabs' | 'assemblyai'
|
|
29
29
|
```
|
|
30
30
|
|
|
31
31
|
**Division of responsibility** (from the `BaseRealtimeClient` doc header):
|
|
@@ -52,7 +52,7 @@ BaseRealtimeModel driver BaseRealtimeClient driver
|
|
|
52
52
|
|
|
53
53
|
States (`RealtimeClientState`): `connecting → connected → listening ⇄ speaking → closed | error`. There is deliberately **no `thinking` state** — "the host is executing a tool" is host policy, not wire state.
|
|
54
54
|
|
|
55
|
-
Transcripts (`RealtimeClientTranscript`) carry `Role`, `Text` (interim events are incremental **deltas**, finals are the complete turn), `IsFinal`, and `Kind: 'normal' | 'narration'` — narration transcripts are ephemeral by product decision (never captions, never persisted).
|
|
55
|
+
Transcripts (`RealtimeClientTranscript`): carry `Role`, `Text` (interim events are incremental **deltas**, finals are the complete turn), `IsFinal`, and `Kind: 'normal' | 'narration'` — narration transcripts are ephemeral by product decision (never captions, never persisted).
|
|
56
56
|
|
|
57
57
|
Errors (`RealtimeClientError`): `Fatal: true` means the session is unusable (transport failure, credential expiry) and is also followed by an `'error'` state; `Fatal: false` is a recoverable provider error frame.
|
|
58
58
|
|
|
@@ -66,6 +66,17 @@ Errors (`RealtimeClientError`): `Fatal: true` means the session is unusable (tra
|
|
|
66
66
|
- **Narration tagging**: `RequestSpokenUpdate` marks the next response so its transcripts emit with `Kind: 'narration'`.
|
|
67
67
|
- **Playback tracking**: `IsAudioPlaying` from the WebRTC `output_audio_buffer` started/stopped events.
|
|
68
68
|
|
|
69
|
+
### `OpenAILiveClient` — `@RegisterClass(BaseRealtimeClient, 'openAILive')`
|
|
70
|
+
|
|
71
|
+
- **Model**: `gpt-live-1` (OpenAI Live API).
|
|
72
|
+
- **Transport**: WebRTC via `/v1/realtime/calls` data channel.
|
|
73
|
+
- **Wire protocol specifics**:
|
|
74
|
+
- **Tool Outputs**: Emits `response.item.create` (item type: `function_call_output`) followed by `response.create` per batch (coordinated via `RealtimeToolBatchBarrier`). `conversation.item.create` is not supported by OpenAI Live and is explicitly rejected.
|
|
75
|
+
- **Floor Control**: No wire `response.cancel` event exists in OpenAI Live WebRTC; `CancelActiveResponse()` performs local audio playback draining and state reset.
|
|
76
|
+
- **Tool Call Deduplication**: Tool calls arriving via `output_item.done` or `response.function_call_arguments.done` are deduplicated by `CallID` via an `emittedToolCallIds` Set. The deduplication Set deliberately survives local cancellations so late-arriving frames within an in-flight turn cannot double-emit an already-handled tool call.
|
|
77
|
+
- **Context & Commentary**: `SendContextNote` appends to model thinking via `session.thinking.append` (with `delegation_id: null`). Spoken interim updates use `session.commentary.append`.
|
|
78
|
+
- **Outbound Queuing**: Frames sent while the WebRTC data channel is connecting are buffered in a bounded FIFO queue (`MAX_OUTBOUND_QUEUE_SIZE = 100`, drop-oldest with warnings).
|
|
79
|
+
|
|
69
80
|
### `GeminiRealtimeClient` — `@RegisterClass(BaseRealtimeClient, 'gemini')`
|
|
70
81
|
|
|
71
82
|
- **Transport**: WebSocket via the `@google/genai` Live SDK, authenticated with the server-minted ephemeral token (a `v1alpha` client).
|
|
@@ -107,13 +118,14 @@ Drivers expose these through overridable `protected` creation seams (`createMicC
|
|
|
107
118
|
import { MJGlobal } from '@memberjunction/global';
|
|
108
119
|
import {
|
|
109
120
|
BaseRealtimeClient,
|
|
110
|
-
LoadOpenAIRealtimeClient, LoadGeminiRealtimeClient,
|
|
121
|
+
LoadOpenAIRealtimeClient, LoadOpenAILiveClient, LoadGeminiRealtimeClient,
|
|
111
122
|
LoadElevenLabsRealtimeClient, LoadAssemblyAIRealtimeClient
|
|
112
123
|
} from '@memberjunction/ai-realtime-client';
|
|
113
124
|
|
|
114
125
|
// Tree-shaking prevention — drivers are resolved dynamically, so a static call path
|
|
115
126
|
// must keep their @RegisterClass side effects alive:
|
|
116
127
|
LoadOpenAIRealtimeClient();
|
|
128
|
+
LoadOpenAILiveClient();
|
|
117
129
|
LoadGeminiRealtimeClient();
|
|
118
130
|
LoadElevenLabsRealtimeClient();
|
|
119
131
|
LoadAssemblyAIRealtimeClient();
|
|
@@ -0,0 +1,137 @@
|
|
|
1
|
+
import { ClientRealtimeSessionConfig, JSONObject } from '@memberjunction/ai';
|
|
2
|
+
import { BaseRealtimeClient, RealtimeClientState } from '../generic/baseRealtimeClient.js';
|
|
3
|
+
import { IRealtimeAudioSink, IRealtimeDataChannel, IRealtimePeerConnection } from './openAIRealtimeClient.js';
|
|
4
|
+
/**
|
|
5
|
+
* Extended peer connection interface for OpenAILiveClient to support non-trickle ICE gathering.
|
|
6
|
+
*/
|
|
7
|
+
export interface IRealtimeLivePeerConnection extends IRealtimePeerConnection {
|
|
8
|
+
iceGatheringState?: RTCIceGatheringState;
|
|
9
|
+
addEventListener?(type: string, listener: (event: Event) => void): void;
|
|
10
|
+
removeEventListener?(type: string, listener: (event: Event) => void): void;
|
|
11
|
+
}
|
|
12
|
+
/**
|
|
13
|
+
* Browser WebRTC client driver for the OpenAI Live Realtime API (`gpt-live-1`).
|
|
14
|
+
*
|
|
15
|
+
* Implements the browser-direct WebRTC topology specified in Section 5 of the OpenAI Live plan:
|
|
16
|
+
* 1. Exchanges SDP offer for answer with an application server SDP broker (`POST /v1/live/sessions`),
|
|
17
|
+
* keeping the project API key and session configuration on the trusted server.
|
|
18
|
+
* 2. Creates the `'oai-events'` data channel **before** `createOffer()`.
|
|
19
|
+
* 3. Waits for non-trickle ICE (`iceGatheringState === 'complete'`).
|
|
20
|
+
* 4. The HTTP SDP exchange starts the session — never sends `session.start` on the data channel.
|
|
21
|
+
* 5. Media rides WebRTC tracks: `session.input_audio.append` and `session.output_audio.delta` are forbidden on the data channel.
|
|
22
|
+
* 6. Omits `audio.format` entirely (SDP negotiates codecs).
|
|
23
|
+
* 7. Accounts for OpenAI Live's 15-second WebRTC session pre-bill.
|
|
24
|
+
*
|
|
25
|
+
* Registered under `'openai-live'` and `'OpenAILiveRealtime'`.
|
|
26
|
+
*/
|
|
27
|
+
export declare class OpenAILiveClient extends BaseRealtimeClient {
|
|
28
|
+
static readonly PREBILL_DURATION_SECONDS = 15;
|
|
29
|
+
static readonly PLAYBACK_DRAIN_SILENCE_MS = 200;
|
|
30
|
+
static readonly PLAYBACK_DRAIN_TRANSCRIPT_GAP_MS = 300;
|
|
31
|
+
static readonly PLAYBACK_DRAIN_SILENCE_THRESHOLD = 0.01;
|
|
32
|
+
static readonly ASSISTANT_SAFETY_BACKSTOP_MS = 3000;
|
|
33
|
+
static readonly MAX_OUTBOUND_QUEUE_SIZE = 100;
|
|
34
|
+
private peerConnection;
|
|
35
|
+
protected dataChannel: IRealtimeDataChannel | null;
|
|
36
|
+
private remoteAudioEl;
|
|
37
|
+
private remoteStream;
|
|
38
|
+
private remoteStreamHandlers;
|
|
39
|
+
private micStream;
|
|
40
|
+
protected sessionConfig: JSONObject | null;
|
|
41
|
+
private currentState;
|
|
42
|
+
private responseActive;
|
|
43
|
+
private audioPlaying;
|
|
44
|
+
private pendingNarration;
|
|
45
|
+
private totalDurationSeconds;
|
|
46
|
+
private pendingUserText;
|
|
47
|
+
private pendingAssistantText;
|
|
48
|
+
private lastOutputTranscriptTime;
|
|
49
|
+
private drainSilenceStartTime;
|
|
50
|
+
private playbackDrainInterval;
|
|
51
|
+
private assistantSafetyBackstopTimer;
|
|
52
|
+
private clientDelegationCallIds;
|
|
53
|
+
private emittedToolCallIds;
|
|
54
|
+
private toolBatchBarrier;
|
|
55
|
+
private outboundQueue;
|
|
56
|
+
get IsBusy(): boolean;
|
|
57
|
+
get IsAudioPlaying(): boolean;
|
|
58
|
+
get PrebillSeconds(): number;
|
|
59
|
+
get State(): RealtimeClientState;
|
|
60
|
+
/**
|
|
61
|
+
* Connects to the OpenAI Live WebRTC endpoint via the application server SDP broker.
|
|
62
|
+
*/
|
|
63
|
+
Connect(config: ClientRealtimeSessionConfig, micStream: MediaStream, _cameraStream?: MediaStream): Promise<void>;
|
|
64
|
+
/**
|
|
65
|
+
* Tears down the connection, data channel, audio tracks, and audio sink.
|
|
66
|
+
*/
|
|
67
|
+
Disconnect(): Promise<void>;
|
|
68
|
+
/**
|
|
69
|
+
* Injects user text into the live session and requests a model response.
|
|
70
|
+
* SendText implies barge-in and must NOT synthesize a user transcript (obligation #4).
|
|
71
|
+
*/
|
|
72
|
+
SendText(text: string): void;
|
|
73
|
+
/**
|
|
74
|
+
* Cancels an active model response and flushes local playback.
|
|
75
|
+
*/
|
|
76
|
+
CancelActiveResponse(): void;
|
|
77
|
+
/**
|
|
78
|
+
* Injects background context without triggering model speech.
|
|
79
|
+
* Uses `session.thinking.append` with `content` parameter and `delegation_id: null`.
|
|
80
|
+
*/
|
|
81
|
+
SendContextNote(text: string): void;
|
|
82
|
+
/**
|
|
83
|
+
* Requests a brief interim narration utterance during delegated background work.
|
|
84
|
+
*/
|
|
85
|
+
RequestSpokenUpdate(instructions: string): void;
|
|
86
|
+
/**
|
|
87
|
+
* Delivers a completed tool execution result back to the model.
|
|
88
|
+
* OpenAI Live remote reasoning plane requires response.item.create followed by response.create.
|
|
89
|
+
* Client delegation plane uses session.commentary.append with the delegation_id.
|
|
90
|
+
*/
|
|
91
|
+
SendToolResult(callID: string, outputJson: string): void;
|
|
92
|
+
/**
|
|
93
|
+
* Mutes or unmutes the microphone capture tracks.
|
|
94
|
+
*/
|
|
95
|
+
SetMuted(muted: boolean): void;
|
|
96
|
+
/**
|
|
97
|
+
* Returns the agent's remote audio MediaStream if available.
|
|
98
|
+
*/
|
|
99
|
+
GetRemoteMediaStream(): MediaStream | null;
|
|
100
|
+
/**
|
|
101
|
+
* Registers a callback for when the agent's remote audio MediaStream lands.
|
|
102
|
+
*/
|
|
103
|
+
OnRemoteMediaStream(handler: (stream: MediaStream) => void): void;
|
|
104
|
+
/** Creates the peer connection. Production returns a real `RTCPeerConnection`. */
|
|
105
|
+
protected createPeerConnection(): IRealtimeLivePeerConnection;
|
|
106
|
+
/** Creates a hidden `<audio>` element to sink the model's audio output. */
|
|
107
|
+
protected createAudioSink(): IRealtimeAudioSink;
|
|
108
|
+
/**
|
|
109
|
+
* Exchanges local SDP offer for remote answer SDP.
|
|
110
|
+
* Posts to application server broker or direct endpoint.
|
|
111
|
+
*/
|
|
112
|
+
protected postSdpOffer(offerSdp: string, config: ClientRealtimeSessionConfig): Promise<string>;
|
|
113
|
+
private setState;
|
|
114
|
+
private attachMicrophone;
|
|
115
|
+
private attachRemoteAudio;
|
|
116
|
+
private waitForIceGatheringComplete;
|
|
117
|
+
protected adoptDataChannel(channel: IRealtimeDataChannel): void;
|
|
118
|
+
private finalizeAssistantTranscript;
|
|
119
|
+
private finalizeUserTranscript;
|
|
120
|
+
private startPlaybackDrainMonitoring;
|
|
121
|
+
private stopPlaybackDrainMonitoring;
|
|
122
|
+
private scheduleAssistantSafetyBackstop;
|
|
123
|
+
private checkPlaybackDrain;
|
|
124
|
+
private handleToolCallItem;
|
|
125
|
+
private handleOutputItemDone;
|
|
126
|
+
private handleResponseCompleted;
|
|
127
|
+
private isResponsesDelegation;
|
|
128
|
+
private handleDataChannelMessage;
|
|
129
|
+
private isActive;
|
|
130
|
+
private sendDataChannelFrame;
|
|
131
|
+
private flushOutboundQueue;
|
|
132
|
+
}
|
|
133
|
+
/**
|
|
134
|
+
* Tree-shaking prevention function for OpenAILiveClient.
|
|
135
|
+
*/
|
|
136
|
+
export declare function LoadOpenAILiveClient(): void;
|
|
137
|
+
//# sourceMappingURL=openAILiveClient.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"openAILiveClient.d.ts","sourceRoot":"","sources":["../../src/drivers/openAILiveClient.ts"],"names":[],"mappings":"AACA,OAAO,EAAE,2BAA2B,EAAE,UAAU,EAA4B,MAAM,oBAAoB,CAAC;AAEvG,OAAO,EACH,kBAAkB,EAElB,mBAAmB,EAItB,MAAM,+BAA+B,CAAC;AACvC,OAAO,EACH,kBAAkB,EAClB,oBAAoB,EACpB,uBAAuB,EAC1B,MAAM,wBAAwB,CAAC;AAEhC;;GAEG;AACH,MAAM,WAAW,2BAA4B,SAAQ,uBAAuB;IACxE,iBAAiB,CAAC,EAAE,oBAAoB,CAAC;IACzC,gBAAgB,CAAC,CAAC,IAAI,EAAE,MAAM,EAAE,QAAQ,EAAE,CAAC,KAAK,EAAE,KAAK,KAAK,IAAI,GAAG,IAAI,CAAC;IACxE,mBAAmB,CAAC,CAAC,IAAI,EAAE,MAAM,EAAE,QAAQ,EAAE,CAAC,KAAK,EAAE,KAAK,KAAK,IAAI,GAAG,IAAI,CAAC;CAC9E;AAED;;;;;;;;;;;;;;GAcG;AACH,qBAEa,gBAAiB,SAAQ,kBAAkB;IACpD,gBAAuB,wBAAwB,MAAM;IACrD,gBAAuB,yBAAyB,OAAO;IACvD,gBAAuB,gCAAgC,OAAO;IAC9D,gBAAuB,gCAAgC,QAAQ;IAC/D,gBAAuB,4BAA4B,QAAQ;IAC3D,gBAAuB,uBAAuB,OAAO;IAGrD,OAAO,CAAC,cAAc,CAA4C;IAClE,SAAS,CAAC,WAAW,EAAE,oBAAoB,GAAG,IAAI,CAAQ;IAC1D,OAAO,CAAC,aAAa,CAAmC;IACxD,OAAO,CAAC,YAAY,CAA4B;IAChD,OAAO,CAAC,oBAAoB,CAA4C;IACxE,OAAO,CAAC,SAAS,CAA4B;IAC7C,SAAS,CAAC,aAAa,EAAE,UAAU,GAAG,IAAI,CAAQ;IAGlD,OAAO,CAAC,YAAY,CAAiC;IACrD,OAAO,CAAC,cAAc,CAAS;IAC/B,OAAO,CAAC,YAAY,CAAS;IAC7B,OAAO,CAAC,gBAAgB,CAAS;IACjC,OAAO,CAAC,oBAAoB,CAA6C;IACzE,OAAO,CAAC,eAAe,CAAM;IAC7B,OAAO,CAAC,oBAAoB,CAAM;IAClC,OAAO,CAAC,wBAAwB,CAAK;IACrC,OAAO,CAAC,qBAAqB,CAAuB;IACpD,OAAO,CAAC,qBAAqB,CAA+C;IAC5E,OAAO,CAAC,4BAA4B,CAA8C;IAClF,OAAO,CAAC,uBAAuB,CAAqB;IACpD,OAAO,CAAC,kBAAkB,CAAqB;IAC/C,OAAO,CAAC,gBAAgB,CAAkC;IAC1D,OAAO,CAAC,aAAa,CAAsC;IAI3D,IAAW,MAAM,IAAI,OAAO,CAE3B;IAED,IAAW,cAAc,IAAI,OAAO,CAEnC;IAED,IAAW,cAAc,IAAI,MAAM,CAElC;IAED,IAAW,KAAK,IAAI,mBAAmB,CAEtC;IAED;;OAEG;IACU,OAAO,CAChB,MAAM,EAAE,2BAA2B,EACnC,SAAS,EAAE,WAAW,EACtB,aAAa,CAAC,EAAE,WAAW,GAC5B,OAAO,CAAC,IAAI,CAAC;IAuChB;;OAEG;IACU,UAAU,IAAI,OAAO,CAAC,IAAI,CAAC;IAmDxC;;;OAGG;IACI,QAAQ,CAAC,IAAI,EAAE,MAAM,GAAG,IAAI;IAYnC;;OAEG;IACI,oBAAoB,IAAI,IAAI;IAqBnC;;;OAGG;IACI,eAAe,CAAC,IAAI,EAAE,MAAM,GAAG,IAAI;IAQ1C;;OAEG;IACI,mBAAmB,CAAC,YAAY,EAAE,MAAM,GAAG,IAAI;IAkBtD;;;;OAIG;IACI,cAAc,CAAC,MAAM,EAAE,MAAM,EAAE,UAAU,EAAE,MAAM,GAAG,IAAI;IA4B/D;;OAEG;IACI,QAAQ,CAAC,KAAK,EAAE,OAAO,GAAG,IAAI;IAMrC;;OAEG;IACa,oBAAoB,IAAI,WAAW,GAAG,IAAI;IAI1D;;OAEG;IACa,mBAAmB,CAAC,OAAO,EAAE,CAAC,MAAM,EAAE,WAAW,KAAK,IAAI,GAAG,IAAI;IAajF,kFAAkF;IAClF,SAAS,CAAC,oBAAoB,IAAI,2BAA2B;IAI7D,2EAA2E;IAC3E,SAAS,CAAC,eAAe,IAAI,kBAAkB;IAQ/C;;;OAGG;cACa,YAAY,CAAC,QAAQ,EAAE,MAAM,EAAE,MAAM,EAAE,2BAA2B,GAAG,OAAO,CAAC,MAAM,CAAC;IAyDpG,OAAO,CAAC,QAAQ;IAOhB,OAAO,CAAC,gBAAgB;IAMxB,OAAO,CAAC,iBAAiB;YAmBX,2BAA2B;IAmBzC,SAAS,CAAC,gBAAgB,CAAC,OAAO,EAAE,oBAAoB,GAAG,IAAI;IAwB/D,OAAO,CAAC,2BAA2B;IAmBnC,OAAO,CAAC,sBAAsB;IAe9B,OAAO,CAAC,4BAA4B;IASpC,OAAO,CAAC,2BAA2B;IAYnC,OAAO,CAAC,+BAA+B;IAUvC,OAAO,CAAC,kBAAkB;IA0B1B,OAAO,CAAC,kBAAkB;IAyB1B,OAAO,CAAC,oBAAoB;IAS5B,OAAO,CAAC,uBAAuB;IAgB/B,OAAO,CAAC,qBAAqB;IAQ7B,OAAO,CAAC,wBAAwB;IA+IhC,OAAO,CAAC,QAAQ;IAIhB,OAAO,CAAC,oBAAoB;IAoB5B,OAAO,CAAC,kBAAkB;CAQ7B;AAED;;GAEG;AACH,wBAAgB,oBAAoB,IAAI,IAAI,CAE3C"}
|
|
@@ -0,0 +1,731 @@
|
|
|
1
|
+
var __decorate = (this && this.__decorate) || function (decorators, target, key, desc) {
|
|
2
|
+
var c = arguments.length, r = c < 3 ? target : desc === null ? desc = Object.getOwnPropertyDescriptor(target, key) : desc, d;
|
|
3
|
+
if (typeof Reflect === "object" && typeof Reflect.decorate === "function") r = Reflect.decorate(decorators, target, key, desc);
|
|
4
|
+
else for (var i = decorators.length - 1; i >= 0; i--) if (d = decorators[i]) r = (c < 3 ? d(r) : c > 3 ? d(target, key, r) : d(target, key)) || r;
|
|
5
|
+
return c > 3 && r && Object.defineProperty(target, key, r), r;
|
|
6
|
+
};
|
|
7
|
+
var OpenAILiveClient_1;
|
|
8
|
+
import { RegisterClass } from '@memberjunction/global';
|
|
9
|
+
import { RealtimeToolBatchBarrier } from '@memberjunction/ai';
|
|
10
|
+
import { RealtimeAudioMeter } from '../audio/audioMeter.js';
|
|
11
|
+
import { BaseRealtimeClient, } from '../generic/baseRealtimeClient.js';
|
|
12
|
+
/**
|
|
13
|
+
* Browser WebRTC client driver for the OpenAI Live Realtime API (`gpt-live-1`).
|
|
14
|
+
*
|
|
15
|
+
* Implements the browser-direct WebRTC topology specified in Section 5 of the OpenAI Live plan:
|
|
16
|
+
* 1. Exchanges SDP offer for answer with an application server SDP broker (`POST /v1/live/sessions`),
|
|
17
|
+
* keeping the project API key and session configuration on the trusted server.
|
|
18
|
+
* 2. Creates the `'oai-events'` data channel **before** `createOffer()`.
|
|
19
|
+
* 3. Waits for non-trickle ICE (`iceGatheringState === 'complete'`).
|
|
20
|
+
* 4. The HTTP SDP exchange starts the session — never sends `session.start` on the data channel.
|
|
21
|
+
* 5. Media rides WebRTC tracks: `session.input_audio.append` and `session.output_audio.delta` are forbidden on the data channel.
|
|
22
|
+
* 6. Omits `audio.format` entirely (SDP negotiates codecs).
|
|
23
|
+
* 7. Accounts for OpenAI Live's 15-second WebRTC session pre-bill.
|
|
24
|
+
*
|
|
25
|
+
* Registered under `'openai-live'` and `'OpenAILiveRealtime'`.
|
|
26
|
+
*/
|
|
27
|
+
let OpenAILiveClient = class OpenAILiveClient extends BaseRealtimeClient {
|
|
28
|
+
constructor() {
|
|
29
|
+
super(...arguments);
|
|
30
|
+
// ── Transport ──────────────────────────────────────────────────────────────
|
|
31
|
+
this.peerConnection = null;
|
|
32
|
+
this.dataChannel = null;
|
|
33
|
+
this.remoteAudioEl = null;
|
|
34
|
+
this.remoteStream = null;
|
|
35
|
+
this.remoteStreamHandlers = [];
|
|
36
|
+
this.micStream = null;
|
|
37
|
+
this.sessionConfig = null;
|
|
38
|
+
// ── State Machine ──────────────────────────────────────────────────────────
|
|
39
|
+
this.currentState = 'closed';
|
|
40
|
+
this.responseActive = false;
|
|
41
|
+
this.audioPlaying = false;
|
|
42
|
+
this.pendingNarration = false;
|
|
43
|
+
this.totalDurationSeconds = OpenAILiveClient_1.PREBILL_DURATION_SECONDS;
|
|
44
|
+
this.pendingUserText = '';
|
|
45
|
+
this.pendingAssistantText = '';
|
|
46
|
+
this.lastOutputTranscriptTime = 0;
|
|
47
|
+
this.drainSilenceStartTime = null;
|
|
48
|
+
this.playbackDrainInterval = null;
|
|
49
|
+
this.assistantSafetyBackstopTimer = null;
|
|
50
|
+
this.clientDelegationCallIds = new Set();
|
|
51
|
+
this.emittedToolCallIds = new Set();
|
|
52
|
+
this.toolBatchBarrier = new RealtimeToolBatchBarrier();
|
|
53
|
+
this.outboundQueue = [];
|
|
54
|
+
}
|
|
55
|
+
static { OpenAILiveClient_1 = this; }
|
|
56
|
+
static { this.PREBILL_DURATION_SECONDS = 15; }
|
|
57
|
+
static { this.PLAYBACK_DRAIN_SILENCE_MS = 200; }
|
|
58
|
+
static { this.PLAYBACK_DRAIN_TRANSCRIPT_GAP_MS = 300; }
|
|
59
|
+
static { this.PLAYBACK_DRAIN_SILENCE_THRESHOLD = 0.01; }
|
|
60
|
+
static { this.ASSISTANT_SAFETY_BACKSTOP_MS = 3000; }
|
|
61
|
+
static { this.MAX_OUTBOUND_QUEUE_SIZE = 100; }
|
|
62
|
+
// ── BaseRealtimeClient implementation ──────────────────────────────────────
|
|
63
|
+
get IsBusy() {
|
|
64
|
+
return this.responseActive;
|
|
65
|
+
}
|
|
66
|
+
get IsAudioPlaying() {
|
|
67
|
+
return this.audioPlaying;
|
|
68
|
+
}
|
|
69
|
+
get PrebillSeconds() {
|
|
70
|
+
return OpenAILiveClient_1.PREBILL_DURATION_SECONDS;
|
|
71
|
+
}
|
|
72
|
+
get State() {
|
|
73
|
+
return this.currentState;
|
|
74
|
+
}
|
|
75
|
+
/**
|
|
76
|
+
* Connects to the OpenAI Live WebRTC endpoint via the application server SDP broker.
|
|
77
|
+
*/
|
|
78
|
+
async Connect(config, micStream, _cameraStream) {
|
|
79
|
+
this.sessionConfig = config.SessionConfig;
|
|
80
|
+
this.micStream = micStream;
|
|
81
|
+
this.setState('connecting');
|
|
82
|
+
const pc = this.createPeerConnection();
|
|
83
|
+
this.peerConnection = pc;
|
|
84
|
+
this.attachMicrophone(pc, micStream);
|
|
85
|
+
this.attachRemoteAudio(pc);
|
|
86
|
+
// Capability obligation #9: input mic audio meter
|
|
87
|
+
this.attachInputAudioMeter(RealtimeAudioMeter.ForMicStream(micStream));
|
|
88
|
+
// Mechanics rule 2: create data channel BEFORE createOffer()
|
|
89
|
+
const channel = pc.createDataChannel('oai-events');
|
|
90
|
+
this.adoptDataChannel(channel);
|
|
91
|
+
// Create local offer
|
|
92
|
+
const offer = await pc.createOffer();
|
|
93
|
+
await pc.setLocalDescription(offer);
|
|
94
|
+
// Mechanics rule 3: Non-trickle ICE (wait for iceGatheringState === 'complete')
|
|
95
|
+
await this.waitForIceGatheringComplete(pc);
|
|
96
|
+
// Mechanics rule 1: Exchange offer SDP for answer SDP via trusted server broker
|
|
97
|
+
const answerSdp = await this.postSdpOffer(offer.sdp ?? '', config);
|
|
98
|
+
await pc.setRemoteDescription({ type: 'answer', sdp: answerSdp });
|
|
99
|
+
this.setState('connected');
|
|
100
|
+
// Mechanics rule 7: account for 15s session pre-bill
|
|
101
|
+
this.totalDurationSeconds = OpenAILiveClient_1.PREBILL_DURATION_SECONDS;
|
|
102
|
+
this.emitUsage({
|
|
103
|
+
DurationSeconds: OpenAILiveClient_1.PREBILL_DURATION_SECONDS,
|
|
104
|
+
InputTokens: 0,
|
|
105
|
+
OutputTokens: 0,
|
|
106
|
+
});
|
|
107
|
+
}
|
|
108
|
+
/**
|
|
109
|
+
* Tears down the connection, data channel, audio tracks, and audio sink.
|
|
110
|
+
*/
|
|
111
|
+
async Disconnect() {
|
|
112
|
+
this.finalizeUserTranscript();
|
|
113
|
+
this.finalizeAssistantTranscript();
|
|
114
|
+
this.closeAudioMeters();
|
|
115
|
+
this.micStream?.getTracks().forEach((t) => t.stop());
|
|
116
|
+
this.micStream = null;
|
|
117
|
+
if (this.dataChannel) {
|
|
118
|
+
try {
|
|
119
|
+
this.dataChannel.close();
|
|
120
|
+
}
|
|
121
|
+
catch {
|
|
122
|
+
/* already closed */
|
|
123
|
+
}
|
|
124
|
+
this.dataChannel = null;
|
|
125
|
+
}
|
|
126
|
+
if (this.peerConnection) {
|
|
127
|
+
try {
|
|
128
|
+
this.peerConnection.close();
|
|
129
|
+
}
|
|
130
|
+
catch {
|
|
131
|
+
/* already closed */
|
|
132
|
+
}
|
|
133
|
+
this.peerConnection = null;
|
|
134
|
+
}
|
|
135
|
+
if (this.remoteAudioEl) {
|
|
136
|
+
this.remoteAudioEl.srcObject = null;
|
|
137
|
+
this.remoteAudioEl.remove();
|
|
138
|
+
this.remoteAudioEl = null;
|
|
139
|
+
}
|
|
140
|
+
this.remoteStream = null;
|
|
141
|
+
this.remoteStreamHandlers = [];
|
|
142
|
+
this.sessionConfig = null;
|
|
143
|
+
this.responseActive = false;
|
|
144
|
+
this.audioPlaying = false;
|
|
145
|
+
this.pendingNarration = false;
|
|
146
|
+
this.stopPlaybackDrainMonitoring();
|
|
147
|
+
this.pendingUserText = '';
|
|
148
|
+
this.pendingAssistantText = '';
|
|
149
|
+
this.clientDelegationCallIds.clear();
|
|
150
|
+
this.emittedToolCallIds.clear();
|
|
151
|
+
this.toolBatchBarrier.Clear();
|
|
152
|
+
this.outboundQueue = [];
|
|
153
|
+
if (this.currentState !== 'error') {
|
|
154
|
+
this.setState('closed');
|
|
155
|
+
}
|
|
156
|
+
}
|
|
157
|
+
/**
|
|
158
|
+
* Injects user text into the live session and requests a model response.
|
|
159
|
+
* SendText implies barge-in and must NOT synthesize a user transcript (obligation #4).
|
|
160
|
+
*/
|
|
161
|
+
SendText(text) {
|
|
162
|
+
this.CancelActiveResponse();
|
|
163
|
+
this.sendDataChannelFrame({
|
|
164
|
+
type: 'session.commentary.append',
|
|
165
|
+
content: text,
|
|
166
|
+
delegation_id: null,
|
|
167
|
+
});
|
|
168
|
+
this.sendDataChannelFrame({
|
|
169
|
+
type: 'response.create',
|
|
170
|
+
});
|
|
171
|
+
}
|
|
172
|
+
/**
|
|
173
|
+
* Cancels an active model response and flushes local playback.
|
|
174
|
+
*/
|
|
175
|
+
CancelActiveResponse() {
|
|
176
|
+
this.stopPlaybackDrainMonitoring();
|
|
177
|
+
this.pendingAssistantText = '';
|
|
178
|
+
// Note: emittedToolCallIds is intentionally NOT cleared here.
|
|
179
|
+
// GPT-Live has no wire response.cancel event, so the server may continue
|
|
180
|
+
// delivering output_item.done / function_call_arguments.done for this turn.
|
|
181
|
+
// Keeping emittedToolCallIds intact prevents double tool emission on cancel.
|
|
182
|
+
// toolBatchBarrier also deliberately survives a cancel so pending tool outputs
|
|
183
|
+
// can be submitted without triggering 'function_call_outputs_required'.
|
|
184
|
+
if (this.responseActive || this.audioPlaying) {
|
|
185
|
+
// Mechanics rule: GPT-Live has no wire response.cancel event on WebRTC;
|
|
186
|
+
// cancellation is local playback drain and state flush only.
|
|
187
|
+
this.responseActive = false;
|
|
188
|
+
this.audioPlaying = false;
|
|
189
|
+
this.pendingNarration = false;
|
|
190
|
+
if (this.currentState === 'speaking') {
|
|
191
|
+
this.setState('listening');
|
|
192
|
+
}
|
|
193
|
+
}
|
|
194
|
+
}
|
|
195
|
+
/**
|
|
196
|
+
* Injects background context without triggering model speech.
|
|
197
|
+
* Uses `session.thinking.append` with `content` parameter and `delegation_id: null`.
|
|
198
|
+
*/
|
|
199
|
+
SendContextNote(text) {
|
|
200
|
+
this.sendDataChannelFrame({
|
|
201
|
+
type: 'session.thinking.append',
|
|
202
|
+
content: text,
|
|
203
|
+
delegation_id: null,
|
|
204
|
+
});
|
|
205
|
+
}
|
|
206
|
+
/**
|
|
207
|
+
* Requests a brief interim narration utterance during delegated background work.
|
|
208
|
+
*/
|
|
209
|
+
RequestSpokenUpdate(instructions) {
|
|
210
|
+
if (this.responseActive || this.audioPlaying) {
|
|
211
|
+
// Collision rule: narration is disposable; skip when response is in flight
|
|
212
|
+
return;
|
|
213
|
+
}
|
|
214
|
+
this.pendingNarration = true;
|
|
215
|
+
this.sendDataChannelFrame({
|
|
216
|
+
type: 'session.commentary.append',
|
|
217
|
+
content: instructions,
|
|
218
|
+
delegation_id: null,
|
|
219
|
+
});
|
|
220
|
+
if (this.toolBatchBarrier.IsEmpty) {
|
|
221
|
+
this.sendDataChannelFrame({
|
|
222
|
+
type: 'response.create',
|
|
223
|
+
});
|
|
224
|
+
}
|
|
225
|
+
}
|
|
226
|
+
/**
|
|
227
|
+
* Delivers a completed tool execution result back to the model.
|
|
228
|
+
* OpenAI Live remote reasoning plane requires response.item.create followed by response.create.
|
|
229
|
+
* Client delegation plane uses session.commentary.append with the delegation_id.
|
|
230
|
+
*/
|
|
231
|
+
SendToolResult(callID, outputJson) {
|
|
232
|
+
if (this.clientDelegationCallIds.has(callID)) {
|
|
233
|
+
this.clientDelegationCallIds.delete(callID);
|
|
234
|
+
this.sendDataChannelFrame({
|
|
235
|
+
type: 'session.commentary.append',
|
|
236
|
+
content: outputJson,
|
|
237
|
+
delegation_id: callID,
|
|
238
|
+
});
|
|
239
|
+
return;
|
|
240
|
+
}
|
|
241
|
+
this.sendDataChannelFrame({
|
|
242
|
+
type: 'response.item.create',
|
|
243
|
+
item: {
|
|
244
|
+
type: 'function_call_output',
|
|
245
|
+
call_id: callID,
|
|
246
|
+
output: outputJson,
|
|
247
|
+
},
|
|
248
|
+
});
|
|
249
|
+
const isBatchComplete = this.toolBatchBarrier.RecordResult(callID);
|
|
250
|
+
if (isBatchComplete) {
|
|
251
|
+
this.sendDataChannelFrame({
|
|
252
|
+
type: 'response.create',
|
|
253
|
+
});
|
|
254
|
+
}
|
|
255
|
+
}
|
|
256
|
+
/**
|
|
257
|
+
* Mutes or unmutes the microphone capture tracks.
|
|
258
|
+
*/
|
|
259
|
+
SetMuted(muted) {
|
|
260
|
+
for (const track of this.micStream?.getAudioTracks() ?? []) {
|
|
261
|
+
track.enabled = !muted;
|
|
262
|
+
}
|
|
263
|
+
}
|
|
264
|
+
/**
|
|
265
|
+
* Returns the agent's remote audio MediaStream if available.
|
|
266
|
+
*/
|
|
267
|
+
GetRemoteMediaStream() {
|
|
268
|
+
return this.remoteStream;
|
|
269
|
+
}
|
|
270
|
+
/**
|
|
271
|
+
* Registers a callback for when the agent's remote audio MediaStream lands.
|
|
272
|
+
*/
|
|
273
|
+
OnRemoteMediaStream(handler) {
|
|
274
|
+
this.remoteStreamHandlers.push(handler);
|
|
275
|
+
if (this.remoteStream) {
|
|
276
|
+
try {
|
|
277
|
+
handler(this.remoteStream);
|
|
278
|
+
}
|
|
279
|
+
catch (error) {
|
|
280
|
+
console.warn('[OpenAILiveClient] remote-stream handler threw:', error);
|
|
281
|
+
}
|
|
282
|
+
}
|
|
283
|
+
}
|
|
284
|
+
// ── Overridable Factory / Network Seams (Tests inject fakes) ───────────────
|
|
285
|
+
/** Creates the peer connection. Production returns a real `RTCPeerConnection`. */
|
|
286
|
+
createPeerConnection() {
|
|
287
|
+
return new RTCPeerConnection();
|
|
288
|
+
}
|
|
289
|
+
/** Creates a hidden `<audio>` element to sink the model's audio output. */
|
|
290
|
+
createAudioSink() {
|
|
291
|
+
const el = document.createElement('audio');
|
|
292
|
+
el.autoplay = true;
|
|
293
|
+
el.style.display = 'none';
|
|
294
|
+
document.body.appendChild(el);
|
|
295
|
+
return el;
|
|
296
|
+
}
|
|
297
|
+
/**
|
|
298
|
+
* Exchanges local SDP offer for remote answer SDP.
|
|
299
|
+
* Posts to application server broker or direct endpoint.
|
|
300
|
+
*/
|
|
301
|
+
async postSdpOffer(offerSdp, config) {
|
|
302
|
+
const endpoint = config.EphemeralToken &&
|
|
303
|
+
(config.EphemeralToken.startsWith('http://') ||
|
|
304
|
+
config.EphemeralToken.startsWith('https://') ||
|
|
305
|
+
config.EphemeralToken.startsWith('/'))
|
|
306
|
+
? config.EphemeralToken
|
|
307
|
+
: 'https://api.openai.com/v1/live/sessions';
|
|
308
|
+
const isBroker = endpoint !== 'https://api.openai.com/v1/live/sessions';
|
|
309
|
+
const headers = {
|
|
310
|
+
'Content-Type': 'application/json',
|
|
311
|
+
};
|
|
312
|
+
if (!isBroker && config.EphemeralToken) {
|
|
313
|
+
headers['Authorization'] = `Bearer ${config.EphemeralToken}`;
|
|
314
|
+
}
|
|
315
|
+
const body = isBroker
|
|
316
|
+
? JSON.stringify({ sdp: offerSdp, session: config.SessionConfig })
|
|
317
|
+
: JSON.stringify({
|
|
318
|
+
session: config.SessionConfig ?? { model: config.Model || 'gpt-live-1' },
|
|
319
|
+
transport: {
|
|
320
|
+
type: 'webrtc',
|
|
321
|
+
sdp: offerSdp,
|
|
322
|
+
},
|
|
323
|
+
});
|
|
324
|
+
const response = await fetch(endpoint, {
|
|
325
|
+
method: 'POST',
|
|
326
|
+
headers,
|
|
327
|
+
body,
|
|
328
|
+
});
|
|
329
|
+
if (!response.ok) {
|
|
330
|
+
const detail = await response.text().catch(() => '');
|
|
331
|
+
throw new Error(`OpenAI Live WebRTC handshake failed (${response.status}): ${detail}`);
|
|
332
|
+
}
|
|
333
|
+
const contentType = response.headers.get('content-type') || '';
|
|
334
|
+
if (contentType.includes('application/json')) {
|
|
335
|
+
const data = (await response.json());
|
|
336
|
+
if (data.transport?.sdp)
|
|
337
|
+
return data.transport.sdp;
|
|
338
|
+
if (data.answerSdp)
|
|
339
|
+
return data.answerSdp;
|
|
340
|
+
if (data.sdp)
|
|
341
|
+
return data.sdp;
|
|
342
|
+
throw new Error('Invalid JSON response from WebRTC SDP broker: missing sdp');
|
|
343
|
+
}
|
|
344
|
+
return response.text();
|
|
345
|
+
}
|
|
346
|
+
// ── Internals ──────────────────────────────────────────────────────────────
|
|
347
|
+
setState(state) {
|
|
348
|
+
if (this.currentState !== state) {
|
|
349
|
+
this.currentState = state;
|
|
350
|
+
this.emitStateChange(state);
|
|
351
|
+
}
|
|
352
|
+
}
|
|
353
|
+
attachMicrophone(pc, micStream) {
|
|
354
|
+
for (const track of micStream.getAudioTracks()) {
|
|
355
|
+
pc.addTrack(track, micStream);
|
|
356
|
+
}
|
|
357
|
+
}
|
|
358
|
+
attachRemoteAudio(pc) {
|
|
359
|
+
this.remoteAudioEl = this.createAudioSink();
|
|
360
|
+
pc.ontrack = (e) => {
|
|
361
|
+
if (this.remoteAudioEl && e.streams[0]) {
|
|
362
|
+
this.remoteAudioEl.srcObject = e.streams[0];
|
|
363
|
+
this.remoteStream = e.streams[0];
|
|
364
|
+
for (const h of this.remoteStreamHandlers) {
|
|
365
|
+
try {
|
|
366
|
+
h(e.streams[0]);
|
|
367
|
+
}
|
|
368
|
+
catch (err) {
|
|
369
|
+
console.warn('[OpenAILiveClient] remote stream handler threw:', err);
|
|
370
|
+
}
|
|
371
|
+
}
|
|
372
|
+
// Capability obligation #9: agent-side audio meter
|
|
373
|
+
this.attachOutputAudioMeter(RealtimeAudioMeter.ForStream(e.streams[0]));
|
|
374
|
+
}
|
|
375
|
+
};
|
|
376
|
+
}
|
|
377
|
+
async waitForIceGatheringComplete(pc) {
|
|
378
|
+
if (!pc.iceGatheringState || pc.iceGatheringState === 'complete') {
|
|
379
|
+
return;
|
|
380
|
+
}
|
|
381
|
+
await new Promise((resolve) => {
|
|
382
|
+
const onIce = () => {
|
|
383
|
+
if (pc.iceGatheringState === 'complete') {
|
|
384
|
+
pc.removeEventListener?.('icegatheringstatechange', onIce);
|
|
385
|
+
resolve();
|
|
386
|
+
}
|
|
387
|
+
};
|
|
388
|
+
pc.addEventListener?.('icegatheringstatechange', onIce);
|
|
389
|
+
if (pc.iceGatheringState === 'complete') {
|
|
390
|
+
resolve();
|
|
391
|
+
}
|
|
392
|
+
});
|
|
393
|
+
}
|
|
394
|
+
adoptDataChannel(channel) {
|
|
395
|
+
this.dataChannel = channel;
|
|
396
|
+
channel.onopen = () => {
|
|
397
|
+
// Mechanics rule 4: never send session.start on the data channel
|
|
398
|
+
this.setState('listening');
|
|
399
|
+
this.flushOutboundQueue();
|
|
400
|
+
};
|
|
401
|
+
channel.onmessage = (e) => {
|
|
402
|
+
this.handleDataChannelMessage(String(e.data));
|
|
403
|
+
};
|
|
404
|
+
channel.onerror = (e) => {
|
|
405
|
+
this.emitError({ Message: `Data channel error: ${String(e)}`, Fatal: true });
|
|
406
|
+
this.setState('error');
|
|
407
|
+
};
|
|
408
|
+
channel.onclose = () => {
|
|
409
|
+
if (this.currentState !== 'error') {
|
|
410
|
+
this.setState('closed');
|
|
411
|
+
}
|
|
412
|
+
};
|
|
413
|
+
}
|
|
414
|
+
finalizeAssistantTranscript() {
|
|
415
|
+
this.stopPlaybackDrainMonitoring();
|
|
416
|
+
if (this.pendingAssistantText.trim().length > 0) {
|
|
417
|
+
this.emitTranscript({
|
|
418
|
+
Role: 'Assistant',
|
|
419
|
+
Text: this.pendingAssistantText,
|
|
420
|
+
IsFinal: true,
|
|
421
|
+
Kind: this.pendingNarration ? 'narration' : 'normal',
|
|
422
|
+
});
|
|
423
|
+
this.pendingAssistantText = '';
|
|
424
|
+
}
|
|
425
|
+
this.pendingNarration = false;
|
|
426
|
+
this.responseActive = false;
|
|
427
|
+
this.audioPlaying = false;
|
|
428
|
+
if (this.currentState === 'speaking') {
|
|
429
|
+
this.setState('listening');
|
|
430
|
+
}
|
|
431
|
+
}
|
|
432
|
+
finalizeUserTranscript() {
|
|
433
|
+
try {
|
|
434
|
+
if (this.pendingUserText.trim().length > 0) {
|
|
435
|
+
this.emitTranscript({
|
|
436
|
+
Role: 'User',
|
|
437
|
+
Text: this.pendingUserText,
|
|
438
|
+
IsFinal: true,
|
|
439
|
+
Kind: 'normal',
|
|
440
|
+
});
|
|
441
|
+
}
|
|
442
|
+
}
|
|
443
|
+
finally {
|
|
444
|
+
this.pendingUserText = '';
|
|
445
|
+
}
|
|
446
|
+
}
|
|
447
|
+
startPlaybackDrainMonitoring() {
|
|
448
|
+
if (this.playbackDrainInterval) {
|
|
449
|
+
return;
|
|
450
|
+
}
|
|
451
|
+
this.playbackDrainInterval = setInterval(() => {
|
|
452
|
+
this.checkPlaybackDrain();
|
|
453
|
+
}, 50);
|
|
454
|
+
}
|
|
455
|
+
stopPlaybackDrainMonitoring() {
|
|
456
|
+
if (this.playbackDrainInterval) {
|
|
457
|
+
clearInterval(this.playbackDrainInterval);
|
|
458
|
+
this.playbackDrainInterval = null;
|
|
459
|
+
}
|
|
460
|
+
if (this.assistantSafetyBackstopTimer) {
|
|
461
|
+
clearTimeout(this.assistantSafetyBackstopTimer);
|
|
462
|
+
this.assistantSafetyBackstopTimer = null;
|
|
463
|
+
}
|
|
464
|
+
this.drainSilenceStartTime = null;
|
|
465
|
+
}
|
|
466
|
+
scheduleAssistantSafetyBackstop(delayMs = OpenAILiveClient_1.ASSISTANT_SAFETY_BACKSTOP_MS) {
|
|
467
|
+
if (this.assistantSafetyBackstopTimer) {
|
|
468
|
+
clearTimeout(this.assistantSafetyBackstopTimer);
|
|
469
|
+
}
|
|
470
|
+
this.assistantSafetyBackstopTimer = setTimeout(() => {
|
|
471
|
+
this.assistantSafetyBackstopTimer = null;
|
|
472
|
+
this.finalizeAssistantTranscript();
|
|
473
|
+
}, delayMs);
|
|
474
|
+
}
|
|
475
|
+
checkPlaybackDrain() {
|
|
476
|
+
if (!this.pendingAssistantText && !this.audioPlaying) {
|
|
477
|
+
this.stopPlaybackDrainMonitoring();
|
|
478
|
+
return;
|
|
479
|
+
}
|
|
480
|
+
const now = Date.now();
|
|
481
|
+
if (now - this.lastOutputTranscriptTime < OpenAILiveClient_1.PLAYBACK_DRAIN_TRANSCRIPT_GAP_MS) {
|
|
482
|
+
this.drainSilenceStartTime = null;
|
|
483
|
+
return;
|
|
484
|
+
}
|
|
485
|
+
if (this.outputAudioMeter) {
|
|
486
|
+
const level = this.outputAudioMeter.Level();
|
|
487
|
+
if (level <= OpenAILiveClient_1.PLAYBACK_DRAIN_SILENCE_THRESHOLD) {
|
|
488
|
+
if (this.drainSilenceStartTime === null) {
|
|
489
|
+
this.drainSilenceStartTime = now;
|
|
490
|
+
}
|
|
491
|
+
else if (now - this.drainSilenceStartTime >= OpenAILiveClient_1.PLAYBACK_DRAIN_SILENCE_MS) {
|
|
492
|
+
this.finalizeAssistantTranscript();
|
|
493
|
+
}
|
|
494
|
+
}
|
|
495
|
+
else {
|
|
496
|
+
this.drainSilenceStartTime = null;
|
|
497
|
+
}
|
|
498
|
+
}
|
|
499
|
+
}
|
|
500
|
+
handleToolCallItem(callId, name, argsJson) {
|
|
501
|
+
if (!callId || !name) {
|
|
502
|
+
return;
|
|
503
|
+
}
|
|
504
|
+
if (this.emittedToolCallIds.has(callId)) {
|
|
505
|
+
return;
|
|
506
|
+
}
|
|
507
|
+
this.emittedToolCallIds.add(callId);
|
|
508
|
+
this.finalizeAssistantTranscript();
|
|
509
|
+
this.responseActive = false;
|
|
510
|
+
this.audioPlaying = false;
|
|
511
|
+
this.toolBatchBarrier.TrackPendingCall(callId, () => {
|
|
512
|
+
this.sendDataChannelFrame({
|
|
513
|
+
type: 'response.create',
|
|
514
|
+
});
|
|
515
|
+
});
|
|
516
|
+
const call = {
|
|
517
|
+
CallID: callId,
|
|
518
|
+
ToolName: name,
|
|
519
|
+
ArgumentsJson: argsJson,
|
|
520
|
+
};
|
|
521
|
+
this.emitToolCall(call);
|
|
522
|
+
}
|
|
523
|
+
handleOutputItemDone(item) {
|
|
524
|
+
if (item?.type === 'function_call') {
|
|
525
|
+
const callId = String(item.call_id ?? '');
|
|
526
|
+
const name = String(item.name ?? '');
|
|
527
|
+
const args = String(item.arguments ?? '{}');
|
|
528
|
+
this.handleToolCallItem(callId, name, args);
|
|
529
|
+
}
|
|
530
|
+
}
|
|
531
|
+
handleResponseCompleted(respOrUsage) {
|
|
532
|
+
this.emittedToolCallIds.clear();
|
|
533
|
+
this.finalizeAssistantTranscript();
|
|
534
|
+
const usage = respOrUsage?.usage ?? respOrUsage;
|
|
535
|
+
const seconds = typeof usage?.seconds === 'number' ? usage.seconds : undefined;
|
|
536
|
+
const duration = seconds !== undefined ? Math.max(OpenAILiveClient_1.PREBILL_DURATION_SECONDS, seconds) : this.totalDurationSeconds;
|
|
537
|
+
this.totalDurationSeconds = duration;
|
|
538
|
+
this.emitUsage({
|
|
539
|
+
InputTokens: typeof usage?.input_tokens === 'number' ? usage.input_tokens : 0,
|
|
540
|
+
OutputTokens: typeof usage?.output_tokens === 'number' ? usage.output_tokens : 0,
|
|
541
|
+
DurationSeconds: duration,
|
|
542
|
+
Raw: usage,
|
|
543
|
+
});
|
|
544
|
+
}
|
|
545
|
+
isResponsesDelegation() {
|
|
546
|
+
const delegation = this.sessionConfig?.['delegation'];
|
|
547
|
+
if (delegation && typeof delegation === 'object' && !Array.isArray(delegation)) {
|
|
548
|
+
return delegation['type'] === 'responses';
|
|
549
|
+
}
|
|
550
|
+
return false;
|
|
551
|
+
}
|
|
552
|
+
handleDataChannelMessage(raw) {
|
|
553
|
+
let event;
|
|
554
|
+
try {
|
|
555
|
+
event = JSON.parse(raw);
|
|
556
|
+
}
|
|
557
|
+
catch {
|
|
558
|
+
return;
|
|
559
|
+
}
|
|
560
|
+
const type = typeof event.type === 'string' ? event.type : '';
|
|
561
|
+
switch (type) {
|
|
562
|
+
case 'session.started': {
|
|
563
|
+
this.setState('listening');
|
|
564
|
+
break;
|
|
565
|
+
}
|
|
566
|
+
case 'session.input_transcript.delta': {
|
|
567
|
+
const delta = typeof event.delta === 'string' ? event.delta : typeof event.text === 'string' ? event.text : '';
|
|
568
|
+
if (!delta) {
|
|
569
|
+
break;
|
|
570
|
+
}
|
|
571
|
+
if (this.pendingAssistantText) {
|
|
572
|
+
this.finalizeAssistantTranscript();
|
|
573
|
+
}
|
|
574
|
+
this.pendingUserText += delta;
|
|
575
|
+
this.emitTranscript({
|
|
576
|
+
Role: 'User',
|
|
577
|
+
Text: delta,
|
|
578
|
+
IsFinal: false,
|
|
579
|
+
Kind: 'normal',
|
|
580
|
+
});
|
|
581
|
+
break;
|
|
582
|
+
}
|
|
583
|
+
case 'session.output_transcript.delta': {
|
|
584
|
+
const delta = typeof event.delta === 'string' ? event.delta : typeof event.text === 'string' ? event.text : '';
|
|
585
|
+
if (!delta) {
|
|
586
|
+
break;
|
|
587
|
+
}
|
|
588
|
+
if (this.pendingUserText) {
|
|
589
|
+
this.finalizeUserTranscript();
|
|
590
|
+
}
|
|
591
|
+
this.responseActive = true;
|
|
592
|
+
this.audioPlaying = true;
|
|
593
|
+
if (this.currentState !== 'speaking') {
|
|
594
|
+
this.setState('speaking');
|
|
595
|
+
}
|
|
596
|
+
this.pendingAssistantText += delta;
|
|
597
|
+
this.emitTranscript({
|
|
598
|
+
Role: 'Assistant',
|
|
599
|
+
Text: delta,
|
|
600
|
+
IsFinal: false,
|
|
601
|
+
Kind: this.pendingNarration ? 'narration' : 'normal',
|
|
602
|
+
});
|
|
603
|
+
this.lastOutputTranscriptTime = Date.now();
|
|
604
|
+
this.drainSilenceStartTime = null;
|
|
605
|
+
this.startPlaybackDrainMonitoring();
|
|
606
|
+
this.scheduleAssistantSafetyBackstop();
|
|
607
|
+
break;
|
|
608
|
+
}
|
|
609
|
+
case 'response.event': {
|
|
610
|
+
const inner = event.event;
|
|
611
|
+
if (inner?.type === 'response.output_item.done') {
|
|
612
|
+
this.handleOutputItemDone(inner.item);
|
|
613
|
+
}
|
|
614
|
+
else if (inner?.type === 'response.function_call_arguments.done') {
|
|
615
|
+
const callId = String(inner.call_id ?? '');
|
|
616
|
+
const name = String(inner.name ?? '');
|
|
617
|
+
const args = String(inner.arguments ?? '{}');
|
|
618
|
+
this.handleToolCallItem(callId, name, args);
|
|
619
|
+
}
|
|
620
|
+
else if (inner?.type === 'response.completed' || inner?.type === 'response.done') {
|
|
621
|
+
this.handleResponseCompleted(inner.response ?? inner);
|
|
622
|
+
}
|
|
623
|
+
break;
|
|
624
|
+
}
|
|
625
|
+
case 'session.delegation.created': {
|
|
626
|
+
this.finalizeAssistantTranscript();
|
|
627
|
+
this.finalizeUserTranscript();
|
|
628
|
+
this.responseActive = false;
|
|
629
|
+
this.audioPlaying = false;
|
|
630
|
+
// In responses delegation mode (remote reasoning), session.delegation.created marks the start
|
|
631
|
+
// of reasoning on the responses model (e.g. gpt-4o). Tool calls are emitted individually
|
|
632
|
+
// via response.event (response.output_item.done) with their real function names.
|
|
633
|
+
// Only emit 'backend_delegation' when in client delegation mode (local plane).
|
|
634
|
+
if (!this.isResponsesDelegation()) {
|
|
635
|
+
const delegationId = String(event.delegation_id ?? '');
|
|
636
|
+
if (delegationId) {
|
|
637
|
+
this.clientDelegationCallIds.add(delegationId);
|
|
638
|
+
const call = {
|
|
639
|
+
CallID: delegationId,
|
|
640
|
+
ToolName: 'backend_delegation',
|
|
641
|
+
ArgumentsJson: JSON.stringify({ delegation_id: delegationId }),
|
|
642
|
+
};
|
|
643
|
+
this.emitToolCall(call);
|
|
644
|
+
}
|
|
645
|
+
}
|
|
646
|
+
break;
|
|
647
|
+
}
|
|
648
|
+
case 'session.usage.updated': {
|
|
649
|
+
const usage = event.usage;
|
|
650
|
+
const seconds = typeof usage?.seconds === 'number' ? usage.seconds : undefined;
|
|
651
|
+
const duration = seconds !== undefined ? Math.max(OpenAILiveClient_1.PREBILL_DURATION_SECONDS, seconds) : this.totalDurationSeconds;
|
|
652
|
+
this.totalDurationSeconds = duration;
|
|
653
|
+
this.emitUsage({
|
|
654
|
+
InputTokens: typeof usage?.input_tokens === 'number' ? usage.input_tokens : 0,
|
|
655
|
+
OutputTokens: typeof usage?.output_tokens === 'number' ? usage.output_tokens : 0,
|
|
656
|
+
DurationSeconds: duration,
|
|
657
|
+
Raw: usage,
|
|
658
|
+
});
|
|
659
|
+
break;
|
|
660
|
+
}
|
|
661
|
+
case 'error': {
|
|
662
|
+
const err = event.error;
|
|
663
|
+
const message = typeof err?.message === 'string' ? err.message : 'Unknown OpenAI Live error';
|
|
664
|
+
const code = typeof err?.code === 'string' ? err.code : undefined;
|
|
665
|
+
const isFatal = !this.isActive() || code === 'session_expired' || code === 'unauthorized';
|
|
666
|
+
// Provider cut-off signal (moderation or provider abort mid-speech)
|
|
667
|
+
if (this.responseActive || this.audioPlaying) {
|
|
668
|
+
this.emitInterruption();
|
|
669
|
+
this.CancelActiveResponse();
|
|
670
|
+
}
|
|
671
|
+
this.emitError({ Message: message, Code: code, Fatal: isFatal });
|
|
672
|
+
if (isFatal) {
|
|
673
|
+
this.setState('error');
|
|
674
|
+
}
|
|
675
|
+
break;
|
|
676
|
+
}
|
|
677
|
+
case 'session.closed': {
|
|
678
|
+
this.setState('closed');
|
|
679
|
+
void this.Disconnect();
|
|
680
|
+
break;
|
|
681
|
+
}
|
|
682
|
+
default:
|
|
683
|
+
break;
|
|
684
|
+
}
|
|
685
|
+
}
|
|
686
|
+
isActive() {
|
|
687
|
+
return this.currentState === 'connected' || this.currentState === 'listening' || this.currentState === 'speaking';
|
|
688
|
+
}
|
|
689
|
+
sendDataChannelFrame(payload) {
|
|
690
|
+
const frameType = typeof payload.type === 'string' ? payload.type : 'unknown';
|
|
691
|
+
if (this.dataChannel && this.dataChannel.readyState === 'open') {
|
|
692
|
+
try {
|
|
693
|
+
this.dataChannel.send(JSON.stringify(payload));
|
|
694
|
+
}
|
|
695
|
+
catch (err) {
|
|
696
|
+
console.warn('[OpenAILiveClient] Failed to send data channel frame:', err);
|
|
697
|
+
}
|
|
698
|
+
}
|
|
699
|
+
else if (!this.dataChannel || this.dataChannel.readyState === 'connecting') {
|
|
700
|
+
if (this.outboundQueue.length >= OpenAILiveClient_1.MAX_OUTBOUND_QUEUE_SIZE) {
|
|
701
|
+
const dropped = this.outboundQueue.shift();
|
|
702
|
+
const droppedType = typeof dropped?.type === 'string' ? dropped.type : 'unknown';
|
|
703
|
+
console.warn(`[OpenAILiveClient] outboundQueue reached max capacity (${OpenAILiveClient_1.MAX_OUTBOUND_QUEUE_SIZE}), dropped oldest frame (${droppedType})`);
|
|
704
|
+
}
|
|
705
|
+
this.outboundQueue.push(payload);
|
|
706
|
+
}
|
|
707
|
+
else {
|
|
708
|
+
console.warn(`[OpenAILiveClient] Dropping frame (${frameType}) because data channel is ${this.dataChannel.readyState}`);
|
|
709
|
+
}
|
|
710
|
+
}
|
|
711
|
+
flushOutboundQueue() {
|
|
712
|
+
while (this.outboundQueue.length > 0 && this.dataChannel && this.dataChannel.readyState === 'open') {
|
|
713
|
+
const frame = this.outboundQueue.shift();
|
|
714
|
+
if (frame) {
|
|
715
|
+
this.sendDataChannelFrame(frame);
|
|
716
|
+
}
|
|
717
|
+
}
|
|
718
|
+
}
|
|
719
|
+
};
|
|
720
|
+
OpenAILiveClient = OpenAILiveClient_1 = __decorate([
|
|
721
|
+
RegisterClass(BaseRealtimeClient, 'openai-live'),
|
|
722
|
+
RegisterClass(BaseRealtimeClient, 'OpenAILiveRealtime')
|
|
723
|
+
], OpenAILiveClient);
|
|
724
|
+
export { OpenAILiveClient };
|
|
725
|
+
/**
|
|
726
|
+
* Tree-shaking prevention function for OpenAILiveClient.
|
|
727
|
+
*/
|
|
728
|
+
export function LoadOpenAILiveClient() {
|
|
729
|
+
// Intentional no-op — ensures ClassFactory registration executes
|
|
730
|
+
}
|
|
731
|
+
//# sourceMappingURL=openAILiveClient.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"openAILiveClient.js","sourceRoot":"","sources":["../../src/drivers/openAILiveClient.ts"],"names":[],"mappings":";;;;;;;AAAA,OAAO,EAAE,aAAa,EAAE,MAAM,wBAAwB,CAAC;AACvD,OAAO,EAA2C,wBAAwB,EAAE,MAAM,oBAAoB,CAAC;AACvG,OAAO,EAAE,kBAAkB,EAAE,MAAM,qBAAqB,CAAC;AACzD,OAAO,EACH,kBAAkB,GAMrB,MAAM,+BAA+B,CAAC;AAgBvC;;;;;;;;;;;;;;GAcG;AAGI,IAAM,gBAAgB,GAAtB,MAAM,gBAAiB,SAAQ,kBAAkB;IAAjD;;QAQH,8EAA8E;QACtE,mBAAc,GAAuC,IAAI,CAAC;QACxD,gBAAW,GAAgC,IAAI,CAAC;QAClD,kBAAa,GAA8B,IAAI,CAAC;QAChD,iBAAY,GAAuB,IAAI,CAAC;QACxC,yBAAoB,GAAyC,EAAE,CAAC;QAChE,cAAS,GAAuB,IAAI,CAAC;QACnC,kBAAa,GAAsB,IAAI,CAAC;QAElD,8EAA8E;QACtE,iBAAY,GAAwB,QAAQ,CAAC;QAC7C,mBAAc,GAAG,KAAK,CAAC;QACvB,iBAAY,GAAG,KAAK,CAAC;QACrB,qBAAgB,GAAG,KAAK,CAAC;QACzB,yBAAoB,GAAG,kBAAgB,CAAC,wBAAwB,CAAC;QACjE,oBAAe,GAAG,EAAE,CAAC;QACrB,yBAAoB,GAAG,EAAE,CAAC;QAC1B,6BAAwB,GAAG,CAAC,CAAC;QAC7B,0BAAqB,GAAkB,IAAI,CAAC;QAC5C,0BAAqB,GAA0C,IAAI,CAAC;QACpE,iCAA4B,GAAyC,IAAI,CAAC;QAC1E,4BAAuB,GAAG,IAAI,GAAG,EAAU,CAAC;QAC5C,uBAAkB,GAAG,IAAI,GAAG,EAAU,CAAC;QACvC,qBAAgB,GAAG,IAAI,wBAAwB,EAAE,CAAC;QAClD,kBAAa,GAAmC,EAAE,CAAC;IA4tB/D,CAAC;;aA3vB0B,6BAAwB,GAAG,EAAE,AAAL,CAAM;aAC9B,8BAAyB,GAAG,GAAG,AAAN,CAAO;aAChC,qCAAgC,GAAG,GAAG,AAAN,CAAO;aACvC,qCAAgC,GAAG,IAAI,AAAP,CAAQ;aACxC,iCAA4B,GAAG,IAAI,AAAP,CAAQ;aACpC,4BAAuB,GAAG,GAAG,AAAN,CAAO;IA4BrD,8EAA8E;IAE9E,IAAW,MAAM;QACb,OAAO,IAAI,CAAC,cAAc,CAAC;IAC/B,CAAC;IAED,IAAW,cAAc;QACrB,OAAO,IAAI,CAAC,YAAY,CAAC;IAC7B,CAAC;IAED,IAAW,cAAc;QACrB,OAAO,kBAAgB,CAAC,wBAAwB,CAAC;IACrD,CAAC;IAED,IAAW,KAAK;QACZ,OAAO,IAAI,CAAC,YAAY,CAAC;IAC7B,CAAC;IAED;;OAEG;IACI,KAAK,CAAC,OAAO,CAChB,MAAmC,EACnC,SAAsB,EACtB,aAA2B;QAE3B,IAAI,CAAC,aAAa,GAAG,MAAM,CAAC,aAAa,CAAC;QAC1C,IAAI,CAAC,SAAS,GAAG,SAAS,CAAC;QAC3B,IAAI,CAAC,QAAQ,CAAC,YAAY,CAAC,CAAC;QAE5B,MAAM,EAAE,GAAG,IAAI,CAAC,oBAAoB,EAAE,CAAC;QACvC,IAAI,CAAC,cAAc,GAAG,EAAE,CAAC;QACzB,IAAI,CAAC,gBAAgB,CAAC,EAAE,EAAE,SAAS,CAAC,CAAC;QACrC,IAAI,CAAC,iBAAiB,CAAC,EAAE,CAAC,CAAC;QAE3B,kDAAkD;QAClD,IAAI,CAAC,qBAAqB,CAAC,kBAAkB,CAAC,YAAY,CAAC,SAAS,CAAC,CAAC,CAAC;QAEvE,6DAA6D;QAC7D,MAAM,OAAO,GAAG,EAAE,CAAC,iBAAiB,CAAC,YAAY,CAAC,CAAC;QACnD,IAAI,CAAC,gBAAgB,CAAC,OAAO,CAAC,CAAC;QAE/B,qBAAqB;QACrB,MAAM,KAAK,GAAG,MAAM,EAAE,CAAC,WAAW,EAAE,CAAC;QACrC,MAAM,EAAE,CAAC,mBAAmB,CAAC,KAAK,CAAC,CAAC;QAEpC,gFAAgF;QAChF,MAAM,IAAI,CAAC,2BAA2B,CAAC,EAAE,CAAC,CAAC;QAE3C,gFAAgF;QAChF,MAAM,SAAS,GAAG,MAAM,IAAI,CAAC,YAAY,CAAC,KAAK,CAAC,GAAG,IAAI,EAAE,EAAE,MAAM,CAAC,CAAC;QACnE,MAAM,EAAE,CAAC,oBAAoB,CAAC,EAAE,IAAI,EAAE,QAAQ,EAAE,GAAG,EAAE,SAAS,EAAE,CAAC,CAAC;QAElE,IAAI,CAAC,QAAQ,CAAC,WAAW,CAAC,CAAC;QAE3B,qDAAqD;QACrD,IAAI,CAAC,oBAAoB,GAAG,kBAAgB,CAAC,wBAAwB,CAAC;QACtE,IAAI,CAAC,SAAS,CAAC;YACX,eAAe,EAAE,kBAAgB,CAAC,wBAAwB;YAC1D,WAAW,EAAE,CAAC;YACd,YAAY,EAAE,CAAC;SAClB,CAAC,CAAC;IACP,CAAC;IAED;;OAEG;IACI,KAAK,CAAC,UAAU;QACnB,IAAI,CAAC,sBAAsB,EAAE,CAAC;QAC9B,IAAI,CAAC,2BAA2B,EAAE,CAAC;QAEnC,IAAI,CAAC,gBAAgB,EAAE,CAAC;QACxB,IAAI,CAAC,SAAS,EAAE,SAAS,EAAE,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC,CAAC;QACrD,IAAI,CAAC,SAAS,GAAG,IAAI,CAAC;QAEtB,IAAI,IAAI,CAAC,WAAW,EAAE,CAAC;YACnB,IAAI,CAAC;gBACD,IAAI,CAAC,WAAW,CAAC,KAAK,EAAE,CAAC;YAC7B,CAAC;YAAC,MAAM,CAAC;gBACL,oBAAoB;YACxB,CAAC;YACD,IAAI,CAAC,WAAW,GAAG,IAAI,CAAC;QAC5B,CAAC;QAED,IAAI,IAAI,CAAC,cAAc,EAAE,CAAC;YACtB,IAAI,CAAC;gBACD,IAAI,CAAC,cAAc,CAAC,KAAK,EAAE,CAAC;YAChC,CAAC;YAAC,MAAM,CAAC;gBACL,oBAAoB;YACxB,CAAC;YACD,IAAI,CAAC,cAAc,GAAG,IAAI,CAAC;QAC/B,CAAC;QAED,IAAI,IAAI,CAAC,aAAa,EAAE,CAAC;YACrB,IAAI,CAAC,aAAa,CAAC,SAAS,GAAG,IAAI,CAAC;YACpC,IAAI,CAAC,aAAa,CAAC,MAAM,EAAE,CAAC;YAC5B,IAAI,CAAC,aAAa,GAAG,IAAI,CAAC;QAC9B,CAAC;QAED,IAAI,CAAC,YAAY,GAAG,IAAI,CAAC;QACzB,IAAI,CAAC,oBAAoB,GAAG,EAAE,CAAC;QAC/B,IAAI,CAAC,aAAa,GAAG,IAAI,CAAC;QAC1B,IAAI,CAAC,cAAc,GAAG,KAAK,CAAC;QAC5B,IAAI,CAAC,YAAY,GAAG,KAAK,CAAC;QAC1B,IAAI,CAAC,gBAAgB,GAAG,KAAK,CAAC;QAC9B,IAAI,CAAC,2BAA2B,EAAE,CAAC;QACnC,IAAI,CAAC,eAAe,GAAG,EAAE,CAAC;QAC1B,IAAI,CAAC,oBAAoB,GAAG,EAAE,CAAC;QAC/B,IAAI,CAAC,uBAAuB,CAAC,KAAK,EAAE,CAAC;QACrC,IAAI,CAAC,kBAAkB,CAAC,KAAK,EAAE,CAAC;QAChC,IAAI,CAAC,gBAAgB,CAAC,KAAK,EAAE,CAAC;QAC9B,IAAI,CAAC,aAAa,GAAG,EAAE,CAAC;QAExB,IAAI,IAAI,CAAC,YAAY,KAAK,OAAO,EAAE,CAAC;YAChC,IAAI,CAAC,QAAQ,CAAC,QAAQ,CAAC,CAAC;QAC5B,CAAC;IACL,CAAC;IAED;;;OAGG;IACI,QAAQ,CAAC,IAAY;QACxB,IAAI,CAAC,oBAAoB,EAAE,CAAC;QAC5B,IAAI,CAAC,oBAAoB,CAAC;YACtB,IAAI,EAAE,2BAA2B;YACjC,OAAO,EAAE,IAAI;YACb,aAAa,EAAE,IAAI;SACtB,CAAC,CAAC;QACH,IAAI,CAAC,oBAAoB,CAAC;YACtB,IAAI,EAAE,iBAAiB;SAC1B,CAAC,CAAC;IACP,CAAC;IAED;;OAEG;IACI,oBAAoB;QACvB,IAAI,CAAC,2BAA2B,EAAE,CAAC;QACnC,IAAI,CAAC,oBAAoB,GAAG,EAAE,CAAC;QAC/B,8DAA8D;QAC9D,yEAAyE;QACzE,4EAA4E;QAC5E,6EAA6E;QAC7E,+EAA+E;QAC/E,wEAAwE;QACxE,IAAI,IAAI,CAAC,cAAc,IAAI,IAAI,CAAC,YAAY,EAAE,CAAC;YAC3C,wEAAwE;YACxE,6DAA6D;YAC7D,IAAI,CAAC,cAAc,GAAG,KAAK,CAAC;YAC5B,IAAI,CAAC,YAAY,GAAG,KAAK,CAAC;YAC1B,IAAI,CAAC,gBAAgB,GAAG,KAAK,CAAC;YAC9B,IAAI,IAAI,CAAC,YAAY,KAAK,UAAU,EAAE,CAAC;gBACnC,IAAI,CAAC,QAAQ,CAAC,WAAW,CAAC,CAAC;YAC/B,CAAC;QACL,CAAC;IACL,CAAC;IAED;;;OAGG;IACI,eAAe,CAAC,IAAY;QAC/B,IAAI,CAAC,oBAAoB,CAAC;YACtB,IAAI,EAAE,yBAAyB;YAC/B,OAAO,EAAE,IAAI;YACb,aAAa,EAAE,IAAI;SACtB,CAAC,CAAC;IACP,CAAC;IAED;;OAEG;IACI,mBAAmB,CAAC,YAAoB;QAC3C,IAAI,IAAI,CAAC,cAAc,IAAI,IAAI,CAAC,YAAY,EAAE,CAAC;YAC3C,2EAA2E;YAC3E,OAAO;QACX,CAAC;QACD,IAAI,CAAC,gBAAgB,GAAG,IAAI,CAAC;QAC7B,IAAI,CAAC,oBAAoB,CAAC;YACtB,IAAI,EAAE,2BAA2B;YACjC,OAAO,EAAE,YAAY;YACrB,aAAa,EAAE,IAAI;SACtB,CAAC,CAAC;QACH,IAAI,IAAI,CAAC,gBAAgB,CAAC,OAAO,EAAE,CAAC;YAChC,IAAI,CAAC,oBAAoB,CAAC;gBACtB,IAAI,EAAE,iBAAiB;aAC1B,CAAC,CAAC;QACP,CAAC;IACL,CAAC;IAED;;;;OAIG;IACI,cAAc,CAAC,MAAc,EAAE,UAAkB;QACpD,IAAI,IAAI,CAAC,uBAAuB,CAAC,GAAG,CAAC,MAAM,CAAC,EAAE,CAAC;YAC3C,IAAI,CAAC,uBAAuB,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC;YAC5C,IAAI,CAAC,oBAAoB,CAAC;gBACtB,IAAI,EAAE,2BAA2B;gBACjC,OAAO,EAAE,UAAU;gBACnB,aAAa,EAAE,MAAM;aACxB,CAAC,CAAC;YACH,OAAO;QACX,CAAC;QAED,IAAI,CAAC,oBAAoB,CAAC;YACtB,IAAI,EAAE,sBAAsB;YAC5B,IAAI,EAAE;gBACF,IAAI,EAAE,sBAAsB;gBAC5B,OAAO,EAAE,MAAM;gBACf,MAAM,EAAE,UAAU;aACrB;SACJ,CAAC,CAAC;QAEH,MAAM,eAAe,GAAG,IAAI,CAAC,gBAAgB,CAAC,YAAY,CAAC,MAAM,CAAC,CAAC;QACnE,IAAI,eAAe,EAAE,CAAC;YAClB,IAAI,CAAC,oBAAoB,CAAC;gBACtB,IAAI,EAAE,iBAAiB;aAC1B,CAAC,CAAC;QACP,CAAC;IACL,CAAC;IAED;;OAEG;IACI,QAAQ,CAAC,KAAc;QAC1B,KAAK,MAAM,KAAK,IAAI,IAAI,CAAC,SAAS,EAAE,cAAc,EAAE,IAAI,EAAE,EAAE,CAAC;YACzD,KAAK,CAAC,OAAO,GAAG,CAAC,KAAK,CAAC;QAC3B,CAAC;IACL,CAAC;IAED;;OAEG;IACa,oBAAoB;QAChC,OAAO,IAAI,CAAC,YAAY,CAAC;IAC7B,CAAC;IAED;;OAEG;IACa,mBAAmB,CAAC,OAAsC;QACtE,IAAI,CAAC,oBAAoB,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC;QACxC,IAAI,IAAI,CAAC,YAAY,EAAE,CAAC;YACpB,IAAI,CAAC;gBACD,OAAO,CAAC,IAAI,CAAC,YAAY,CAAC,CAAC;YAC/B,CAAC;YAAC,OAAO,KAAK,EAAE,CAAC;gBACb,OAAO,CAAC,IAAI,CAAC,iDAAiD,EAAE,KAAK,CAAC,CAAC;YAC3E,CAAC;QACL,CAAC;IACL,CAAC;IAED,8EAA8E;IAE9E,kFAAkF;IACxE,oBAAoB;QAC1B,OAAO,IAAI,iBAAiB,EAAE,CAAC;IACnC,CAAC;IAED,2EAA2E;IACjE,eAAe;QACrB,MAAM,EAAE,GAAG,QAAQ,CAAC,aAAa,CAAC,OAAO,CAAC,CAAC;QAC3C,EAAE,CAAC,QAAQ,GAAG,IAAI,CAAC;QACnB,EAAE,CAAC,KAAK,CAAC,OAAO,GAAG,MAAM,CAAC;QAC1B,QAAQ,CAAC,IAAI,CAAC,WAAW,CAAC,EAAE,CAAC,CAAC;QAC9B,OAAO,EAAE,CAAC;IACd,CAAC;IAED;;;OAGG;IACO,KAAK,CAAC,YAAY,CAAC,QAAgB,EAAE,MAAmC;QAC9E,MAAM,QAAQ,GACV,MAAM,CAAC,cAAc;YACrB,CAAC,MAAM,CAAC,cAAc,CAAC,UAAU,CAAC,SAAS,CAAC;gBACxC,MAAM,CAAC,cAAc,CAAC,UAAU,CAAC,UAAU,CAAC;gBAC5C,MAAM,CAAC,cAAc,CAAC,UAAU,CAAC,GAAG,CAAC,CAAC;YACtC,CAAC,CAAC,MAAM,CAAC,cAAc;YACvB,CAAC,CAAC,yCAAyC,CAAC;QAEpD,MAAM,QAAQ,GAAG,QAAQ,KAAK,yCAAyC,CAAC;QACxE,MAAM,OAAO,GAA2B;YACpC,cAAc,EAAE,kBAAkB;SACrC,CAAC;QAEF,IAAI,CAAC,QAAQ,IAAI,MAAM,CAAC,cAAc,EAAE,CAAC;YACrC,OAAO,CAAC,eAAe,CAAC,GAAG,UAAU,MAAM,CAAC,cAAc,EAAE,CAAC;QACjE,CAAC;QAED,MAAM,IAAI,GAAG,QAAQ;YACjB,CAAC,CAAC,IAAI,CAAC,SAAS,CAAC,EAAE,GAAG,EAAE,QAAQ,EAAE,OAAO,EAAE,MAAM,CAAC,aAAa,EAAE,CAAC;YAClE,CAAC,CAAC,IAAI,CAAC,SAAS,CAAC;gBACX,OAAO,EAAE,MAAM,CAAC,aAAa,IAAI,EAAE,KAAK,EAAE,MAAM,CAAC,KAAK,IAAI,YAAY,EAAE;gBACxE,SAAS,EAAE;oBACP,IAAI,EAAE,QAAQ;oBACd,GAAG,EAAE,QAAQ;iBAChB;aACJ,CAAC,CAAC;QAET,MAAM,QAAQ,GAAG,MAAM,KAAK,CAAC,QAAQ,EAAE;YACnC,MAAM,EAAE,MAAM;YACd,OAAO;YACP,IAAI;SACP,CAAC,CAAC;QAEH,IAAI,CAAC,QAAQ,CAAC,EAAE,EAAE,CAAC;YACf,MAAM,MAAM,GAAG,MAAM,QAAQ,CAAC,IAAI,EAAE,CAAC,KAAK,CAAC,GAAG,EAAE,CAAC,EAAE,CAAC,CAAC;YACrD,MAAM,IAAI,KAAK,CAAC,wCAAwC,QAAQ,CAAC,MAAM,MAAM,MAAM,EAAE,CAAC,CAAC;QAC3F,CAAC;QAED,MAAM,WAAW,GAAG,QAAQ,CAAC,OAAO,CAAC,GAAG,CAAC,cAAc,CAAC,IAAI,EAAE,CAAC;QAC/D,IAAI,WAAW,CAAC,QAAQ,CAAC,kBAAkB,CAAC,EAAE,CAAC;YAC3C,MAAM,IAAI,GAAG,CAAC,MAAM,QAAQ,CAAC,IAAI,EAAE,CAIlC,CAAC;YACF,IAAI,IAAI,CAAC,SAAS,EAAE,GAAG;gBAAE,OAAO,IAAI,CAAC,SAAS,CAAC,GAAG,CAAC;YACnD,IAAI,IAAI,CAAC,SAAS;gBAAE,OAAO,IAAI,CAAC,SAAS,CAAC;YAC1C,IAAI,IAAI,CAAC,GAAG;gBAAE,OAAO,IAAI,CAAC,GAAG,CAAC;YAC9B,MAAM,IAAI,KAAK,CAAC,2DAA2D,CAAC,CAAC;QACjF,CAAC;QAED,OAAO,QAAQ,CAAC,IAAI,EAAE,CAAC;IAC3B,CAAC;IAED,8EAA8E;IAEtE,QAAQ,CAAC,KAA0B;QACvC,IAAI,IAAI,CAAC,YAAY,KAAK,KAAK,EAAE,CAAC;YAC9B,IAAI,CAAC,YAAY,GAAG,KAAK,CAAC;YAC1B,IAAI,CAAC,eAAe,CAAC,KAAK,CAAC,CAAC;QAChC,CAAC;IACL,CAAC;IAEO,gBAAgB,CAAC,EAA2B,EAAE,SAAsB;QACxE,KAAK,MAAM,KAAK,IAAI,SAAS,CAAC,cAAc,EAAE,EAAE,CAAC;YAC7C,EAAE,CAAC,QAAQ,CAAC,KAAK,EAAE,SAAS,CAAC,CAAC;QAClC,CAAC;IACL,CAAC;IAEO,iBAAiB,CAAC,EAA2B;QACjD,IAAI,CAAC,aAAa,GAAG,IAAI,CAAC,eAAe,EAAE,CAAC;QAC5C,EAAE,CAAC,OAAO,GAAG,CAAC,CAAgB,EAAE,EAAE;YAC9B,IAAI,IAAI,CAAC,aAAa,IAAI,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE,CAAC;gBACrC,IAAI,CAAC,aAAa,CAAC,SAAS,GAAG,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC;gBAC5C,IAAI,CAAC,YAAY,GAAG,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC;gBACjC,KAAK,MAAM,CAAC,IAAI,IAAI,CAAC,oBAAoB,EAAE,CAAC;oBACxC,IAAI,CAAC;wBACD,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC;oBACpB,CAAC;oBAAC,OAAO,GAAG,EAAE,CAAC;wBACX,OAAO,CAAC,IAAI,CAAC,iDAAiD,EAAE,GAAG,CAAC,CAAC;oBACzE,CAAC;gBACL,CAAC;gBACD,mDAAmD;gBACnD,IAAI,CAAC,sBAAsB,CAAC,kBAAkB,CAAC,SAAS,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;YAC5E,CAAC;QACL,CAAC,CAAC;IACN,CAAC;IAEO,KAAK,CAAC,2BAA2B,CAAC,EAA+B;QACrE,IAAI,CAAC,EAAE,CAAC,iBAAiB,IAAI,EAAE,CAAC,iBAAiB,KAAK,UAAU,EAAE,CAAC;YAC/D,OAAO;QACX,CAAC;QAED,MAAM,IAAI,OAAO,CAAO,CAAC,OAAO,EAAE,EAAE;YAChC,MAAM,KAAK,GAAG,GAAG,EAAE;gBACf,IAAI,EAAE,CAAC,iBAAiB,KAAK,UAAU,EAAE,CAAC;oBACtC,EAAE,CAAC,mBAAmB,EAAE,CAAC,yBAAyB,EAAE,KAAK,CAAC,CAAC;oBAC3D,OAAO,EAAE,CAAC;gBACd,CAAC;YACL,CAAC,CAAC;YACF,EAAE,CAAC,gBAAgB,EAAE,CAAC,yBAAyB,EAAE,KAAK,CAAC,CAAC;YACxD,IAAI,EAAE,CAAC,iBAAiB,KAAK,UAAU,EAAE,CAAC;gBACtC,OAAO,EAAE,CAAC;YACd,CAAC;QACL,CAAC,CAAC,CAAC;IACP,CAAC;IAES,gBAAgB,CAAC,OAA6B;QACpD,IAAI,CAAC,WAAW,GAAG,OAAO,CAAC;QAC3B,OAAO,CAAC,MAAM,GAAG,GAAG,EAAE;YAClB,iEAAiE;YACjE,IAAI,CAAC,QAAQ,CAAC,WAAW,CAAC,CAAC;YAC3B,IAAI,CAAC,kBAAkB,EAAE,CAAC;QAC9B,CAAC,CAAC;QAEF,OAAO,CAAC,SAAS,GAAG,CAAC,CAAe,EAAE,EAAE;YACpC,IAAI,CAAC,wBAAwB,CAAC,MAAM,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC;QAClD,CAAC,CAAC;QAEF,OAAO,CAAC,OAAO,GAAG,CAAC,CAAQ,EAAE,EAAE;YAC3B,IAAI,CAAC,SAAS,CAAC,EAAE,OAAO,EAAE,uBAAuB,MAAM,CAAC,CAAC,CAAC,EAAE,EAAE,KAAK,EAAE,IAAI,EAAE,CAAC,CAAC;YAC7E,IAAI,CAAC,QAAQ,CAAC,OAAO,CAAC,CAAC;QAC3B,CAAC,CAAC;QAEF,OAAO,CAAC,OAAO,GAAG,GAAG,EAAE;YACnB,IAAI,IAAI,CAAC,YAAY,KAAK,OAAO,EAAE,CAAC;gBAChC,IAAI,CAAC,QAAQ,CAAC,QAAQ,CAAC,CAAC;YAC5B,CAAC;QACL,CAAC,CAAC;IACN,CAAC;IAEO,2BAA2B;QAC/B,IAAI,CAAC,2BAA2B,EAAE,CAAC;QACnC,IAAI,IAAI,CAAC,oBAAoB,CAAC,IAAI,EAAE,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;YAC9C,IAAI,CAAC,cAAc,CAAC;gBAChB,IAAI,EAAE,WAAW;gBACjB,IAAI,EAAE,IAAI,CAAC,oBAAoB;gBAC/B,OAAO,EAAE,IAAI;gBACb,IAAI,EAAE,IAAI,CAAC,gBAAgB,CAAC,CAAC,CAAC,WAAW,CAAC,CAAC,CAAC,QAAQ;aACvD,CAAC,CAAC;YACH,IAAI,CAAC,oBAAoB,GAAG,EAAE,CAAC;QACnC,CAAC;QACD,IAAI,CAAC,gBAAgB,GAAG,KAAK,CAAC;QAC9B,IAAI,CAAC,cAAc,GAAG,KAAK,CAAC;QAC5B,IAAI,CAAC,YAAY,GAAG,KAAK,CAAC;QAC1B,IAAI,IAAI,CAAC,YAAY,KAAK,UAAU,EAAE,CAAC;YACnC,IAAI,CAAC,QAAQ,CAAC,WAAW,CAAC,CAAC;QAC/B,CAAC;IACL,CAAC;IAEO,sBAAsB;QAC1B,IAAI,CAAC;YACD,IAAI,IAAI,CAAC,eAAe,CAAC,IAAI,EAAE,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;gBACzC,IAAI,CAAC,cAAc,CAAC;oBAChB,IAAI,EAAE,MAAM;oBACZ,IAAI,EAAE,IAAI,CAAC,eAAe;oBAC1B,OAAO,EAAE,IAAI;oBACb,IAAI,EAAE,QAAQ;iBACjB,CAAC,CAAC;YACP,CAAC;QACL,CAAC;gBAAS,CAAC;YACP,IAAI,CAAC,eAAe,GAAG,EAAE,CAAC;QAC9B,CAAC;IACL,CAAC;IAEO,4BAA4B;QAChC,IAAI,IAAI,CAAC,qBAAqB,EAAE,CAAC;YAC7B,OAAO;QACX,CAAC;QACD,IAAI,CAAC,qBAAqB,GAAG,WAAW,CAAC,GAAG,EAAE;YAC1C,IAAI,CAAC,kBAAkB,EAAE,CAAC;QAC9B,CAAC,EAAE,EAAE,CAAC,CAAC;IACX,CAAC;IAEO,2BAA2B;QAC/B,IAAI,IAAI,CAAC,qBAAqB,EAAE,CAAC;YAC7B,aAAa,CAAC,IAAI,CAAC,qBAAqB,CAAC,CAAC;YAC1C,IAAI,CAAC,qBAAqB,GAAG,IAAI,CAAC;QACtC,CAAC;QACD,IAAI,IAAI,CAAC,4BAA4B,EAAE,CAAC;YACpC,YAAY,CAAC,IAAI,CAAC,4BAA4B,CAAC,CAAC;YAChD,IAAI,CAAC,4BAA4B,GAAG,IAAI,CAAC;QAC7C,CAAC;QACD,IAAI,CAAC,qBAAqB,GAAG,IAAI,CAAC;IACtC,CAAC;IAEO,+BAA+B,CAAC,OAAO,GAAG,kBAAgB,CAAC,4BAA4B;QAC3F,IAAI,IAAI,CAAC,4BAA4B,EAAE,CAAC;YACpC,YAAY,CAAC,IAAI,CAAC,4BAA4B,CAAC,CAAC;QACpD,CAAC;QACD,IAAI,CAAC,4BAA4B,GAAG,UAAU,CAAC,GAAG,EAAE;YAChD,IAAI,CAAC,4BAA4B,GAAG,IAAI,CAAC;YACzC,IAAI,CAAC,2BAA2B,EAAE,CAAC;QACvC,CAAC,EAAE,OAAO,CAAC,CAAC;IAChB,CAAC;IAEO,kBAAkB;QACtB,IAAI,CAAC,IAAI,CAAC,oBAAoB,IAAI,CAAC,IAAI,CAAC,YAAY,EAAE,CAAC;YACnD,IAAI,CAAC,2BAA2B,EAAE,CAAC;YACnC,OAAO;QACX,CAAC;QAED,MAAM,GAAG,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC;QACvB,IAAI,GAAG,GAAG,IAAI,CAAC,wBAAwB,GAAG,kBAAgB,CAAC,gCAAgC,EAAE,CAAC;YAC1F,IAAI,CAAC,qBAAqB,GAAG,IAAI,CAAC;YAClC,OAAO;QACX,CAAC;QAED,IAAI,IAAI,CAAC,gBAAgB,EAAE,CAAC;YACxB,MAAM,KAAK,GAAG,IAAI,CAAC,gBAAgB,CAAC,KAAK,EAAE,CAAC;YAC5C,IAAI,KAAK,IAAI,kBAAgB,CAAC,gCAAgC,EAAE,CAAC;gBAC7D,IAAI,IAAI,CAAC,qBAAqB,KAAK,IAAI,EAAE,CAAC;oBACtC,IAAI,CAAC,qBAAqB,GAAG,GAAG,CAAC;gBACrC,CAAC;qBAAM,IAAI,GAAG,GAAG,IAAI,CAAC,qBAAqB,IAAI,kBAAgB,CAAC,yBAAyB,EAAE,CAAC;oBACxF,IAAI,CAAC,2BAA2B,EAAE,CAAC;gBACvC,CAAC;YACL,CAAC;iBAAM,CAAC;gBACJ,IAAI,CAAC,qBAAqB,GAAG,IAAI,CAAC;YACtC,CAAC;QACL,CAAC;IACL,CAAC;IAEO,kBAAkB,CAAC,MAAc,EAAE,IAAY,EAAE,QAAgB;QACrE,IAAI,CAAC,MAAM,IAAI,CAAC,IAAI,EAAE,CAAC;YACnB,OAAO;QACX,CAAC;QACD,IAAI,IAAI,CAAC,kBAAkB,CAAC,GAAG,CAAC,MAAM,CAAC,EAAE,CAAC;YACtC,OAAO;QACX,CAAC;QACD,IAAI,CAAC,kBAAkB,CAAC,GAAG,CAAC,MAAM,CAAC,CAAC;QAEpC,IAAI,CAAC,2BAA2B,EAAE,CAAC;QACnC,IAAI,CAAC,cAAc,GAAG,KAAK,CAAC;QAC5B,IAAI,CAAC,YAAY,GAAG,KAAK,CAAC;QAC1B,IAAI,CAAC,gBAAgB,CAAC,gBAAgB,CAAC,MAAM,EAAE,GAAG,EAAE;YAChD,IAAI,CAAC,oBAAoB,CAAC;gBACtB,IAAI,EAAE,iBAAiB;aAC1B,CAAC,CAAC;QACP,CAAC,CAAC,CAAC;QACH,MAAM,IAAI,GAA2B;YACjC,MAAM,EAAE,MAAM;YACd,QAAQ,EAAE,IAAI;YACd,aAAa,EAAE,QAAQ;SAC1B,CAAC;QACF,IAAI,CAAC,YAAY,CAAC,IAAI,CAAC,CAAC;IAC5B,CAAC;IAEO,oBAAoB,CAAC,IAAyC;QAClE,IAAI,IAAI,EAAE,IAAI,KAAK,eAAe,EAAE,CAAC;YACjC,MAAM,MAAM,GAAG,MAAM,CAAC,IAAI,CAAC,OAAO,IAAI,EAAE,CAAC,CAAC;YAC1C,MAAM,IAAI,GAAG,MAAM,CAAC,IAAI,CAAC,IAAI,IAAI,EAAE,CAAC,CAAC;YACrC,MAAM,IAAI,GAAG,MAAM,CAAC,IAAI,CAAC,SAAS,IAAI,IAAI,CAAC,CAAC;YAC5C,IAAI,CAAC,kBAAkB,CAAC,MAAM,EAAE,IAAI,EAAE,IAAI,CAAC,CAAC;QAChD,CAAC;IACL,CAAC;IAEO,uBAAuB,CAAC,WAAgD;QAC5E,IAAI,CAAC,kBAAkB,CAAC,KAAK,EAAE,CAAC;QAChC,IAAI,CAAC,2BAA2B,EAAE,CAAC;QACnC,MAAM,KAAK,GAAI,WAAW,EAAE,KAA6C,IAAI,WAAW,CAAC;QACzF,MAAM,OAAO,GAAG,OAAO,KAAK,EAAE,OAAO,KAAK,QAAQ,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,SAAS,CAAC;QAC/E,MAAM,QAAQ,GAAG,OAAO,KAAK,SAAS,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,kBAAgB,CAAC,wBAAwB,EAAE,OAAO,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,oBAAoB,CAAC;QAClI,IAAI,CAAC,oBAAoB,GAAG,QAAQ,CAAC;QAErC,IAAI,CAAC,SAAS,CAAC;YACX,WAAW,EAAE,OAAO,KAAK,EAAE,YAAY,KAAK,QAAQ,CAAC,CAAC,CAAC,KAAK,CAAC,YAAY,CAAC,CAAC,CAAC,CAAC;YAC7E,YAAY,EAAE,OAAO,KAAK,EAAE,aAAa,KAAK,QAAQ,CAAC,CAAC,CAAC,KAAK,CAAC,aAAa,CAAC,CAAC,CAAC,CAAC;YAChF,eAAe,EAAE,QAAQ;YACzB,GAAG,EAAE,KAAK;SACb,CAAC,CAAC;IACP,CAAC;IAEO,qBAAqB;QACzB,MAAM,UAAU,GAAG,IAAI,CAAC,aAAa,EAAE,CAAC,YAAY,CAAC,CAAC;QACtD,IAAI,UAAU,IAAI,OAAO,UAAU,KAAK,QAAQ,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,UAAU,CAAC,EAAE,CAAC;YAC7E,OAAQ,UAAsC,CAAC,MAAM,CAAC,KAAK,WAAW,CAAC;QAC3E,CAAC;QACD,OAAO,KAAK,CAAC;IACjB,CAAC;IAEO,wBAAwB,CAAC,GAAW;QACxC,IAAI,KAA8B,CAAC;QACnC,IAAI,CAAC;YACD,KAAK,GAAG,IAAI,CAAC,KAAK,CAAC,GAAG,CAA4B,CAAC;QACvD,CAAC;QAAC,MAAM,CAAC;YACL,OAAO;QACX,CAAC;QAED,MAAM,IAAI,GAAG,OAAO,KAAK,CAAC,IAAI,KAAK,QAAQ,CAAC,CAAC,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC;QAE9D,QAAQ,IAAI,EAAE,CAAC;YACX,KAAK,iBAAiB,CAAC,CAAC,CAAC;gBACrB,IAAI,CAAC,QAAQ,CAAC,WAAW,CAAC,CAAC;gBAC3B,MAAM;YACV,CAAC;YAED,KAAK,gCAAgC,CAAC,CAAC,CAAC;gBACpC,MAAM,KAAK,GAAG,OAAO,KAAK,CAAC,KAAK,KAAK,QAAQ,CAAC,CAAC,CAAC,KAAK,CAAC,KAAK,CAAC,CAAC,CAAC,OAAO,KAAK,CAAC,IAAI,KAAK,QAAQ,CAAC,CAAC,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC;gBAC/G,IAAI,CAAC,KAAK,EAAE,CAAC;oBACT,MAAM;gBACV,CAAC;gBACD,IAAI,IAAI,CAAC,oBAAoB,EAAE,CAAC;oBAC5B,IAAI,CAAC,2BAA2B,EAAE,CAAC;gBACvC,CAAC;gBACD,IAAI,CAAC,eAAe,IAAI,KAAK,CAAC;gBAC9B,IAAI,CAAC,cAAc,CAAC;oBAChB,IAAI,EAAE,MAAM;oBACZ,IAAI,EAAE,KAAK;oBACX,OAAO,EAAE,KAAK;oBACd,IAAI,EAAE,QAAQ;iBACjB,CAAC,CAAC;gBACH,MAAM;YACV,CAAC;YAED,KAAK,iCAAiC,CAAC,CAAC,CAAC;gBACrC,MAAM,KAAK,GAAG,OAAO,KAAK,CAAC,KAAK,KAAK,QAAQ,CAAC,CAAC,CAAC,KAAK,CAAC,KAAK,CAAC,CAAC,CAAC,OAAO,KAAK,CAAC,IAAI,KAAK,QAAQ,CAAC,CAAC,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC;gBAC/G,IAAI,CAAC,KAAK,EAAE,CAAC;oBACT,MAAM;gBACV,CAAC;gBACD,IAAI,IAAI,CAAC,eAAe,EAAE,CAAC;oBACvB,IAAI,CAAC,sBAAsB,EAAE,CAAC;gBAClC,CAAC;gBACD,IAAI,CAAC,cAAc,GAAG,IAAI,CAAC;gBAC3B,IAAI,CAAC,YAAY,GAAG,IAAI,CAAC;gBACzB,IAAI,IAAI,CAAC,YAAY,KAAK,UAAU,EAAE,CAAC;oBACnC,IAAI,CAAC,QAAQ,CAAC,UAAU,CAAC,CAAC;gBAC9B,CAAC;gBACD,IAAI,CAAC,oBAAoB,IAAI,KAAK,CAAC;gBACnC,IAAI,CAAC,cAAc,CAAC;oBAChB,IAAI,EAAE,WAAW;oBACjB,IAAI,EAAE,KAAK;oBACX,OAAO,EAAE,KAAK;oBACd,IAAI,EAAE,IAAI,CAAC,gBAAgB,CAAC,CAAC,CAAC,WAAW,CAAC,CAAC,CAAC,QAAQ;iBACvD,CAAC,CAAC;gBACH,IAAI,CAAC,wBAAwB,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC;gBAC3C,IAAI,CAAC,qBAAqB,GAAG,IAAI,CAAC;gBAClC,IAAI,CAAC,4BAA4B,EAAE,CAAC;gBACpC,IAAI,CAAC,+BAA+B,EAAE,CAAC;gBACvC,MAAM;YACV,CAAC;YAED,KAAK,gBAAgB,CAAC,CAAC,CAAC;gBACpB,MAAM,KAAK,GAAG,KAAK,CAAC,KAA4C,CAAC;gBACjE,IAAI,KAAK,EAAE,IAAI,KAAK,2BAA2B,EAAE,CAAC;oBAC9C,IAAI,CAAC,oBAAoB,CAAC,KAAK,CAAC,IAA2C,CAAC,CAAC;gBACjF,CAAC;qBAAM,IAAI,KAAK,EAAE,IAAI,KAAK,uCAAuC,EAAE,CAAC;oBACjE,MAAM,MAAM,GAAG,MAAM,CAAC,KAAK,CAAC,OAAO,IAAI,EAAE,CAAC,CAAC;oBAC3C,MAAM,IAAI,GAAG,MAAM,CAAC,KAAK,CAAC,IAAI,IAAI,EAAE,CAAC,CAAC;oBACtC,MAAM,IAAI,GAAG,MAAM,CAAC,KAAK,CAAC,SAAS,IAAI,IAAI,CAAC,CAAC;oBAC7C,IAAI,CAAC,kBAAkB,CAAC,MAAM,EAAE,IAAI,EAAE,IAAI,CAAC,CAAC;gBAChD,CAAC;qBAAM,IAAI,KAAK,EAAE,IAAI,KAAK,oBAAoB,IAAI,KAAK,EAAE,IAAI,KAAK,eAAe,EAAE,CAAC;oBACjF,IAAI,CAAC,uBAAuB,CAAE,KAAK,CAAC,QAAgD,IAAI,KAAK,CAAC,CAAC;gBACnG,CAAC;gBACD,MAAM;YACV,CAAC;YAED,KAAK,4BAA4B,CAAC,CAAC,CAAC;gBAChC,IAAI,CAAC,2BAA2B,EAAE,CAAC;gBACnC,IAAI,CAAC,sBAAsB,EAAE,CAAC;gBAC9B,IAAI,CAAC,cAAc,GAAG,KAAK,CAAC;gBAC5B,IAAI,CAAC,YAAY,GAAG,KAAK,CAAC;gBAC1B,8FAA8F;gBAC9F,yFAAyF;gBACzF,iFAAiF;gBACjF,+EAA+E;gBAC/E,IAAI,CAAC,IAAI,CAAC,qBAAqB,EAAE,EAAE,CAAC;oBAChC,MAAM,YAAY,GAAG,MAAM,CAAC,KAAK,CAAC,aAAa,IAAI,EAAE,CAAC,CAAC;oBACvD,IAAI,YAAY,EAAE,CAAC;wBACf,IAAI,CAAC,uBAAuB,CAAC,GAAG,CAAC,YAAY,CAAC,CAAC;wBAC/C,MAAM,IAAI,GAA2B;4BACjC,MAAM,EAAE,YAAY;4BACpB,QAAQ,EAAE,oBAAoB;4BAC9B,aAAa,EAAE,IAAI,CAAC,SAAS,CAAC,EAAE,aAAa,EAAE,YAAY,EAAE,CAAC;yBACjE,CAAC;wBACF,IAAI,CAAC,YAAY,CAAC,IAAI,CAAC,CAAC;oBAC5B,CAAC;gBACL,CAAC;gBACD,MAAM;YACV,CAAC;YAED,KAAK,uBAAuB,CAAC,CAAC,CAAC;gBAC3B,MAAM,KAAK,GAAG,KAAK,CAAC,KAA4C,CAAC;gBACjE,MAAM,OAAO,GAAG,OAAO,KAAK,EAAE,OAAO,KAAK,QAAQ,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,SAAS,CAAC;gBAC/E,MAAM,QAAQ,GAAG,OAAO,KAAK,SAAS,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,kBAAgB,CAAC,wBAAwB,EAAE,OAAO,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,oBAAoB,CAAC;gBAClI,IAAI,CAAC,oBAAoB,GAAG,QAAQ,CAAC;gBAErC,IAAI,CAAC,SAAS,CAAC;oBACX,WAAW,EAAE,OAAO,KAAK,EAAE,YAAY,KAAK,QAAQ,CAAC,CAAC,CAAC,KAAK,CAAC,YAAY,CAAC,CAAC,CAAC,CAAC;oBAC7E,YAAY,EAAE,OAAO,KAAK,EAAE,aAAa,KAAK,QAAQ,CAAC,CAAC,CAAC,KAAK,CAAC,aAAa,CAAC,CAAC,CAAC,CAAC;oBAChF,eAAe,EAAE,QAAQ;oBACzB,GAAG,EAAE,KAAK;iBACb,CAAC,CAAC;gBACH,MAAM;YACV,CAAC;YAED,KAAK,OAAO,CAAC,CAAC,CAAC;gBACX,MAAM,GAAG,GAAG,KAAK,CAAC,KAA4C,CAAC;gBAC/D,MAAM,OAAO,GAAG,OAAO,GAAG,EAAE,OAAO,KAAK,QAAQ,CAAC,CAAC,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,2BAA2B,CAAC;gBAC7F,MAAM,IAAI,GAAG,OAAO,GAAG,EAAE,IAAI,KAAK,QAAQ,CAAC,CAAC,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC,CAAC,SAAS,CAAC;gBAClE,MAAM,OAAO,GAAG,CAAC,IAAI,CAAC,QAAQ,EAAE,IAAI,IAAI,KAAK,iBAAiB,IAAI,IAAI,KAAK,cAAc,CAAC;gBAC1F,oEAAoE;gBACpE,IAAI,IAAI,CAAC,cAAc,IAAI,IAAI,CAAC,YAAY,EAAE,CAAC;oBAC3C,IAAI,CAAC,gBAAgB,EAAE,CAAC;oBACxB,IAAI,CAAC,oBAAoB,EAAE,CAAC;gBAChC,CAAC;gBACD,IAAI,CAAC,SAAS,CAAC,EAAE,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,IAAI,EAAE,KAAK,EAAE,OAAO,EAAE,CAAC,CAAC;gBACjE,IAAI,OAAO,EAAE,CAAC;oBACV,IAAI,CAAC,QAAQ,CAAC,OAAO,CAAC,CAAC;gBAC3B,CAAC;gBACD,MAAM;YACV,CAAC;YAED,KAAK,gBAAgB,CAAC,CAAC,CAAC;gBACpB,IAAI,CAAC,QAAQ,CAAC,QAAQ,CAAC,CAAC;gBACxB,KAAK,IAAI,CAAC,UAAU,EAAE,CAAC;gBACvB,MAAM;YACV,CAAC;YAED;gBACI,MAAM;QACd,CAAC;IACL,CAAC;IAEO,QAAQ;QACZ,OAAO,IAAI,CAAC,YAAY,KAAK,WAAW,IAAI,IAAI,CAAC,YAAY,KAAK,WAAW,IAAI,IAAI,CAAC,YAAY,KAAK,UAAU,CAAC;IACtH,CAAC;IAEO,oBAAoB,CAAC,OAAgC;QACzD,MAAM,SAAS,GAAG,OAAO,OAAO,CAAC,IAAI,KAAK,QAAQ,CAAC,CAAC,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,SAAS,CAAC;QAC9E,IAAI,IAAI,CAAC,WAAW,IAAI,IAAI,CAAC,WAAW,CAAC,UAAU,KAAK,MAAM,EAAE,CAAC;YAC7D,IAAI,CAAC;gBACD,IAAI,CAAC,WAAW,CAAC,IAAI,CAAC,IAAI,CAAC,SAAS,CAAC,OAAO,CAAC,CAAC,CAAC;YACnD,CAAC;YAAC,OAAO,GAAG,EAAE,CAAC;gBACX,OAAO,CAAC,IAAI,CAAC,uDAAuD,EAAE,GAAG,CAAC,CAAC;YAC/E,CAAC;QACL,CAAC;aAAM,IAAI,CAAC,IAAI,CAAC,WAAW,IAAI,IAAI,CAAC,WAAW,CAAC,UAAU,KAAK,YAAY,EAAE,CAAC;YAC3E,IAAI,IAAI,CAAC,aAAa,CAAC,MAAM,IAAI,kBAAgB,CAAC,uBAAuB,EAAE,CAAC;gBACxE,MAAM,OAAO,GAAG,IAAI,CAAC,aAAa,CAAC,KAAK,EAAE,CAAC;gBAC3C,MAAM,WAAW,GAAG,OAAO,OAAO,EAAE,IAAI,KAAK,QAAQ,CAAC,CAAC,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,SAAS,CAAC;gBACjF,OAAO,CAAC,IAAI,CAAC,0DAA0D,kBAAgB,CAAC,uBAAuB,4BAA4B,WAAW,GAAG,CAAC,CAAC;YAC/J,CAAC;YACD,IAAI,CAAC,aAAa,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC;QACrC,CAAC;aAAM,CAAC;YACJ,OAAO,CAAC,IAAI,CAAC,sCAAsC,SAAS,6BAA6B,IAAI,CAAC,WAAW,CAAC,UAAU,EAAE,CAAC,CAAC;QAC5H,CAAC;IACL,CAAC;IAEO,kBAAkB;QACtB,OAAO,IAAI,CAAC,aAAa,CAAC,MAAM,GAAG,CAAC,IAAI,IAAI,CAAC,WAAW,IAAI,IAAI,CAAC,WAAW,CAAC,UAAU,KAAK,MAAM,EAAE,CAAC;YACjG,MAAM,KAAK,GAAG,IAAI,CAAC,aAAa,CAAC,KAAK,EAAE,CAAC;YACzC,IAAI,KAAK,EAAE,CAAC;gBACR,IAAI,CAAC,oBAAoB,CAAC,KAAK,CAAC,CAAC;YACrC,CAAC;QACL,CAAC;IACL,CAAC;;AA3vBQ,gBAAgB;IAF5B,aAAa,CAAC,kBAAkB,EAAE,aAAa,CAAC;IAChD,aAAa,CAAC,kBAAkB,EAAE,oBAAoB,CAAC;GAC3C,gBAAgB,CA4vB5B;;AAED;;GAEG;AACH,MAAM,UAAU,oBAAoB;IAChC,iEAAiE;AACrE,CAAC"}
|
|
@@ -86,6 +86,11 @@ export interface RealtimeClientUsage {
|
|
|
86
86
|
InputTokens?: number;
|
|
87
87
|
/** Output tokens reported in this update (a delta for the completed response/turn). */
|
|
88
88
|
OutputTokens?: number;
|
|
89
|
+
/**
|
|
90
|
+
* Session duration in seconds reported by the provider or transport (e.g. OpenAI Live's
|
|
91
|
+
* 15 s WebRTC session pre-bill and cumulative voice duration snapshots).
|
|
92
|
+
*/
|
|
93
|
+
DurationSeconds?: number;
|
|
89
94
|
/** The raw provider usage payload, for hosts that want provider-specific detail. */
|
|
90
95
|
Raw?: unknown;
|
|
91
96
|
}
|
|
@@ -202,9 +207,9 @@ export declare abstract class BaseRealtimeClient {
|
|
|
202
207
|
private usageHandler?;
|
|
203
208
|
private remoteVideoHandler?;
|
|
204
209
|
/** Meter over the USER's microphone, when the driver attached one. */
|
|
205
|
-
|
|
210
|
+
protected inputAudioMeter: IRealtimeAudioMeter | null;
|
|
206
211
|
/** Meter over the AGENT's audio output, when the driver attached one. */
|
|
207
|
-
|
|
212
|
+
protected outputAudioMeter: IRealtimeAudioMeter | null;
|
|
208
213
|
/**
|
|
209
214
|
* The session's current audible activity, or `null` when this driver attached no
|
|
210
215
|
* meters at all (hosts then keep their turn-state-driven visuals). Sampled by hosts
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"baseRealtimeClient.d.ts","sourceRoot":"","sources":["../../src/generic/baseRealtimeClient.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,2BAA2B,EAAE,MAAM,oBAAoB,CAAC;AACjE,OAAO,EAAE,mBAAmB,EAA4B,MAAM,qBAAqB,CAAC;AAEpF;;;;;;;;;GASG;AACH,MAAM,WAAW,qBAAqB;IAClC,4EAA4E;IAC5E,UAAU,EAAE,MAAM,GAAG,IAAI,CAAC;IAC1B,+EAA+E;IAC/E,WAAW,EAAE,MAAM,GAAG,IAAI,CAAC;IAC3B,6FAA6F;IAC7F,SAAS,EAAE,MAAM,EAAE,GAAG,IAAI,CAAC;IAC3B,sGAAsG;IACtG,UAAU,EAAE,MAAM,EAAE,GAAG,IAAI,CAAC;CAC/B;AAED;;;;;;;GAOG;AACH,MAAM,WAAW,wBAAwB;IACrC,6CAA6C;IAC7C,IAAI,EAAE,MAAM,GAAG,WAAW,CAAC;IAC3B;;;OAGG;IACH,IAAI,EAAE,MAAM,CAAC;IACb,gGAAgG;IAChG,OAAO,EAAE,OAAO,CAAC;IACjB;;;OAGG;IACH,IAAI,EAAE,QAAQ,GAAG,WAAW,CAAC;IAC7B;;;;;;;OAOG;IACH,gBAAgB,CAAC,EAAE,OAAO,CAAC;CAC9B;AAED;;;GAGG;AACH,MAAM,WAAW,sBAAsB;IACnC,8FAA8F;IAC9F,MAAM,EAAE,MAAM,CAAC;IACf,8DAA8D;IAC9D,QAAQ,EAAE,MAAM,CAAC;IACjB,yFAAyF;IACzF,aAAa,EAAE,MAAM,CAAC;CACzB;AAED;;;;;;;;;;;;;;;GAeG;AACH,MAAM,WAAW,mBAAmB;IAChC,sFAAsF;IACtF,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB,uFAAuF;IACvF,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB,oFAAoF;IACpF,GAAG,CAAC,EAAE,OAAO,CAAC;CACjB;AAED;;;;;;;;;;;;GAYG;AACH,MAAM,MAAM,mBAAmB,GAAG,YAAY,GAAG,WAAW,GAAG,WAAW,GAAG,UAAU,GAAG,QAAQ,GAAG,OAAO,CAAC;AAE7G;;;GAGG;AACH,MAAM,WAAW,mBAAmB;IAChC,oCAAoC;IACpC,OAAO,EAAE,MAAM,CAAC;IAChB,6CAA6C;IAC7C,IAAI,CAAC,EAAE,MAAM,CAAC;IACd,gDAAgD;IAChD,KAAK,EAAE,OAAO,CAAC;CAClB;AAED;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GA6EG;AACH,8BAAsB,kBAAkB;IAEpC,OAAO,CAAC,iBAAiB,CAAC,CAAiD;IAC3E,OAAO,CAAC,eAAe,CAAC,CAAyC;IACjE,OAAO,CAAC,kBAAkB,CAAC,CAAuC;IAClE,OAAO,CAAC,YAAY,CAAC,CAAuC;IAC5D,OAAO,CAAC,mBAAmB,CAAC,CAAa;IACzC,OAAO,CAAC,YAAY,CAAC,CAAuC;IAC5D,OAAO,CAAC,kBAAkB,CAAC,CAAgC;IAG3D,sEAAsE;IACtE,
|
|
1
|
+
{"version":3,"file":"baseRealtimeClient.d.ts","sourceRoot":"","sources":["../../src/generic/baseRealtimeClient.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,2BAA2B,EAAE,MAAM,oBAAoB,CAAC;AACjE,OAAO,EAAE,mBAAmB,EAA4B,MAAM,qBAAqB,CAAC;AAEpF;;;;;;;;;GASG;AACH,MAAM,WAAW,qBAAqB;IAClC,4EAA4E;IAC5E,UAAU,EAAE,MAAM,GAAG,IAAI,CAAC;IAC1B,+EAA+E;IAC/E,WAAW,EAAE,MAAM,GAAG,IAAI,CAAC;IAC3B,6FAA6F;IAC7F,SAAS,EAAE,MAAM,EAAE,GAAG,IAAI,CAAC;IAC3B,sGAAsG;IACtG,UAAU,EAAE,MAAM,EAAE,GAAG,IAAI,CAAC;CAC/B;AAED;;;;;;;GAOG;AACH,MAAM,WAAW,wBAAwB;IACrC,6CAA6C;IAC7C,IAAI,EAAE,MAAM,GAAG,WAAW,CAAC;IAC3B;;;OAGG;IACH,IAAI,EAAE,MAAM,CAAC;IACb,gGAAgG;IAChG,OAAO,EAAE,OAAO,CAAC;IACjB;;;OAGG;IACH,IAAI,EAAE,QAAQ,GAAG,WAAW,CAAC;IAC7B;;;;;;;OAOG;IACH,gBAAgB,CAAC,EAAE,OAAO,CAAC;CAC9B;AAED;;;GAGG;AACH,MAAM,WAAW,sBAAsB;IACnC,8FAA8F;IAC9F,MAAM,EAAE,MAAM,CAAC;IACf,8DAA8D;IAC9D,QAAQ,EAAE,MAAM,CAAC;IACjB,yFAAyF;IACzF,aAAa,EAAE,MAAM,CAAC;CACzB;AAED;;;;;;;;;;;;;;;GAeG;AACH,MAAM,WAAW,mBAAmB;IAChC,sFAAsF;IACtF,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB,uFAAuF;IACvF,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB;;;OAGG;IACH,eAAe,CAAC,EAAE,MAAM,CAAC;IACzB,oFAAoF;IACpF,GAAG,CAAC,EAAE,OAAO,CAAC;CACjB;AAED;;;;;;;;;;;;GAYG;AACH,MAAM,MAAM,mBAAmB,GAAG,YAAY,GAAG,WAAW,GAAG,WAAW,GAAG,UAAU,GAAG,QAAQ,GAAG,OAAO,CAAC;AAE7G;;;GAGG;AACH,MAAM,WAAW,mBAAmB;IAChC,oCAAoC;IACpC,OAAO,EAAE,MAAM,CAAC;IAChB,6CAA6C;IAC7C,IAAI,CAAC,EAAE,MAAM,CAAC;IACd,gDAAgD;IAChD,KAAK,EAAE,OAAO,CAAC;CAClB;AAED;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GA6EG;AACH,8BAAsB,kBAAkB;IAEpC,OAAO,CAAC,iBAAiB,CAAC,CAAiD;IAC3E,OAAO,CAAC,eAAe,CAAC,CAAyC;IACjE,OAAO,CAAC,kBAAkB,CAAC,CAAuC;IAClE,OAAO,CAAC,YAAY,CAAC,CAAuC;IAC5D,OAAO,CAAC,mBAAmB,CAAC,CAAa;IACzC,OAAO,CAAC,YAAY,CAAC,CAAuC;IAC5D,OAAO,CAAC,kBAAkB,CAAC,CAAgC;IAG3D,sEAAsE;IACtE,SAAS,CAAC,eAAe,EAAE,mBAAmB,GAAG,IAAI,CAAQ;IAC7D,yEAAyE;IACzE,SAAS,CAAC,gBAAgB,EAAE,mBAAmB,GAAG,IAAI,CAAQ;IAE9D;;;;;OAKG;IACI,gBAAgB,IAAI,qBAAqB,GAAG,IAAI;IAYvD,8FAA8F;IAC9F,SAAS,CAAC,qBAAqB,CAAC,KAAK,EAAE,mBAAmB,GAAG,IAAI,GAAG,IAAI;IAKxE,2FAA2F;IAC3F,SAAS,CAAC,sBAAsB,CAAC,KAAK,EAAE,mBAAmB,GAAG,IAAI,GAAG,IAAI;IAKzE,wFAAwF;IACxF,SAAS,CAAC,gBAAgB,IAAI,IAAI;IAKlC;;;;;;;;;;;;;;;OAeG;aACa,OAAO,CAAC,MAAM,EAAE,2BAA2B,EAAE,SAAS,EAAE,WAAW,EAAE,YAAY,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,IAAI,CAAC;IAE/H;;;;;;;;;;OAUG;IACI,oBAAoB,CAAC,IAAI,WAAW,GAAG,IAAI;IAElD;;;;;;OAMG;IACI,mBAAmB,CAAC,CAAC,OAAO,EAAE,CAAC,MAAM,EAAE,WAAW,KAAK,IAAI,GAAG,IAAI;IAEzE;;;;;;;;;;;;;;OAcG;aACa,QAAQ,CAAC,IAAI,EAAE,MAAM,GAAG,IAAI;IAE5C;;;;;;;;;OASG;aACa,oBAAoB,IAAI,IAAI;IAE5C;;;;;OAKG;aACa,eAAe,CAAC,IAAI,EAAE,MAAM,GAAG,IAAI;IAEnD;;;;;;;;;;;;;;;OAeG;aACa,mBAAmB,CAAC,YAAY,EAAE,MAAM,GAAG,IAAI;IAE/D;;;;;;;;OAQG;aACa,cAAc,CAAC,MAAM,EAAE,MAAM,EAAE,UAAU,EAAE,MAAM,GAAG,IAAI;IAExE;;;;;OAKG;aACa,QAAQ,CAAC,KAAK,EAAE,OAAO,GAAG,IAAI;IAE9C;;;;OAIG;aACa,UAAU,IAAI,OAAO,CAAC,IAAI,CAAC;IAE3C;;;;OAIG;IACH,aAAoB,MAAM,IAAI,OAAO,CAAC;IAEtC;;;;;OAKG;IACH,aAAoB,cAAc,IAAI,OAAO,CAAC;IAI9C;;;;OAIG;IACI,YAAY,CAAC,OAAO,EAAE,CAAC,UAAU,EAAE,wBAAwB,KAAK,IAAI,GAAG,IAAI;IAIlF;;;OAGG;IACI,UAAU,CAAC,OAAO,EAAE,CAAC,IAAI,EAAE,sBAAsB,KAAK,IAAI,GAAG,IAAI;IAIxE;;;OAGG;IACI,aAAa,CAAC,OAAO,EAAE,CAAC,KAAK,EAAE,mBAAmB,KAAK,IAAI,GAAG,IAAI;IAIzE;;;OAGG;IACI,OAAO,CAAC,OAAO,EAAE,CAAC,KAAK,EAAE,mBAAmB,KAAK,IAAI,GAAG,IAAI;IAInE;;;;;;;;;;;;;OAaG;IACI,cAAc,CAAC,OAAO,EAAE,MAAM,IAAI,GAAG,IAAI;IAIhD;;;;;;;;;;;OAWG;IACI,OAAO,CAAC,OAAO,EAAE,CAAC,KAAK,EAAE,mBAAmB,KAAK,IAAI,GAAG,IAAI;IAInE;;;;;;;;;;OAUG;IACI,aAAa,CAAC,OAAO,EAAE,CAAC,MAAM,EAAE,WAAW,KAAK,IAAI,GAAG,IAAI;IAMlE,mEAAmE;IACnE,SAAS,CAAC,cAAc,CAAC,UAAU,EAAE,wBAAwB,GAAG,IAAI;IAIpE,oEAAoE;IACpE,SAAS,CAAC,YAAY,CAAC,IAAI,EAAE,sBAAsB,GAAG,IAAI;IAI1D,+DAA+D;IAC/D,SAAS,CAAC,eAAe,CAAC,KAAK,EAAE,mBAAmB,GAAG,IAAI;IAI3D,yDAAyD;IACzD,SAAS,CAAC,SAAS,CAAC,KAAK,EAAE,mBAAmB,GAAG,IAAI;IAIrD,6EAA6E;IAC7E,SAAS,CAAC,gBAAgB,IAAI,IAAI;IAIlC,4GAA4G;IAC5G,SAAS,CAAC,SAAS,CAAC,KAAK,EAAE,mBAAmB,GAAG,IAAI;IAIrD,mGAAmG;IACnG,SAAS,CAAC,eAAe,CAAC,MAAM,EAAE,WAAW,GAAG,IAAI;CAGvD"}
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"baseRealtimeClient.js","sourceRoot":"","sources":["../../src/generic/baseRealtimeClient.ts"],"names":[],"mappings":"AACA,OAAO,EAAuB,wBAAwB,EAAE,MAAM,qBAAqB,CAAC;
|
|
1
|
+
{"version":3,"file":"baseRealtimeClient.js","sourceRoot":"","sources":["../../src/generic/baseRealtimeClient.ts"],"names":[],"mappings":"AACA,OAAO,EAAuB,wBAAwB,EAAE,MAAM,qBAAqB,CAAC;AAgIpF;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GA6EG;AACH,MAAM,OAAgB,kBAAkB;IAAxC;QAUI,+EAA+E;QAC/E,sEAAsE;QAC5D,oBAAe,GAA+B,IAAI,CAAC;QAC7D,yEAAyE;QAC/D,qBAAgB,GAA+B,IAAI,CAAC;IAsSlE,CAAC;IApSG;;;;;OAKG;IACI,gBAAgB;QACnB,IAAI,CAAC,IAAI,CAAC,eAAe,IAAI,CAAC,IAAI,CAAC,gBAAgB,EAAE,CAAC;YAClD,OAAO,IAAI,CAAC;QAChB,CAAC;QACD,OAAO;YACH,UAAU,EAAE,IAAI,CAAC,eAAe,CAAC,CAAC,CAAC,IAAI,CAAC,eAAe,CAAC,KAAK,EAAE,CAAC,CAAC,CAAC,IAAI;YACtE,WAAW,EAAE,IAAI,CAAC,gBAAgB,CAAC,CAAC,CAAC,IAAI,CAAC,gBAAgB,CAAC,KAAK,EAAE,CAAC,CAAC,CAAC,IAAI;YACzE,SAAS,EAAE,IAAI,CAAC,eAAe,CAAC,CAAC,CAAC,IAAI,CAAC,eAAe,CAAC,IAAI,CAAC,wBAAwB,CAAC,CAAC,CAAC,CAAC,IAAI;YAC5F,UAAU,EAAE,IAAI,CAAC,gBAAgB,CAAC,CAAC,CAAC,IAAI,CAAC,gBAAgB,CAAC,IAAI,CAAC,wBAAwB,CAAC,CAAC,CAAC,CAAC,IAAI;SAClG,CAAC;IACN,CAAC;IAED,8FAA8F;IACpF,qBAAqB,CAAC,KAAiC;QAC7D,IAAI,CAAC,eAAe,EAAE,KAAK,EAAE,CAAC;QAC9B,IAAI,CAAC,eAAe,GAAG,KAAK,CAAC;IACjC,CAAC;IAED,2FAA2F;IACjF,sBAAsB,CAAC,KAAiC;QAC9D,IAAI,CAAC,gBAAgB,EAAE,KAAK,EAAE,CAAC;QAC/B,IAAI,CAAC,gBAAgB,GAAG,KAAK,CAAC;IAClC,CAAC;IAED,wFAAwF;IAC9E,gBAAgB;QACtB,IAAI,CAAC,qBAAqB,CAAC,IAAI,CAAC,CAAC;QACjC,IAAI,CAAC,sBAAsB,CAAC,IAAI,CAAC,CAAC;IACtC,CAAC;IA0ID,6EAA6E;IAE7E;;;;OAIG;IACI,YAAY,CAAC,OAAuD;QACvE,IAAI,CAAC,iBAAiB,GAAG,OAAO,CAAC;IACrC,CAAC;IAED;;;OAGG;IACI,UAAU,CAAC,OAA+C;QAC7D,IAAI,CAAC,eAAe,GAAG,OAAO,CAAC;IACnC,CAAC;IAED;;;OAGG;IACI,aAAa,CAAC,OAA6C;QAC9D,IAAI,CAAC,kBAAkB,GAAG,OAAO,CAAC;IACtC,CAAC;IAED;;;OAGG;IACI,OAAO,CAAC,OAA6C;QACxD,IAAI,CAAC,YAAY,GAAG,OAAO,CAAC;IAChC,CAAC;IAED;;;;;;;;;;;;;OAaG;IACI,cAAc,CAAC,OAAmB;QACrC,IAAI,CAAC,mBAAmB,GAAG,OAAO,CAAC;IACvC,CAAC;IAED;;;;;;;;;;;OAWG;IACI,OAAO,CAAC,OAA6C;QACxD,IAAI,CAAC,YAAY,GAAG,OAAO,CAAC;IAChC,CAAC;IAED;;;;;;;;;;OAUG;IACI,aAAa,CAAC,OAAsC;QACvD,IAAI,CAAC,kBAAkB,GAAG,OAAO,CAAC;IACtC,CAAC;IAED,6EAA6E;IAE7E,mEAAmE;IACzD,cAAc,CAAC,UAAoC;QACzD,IAAI,CAAC,iBAAiB,EAAE,CAAC,UAAU,CAAC,CAAC;IACzC,CAAC;IAED,oEAAoE;IAC1D,YAAY,CAAC,IAA4B;QAC/C,IAAI,CAAC,eAAe,EAAE,CAAC,IAAI,CAAC,CAAC;IACjC,CAAC;IAED,+DAA+D;IACrD,eAAe,CAAC,KAA0B;QAChD,IAAI,CAAC,kBAAkB,EAAE,CAAC,KAAK,CAAC,CAAC;IACrC,CAAC;IAED,yDAAyD;IAC/C,SAAS,CAAC,KAA0B;QAC1C,IAAI,CAAC,YAAY,EAAE,CAAC,KAAK,CAAC,CAAC;IAC/B,CAAC;IAED,6EAA6E;IACnE,gBAAgB;QACtB,IAAI,CAAC,mBAAmB,EAAE,EAAE,CAAC;IACjC,CAAC;IAED,4GAA4G;IAClG,SAAS,CAAC,KAA0B;QAC1C,IAAI,CAAC,YAAY,EAAE,CAAC,KAAK,CAAC,CAAC;IAC/B,CAAC;IAED,mGAAmG;IACzF,eAAe,CAAC,MAAmB;QACzC,IAAI,CAAC,kBAAkB,EAAE,CAAC,MAAM,CAAC,CAAC;IACtC,CAAC;CACJ"}
|
package/dist/index.d.ts
CHANGED
|
@@ -10,4 +10,5 @@ export * from './drivers/elevenLabsRealtimeClient.js';
|
|
|
10
10
|
export * from './drivers/assemblyAIRealtimeClient.js';
|
|
11
11
|
export * from './drivers/xaiRealtimeClient.js';
|
|
12
12
|
export * from './drivers/huggingFaceRealtimeClient.js';
|
|
13
|
+
export * from './drivers/openAILiveClient.js';
|
|
13
14
|
//# sourceMappingURL=index.d.ts.map
|
package/dist/index.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,cAAc,8BAA8B,CAAC;AAC7C,cAAc,gCAAgC,CAAC;AAC/C,cAAc,kBAAkB,CAAC;AACjC,cAAc,oBAAoB,CAAC;AACnC,cAAc,qBAAqB,CAAC;AACpC,cAAc,oBAAoB,CAAC;AACnC,cAAc,gCAAgC,CAAC;AAC/C,cAAc,gCAAgC,CAAC;AAC/C,cAAc,oCAAoC,CAAC;AACnD,cAAc,oCAAoC,CAAC;AACnD,cAAc,6BAA6B,CAAC;AAC5C,cAAc,qCAAqC,CAAC"}
|
|
1
|
+
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,cAAc,8BAA8B,CAAC;AAC7C,cAAc,gCAAgC,CAAC;AAC/C,cAAc,kBAAkB,CAAC;AACjC,cAAc,oBAAoB,CAAC;AACnC,cAAc,qBAAqB,CAAC;AACpC,cAAc,oBAAoB,CAAC;AACnC,cAAc,gCAAgC,CAAC;AAC/C,cAAc,gCAAgC,CAAC;AAC/C,cAAc,oCAAoC,CAAC;AACnD,cAAc,oCAAoC,CAAC;AACnD,cAAc,6BAA6B,CAAC;AAC5C,cAAc,qCAAqC,CAAC;AACpD,cAAc,4BAA4B,CAAC"}
|
package/dist/index.js
CHANGED
|
@@ -10,4 +10,5 @@ export * from './drivers/elevenLabsRealtimeClient.js';
|
|
|
10
10
|
export * from './drivers/assemblyAIRealtimeClient.js';
|
|
11
11
|
export * from './drivers/xaiRealtimeClient.js';
|
|
12
12
|
export * from './drivers/huggingFaceRealtimeClient.js';
|
|
13
|
+
export * from './drivers/openAILiveClient.js';
|
|
13
14
|
//# sourceMappingURL=index.js.map
|
package/dist/index.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,cAAc,8BAA8B,CAAC;AAC7C,cAAc,gCAAgC,CAAC;AAC/C,cAAc,kBAAkB,CAAC;AACjC,cAAc,oBAAoB,CAAC;AACnC,cAAc,qBAAqB,CAAC;AACpC,cAAc,oBAAoB,CAAC;AACnC,cAAc,gCAAgC,CAAC;AAC/C,cAAc,gCAAgC,CAAC;AAC/C,cAAc,oCAAoC,CAAC;AACnD,cAAc,oCAAoC,CAAC;AACnD,cAAc,6BAA6B,CAAC;AAC5C,cAAc,qCAAqC,CAAC"}
|
|
1
|
+
{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,cAAc,8BAA8B,CAAC;AAC7C,cAAc,gCAAgC,CAAC;AAC/C,cAAc,kBAAkB,CAAC;AACjC,cAAc,oBAAoB,CAAC;AACnC,cAAc,qBAAqB,CAAC;AACpC,cAAc,oBAAoB,CAAC;AACnC,cAAc,gCAAgC,CAAC;AAC/C,cAAc,gCAAgC,CAAC;AAC/C,cAAc,oCAAoC,CAAC;AACnD,cAAc,oCAAoC,CAAC;AACnD,cAAc,6BAA6B,CAAC;AAC5C,cAAc,qCAAqC,CAAC;AACpD,cAAc,4BAA4B,CAAC"}
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@memberjunction/ai-realtime-client",
|
|
3
3
|
"type": "module",
|
|
4
|
-
"version": "6.1.0
|
|
4
|
+
"version": "6.1.0",
|
|
5
5
|
"description": "MemberJunction: Framework-agnostic browser client abstraction for provider-direct realtime (voice) sessions - BaseRealtimeClient + provider drivers",
|
|
6
6
|
"main": "dist/index.js",
|
|
7
7
|
"types": "dist/index.d.ts",
|
|
@@ -11,8 +11,8 @@
|
|
|
11
11
|
"author": "MemberJunction.com",
|
|
12
12
|
"license": "BUSL-1.1",
|
|
13
13
|
"dependencies": {
|
|
14
|
-
"@memberjunction/global": "6.1.0
|
|
15
|
-
"@memberjunction/ai": "6.1.0
|
|
14
|
+
"@memberjunction/global": "6.1.0",
|
|
15
|
+
"@memberjunction/ai": "6.1.0",
|
|
16
16
|
"@google/genai": "^1.40.0"
|
|
17
17
|
},
|
|
18
18
|
"devDependencies": {
|