@breeze.blue/sdk 0.6.2 → 0.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +16 -0
- package/README.md +46 -5
- package/dist/client.d.ts +23 -1
- package/dist/client.js +169 -11
- package/dist/errors.d.ts +5 -4
- package/dist/errors.js +5 -4
- package/dist/types.d.ts +5 -6
- package/dist/version.d.ts +1 -1
- package/dist/version.js +1 -1
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,22 @@
|
|
|
2
2
|
|
|
3
3
|
## Unreleased
|
|
4
4
|
|
|
5
|
+
## 0.7.0
|
|
6
|
+
|
|
7
|
+
- Realtime TTS connections can update synthesis instructions between turns
|
|
8
|
+
with `updateInstructions(...)`. The matching `session.updated` event remains
|
|
9
|
+
on the existing ordered stream, and managed connections carry confirmed
|
|
10
|
+
instructions across physical-session rotation and idle reconnects.
|
|
11
|
+
`BreezeBlueRealtimeError.recoverable` means the same connection remains
|
|
12
|
+
usable: most recoverable turn-command errors cancel the active turn, while
|
|
13
|
+
an active-turn `session.update` rejection leaves that turn running and
|
|
14
|
+
callers continue consuming it on the existing connection.
|
|
15
|
+
|
|
16
|
+
## 0.6.3
|
|
17
|
+
|
|
18
|
+
- Simplified model listings to Breeze-native fields: model ID, name, supported
|
|
19
|
+
languages, and description.
|
|
20
|
+
|
|
5
21
|
## 0.6.2
|
|
6
22
|
|
|
7
23
|
- Voice search now uses the Voice Library's hybrid semantic and text ranking
|
package/README.md
CHANGED
|
@@ -152,7 +152,14 @@ a `BreezeBlueRealtimeError`. The `audio()` helper also throws
|
|
|
152
152
|
`BreezeBlueRealtimeError` for server `error` events and abnormal WebSocket
|
|
153
153
|
closes instead of silently ending.
|
|
154
154
|
|
|
155
|
-
|
|
155
|
+
When `BreezeBlueRealtimeError.recoverable` is `true`, the same connection
|
|
156
|
+
remains usable. Most recoverable turn-command errors cancel the active turn.
|
|
157
|
+
An active-turn `session.update` rejection is the exception: the turn keeps
|
|
158
|
+
running. If `connection.audio()` throws that error, start a fresh `audio()`
|
|
159
|
+
iterator immediately on the same connection and continue collecting the
|
|
160
|
+
current turn; do not start a replacement turn.
|
|
161
|
+
|
|
162
|
+
Events are a typed discriminated union (`session.ready`, `session.updated`, `turn.started`,
|
|
156
163
|
`audio.started`, `turn.done`, `turn.cancelled`, `usage.committed`,
|
|
157
164
|
`session.expiring`, `session.closed`, `error`, `pong`), with camelCase fields
|
|
158
165
|
such as `turnId`, `historyItemId`, `expiresAt`, and `ttfaMs`. Treat the union as
|
|
@@ -160,6 +167,32 @@ non-exhaustive: the server may add event types, and the SDK delivers unknown
|
|
|
160
167
|
JSON events unchanged — ignore event types you do not recognize instead of
|
|
161
168
|
switching exhaustively.
|
|
162
169
|
|
|
170
|
+
Between turns, update the synthesis instructions without replacing the
|
|
171
|
+
WebSocket. `updateInstructions(...)` only sends the request; the matching
|
|
172
|
+
`session.updated` acknowledgement remains on the same ordered iterator. Wait
|
|
173
|
+
for it before starting the next turn:
|
|
174
|
+
|
|
175
|
+
```ts
|
|
176
|
+
connection.updateInstructions("Speak faster and with more energy.");
|
|
177
|
+
for await (const message of connection) {
|
|
178
|
+
if (message.type === "session.updated") {
|
|
179
|
+
if (message.instructions !== "Speak faster and with more energy.") {
|
|
180
|
+
throw new Error("Unexpected instructions acknowledgement");
|
|
181
|
+
}
|
|
182
|
+
break;
|
|
183
|
+
}
|
|
184
|
+
}
|
|
185
|
+
|
|
186
|
+
connection.startTurn("turn_2");
|
|
187
|
+
```
|
|
188
|
+
|
|
189
|
+
Instructions must be a non-empty string of at most 1,000 characters. Only one
|
|
190
|
+
update may await acknowledgement, and updates are accepted only while no turn
|
|
191
|
+
is active. An update attempted during a turn is rejected without interrupting
|
|
192
|
+
that turn. Keep one stream consumer: if a long-lived consumer owns the
|
|
193
|
+
iterator, have it signal your turn producer when `session.updated` arrives
|
|
194
|
+
instead of starting a second iterator.
|
|
195
|
+
|
|
163
196
|
Send a keepalive ping inside the session's `inactivityTimeoutSeconds` window;
|
|
164
197
|
the server answers with a `pong` event. Keep it running during active turns too:
|
|
165
198
|
a long TTFA or upstream stall with no audio does not pause the idle deadline.
|
|
@@ -291,10 +324,18 @@ Honor the factory `signal`: the SDK aborts it when the logical connection closes
|
|
|
291
324
|
or the physical-epoch timeout expires. A custom callback that ignores the
|
|
292
325
|
signal must still enforce its own bounded request timeout.
|
|
293
326
|
|
|
294
|
-
|
|
295
|
-
|
|
296
|
-
|
|
297
|
-
|
|
327
|
+
After a dynamic instruction update receives `session.updated`, the managed SDK
|
|
328
|
+
carries the confirmed value to later physical epochs. SDK-created replacements
|
|
329
|
+
include it in their session request; factory-created replacements receive an
|
|
330
|
+
automatic `session.update` before their next `session.ready` is exposed to the
|
|
331
|
+
logical iterator.
|
|
332
|
+
|
|
333
|
+
`modelId`, `languageCode`, `voiceSettings`, `inactivityTimeoutSeconds`, and
|
|
334
|
+
`enableLogging` are fixed when the session is created. The initial
|
|
335
|
+
`instructions` value is also supplied at creation, but it can later be
|
|
336
|
+
replaced between turns with `updateInstructions(...)`. `connect` ignores
|
|
337
|
+
initial session options when `clientSecret` or `websocketUrl` is provided and
|
|
338
|
+
logs a warning.
|
|
298
339
|
|
|
299
340
|
If a realtime WebSocket is interrupted by a network change, service deployment,
|
|
300
341
|
or upstream realtime worker restart, `connectManaged(...)` handles bounded
|
package/dist/client.d.ts
CHANGED
|
@@ -48,6 +48,7 @@ export declare class RealtimeTextToSpeechConnection implements AsyncIterable<Rea
|
|
|
48
48
|
private closed;
|
|
49
49
|
private queuedBytes;
|
|
50
50
|
private failure;
|
|
51
|
+
private pendingInstructions;
|
|
51
52
|
private closeEventGraceTimer;
|
|
52
53
|
private constructor();
|
|
53
54
|
static open(socket: BreezeBlueWebSocket, options?: {
|
|
@@ -59,6 +60,14 @@ export declare class RealtimeTextToSpeechConnection implements AsyncIterable<Rea
|
|
|
59
60
|
flush(): void;
|
|
60
61
|
endTurn(): void;
|
|
61
62
|
cancelTurn(): void;
|
|
63
|
+
/**
|
|
64
|
+
* Request new synthesis instructions for subsequent turns.
|
|
65
|
+
*
|
|
66
|
+
* The matching `session.updated` acknowledgement stays on this
|
|
67
|
+
* connection's existing message stream. Keep the single consumer running
|
|
68
|
+
* and wait for that event before starting the next turn.
|
|
69
|
+
*/
|
|
70
|
+
updateInstructions(instructions: string): void;
|
|
62
71
|
/** Send a keepalive ping. The server answers with a `pong` event. */
|
|
63
72
|
ping(): void;
|
|
64
73
|
close(): void;
|
|
@@ -73,6 +82,7 @@ export declare class RealtimeTextToSpeechConnection implements AsyncIterable<Rea
|
|
|
73
82
|
private fail;
|
|
74
83
|
private finish;
|
|
75
84
|
private clearCloseEventGraceTimer;
|
|
85
|
+
private handleInstructionsEvent;
|
|
76
86
|
}
|
|
77
87
|
type RealtimeConnectionFactory = (signal: AbortSignal) => Promise<RealtimeTextToSpeechConnection>;
|
|
78
88
|
/**
|
|
@@ -88,6 +98,7 @@ type RealtimeConnectionFactory = (signal: AbortSignal) => Promise<RealtimeTextTo
|
|
|
88
98
|
*/
|
|
89
99
|
export declare class ManagedRealtimeTextToSpeechConnection implements AsyncIterable<RealtimeTextToSpeechMessage> {
|
|
90
100
|
private readonly createConnection;
|
|
101
|
+
private readonly onInstructionsConfirmed;
|
|
91
102
|
private readonly queue;
|
|
92
103
|
private readonly drainingEpochs;
|
|
93
104
|
private readonly openAttempts;
|
|
@@ -99,6 +110,7 @@ export declare class ManagedRealtimeTextToSpeechConnection implements AsyncItera
|
|
|
99
110
|
private readonly maxPhysicalSessionMs;
|
|
100
111
|
private readonly maxReconnectAttempts;
|
|
101
112
|
private readonly reconnectBaseDelayMs;
|
|
113
|
+
private readonly reapplyInstructionsOnOpen;
|
|
102
114
|
private current;
|
|
103
115
|
private heartbeatTimer;
|
|
104
116
|
private heartbeatAckTimer;
|
|
@@ -109,11 +121,13 @@ export declare class ManagedRealtimeTextToSpeechConnection implements AsyncItera
|
|
|
109
121
|
private epochSequence;
|
|
110
122
|
private queuedBytes;
|
|
111
123
|
private activeTurn;
|
|
124
|
+
private instructions;
|
|
125
|
+
private pendingInstructions;
|
|
112
126
|
private rotationPending;
|
|
113
127
|
private closed;
|
|
114
128
|
private failure;
|
|
115
129
|
private constructor();
|
|
116
|
-
static open(createConnection: RealtimeConnectionFactory, options?: RealtimeTextToSpeechManagedConnectOptions): Promise<ManagedRealtimeTextToSpeechConnection>;
|
|
130
|
+
static open(createConnection: RealtimeConnectionFactory, options?: RealtimeTextToSpeechManagedConnectOptions, onInstructionsConfirmed?: (instructions: string) => void): Promise<ManagedRealtimeTextToSpeechConnection>;
|
|
117
131
|
startTurn(turnId?: string): void;
|
|
118
132
|
/**
|
|
119
133
|
* Start a turn on the current physical epoch, waiting only when an idle
|
|
@@ -129,6 +143,14 @@ export declare class ManagedRealtimeTextToSpeechConnection implements AsyncItera
|
|
|
129
143
|
flush(): void;
|
|
130
144
|
endTurn(): void;
|
|
131
145
|
cancelTurn(): void;
|
|
146
|
+
/**
|
|
147
|
+
* Update the synthesis instructions used by subsequent turns.
|
|
148
|
+
*
|
|
149
|
+
* The matching `session.updated` acknowledgement stays on this logical
|
|
150
|
+
* connection's ordered event stream. Confirmed instructions are carried to
|
|
151
|
+
* later physical WebSocket epochs.
|
|
152
|
+
*/
|
|
153
|
+
updateInstructions(instructions: string): void;
|
|
132
154
|
/** Send an immediate ping in addition to the managed keepalive schedule. */
|
|
133
155
|
ping(): void;
|
|
134
156
|
close(): void;
|
package/dist/client.js
CHANGED
|
@@ -14,6 +14,7 @@ const DEFAULT_REALTIME_RECONNECT_BASE_DELAY_MS = 250;
|
|
|
14
14
|
const MAX_REALTIME_RECONNECT_DELAY_MS = 4_000;
|
|
15
15
|
const REALTIME_SETTLEMENT_DRAIN_TIMEOUT_MS = 5_000;
|
|
16
16
|
const REALTIME_CLOSE_EVENT_GRACE_MS = 50;
|
|
17
|
+
const REALTIME_MAX_INSTRUCTIONS_CHARACTERS = 1_000;
|
|
17
18
|
const WEBSOCKET_OPEN = 1;
|
|
18
19
|
export class BreezeBlueClient {
|
|
19
20
|
apiKey;
|
|
@@ -131,7 +132,8 @@ class RealtimeTextToSpeechResource {
|
|
|
131
132
|
if ((options.clientSecret !== undefined || options.websocketUrl !== undefined) && hasRealtimeSessionParams(options)) {
|
|
132
133
|
console.warn("@breeze.blue/sdk: realtime.connect ignores modelId, languageCode, instructions, voiceSettings, " +
|
|
133
134
|
"inactivityTimeoutSeconds, and enableLogging when clientSecret or websocketUrl is provided. " +
|
|
134
|
-
"Configure these when creating the realtime session instead."
|
|
135
|
+
"Configure these initial values when creating the realtime session instead. " +
|
|
136
|
+
"To change instructions between turns, call connection.updateInstructions() and wait for session.updated.");
|
|
135
137
|
}
|
|
136
138
|
const websocketUrl = options.websocketUrl ??
|
|
137
139
|
(options.clientSecret
|
|
@@ -164,7 +166,8 @@ class RealtimeTextToSpeechResource {
|
|
|
164
166
|
if (options.sessionFactory !== undefined && hasRealtimeSessionParams(options)) {
|
|
165
167
|
console.warn("@breeze.blue/sdk: realtime.connectManaged cannot apply modelId, languageCode, instructions, " +
|
|
166
168
|
"voiceSettings, inactivityTimeoutSeconds, or enableLogging when sessionFactory is provided. " +
|
|
167
|
-
"Configure these when the factory creates each realtime session."
|
|
169
|
+
"Configure these when the factory creates each realtime session. After connecting, confirmed " +
|
|
170
|
+
"updateInstructions() values are carried across managed physical sessions.");
|
|
168
171
|
}
|
|
169
172
|
const sessionRequest = {
|
|
170
173
|
modelId: options.modelId,
|
|
@@ -196,6 +199,8 @@ class RealtimeTextToSpeechResource {
|
|
|
196
199
|
return ManagedRealtimeTextToSpeechConnection.open(createConnection, {
|
|
197
200
|
...options,
|
|
198
201
|
timeout: connectionTimeout,
|
|
202
|
+
}, (instructions) => {
|
|
203
|
+
sessionRequest.instructions = instructions;
|
|
199
204
|
});
|
|
200
205
|
}
|
|
201
206
|
}
|
|
@@ -214,6 +219,7 @@ export class RealtimeTextToSpeechConnection {
|
|
|
214
219
|
closed = false;
|
|
215
220
|
queuedBytes = 0;
|
|
216
221
|
failure;
|
|
222
|
+
pendingInstructions;
|
|
217
223
|
closeEventGraceTimer;
|
|
218
224
|
constructor(socket) {
|
|
219
225
|
this.socket = socket;
|
|
@@ -271,6 +277,9 @@ export class RealtimeTextToSpeechConnection {
|
|
|
271
277
|
return connection;
|
|
272
278
|
}
|
|
273
279
|
startTurn(turnId) {
|
|
280
|
+
if (this.pendingInstructions !== undefined) {
|
|
281
|
+
throw new BreezeBlueRealtimeError("Realtime TTS instructions update is awaiting session.updated; wait for that event before starting the next turn.", { code: "BAD_REQUEST", recoverable: true });
|
|
282
|
+
}
|
|
274
283
|
this.sendJson({ type: "turn.start", ...(turnId ? { turn_id: turnId } : {}) });
|
|
275
284
|
}
|
|
276
285
|
appendText(text) {
|
|
@@ -285,6 +294,29 @@ export class RealtimeTextToSpeechConnection {
|
|
|
285
294
|
cancelTurn() {
|
|
286
295
|
this.sendJson({ type: "turn.cancel" });
|
|
287
296
|
}
|
|
297
|
+
/**
|
|
298
|
+
* Request new synthesis instructions for subsequent turns.
|
|
299
|
+
*
|
|
300
|
+
* The matching `session.updated` acknowledgement stays on this
|
|
301
|
+
* connection's existing message stream. Keep the single consumer running
|
|
302
|
+
* and wait for that event before starting the next turn.
|
|
303
|
+
*/
|
|
304
|
+
updateInstructions(instructions) {
|
|
305
|
+
const validated = validateRealtimeInstructions(instructions);
|
|
306
|
+
if (this.pendingInstructions !== undefined) {
|
|
307
|
+
throw new BreezeBlueRealtimeError("Realtime TTS already has an instructions update awaiting session.updated.", { code: "BAD_REQUEST", recoverable: true });
|
|
308
|
+
}
|
|
309
|
+
this.pendingInstructions = validated;
|
|
310
|
+
try {
|
|
311
|
+
this.sendJson({ type: "session.update", instructions: validated });
|
|
312
|
+
}
|
|
313
|
+
catch (error) {
|
|
314
|
+
if (this.pendingInstructions === validated) {
|
|
315
|
+
this.pendingInstructions = undefined;
|
|
316
|
+
}
|
|
317
|
+
throw error;
|
|
318
|
+
}
|
|
319
|
+
}
|
|
288
320
|
/** Send a keepalive ping. The server answers with a `pong` event. */
|
|
289
321
|
ping() {
|
|
290
322
|
this.sendJson({ type: "ping" });
|
|
@@ -296,6 +328,7 @@ export class RealtimeTextToSpeechConnection {
|
|
|
296
328
|
if (this.socket.readyState === WEBSOCKET_OPEN) {
|
|
297
329
|
this.sendJson({ type: "session.close" });
|
|
298
330
|
}
|
|
331
|
+
this.pendingInstructions = undefined;
|
|
299
332
|
this.socket.close(1000);
|
|
300
333
|
this.finish();
|
|
301
334
|
}
|
|
@@ -351,7 +384,10 @@ export class RealtimeTextToSpeechConnection {
|
|
|
351
384
|
this.fail(new BreezeBlueRealtimeError("Realtime TTS returned an invalid JSON event.", { cause }));
|
|
352
385
|
return;
|
|
353
386
|
}
|
|
354
|
-
|
|
387
|
+
const message = camelizeKeys(parsed);
|
|
388
|
+
if (this.handleInstructionsEvent(message)) {
|
|
389
|
+
this.push(message, data.length);
|
|
390
|
+
}
|
|
355
391
|
return;
|
|
356
392
|
}
|
|
357
393
|
if (data instanceof ArrayBuffer) {
|
|
@@ -363,6 +399,7 @@ export class RealtimeTextToSpeechConnection {
|
|
|
363
399
|
}
|
|
364
400
|
}
|
|
365
401
|
handleClose(event) {
|
|
402
|
+
this.pendingInstructions = undefined;
|
|
366
403
|
this.clearCloseEventGraceTimer();
|
|
367
404
|
if (!this.closed && event.code !== 1000) {
|
|
368
405
|
this.push({
|
|
@@ -409,6 +446,7 @@ export class RealtimeTextToSpeechConnection {
|
|
|
409
446
|
this.clearCloseEventGraceTimer();
|
|
410
447
|
this.closed = true;
|
|
411
448
|
this.failure = error;
|
|
449
|
+
this.pendingInstructions = undefined;
|
|
412
450
|
this.queue.length = 0;
|
|
413
451
|
this.queuedBytes = 0;
|
|
414
452
|
this.socket.close(1000);
|
|
@@ -422,6 +460,7 @@ export class RealtimeTextToSpeechConnection {
|
|
|
422
460
|
}
|
|
423
461
|
this.clearCloseEventGraceTimer();
|
|
424
462
|
this.closed = true;
|
|
463
|
+
this.pendingInstructions = undefined;
|
|
425
464
|
for (const waiter of this.waiters.splice(0)) {
|
|
426
465
|
waiter.resolve({ value: undefined, done: true });
|
|
427
466
|
}
|
|
@@ -432,6 +471,25 @@ export class RealtimeTextToSpeechConnection {
|
|
|
432
471
|
this.closeEventGraceTimer = undefined;
|
|
433
472
|
}
|
|
434
473
|
}
|
|
474
|
+
handleInstructionsEvent(message) {
|
|
475
|
+
if (message.type === "session.updated") {
|
|
476
|
+
if (this.pendingInstructions === undefined ||
|
|
477
|
+
message.instructions !== this.pendingInstructions) {
|
|
478
|
+
this.fail(new BreezeBlueRealtimeError("Realtime TTS returned a session.updated acknowledgement that did not match the pending instructions update.", { code: "GENERATION_INVALID_RESPONSE" }));
|
|
479
|
+
return false;
|
|
480
|
+
}
|
|
481
|
+
this.pendingInstructions = undefined;
|
|
482
|
+
}
|
|
483
|
+
else if (message.type === "error" && this.pendingInstructions !== undefined) {
|
|
484
|
+
if (!realtimeErrorFromEvent(message).reconnect) {
|
|
485
|
+
this.pendingInstructions = undefined;
|
|
486
|
+
}
|
|
487
|
+
}
|
|
488
|
+
else if (message.type === "session.closed") {
|
|
489
|
+
this.pendingInstructions = undefined;
|
|
490
|
+
}
|
|
491
|
+
return true;
|
|
492
|
+
}
|
|
435
493
|
}
|
|
436
494
|
/**
|
|
437
495
|
* A logical realtime TTS connection backed by bounded physical WebSocket
|
|
@@ -446,6 +504,7 @@ export class RealtimeTextToSpeechConnection {
|
|
|
446
504
|
*/
|
|
447
505
|
export class ManagedRealtimeTextToSpeechConnection {
|
|
448
506
|
createConnection;
|
|
507
|
+
onInstructionsConfirmed;
|
|
449
508
|
queue = [];
|
|
450
509
|
drainingEpochs = new Set();
|
|
451
510
|
openAttempts = new Set();
|
|
@@ -457,6 +516,7 @@ export class ManagedRealtimeTextToSpeechConnection {
|
|
|
457
516
|
maxPhysicalSessionMs;
|
|
458
517
|
maxReconnectAttempts;
|
|
459
518
|
reconnectBaseDelayMs;
|
|
519
|
+
reapplyInstructionsOnOpen;
|
|
460
520
|
current;
|
|
461
521
|
heartbeatTimer;
|
|
462
522
|
heartbeatAckTimer;
|
|
@@ -467,11 +527,16 @@ export class ManagedRealtimeTextToSpeechConnection {
|
|
|
467
527
|
epochSequence = 0;
|
|
468
528
|
queuedBytes = 0;
|
|
469
529
|
activeTurn = false;
|
|
530
|
+
instructions;
|
|
531
|
+
pendingInstructions;
|
|
470
532
|
rotationPending = false;
|
|
471
533
|
closed = false;
|
|
472
534
|
failure;
|
|
473
|
-
constructor(createConnection, options) {
|
|
535
|
+
constructor(createConnection, options, onInstructionsConfirmed) {
|
|
474
536
|
this.createConnection = createConnection;
|
|
537
|
+
this.onInstructionsConfirmed = onInstructionsConfirmed;
|
|
538
|
+
this.reapplyInstructionsOnOpen = options.sessionFactory !== undefined;
|
|
539
|
+
this.instructions = this.reapplyInstructionsOnOpen ? undefined : options.instructions;
|
|
475
540
|
this.readyTimeoutMs = positiveNumber(options.timeout ?? DEFAULT_REALTIME_HANDSHAKE_TIMEOUT_MS, "timeout");
|
|
476
541
|
this.heartbeatIntervalMs = optionalNonNegativeNumber(options.heartbeatIntervalMs, "heartbeatIntervalMs");
|
|
477
542
|
this.heartbeatTimeoutMs = positiveNumber(options.heartbeatTimeoutMs ?? DEFAULT_REALTIME_HEARTBEAT_TIMEOUT_MS, "heartbeatTimeoutMs");
|
|
@@ -480,17 +545,22 @@ export class ManagedRealtimeTextToSpeechConnection {
|
|
|
480
545
|
this.maxReconnectAttempts = nonNegativeInteger(options.maxReconnectAttempts ?? DEFAULT_REALTIME_MAX_RECONNECT_ATTEMPTS, "maxReconnectAttempts");
|
|
481
546
|
this.reconnectBaseDelayMs = nonNegativeNumber(options.reconnectBaseDelayMs ?? DEFAULT_REALTIME_RECONNECT_BASE_DELAY_MS, "reconnectBaseDelayMs");
|
|
482
547
|
}
|
|
483
|
-
static async open(createConnection, options = {}) {
|
|
484
|
-
const managed = new ManagedRealtimeTextToSpeechConnection(createConnection, options);
|
|
548
|
+
static async open(createConnection, options = {}, onInstructionsConfirmed = () => { }) {
|
|
549
|
+
const managed = new ManagedRealtimeTextToSpeechConnection(createConnection, options, onInstructionsConfirmed);
|
|
485
550
|
const epoch = await managed.openEpoch();
|
|
486
551
|
managed.installEpoch(epoch);
|
|
487
552
|
return managed;
|
|
488
553
|
}
|
|
489
554
|
startTurn(turnId) {
|
|
555
|
+
if (this.pendingInstructions !== undefined) {
|
|
556
|
+
throw new BreezeBlueRealtimeError("Realtime TTS instructions update is awaiting session.updated; wait for that event before starting the next turn.", { code: "BAD_REQUEST", recoverable: true });
|
|
557
|
+
}
|
|
490
558
|
const connection = this.requireCurrentConnection();
|
|
559
|
+
// Claim the turn before the first WebSocket write so a custom WebSocket
|
|
560
|
+
// implementation cannot re-enter updateInstructions() from send().
|
|
561
|
+
this.activeTurn = true;
|
|
491
562
|
try {
|
|
492
563
|
connection.startTurn(turnId);
|
|
493
|
-
this.activeTurn = true;
|
|
494
564
|
}
|
|
495
565
|
catch (error) {
|
|
496
566
|
this.activeTurn = false;
|
|
@@ -536,6 +606,36 @@ export class ManagedRealtimeTextToSpeechConnection {
|
|
|
536
606
|
cancelTurn() {
|
|
537
607
|
this.requireCurrentConnection().cancelTurn();
|
|
538
608
|
}
|
|
609
|
+
/**
|
|
610
|
+
* Update the synthesis instructions used by subsequent turns.
|
|
611
|
+
*
|
|
612
|
+
* The matching `session.updated` acknowledgement stays on this logical
|
|
613
|
+
* connection's ordered event stream. Confirmed instructions are carried to
|
|
614
|
+
* later physical WebSocket epochs.
|
|
615
|
+
*/
|
|
616
|
+
updateInstructions(instructions) {
|
|
617
|
+
const validated = validateRealtimeInstructions(instructions);
|
|
618
|
+
if (this.activeTurn) {
|
|
619
|
+
throw new BreezeBlueRealtimeError("Realtime TTS instructions can only be updated between turns.", { code: "BAD_REQUEST", recoverable: true });
|
|
620
|
+
}
|
|
621
|
+
if (this.pendingInstructions !== undefined) {
|
|
622
|
+
throw new BreezeBlueRealtimeError("Realtime TTS already has an instructions update awaiting session.updated.", { code: "BAD_REQUEST", recoverable: true });
|
|
623
|
+
}
|
|
624
|
+
if (this.transitionPromise !== undefined) {
|
|
625
|
+
throw new BreezeBlueRealtimeError("Realtime TTS is rotating its physical session; wait for the next session.ready event before updating instructions.", { code: "BAD_REQUEST", recoverable: true });
|
|
626
|
+
}
|
|
627
|
+
const connection = this.requireCurrentConnection();
|
|
628
|
+
this.pendingInstructions = validated;
|
|
629
|
+
try {
|
|
630
|
+
connection.updateInstructions(validated);
|
|
631
|
+
}
|
|
632
|
+
catch (error) {
|
|
633
|
+
if (this.pendingInstructions === validated) {
|
|
634
|
+
this.pendingInstructions = undefined;
|
|
635
|
+
}
|
|
636
|
+
throw error;
|
|
637
|
+
}
|
|
638
|
+
}
|
|
539
639
|
/** Send an immediate ping in addition to the managed keepalive schedule. */
|
|
540
640
|
ping() {
|
|
541
641
|
this.requireCurrentConnection().ping();
|
|
@@ -545,6 +645,7 @@ export class ManagedRealtimeTextToSpeechConnection {
|
|
|
545
645
|
return;
|
|
546
646
|
}
|
|
547
647
|
this.closed = true;
|
|
648
|
+
this.pendingInstructions = undefined;
|
|
548
649
|
this.clearTimers();
|
|
549
650
|
this.cancelRetryDelay();
|
|
550
651
|
this.cancelOpenAttempts();
|
|
@@ -636,6 +737,26 @@ export class ManagedRealtimeTextToSpeechConnection {
|
|
|
636
737
|
}
|
|
637
738
|
const message = result.value;
|
|
638
739
|
if (message.type === "session.ready") {
|
|
740
|
+
if (this.reapplyInstructionsOnOpen && this.instructions !== undefined) {
|
|
741
|
+
connection.updateInstructions(this.instructions);
|
|
742
|
+
while (true) {
|
|
743
|
+
const confirmation = await Promise.race([iterator.next(), boundary]);
|
|
744
|
+
if (confirmation.done) {
|
|
745
|
+
throw new BreezeBlueRealtimeError("Realtime TTS connection closed before confirming the instructions update.", { reconnect: true });
|
|
746
|
+
}
|
|
747
|
+
const confirmationMessage = confirmation.value;
|
|
748
|
+
if (confirmationMessage.type === "session.updated") {
|
|
749
|
+
break;
|
|
750
|
+
}
|
|
751
|
+
if (confirmationMessage.type === "error") {
|
|
752
|
+
throw realtimeErrorFromEvent(confirmationMessage);
|
|
753
|
+
}
|
|
754
|
+
if (confirmationMessage.type === "session.closed") {
|
|
755
|
+
throw realtimeErrorFromClose(confirmationMessage);
|
|
756
|
+
}
|
|
757
|
+
pending.push(confirmationMessage);
|
|
758
|
+
}
|
|
759
|
+
}
|
|
639
760
|
this.completeOpenAttempt(attempt);
|
|
640
761
|
return {
|
|
641
762
|
id: ++this.epochSequence,
|
|
@@ -748,6 +869,17 @@ export class ManagedRealtimeTextToSpeechConnection {
|
|
|
748
869
|
if (message.type === "turn.started") {
|
|
749
870
|
this.activeTurn = true;
|
|
750
871
|
}
|
|
872
|
+
else if (message.type === "session.updated") {
|
|
873
|
+
if (this.pendingInstructions === undefined ||
|
|
874
|
+
message.instructions !== this.pendingInstructions) {
|
|
875
|
+
this.fail(new BreezeBlueRealtimeError("Realtime TTS returned a session.updated acknowledgement that did not match the pending instructions update.", { code: "GENERATION_INVALID_RESPONSE" }));
|
|
876
|
+
return false;
|
|
877
|
+
}
|
|
878
|
+
const confirmedInstructions = this.pendingInstructions;
|
|
879
|
+
this.instructions = confirmedInstructions;
|
|
880
|
+
this.pendingInstructions = undefined;
|
|
881
|
+
this.onInstructionsConfirmed(confirmedInstructions);
|
|
882
|
+
}
|
|
751
883
|
else if (message.type === "turn.done") {
|
|
752
884
|
this.activeTurn = false;
|
|
753
885
|
epoch.pendingUsageHistoryIds.add(message.historyItemId);
|
|
@@ -767,13 +899,16 @@ export class ManagedRealtimeTextToSpeechConnection {
|
|
|
767
899
|
epoch.retryAfterMs = Math.max(epoch.retryAfterMs, message.meta.retryAfterMs);
|
|
768
900
|
}
|
|
769
901
|
if (error.recoverable) {
|
|
902
|
+
if (this.pendingInstructions !== undefined && !this.activeTurn) {
|
|
903
|
+
this.pendingInstructions = undefined;
|
|
904
|
+
}
|
|
770
905
|
if (message.code === "GENERATION_CONCURRENCY_EXCEEDED") {
|
|
771
906
|
// The server rejected turn.start before establishing an active turn,
|
|
772
907
|
// so no turn.cancelled event will follow to clear local state.
|
|
773
908
|
this.activeTurn = false;
|
|
774
909
|
}
|
|
775
910
|
this.push(message);
|
|
776
|
-
if (!this.activeTurn && this.rotationPending) {
|
|
911
|
+
if (!this.activeTurn && this.pendingInstructions === undefined && this.rotationPending) {
|
|
777
912
|
this.beginTransition("planned", epoch);
|
|
778
913
|
}
|
|
779
914
|
return true;
|
|
@@ -805,7 +940,7 @@ export class ManagedRealtimeTextToSpeechConnection {
|
|
|
805
940
|
return true;
|
|
806
941
|
}
|
|
807
942
|
this.push(message);
|
|
808
|
-
if (!this.activeTurn && this.rotationPending) {
|
|
943
|
+
if (!this.activeTurn && this.pendingInstructions === undefined && this.rotationPending) {
|
|
809
944
|
this.beginTransition("planned", epoch);
|
|
810
945
|
}
|
|
811
946
|
return true;
|
|
@@ -818,6 +953,7 @@ export class ManagedRealtimeTextToSpeechConnection {
|
|
|
818
953
|
this.fail(activeTurnInterruptedError());
|
|
819
954
|
return;
|
|
820
955
|
}
|
|
956
|
+
this.pendingInstructions = undefined;
|
|
821
957
|
if (epoch.reconnectRequested) {
|
|
822
958
|
this.requestUnexpectedReconnect(epoch);
|
|
823
959
|
return;
|
|
@@ -834,6 +970,7 @@ export class ManagedRealtimeTextToSpeechConnection {
|
|
|
834
970
|
this.fail(activeTurnInterruptedError(cause));
|
|
835
971
|
return;
|
|
836
972
|
}
|
|
973
|
+
this.pendingInstructions = undefined;
|
|
837
974
|
this.clearTimers();
|
|
838
975
|
this.current = undefined;
|
|
839
976
|
epoch.connection.close();
|
|
@@ -843,6 +980,10 @@ export class ManagedRealtimeTextToSpeechConnection {
|
|
|
843
980
|
if (this.closed || this.transitionPromise !== undefined) {
|
|
844
981
|
return;
|
|
845
982
|
}
|
|
983
|
+
if (kind === "planned" && this.pendingInstructions !== undefined) {
|
|
984
|
+
this.rotationPending = true;
|
|
985
|
+
return;
|
|
986
|
+
}
|
|
846
987
|
this.transitionKind = kind;
|
|
847
988
|
this.transitionPromise = (async () => {
|
|
848
989
|
try {
|
|
@@ -855,7 +996,11 @@ export class ManagedRealtimeTextToSpeechConnection {
|
|
|
855
996
|
this.transitionPromise = undefined;
|
|
856
997
|
this.transitionKind = undefined;
|
|
857
998
|
const current = this.current;
|
|
858
|
-
if (!this.closed &&
|
|
999
|
+
if (!this.closed &&
|
|
1000
|
+
this.rotationPending &&
|
|
1001
|
+
!this.activeTurn &&
|
|
1002
|
+
this.pendingInstructions === undefined &&
|
|
1003
|
+
current) {
|
|
859
1004
|
this.beginTransition("planned", current);
|
|
860
1005
|
}
|
|
861
1006
|
}
|
|
@@ -949,7 +1094,7 @@ export class ManagedRealtimeTextToSpeechConnection {
|
|
|
949
1094
|
return;
|
|
950
1095
|
}
|
|
951
1096
|
this.rotationPending = true;
|
|
952
|
-
if (!this.activeTurn) {
|
|
1097
|
+
if (!this.activeTurn && this.pendingInstructions === undefined) {
|
|
953
1098
|
this.beginTransition("planned", epoch);
|
|
954
1099
|
}
|
|
955
1100
|
}, rotationDelay);
|
|
@@ -1050,6 +1195,7 @@ export class ManagedRealtimeTextToSpeechConnection {
|
|
|
1050
1195
|
return;
|
|
1051
1196
|
}
|
|
1052
1197
|
this.failure = error;
|
|
1198
|
+
this.pendingInstructions = undefined;
|
|
1053
1199
|
this.closed = true;
|
|
1054
1200
|
this.clearTimers();
|
|
1055
1201
|
this.cancelRetryDelay();
|
|
@@ -1410,6 +1556,18 @@ function requireWebSocket(WebSocketImpl) {
|
|
|
1410
1556
|
}
|
|
1411
1557
|
return WebSocketImpl;
|
|
1412
1558
|
}
|
|
1559
|
+
function validateRealtimeInstructions(instructions) {
|
|
1560
|
+
if (typeof instructions !== "string") {
|
|
1561
|
+
throw new TypeError("instructions must be a string.");
|
|
1562
|
+
}
|
|
1563
|
+
if (instructions.trim().length === 0) {
|
|
1564
|
+
throw new RangeError("instructions must not be empty.");
|
|
1565
|
+
}
|
|
1566
|
+
if (Array.from(instructions).length > REALTIME_MAX_INSTRUCTIONS_CHARACTERS) {
|
|
1567
|
+
throw new RangeError(`instructions must be at most ${REALTIME_MAX_INSTRUCTIONS_CHARACTERS} characters.`);
|
|
1568
|
+
}
|
|
1569
|
+
return instructions;
|
|
1570
|
+
}
|
|
1413
1571
|
function normalizeBaseUrl(value) {
|
|
1414
1572
|
return value.replace(/\/+$/, "");
|
|
1415
1573
|
}
|
package/dist/errors.d.ts
CHANGED
|
@@ -16,10 +16,11 @@ export declare class BreezeBlueConfigurationError extends BreezeBlueError {
|
|
|
16
16
|
* message iterator rejects with this error.
|
|
17
17
|
*
|
|
18
18
|
* `recoverable` mirrors the Python SDK's `RealtimeError.recoverable`: it is
|
|
19
|
-
* `true`
|
|
20
|
-
*
|
|
21
|
-
*
|
|
22
|
-
*
|
|
19
|
+
* `true` when the same connection remains usable. Most recoverable
|
|
20
|
+
* turn-command errors cancel the active turn; a `session.update` rejected
|
|
21
|
+
* while a turn is active does not, so keep consuming that turn. SDK-detected
|
|
22
|
+
* connection failures are never recoverable. `reconnect` tells you whether
|
|
23
|
+
* opening a new session is expected to succeed.
|
|
23
24
|
*/
|
|
24
25
|
export declare class BreezeBlueRealtimeError extends BreezeBlueError {
|
|
25
26
|
readonly code: string | undefined;
|
package/dist/errors.js
CHANGED
|
@@ -12,10 +12,11 @@ export class BreezeBlueConfigurationError extends BreezeBlueError {
|
|
|
12
12
|
* message iterator rejects with this error.
|
|
13
13
|
*
|
|
14
14
|
* `recoverable` mirrors the Python SDK's `RealtimeError.recoverable`: it is
|
|
15
|
-
* `true`
|
|
16
|
-
*
|
|
17
|
-
*
|
|
18
|
-
*
|
|
15
|
+
* `true` when the same connection remains usable. Most recoverable
|
|
16
|
+
* turn-command errors cancel the active turn; a `session.update` rejected
|
|
17
|
+
* while a turn is active does not, so keep consuming that turn. SDK-detected
|
|
18
|
+
* connection failures are never recoverable. `reconnect` tells you whether
|
|
19
|
+
* opening a new session is expected to succeed.
|
|
19
20
|
*/
|
|
20
21
|
export class BreezeBlueRealtimeError extends BreezeBlueError {
|
|
21
22
|
code;
|
package/dist/types.d.ts
CHANGED
|
@@ -134,6 +134,10 @@ export interface RealtimeTextToSpeechSessionReadyEvent {
|
|
|
134
134
|
inactivityTimeoutSeconds: number;
|
|
135
135
|
maxSessionSeconds: number;
|
|
136
136
|
}
|
|
137
|
+
export interface RealtimeTextToSpeechSessionUpdatedEvent {
|
|
138
|
+
type: "session.updated";
|
|
139
|
+
instructions: string;
|
|
140
|
+
}
|
|
137
141
|
export interface RealtimeTextToSpeechSessionExpiringEvent {
|
|
138
142
|
type: "session.expiring";
|
|
139
143
|
sessionId: string;
|
|
@@ -209,7 +213,7 @@ export interface RealtimeTextToSpeechErrorEvent {
|
|
|
209
213
|
* (with camelCase keys). Do not rely on an exhaustive `switch` over `type`;
|
|
210
214
|
* ignore event types you do not recognize.
|
|
211
215
|
*/
|
|
212
|
-
export type RealtimeTextToSpeechEvent = RealtimeTextToSpeechSessionReadyEvent | RealtimeTextToSpeechSessionExpiringEvent | RealtimeTextToSpeechPongEvent | RealtimeTextToSpeechTurnStartedEvent | RealtimeTextToSpeechAudioStartedEvent | RealtimeTextToSpeechTurnDoneEvent | RealtimeTextToSpeechTurnCancelledEvent | RealtimeTextToSpeechUsageCommittedEvent | RealtimeTextToSpeechSessionClosedEvent | RealtimeTextToSpeechErrorEvent;
|
|
216
|
+
export type RealtimeTextToSpeechEvent = RealtimeTextToSpeechSessionReadyEvent | RealtimeTextToSpeechSessionUpdatedEvent | RealtimeTextToSpeechSessionExpiringEvent | RealtimeTextToSpeechPongEvent | RealtimeTextToSpeechTurnStartedEvent | RealtimeTextToSpeechAudioStartedEvent | RealtimeTextToSpeechTurnDoneEvent | RealtimeTextToSpeechTurnCancelledEvent | RealtimeTextToSpeechUsageCommittedEvent | RealtimeTextToSpeechSessionClosedEvent | RealtimeTextToSpeechErrorEvent;
|
|
213
217
|
export interface RealtimeTextToSpeechAudioMessage {
|
|
214
218
|
type: "audio";
|
|
215
219
|
audio: Uint8Array;
|
|
@@ -285,11 +289,6 @@ export interface ModelLanguage {
|
|
|
285
289
|
export interface Model {
|
|
286
290
|
modelId: string;
|
|
287
291
|
name: string;
|
|
288
|
-
canDoTextToSpeech: boolean;
|
|
289
|
-
canDoVoiceConversion: boolean;
|
|
290
|
-
canUseStyle: boolean;
|
|
291
|
-
canUseSpeakerBoost: boolean;
|
|
292
|
-
servesProVoices: boolean;
|
|
293
292
|
languages: ModelLanguage[];
|
|
294
293
|
description: string;
|
|
295
294
|
}
|
package/dist/version.d.ts
CHANGED
|
@@ -1,2 +1,2 @@
|
|
|
1
|
-
export declare const SDK_VERSION = "0.
|
|
1
|
+
export declare const SDK_VERSION = "0.7.0";
|
|
2
2
|
export declare const SDK_NAME = "@breeze.blue/sdk";
|
package/dist/version.js
CHANGED
|
@@ -1,2 +1,2 @@
|
|
|
1
|
-
export const SDK_VERSION = "0.
|
|
1
|
+
export const SDK_VERSION = "0.7.0";
|
|
2
2
|
export const SDK_NAME = "@breeze.blue/sdk";
|