@vellumai/assistant 0.11.2-dev.202608072144.9eaee43 → 0.11.2-dev.202608080025.b2695c7
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/docs/flux-turn-detection-spike.md +10 -10
- package/docs/stt-provider-onboarding.md +1 -0
- package/openapi.yaml +8 -0
- package/package.json +1 -1
- package/src/live-voice/__tests__/live-voice-flux-turn-end.test.ts +2 -2
- package/src/live-voice/__tests__/live-voice-metrics.test.ts +4 -4
- package/src/live-voice/live-voice-metrics.ts +2 -2
- package/src/live-voice/live-voice-session.ts +131 -125
- package/src/live-voice/protocol.ts +1 -1
- package/src/providers/speech-to-text/provider-catalog.ts +30 -0
- package/src/stt/types.ts +20 -0
|
@@ -22,10 +22,10 @@ It is also the only endpoint number that the Flux socket teardown stays out of:
|
|
|
22
22
|
|
|
23
23
|
| Path | What `endpointDecisionMaxLatencyMs` actually spans |
|
|
24
24
|
| -------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
25
|
-
| `endpointDecisionSource: "
|
|
25
|
+
| `endpointDecisionSource: "provider"` | Local VAD speech-stop mark to the `turn-end` event. This is the **whole** end-of-turn latency. |
|
|
26
26
|
| `endpointDecisionSource: "front-door"` | Speculative dispatch to the hold verdict, i.e. the endpoint-decider LLM roundtrip **only**. The trailing-silence wait that had to elapse before that dispatch is not in the number, and neither is the hold extension that follows it. |
|
|
27
27
|
|
|
28
|
-
Source: `
|
|
28
|
+
Source: `handleProviderTurnEnd` passes `this.msSinceLocalSpeechStop()` (`live-voice-session.ts`), while `holdSpeculativeTurn` passes `Date.now() - turn.speculativeDispatchedAtMs`, and the dispatch happens at the silence boundary.
|
|
29
29
|
|
|
30
30
|
**It samples different populations.** `markEndpointDecision` is called from exactly two places: the Flux commit, and the hold verdict. A front-door turn that committed straight through the boundary emits **no** `endpoint_decision` at all, and its metrics frame carries none of the three endpoint fields. So the front-door sample is "turns the model judged mid-thought", the slow tail by construction, while the Flux sample is every turn. Two denominators that cannot be reconciled after the fact is exactly why `endpointCommitLatencyMs` exists.
|
|
31
31
|
|
|
@@ -129,7 +129,7 @@ The daemon sends a `metrics` frame over the live-voice WebSocket on `turn_comple
|
|
|
129
129
|
| Field | Meaning |
|
|
130
130
|
| ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------- |
|
|
131
131
|
| `endpointCommitLatencyMs` | **The headline.** Local VAD speech-stop mark to the commit. Present on every committed turn on both arms. Compare medians across arms. |
|
|
132
|
-
| `endpointDecisionSource` | `"
|
|
132
|
+
| `endpointDecisionSource` | `"provider"` or `"front-door"`. Present only when an endpoint decision was recorded. |
|
|
133
133
|
| `endpointDecisionMaxLatencyMs` | Worst single decision latency in the turn. A breakdown of the headline, never a cross-arm comparison. See the first section. |
|
|
134
134
|
| `endpointHoldCount` | Hold verdicts in the turn. Always 0 on a Flux-committed turn. |
|
|
135
135
|
|
|
@@ -141,21 +141,21 @@ The last three are **absent** unless a decision was recorded, which is deliberat
|
|
|
141
141
|
|
|
142
142
|
| What you see | What happened |
|
|
143
143
|
| -------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
144
|
-
| `endpointDecisionSource: "
|
|
144
|
+
| `endpointDecisionSource: "provider"` | Flux committed the turn. This is the measurement you came for. |
|
|
145
145
|
| No endpoint fields at all | The fail-open deadline fired, the utterance fell back to the silence path, and the front door released it without holding. Not a Flux sample. |
|
|
146
146
|
| `endpointDecisionSource: "front-door"` | Same fallback, but the front door then held. Not a Flux sample either. |
|
|
147
147
|
|
|
148
148
|
Fallbacks are logged at `warn`, so they are visible at the default log level:
|
|
149
149
|
|
|
150
150
|
```
|
|
151
|
-
No
|
|
151
|
+
No provider end-of-turn is coming; falling back to the silence boundary for this utterance
|
|
152
152
|
```
|
|
153
153
|
|
|
154
|
-
The deadline is `liveVoice.flux.eotTimeoutMs` plus a 1000ms margin (`
|
|
154
|
+
The deadline is `liveVoice.flux.eotTimeoutMs` plus a 1000ms margin (`PROVIDER_TURN_END_FALLBACK_MARGIN_MS`), measured from the local speech-stop mark. If you see these routinely, you are measuring the fallback path, not Flux.
|
|
155
155
|
|
|
156
156
|
#### Lower `eotTimeoutMs` for the run
|
|
157
157
|
|
|
158
|
-
With the shipped defaults the fail-open budget is `eotTimeoutMs` (**5000**) plus `
|
|
158
|
+
With the shipped defaults the fail-open budget is `eotTimeoutMs` (**5000**) plus `PROVIDER_TURN_END_FALLBACK_MARGIN_MS` (**1000**), so **6000ms from the local speech-stop mark**. The local silence boundary fires at ~1200ms and then hands the turn to Flux, so a stalled turn is roughly **4.8 seconds of dead air** before the utterance replays onto the hold path and anyone answers.
|
|
159
159
|
|
|
160
160
|
That is the worst case by design (the budget has to clear Flux's own force-end), but it is a bad property to carry through a measurement run. It makes a stall expensive to sit through, which biases you toward not reproducing one, and it means arm B's slow turns are dominated by a timeout rather than by turn detection.
|
|
161
161
|
|
|
@@ -163,14 +163,14 @@ Set `liveVoice.flux.eotTimeoutMs` to **1500 to 2000** for the duration of the ru
|
|
|
163
163
|
|
|
164
164
|
### Prefer medians, and do not chase a single bad sample
|
|
165
165
|
|
|
166
|
-
`
|
|
166
|
+
`isStaleProviderTurnEnd` decides on Flux's own `turn_index`, which `parseFluxFrame` carries onto `turn-start` and `turn-end`. The cycle records the newest turn Flux opened, so a delayed end-of-turn for an older index is recognized as stale and dropped, even though the caller has resumed speaking since. An end-of-turn for the turn still in progress is never stale: the mid-thought pause is what Flux's turn model exists to judge, its verdict covers the resumed speech, and the newest speech-stop mark is the right anchor for it.
|
|
167
167
|
|
|
168
|
-
The local VAD generation counter is only the fallback, for an event that carries no turn number. There the outlier mode survives: if the caller resumes and stops again before a delayed `turn-end` lands, the second boundary re-stamps `
|
|
168
|
+
The local VAD generation counter is only the fallback, for an event that carries no turn number. There the outlier mode survives: if the caller resumes and stops again before a delayed `turn-end` lands, the second boundary re-stamps `turnBoundaryGeneration` to the current generation, the event stops looking stale, and it is accepted against the newer speech-stop mark. The **commit is still correct**, but the recorded latency is understated for that turn, in `endpointCommitLatencyMs` and `endpointDecisionMaxLatencyMs` alike, since they share the anchor.
|
|
169
169
|
|
|
170
170
|
Drops log at `info` and so are visible at the default log level, carrying `turnIndex`, `openTurnIndex`, `boundaryGeneration`, and `speechGeneration`:
|
|
171
171
|
|
|
172
172
|
```
|
|
173
|
-
Dropping a stale
|
|
173
|
+
Dropping a stale provider end-of-turn: the caller resumed speaking past the boundary it closed
|
|
174
174
|
```
|
|
175
175
|
|
|
176
176
|
A drop is not itself an error: the cycle stays open and still commits, on the end-of-turn for the resumed speech or on the fail-open deadline. Read the line's fields. A drop with both `turnIndex` and `openTurnIndex` present is the turn-index path working, and it also drops the silent variant of the same race. A drop with `turnIndex` absent means Deepgram is sending unnumbered events and the run is on the generation fallback, where individual samples can be skewed and legitimate fast end-of-turns are dropped conservatively. Either way, report medians over a run of turns and ignore individual extremes.
|
|
@@ -13,6 +13,7 @@ Add a new entry to the `CATALOG` map with:
|
|
|
13
13
|
- `supportedBoundaries` — the set of `SttBoundaryId` values the provider supports. Valid values are `"daemon-batch"` (post-recording transcription) and `"daemon-streaming"` (real-time streaming transcription during conversation).
|
|
14
14
|
- `conversationStreamingMode` — how the provider handles streaming transcription in conversation mode: `"realtime-ws"` (provider supports real-time streaming natively via WebSocket), `"incremental-batch"` (streaming emulated via throttled polling), or `"none"` (no streaming support). Required for all providers.
|
|
15
15
|
- `telephonyMode` — how the provider participates in real-time telephony STT: `"realtime-ws"`, `"batch-only"`, or `"none"`. The telephony capability resolver (`resolveTelephonySttCapability()` in `src/providers/speech-to-text/resolve.ts`) reads this field plus credential availability to decide whether phone calls can run with the provider.
|
|
16
|
+
- `turnDetection`: whether the provider decides end-of-turn itself, `"provider"` or `"none"`. Use `"provider"` only when the adapter emits `turn-start` / `turn-end` on its transcript stream; a live-voice session reads this via `supportsProviderTurnDetection()` to decide whether to arm its provider turn-end path. Default to `"none"`: a provider that declares `"provider"` but never emits the events makes every turn wait out the fail-open deadline before falling back to the silence boundary. A provider declaring `"provider"` should also number its turns, because the staleness check prefers the turn index on the event and degrades to the session's VAD generation counter without one.
|
|
16
17
|
|
|
17
18
|
## 2. Type-system registration
|
|
18
19
|
|
package/openapi.yaml
CHANGED
|
@@ -4354,6 +4354,7 @@ paths:
|
|
|
4354
4354
|
- platform
|
|
4355
4355
|
- a2a
|
|
4356
4356
|
- discord
|
|
4357
|
+
- plugin
|
|
4357
4358
|
ready:
|
|
4358
4359
|
type: boolean
|
|
4359
4360
|
setupStatus:
|
|
@@ -4444,6 +4445,7 @@ paths:
|
|
|
4444
4445
|
- platform
|
|
4445
4446
|
- a2a
|
|
4446
4447
|
- discord
|
|
4448
|
+
- plugin
|
|
4447
4449
|
includeRemote:
|
|
4448
4450
|
description: Include remote checks (default true)
|
|
4449
4451
|
type: boolean
|
|
@@ -4474,6 +4476,7 @@ paths:
|
|
|
4474
4476
|
- platform
|
|
4475
4477
|
- a2a
|
|
4476
4478
|
- discord
|
|
4479
|
+
- plugin
|
|
4477
4480
|
ready:
|
|
4478
4481
|
type: boolean
|
|
4479
4482
|
setupStatus:
|
|
@@ -6578,6 +6581,7 @@ paths:
|
|
|
6578
6581
|
- platform
|
|
6579
6582
|
- a2a
|
|
6580
6583
|
- discord
|
|
6584
|
+
- plugin
|
|
6581
6585
|
description:
|
|
6582
6586
|
Filter by origin channel. When provided, only conversations with this exact origin_channel value are
|
|
6583
6587
|
returned. Omit to include all channels.
|
|
@@ -6714,6 +6718,7 @@ paths:
|
|
|
6714
6718
|
- platform
|
|
6715
6719
|
- a2a
|
|
6716
6720
|
- discord
|
|
6721
|
+
- plugin
|
|
6717
6722
|
- type: "null"
|
|
6718
6723
|
assistantAttention:
|
|
6719
6724
|
type: object
|
|
@@ -7077,6 +7082,7 @@ paths:
|
|
|
7077
7082
|
- platform
|
|
7078
7083
|
- a2a
|
|
7079
7084
|
- discord
|
|
7085
|
+
- plugin
|
|
7080
7086
|
- type: "null"
|
|
7081
7087
|
assistantAttention:
|
|
7082
7088
|
type: object
|
|
@@ -8290,6 +8296,7 @@ paths:
|
|
|
8290
8296
|
- platform
|
|
8291
8297
|
- a2a
|
|
8292
8298
|
- discord
|
|
8299
|
+
- plugin
|
|
8293
8300
|
- type: "null"
|
|
8294
8301
|
assistantAttention:
|
|
8295
8302
|
type: object
|
|
@@ -21958,6 +21965,7 @@ paths:
|
|
|
21958
21965
|
- platform
|
|
21959
21966
|
- a2a
|
|
21960
21967
|
- discord
|
|
21968
|
+
- plugin
|
|
21961
21969
|
- scheduler
|
|
21962
21970
|
- watcher
|
|
21963
21971
|
sourceContextId:
|
package/package.json
CHANGED
|
@@ -377,7 +377,7 @@ describe("LiveVoiceSession Flux end-of-turn", () => {
|
|
|
377
377
|
await waitFor(() => turnCalls.length === 1);
|
|
378
378
|
|
|
379
379
|
const metricsFrame = await waitForTurnMetrics(frames);
|
|
380
|
-
expect(metricsFrame.endpointDecisionSource).toBe("
|
|
380
|
+
expect(metricsFrame.endpointDecisionSource).toBe("provider");
|
|
381
381
|
expect(metricsFrame.endpointHoldCount).toBe(0);
|
|
382
382
|
|
|
383
383
|
await session.close("client_end");
|
|
@@ -606,7 +606,7 @@ describe("LiveVoiceSession Flux end-of-turn", () => {
|
|
|
606
606
|
expect(frames.some((frame) => frame.type === "utterance_end")).toBe(true);
|
|
607
607
|
|
|
608
608
|
const metricsFrame = await waitForTurnMetrics(frames);
|
|
609
|
-
expect(metricsFrame.endpointDecisionSource).toBe("
|
|
609
|
+
expect(metricsFrame.endpointDecisionSource).toBe("provider");
|
|
610
610
|
|
|
611
611
|
await session.close("client_end");
|
|
612
612
|
});
|
|
@@ -376,19 +376,19 @@ describe("LiveVoiceMetricsCollector", () => {
|
|
|
376
376
|
collector.markEndpointDecision("turn-src-flux", {
|
|
377
377
|
action: "hold",
|
|
378
378
|
latencyMs: 40,
|
|
379
|
-
source: "
|
|
379
|
+
source: "provider",
|
|
380
380
|
});
|
|
381
381
|
collector.markEndpointDecision("turn-src-flux", {
|
|
382
382
|
action: "release",
|
|
383
383
|
latencyMs: 120,
|
|
384
|
-
source: "
|
|
384
|
+
source: "provider",
|
|
385
385
|
});
|
|
386
386
|
const completed = collector.completeTurn();
|
|
387
387
|
|
|
388
388
|
expect(completed).toMatchObject({
|
|
389
389
|
endpointHoldCount: 1,
|
|
390
390
|
endpointDecisionMaxLatencyMs: 120,
|
|
391
|
-
endpointDecisionSource: "
|
|
391
|
+
endpointDecisionSource: "provider",
|
|
392
392
|
});
|
|
393
393
|
expect(
|
|
394
394
|
getLiveVoiceMetricsAggregateFields(
|
|
@@ -398,7 +398,7 @@ describe("LiveVoiceMetricsCollector", () => {
|
|
|
398
398
|
).toMatchObject({
|
|
399
399
|
endpointHoldCount: 1,
|
|
400
400
|
endpointDecisionMaxLatencyMs: 120,
|
|
401
|
-
endpointDecisionSource: "
|
|
401
|
+
endpointDecisionSource: "provider",
|
|
402
402
|
});
|
|
403
403
|
});
|
|
404
404
|
|
|
@@ -31,8 +31,8 @@ export type LiveVoiceMetricsEvent =
|
|
|
31
31
|
export type VoiceEndpointAction = "release" | "hold";
|
|
32
32
|
|
|
33
33
|
// Which decider produced the endpoint outcome: the speculative front-door
|
|
34
|
-
// leg, or the
|
|
35
|
-
export type VoiceEndpointSource = "front-door" | "
|
|
34
|
+
// leg, or the STT provider's own turn-detection stream.
|
|
35
|
+
export type VoiceEndpointSource = "front-door" | "provider";
|
|
36
36
|
|
|
37
37
|
const DEFAULT_ENDPOINT_SOURCE: VoiceEndpointSource = "front-door";
|
|
38
38
|
|
|
@@ -50,6 +50,7 @@ import {
|
|
|
50
50
|
listProviderIds,
|
|
51
51
|
pinnedListeningLanguage,
|
|
52
52
|
supportsBoundary,
|
|
53
|
+
supportsProviderTurnDetection,
|
|
53
54
|
} from "../providers/speech-to-text/provider-catalog.js";
|
|
54
55
|
import type { ResolveStreamingTranscriberOptions } from "../providers/speech-to-text/resolve.js";
|
|
55
56
|
import { broadcastMessage } from "../runtime/assistant-event-hub.js";
|
|
@@ -213,12 +214,12 @@ const BARGE_IN_GAP_TOLERANCE_MS = 200;
|
|
|
213
214
|
// (1 / (1 + ratio) ≈ 20%): once the run is mostly silence it resets, so genuine
|
|
214
215
|
// choppy speech still lands but periodic noise cannot accumulate into one.
|
|
215
216
|
const BARGE_IN_MAX_TOLERATED_SILENCE_RATIO = 4;
|
|
216
|
-
// Slack added to
|
|
217
|
-
// waiting for an
|
|
218
|
-
//
|
|
219
|
-
//
|
|
220
|
-
//
|
|
221
|
-
const
|
|
217
|
+
// Slack added to the configured end-of-turn timeout before the session stops
|
|
218
|
+
// waiting for an event that is not coming and falls the utterance back onto
|
|
219
|
+
// the silence-boundary path. A turn-detecting provider force-ends its own turn
|
|
220
|
+
// at that timeout, so anything past it plus a frame's network and parse hop
|
|
221
|
+
// means the stream is gone, not slow.
|
|
222
|
+
const PROVIDER_TURN_END_FALLBACK_MARGIN_MS = 1_000;
|
|
222
223
|
// At most this many TTS segment jobs are open (provider stream started,
|
|
223
224
|
// frames not yet fully emitted) per turn: the emitting job plus one
|
|
224
225
|
// prefetching job. The prefetch buffers its chunks in memory until promoted;
|
|
@@ -443,18 +444,19 @@ interface UtteranceCycle {
|
|
|
443
444
|
// The provider's end-of-turn never arrived for this cycle, so it fell back
|
|
444
445
|
// to the silence-boundary path and stays there: a late event must not
|
|
445
446
|
// commit a boundary the fallback already owns.
|
|
446
|
-
|
|
447
|
+
providerTurnEndTimedOut: boolean;
|
|
447
448
|
// `vadSpeechGeneration` as of the local silence boundary that handed this
|
|
448
|
-
// cycle to
|
|
449
|
-
// beats the trailing-silence countdown, and that fast commit
|
|
449
|
+
// cycle to the provider, or null while no boundary has fired yet (a turn
|
|
450
|
+
// model routinely beats the trailing-silence countdown, and that fast commit
|
|
451
|
+
// is the point).
|
|
450
452
|
// The staleness signal of last resort: it stands in for the provider's own
|
|
451
|
-
// turn numbering when the provider sends none (see
|
|
452
|
-
|
|
453
|
+
// turn numbering when the provider sends none (see isStaleProviderTurnEnd).
|
|
454
|
+
turnBoundaryGeneration: number | null;
|
|
453
455
|
// Index of the newest provider turn opened in this cycle, from the turn
|
|
454
456
|
// model's own numbering, or null when the provider does not number its
|
|
455
457
|
// turns. An end-of-turn closing an older index describes a turn the
|
|
456
|
-
// provider has already superseded (see
|
|
457
|
-
|
|
458
|
+
// provider has already superseded (see isStaleProviderTurnEnd).
|
|
459
|
+
openProviderTurnIndex: number | null;
|
|
458
460
|
// The transcript the most recent hold verdict judged (unified front-door).
|
|
459
461
|
// A final segment arriving during the extension window that extends this
|
|
460
462
|
// text replays the boundary immediately — the hold was judged on stale
|
|
@@ -955,9 +957,9 @@ function createUtteranceCycle(): UtteranceCycle {
|
|
|
955
957
|
finalTranscriptSegments: [],
|
|
956
958
|
latestPartialText: null,
|
|
957
959
|
endpointExtensionCount: 0,
|
|
958
|
-
|
|
959
|
-
|
|
960
|
-
|
|
960
|
+
providerTurnEndTimedOut: false,
|
|
961
|
+
turnBoundaryGeneration: null,
|
|
962
|
+
openProviderTurnIndex: null,
|
|
961
963
|
heldSpeculativeContent: null,
|
|
962
964
|
languageTally: new Map(),
|
|
963
965
|
latestPartialLanguages: null,
|
|
@@ -1267,24 +1269,25 @@ export class LiveVoiceSession implements LiveVoiceSessionContract {
|
|
|
1267
1269
|
/**
|
|
1268
1270
|
* Per-session latch: the provider's committed end-of-turn owns the turn
|
|
1269
1271
|
* boundary instead of the silence boundary's front-door hold verdict. Set
|
|
1270
|
-
* when the config flag is on AND the resolved streaming provider
|
|
1271
|
-
* `
|
|
1272
|
-
* deliberately: there the client's release IS
|
|
1273
|
-
* while the caller still holds the button is not
|
|
1274
|
-
* bug. False leaves every other code path exactly as
|
|
1275
|
-
* the picture.
|
|
1272
|
+
* when the config flag is on AND the resolved streaming provider declares
|
|
1273
|
+
* `turnDetection: "provider"` in the STT catalog AND the session runs server
|
|
1274
|
+
* VAD. Push-to-talk is excluded deliberately: there the client's release IS
|
|
1275
|
+
* the boundary, and answering while the caller still holds the button is not
|
|
1276
|
+
* turn detection, it is a bug. False leaves every other code path exactly as
|
|
1277
|
+
* it is, with no provider turn detection in the picture.
|
|
1276
1278
|
*/
|
|
1277
|
-
private
|
|
1279
|
+
private providerTurnEndActive = false;
|
|
1278
1280
|
// Wall-clock of the newest above-gate audio chunk, tracked in every
|
|
1279
1281
|
// server-VAD session. It is the local VAD's speech-stop mark: the one anchor
|
|
1280
1282
|
// the reported end-of-turn latency is measured from whichever decider
|
|
1281
|
-
// commits the turn, and the anchor for the
|
|
1282
|
-
// push-to-talk, where no local VAD runs.
|
|
1283
|
+
// commits the turn, and the anchor for the provider fallback deadline. Null
|
|
1284
|
+
// in push-to-talk, where no local VAD runs.
|
|
1283
1285
|
private localSpeechStopAtMs: number | null = null;
|
|
1284
|
-
// Fail-open deadline for a
|
|
1285
|
-
// the utterance falls back to the silence-boundary path, so a dead
|
|
1286
|
-
// stream degrades to
|
|
1287
|
-
|
|
1286
|
+
// Fail-open deadline for a provider end-of-turn that never arrives. On
|
|
1287
|
+
// expiry the utterance falls back to the silence-boundary path, so a dead
|
|
1288
|
+
// provider stream degrades to the silence-path behavior instead of a hung
|
|
1289
|
+
// turn.
|
|
1290
|
+
private providerTurnEndTimer: ReturnType<typeof setTimeout> | null = null;
|
|
1288
1291
|
// Effective trailing-silence threshold, mirroring the detector's private
|
|
1289
1292
|
// copy (constructor seed + update_config), reported to the endpoint decider.
|
|
1290
1293
|
private silenceThresholdMs: number;
|
|
@@ -1534,7 +1537,7 @@ export class LiveVoiceSession implements LiveVoiceSessionContract {
|
|
|
1534
1537
|
this.liveActivityReporter.end();
|
|
1535
1538
|
this.turnDetector?.dispose();
|
|
1536
1539
|
this.clearEndpointExtensionTimer();
|
|
1537
|
-
this.
|
|
1540
|
+
this.clearProviderTurnEndTimer();
|
|
1538
1541
|
// There is no longer anyone on the call to speak to, so a queued
|
|
1539
1542
|
// announcement cannot be spoken — but the answer behind it is finished
|
|
1540
1543
|
// work the user asked for, so it takes the same conversation route a
|
|
@@ -1598,13 +1601,14 @@ export class LiveVoiceSession implements LiveVoiceSessionContract {
|
|
|
1598
1601
|
// waiting on this resolve, so the caller can speak and close an entire
|
|
1599
1602
|
// silence boundary while the provider handshake is still in flight. A
|
|
1600
1603
|
// latch still on its default `false` at that boundary hands the first
|
|
1601
|
-
// turn of the session to the
|
|
1602
|
-
//
|
|
1603
|
-
// the session looks like a
|
|
1604
|
-
// one. The resolved provider reconciles this guess
|
|
1605
|
-
|
|
1604
|
+
// turn of the session to the silence path and then ignores the
|
|
1605
|
+
// provider end-of-turn that follows, which is invisible from the
|
|
1606
|
+
// outside: the session looks like a turn-detecting session while its
|
|
1607
|
+
// opening turn is not one. The resolved provider reconciles this guess
|
|
1608
|
+
// below.
|
|
1609
|
+
this.setProviderTurnEndActive(
|
|
1606
1610
|
this.fluxConfig.turnEnd.enabled &&
|
|
1607
|
-
stt.provider
|
|
1611
|
+
supportsProviderTurnDetection(stt.provider as SttProviderId) &&
|
|
1608
1612
|
this.turnDetector !== null,
|
|
1609
1613
|
);
|
|
1610
1614
|
const transcriber = await this.resolveTranscriber({
|
|
@@ -1620,7 +1624,7 @@ export class LiveVoiceSession implements LiveVoiceSessionContract {
|
|
|
1620
1624
|
if (!transcriber) {
|
|
1621
1625
|
// No stream answered, so no end-of-turn ever will: the guess above
|
|
1622
1626
|
// must not outlive the dial that disproved it.
|
|
1623
|
-
this.
|
|
1627
|
+
this.setProviderTurnEndActive(false);
|
|
1624
1628
|
return {
|
|
1625
1629
|
status: "unavailable",
|
|
1626
1630
|
message: unavailableTranscriberMessage(),
|
|
@@ -1634,9 +1638,9 @@ export class LiveVoiceSession implements LiveVoiceSessionContract {
|
|
|
1634
1638
|
// change under a live session, so this normally confirms the guess; it
|
|
1635
1639
|
// clears it when the dial fell back to another provider or resolved
|
|
1636
1640
|
// one the config did not name.
|
|
1637
|
-
this.
|
|
1641
|
+
this.setProviderTurnEndActive(
|
|
1638
1642
|
this.fluxConfig.turnEnd.enabled &&
|
|
1639
|
-
transcriber.providerId
|
|
1643
|
+
supportsProviderTurnDetection(transcriber.providerId) &&
|
|
1640
1644
|
this.turnDetector !== null,
|
|
1641
1645
|
);
|
|
1642
1646
|
if (
|
|
@@ -1674,7 +1678,7 @@ export class LiveVoiceSession implements LiveVoiceSessionContract {
|
|
|
1674
1678
|
}
|
|
1675
1679
|
// The dial threw, so the pre-dial guess is disproved the same way the
|
|
1676
1680
|
// unavailable case disproves it: nothing will send an end-of-turn.
|
|
1677
|
-
this.
|
|
1681
|
+
this.setProviderTurnEndActive(false);
|
|
1678
1682
|
return {
|
|
1679
1683
|
status: "error",
|
|
1680
1684
|
message: `Live voice transcription could not be started: ${errorMessage(
|
|
@@ -1802,7 +1806,7 @@ export class LiveVoiceSession implements LiveVoiceSessionContract {
|
|
|
1802
1806
|
|
|
1803
1807
|
this.state = "failed";
|
|
1804
1808
|
this.clearEndpointExtensionTimer();
|
|
1805
|
-
this.
|
|
1809
|
+
this.clearProviderTurnEndTimer();
|
|
1806
1810
|
await this.sendFrame({
|
|
1807
1811
|
type: "error",
|
|
1808
1812
|
code:
|
|
@@ -2281,7 +2285,7 @@ export class LiveVoiceSession implements LiveVoiceSessionContract {
|
|
|
2281
2285
|
this.clearEndpointExtensionTimer();
|
|
2282
2286
|
// ...and so is a fallback deadline armed for the boundary the caller just
|
|
2283
2287
|
// spoke through. The next silence boundary arms a fresh one.
|
|
2284
|
-
this.
|
|
2288
|
+
this.clearProviderTurnEndTimer();
|
|
2285
2289
|
|
|
2286
2290
|
// Speech resumed while a speculative leg was awaiting its verdict: the
|
|
2287
2291
|
// pause was mid-thought after all. Discard silently (no frames were ever
|
|
@@ -3191,10 +3195,10 @@ export class LiveVoiceSession implements LiveVoiceSessionContract {
|
|
|
3191
3195
|
this.pendingBargeIn = null;
|
|
3192
3196
|
if (reason === "max-duration") {
|
|
3193
3197
|
// A max-duration boundary always releases: drop any pending hold
|
|
3194
|
-
// replay (or
|
|
3195
|
-
// this one already owns.
|
|
3198
|
+
// replay (or provider fallback deadline) so it cannot re-fire a
|
|
3199
|
+
// boundary this one already owns.
|
|
3196
3200
|
this.clearEndpointExtensionTimer();
|
|
3197
|
-
this.
|
|
3201
|
+
this.clearProviderTurnEndTimer();
|
|
3198
3202
|
}
|
|
3199
3203
|
const utterance = this.currentUtterance;
|
|
3200
3204
|
if (!utterance || utterance.released || utterance.completed) {
|
|
@@ -3216,17 +3220,18 @@ export class LiveVoiceSession implements LiveVoiceSessionContract {
|
|
|
3216
3220
|
// release (ptt_release forced the boundary) is the user saying "answer
|
|
3217
3221
|
// now" — never second-guess it.
|
|
3218
3222
|
if (reason === "silence" && !manualRelease) {
|
|
3219
|
-
//
|
|
3220
|
-
// the silence timer only arms the fail-open deadline. An
|
|
3221
|
-
// whose deadline already elapsed falls through to the hold
|
|
3222
|
-
// below, which is the whole point of the fallback.
|
|
3223
|
-
if (this.
|
|
3223
|
+
// The provider owns this boundary: its end-of-turn commits the
|
|
3224
|
+
// utterance, so the silence timer only arms the fail-open deadline. An
|
|
3225
|
+
// utterance whose deadline already elapsed falls through to the hold
|
|
3226
|
+
// path below, which is the whole point of the fallback.
|
|
3227
|
+
if (this.providerTurnEndActive && !utterance.providerTurnEndTimedOut) {
|
|
3224
3228
|
// Stamp the speech run this boundary closed. This is the staleness
|
|
3225
3229
|
// signal of last resort, used only for a provider that does not
|
|
3226
|
-
// number its turns: where
|
|
3227
|
-
// answers the question directly (see
|
|
3228
|
-
|
|
3229
|
-
this.
|
|
3230
|
+
// number its turns: where the provider's own turn index is on the
|
|
3231
|
+
// wire it answers the question directly (see
|
|
3232
|
+
// isStaleProviderTurnEnd).
|
|
3233
|
+
utterance.turnBoundaryGeneration = this.vadSpeechGeneration;
|
|
3234
|
+
this.armProviderTurnEndFallbackTimer(utterance);
|
|
3230
3235
|
return;
|
|
3231
3236
|
}
|
|
3232
3237
|
if (await this.launchSpeculativeAssistantTurn(utterance)) {
|
|
@@ -3259,23 +3264,24 @@ export class LiveVoiceSession implements LiveVoiceSessionContract {
|
|
|
3259
3264
|
}, this.frontModelConfig.endpointExtensionMs);
|
|
3260
3265
|
}
|
|
3261
3266
|
|
|
3262
|
-
// Arms the fail-open deadline for a
|
|
3263
|
-
// own turn after `eotTimeoutMs` of silence, so once
|
|
3264
|
-
// margin has passed since the caller stopped speaking, no
|
|
3265
|
-
// coming and the utterance replays this boundary on the
|
|
3267
|
+
// Arms the fail-open deadline for a provider end-of-turn. A turn-detecting
|
|
3268
|
+
// provider force-ends its own turn after `eotTimeoutMs` of silence, so once
|
|
3269
|
+
// that budget plus a margin has passed since the caller stopped speaking, no
|
|
3270
|
+
// end-of-turn is coming and the utterance replays this boundary on the
|
|
3271
|
+
// silence path.
|
|
3266
3272
|
// `waitMsOverride` collapses that wait when the caller already knows no
|
|
3267
|
-
// end-of-turn is coming (see
|
|
3268
|
-
private
|
|
3273
|
+
// end-of-turn is coming (see setProviderTurnEndActive).
|
|
3274
|
+
private armProviderTurnEndFallbackTimer(
|
|
3269
3275
|
utterance: UtteranceCycle,
|
|
3270
3276
|
waitMsOverride?: number,
|
|
3271
3277
|
): void {
|
|
3272
|
-
this.
|
|
3278
|
+
this.clearProviderTurnEndTimer();
|
|
3273
3279
|
const budgetMs =
|
|
3274
|
-
this.fluxConfig.eotTimeoutMs +
|
|
3280
|
+
this.fluxConfig.eotTimeoutMs + PROVIDER_TURN_END_FALLBACK_MARGIN_MS;
|
|
3275
3281
|
const waitMs =
|
|
3276
3282
|
waitMsOverride ?? Math.max(0, budgetMs - this.msSinceLocalSpeechStop());
|
|
3277
|
-
this.
|
|
3278
|
-
this.
|
|
3283
|
+
this.providerTurnEndTimer = setTimeout(() => {
|
|
3284
|
+
this.providerTurnEndTimer = null;
|
|
3279
3285
|
if (
|
|
3280
3286
|
this.currentUtterance !== utterance ||
|
|
3281
3287
|
utterance.released ||
|
|
@@ -3287,23 +3293,23 @@ export class LiveVoiceSession implements LiveVoiceSessionContract {
|
|
|
3287
3293
|
) {
|
|
3288
3294
|
return;
|
|
3289
3295
|
}
|
|
3290
|
-
utterance.
|
|
3296
|
+
utterance.providerTurnEndTimedOut = true;
|
|
3291
3297
|
log.warn(
|
|
3292
3298
|
{ budgetMs, waitMs },
|
|
3293
|
-
"No
|
|
3299
|
+
"No provider end-of-turn is coming; falling back to the silence boundary for this utterance",
|
|
3294
3300
|
);
|
|
3295
3301
|
this.handleVadUtteranceEnd("silence");
|
|
3296
3302
|
}, waitMs);
|
|
3297
3303
|
}
|
|
3298
3304
|
|
|
3299
3305
|
/**
|
|
3300
|
-
* Flips the
|
|
3306
|
+
* Flips the provider end-of-turn latch, unwinding an optimistic arm.
|
|
3301
3307
|
*
|
|
3302
3308
|
* `beginUtterance` arms the latch from the configured provider before the
|
|
3303
|
-
* dial resolves, so a silence boundary can already have deferred to
|
|
3304
|
-
* the time the resolved
|
|
3305
|
-
* parked on the fail-open deadline, a whole
|
|
3306
|
-
* waiting for an event that will now never arrive. Collapse the wait to
|
|
3309
|
+
* dial resolves, so a silence boundary can already have deferred to the
|
|
3310
|
+
* provider by the time the resolved one says otherwise. That deferred
|
|
3311
|
+
* boundary is parked on the fail-open deadline, a whole end-of-turn budget
|
|
3312
|
+
* away, waiting for an event that will now never arrive. Collapse the wait to
|
|
3307
3313
|
* zero rather than burn the budget: the deadline body re-checks the cycle
|
|
3308
3314
|
* and replays the silence boundary, which with the latch down takes the
|
|
3309
3315
|
* ordinary hold path. Replaying through the deadline instead of calling
|
|
@@ -3313,38 +3319,38 @@ export class LiveVoiceSession implements LiveVoiceSessionContract {
|
|
|
3313
3319
|
* A cleared latch with no deadline armed needs no unwind: either no
|
|
3314
3320
|
* boundary ever deferred, or the caller resumed speaking and
|
|
3315
3321
|
* `handleVadSpeechStart` already dropped the deadline, leaving the detector
|
|
3316
|
-
* owning the next boundary. The cycle's `
|
|
3317
|
-
* left as it is: `
|
|
3318
|
-
* `
|
|
3322
|
+
* owning the next boundary. The cycle's `turnBoundaryGeneration` stamp is
|
|
3323
|
+
* left as it is: `isStaleProviderTurnEnd` is only ever consulted from
|
|
3324
|
+
* `handleProviderTurnEnd`, which returns immediately once the latch is down.
|
|
3319
3325
|
*/
|
|
3320
|
-
private
|
|
3321
|
-
if (active === this.
|
|
3326
|
+
private setProviderTurnEndActive(active: boolean): void {
|
|
3327
|
+
if (active === this.providerTurnEndActive) {
|
|
3322
3328
|
return;
|
|
3323
3329
|
}
|
|
3324
|
-
this.
|
|
3325
|
-
if (active || this.
|
|
3330
|
+
this.providerTurnEndActive = active;
|
|
3331
|
+
if (active || this.providerTurnEndTimer === null) {
|
|
3326
3332
|
return;
|
|
3327
3333
|
}
|
|
3328
3334
|
const utterance = this.currentUtterance;
|
|
3329
|
-
this.
|
|
3335
|
+
this.clearProviderTurnEndTimer();
|
|
3330
3336
|
if (utterance && !utterance.released && !utterance.completed) {
|
|
3331
|
-
this.
|
|
3337
|
+
this.armProviderTurnEndFallbackTimer(utterance, 0);
|
|
3332
3338
|
}
|
|
3333
3339
|
}
|
|
3334
3340
|
|
|
3335
3341
|
/**
|
|
3336
3342
|
* Record the provider turn a cycle is currently inside. Providers without
|
|
3337
3343
|
* turn numbering send no index, which leaves the cycle on the local speech
|
|
3338
|
-
* generation as its only staleness signal (see
|
|
3344
|
+
* generation as its only staleness signal (see isStaleProviderTurnEnd).
|
|
3339
3345
|
*/
|
|
3340
|
-
private
|
|
3346
|
+
private recordProviderTurnStart(
|
|
3341
3347
|
utterance: UtteranceCycle,
|
|
3342
3348
|
turnIndex: number | undefined,
|
|
3343
3349
|
): void {
|
|
3344
3350
|
if (turnIndex === undefined) {
|
|
3345
3351
|
return;
|
|
3346
3352
|
}
|
|
3347
|
-
utterance.
|
|
3353
|
+
utterance.openProviderTurnIndex = turnIndex;
|
|
3348
3354
|
}
|
|
3349
3355
|
|
|
3350
3356
|
/**
|
|
@@ -3354,13 +3360,13 @@ export class LiveVoiceSession implements LiveVoiceSessionContract {
|
|
|
3354
3360
|
* still routing the resumed speech into this same cycle, cutting off the
|
|
3355
3361
|
* next words or folding them into the wrong turn.
|
|
3356
3362
|
*
|
|
3357
|
-
*
|
|
3358
|
-
* wherever it is available: the cycle records the newest turn
|
|
3359
|
-
* opened, so an end-of-turn for an older index closes a turn
|
|
3360
|
-
* has already superseded. An end-of-turn for the turn still
|
|
3361
|
-
* never stale, however many times the caller drew breath
|
|
3362
|
-
* mid-thought pause is exactly the case
|
|
3363
|
-
* and its verdict covers the resumed speech too.
|
|
3363
|
+
* The provider's own turn numbering answers this directly and is preferred
|
|
3364
|
+
* wherever it is available: the cycle records the newest turn the provider
|
|
3365
|
+
* has opened, so an end-of-turn for an older index closes a turn the
|
|
3366
|
+
* provider itself has already superseded. An end-of-turn for the turn still
|
|
3367
|
+
* in progress is never stale, however many times the caller drew breath
|
|
3368
|
+
* inside it: the mid-thought pause is exactly the case a provider's turn
|
|
3369
|
+
* model exists to judge, and its verdict covers the resumed speech too.
|
|
3364
3370
|
*
|
|
3365
3371
|
* With no turn numbering the local speech generation is the only signal
|
|
3366
3372
|
* left. The silence boundary stamps the cycle with the generation it
|
|
@@ -3370,23 +3376,23 @@ export class LiveVoiceSession implements LiveVoiceSessionContract {
|
|
|
3370
3376
|
* feature exists for, which the turn index would have committed. Before
|
|
3371
3377
|
* any boundary has fired the stamp is null and nothing is stale.
|
|
3372
3378
|
*/
|
|
3373
|
-
private
|
|
3379
|
+
private isStaleProviderTurnEnd(
|
|
3374
3380
|
utterance: UtteranceCycle,
|
|
3375
3381
|
turnIndex: number | undefined,
|
|
3376
3382
|
): boolean {
|
|
3377
|
-
if (turnIndex !== undefined && utterance.
|
|
3378
|
-
return turnIndex < utterance.
|
|
3383
|
+
if (turnIndex !== undefined && utterance.openProviderTurnIndex !== null) {
|
|
3384
|
+
return turnIndex < utterance.openProviderTurnIndex;
|
|
3379
3385
|
}
|
|
3380
3386
|
return (
|
|
3381
|
-
utterance.
|
|
3382
|
-
utterance.
|
|
3387
|
+
utterance.turnBoundaryGeneration !== null &&
|
|
3388
|
+
utterance.turnBoundaryGeneration !== this.vadSpeechGeneration
|
|
3383
3389
|
);
|
|
3384
3390
|
}
|
|
3385
3391
|
|
|
3386
|
-
private
|
|
3387
|
-
if (this.
|
|
3388
|
-
clearTimeout(this.
|
|
3389
|
-
this.
|
|
3392
|
+
private clearProviderTurnEndTimer(): void {
|
|
3393
|
+
if (this.providerTurnEndTimer !== null) {
|
|
3394
|
+
clearTimeout(this.providerTurnEndTimer);
|
|
3395
|
+
this.providerTurnEndTimer = null;
|
|
3390
3396
|
}
|
|
3391
3397
|
}
|
|
3392
3398
|
|
|
@@ -3401,7 +3407,7 @@ export class LiveVoiceSession implements LiveVoiceSessionContract {
|
|
|
3401
3407
|
}
|
|
3402
3408
|
|
|
3403
3409
|
/**
|
|
3404
|
-
*
|
|
3410
|
+
* The provider committed an end of turn: the caller has finished, so the utterance
|
|
3405
3411
|
* commits now instead of waiting out the trailing-silence timer and asking
|
|
3406
3412
|
* the front door whether the pause was mid-thought. This runs the ordinary
|
|
3407
3413
|
* post-release path (utterance_end, then release), so the front-door leg it
|
|
@@ -3410,33 +3416,33 @@ export class LiveVoiceSession implements LiveVoiceSessionContract {
|
|
|
3410
3416
|
* the caller has finished. The escalate verdict and the spoken ack /
|
|
3411
3417
|
* progress phrasing are untouched; only the hold verdict is bypassed.
|
|
3412
3418
|
*/
|
|
3413
|
-
private async
|
|
3419
|
+
private async handleProviderTurnEnd(
|
|
3414
3420
|
utterance: UtteranceCycle,
|
|
3415
3421
|
turnIndex: number | undefined,
|
|
3416
3422
|
): Promise<void> {
|
|
3417
3423
|
if (
|
|
3418
|
-
!this.
|
|
3424
|
+
!this.providerTurnEndActive ||
|
|
3419
3425
|
this.currentUtterance !== utterance ||
|
|
3420
3426
|
utterance.released ||
|
|
3421
3427
|
utterance.completed ||
|
|
3422
3428
|
utterance.assistantTurnStarted ||
|
|
3423
3429
|
// The fallback already took this utterance back to the silence path.
|
|
3424
|
-
utterance.
|
|
3430
|
+
utterance.providerTurnEndTimedOut
|
|
3425
3431
|
) {
|
|
3426
3432
|
return;
|
|
3427
3433
|
}
|
|
3428
|
-
if (this.
|
|
3434
|
+
if (this.isStaleProviderTurnEnd(utterance, turnIndex)) {
|
|
3429
3435
|
// At `info`, not `debug`: a drop is rare and is the only signal of the
|
|
3430
3436
|
// one known latency-outlier mode, so an operator measuring the two
|
|
3431
3437
|
// paths has to be able to see it at the default log level.
|
|
3432
3438
|
log.info(
|
|
3433
3439
|
{
|
|
3434
3440
|
turnIndex,
|
|
3435
|
-
openTurnIndex: utterance.
|
|
3436
|
-
boundaryGeneration: utterance.
|
|
3441
|
+
openTurnIndex: utterance.openProviderTurnIndex,
|
|
3442
|
+
boundaryGeneration: utterance.turnBoundaryGeneration,
|
|
3437
3443
|
speechGeneration: this.vadSpeechGeneration,
|
|
3438
3444
|
},
|
|
3439
|
-
"Dropping a stale
|
|
3445
|
+
"Dropping a stale provider end-of-turn: the caller resumed speaking past the boundary it closed",
|
|
3440
3446
|
);
|
|
3441
3447
|
// The cycle stays open and local VAD keeps routing the resumed speech
|
|
3442
3448
|
// into it. The detector's next silence boundary re-stamps the
|
|
@@ -3446,23 +3452,23 @@ export class LiveVoiceSession implements LiveVoiceSessionContract {
|
|
|
3446
3452
|
// idle, so the deadline handleVadSpeechStart cleared can never strand
|
|
3447
3453
|
// the cycle with nothing left to close it.
|
|
3448
3454
|
if (this.turnDetector?.isActive !== true) {
|
|
3449
|
-
utterance.
|
|
3450
|
-
this.
|
|
3455
|
+
utterance.turnBoundaryGeneration = this.vadSpeechGeneration;
|
|
3456
|
+
this.armProviderTurnEndFallbackTimer(utterance);
|
|
3451
3457
|
}
|
|
3452
3458
|
return;
|
|
3453
3459
|
}
|
|
3454
|
-
this.
|
|
3460
|
+
this.clearProviderTurnEndTimer();
|
|
3455
3461
|
this.markEndpointDecision(
|
|
3456
3462
|
utterance,
|
|
3457
3463
|
"release",
|
|
3458
3464
|
this.msSinceLocalSpeechStop(),
|
|
3459
|
-
"
|
|
3465
|
+
"provider",
|
|
3460
3466
|
);
|
|
3461
3467
|
await this.sendFrame({ type: "utterance_end", reason: "silence" });
|
|
3462
3468
|
await this.releaseUtterance();
|
|
3463
3469
|
// Leave the local detector idle, which is where every other commit path
|
|
3464
|
-
// leaves it.
|
|
3465
|
-
// is still running, and barge-in fires from the detector's
|
|
3470
|
+
// leaves it. A provider can close a turn while the trailing-silence
|
|
3471
|
+
// countdown is still running, and barge-in fires from the detector's speech
|
|
3466
3472
|
// ONSET: a detector left mid-turn reports no onset, so the caller could
|
|
3467
3473
|
// not interrupt the reply they just triggered. The forced boundary
|
|
3468
3474
|
// reaches an already-released utterance and returns.
|
|
@@ -3688,7 +3694,7 @@ export class LiveVoiceSession implements LiveVoiceSessionContract {
|
|
|
3688
3694
|
!utterance.completed
|
|
3689
3695
|
) {
|
|
3690
3696
|
this.clearEndpointExtensionTimer();
|
|
3691
|
-
this.
|
|
3697
|
+
this.clearProviderTurnEndTimer();
|
|
3692
3698
|
await this.sendFrame({ type: "utterance_end", reason: "silence" });
|
|
3693
3699
|
}
|
|
3694
3700
|
await this.releaseUtterance();
|
|
@@ -3898,8 +3904,8 @@ export class LiveVoiceSession implements LiveVoiceSessionContract {
|
|
|
3898
3904
|
// because a provider roundtrip cannot beat a local energy gate on an
|
|
3899
3905
|
// interrupt during playback (see DEFAULT_BARGE_IN_MIN_SPEECH_MS). The
|
|
3900
3906
|
// index is recorded so a later end-of-turn can be told apart from one
|
|
3901
|
-
// this turn superseded (see
|
|
3902
|
-
this.
|
|
3907
|
+
// this turn superseded (see isStaleProviderTurnEnd).
|
|
3908
|
+
this.recordProviderTurnStart(utterance, event.turnIndex);
|
|
3903
3909
|
return;
|
|
3904
3910
|
case "eager-turn-end":
|
|
3905
3911
|
case "turn-resumed":
|
|
@@ -3909,7 +3915,7 @@ export class LiveVoiceSession implements LiveVoiceSessionContract {
|
|
|
3909
3915
|
// follow-up.
|
|
3910
3916
|
return;
|
|
3911
3917
|
case "turn-end":
|
|
3912
|
-
await this.
|
|
3918
|
+
await this.handleProviderTurnEnd(utterance, event.turnIndex);
|
|
3913
3919
|
return;
|
|
3914
3920
|
case "error":
|
|
3915
3921
|
await this.sendTranscriberErrorFrame(event);
|
|
@@ -4044,10 +4050,10 @@ export class LiveVoiceSession implements LiveVoiceSessionContract {
|
|
|
4044
4050
|
// because a provider roundtrip cannot beat a local energy gate on an
|
|
4045
4051
|
// interrupt during playback (see DEFAULT_BARGE_IN_MIN_SPEECH_MS). The
|
|
4046
4052
|
// index is recorded so a later end-of-turn can be told apart from one
|
|
4047
|
-
// this turn superseded (see
|
|
4053
|
+
// this turn superseded (see isStaleProviderTurnEnd).
|
|
4048
4054
|
const target = this.pendingTranscriptCycle();
|
|
4049
4055
|
if (target) {
|
|
4050
|
-
this.
|
|
4056
|
+
this.recordProviderTurnStart(target, event.turnIndex);
|
|
4051
4057
|
}
|
|
4052
4058
|
return;
|
|
4053
4059
|
}
|
|
@@ -4063,7 +4069,7 @@ export class LiveVoiceSession implements LiveVoiceSessionContract {
|
|
|
4063
4069
|
// transcript owns the turn the provider just closed.
|
|
4064
4070
|
const target = this.pendingTranscriptCycle();
|
|
4065
4071
|
if (target) {
|
|
4066
|
-
await this.
|
|
4072
|
+
await this.handleProviderTurnEnd(target, event.turnIndex);
|
|
4067
4073
|
}
|
|
4068
4074
|
return;
|
|
4069
4075
|
}
|
|
@@ -4246,9 +4252,9 @@ export class LiveVoiceSession implements LiveVoiceSessionContract {
|
|
|
4246
4252
|
this.takeVadPreRoll();
|
|
4247
4253
|
this.vadPendingTurnEnd = null;
|
|
4248
4254
|
// ...and abandons any semantic-endpointing hold still awaiting replay,
|
|
4249
|
-
// along with any
|
|
4255
|
+
// along with any provider end-of-turn the session was still waiting on.
|
|
4250
4256
|
this.clearEndpointExtensionTimer();
|
|
4251
|
-
this.
|
|
4257
|
+
this.clearProviderTurnEndTimer();
|
|
4252
4258
|
// A client interrupt is a hard reset: any barge-in merge context waiting for
|
|
4253
4259
|
// the next turn is now stale (the interrupted utterance may be discarded
|
|
4254
4260
|
// without ever reaching finalizePendingUtterance).
|
|
@@ -6128,7 +6134,7 @@ export class LiveVoiceSession implements LiveVoiceSessionContract {
|
|
|
6128
6134
|
action: VoiceEndpointAction,
|
|
6129
6135
|
latencyMs: number,
|
|
6130
6136
|
// Absent means the front door decided, which is what the collector
|
|
6131
|
-
// defaults to; the
|
|
6137
|
+
// defaults to; the provider path names itself so one metrics frame can
|
|
6132
6138
|
// compare the two.
|
|
6133
6139
|
source?: VoiceEndpointSource,
|
|
6134
6140
|
): void {
|
|
@@ -6146,7 +6152,7 @@ export class LiveVoiceSession implements LiveVoiceSessionContract {
|
|
|
6146
6152
|
/**
|
|
6147
6153
|
* The turn's end-of-turn latency, stamped at the moment it commits. Called
|
|
6148
6154
|
* from `releaseUtterance`, which every committed turn passes through
|
|
6149
|
-
* whichever decider released it, so the front-door and
|
|
6155
|
+
* whichever decider released it, so the front-door and provider samples span
|
|
6150
6156
|
* the same thing from the same anchor over the same population. That is
|
|
6151
6157
|
* what `endpointDecisionMaxLatencyMs` cannot offer, and why this exists
|
|
6152
6158
|
* alongside it.
|
|
@@ -6563,7 +6569,7 @@ export function createLiveVoiceSession(
|
|
|
6563
6569
|
options.echoEmaHalfLifeMs ?? vadConfig?.echoEmaHalfLifeMs,
|
|
6564
6570
|
echoDrainSlackMs: options.echoDrainSlackMs ?? vadConfig?.echoDrainSlackMs,
|
|
6565
6571
|
frontModelConfig,
|
|
6566
|
-
// Absent config leaves the schema defaults, which keep
|
|
6572
|
+
// Absent config leaves the schema defaults, which keep provider turn
|
|
6567
6573
|
// detection off.
|
|
6568
6574
|
fluxConfig: options.fluxConfig ?? liveVoiceConfig?.flux,
|
|
6569
6575
|
// Eager construction is safe even when the `liveVoice.frontModel` config
|
|
@@ -368,7 +368,7 @@ export interface LiveVoiceMetricsServerFrame extends LiveVoiceServerFrameBase {
|
|
|
368
368
|
* the STT provider's model-integrated end-of-turn. Present under the same
|
|
369
369
|
* condition as the two fields above.
|
|
370
370
|
*/
|
|
371
|
-
readonly endpointDecisionSource?: "front-door" | "
|
|
371
|
+
readonly endpointDecisionSource?: "front-door" | "provider";
|
|
372
372
|
/** Which floor-holding ack actually spoke during the turn, if any. */
|
|
373
373
|
readonly ackSpoken?: "first_delta" | "tool_use";
|
|
374
374
|
/**
|
|
@@ -14,6 +14,7 @@ import type {
|
|
|
14
14
|
ConversationStreamingMode,
|
|
15
15
|
SttBoundaryId,
|
|
16
16
|
SttProviderId,
|
|
17
|
+
SttTurnDetectionMode,
|
|
17
18
|
TelephonySttMode,
|
|
18
19
|
} from "../../stt/types.js";
|
|
19
20
|
import { baseLanguageSubtag } from "../../util/language-subtag.js";
|
|
@@ -87,6 +88,14 @@ interface SttProviderEntry {
|
|
|
87
88
|
*/
|
|
88
89
|
readonly conversationStreamingMode: ConversationStreamingMode;
|
|
89
90
|
|
|
91
|
+
/**
|
|
92
|
+
* Whether the provider decides end-of-turn itself, in-band with the audio
|
|
93
|
+
* it transcribes, or leaves the boundary to the session's local silence
|
|
94
|
+
* timer. A live-voice session reads this to decide whether to arm its
|
|
95
|
+
* provider turn-end path; it never names a provider directly.
|
|
96
|
+
*/
|
|
97
|
+
readonly turnDetection: SttTurnDetectionMode;
|
|
98
|
+
|
|
90
99
|
/**
|
|
91
100
|
* Whether the provider can attribute transcribed speech to distinct
|
|
92
101
|
* speakers (speaker diarization). When `true`, callers may opt in to
|
|
@@ -159,6 +168,7 @@ const CATALOG: ReadonlyMap<SttProviderId, SttProviderEntry> = new Map<
|
|
|
159
168
|
]),
|
|
160
169
|
telephonyMode: "realtime-ws",
|
|
161
170
|
conversationStreamingMode: "realtime-ws",
|
|
171
|
+
turnDetection: "none",
|
|
162
172
|
supportsDiarization: true,
|
|
163
173
|
languageSelection: "manual",
|
|
164
174
|
credentialsGuide: DEEPGRAM_CREDENTIALS_GUIDE,
|
|
@@ -184,6 +194,9 @@ const CATALOG: ReadonlyMap<SttProviderId, SttProviderEntry> = new Map<
|
|
|
184
194
|
// calls at all.
|
|
185
195
|
telephonyMode: "none",
|
|
186
196
|
conversationStreamingMode: "realtime-ws",
|
|
197
|
+
// Flux emits turn lifecycle events on its transcript stream and numbers
|
|
198
|
+
// them, so a live-voice session may let them commit the turn.
|
|
199
|
+
turnDetection: "provider",
|
|
187
200
|
supportsDiarization: false,
|
|
188
201
|
// "no picker", not native detection: the Flux model is English-only and
|
|
189
202
|
// takes no language parameter, so audio in another language transcribes
|
|
@@ -209,6 +222,7 @@ const CATALOG: ReadonlyMap<SttProviderId, SttProviderEntry> = new Map<
|
|
|
209
222
|
]),
|
|
210
223
|
telephonyMode: "batch-only",
|
|
211
224
|
conversationStreamingMode: "realtime-ws",
|
|
225
|
+
turnDetection: "none",
|
|
212
226
|
supportsDiarization: false,
|
|
213
227
|
languageSelection: "auto",
|
|
214
228
|
credentialsGuide: {
|
|
@@ -235,6 +249,7 @@ const CATALOG: ReadonlyMap<SttProviderId, SttProviderEntry> = new Map<
|
|
|
235
249
|
]),
|
|
236
250
|
telephonyMode: "batch-only",
|
|
237
251
|
conversationStreamingMode: "incremental-batch",
|
|
252
|
+
turnDetection: "none",
|
|
238
253
|
supportsDiarization: false,
|
|
239
254
|
languageSelection: "auto",
|
|
240
255
|
credentialsGuide: {
|
|
@@ -261,6 +276,7 @@ const CATALOG: ReadonlyMap<SttProviderId, SttProviderEntry> = new Map<
|
|
|
261
276
|
]),
|
|
262
277
|
telephonyMode: "realtime-ws",
|
|
263
278
|
conversationStreamingMode: "realtime-ws",
|
|
279
|
+
turnDetection: "none",
|
|
264
280
|
supportsDiarization: false,
|
|
265
281
|
// The relay dials Deepgram nova-3 server-side, which takes an explicit
|
|
266
282
|
// language parameter.
|
|
@@ -283,6 +299,7 @@ const CATALOG: ReadonlyMap<SttProviderId, SttProviderEntry> = new Map<
|
|
|
283
299
|
]),
|
|
284
300
|
telephonyMode: "batch-only",
|
|
285
301
|
conversationStreamingMode: "realtime-ws",
|
|
302
|
+
turnDetection: "none",
|
|
286
303
|
supportsDiarization: true,
|
|
287
304
|
languageSelection: "manual",
|
|
288
305
|
credentialsGuide: {
|
|
@@ -414,6 +431,19 @@ export function supportsDiarization(id: SttProviderId): boolean {
|
|
|
414
431
|
return CATALOG.get(id)?.supportsDiarization ?? false;
|
|
415
432
|
}
|
|
416
433
|
|
|
434
|
+
/**
|
|
435
|
+
* Check whether a provider decides end-of-turn itself.
|
|
436
|
+
*
|
|
437
|
+
* Returns `false` for unknown provider IDs, which keeps an unrecognized
|
|
438
|
+
* provider on the local silence boundary rather than waiting for turn events
|
|
439
|
+
* that will never arrive. A live-voice session reads this instead of naming a
|
|
440
|
+
* provider, so a new turn-detecting provider is a catalog entry rather than a
|
|
441
|
+
* session change.
|
|
442
|
+
*/
|
|
443
|
+
export function supportsProviderTurnDetection(id: SttProviderId): boolean {
|
|
444
|
+
return CATALOG.get(id)?.turnDetection === "provider";
|
|
445
|
+
}
|
|
446
|
+
|
|
417
447
|
/**
|
|
418
448
|
* Return all canonical provider IDs in deterministic (insertion) order.
|
|
419
449
|
*/
|
package/src/stt/types.ts
CHANGED
|
@@ -70,6 +70,26 @@ export type ConversationStreamingMode =
|
|
|
70
70
|
| "incremental-batch"
|
|
71
71
|
| "none";
|
|
72
72
|
|
|
73
|
+
/**
|
|
74
|
+
* Turn-detection support mode.
|
|
75
|
+
*
|
|
76
|
+
* Describes whether a provider decides end-of-turn itself, in-band with the
|
|
77
|
+
* audio it is already transcribing, rather than leaving the boundary to the
|
|
78
|
+
* session's local silence timer and front-door hold verdict.
|
|
79
|
+
*
|
|
80
|
+
* - `"provider"`: the provider emits turn lifecycle events (`turn-start`,
|
|
81
|
+
* `turn-end`) on its transcript stream, and a live-voice session may let
|
|
82
|
+
* those events commit the turn.
|
|
83
|
+
* - `"none"`: the provider transcribes only; the local silence boundary owns
|
|
84
|
+
* end-of-turn.
|
|
85
|
+
*
|
|
86
|
+
* A provider declaring `"provider"` should number its turns: the staleness
|
|
87
|
+
* check prefers the turn index carried on the event, and degrades to the
|
|
88
|
+
* session's local VAD generation counter when one is absent, which understates
|
|
89
|
+
* the recorded latency for a turn the caller resumed and stopped again.
|
|
90
|
+
*/
|
|
91
|
+
export type SttTurnDetectionMode = "provider" | "none";
|
|
92
|
+
|
|
73
93
|
// ---------------------------------------------------------------------------
|
|
74
94
|
// Boundary identifier
|
|
75
95
|
// ---------------------------------------------------------------------------
|