arcane-os 0.75.0 → 0.77.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +32 -0
- package/README.md +5 -4
- package/docs/reference/ai/browser-speech.md +93 -7
- package/docs/reference/inventory/package-api.json +1 -1
- package/docs/reference/inventory/runtime-modules.json +6 -5
- package/docs/reference/runtime-modules.md +61 -10
- package/docs/reference/sdk-api.md +2 -1
- package/package.json +2 -2
- package/runtime/arcane/modules/AI.js +316 -24
- package/runtime/arcane/modules/SpeechPlayback.js +18 -4
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,37 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.77.0
|
|
4
|
+
|
|
5
|
+
- Add `AI.prepareAudioOutput()` for synchronous, silent preparation of a reusable
|
|
6
|
+
WebAudio output. Observe its actual context state through immediate subscription
|
|
7
|
+
replay and `AI_AUDIO_OUTPUT_STATE_EVENT`; resume and dispose explicitly.
|
|
8
|
+
- Add optional `audioOutput` to `playPreparedTTS()` so prepared narration borrows
|
|
9
|
+
that same output across plays. Completion and stop leave the borrowed context
|
|
10
|
+
open; omission preserves the existing per-playback context lifecycle.
|
|
11
|
+
- Keep replacement, pending suspension, decoding, cancellation and disposal tied
|
|
12
|
+
to their actual output and playback owners. Cleanup failures remain observable
|
|
13
|
+
and failed context closure can be retried. Independent native speech, saved
|
|
14
|
+
audio and detached preparation retain their existing ownership.
|
|
15
|
+
- Preparation plays no sound, creates no audio source, resumes no context,
|
|
16
|
+
changes no mute setting and loads no model. A running WebAudio context is not
|
|
17
|
+
a guarantee of future autoplay, native speech or HTML audio readiness. No
|
|
18
|
+
physical-GPU placement or native-engine change is included.
|
|
19
|
+
|
|
20
|
+
## 0.76.0
|
|
21
|
+
|
|
22
|
+
- Add public `textFormat:'plain'` to the `SpeechPlayback` constructor and
|
|
23
|
+
`prepare()` options. Preserve every prepared part's exact input through
|
|
24
|
+
audio-file, native and custom speech clients, including queued narration;
|
|
25
|
+
omitted selection retains the existing formatting cleanup.
|
|
26
|
+
- Distinguish a current native utterance's non-error stopped completion from
|
|
27
|
+
initial preparation with `reason:'native-playback-stopped'` on the existing
|
|
28
|
+
`ready` event. Natural final completion remains `ended`; an error does not
|
|
29
|
+
establish native resource release.
|
|
30
|
+
- Preserve per-part voices, ordered playback, cancellation, replay and silent
|
|
31
|
+
preparation. No model, dependency, native host or activation changes are
|
|
32
|
+
included. Focused source fixtures are authored separately from the selected
|
|
33
|
+
package verification; no live speech result is claimed.
|
|
34
|
+
|
|
3
35
|
## 0.75.0
|
|
4
36
|
|
|
5
37
|
- Advertise whole-image `img2img` alongside `txt2img` for the built-in
|
package/README.md
CHANGED
|
@@ -20,7 +20,7 @@ selected SDK projection or an explicit live SDK source mount. Both profiles
|
|
|
20
20
|
share the theme, packaging, event, cancellation,
|
|
21
21
|
and browser run contracts while retaining their selected application layout.
|
|
22
22
|
|
|
23
|
-
This checkout defines the `0.
|
|
23
|
+
This checkout defines the `0.77.0` SDK contract. Development applications declare
|
|
24
24
|
`arcane-os` as `latest` and retain their lockfile. Exact SDK pins begin when Roshi
|
|
25
25
|
designates a production build. Follow [Consumer updates](docs/consumer-updates.md)
|
|
26
26
|
to refresh and compare the installed SDK and lockfile with the published channel;
|
|
@@ -476,9 +476,10 @@ ordinary repeated punctuation remain literal. Complete synthesis calls may
|
|
|
476
476
|
select `textFormat:'plain'` to preserve exact input: supply it in `prepareTTS`
|
|
477
477
|
options or the payload of `fetchTTS`, provider-runtime TTS requests, or direct
|
|
478
478
|
Kokoro requests. The SDK consumes this control before engine/host dispatch;
|
|
479
|
-
omission keeps the default cleanup.
|
|
480
|
-
|
|
481
|
-
|
|
479
|
+
omission keeps the default cleanup. `SpeechPlayback` accepts the same option in
|
|
480
|
+
its constructor or `prepare()` call and forwards each part's exact input.
|
|
481
|
+
Streaming narration retains its existing formatting behavior. This option does
|
|
482
|
+
not change the selected engine's input capacity or establish complete upstream synthesis. Original
|
|
482
483
|
messages and caller payloads remain unchanged for display, storage, and model
|
|
483
484
|
input. The `arcane-os/speech-text` entrypoint exports the same
|
|
484
485
|
`MarkdownSpeech` streaming class and `stripSpeechFormatting()` one-pass helper
|
|
@@ -889,7 +889,7 @@ provider cleanup remains observable through that operation's settlement.
|
|
|
889
889
|
|
|
890
890
|
### Playback controls and independent lifetimes
|
|
891
891
|
|
|
892
|
-
`ai.playPreparedTTS(prepared,{signal,onState})` returns
|
|
892
|
+
`ai.playPreparedTTS(prepared,{signal,onState,audioOutput})` returns
|
|
893
893
|
`{state,error,finished,pause(),resume(),stop()}` immediately. It uses the
|
|
894
894
|
existing AI audio-clock scheduler and waits for each earlier segment before scheduling
|
|
895
895
|
later audio. Ready adjacent buffers retain contiguous scheduling. Requested
|
|
@@ -902,8 +902,9 @@ states are `waiting`, `waiting-for-gesture`, `scheduled`, `paused`, `complete`,
|
|
|
902
902
|
`stopped`, and `error`. Use `waiting-for-gesture` to present an audio-unlock
|
|
903
903
|
control. A `false` result from `resume()` is not a first-segment-ready signal;
|
|
904
904
|
inspect `state` and `error` to distinguish a stopped or unavailable context
|
|
905
|
-
from browser gesture waiting.
|
|
906
|
-
context immediately, before the first
|
|
905
|
+
from browser gesture waiting. With `audioOutput` omitted or `null`, playback
|
|
906
|
+
creates its own audio context immediately, before the first segment is ready,
|
|
907
|
+
and closes that context after completion, stop, cancellation, or failure.
|
|
907
908
|
|
|
908
909
|
`finished` resolves `true` after natural playback completion and `false` after
|
|
909
910
|
stop, playback cancellation, or failure. Real failures also reach the existing
|
|
@@ -931,6 +932,87 @@ preparations. A page that starts several preparations owns and observes every
|
|
|
931
932
|
`ready` promise, even when it plays only one handle. Browser autoplay permission
|
|
932
933
|
still applies; neither promise proves that a person heard the audio.
|
|
933
934
|
|
|
935
|
+
### Prepare a silent reusable WebAudio output
|
|
936
|
+
|
|
937
|
+
`ai.prepareAudioOutput()` synchronously returns an AI-owned output handle.
|
|
938
|
+
It creates or reuses that AI's persistent WebAudio context and observes its
|
|
939
|
+
actual state. Preparation stays silent: it does not resume the context, unmute
|
|
940
|
+
speech, load a model, create an audio source, play a test sound, or install
|
|
941
|
+
gesture listeners. It does not wait for speech preparation or provider readiness.
|
|
942
|
+
|
|
943
|
+
The handle exposes `kind:'web-audio'`, live `state`, `contextState`, and `error`
|
|
944
|
+
getters, `subscribe(listener)`, `resume()`, and `dispose()`. Repeated preparation
|
|
945
|
+
returns the same handle until its output is disposed or its context closes.
|
|
946
|
+
`contextState` is the platform context's current state, or `null` when unavailable;
|
|
947
|
+
`error` retains the complete actual error, or `null`.
|
|
948
|
+
|
|
949
|
+
| Output state | Meaning |
|
|
950
|
+
| --- | --- |
|
|
951
|
+
| `ready` | This WebAudio context is currently `running`. |
|
|
952
|
+
| `suspended`, `interrupted`, or another platform state | The context reports that state; suspension alone does not establish a gesture requirement. |
|
|
953
|
+
| `gesture-required` | A resume operation actually reported `NotAllowedError` while the context was not running. |
|
|
954
|
+
| `unavailable` | Context creation failed; `error` contains the actual failure. |
|
|
955
|
+
| `error` | An output operation failed; inspect the complete `error`. |
|
|
956
|
+
| `disposing`, `disposed` | Owner disposal is in progress or has completed. |
|
|
957
|
+
| `closed` | The underlying context has closed outside completed owner disposal. |
|
|
958
|
+
|
|
959
|
+
`subscribe(listener)` immediately replays `{kind,state,contextState,error}` and
|
|
960
|
+
then delivers state changes through the canonical AI event owner. It returns
|
|
961
|
+
an idempotent unsubscribe function. The named export
|
|
962
|
+
`AI_AUDIO_OUTPUT_STATE_EVENT` from `AI.js` is `'ai-audio-output-state'`; its
|
|
963
|
+
canonical occurrences carry the same snapshot in `detail` and an output-specific
|
|
964
|
+
`operationId`. Handle subscriptions select only their own output. Disposal or
|
|
965
|
+
context closure releases those subscriptions after the terminal notification;
|
|
966
|
+
a later handle subscription still receives the current snapshot immediately.
|
|
967
|
+
|
|
968
|
+
`resume()` is an explicit asynchronous action for the application's user-gesture
|
|
969
|
+
flow. It resolves `true` only when the context is actually running, and `false`
|
|
970
|
+
when unavailable, disposed, superseded, or held by a paused prepared playback.
|
|
971
|
+
A platform rejection rejects with its actual error and retains that failure in
|
|
972
|
+
the output state. It does not unmute speech or load a model. Output readiness
|
|
973
|
+
does not establish HTML audio readiness, native-speech readiness, future autoplay
|
|
974
|
+
permission, available speech audio, or audible delivery.
|
|
975
|
+
|
|
976
|
+
Pass the handle explicitly to borrow that exact output for prepared playback:
|
|
977
|
+
|
|
978
|
+
```javascript
|
|
979
|
+
// ai and prepared are the same AI owner and preparation used above.
|
|
980
|
+
const audioOutput = ai.prepareAudioOutput();
|
|
981
|
+
|
|
982
|
+
function observeNarrationOutput(listener) {
|
|
983
|
+
return audioOutput.subscribe(listener);
|
|
984
|
+
}
|
|
985
|
+
|
|
986
|
+
function resumeNarrationOutput() {
|
|
987
|
+
// Call from the application's user action and observe the returned promise.
|
|
988
|
+
return audioOutput.resume();
|
|
989
|
+
}
|
|
990
|
+
|
|
991
|
+
function playNarration() {
|
|
992
|
+
return ai.playPreparedTTS(prepared, {audioOutput});
|
|
993
|
+
}
|
|
994
|
+
|
|
995
|
+
function disposeNarrationOutput() {
|
|
996
|
+
// Call when this owner no longer needs the output; observe its promise.
|
|
997
|
+
return audioOutput.dispose();
|
|
998
|
+
}
|
|
999
|
+
```
|
|
1000
|
+
|
|
1001
|
+
The selected handle must be live and belong to the same AI instance; another
|
|
1002
|
+
owner's, closed, or disposing output throws `TypeError`. Borrowed playback
|
|
1003
|
+
retains the same ordered scheduling, pause/resume, cancellation, and single-lane
|
|
1004
|
+
replacement behavior. Completion or stopping playback leaves the output open
|
|
1005
|
+
for reuse. Merely preparing an output does not change the omitted-option path.
|
|
1006
|
+
|
|
1007
|
+
The application owns the output lifetime and calls `dispose()` when finished
|
|
1008
|
+
with it. Disposal stops playback bound to that context and any following work
|
|
1009
|
+
waiting on its clock, observes cleanup, and awaits the actual context close.
|
|
1010
|
+
It preserves detached preparation, saved audio, unrelated contexts, and model
|
|
1011
|
+
selection. Concurrent calls share the disposal promise. Actual cleanup or close
|
|
1012
|
+
errors reject and remain observable; a failed context close permits a later
|
|
1013
|
+
disposal retry. Stop/mute controls retain their existing playback/provider scope
|
|
1014
|
+
and do not replace this explicit output disposal.
|
|
1015
|
+
|
|
934
1016
|
## Stream chunks as they arrive
|
|
935
1017
|
|
|
936
1018
|
Use the configured `ai` created above. Run this snippet from an owned user
|
|
@@ -1008,8 +1090,11 @@ waits for its next character or the final flush.
|
|
|
1008
1090
|
state. `fetchTTS()`, provider-runtime TTS requests, direct Kokoro provider
|
|
1009
1091
|
requests, and `SpeechPlayback` apply the same default cleanup to complete input.
|
|
1010
1092
|
Complete-input APIs can explicitly select `textFormat:'plain'` as described
|
|
1011
|
-
below.
|
|
1012
|
-
|
|
1093
|
+
below. `SpeechPlayback` accepts `textFormat:'plain'` in its constructor or
|
|
1094
|
+
`prepare()` options and forwards each prepared part's exact input through every
|
|
1095
|
+
supported speech client. Omitted options reuse its current selection; explicit
|
|
1096
|
+
`null` restores default cleanup. The streaming `streamTTS()` options still ignore
|
|
1097
|
+
`textFormat` and streaming narration keeps its existing cleanup.
|
|
1013
1098
|
SDK-internal delegation carries `{speechInputPrepared:true}` outside the speech
|
|
1014
1099
|
payload after its owner selects text treatment, preventing a second
|
|
1015
1100
|
non-idempotent cleanup pass or cleanup of explicitly plain input.
|
|
@@ -1019,8 +1104,9 @@ playback timing are not changed by the filter.
|
|
|
1019
1104
|
|
|
1020
1105
|
Direct native `provider.prepare()` and `provider.play()` preserve their complete
|
|
1021
1106
|
`input` as supplied. The native route of `ai.prepareTTSPlayback()` also preserves
|
|
1022
|
-
that input; shared streaming and `SpeechPlayback` perform
|
|
1023
|
-
|
|
1107
|
+
that input; shared streaming and default `SpeechPlayback` perform cleanup before
|
|
1108
|
+
calling it. Plain `SpeechPlayback` preserves the complete input instead.
|
|
1109
|
+
Choose the high-level narration path when that speech-only
|
|
1024
1110
|
formatting cleanup is desired.
|
|
1025
1111
|
|
|
1026
1112
|
The shared helper is public for code that needs the speech-only
|
|
@@ -10,13 +10,14 @@
|
|
|
10
10
|
},
|
|
11
11
|
"artifactCount": 93,
|
|
12
12
|
"javascriptArtifactCount": 91,
|
|
13
|
-
"esmExportCount":
|
|
13
|
+
"esmExportCount": 374,
|
|
14
14
|
"artifacts": [
|
|
15
15
|
{
|
|
16
16
|
"file": "runtime/arcane/modules/AI.js",
|
|
17
17
|
"name": "AI.js",
|
|
18
18
|
"kind": "esm",
|
|
19
19
|
"exports": [
|
|
20
|
+
"AI_AUDIO_OUTPUT_STATE_EVENT",
|
|
20
21
|
"AI_BROWSER_SPEECH_CONFIGURATION_PROTOCOL",
|
|
21
22
|
"AI_BROWSER_SPEECH_ERROR_CODES",
|
|
22
23
|
"AI_BROWSER_SPEECH_EVENT_TYPES",
|
|
@@ -30,7 +31,7 @@
|
|
|
30
31
|
"availability": "Browser + native bridge + TWiN Cloud",
|
|
31
32
|
"protocol": "arcane-ai-browser-speech-configuration/1, AIProviderRuntime arcane-ai-provider/2 routes, globalThis.arcaneEvents, TWiN Cloud HTTPS, Arcane.ollama, Arcane.speech, Android WebView bridge",
|
|
32
33
|
"normalization": "Complete mutable caller-owned browser speech authority, SDK-owned provider registration/replacement/disposal, explicit STT/TTS activation, normalized Whisper STT and Kokoro TTS execution with default capacities 1 and 4 respectively, explicit provider-neutral execution snapshots through status(role,{execution:true}), sticky readiness, canonical window.user readiness without compatibility-event object-identity admission or polling, exact ordered structural calls with required arguments.message, atomic nonblank all-ID tool-result sequencing, complete all-choice streaming/data validation, native Ollama structural adaptation, complete-response then all validated tool callbacks then completion ordering, observed async callbacks, Blob/File transcription, default repeated-formatting-mark removal from cloned outbound TTS input, explicit textFormat:'plain' preservation for complete synthesis/preparation while streaming cleanup remains automatic and caller/visible content stays exact, immediate exact-segment synthesis, per-call voice/speed capture, optional final-segment pause on the existing audio clock, optional per-call terminal playback completion, detached preparation with complete semantic DBOPFS audio reuse and same-owner pending request sharing; each preparation owns its pending activation interest, final-interest cancellation aborts that activation while other interests preserve it, complete cache hits start no model, and cancellation after readiness leaves the ready model loaded; independent prepared playback without a model load for cached audio, ordered audio-clock scheduling, playable audio, active-generation TTS operation-failure routing, mute, and cancellation are normalized; provider/model/runtime/voice policy remains caller-owned. Both speech roles default execution.device to auto: webnn-npu when navigator.ml.createContext is exposed, then webgpu when navigator.gpu is exposed, then wasm for CPU execution. Failed backend loads clean up the pool before trying fresh Workers with the same prepared model and dtype; explicit devices never fall back. STT capacity only accepts 1; TTS accepts 1 through 4. Execution snapshots expose requestedDevice, selectedDevice, maxConcurrentRequests, and activeRequestCount for both roles; selectedDevice is null while unloaded and names the successful upstream session backend request when loaded. It does not prove every operation used the physical NPU: WebNN unsupported operations may use WASM, and compatibility depends on the browser, driver, hardware, and graph.",
|
|
33
|
-
"surface": "Native fetchTTS(payload,signal) and fetchSTT(audio,signal) forward caller cancellation separately from their complete Core speech payloads, normalize aborts to AbortError and suppress late results; actual engine interruption remains service-owned. prepareTTSPlayback(payload,signal) silently returns a real Blob or native-speech descriptor; native controls expose finished/released/state/error/pause/resume/stop, and actual release owns the playback lane. fetchTTS and durable prepareTTS report ARCANE_AI_TTS_AUDIO_EXPORT_UNAVAILABLE for native voices. Request-local model overrides remain exact and never mutate the configured model. Browser-speech protocol/event/error/reason constants; default `AI`; read-only `providerRuntime`, `browserSpeechConfiguration`, and `browserSpeechDescriptor`; explicit `providerRuntime.status(role,{execution:true})` snapshots; `streamTTS(text,end,options={})` with optional voice, speed, pauseAfterMs, and waitForPlayback; default speech-input cleanup with SDK-internal preparation metadata and explicit complete-call textFormat:'plain' consumed before engine dispatch; `prepareTTS({parts,storage,identity,textFormat,signal,onState})` returning ordered segments, state, ready, getAudio(index), and cancel(); `playPreparedTTS(prepared,{signal,onState})` returning state, error, finished, pause(), resume(), and stop() with one playback lane per AI and observational state/error callbacks; configure/dispose speech, route lifecycle, declaration-validated chat/stream requests with exact ordered calls; optional streamRequest toolText:{name,field} and onToolText(text,call,displayId) for decoded selected argument text with actual call identity, distinct from ordinary prose and terminal tool execution; synthesis/transcription and playback controls; initializes from current canonical user readiness, installs `window.ai`, and projects `ai-ready` plus active-generation `ai-tts-failure`."
|
|
34
|
+
"surface": "Native fetchTTS(payload,signal) and fetchSTT(audio,signal) forward caller cancellation separately from their complete Core speech payloads, normalize aborts to AbortError and suppress late results; actual engine interruption remains service-owned. prepareTTSPlayback(payload,signal) silently returns a real Blob or native-speech descriptor; native controls expose finished/released/state/error/pause/resume/stop, and actual release owns the playback lane. fetchTTS and durable prepareTTS report ARCANE_AI_TTS_AUDIO_EXPORT_UNAVAILABLE for native voices. Request-local model overrides remain exact and never mutate the configured model. Browser-speech protocol/event/error/reason constants; default `AI`; read-only `providerRuntime`, `browserSpeechConfiguration`, and `browserSpeechDescriptor`; explicit `providerRuntime.status(role,{execution:true})` snapshots; `streamTTS(text,end,options={})` with optional voice, speed, pauseAfterMs, and waitForPlayback; default speech-input cleanup with SDK-internal preparation metadata and explicit complete-call textFormat:'plain' consumed before engine dispatch; `prepareTTS({parts,storage,identity,textFormat,signal,onState})` returning ordered segments, state, ready, getAudio(index), and cancel(); `prepareAudioOutput()` returning the persistent WebAudio handle {kind,state,contextState,error,subscribe,resume,dispose} without resume, unmute, model load, sources, sound, or gesture listeners; current-state replay and output-specific canonical `AI_AUDIO_OUTPUT_STATE_EVENT` (`ai-audio-output-state`), with ready only for an actually running context; explicit owner disposal observes context-bound cleanup and close; `playPreparedTTS(prepared,{signal,onState,audioOutput})` returning state, error, finished, pause(), resume(), and stop() with one playback lane per AI and observational state/error callbacks; explicit live same-AI audioOutput borrows its context until owner disposal, while omission or null preserves per-playback terminal close; configure/dispose speech, route lifecycle, declaration-validated chat/stream requests with exact ordered calls; optional streamRequest toolText:{name,field} and onToolText(text,call,displayId) for decoded selected argument text with actual call identity, distinct from ordinary prose and terminal tool execution; synthesis/transcription and playback controls; initializes from current canonical user readiness, installs `window.ai`, and projects `ai-ready` plus active-generation `ai-tts-failure`."
|
|
34
35
|
},
|
|
35
36
|
{
|
|
36
37
|
"file": "runtime/arcane/modules/AIModelSelectionController.js",
|
|
@@ -1262,11 +1263,11 @@
|
|
|
1262
1263
|
"default",
|
|
1263
1264
|
"splitSpeechText"
|
|
1264
1265
|
],
|
|
1265
|
-
"summary": "Preserves exact nonblank stored segment text; removes repeated formatting marks from
|
|
1266
|
+
"summary": "Preserves exact nonblank stored segment text; removes repeated formatting marks from the outbound synthesis copy by default or forwards exact input with constructor/prepare textFormat:'plain'; eagerly submits complete parts to a capacity-advertising fetchTTS provider while keeping playback indexed; and retains serialized one-segment lookahead for other clients.",
|
|
1266
1267
|
"availability": "Browser + compatible AI/native bridge",
|
|
1267
1268
|
"protocol": "AI.prepareTTSPlayback or AI.fetchTTS plus providerRuntime TTS execution capacity, native provider.prepare, or compatible serialized Arcane.speech.synthesize; globalThis.arcaneEvents, Blob URLs/audio element, and native playback controls",
|
|
1268
|
-
"normalization": "Exact nonblank split/part input strings are preserved in mutable records. Each synthesis request
|
|
1269
|
-
"surface": "prepareTTSPlayback or direct native provider.prepare may return silent native-speech descriptors. autoplay:false stays silent; actual completion advances ordered parts and failed cancellation retains controls until released. Replay preserves completed segments and retries only missing failed generation. SpeechPlayback class/default, mutable `SPEECH_VOICE_OPTIONS` records, mutable `SPEECH_VOICE_ALIASES` membership, `SPEECH_PLAYBACK_STATE_EVENT`, optional constructor `onState(detail)`, `splitSpeechText()`, canonical playback lifecycle events, cancellation, and destroy APIs."
|
|
1269
|
+
"normalization": "Exact nonblank split/part input strings are preserved in mutable records. Each synthesis request copies that input, preserves it exactly for constructor/prepare textFormat:'plain' or applies default formatting cleanup, and carries SDK-internal preparation metadata to prevent a second cleanup pass. A capable provider receives complete segments immediately and owns bounded FIFO admission; URLs and playback stay in exact input order. Native/custom clients remain serialized. Canonical state dispatch precedes the optional synchronous onState callback. Cancellation, Replay recovery, Blob results, and lifecycle state are normalized; provider failures remain external.",
|
|
1270
|
+
"surface": "prepareTTSPlayback or direct native provider.prepare may return silent native-speech descriptors. autoplay:false stays silent; actual completion advances ordered parts and failed cancellation retains controls until released. Current native finished:false without error reports ready with reason native-playback-stopped; natural final completion reports ended. Replay preserves completed segments and retries only missing failed generation. SpeechPlayback class/default, mutable `SPEECH_VOICE_OPTIONS` records, mutable `SPEECH_VOICE_ALIASES` membership, `SPEECH_PLAYBACK_STATE_EVENT`, optional constructor `onState(detail)`, `splitSpeechText()`, canonical playback lifecycle events, cancellation, and destroy APIs."
|
|
1270
1271
|
},
|
|
1271
1272
|
{
|
|
1272
1273
|
"file": "runtime/arcane/modules/StaticDocumentCatalog.js",
|
|
@@ -132,7 +132,7 @@ own asynchronous work, cancellation, and backpressure.
|
|
|
132
132
|
| [`ScamRiskPolicy.js`](#scamriskpolicyjs) | esm | Combines deterministic scam signals with optional Arcane blocked-domain evidence and safety guidance. | Cross-host | Complete mutable results; blocked-domain policy requires `secure:true`. |
|
|
133
133
|
| [`ScopedOPFSCache.js`](#scopedopfscachejs) | esm | Provides a narrow exact-key JSON cache inside one app-owned OPFS namespace. | Browser / native WebView | Filename-safe keys, complete JSON values, and malformed-cache cleanup normalized; storage errors mixed. |
|
|
134
134
|
| [`ScreenCapture.js`](#screencapturejs) | esm | Captures a display surface as image, video, or GIF with explicit lifecycle events. | Browser / native WebView | State/events normalized; permission and codec errors mixed. |
|
|
135
|
-
| [`SpeechPlayback.js`](#speechplaybackjs) | esm | Admits complete speech segments to a capacity-advertising provider immediately, retains serialized native/custom lookahead, and plays every result in exact indexed order. | Browser + native bridge | Stored part text stays exact;
|
|
135
|
+
| [`SpeechPlayback.js`](#speechplaybackjs) | esm | Admits complete speech segments to a capacity-advertising provider immediately, retains serialized native/custom lookahead, and plays every result in exact indexed order. | Browser + native bridge | Stored part text stays exact; outbound input receives default formatting cleanup or explicit plain-text preservation; provider/media failures remain mixed. |
|
|
136
136
|
| [`StaticDocumentCatalog.js`](#staticdocumentcatalogjs) | esm | Loads a positive static document inventory with cache, search, and complete context. | Browser / native WebView / server with fetch | Mutable complete catalog/content normalization; malformed data and transport failures remain visible. |
|
|
137
137
|
| [`SystemAppearance.js`](#systemappearancejs) | esm | Reads or applies native appearance, returning an explicit unsupported result when the bridge or optional namespace is absent. | Browser/native hybrid | Exact absent optional namespace normalized with its complete error; partial-method and genuine native errors preserved. |
|
|
138
138
|
| [`SystemPlatformPresentation.js`](#systemplatformpresentationjs) | classic-script | Maps kernel names to presentation labels/classes without granting platform authority. | Browser / native WebView classic script | Fully normalized presentation only. |
|
|
@@ -171,10 +171,11 @@ default `AI`; read-only `providerRuntime`, `speechActivationPending`, `browserSp
|
|
|
171
171
|
read-only `ttsSegmentation`, `configureTTSSegmentation()`,
|
|
172
172
|
`streamTTS(text='',end=false,options={})`,
|
|
173
173
|
`prepareTTS({parts,textFormat,storage,identity,signal,onState})`,
|
|
174
|
-
`playPreparedTTS(prepared,{signal,onState})`,
|
|
174
|
+
`prepareAudioOutput()`, `playPreparedTTS(prepared,{signal,onState,audioOutput})`,
|
|
175
175
|
`finishTTS()`, `prepareTTSPlayback()`, `fetchTTS()`, `fetchSTT()`, `stopAudio()`, `resumeAudio()`,
|
|
176
176
|
`playAudio()`; consumes `user-entity-loaded` and `arcane-ollama-ready`,
|
|
177
|
-
installs `window.ai`, and emits `ai-ready
|
|
177
|
+
installs `window.ai`, and emits `ai-ready`, `ai-tts-failure`, and
|
|
178
|
+
`AI_AUDIO_OUTPUT_STATE_EVENT` (`'ai-audio-output-state'`).
|
|
178
179
|
|
|
179
180
|
`fetch(...)` and `fetchRequest(options)` are asynchronous complete-response
|
|
180
181
|
entry points. `streamMessage(...)` and `streamRequest(options)` deliver
|
|
@@ -496,7 +497,7 @@ cancelling one handle does not cancel another active matching caller.
|
|
|
496
497
|
Preparation cancellation prevents later synthesis and releases its activation
|
|
497
498
|
interest. The provider's operation remains responsible for actual cleanup.
|
|
498
499
|
|
|
499
|
-
`playPreparedTTS(prepared,{signal,onState})` can attach immediately. Its returned
|
|
500
|
+
`playPreparedTTS(prepared,{signal,onState,audioOutput})` can attach immediately. Its returned
|
|
500
501
|
`{state,error,finished,pause(),resume(),stop()}` handle schedules segments in
|
|
501
502
|
their original order using the existing AI audio clock. `finished` resolves `true` after
|
|
502
503
|
natural completion and `false` after stop, cancellation, or failure; genuine
|
|
@@ -521,6 +522,38 @@ model even when the AI starts muted. Missing audio alone requests the shared
|
|
|
521
522
|
TTS readiness path. See [prepared narration](ai/browser-speech.md#prepare-narration-once-and-replay-stored-audio)
|
|
522
523
|
for full record shapes, storage ownership, and a complete example.
|
|
523
524
|
|
|
525
|
+
`prepareAudioOutput()` synchronously creates or reuses this AI's persistent
|
|
526
|
+
WebAudio context and returns the same handle until disposed or closed. It is
|
|
527
|
+
silent: no resume, unmute, model load, source creation, test sound, or gesture
|
|
528
|
+
listener registration. The handle exposes `kind:'web-audio'`, current
|
|
529
|
+
`state`, `contextState`, and complete `error` getters, plus
|
|
530
|
+
`subscribe(listener)`, `resume()`, and `dispose()`.
|
|
531
|
+
`subscribe` immediately replays `{kind,state,contextState,error}`, observes
|
|
532
|
+
future changes for this output, and returns an idempotent unsubscribe function.
|
|
533
|
+
The named `AI_AUDIO_OUTPUT_STATE_EVENT` export identifies canonical occurrences
|
|
534
|
+
with that snapshot as `detail` and an output-specific `operationId`.
|
|
535
|
+
|
|
536
|
+
Output `ready` means its context is actually `running`, independently of model
|
|
537
|
+
readiness, HTML audio, native speech, future autoplay permission, or audible
|
|
538
|
+
delivery. Other states are the platform's current state (including `suspended`
|
|
539
|
+
or `interrupted`), `gesture-required` after an actual `NotAllowedError`,
|
|
540
|
+
`unavailable` when context creation fails, `error`, `disposing`, `disposed`,
|
|
541
|
+
or `closed`. Explicit `resume()` resolves a boolean for actual readiness;
|
|
542
|
+
unavailable/disposed, superseded, or paused-playback cases resolve `false`.
|
|
543
|
+
Platform rejection rejects with the actual error and retains observable state.
|
|
544
|
+
|
|
545
|
+
Pass `{audioOutput}` to `playPreparedTTS` to borrow that same AI's live output;
|
|
546
|
+
another owner's, closed, or disposing handle throws `TypeError`. Borrowed
|
|
547
|
+
playback leaves the context open after completion or stop. Omission or `null`
|
|
548
|
+
preserves the existing per-playback context and terminal context close.
|
|
549
|
+
Owner `dispose()` stops context-bound playback and dependent clock work,
|
|
550
|
+
observes cleanup, and awaits actual close without cancelling detached speech
|
|
551
|
+
preparation or deleting saved audio. Concurrent disposal calls share their
|
|
552
|
+
promise; failures remain observable and a failed close can be retried.
|
|
553
|
+
Disposal/closure releases handle subscriptions after their final notification.
|
|
554
|
+
See [silent reusable output](ai/browser-speech.md#prepare-a-silent-reusable-webaudio-output)
|
|
555
|
+
for state details and usage.
|
|
556
|
+
|
|
524
557
|
Streaming speech retains sentence
|
|
525
558
|
segmentation by default. `configureTTSSegmentation({punctuation,wordCadence})`
|
|
526
559
|
accepts `punctuation:'sentence'|'any'|'none'` and a `wordCadence` that is either
|
|
@@ -3973,8 +4006,8 @@ console.log(Object.keys(module));
|
|
|
3973
4006
|
|
|
3974
4007
|
Preserves exact nonblank text, admits complete speech segments according to the
|
|
3975
4008
|
selected client's advertised capacity, and plays indexed HTML audio in exact
|
|
3976
|
-
input order. Stored parts remain exact;
|
|
3977
|
-
|
|
4009
|
+
input order. Stored parts remain exact; each outbound synthesis payload copy
|
|
4010
|
+
receives default formatting-mark cleanup or explicit plain-text preservation.
|
|
3978
4011
|
|
|
3979
4012
|
### Public surface
|
|
3980
4013
|
|
|
@@ -3992,6 +4025,7 @@ new SpeechPlayback({
|
|
|
3992
4025
|
model=null,
|
|
3993
4026
|
voice=null,
|
|
3994
4027
|
responseFormat=null,
|
|
4028
|
+
textFormat=null,
|
|
3995
4029
|
speed=1,
|
|
3996
4030
|
onState=()=>{},
|
|
3997
4031
|
createObjectURL,
|
|
@@ -4006,7 +4040,7 @@ native descriptor, a native provider's silent `prepare(payload,{signal})`,
|
|
|
4006
4040
|
`fetchTTS(payload, signal)`, or `synthesize(payload, {signal})`.
|
|
4007
4041
|
The playback-oriented method is preferred. `SpeechPlayback` also supplies a third
|
|
4008
4042
|
SDK-internal preparation object; existing two-argument clients may ignore it.
|
|
4009
|
-
`prepare({key,parts,model,voice,responseFormat,
|
|
4043
|
+
`prepare({key,parts,model,voice,responseFormat,textFormat,
|
|
4010
4044
|
speed,autoplay=true})` uses only caller-supplied model, voice, and response-format
|
|
4011
4045
|
values; those three omitted values remain omitted so the selected AI/model
|
|
4012
4046
|
catalog may provide its documented defaults. Speed defaults to `1`, is normalized
|
|
@@ -4020,13 +4054,25 @@ There is no hard-coded model, response format, voice, or cloud/browser fallback.
|
|
|
4020
4054
|
the caller's exact string in one mutable array without trimming, splitting, or
|
|
4021
4055
|
freezing it. `prepare()` likewise preserves each nonblank part's exact `input`
|
|
4022
4056
|
string while normalizing its other playback fields into a new mutable record.
|
|
4023
|
-
At synthesis time, `requestSpeech()` copies that record, removes
|
|
4024
|
-
formatting marks from only the outbound `input`, and delegates with the
|
|
4057
|
+
At synthesis time, default `requestSpeech()` behavior copies that record, removes
|
|
4058
|
+
repeated same formatting marks from only the outbound `input`, and delegates with the
|
|
4025
4059
|
SDK-internal `{speechInputPrepared:true}` argument so downstream SDK boundaries
|
|
4026
4060
|
do not apply the non-idempotent filter again. Original part objects, stored
|
|
4027
4061
|
parts, displayed text, and all non-input payload fields remain unchanged. The
|
|
4028
4062
|
class applies no part-count, character-count, pause, or input upper cap.
|
|
4029
4063
|
|
|
4064
|
+
Set `textFormat:'plain'` in the constructor or `prepare()` options to forward
|
|
4065
|
+
every part's exact `input` without formatting cleanup. The selection is captured
|
|
4066
|
+
with the prepared parts, including queued and lookahead requests. A `prepare()`
|
|
4067
|
+
override becomes the instance's selection for later omitted options, like its
|
|
4068
|
+
voice and speed options; explicit `null` or another non-`undefined` value other
|
|
4069
|
+
than `'plain'` selects the existing cleanup. Plain requests carry `textFormat:'plain'` to every
|
|
4070
|
+
supported speech client along with the existing internal preparation marker;
|
|
4071
|
+
default requests keep their existing payload shape. Original part objects stay
|
|
4072
|
+
unchanged, and per-part voices, ordering, cancellation, and silent
|
|
4073
|
+
`autoplay:false` preparation retain their existing behavior. This controls SDK
|
|
4074
|
+
text treatment, not the selected speech engine's pronunciation or input capacity.
|
|
4075
|
+
|
|
4030
4076
|
### Admission and playback order
|
|
4031
4077
|
|
|
4032
4078
|
When `speech` exposes both `fetchTTS(payload, signal)` and
|
|
@@ -4053,6 +4099,11 @@ Replay use the native control; only a successful current native `finished`
|
|
|
4053
4099
|
advances the next part. Native errors remain observable, and a failed native
|
|
4054
4100
|
control stays owned until its optional `released` promise confirms cleanup.
|
|
4055
4101
|
Stale completion after stop/restart/disposal cannot advance replacement content.
|
|
4102
|
+
When current native `finished` resolves `false` without an error, the existing
|
|
4103
|
+
`ready` event carries `reason:'native-playback-stopped'` and its operation ID;
|
|
4104
|
+
initial prepared `ready` has no such reason. Only natural completion of the last
|
|
4105
|
+
part emits `ended`. Errors never establish native resource release; failed
|
|
4106
|
+
controls remain retained through their existing `released` boundary.
|
|
4056
4107
|
Selecting a native part hides and clears the HTML audio source while preserving
|
|
4057
4108
|
the same shared playback state events and complete original parts.
|
|
4058
4109
|
|
|
@@ -4087,7 +4138,7 @@ Exact lifecycle reasons are `playback-replaced`, `playback-stopped`,
|
|
|
4087
4138
|
`playback-destroyed`, `speech-playback-cancelled`,
|
|
4088
4139
|
`speech-synthesis-superseded`, `speech-synthesis-cancelled`,
|
|
4089
4140
|
`speech-synthesizer-unavailable`, `synthesized-audio-contract-mismatch`,
|
|
4090
|
-
`audio-playback-rejected`, `audio-autoplay-rejected`,
|
|
4141
|
+
`audio-playback-rejected`, `audio-autoplay-rejected`, `native-playback-stopped`,
|
|
4091
4142
|
`speech-playback-request-contract-mismatch`, and
|
|
4092
4143
|
`speech-synthesis-rejected`, as applicable to the emitted state.
|
|
4093
4144
|
|
|
@@ -8357,7 +8357,8 @@ mediaType}` and copies the supplied view at acceptance. Its explicit finite
|
|
|
8357
8357
|
`strength` is from `0` through `1`, takes precedence over model/parameter
|
|
8358
8358
|
defaults and is never clamped. It decodes in the existing worker and uses the
|
|
8359
8359
|
same selected context, queue, progress, cancellation and complete result
|
|
8360
|
-
contract. SD1.4
|
|
8360
|
+
contract. SD1.4 and [SDXL Base 1.0](local-image-sdxl.md#edit-a-whole-image)
|
|
8361
|
+
advertise `txt2img` and whole-image `img2img`. See the
|
|
8361
8362
|
[editing contract](local-image-editing.md#core-and-native-request) for supported
|
|
8362
8363
|
PNG input, native preprocessing, alpha and strength-zero semantics; the
|
|
8363
8364
|
original file remains unchanged.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "arcane-os",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.77.0",
|
|
4
4
|
"description": "Arcane OS JavaScript SDK, project-local CLI, browser runtime, and repository-portable application packager.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "./src/index.mjs",
|
|
@@ -125,7 +125,7 @@
|
|
|
125
125
|
"test:unit": "node ./bin/arcane-test.mjs test/app-descriptor.test.mjs test/app-schema.test.mjs test/app-selection.test.mjs test/contracts.test.mjs test/conversation-closing-report.test.mjs test/doctor.test.mjs test/mail-credentials.test.mjs test/mail-outbox.test.mjs test/mail-public-api.test.mjs test/mail-send.test.mjs test/mail-transport.test.mjs test/native-windows-icon.test.mjs test/targets.test.mjs test/workspace-operation-lock.test.mjs",
|
|
126
126
|
"test:functional": "node ./bin/arcane-test.mjs test/ai-ollama-readiness.test.mjs test/core-ollama-lifecycle.test.mjs test/ai-request-model.test.mjs test/ai-request-retry.test.mjs test/ai-stream-cancellation.test.mjs test/browser-decisions.test.mjs test/browser-kokoro-complete-input.test.mjs test/browser-speech-providers.test.mjs test/browser-speech-artifacts.test.mjs test/browser-speech-resource-bridge.test.mjs test/browser-wasm-gpu-notice.test.mjs test/browser-wasm-download-resume.test.mjs test/chat-input.test.mjs test/chat-stream-speech.test.mjs test/chat-transcript-scroll.test.mjs test/chat-transient-retention.test.mjs test/cli.test.mjs test/continuous-voice-capture.test.mjs test/codex-app-server.test.mjs test/mcp-stdio-server.test.mjs test/core-classic-source.test.mjs test/core-client.test.mjs test/core-client-installation.test.mjs test/core-ai-installation.test.mjs test/core-llama-lifecycle.test.mjs test/core-development.test.mjs test/core-host.test.mjs test/core-shared-host.test.mjs test/core-model-assets.test.mjs test/core-native-services.test.mjs test/core-packaged-web.test.mjs test/core-runtime.test.mjs test/core-stdio.test.mjs test/dbopfs-committed-events.test.mjs test/dbopfs-document-library.test.mjs test/dbopfs-model-resources.test.mjs test/dbopfs-png.test.mjs test/dbopfs.test.mjs test/dev-server.test.mjs test/dev-pwa.test.mjs test/dom-event-instrumentation.test.mjs test/event-manager.test.mjs test/events.test.mjs test/file-preview.test.mjs test/import-map.test.mjs test/local-image-editing.test.mjs test/mail-cli.test.mjs test/mail-runtime.test.mjs test/mail-server.test.mjs test/markdown-editor.test.mjs test/markdown-media.test.mjs test/markdown-source-map.test.mjs test/modal.test.mjs test/model-resource-bridge.test.mjs test/native-decisions.test.mjs test/native-onnx.test.mjs test/native-module-packaging.test.mjs test/native-module-resource-references.test.mjs test/packaging.test.mjs test/pwa-packaging.test.mjs test/pwa-client.test.mjs test/pwa-install.test.mjs test/pwa-worker.test.mjs test/persistent-ai-chat-session.test.mjs test/reference-completeness.test.mjs test/rendered-print.test.mjs test/runtime-api-behavior.test.mjs test/runtime.test.mjs test/scaffold.test.mjs test/speech-playback.test.mjs test/site.test.mjs test/theme-manager.test.mjs test/twin-cloud-images.test.mjs test/twin-cloud.test.mjs test/websocket-client-import-map.test.mjs test/websocket.test.mjs test/update-check.test.mjs test/user-explicit-updates.test.mjs",
|
|
127
127
|
"test:integration": "node ./bin/arcane-test.mjs test/installed-package-runtime.test.mjs test/root-app-layout.test.mjs test/integrated-shared.test.mjs test/integrated-workspace.test.mjs test/mail-browser.test.mjs test/native-macos-provider.test.mjs test/native-plan.test.mjs test/native-portable-provider.test.mjs test/native-provider-loader.test.mjs test/npm-release.test.mjs test/release-bundle.test.mjs test/release-capability-smoke.test.mjs test/shared-payload-batch.test.mjs test/tarball.test.mjs test/browser-wasm-cpu.test.mjs test/wllama-webgpu-runtime.test.mjs",
|
|
128
|
-
"test:regression": "node ./bin/arcane-test.mjs test/channel-workflows.test.mjs test/html-import-registration.test.mjs test/html-import-packaged-script.test.mjs test/logging-regression.test.mjs test/markdown-speech.test.mjs test/prepared-speech.test.mjs test/native-provider-generation.test.mjs test/speech-queue-regression.test.mjs test/testing.test.mjs test/test-sets.test.mjs",
|
|
128
|
+
"test:regression": "node ./bin/arcane-test.mjs test/channel-workflows.test.mjs test/html-import-registration.test.mjs test/html-import-packaged-script.test.mjs test/logging-regression.test.mjs test/markdown-speech.test.mjs test/prepared-speech.test.mjs test/ai-audio-output.test.mjs test/native-provider-generation.test.mjs test/speech-queue-regression.test.mjs test/testing.test.mjs test/test-sets.test.mjs",
|
|
129
129
|
"check": "node tools/check-source.mjs && npm test",
|
|
130
130
|
"check:release": "node tools/check-source.mjs --package-only && npm run test:release",
|
|
131
131
|
"pack:local": "npm pack --ignore-scripts",
|
|
@@ -51,6 +51,7 @@ credentials='omit';
|
|
|
51
51
|
const BUILT_IN_AI_SERVICES=new Set(['TWIN','OLLAMA','LOCAL_SPEACH']);
|
|
52
52
|
const AI_REASONING_EFFORTS=new Set(['none','low','medium','high','max']);
|
|
53
53
|
export const AI_READY_EVENT='ai-ready';
|
|
54
|
+
export const AI_AUDIO_OUTPUT_STATE_EVENT='ai-audio-output-state';
|
|
54
55
|
const AI_TTS_FAILURE_EVENT='ai-tts-failure';
|
|
55
56
|
export const AI_INITIALIZATION_ERROR_CODES=completeValue({
|
|
56
57
|
userReadyRegistrationCollision:
|
|
@@ -1128,6 +1129,7 @@ class AI {
|
|
|
1128
1129
|
eventTypes:completeValue(
|
|
1129
1130
|
[
|
|
1130
1131
|
AI_READY_EVENT,
|
|
1132
|
+
AI_AUDIO_OUTPUT_STATE_EVENT,
|
|
1131
1133
|
AI_TTS_FAILURE_EVENT,
|
|
1132
1134
|
...Object.values(AI_BROWSER_SPEECH_EVENT_TYPES)
|
|
1133
1135
|
]
|
|
@@ -1183,8 +1185,12 @@ class AI {
|
|
|
1183
1185
|
#stopOllamaReady=null;
|
|
1184
1186
|
#ttsSegmentation={...DEFAULT_TTS_SEGMENTATION};
|
|
1185
1187
|
#preparedSpeechPlaybacks = new Set();
|
|
1188
|
+
#activePreparedSpeechPlayback = null;
|
|
1186
1189
|
#nativeSpeechPlaybacks = new Set();
|
|
1187
1190
|
#speechUnlockContext = null;
|
|
1191
|
+
#speechResumeContext = null;
|
|
1192
|
+
#audioOutput = null;
|
|
1193
|
+
#audioOutputSequence = 0;
|
|
1188
1194
|
|
|
1189
1195
|
#traceSpeech(phase,detail={}){
|
|
1190
1196
|
if(!arcaneLogging.enabled)return null;
|
|
@@ -6434,7 +6440,7 @@ class AI {
|
|
|
6434
6440
|
);
|
|
6435
6441
|
}
|
|
6436
6442
|
|
|
6437
|
-
playPreparedTTS(prepared, {signal = null, onState = null} = {}) {
|
|
6443
|
+
playPreparedTTS(prepared, {signal = null, onState = null, audioOutput = null} = {}) {
|
|
6438
6444
|
if(!is.array(prepared?.segments) || !is.function(prepared?.getAudio)) {
|
|
6439
6445
|
throw new TypeError('AI.playPreparedTTS requires a speech preparation handle.');
|
|
6440
6446
|
}
|
|
@@ -6442,9 +6448,19 @@ class AI {
|
|
|
6442
6448
|
if(onState !== null && !is.function(onState)) {
|
|
6443
6449
|
throw new TypeError('Prepared playback onState must be a function when provided.');
|
|
6444
6450
|
}
|
|
6451
|
+
const output = audioOutput === null ? null : this.#audioOutput;
|
|
6452
|
+
if(audioOutput !== null && (output?.handle !== audioOutput
|
|
6453
|
+
|| output.disposing || output.disposed || !output.context
|
|
6454
|
+
|| output.context.state === 'closed')) {
|
|
6455
|
+
throw new TypeError('Prepared playback requires a live audio output from this AI.');
|
|
6456
|
+
}
|
|
6445
6457
|
// One playback lane per AI; detached preparation has its own lifetime.
|
|
6446
6458
|
this.stopAudio();
|
|
6447
|
-
const group = {
|
|
6459
|
+
const group = {
|
|
6460
|
+
jobs: [], context: output?.context ?? null, output,
|
|
6461
|
+
paused: false, stopped: false, state: 'waiting', error: null
|
|
6462
|
+
};
|
|
6463
|
+
this.#activePreparedSpeechPlayback = group;
|
|
6448
6464
|
const completions = [];
|
|
6449
6465
|
const generation = this.speechGeneration;
|
|
6450
6466
|
this.#traceSpeech('playPreparedTTS.call', {segments: prepared.segments});
|
|
@@ -6464,8 +6480,16 @@ class AI {
|
|
|
6464
6480
|
function stopPreparedPlayback() {
|
|
6465
6481
|
if(group.stopped) return false;
|
|
6466
6482
|
group.stopped = true;
|
|
6467
|
-
runtime
|
|
6468
|
-
|
|
6483
|
+
if(runtime.#activePreparedSpeechPlayback === group) {
|
|
6484
|
+
runtime.#activePreparedSpeechPlayback = null;
|
|
6485
|
+
runtime.speechResumeAttempt += 1;
|
|
6486
|
+
runtime.speechResumePending = false;
|
|
6487
|
+
if(runtime.#speechUnlockContext === group.context) runtime.#clearSpeechUnlock();
|
|
6488
|
+
if(runtime.speechScheduleContext === group.context) {
|
|
6489
|
+
runtime.speechScheduleContext = null;
|
|
6490
|
+
runtime.speechScheduleTime = 0;
|
|
6491
|
+
}
|
|
6492
|
+
}
|
|
6469
6493
|
publishPreparedPlaybackState('stopped');
|
|
6470
6494
|
for(const job of group.jobs) {
|
|
6471
6495
|
if(job.state === 'complete') continue;
|
|
@@ -6481,13 +6505,10 @@ class AI {
|
|
|
6481
6505
|
}
|
|
6482
6506
|
runtime.#cancelSpeechJob(job);
|
|
6483
6507
|
}
|
|
6484
|
-
if(runtime.speechScheduleContext === group.context) {
|
|
6485
|
-
runtime.speechScheduleContext = null;
|
|
6486
|
-
runtime.speechScheduleTime = 0;
|
|
6487
|
-
}
|
|
6488
6508
|
runtime.#requestSpeechPlayback();
|
|
6489
6509
|
return true;
|
|
6490
6510
|
}
|
|
6511
|
+
group.stop = stopPreparedPlayback;
|
|
6491
6512
|
const control = {
|
|
6492
6513
|
finished: null,
|
|
6493
6514
|
get state() { return group.state; },
|
|
@@ -6496,9 +6517,23 @@ class AI {
|
|
|
6496
6517
|
if(group.stopped || !group.context || group.context.state === 'closed') return false;
|
|
6497
6518
|
group.paused = true;
|
|
6498
6519
|
if(runtime.#speechUnlockContext === group.context) runtime.#clearSpeechUnlock();
|
|
6499
|
-
|
|
6500
|
-
|
|
6501
|
-
|
|
6520
|
+
let suspension;
|
|
6521
|
+
try {
|
|
6522
|
+
suspension = Promise.resolve(group.context.suspend());
|
|
6523
|
+
if(output) output.suspending = suspension;
|
|
6524
|
+
await suspension;
|
|
6525
|
+
if(group.stopped || generation !== runtime.speechGeneration) return false;
|
|
6526
|
+
publishPreparedPlaybackState('paused');
|
|
6527
|
+
return true;
|
|
6528
|
+
} catch(error) {
|
|
6529
|
+
if(output) {
|
|
6530
|
+
output.reportError(error);
|
|
6531
|
+
if(!group.stopped) group.fail(error);
|
|
6532
|
+
}
|
|
6533
|
+
throw error;
|
|
6534
|
+
} finally {
|
|
6535
|
+
if(output?.suspending === suspension) output.suspending = null;
|
|
6536
|
+
}
|
|
6502
6537
|
},
|
|
6503
6538
|
async resume() {
|
|
6504
6539
|
if(group.stopped || !group.context || group.context.state === 'closed') return false;
|
|
@@ -6558,15 +6593,18 @@ class AI {
|
|
|
6558
6593
|
async function finishPreparedPlayback(results) {
|
|
6559
6594
|
const completed = !group.stopped && results.every(Boolean);
|
|
6560
6595
|
group.stopped = true;
|
|
6596
|
+
if(runtime.#activePreparedSpeechPlayback === group) {
|
|
6597
|
+
runtime.#activePreparedSpeechPlayback = null;
|
|
6598
|
+
if(runtime.#speechUnlockContext === group.context) runtime.#clearSpeechUnlock();
|
|
6599
|
+
if(runtime.speechScheduleContext === group.context) {
|
|
6600
|
+
runtime.speechScheduleContext = null;
|
|
6601
|
+
runtime.speechScheduleTime = 0;
|
|
6602
|
+
}
|
|
6603
|
+
}
|
|
6561
6604
|
publishPreparedPlaybackState(completed ? 'complete' : group.error ? 'error' : 'stopped');
|
|
6562
6605
|
signal?.removeEventListener('abort', stopPreparedPlayback);
|
|
6563
6606
|
runtime.#preparedSpeechPlaybacks.delete(control);
|
|
6564
|
-
if(
|
|
6565
|
-
if(runtime.speechScheduleContext === group.context) {
|
|
6566
|
-
runtime.speechScheduleContext = null;
|
|
6567
|
-
runtime.speechScheduleTime = 0;
|
|
6568
|
-
}
|
|
6569
|
-
if(group.context && group.context.state !== 'closed') {
|
|
6607
|
+
if(!output && group.context && group.context.state !== 'closed') {
|
|
6570
6608
|
try {
|
|
6571
6609
|
await group.context.close();
|
|
6572
6610
|
} catch(error) {
|
|
@@ -6577,17 +6615,25 @@ class AI {
|
|
|
6577
6615
|
return completed;
|
|
6578
6616
|
}
|
|
6579
6617
|
);
|
|
6618
|
+
group.finished = control.finished;
|
|
6580
6619
|
signal?.addEventListener('abort', stopPreparedPlayback, {once: true});
|
|
6581
6620
|
publishPreparedPlaybackState('waiting');
|
|
6621
|
+
if(group.stopped) return control;
|
|
6582
6622
|
if(signal?.aborted) {
|
|
6583
6623
|
stopPreparedPlayback();
|
|
6584
6624
|
} else if(group.jobs.length) {
|
|
6585
6625
|
// Playing saved audio does not require activating a synthesis provider.
|
|
6586
6626
|
this.muted = false;
|
|
6587
6627
|
try {
|
|
6588
|
-
|
|
6589
|
-
|
|
6590
|
-
|
|
6628
|
+
if(output) {
|
|
6629
|
+
if(output.disposing || output.disposed || output.context.state === 'closed') {
|
|
6630
|
+
throw new Error('The prepared playback audio output has been disposed.');
|
|
6631
|
+
}
|
|
6632
|
+
} else {
|
|
6633
|
+
const AudioContext = globalThis.AudioContext || globalThis.webkitAudioContext;
|
|
6634
|
+
if(!is.function(AudioContext)) throw new TypeError('Audio playback is unavailable in this browser.');
|
|
6635
|
+
group.context = new AudioContext();
|
|
6636
|
+
}
|
|
6591
6637
|
this.resumeAudio(group.context, true).catch(
|
|
6592
6638
|
function failPreparedSpeechUnlock(error) {
|
|
6593
6639
|
for(const job of group.jobs) runtime.#failSpeechJob(job, error, 'playback-resume');
|
|
@@ -7096,12 +7142,209 @@ class AI {
|
|
|
7096
7142
|
return new Blob(normalized.chunks,{type:normalized.type||this.audioType});
|
|
7097
7143
|
}
|
|
7098
7144
|
|
|
7145
|
+
prepareAudioOutput(){
|
|
7146
|
+
if(this.#audioOutput && !this.#audioOutput.disposed
|
|
7147
|
+
&& this.#audioOutput.context?.state !== 'closed') {
|
|
7148
|
+
return this.#audioOutput.handle;
|
|
7149
|
+
}
|
|
7150
|
+
const runtime = this;
|
|
7151
|
+
const output = {
|
|
7152
|
+
context: null, handle: null, error: null,
|
|
7153
|
+
disposing: false, disposed: false, suspending: null,
|
|
7154
|
+
resumeAttempt: 0, disposal: null, reportError: null, publish: null
|
|
7155
|
+
};
|
|
7156
|
+
const operationId = `${this.#events.instanceId}:audio-output:${++this.#audioOutputSequence}`;
|
|
7157
|
+
const subscriptions = new Set();
|
|
7158
|
+
function snapshot(){
|
|
7159
|
+
const contextState = output.context?.state ?? null;
|
|
7160
|
+
const state = output.disposed ? 'disposed'
|
|
7161
|
+
: output.disposing ? 'disposing'
|
|
7162
|
+
: !output.context ? 'unavailable'
|
|
7163
|
+
: contextState === 'closed' ? 'closed'
|
|
7164
|
+
: output.error?.name === 'NotAllowedError' && contextState !== 'running' ? 'gesture-required'
|
|
7165
|
+
: output.error ? 'error'
|
|
7166
|
+
: contextState === 'running' ? 'ready'
|
|
7167
|
+
: contextState;
|
|
7168
|
+
return {kind: 'web-audio', state, contextState, error: output.error};
|
|
7169
|
+
}
|
|
7170
|
+
function publish(){
|
|
7171
|
+
runtime.#events.dispatch(AI_AUDIO_OUTPUT_STATE_EVENT, snapshot(), {operationId});
|
|
7172
|
+
}
|
|
7173
|
+
output.publish = publish;
|
|
7174
|
+
function reportError(error){
|
|
7175
|
+
output.error = error;
|
|
7176
|
+
publish();
|
|
7177
|
+
if(error?.name !== 'NotAllowedError') {
|
|
7178
|
+
arcaneLogging.error('AI audio output operation failed.', error);
|
|
7179
|
+
}
|
|
7180
|
+
}
|
|
7181
|
+
output.reportError = reportError;
|
|
7182
|
+
function observeContextState(){
|
|
7183
|
+
if(output.context.state === 'running') output.error = null;
|
|
7184
|
+
publish();
|
|
7185
|
+
if(output.context.state === 'closed' && !output.disposing) {
|
|
7186
|
+
output.context.removeEventListener?.('statechange', observeContextState);
|
|
7187
|
+
for(const unsubscribe of [...subscriptions]) unsubscribe();
|
|
7188
|
+
}
|
|
7189
|
+
}
|
|
7190
|
+
function observeRejection(promise, attempt = null){
|
|
7191
|
+
// The owner retains failures even when a caller observes state only.
|
|
7192
|
+
promise.catch(function observeOutputFailure(error){
|
|
7193
|
+
if(attempt === null || (attempt === output.resumeAttempt && !output.disposing && !output.disposed)) {
|
|
7194
|
+
reportError(error);
|
|
7195
|
+
} else {
|
|
7196
|
+
arcaneLogging.error('A retired AI audio output resume failed.', error);
|
|
7197
|
+
}
|
|
7198
|
+
});
|
|
7199
|
+
return promise;
|
|
7200
|
+
}
|
|
7201
|
+
async function resumeOutput(attempt){
|
|
7202
|
+
const context = output.context;
|
|
7203
|
+
if(!context || output.disposing || output.disposed || context.state === 'closed') return false;
|
|
7204
|
+
// A pending suspend targets this exact context; resume must follow it.
|
|
7205
|
+
if(output.suspending) await output.suspending;
|
|
7206
|
+
if(output.disposing || output.disposed || attempt !== output.resumeAttempt) return false;
|
|
7207
|
+
const paused = runtime.speechJobs.some(function hasPausedOutputPlayback(job){
|
|
7208
|
+
return job.preparedPlayback?.context === context && job.preparedPlayback.paused;
|
|
7209
|
+
});
|
|
7210
|
+
if(paused) return false;
|
|
7211
|
+
await context.resume();
|
|
7212
|
+
if(output.disposing || output.disposed || attempt !== output.resumeAttempt) return false;
|
|
7213
|
+
output.error = null;
|
|
7214
|
+
publish();
|
|
7215
|
+
if(context.state !== 'running') return false;
|
|
7216
|
+
if(runtime.#speechUnlockContext === context) runtime.#clearSpeechUnlock();
|
|
7217
|
+
if(runtime.speechJobs.some(function usesResumedOutput(job){
|
|
7218
|
+
return job.audioContext === context || job.preparedPlayback?.context === context;
|
|
7219
|
+
})) runtime.#requestSpeechPlayback();
|
|
7220
|
+
return true;
|
|
7221
|
+
}
|
|
7222
|
+
async function disposeOutput(){
|
|
7223
|
+
const context = output.context;
|
|
7224
|
+
const errors = [];
|
|
7225
|
+
const groups = new Set();
|
|
7226
|
+
output.disposing = true;
|
|
7227
|
+
output.resumeAttempt += 1;
|
|
7228
|
+
publish();
|
|
7229
|
+
if(context) {
|
|
7230
|
+
if(runtime.#speechUnlockContext === context) runtime.#clearSpeechUnlock();
|
|
7231
|
+
if(runtime.#speechResumeContext === context) {
|
|
7232
|
+
runtime.speechResumeAttempt += 1;
|
|
7233
|
+
runtime.speechResumePending = false;
|
|
7234
|
+
runtime.#speechResumeContext = null;
|
|
7235
|
+
}
|
|
7236
|
+
for(const job of [...runtime.speechJobs]) {
|
|
7237
|
+
if(job.preparedPlayback?.context === context) {
|
|
7238
|
+
groups.add(job.preparedPlayback);
|
|
7239
|
+
} else if(job.audioContext === context || job.sourceNode?.context === context
|
|
7240
|
+
|| job.precedingSpeechContext === context) {
|
|
7241
|
+
if(job.sourceNode) {
|
|
7242
|
+
job.sourceNode.onended = null;
|
|
7243
|
+
if(job.sourceNode.__arcaneStarted) {
|
|
7244
|
+
try { job.sourceNode.stop(); }
|
|
7245
|
+
catch(error) { errors.push(error); }
|
|
7246
|
+
}
|
|
7247
|
+
}
|
|
7248
|
+
try { runtime.#cancelSpeechJob(job); }
|
|
7249
|
+
catch(error) { errors.push(error); }
|
|
7250
|
+
}
|
|
7251
|
+
}
|
|
7252
|
+
for(const group of groups) {
|
|
7253
|
+
try { group.stop(); }
|
|
7254
|
+
catch(error) { errors.push(error); }
|
|
7255
|
+
}
|
|
7256
|
+
if(runtime.speechScheduleContext === context) {
|
|
7257
|
+
runtime.speechScheduleContext = null;
|
|
7258
|
+
runtime.speechScheduleTime = 0;
|
|
7259
|
+
}
|
|
7260
|
+
if(runtime.audioContext === context) runtime.audioContext = null;
|
|
7261
|
+
const completions = await Promise.allSettled(
|
|
7262
|
+
[...groups].map(function finishOutputPlayback(group){ return group.finished; })
|
|
7263
|
+
);
|
|
7264
|
+
for(const completion of completions) {
|
|
7265
|
+
if(completion.status === 'rejected') errors.push(completion.reason);
|
|
7266
|
+
}
|
|
7267
|
+
try {
|
|
7268
|
+
if(context.state !== 'closed') await context.close();
|
|
7269
|
+
} catch(error) {
|
|
7270
|
+
errors.push(error);
|
|
7271
|
+
output.disposing = false;
|
|
7272
|
+
output.disposal = null;
|
|
7273
|
+
if(runtime.#audioOutput === output && runtime.audioContext === null) {
|
|
7274
|
+
runtime.audioContext = context;
|
|
7275
|
+
}
|
|
7276
|
+
throw errors.length === 1 ? errors[0] : new AggregateError(errors, 'AI audio output disposal failed.');
|
|
7277
|
+
}
|
|
7278
|
+
context.removeEventListener?.('statechange', observeContextState);
|
|
7279
|
+
}
|
|
7280
|
+
output.disposing = false;
|
|
7281
|
+
output.disposed = true;
|
|
7282
|
+
output.error = !errors.length ? null : errors.length === 1
|
|
7283
|
+
? errors[0] : new AggregateError(errors, 'AI audio output playback cleanup failed.');
|
|
7284
|
+
publish();
|
|
7285
|
+
for(const unsubscribe of [...subscriptions]) unsubscribe();
|
|
7286
|
+
runtime.#requestSpeechPlayback();
|
|
7287
|
+
if(errors.length) throw output.error;
|
|
7288
|
+
return true;
|
|
7289
|
+
}
|
|
7290
|
+
output.handle = {
|
|
7291
|
+
kind: 'web-audio',
|
|
7292
|
+
get state(){ return snapshot().state; },
|
|
7293
|
+
get contextState(){ return output.context?.state ?? null; },
|
|
7294
|
+
get error(){ return output.error; },
|
|
7295
|
+
subscribe(listener){
|
|
7296
|
+
if(!is.function(listener)) throw new TypeError('Audio output subscriber must be a function.');
|
|
7297
|
+
const remove = output.disposed || output.context?.state === 'closed' ? null : runtime.#events.subscribe(
|
|
7298
|
+
AI_AUDIO_OUTPUT_STATE_EVENT,
|
|
7299
|
+
function observeAudioOutput(occurrence){
|
|
7300
|
+
if(occurrence.operationId === operationId) listener(occurrence.detail);
|
|
7301
|
+
}
|
|
7302
|
+
);
|
|
7303
|
+
function unsubscribe(){
|
|
7304
|
+
remove?.();
|
|
7305
|
+
subscriptions.delete(unsubscribe);
|
|
7306
|
+
}
|
|
7307
|
+
if(remove) subscriptions.add(unsubscribe);
|
|
7308
|
+
try { listener(snapshot()); }
|
|
7309
|
+
catch(error) { arcaneLogging.error('AI audio output state observer failed.', error); }
|
|
7310
|
+
return unsubscribe;
|
|
7311
|
+
},
|
|
7312
|
+
resume(){
|
|
7313
|
+
const attempt = ++output.resumeAttempt;
|
|
7314
|
+
return observeRejection(resumeOutput(attempt), attempt);
|
|
7315
|
+
},
|
|
7316
|
+
dispose(){
|
|
7317
|
+
if(output.disposed) return output.disposal ?? Promise.resolve(true);
|
|
7318
|
+
if(!output.disposal) {
|
|
7319
|
+
let resolve;
|
|
7320
|
+
let reject;
|
|
7321
|
+
output.disposal = observeRejection(new Promise(function retainOutputDisposal(accept, decline){
|
|
7322
|
+
resolve = accept;
|
|
7323
|
+
reject = decline;
|
|
7324
|
+
}));
|
|
7325
|
+
disposeOutput().then(resolve, reject);
|
|
7326
|
+
}
|
|
7327
|
+
return output.disposal;
|
|
7328
|
+
}
|
|
7329
|
+
};
|
|
7330
|
+
this.#audioOutput = output;
|
|
7331
|
+
try {
|
|
7332
|
+
output.context = this.#getSpeechAudioContext();
|
|
7333
|
+
output.context.addEventListener?.('statechange', observeContextState);
|
|
7334
|
+
publish();
|
|
7335
|
+
} catch(error) {
|
|
7336
|
+
reportError(error);
|
|
7337
|
+
}
|
|
7338
|
+
return output.handle;
|
|
7339
|
+
}
|
|
7340
|
+
|
|
7099
7341
|
#getSpeechAudioContext(){
|
|
7100
7342
|
if(this.audioContext&&this.audioContext.state!=='closed'){
|
|
7101
7343
|
return this.audioContext;
|
|
7102
7344
|
}
|
|
7103
7345
|
|
|
7104
|
-
const AudioContext=
|
|
7346
|
+
const AudioContext=globalThis.AudioContext||globalThis.webkitAudioContext
|
|
7347
|
+
||window.AudioContext||window.webkitAudioContext;
|
|
7105
7348
|
|
|
7106
7349
|
if(!is.function(AudioContext)){
|
|
7107
7350
|
throw new TypeError('Audio playback is unavailable in this browser.');
|
|
@@ -7396,6 +7639,7 @@ class AI {
|
|
|
7396
7639
|
this.speechGeneration+=1;
|
|
7397
7640
|
this.speechResumeAttempt+=1;
|
|
7398
7641
|
this.speechResumePending=false;
|
|
7642
|
+
this.#speechResumeContext=null;
|
|
7399
7643
|
this.speechScheduleGeneration=this.speechGeneration;
|
|
7400
7644
|
this.speechScheduleContext=null;
|
|
7401
7645
|
this.speechScheduleTime=0;
|
|
@@ -7461,13 +7705,20 @@ class AI {
|
|
|
7461
7705
|
let attempt=0;
|
|
7462
7706
|
let context;
|
|
7463
7707
|
let preparedGroup = null;
|
|
7708
|
+
let output = null;
|
|
7464
7709
|
|
|
7465
7710
|
try{
|
|
7466
7711
|
context=audioContext||this.#getSpeechAudioContext();
|
|
7712
|
+
output = this.#audioOutput?.context === context ? this.#audioOutput : null;
|
|
7713
|
+
if(output?.disposing || output?.disposed) return false;
|
|
7467
7714
|
preparedGroup = this.speechJobs.find(function findPreparedResumeOwner(job) {
|
|
7468
7715
|
return job.preparedPlayback?.context === context;
|
|
7469
7716
|
})?.preparedPlayback ?? null;
|
|
7470
7717
|
if(preparedGroup?.paused || preparedGroup?.stopped) return false;
|
|
7718
|
+
if(output?.suspending) {
|
|
7719
|
+
await output.suspending;
|
|
7720
|
+
if(output.disposing || output.disposed || preparedGroup?.paused || preparedGroup?.stopped) return false;
|
|
7721
|
+
}
|
|
7471
7722
|
this.#traceSpeech('resumeAudio.context',{
|
|
7472
7723
|
callId,state:context.state,audioTime:context.currentTime,
|
|
7473
7724
|
sampleRate:context.sampleRate
|
|
@@ -7490,11 +7741,26 @@ class AI {
|
|
|
7490
7741
|
|
|
7491
7742
|
attempt=++this.speechResumeAttempt;
|
|
7492
7743
|
this.speechResumePending=true;
|
|
7744
|
+
this.#speechResumeContext=context;
|
|
7493
7745
|
await context.resume();
|
|
7746
|
+
if(output?.disposing || output?.disposed) return false;
|
|
7494
7747
|
if(preparedGroup?.stopped) return false;
|
|
7495
7748
|
if(preparedGroup?.paused) {
|
|
7496
|
-
|
|
7497
|
-
|
|
7749
|
+
let suspension;
|
|
7750
|
+
try {
|
|
7751
|
+
suspension = Promise.resolve(context.suspend());
|
|
7752
|
+
if(output) output.suspending = suspension;
|
|
7753
|
+
await suspension;
|
|
7754
|
+
} catch(error) {
|
|
7755
|
+
output?.reportError(error);
|
|
7756
|
+
throw error;
|
|
7757
|
+
} finally {
|
|
7758
|
+
if(output?.suspending === suspension) output.suspending = null;
|
|
7759
|
+
}
|
|
7760
|
+
if(attempt === this.speechResumeAttempt) {
|
|
7761
|
+
this.speechResumePending = false;
|
|
7762
|
+
this.#speechResumeContext = null;
|
|
7763
|
+
}
|
|
7498
7764
|
return false;
|
|
7499
7765
|
}
|
|
7500
7766
|
this.#traceSpeech('resumeAudio.resumed',{
|
|
@@ -7509,6 +7775,11 @@ class AI {
|
|
|
7509
7775
|
}
|
|
7510
7776
|
|
|
7511
7777
|
this.speechResumePending=false;
|
|
7778
|
+
this.#speechResumeContext=null;
|
|
7779
|
+
if(output) {
|
|
7780
|
+
output.error = null;
|
|
7781
|
+
output.publish();
|
|
7782
|
+
}
|
|
7512
7783
|
|
|
7513
7784
|
if(context.state==='running'){
|
|
7514
7785
|
this.#clearSpeechUnlock();
|
|
@@ -7518,6 +7789,7 @@ class AI {
|
|
|
7518
7789
|
}
|
|
7519
7790
|
}catch(error){
|
|
7520
7791
|
this.#traceSpeech('resumeAudio.error',{callId,error,state:context?.state});
|
|
7792
|
+
if(output?.disposing || output?.disposed) return false;
|
|
7521
7793
|
if(preparedGroup?.stopped || preparedGroup?.paused) return false;
|
|
7522
7794
|
if(attempt&&attempt!==this.speechResumeAttempt){
|
|
7523
7795
|
return context?.state==='running';
|
|
@@ -7525,7 +7797,9 @@ class AI {
|
|
|
7525
7797
|
|
|
7526
7798
|
if(attempt===this.speechResumeAttempt){
|
|
7527
7799
|
this.speechResumePending=false;
|
|
7800
|
+
this.#speechResumeContext=null;
|
|
7528
7801
|
}
|
|
7802
|
+
output?.reportError(error);
|
|
7529
7803
|
if(preparedGroup && error?.name !== 'NotAllowedError') {
|
|
7530
7804
|
for(const job of preparedGroup.jobs) this.#failSpeechJob(job, error, 'playback-resume');
|
|
7531
7805
|
return false;
|
|
@@ -7600,6 +7874,9 @@ class AI {
|
|
|
7600
7874
|
const playbackContext=sourceNode?.context
|
|
7601
7875
|
||audioContext
|
|
7602
7876
|
||this.#getSpeechAudioContext();
|
|
7877
|
+
// Retain the actual output before asynchronous decoding so disposal
|
|
7878
|
+
// can cancel this job without cancelling unrelated synthesis.
|
|
7879
|
+
job.audioContext=playbackContext;
|
|
7603
7880
|
const audioBlob=new Blob(audioChunks,{type:audioType});
|
|
7604
7881
|
const arrayBuffer=await audioBlob.arrayBuffer();
|
|
7605
7882
|
const audioBuffer=await playbackContext.decodeAudioData(arrayBuffer);
|
|
@@ -7683,12 +7960,17 @@ class AI {
|
|
|
7683
7960
|
const deadline=this.speechScheduleTime;
|
|
7684
7961
|
if(!context||deadline<=context.currentTime)return Promise.resolve(true);
|
|
7685
7962
|
const signal=job.abortController.signal;
|
|
7963
|
+
job.precedingSpeechContext=context;
|
|
7686
7964
|
return new Promise(function waitForScheduledSpeechGap(resolve,reject){
|
|
7687
7965
|
let timer=null;
|
|
7966
|
+
let settled=false;
|
|
7688
7967
|
function finish(value,error=null){
|
|
7968
|
+
if(settled)return;
|
|
7969
|
+
settled=true;
|
|
7689
7970
|
globalThis.clearTimeout(timer);
|
|
7690
7971
|
signal.removeEventListener('abort',cancel);
|
|
7691
7972
|
context.removeEventListener('statechange',schedule);
|
|
7973
|
+
if(job.precedingSpeechContext===context)job.precedingSpeechContext=null;
|
|
7692
7974
|
if(error)reject(error);
|
|
7693
7975
|
else resolve(value);
|
|
7694
7976
|
}
|
|
@@ -7755,7 +8037,8 @@ class AI {
|
|
|
7755
8037
|
// the next turn, including when it follows buffered audio.
|
|
7756
8038
|
if(index!==0||job.state!=='ready')break;
|
|
7757
8039
|
if(!await this.#waitForPrecedingSpeechClock(job)
|
|
7758
|
-
||job.generation!==this.speechGeneration||this.muted
|
|
8040
|
+
||job.generation!==this.speechGeneration||this.muted
|
|
8041
|
+
||['cancelled','failed'].includes(job.state)){
|
|
7759
8042
|
return this.#cancelSpeechJob(job);
|
|
7760
8043
|
}
|
|
7761
8044
|
job.state='scheduled';
|
|
@@ -7783,6 +8066,10 @@ class AI {
|
|
|
7783
8066
|
const audioContext=job.sourceNode.context||job.audioContext;
|
|
7784
8067
|
|
|
7785
8068
|
if(job.preparedPlayback?.paused) break;
|
|
8069
|
+
if(this.#audioOutput?.context === audioContext && this.#audioOutput.suspending) {
|
|
8070
|
+
this.resumeAudio(audioContext, false);
|
|
8071
|
+
return scheduled;
|
|
8072
|
+
}
|
|
7786
8073
|
|
|
7787
8074
|
if(audioContext.state!=='running'){
|
|
7788
8075
|
this.#waitForSpeechGesture(null,audioContext);
|
|
@@ -7871,6 +8158,7 @@ class AI {
|
|
|
7871
8158
|
return false;
|
|
7872
8159
|
}finally{
|
|
7873
8160
|
this.speechPlaybackStarting=false;
|
|
8161
|
+
const output = this.#audioOutput;
|
|
7874
8162
|
const nextJob=this.speechJobs.find(function findFirstUnscheduledSpeechJob(job){
|
|
7875
8163
|
return job.state!=='scheduled';
|
|
7876
8164
|
});
|
|
@@ -7879,6 +8167,10 @@ class AI {
|
|
|
7879
8167
|
!this.muted
|
|
7880
8168
|
&&!this.speechAwaitingGesture
|
|
7881
8169
|
&&!this.speechResumePending
|
|
8170
|
+
&&!this.speechJobs.some(function hasPendingOutputSuspension(job){
|
|
8171
|
+
return output?.suspending && (job.audioContext === output.context
|
|
8172
|
+
|| job.preparedPlayback?.context === output.context);
|
|
8173
|
+
})
|
|
7882
8174
|
&&!this.speechJobs.some(function hasPausedPreparedPlayback(job) {
|
|
7883
8175
|
return job.preparedPlayback?.paused;
|
|
7884
8176
|
})
|
|
@@ -42,7 +42,7 @@ function splitSpeechText(value=''){
|
|
|
42
42
|
return text.trim()?[text]:[];
|
|
43
43
|
}
|
|
44
44
|
|
|
45
|
-
function normalizeParts(parts,defaultVoice=null,defaultSpeed=1){
|
|
45
|
+
function normalizeParts(parts,defaultVoice=null,defaultSpeed=1,textFormat=null){
|
|
46
46
|
if(!is.array(parts)||!parts.length){
|
|
47
47
|
throw new TypeError('Narration text cannot be blank. The full visual content remains available.');
|
|
48
48
|
}
|
|
@@ -63,6 +63,7 @@ function normalizeParts(parts,defaultVoice=null,defaultSpeed=1){
|
|
|
63
63
|
if(!is.finite(speed)||speed<=0)throw new RangeError('Every speech segment requires a positive speech speed.');
|
|
64
64
|
return {
|
|
65
65
|
input,
|
|
66
|
+
...(textFormat==='plain'?{textFormat:'plain'}:{}),
|
|
66
67
|
pauseAfterMs,
|
|
67
68
|
role:String(candidate?.role||'narration'),
|
|
68
69
|
speed,
|
|
@@ -425,6 +426,7 @@ class SpeechPlayback{
|
|
|
425
426
|
model=null,
|
|
426
427
|
voice=null,
|
|
427
428
|
responseFormat=null,
|
|
429
|
+
textFormat=null,
|
|
428
430
|
speed=1,
|
|
429
431
|
onState=function noop(){},
|
|
430
432
|
createObjectURL,
|
|
@@ -485,6 +487,7 @@ class SpeechPlayback{
|
|
|
485
487
|
this.responseFormat=responseFormat===null||responseFormat===undefined
|
|
486
488
|
?null
|
|
487
489
|
:String(responseFormat);
|
|
490
|
+
this.textFormat=textFormat==='plain'?'plain':null;
|
|
488
491
|
this.speed=normalizedSpeed;
|
|
489
492
|
this.key='';
|
|
490
493
|
this.state='idle';
|
|
@@ -622,10 +625,13 @@ class SpeechPlayback{
|
|
|
622
625
|
}
|
|
623
626
|
|
|
624
627
|
async requestSpeech(part,signal){
|
|
625
|
-
const input=
|
|
628
|
+
const input=part.textFormat==='plain'
|
|
629
|
+
?part.input
|
|
630
|
+
:stripSpeechFormatting(part.input);
|
|
626
631
|
const preparation={speechInputPrepared:true};
|
|
627
632
|
const payload={
|
|
628
633
|
input,
|
|
634
|
+
...(part.textFormat==='plain'?{textFormat:'plain'}:{}),
|
|
629
635
|
speed:part.speed,
|
|
630
636
|
...(this.model?{model:this.model}:{}),
|
|
631
637
|
...(part.voice?{voice:part.voice}:{}),
|
|
@@ -799,6 +805,7 @@ class SpeechPlayback{
|
|
|
799
805
|
model=this.model,
|
|
800
806
|
voice=this.voice,
|
|
801
807
|
responseFormat=this.responseFormat,
|
|
808
|
+
textFormat=this.textFormat,
|
|
802
809
|
speed=this.speed,
|
|
803
810
|
autoplay=true
|
|
804
811
|
}={}){
|
|
@@ -843,10 +850,12 @@ class SpeechPlayback{
|
|
|
843
850
|
this.model=selectedModel;
|
|
844
851
|
this.voice=selectedVoice;
|
|
845
852
|
this.responseFormat=selectedResponseFormat;
|
|
853
|
+
this.textFormat=textFormat==='plain'?'plain':null;
|
|
846
854
|
normalized=normalizeParts(
|
|
847
855
|
is.function(parts)?parts():parts,
|
|
848
856
|
selectedVoice,
|
|
849
|
-
numericSpeed
|
|
857
|
+
numericSpeed,
|
|
858
|
+
this.textFormat
|
|
850
859
|
);
|
|
851
860
|
this.parts=normalized;
|
|
852
861
|
}catch(error){
|
|
@@ -1036,7 +1045,12 @@ class SpeechPlayback{
|
|
|
1036
1045
|
playback.abortControllers.delete(active.controller);
|
|
1037
1046
|
if(completed===true)return await playback.advance();
|
|
1038
1047
|
if(active.control.error)playback.fail(active.control.error);
|
|
1039
|
-
else playback.emit(
|
|
1048
|
+
else playback.emit(
|
|
1049
|
+
'ready',
|
|
1050
|
+
playback.message('stopped'),
|
|
1051
|
+
playback.key,
|
|
1052
|
+
{reason:'native-playback-stopped'}
|
|
1053
|
+
);
|
|
1040
1054
|
return false;
|
|
1041
1055
|
}catch(error){
|
|
1042
1056
|
console.error('[Arcane speech playback] Native speech failed.',error);
|