arcane-os 0.93.0 → 0.95.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +33 -0
- package/README.md +1 -1
- package/browser-runtime/ai/twin-cloud.mjs +109 -46
- package/browser-runtime/core/client.mjs +197 -0
- package/docs/reference/ai/twin-cloud.md +35 -14
- package/docs/reference/availability-and-normalization.md +1 -1
- package/docs/reference/inventory/package-api.json +96 -12
- package/docs/reference/local-ai.md +81 -5
- package/docs/reference/native-magpie.md +98 -0
- package/docs/reference/native-nemotron.md +204 -0
- package/docs/reference/native-speech.md +364 -11
- package/docs/reference/runtime-modules.md +21 -18
- package/docs/reference/sdk-api.md +152 -11
- package/package.json +6 -1
- package/runtime/arcane/modules/AI.js +114 -14
- package/runtime/arcane/modules/AIProviderRuntime.js +32 -15
- package/runtime/arcane/modules/ContinuousVoiceCapture.js +118 -21
- package/runtime/arcane/modules/NativeSpeechCapture.js +198 -0
- package/runtime/arcane/modules/NativeSpeechPlayback.js +304 -0
- package/runtime/arcane/modules/VoiceCaptureWorklet.js +62 -2
- package/schemas/arcane-app.schema.json +2 -2
- package/schemas/arcane-package.schema.json +2 -2
- package/src/core/services/speech.mjs +320 -25
- package/src/local-ai/config.mjs +12 -1
- package/src/local-ai/dev.mjs +71 -4
- package/src/local-ai/install.mjs +113 -5
- package/src/local-ai/kokoro/frontend.mjs +219 -0
- package/src/local-ai/kokoro/helper.mjs +168 -0
- package/src/local-ai/kokoro/index.mjs +645 -0
- package/src/local-ai/kokoro/native/CMakeLists.txt +45 -0
- package/src/local-ai/kokoro/native/main.cpp +249 -0
- package/src/local-ai/magpie/build.mjs +36 -0
- package/src/local-ai/magpie/helper.mjs +254 -0
- package/src/local-ai/magpie/index.mjs +488 -0
- package/src/local-ai/magpie/native/CMakeLists.txt +10 -0
- package/src/local-ai/magpie/native/main.cpp +270 -0
- package/src/local-ai/nemotron/build.mjs +53 -0
- package/src/local-ai/nemotron/helper.mjs +308 -0
- package/src/local-ai/nemotron/index.mjs +637 -0
- package/src/local-ai/nemotron/native/CMakeLists.txt +17 -0
- package/src/local-ai/nemotron/native/main.cpp +282 -0
- package/src/native/portable-layout.mjs +67 -9
|
@@ -6,8 +6,10 @@ from `arcane-os/core/speech`.
|
|
|
6
6
|
|
|
7
7
|
The host supplies an STT engine, a TTS engine, or both. Construction performs no
|
|
8
8
|
installation, model download, model loading, or inference. Core registration
|
|
9
|
-
exposes `speech.status`, `speech.
|
|
10
|
-
|
|
9
|
+
exposes `speech.status`, `speech.load`, `speech.unload`, `speech.transcribe`,
|
|
10
|
+
and `speech.synthesize`, plus the optional streaming operations described below;
|
|
11
|
+
an ordinary operation whose engine was omitted rejects with
|
|
12
|
+
`SPEECH_ENGINE_UNAVAILABLE`.
|
|
11
13
|
The optional `signal` controls the service lifetime.
|
|
12
14
|
|
|
13
15
|
```js
|
|
@@ -20,11 +22,11 @@ const core = createCoreRuntime({services: [speech]});
|
|
|
20
22
|
core.start();
|
|
21
23
|
```
|
|
22
24
|
|
|
23
|
-
This service is the shared integration layer.
|
|
24
|
-
Kokoro
|
|
25
|
-
selection, supported formats, and
|
|
26
|
-
selected engine and its host
|
|
27
|
-
independent of this native service.
|
|
25
|
+
This service is the shared integration layer. The SDK separately exports the
|
|
26
|
+
[native Kokoro engine](#native-kokoro-engine) from `arcane-os/local-ai/kokoro`.
|
|
27
|
+
Engine preparation, model resources, device selection, supported formats, and
|
|
28
|
+
native process ownership remain with the selected engine and its host
|
|
29
|
+
composition. Browser-only operation remains independent of this native service.
|
|
28
30
|
|
|
29
31
|
The SDK's [retained Whisper engine](local-whisper.md) supplies native STT.
|
|
30
32
|
Its optional [Intel NPU encoder](../guides/native-whisper-openvino.md) uses the
|
|
@@ -34,6 +36,15 @@ continues calling `speech.transcribe` without loading native libraries itself.
|
|
|
34
36
|
|
|
35
37
|
## Startup and status
|
|
36
38
|
|
|
39
|
+
A speech-only interface can construct the shared browser AI API with explicit
|
|
40
|
+
`null` for both the language-model service and model:
|
|
41
|
+
`new AI(null, 'LOCAL_SPEACH', 'LOCAL_SPEACH', null, ttsModel, sttModel)`.
|
|
42
|
+
This leaves both LLM routes unselected from construction onward; it does not
|
|
43
|
+
register or activate the historical default language provider. Select a real
|
|
44
|
+
language model later through `configureProviders`. Omitted or empty constructor
|
|
45
|
+
arguments keep their historical defaults, and `setAI` retains its existing
|
|
46
|
+
omitted/null/empty-field semantics.
|
|
47
|
+
|
|
37
48
|
`start(context)` subscribes to each configured engine and starts their loads
|
|
38
49
|
independently. It returns immediately, allowing Core and `speech.status` to
|
|
39
50
|
respond while models load. A failure in one role leaves the other role usable.
|
|
@@ -59,6 +70,30 @@ model readiness. Existing readiness consumers additionally recognize
|
|
|
59
70
|
the engine IDs `whisper.cpp` and `kokoro-onnx`; another engine ID requires the
|
|
60
71
|
corresponding consumer integration.
|
|
61
72
|
|
|
73
|
+
Each role also reports `supportsStreaming`: STT requires the configured
|
|
74
|
+
engine's `openStream()` method, and TTS requires its `synthesizeStream()` method.
|
|
75
|
+
This capability is independent of `loaded` and `available`; a facade method
|
|
76
|
+
alone does not establish support in the selected engine. Complete engine state,
|
|
77
|
+
including `defaultVoice` and `voices` when supplied, remains present.
|
|
78
|
+
|
|
79
|
+
## Explicit role activation
|
|
80
|
+
|
|
81
|
+
`speech.load({role, ...selection})` and `speech.unload({role, ...selection})`
|
|
82
|
+
route only to the selected `stt` or `tts` engine. The service separates the
|
|
83
|
+
`role` routing field and forwards the remaining selection fields unchanged,
|
|
84
|
+
with the operation's signal. Each call returns that engine's complete result.
|
|
85
|
+
An invalid role reports `SPEECH_ROLE_INVALID`; an engine without the requested
|
|
86
|
+
method reports `SPEECH_OPERATION_UNAVAILABLE`. These calls do not alter the
|
|
87
|
+
other speech role or select a browser provider.
|
|
88
|
+
|
|
89
|
+
The browser facade is `Arcane.speech.load(request, {signal, onRequest})` and
|
|
90
|
+
`Arcane.speech.unload(request, {signal, onRequest})`. Both use an unlimited Core
|
|
91
|
+
request duration (`timeoutMs: 0`). Synthesis retains its existing
|
|
92
|
+
`Arcane.speech.synthesize(request, {signal})` facade and 180-second timeout.
|
|
93
|
+
Call `client.invoke('speech.synthesize', request, {signal, timeoutMs: 0, onRequest})`
|
|
94
|
+
when the owning application explicitly needs an unlimited request duration or
|
|
95
|
+
the actual request ID. The synthesis payload is the same through either path.
|
|
96
|
+
|
|
62
97
|
## Events and subscriptions
|
|
63
98
|
|
|
64
99
|
`subscribe(listener, {replay = true, signal} = {})` immediately delivers the
|
|
@@ -78,6 +113,129 @@ that initial request is pending.
|
|
|
78
113
|
The service emits `progress: {phase: 'accepted'}` synchronously when accepting
|
|
79
114
|
a speech request. Subsequent progress is the selected engine's actual report;
|
|
80
115
|
cancelled requests publish no later progress.
|
|
116
|
+
Role load/unload acknowledgements additionally carry `operation: 'load'` or
|
|
117
|
+
`'unload'` in that progress record.
|
|
118
|
+
|
|
119
|
+
## Continuous transcription and streaming synthesis
|
|
120
|
+
|
|
121
|
+
`Arcane.speech.openStream(request, {signal, onUpdate, onProgress, onRequest})`
|
|
122
|
+
returns an owned handle immediately:
|
|
123
|
+
|
|
124
|
+
```js
|
|
125
|
+
const recognition = Arcane.speech.openStream({sampleRate: 16000}, {
|
|
126
|
+
signal,
|
|
127
|
+
onUpdate: function retainNativeSnapshot(result) {
|
|
128
|
+
// Keep the complete snapshot. The application decides when to take a turn.
|
|
129
|
+
showRecognition(result);
|
|
130
|
+
}
|
|
131
|
+
});
|
|
132
|
+
await recognition.ready;
|
|
133
|
+
await recognition.push(pcm); // Float32Array, supplied in capture order.
|
|
134
|
+
await recognition.forceEndpoint(); // Finalizes recognition, keeps the stream.
|
|
135
|
+
const complete = await recognition.finish(); // Drains and closes this stream.
|
|
136
|
+
```
|
|
137
|
+
|
|
138
|
+
The handle exposes `streamId`, `ready`, `completion`, `current()`,
|
|
139
|
+
`push(Float32Array)`, `forceEndpoint()`, `finish()`, and `cancel(reason)`.
|
|
140
|
+
`ready` resolves with the native-open acknowledgement; `completion` resolves
|
|
141
|
+
with the engine's complete terminal result or rejects with its actual failure.
|
|
142
|
+
`current()` returns the latest delivered complete snapshot. Pushes and endpoint
|
|
143
|
+
controls are ordered, and each returned promise lets the producer apply
|
|
144
|
+
backpressure without dropping captured samples. `forceEndpoint()` waits for
|
|
145
|
+
ordered update callbacks preceding its control response, so an external Reply
|
|
146
|
+
control can consume those final segments. `finish()` follows every accepted
|
|
147
|
+
push and waits for final callback delivery. An update callback must not await
|
|
148
|
+
its own `forceEndpoint()`, `finish()`, or `completion` delivery join; invoke these
|
|
149
|
+
controls outside the callback. `cancel()` targets this
|
|
150
|
+
stream, suppresses later callbacks immediately, and awaits its explicit native
|
|
151
|
+
cleanup control; it leaves the other speech role alone. Aborting the caller's
|
|
152
|
+
signal uses ordinary Core request cancellation and can settle the browser's
|
|
153
|
+
request before native cleanup finishes.
|
|
154
|
+
Creation opens no microphone: the selected capture owner supplies PCM.
|
|
155
|
+
|
|
156
|
+
Updates are the engine's unchanged snapshots, including provisional/final
|
|
157
|
+
status, words, timestamps, and speaker scope when supplied. A provisional
|
|
158
|
+
snapshot replaces the preceding provisional snapshot; it is not appended text.
|
|
159
|
+
An ASR final or silence boundary is recognition output, not a completed human
|
|
160
|
+
turn, an instruction, or approval. `forceEndpoint()` preserves the native
|
|
161
|
+
stream's speaker scope according to its engine contract. The application owns
|
|
162
|
+
conversational meaning, turn completion, tool selection, and whether to issue
|
|
163
|
+
an LLM request. The SDK starts no chat request from an update.
|
|
164
|
+
|
|
165
|
+
The underlying `speech.open` request remains pending for the entire stream,
|
|
166
|
+
with no fixed request timeout. `speech.opened` and `speech.transcription`
|
|
167
|
+
events use that request's scoped emitter, carrying `{streamId, ...metadata}`
|
|
168
|
+
and `{streamId, result}` respectively. They reach only the owning connection.
|
|
169
|
+
`speech.push`, `speech.endpoint`, `speech.finish`, and `speech.cancel` address
|
|
170
|
+
the same stream. Client close, document retirement, transport disconnect, or
|
|
171
|
+
the request signal cancels the native stream through normal Core request
|
|
172
|
+
ownership. This stream is not accepted service-lifetime background work.
|
|
173
|
+
|
|
174
|
+
`Arcane.speech.synthesizeStream(request, {signal, onAudio, onProgress,
|
|
175
|
+
onRequest})` returns a promise for the complete native terminal metadata.
|
|
176
|
+
The complete synthesis payload reaches the selected engine unchanged.
|
|
177
|
+
The required `onAudio(audio)` consumer receives every ordered native PCM contribution, preserving
|
|
178
|
+
its metadata; `audio.pcm` is a `Uint8Array`, interpreted using the supplied
|
|
179
|
+
`format`, `channels`, and `sampleRate`. Magpie supplies signed little-endian
|
|
180
|
+
16-bit mono PCM. The facade does not play audio, infer audible completion, or
|
|
181
|
+
add a voice, language, speed, or output limit. The selected playback owner
|
|
182
|
+
consumes this stream and owns its actual playback completion and interruption.
|
|
183
|
+
Give each utterance its own signal to stop precisely that utterance without
|
|
184
|
+
stopping microphone capture or unrelated synthesis.
|
|
185
|
+
|
|
186
|
+
The synthesis request has no fixed timeout. Its request-scoped `speech.audio`
|
|
187
|
+
events contain `{streamId, sequence, audio}`; `speech.progress` also carries the
|
|
188
|
+
stream correlation. The facade serializes asynchronous callbacks and sends
|
|
189
|
+
`speech.audioConsumed` for that exact stream and sequence only after `onAudio`
|
|
190
|
+
completes. The native producer awaits this acknowledgement before contributing
|
|
191
|
+
the next PCM chunk; the event parser remains free to handle control messages.
|
|
192
|
+
Sequence and acknowledgement metadata stay outside the audio payload.
|
|
193
|
+
|
|
194
|
+
Aborting the utterance signal suppresses later callback delivery and sends
|
|
195
|
+
`speech.cancelSynthesis`. That scoped control releases pending audio
|
|
196
|
+
acknowledgements and joins the native synthesis task. The facade keeps the
|
|
197
|
+
original response pending and joins this cleanup control before settling;
|
|
198
|
+
ordinary cancellation rejects with the caller's reason. Callback failure
|
|
199
|
+
cancels the same owned operation and retains its complete failure together with
|
|
200
|
+
any native cleanup failure. Successful completion waits for all callback
|
|
201
|
+
delivery. A callback may abort its caller-owned signal, while its own completion
|
|
202
|
+
must not await the same synthesis promise. A transport failure remains an error,
|
|
203
|
+
not evidence of completed native cleanup. Playback completion remains with the
|
|
204
|
+
selected playback owner, independently of synthesis completion.
|
|
205
|
+
|
|
206
|
+
The Core transport uses base64 only for PCM framing: STT transfers exact
|
|
207
|
+
Float32 bit patterns as little-endian samples, and TTS transfers the engine's
|
|
208
|
+
complete PCM before decoding it back to the original sample representation.
|
|
209
|
+
Routing and callbacks stay outside transcript and synthesis payloads.
|
|
210
|
+
Existing file transcription, complete-audio synthesis, model loading, and
|
|
211
|
+
unloading methods keep their original contracts. A selected engine without the
|
|
212
|
+
requested streaming method reports `SPEECH_OPERATION_UNAVAILABLE`; the SDK
|
|
213
|
+
selects no replacement engine or capture path.
|
|
214
|
+
|
|
215
|
+
### Shared AI capture and playback
|
|
216
|
+
|
|
217
|
+
The built-in native AI adapter advertises continuous capture only when the
|
|
218
|
+
selected STT engine reports streaming support. `ai.createTranscriptionCapture`
|
|
219
|
+
uses the shared microphone owner, forwards the caller's media constraints and
|
|
220
|
+
actual sample rate, and preserves every ordered PCM sample. Its `onUpdate`,
|
|
221
|
+
`onInterim`, and `onSegment` callbacks receive the complete recognition record
|
|
222
|
+
and a second `{requestStop}` argument. A callback can call `requestStop()` and
|
|
223
|
+
return; the external owner joins `capture.stop()` or `capture.done`. Do not
|
|
224
|
+
await those joins or `forceEndpoint()` from their own recognition callback.
|
|
225
|
+
Stop drains accepted audio and final recognition; cancel joins microphone and
|
|
226
|
+
native cleanup without inventing a completed transcript. Neither action sends
|
|
227
|
+
a chat turn automatically.
|
|
228
|
+
|
|
229
|
+
For a selected streaming TTS engine, `ai.prepareTTSPlayback()` returns an inert
|
|
230
|
+
playback descriptor. Its `play()` consumes native PCM through the existing AI
|
|
231
|
+
provider lane and shared audio output. Playback stop affects only that utterance;
|
|
232
|
+
the control's `released` promise joins native cleanup. Existing full-audio
|
|
233
|
+
`fetchTTS()` remains available. `streamTTS(text, end, {languageCode})`,
|
|
234
|
+
`prepareTTSPlayback({input, languageCode, ...})`, and
|
|
235
|
+
`fetchTTS({input, languageCode, ...})` carry the caller's per-utterance language
|
|
236
|
+
to the native engine separately from the unchanged text. The SDK does not infer
|
|
237
|
+
language or create conversation replies. See the
|
|
238
|
+
[Nemotron](native-nemotron.md) and [Magpie](native-magpie.md) engine contracts.
|
|
81
239
|
|
|
82
240
|
## Engine interface
|
|
83
241
|
|
|
@@ -86,15 +244,25 @@ Each supplied engine implements:
|
|
|
86
244
|
- `current()` returning the actual selected-model state.
|
|
87
245
|
- `subscribe(listener)` with immediate current-state replay and an unsubscribe
|
|
88
246
|
function.
|
|
89
|
-
- `load({signal})`,
|
|
90
|
-
|
|
247
|
+
- `load({signal, ...selection})`, applying that engine's explicit selection
|
|
248
|
+
contract. Independent engines may load concurrently. Kokoro without selected
|
|
249
|
+
assets resolves to its ordinary unloaded state.
|
|
250
|
+
- Optional `unload(selection)`, releasing the selected engine's model through
|
|
251
|
+
`speech.unload`; engines without this method report it unavailable.
|
|
91
252
|
- `transcribe(request, {signal, onProgress, requestId})` for STT, or
|
|
92
253
|
`synthesize(request, {signal, onProgress, requestId})` for TTS. `requestId` is
|
|
93
254
|
the actual Core request's control metadata, separate from the unchanged payload.
|
|
255
|
+
- Optional `openStream({...options, signal, onUpdate})` for continuous STT,
|
|
256
|
+
returning `{push, forceEndpoint, finish, cancel, completion, current}`.
|
|
257
|
+
Its cancellation must settle `completion` without requiring service shutdown.
|
|
258
|
+
- Optional `synthesizeStream(request, {signal, onProgress, onAudio, requestId})`
|
|
259
|
+
for streaming TTS. Each `onAudio` value contains the complete `pcm` Buffer
|
|
260
|
+
and its format metadata; the engine awaits each callback before producing the
|
|
261
|
+
next contribution and joins native cleanup before settling cancellation.
|
|
94
262
|
- `close()`, cancelling and joining its native work and releasing its resources.
|
|
95
263
|
|
|
96
|
-
The service forwards each complete request and result
|
|
97
|
-
no input trimming, content limits, model or voice substitution, transcript
|
|
264
|
+
The service forwards each complete transcription/synthesis request and result
|
|
265
|
+
unchanged. It applies no input trimming, content limits, model or voice substitution, transcript
|
|
98
266
|
rewriting, or audio conversion. The engine owns any format decoding,
|
|
99
267
|
phonemization, required model segmentation, output encoding, and supported
|
|
100
268
|
model selection. Necessary processing must preserve complete ordered input
|
|
@@ -120,6 +288,191 @@ share one shutdown promise. Actual engine cleanup failures reject with an
|
|
|
120
288
|
releases the service's event subscriptions. New work after shutdown begins
|
|
121
289
|
rejects with `CORE_CLOSING`.
|
|
122
290
|
|
|
291
|
+
## Native Kokoro engine
|
|
292
|
+
|
|
293
|
+
`createNativeKokoroRuntime` is the named and default export from
|
|
294
|
+
`arcane-os/local-ai/kokoro`. It uses the existing native ONNX owner for model
|
|
295
|
+
execution and one retained C++ helper for eSpeak phonemization and Ogg Opus
|
|
296
|
+
encoding. The Kokoro route requires no Python, Sherpa-ONNX or FFmpeg runtime.
|
|
297
|
+
|
|
298
|
+
```js
|
|
299
|
+
import {createNativeKokoroRuntime} from 'arcane-os/local-ai/kokoro';
|
|
300
|
+
import {createSpeechService} from 'arcane-os/core/speech';
|
|
301
|
+
|
|
302
|
+
const tts = createNativeKokoroRuntime({
|
|
303
|
+
onnx, // The existing Core local-ai owner's getONNXRuntime() result.
|
|
304
|
+
modelAssets, // The existing Core model-assets service.
|
|
305
|
+
runtime, // Prepared helperExecutable, espeakDataDirectory, libraryDirectory.
|
|
306
|
+
executionPreference: 'cpu',
|
|
307
|
+
sessionOptions: {executionProviders: ['cpu']}
|
|
308
|
+
});
|
|
309
|
+
const speech = createSpeechService({tts});
|
|
310
|
+
```
|
|
311
|
+
|
|
312
|
+
The constructor accepts `{onnx, modelAssets, runtime, prepare, modelId, model,
|
|
313
|
+
revision, dtype, paths, assetProjectionId, resourcePaths, sessionOptions,
|
|
314
|
+
executionPreference, executionTarget, signal, onEvent}`. At least `onnx` or
|
|
315
|
+
`prepare` must be supplied. Optional `prepare({signal})` returns `{onnx, runtime}`
|
|
316
|
+
and runs only when an activation has selected assets and either its ONNX owner
|
|
317
|
+
is absent or closed, or its helper descriptor is absent. It returns the host's
|
|
318
|
+
existing ONNX owner; it does not construct another one. Construction performs
|
|
319
|
+
no acquisition or loading.
|
|
320
|
+
The helper data path is the **parent** containing `espeak-ng-data`.
|
|
321
|
+
|
|
322
|
+
The provider ID is `kokoro-onnx`; the public `modelId` alias defaults to `kokoro`.
|
|
323
|
+
The implemented model is `onnx-community/Kokoro-82M-v1.0-ONNX`, revision
|
|
324
|
+
`1939ad2a8e416c0acfeecc08a694d14ef25f2231`, with `dtype: 'fp32'`. Another model,
|
|
325
|
+
revision or dtype reports incompatibility. The module also exports
|
|
326
|
+
`KOKORO_MODEL`, `KOKORO_REVISION`, `KOKORO_SAMPLE_RATE` (`24000`) and
|
|
327
|
+
`KOKORO_VOICES`.
|
|
328
|
+
|
|
329
|
+
### Resources and loading
|
|
330
|
+
|
|
331
|
+
Supply either prepared native `paths` or an existing completed
|
|
332
|
+
[`model-assets` projection](model-assets.md). Both mappings have
|
|
333
|
+
`{model, tokenizer, tokenizerConfig, voices: {[voiceId]: path}}`: native paths
|
|
334
|
+
identify prepared files, while `resourcePaths` identifies projection members.
|
|
335
|
+
Projection loading requires the existing `modelAssets` service. Direct-path
|
|
336
|
+
callers keep their prepared files available until engine cleanup completes.
|
|
337
|
+
The complete selection contains `onnx/model.onnx`, `tokenizer.json`,
|
|
338
|
+
`tokenizer_config.json`, and all 28 `voices/<voiceId>.bin` files from the same
|
|
339
|
+
revision. The engine does not download models or create another model store.
|
|
340
|
+
|
|
341
|
+
`load({modelId, assetProjectionId, resourcePaths, executionTarget, signal})`
|
|
342
|
+
activates that selection. Omitted selection fields retain the previous choice;
|
|
343
|
+
an omitted or undefined target retains the previous target, and `null` requests
|
|
344
|
+
automatic selection. `modelId`, when supplied, must match this engine's alias.
|
|
345
|
+
With neither constructor paths nor a projection selected, loading resolves
|
|
346
|
+
unloaded without preparing the helper. Actual readiness requires the frontend
|
|
347
|
+
resources, live helper and loaded ONNX session.
|
|
348
|
+
|
|
349
|
+
For a complete DBOPFS-to-Core projection already prepared by the application:
|
|
350
|
+
|
|
351
|
+
```js
|
|
352
|
+
await Arcane.speech.load({
|
|
353
|
+
role: 'tts',
|
|
354
|
+
modelId: 'kokoro',
|
|
355
|
+
assetProjectionId: projection.id,
|
|
356
|
+
resourcePaths,
|
|
357
|
+
executionTarget: {deviceId: 'cpu'}
|
|
358
|
+
}, {signal});
|
|
359
|
+
```
|
|
360
|
+
|
|
361
|
+
The engine retains the incoming projection before retiring a previous
|
|
362
|
+
activation, including replacement using the same projection. It releases that
|
|
363
|
+
retain only after frontend loading settles and the actual ONNX worker and
|
|
364
|
+
helper exit. The preparation owner retains its own independent release duty.
|
|
365
|
+
Cancelling a caller's `load` signal stops that caller's wait; explicit unload,
|
|
366
|
+
replacement or engine close owns cancellation of the shared activation.
|
|
367
|
+
After `localai.services.recover({runtimes: ['onnx']})` completes, a subsequent
|
|
368
|
+
explicit `speech.load` uses the composition's `prepare` callback to reacquire
|
|
369
|
+
the current ONNX owner after the previous activation's cleanup has joined.
|
|
370
|
+
|
|
371
|
+
`executionPreference` defaults to `'gpu'` and accepts the existing ONNX CPU/GPU
|
|
372
|
+
selection contract. `sessionOptions` and `executionTarget` reach that same
|
|
373
|
+
owner unchanged. Explicit providers take precedence, and an explicit physical
|
|
374
|
+
target is never retried on another device or provider. The automatic route's
|
|
375
|
+
provider attempts and possible CPU selection remain visible in `execution`.
|
|
376
|
+
STT keeps its own independent device selection. See
|
|
377
|
+
[native ONNX execution targets](local-ai.md#physical-execution-targets);
|
|
378
|
+
accepted configuration and actual physical execution are separate facts.
|
|
379
|
+
|
|
380
|
+
### Synthesis, voices and state
|
|
381
|
+
|
|
382
|
+
`synthesize(request, {signal, onProgress, requestId} = {})` accepts
|
|
383
|
+
`{input, model?, voice?, speed?, responseFormat?}`. `input` is the complete
|
|
384
|
+
string; optional `model` must match the alias. Defaults are voice `af_heart`,
|
|
385
|
+
speed `1`, and format `wav`. Speed must be a positive finite float32 value.
|
|
386
|
+
|
|
387
|
+
| English voice set | Exact supported IDs |
|
|
388
|
+
| --- | --- |
|
|
389
|
+
| US female | `af_heart`, `af_alloy`, `af_aoede`, `af_bella`, `af_jessica`, `af_kore`, `af_nicole`, `af_nova`, `af_river`, `af_sarah`, `af_sky` |
|
|
390
|
+
| US male | `am_adam`, `am_echo`, `am_eric`, `am_fenrir`, `am_liam`, `am_michael`, `am_onyx`, `am_puck`, `am_santa` |
|
|
391
|
+
| GB female | `bf_emma`, `bf_isabella`, `bf_alice`, `bf_lily` |
|
|
392
|
+
| GB male | `bm_george`, `bm_lewis`, `bm_daniel`, `bm_fable` |
|
|
393
|
+
|
|
394
|
+
The selected voice determines US or GB phonemization. Unsupported voices,
|
|
395
|
+
formats and emitted phonemes report their actual errors; there is no silent
|
|
396
|
+
voice substitution or phoneme omission. An embedded NUL or unpaired Unicode
|
|
397
|
+
surrogate reports the public eSpeak interface's inability to consume that
|
|
398
|
+
complete string. The engine preserves source text, consumes all returned
|
|
399
|
+
clauses and divides model tokens into ordered segments according to the model's
|
|
400
|
+
context. Each segment's complete waveform contributes to the final audio.
|
|
401
|
+
|
|
402
|
+
The result is `{audioBase64, contentType, sampleRate: 24000, channels: 1,
|
|
403
|
+
model, voice, speed}`. `wav` returns IEEE float32 WAV with `audio/wav`;
|
|
404
|
+
`opus` and `ogg` both return finalized Ogg Opus with
|
|
405
|
+
`audio/ogg; codecs=opus`. The result contains the complete encoded audio rather
|
|
406
|
+
than a playback side effect. Synthesis requires a ready activation and proceeds
|
|
407
|
+
in FIFO order. Progress reports the actual `phonemizing`, `synthesizing` and
|
|
408
|
+
`encoding` phases; segment progress is not streamed audio.
|
|
409
|
+
|
|
410
|
+
`current()` returns the selected model/provider, `state`, `loaded`, `busy`,
|
|
411
|
+
active `requestId` (or `null` when omitted), `queuedRequests`, `pendingActivation`, `execution`,
|
|
412
|
+
complete `progress` and serialized `error`, plus `defaultVoice` and `voices`.
|
|
413
|
+
`subscribe(listener, {replay = true, signal} = {})` returns an unsubscribe
|
|
414
|
+
function and replays that current state. Through Core, it appears in
|
|
415
|
+
`speech.status().roles.tts` and `speech.state`.
|
|
416
|
+
|
|
417
|
+
Cancelling a queued synthesis removes only that request. Cancelling active
|
|
418
|
+
synthesis retires its native session/helper and joins their actual exit before
|
|
419
|
+
settlement. Surviving queued requests retain their complete input and order;
|
|
420
|
+
the next request reopens the same activation with the accepted provider options
|
|
421
|
+
and the resolved device when known. It does not replay the cancelled synthesis or search for
|
|
422
|
+
another provider. With no queued work, explicitly `load()` before new synthesis.
|
|
423
|
+
Native ONNX cancellation can wait for an in-flight native call to return.
|
|
424
|
+
|
|
425
|
+
`unload()` cancels the activation and its queued work, joins native cleanup and
|
|
426
|
+
releases retained model assets. `close()` also disposes engine subscriptions;
|
|
427
|
+
both preserve the host's shared ONNX owner and other engines. Cleanup errors
|
|
428
|
+
remain complete and observable. The Core equivalent is
|
|
429
|
+
`Arcane.speech.unload({role: 'tts'})`.
|
|
430
|
+
|
|
431
|
+
### Native helper production and availability
|
|
432
|
+
|
|
433
|
+
Windows x64 has a selected SDK helper archive, `arcane-kokoro-windows-x64.tar.gz`.
|
|
434
|
+
[Runtime preparation](local-ai.md#native-kokoro-selection) resolves it from the
|
|
435
|
+
installed SDK version's release. Linux and macOS use the shared JavaScript
|
|
436
|
+
engine and native CMake source with an explicitly prepared, platform-matching
|
|
437
|
+
complete helper tree or archive. Automatic Linux/macOS helper distributions
|
|
438
|
+
are not provided; execution on those hosts remains unverified.
|
|
439
|
+
|
|
440
|
+
SDK selected-output verification on October 7, 2026 used Node `26.7.0` and
|
|
441
|
+
ONNX Runtime Node `1.30.0` with explicit CPU selection for the selected Windows
|
|
442
|
+
helper and FP32 model. It covered US `af_heart` and GB `bf_emma` synthesis to WAV and Ogg Opus,
|
|
443
|
+
including queued/active cancellation and a surviving queued request. This
|
|
444
|
+
establishes that selected CPU output boundary, not GPU execution, other voices'
|
|
445
|
+
pronunciation, listening acceptance or consumer application behavior.
|
|
446
|
+
The subsequent Core-recovery reacquisition correction received source review
|
|
447
|
+
only and was not part of those execution runs.
|
|
448
|
+
|
|
449
|
+
The repository's [`tools/build-kokoro-runtime-windows.ps1`](https://github.com/TheWizardNexus/arcane-os-sdk/blob/main/tools/build-kokoro-runtime-windows.ps1)
|
|
450
|
+
builds from prepared `espeak-ng`, `opus` and `libopusenc` source directories.
|
|
451
|
+
Its selected inputs are the `csukuangfj/espeak-ng` fork at
|
|
452
|
+
`ed530aa113046142eb5115cf2fc9157854d0ffe1` with its matching UCD and language data,
|
|
453
|
+
libopus `1.6.1`, and libopusenc `0.3`. It uses installed CMake and Visual Studio
|
|
454
|
+
2022 Build Tools, defaulting to MSVC `14.44.35207` and Windows SDK
|
|
455
|
+
`10.0.26100.0`; it acquires no dependencies or model files.
|
|
456
|
+
|
|
457
|
+
```powershell
|
|
458
|
+
& ./tools/build-kokoro-runtime-windows.ps1 `
|
|
459
|
+
-SourceRoot ./.arcane/kokoro-source `
|
|
460
|
+
-OutputDirectory ./.arcane/kokoro-build `
|
|
461
|
+
-Component espeak
|
|
462
|
+
```
|
|
463
|
+
|
|
464
|
+
Run `espeak`, `opus` and `opusenc` components independently, concurrently when
|
|
465
|
+
desired, using the same output directory. After all three complete, run
|
|
466
|
+
`helper`, then `stage`. Each component retains complete command/result logs;
|
|
467
|
+
do not run the same component concurrently in that output. The output must
|
|
468
|
+
belong to `ProjectDirectory` and be separate from the prepared source tree.
|
|
469
|
+
The staged helper, matching complete data, corresponding source and included
|
|
470
|
+
terms travel together. The recipe statically links its selected libraries and
|
|
471
|
+
retains their upstream notices alongside the SDK license. Model/voice assets
|
|
472
|
+
and the separately selected ONNX runtime remain outside the helper package.
|
|
473
|
+
This reusable recipe was source-reviewed; its publication does not claim a
|
|
474
|
+
fresh execution of every component or a build on another operating system.
|
|
475
|
+
|
|
123
476
|
## AI built-in native speech readiness
|
|
124
477
|
|
|
125
478
|
The `AI` module's `LOCAL_SPEACH` adapter observes `speech.state` before reading
|
|
@@ -210,30 +210,33 @@ Built-in cloud chat decodes an HTTP error body once as JSON or text and rejects
|
|
|
210
210
|
with that complete value unchanged. It does not reconstruct an Error, replace
|
|
211
211
|
the message, or add `providerMessage`, `status`, or an SDK failure code to the
|
|
212
212
|
provider body. Network errors pass through after the retry policy below;
|
|
213
|
-
decoding errors pass through immediately. Cancellation
|
|
213
|
+
successful-response decoding errors pass through immediately. Cancellation
|
|
214
214
|
retains the existing `ARCANE_AI_REQUEST_ABORTED` contract.
|
|
215
215
|
|
|
216
|
-
|
|
217
|
-
|
|
218
|
-
|
|
219
|
-
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
|
|
224
|
-
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
|
|
228
|
-
|
|
229
|
-
|
|
230
|
-
|
|
216
|
+
Rejected Fetch calls and all HTTP `429`/`529` responses share three retries.
|
|
217
|
+
Quota messages need no `overload` keyword. A readable `Retry-After` controls a
|
|
218
|
+
`429` wait; otherwise its upcoming retry number selects `3000`, `6000`, or
|
|
219
|
+
`12000` milliseconds. Fetch rejection and `529` retain `3000` milliseconds.
|
|
220
|
+
After a `429`, active requests to the same endpoint using the same credential
|
|
221
|
+
share its cooldown and dispatch three seconds apart; other endpoints and
|
|
222
|
+
credentials remain independent. The group is released when its last HTTP
|
|
223
|
+
operation settles. See [TWiN recovery](ai/twin-cloud.md#shared-request-behavior)
|
|
224
|
+
for the scope and lifetime.
|
|
225
|
+
|
|
226
|
+
Every attempt uses the same destination, headers, complete serialized body,
|
|
227
|
+
and signal. `onRequest` runs once. Cancellation stops waits and prevents another
|
|
228
|
+
attempt. Recovery precedes successful-body consumption, so partial streams and
|
|
229
|
+
tool callbacks are never replayed. Complete errors, including provider
|
|
230
|
+
`status_code` and nested records, pass through unchanged on exhaustion.
|
|
231
|
+
Diagnostic-body reading failures for `429`/`529` participate in recovery;
|
|
232
|
+
other HTTP errors, successful-body reads, decoding, and callbacks do not.
|
|
231
233
|
|
|
232
234
|
`fetchRequest()` and `streamRequest()` accept
|
|
233
235
|
`onRetry({phase,attempt,delayMs,status,error})`. The observer receives
|
|
234
236
|
`phase:'waiting'` before the abortable delay and `phase:'requesting'` immediately
|
|
235
|
-
before the next Fetch. `attempt` is the one-based upcoming retry number
|
|
236
|
-
|
|
237
|
+
before the next Fetch. `attempt` is the one-based upcoming retry number, or zero
|
|
238
|
+
for an initial request waiting behind a shared quota cooldown. `delayMs` is
|
|
239
|
+
the initially selected wait; shared pacing can extend it. `status` is the HTTP status or `null`
|
|
237
240
|
for rejected Fetch; `error` is the complete original failure value. The SDK
|
|
238
241
|
invokes this observational callback synchronously and observes a returned
|
|
239
242
|
promise without awaiting it. Callback throws and rejections are logged through
|