arcane-os 0.94.0 → 0.96.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +36 -0
- package/README.md +1 -1
- package/browser-runtime/ai/core-model-assets.mjs +74 -0
- package/browser-runtime/core/client.mjs +195 -0
- package/docs/reference/README.md +9 -0
- package/docs/reference/ai/browser-speech.md +61 -10
- package/docs/reference/core-client.md +35 -0
- package/docs/reference/core-native-packaging.md +38 -6
- package/docs/reference/inventory/package-api.json +133 -5
- package/docs/reference/inventory/runtime-modules.json +39 -13
- package/docs/reference/model-assets.md +164 -3
- package/docs/reference/native-magpie.md +98 -0
- package/docs/reference/native-nemotron.md +204 -0
- package/docs/reference/native-speech.md +250 -5
- package/docs/reference/runtime-modules.md +157 -11
- package/docs/reference/sdk-api.md +247 -4
- package/package.json +7 -3
- package/runtime/arcane/modules/AI.js +114 -14
- package/runtime/arcane/modules/AIProviderRuntime.js +32 -15
- package/runtime/arcane/modules/ContinuousVoiceCapture.js +118 -21
- package/runtime/arcane/modules/NativeSpeechCapture.js +198 -0
- package/runtime/arcane/modules/NativeSpeechPlayback.js +304 -0
- package/runtime/arcane/modules/VoiceCaptureWorklet.js +62 -2
- package/src/core/host.mjs +1 -0
- package/src/core/services/local-ai.mjs +46 -12
- package/src/core/services/model-assets.mjs +491 -59
- package/src/core/services/speech.mjs +237 -4
- package/src/local-ai/magpie/build.mjs +36 -0
- package/src/local-ai/magpie/helper.mjs +254 -0
- package/src/local-ai/magpie/index.mjs +488 -0
- package/src/local-ai/magpie/native/CMakeLists.txt +10 -0
- package/src/local-ai/magpie/native/main.cpp +270 -0
- package/src/local-ai/nemo-models.mjs +160 -0
- package/src/local-ai/nemotron/build.mjs +53 -0
- package/src/local-ai/nemotron/helper.mjs +308 -0
- package/src/local-ai/nemotron/index.mjs +693 -0
- package/src/local-ai/nemotron/native/CMakeLists.txt +17 -0
- package/src/local-ai/nemotron/native/main.cpp +282 -0
- package/src/native/macos-provider.mjs +14 -7
- package/src/native/portable-assembly.mjs +180 -0
- package/src/native/portable-provider.mjs +17 -161
- package/src/native/publish-output.mjs +101 -0
- package/src/native/windows-provider.mjs +12 -6
|
@@ -0,0 +1,204 @@
|
|
|
1
|
+
# Native Nemotron streaming transcription
|
|
2
|
+
|
|
3
|
+
`arcane-os/local-ai/nemotron` provides `createNemotronRuntime`, a Node/Core
|
|
4
|
+
adapter for NVIDIA's public NeMo Speech ASR C ABI. One owned helper retains the
|
|
5
|
+
selected recognizer and optional diarization model. One active audio stream
|
|
6
|
+
belongs to that engine at a time. Existing Whisper transcription and standalone
|
|
7
|
+
diarization remain separate supported capabilities.
|
|
8
|
+
|
|
9
|
+
This adapter has no microphone, playback, conversational endpoint detector,
|
|
10
|
+
keyword classifier, cloud fallback or automatic engine replacement. The host
|
|
11
|
+
owns capture and the model owns conversational interpretation. A final ASR
|
|
12
|
+
segment does not establish that the person has finished thinking, yielded the
|
|
13
|
+
conversation, approved an action or issued an instruction.
|
|
14
|
+
|
|
15
|
+
## Preparation and selection
|
|
16
|
+
|
|
17
|
+
Select an upstream NeMo Speech runtime with its public ASR headers, library,
|
|
18
|
+
runtime dependencies and CMake package configuration. This implementation was
|
|
19
|
+
source-reviewed against the installed NVIDIA NeMo-Speech.cpp 0.2.0 public ABI,
|
|
20
|
+
`nemotron-3.5-asr-streaming-0.6b` and `Nemotron-3-Diarization`. Acquire runtime and
|
|
21
|
+
model assets through their upstream distribution; the SDK adapter does not
|
|
22
|
+
download or redistribute them.
|
|
23
|
+
|
|
24
|
+
`arcane-os/local-ai/nemotron/build` exports `buildNemotronHelper`:
|
|
25
|
+
|
|
26
|
+
```js
|
|
27
|
+
import {buildNemotronHelper} from 'arcane-os/local-ai/nemotron/build';
|
|
28
|
+
import {createNemotronRuntime} from 'arcane-os/local-ai/nemotron';
|
|
29
|
+
|
|
30
|
+
const built = await buildNemotronHelper(
|
|
31
|
+
{
|
|
32
|
+
runtime: {cmakeDirectory: selectedNeMoCmakeDirectory},
|
|
33
|
+
outputRoot: selectedHelperOutputDirectory,
|
|
34
|
+
signal,
|
|
35
|
+
onEvent
|
|
36
|
+
}
|
|
37
|
+
);
|
|
38
|
+
const engine = createNemotronRuntime(
|
|
39
|
+
{
|
|
40
|
+
runtime: {
|
|
41
|
+
helperExecutable: built.helperExecutable,
|
|
42
|
+
binaryDirectory: selectedNeMoBinaryDirectory,
|
|
43
|
+
libraryDirectory: selectedNeMoLibraryDirectory
|
|
44
|
+
},
|
|
45
|
+
modelPath: selectedAsrModelPath,
|
|
46
|
+
diarizationModelPath: selectedDiarizationModelPath,
|
|
47
|
+
gpu: selectedNativeGpuOrdinal,
|
|
48
|
+
onEvent
|
|
49
|
+
}
|
|
50
|
+
);
|
|
51
|
+
await engine.load();
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
The builder requires CMake 3.26 or later and a platform C++17 toolchain. It
|
|
55
|
+
configures, builds and installs only the first-party helper, returning
|
|
56
|
+
`helperExecutable`, its `executable` alias, `root`, `platform`, `architecture`
|
|
57
|
+
and complete configure/build/install diagnostics. The caller selects the output
|
|
58
|
+
directory and supplies the appropriate installed upstream runtime. Windows
|
|
59
|
+
loads its runtime through the child's `PATH`; Linux uses `LD_LIBRARY_PATH` and
|
|
60
|
+
macOS uses `DYLD_LIBRARY_PATH`, alongside the helper's loader-relative paths.
|
|
61
|
+
`runtime.env` supplies additional child-only environment settings.
|
|
62
|
+
|
|
63
|
+
`gpu` is required: supply the native runtime's selected GPU ordinal or explicitly
|
|
64
|
+
select `-1` for CPU. Device selection belongs to the host. When the host changes
|
|
65
|
+
`CUDA_VISIBLE_DEVICES`, the ordinal must describe that child's visible devices.
|
|
66
|
+
The adapter reports `requestedGpu`; `observedBackend` remains `null` because a
|
|
67
|
+
requested device alone does not prove GPU inference.
|
|
68
|
+
|
|
69
|
+
Instead of `runtime`, supply an asynchronous
|
|
70
|
+
`prepare({signal, onEvent})` function returning that prepared descriptor.
|
|
71
|
+
Preparation runs during `load`, never at construction. `modelPath`,
|
|
72
|
+
`diarizationModelPath` and `gpu` may also be supplied by the returned descriptor.
|
|
73
|
+
Omit `diarizationModelPath` when speaker labels are not requested.
|
|
74
|
+
|
|
75
|
+
## Engine lifecycle
|
|
76
|
+
|
|
77
|
+
- `current()` returns selected provider/model, lifecycle `state`, `loaded`,
|
|
78
|
+
`busy`, native version, requested device, active speaker scope, errors and any
|
|
79
|
+
explicit reset requirement.
|
|
80
|
+
- `subscribe(listener, {replay: true, signal} = {})` immediately replays current
|
|
81
|
+
state and observes subsequent lifecycle events; it returns an unsubscribe
|
|
82
|
+
function.
|
|
83
|
+
- `load({modelId, modelPath, diarizationModelPath, gpu, signal} = {})` loads the
|
|
84
|
+
selected models and resolves only when native readiness is reported.
|
|
85
|
+
`modelId` defaults to `nemotron-3.5-asr-streaming-0.6b`. Concurrent callers share
|
|
86
|
+
that load; one caller's signal cancels its wait, not the shared load. Unload
|
|
87
|
+
before replacing an active selection.
|
|
88
|
+
- `unload()` cancels and joins the active stream and model preparation, closes
|
|
89
|
+
the owned helper, and retains selection for a later explicit load.
|
|
90
|
+
- `close()` joins unload and disposes lifecycle subscriptions.
|
|
91
|
+
|
|
92
|
+
The host can compose this engine into the existing
|
|
93
|
+
[`createSpeechService`](./native-speech.md) STT engine seam. Readiness follows
|
|
94
|
+
the selected engine, not an application startup barrier.
|
|
95
|
+
|
|
96
|
+
## Ordered streaming
|
|
97
|
+
|
|
98
|
+
```js
|
|
99
|
+
const stream = await engine.openStream(
|
|
100
|
+
{
|
|
101
|
+
model: 'nemotron-3.5-asr-streaming-0.6b',
|
|
102
|
+
sampleRate: actualCaptureSampleRate,
|
|
103
|
+
signal,
|
|
104
|
+
onUpdate: async function showRecognition(result) {
|
|
105
|
+
await displayCompleteSnapshot(result);
|
|
106
|
+
}
|
|
107
|
+
}
|
|
108
|
+
);
|
|
109
|
+
await stream.push(monoFloat32Samples);
|
|
110
|
+
await stream.forceEndpoint();
|
|
111
|
+
// The same stream accepts the next utterance, preserving its speaker scope.
|
|
112
|
+
await stream.push(nextMonoFloat32Samples);
|
|
113
|
+
await stream.finish();
|
|
114
|
+
await stream.completion;
|
|
115
|
+
```
|
|
116
|
+
|
|
117
|
+
`openStream({sampleRate = 16000, languageCode, model, onUpdate, signal} = {})`
|
|
118
|
+
requires the selected model to be ready, or waits for its already-started load.
|
|
119
|
+
It never silently reloads after a reset. An explicit `model` must match the
|
|
120
|
+
loaded selection. `sampleRate` is the actual input rate: an integer from 8000
|
|
121
|
+
through 96000, or `0` for the model rate. It is passed unchanged to the native
|
|
122
|
+
API. The transport's ordered 100 ms scheduling does not reinterpret a 48 kHz
|
|
123
|
+
recording as 16 kHz or discard audio.
|
|
124
|
+
|
|
125
|
+
The returned handle contains `push`, `forceEndpoint`, `finish`, `cancel`,
|
|
126
|
+
`completion`, `current`, `sampleRate` and `speakerScope`:
|
|
127
|
+
|
|
128
|
+
- `push(audio)` accepts mono `Float32Array` or
|
|
129
|
+
`{encoding: 'f32le', data: BASE64}` and resolves to all ordered snapshots from
|
|
130
|
+
that input. The encoded form carries exact little-endian float32 samples.
|
|
131
|
+
Keep a supplied array unchanged until its push settles. Other encoded
|
|
132
|
+
recordings produce `NEMOTRON_AUDIO_FORMAT_UNSUPPORTED`; caller-owned decoding
|
|
133
|
+
is required.
|
|
134
|
+
- `forceEndpoint()` drains the current ASR utterance while retaining the native
|
|
135
|
+
stream, recognizer and diarization scope. Capture closure and conversational
|
|
136
|
+
turn completion remain the host's decisions.
|
|
137
|
+
- `finish()` queues behind all accepted input, terminally finishes and drains
|
|
138
|
+
the native stream, delivers remaining snapshots, and resolves with that final
|
|
139
|
+
drain's snapshot array. The loaded engine remains available for another
|
|
140
|
+
stream.
|
|
141
|
+
- `current()` returns the latest delivered snapshot, or `null` before one is
|
|
142
|
+
available.
|
|
143
|
+
- `completion` settles after terminal finish or joined cancellation. Successful
|
|
144
|
+
finish resolves to the final drain's snapshots; earlier snapshots have already
|
|
145
|
+
been delivered through `onUpdate` and each operation's result.
|
|
146
|
+
|
|
147
|
+
Each snapshot includes `sequence`, `final`, `audioProcessedSeconds`,
|
|
148
|
+
`channelTag`, `speakerScope`, and every native `alternatives` record with its
|
|
149
|
+
exact `text`, `confidence`, `languageCodes` and `words`. Every word preserves
|
|
150
|
+
`word`, `startTimeMs`, `endTimeMs`, `confidence` and numeric `speaker`. Top-level
|
|
151
|
+
`text` and `words` expose the first alternative without changing its content.
|
|
152
|
+
|
|
153
|
+
Partial snapshots are provisional replacements, not appended transcript text.
|
|
154
|
+
Native endpoint finals describe individual utterances in a continuing stream;
|
|
155
|
+
they are forwarded unchanged. Keep their boundaries distinct. Speaker numbers
|
|
156
|
+
distinguish voices within the reported scope, not named people or biometric
|
|
157
|
+
identity. A new stream has a new scope. Timestamp values retain the native
|
|
158
|
+
runtime's units and origin without an invented offset.
|
|
159
|
+
|
|
160
|
+
Callbacks run serially in native result order and may be asynchronous. A
|
|
161
|
+
callback failure stops the stream, remains observable, and retains complete
|
|
162
|
+
native result records in diagnostics. Inside an active `onUpdate` callback,
|
|
163
|
+
calling `push`, `forceEndpoint`, `finish`, `cancel`, `unload` or `close` raises
|
|
164
|
+
`NEMOTRON_CALLBACK_WAIT`: those joined operations would wait for the callback
|
|
165
|
+
itself. Abort the caller-owned signal synchronously if cancellation is needed
|
|
166
|
+
there, return from the callback, then join through the external owner.
|
|
167
|
+
|
|
168
|
+
## Cancellation and complete recordings
|
|
169
|
+
|
|
170
|
+
`cancel(reason)` and an aborted stream signal first request ordinary native
|
|
171
|
+
stream closure. If a noninterruptible public-ABI call is in flight, cancellation
|
|
172
|
+
terminates only this engine's owned helper and joins it. It also joins accepted
|
|
173
|
+
JavaScript operations and output callbacks before releasing stream ownership.
|
|
174
|
+
Explicit cancellation resolves `completion` and `cancel` with
|
|
175
|
+
`{cancelled: true, resetRequired, speakerScope, diagnostics?}`. Actual native,
|
|
176
|
+
callback or cleanup failures reject and retain their diagnostic causes.
|
|
177
|
+
|
|
178
|
+
After helper termination, `current()` reports `resetRequired: true`, a reset
|
|
179
|
+
record and `state: 'unloaded'`. The next caller must explicitly `load()`;
|
|
180
|
+
recognizer replacement and speaker-scope reset are never silent. Ordinary
|
|
181
|
+
stream closure keeps the loaded helper. No cancellation claims that unfinished
|
|
182
|
+
recognition produced a complete final.
|
|
183
|
+
|
|
184
|
+
`transcribe(request, {signal, onProgress, onUpdate} = {})` uses the same stream
|
|
185
|
+
implementation for complete recordings. Supply `{audio, sampleRate, model?,
|
|
186
|
+
languageCode?}` or a direct supported audio value at the default sample rate.
|
|
187
|
+
It returns `{text, words, results, speakerScope}`, retaining all final records
|
|
188
|
+
in `results`, and emits accepted/recognizing progress. Its final text is the
|
|
189
|
+
ordered concatenation of native final text without inserted separators. It
|
|
190
|
+
does not decode WAV, MP3 or other encoded recording formats.
|
|
191
|
+
|
|
192
|
+
## Evidence boundary
|
|
193
|
+
|
|
194
|
+
The adapter, transport and platform paths have been source-reviewed. On October
|
|
195
|
+
17, 2026, the selected Windows x64 helper configured, compiled and installed
|
|
196
|
+
successfully against installed NeMo-Speech.cpp 0.2.0 using CMake 4.4.3, Visual
|
|
197
|
+
Studio 2022, MSVC 19.44.35217 and Windows SDK 10.0.26100. Each operation exited 0;
|
|
198
|
+
complete diagnostics remain in the SDK's ignored selected-output directory. The
|
|
199
|
+
helper was not launched, and there is no measured latency, GPU-inference,
|
|
200
|
+
microphone, diarization, audio playback or multi-utterance acceptance result for
|
|
201
|
+
this adapter. Linux and macOS compilation and execution remain unverified.
|
|
202
|
+
Compilation, header review and upstream documentation remain distinct from a
|
|
203
|
+
built native application's observed behavior. Android requires the necessary
|
|
204
|
+
host integration and a matching native runtime.
|
|
@@ -1,13 +1,14 @@
|
|
|
1
1
|
# Native speech service
|
|
2
2
|
|
|
3
|
-
`createSpeechService({stt, tts, signal} = {})` composes independently selected
|
|
3
|
+
`createSpeechService({stt, tts, signal, autoLoad = true} = {})` composes independently selected
|
|
4
4
|
native speech engines into one Core service. It is the named and default export
|
|
5
5
|
from `arcane-os/core/speech`.
|
|
6
6
|
|
|
7
7
|
The host supplies an STT engine, a TTS engine, or both. Construction performs no
|
|
8
8
|
installation, model download, model loading, or inference. Core registration
|
|
9
9
|
exposes `speech.status`, `speech.load`, `speech.unload`, `speech.transcribe`,
|
|
10
|
-
and `speech.synthesize
|
|
10
|
+
and `speech.synthesize`, plus the optional streaming operations described below;
|
|
11
|
+
an ordinary operation whose engine was omitted rejects with
|
|
11
12
|
`SPEECH_ENGINE_UNAVAILABLE`.
|
|
12
13
|
The optional `signal` controls the service lifetime.
|
|
13
14
|
|
|
@@ -35,11 +36,27 @@ continues calling `speech.transcribe` without loading native libraries itself.
|
|
|
35
36
|
|
|
36
37
|
## Startup and status
|
|
37
38
|
|
|
39
|
+
A speech-only interface can construct the shared browser AI API with explicit
|
|
40
|
+
`null` for both the language-model service and model:
|
|
41
|
+
`new AI(null, 'LOCAL_SPEACH', 'LOCAL_SPEACH', null, ttsModel, sttModel)`.
|
|
42
|
+
This leaves both LLM routes unselected from construction onward; it does not
|
|
43
|
+
register or activate the historical default language provider. Select a real
|
|
44
|
+
language model later through `configureProviders`. Omitted or empty constructor
|
|
45
|
+
arguments keep their historical defaults, and `setAI` retains its existing
|
|
46
|
+
omitted/null/empty-field semantics.
|
|
47
|
+
|
|
38
48
|
`start(context)` subscribes to each configured engine and starts their loads
|
|
39
49
|
independently. It returns immediately, allowing Core and `speech.status` to
|
|
40
50
|
respond while models load. A failure in one role leaves the other role usable.
|
|
41
51
|
Each engine publishes the actual state of its selected model.
|
|
42
52
|
|
|
53
|
+
Use `autoLoad: false` when the host first needs the user to select or download
|
|
54
|
+
model files. Startup still subscribes and exposes the ordinary unloaded state;
|
|
55
|
+
it does not call either engine's `load()`. Explicit `speech.load({role,
|
|
56
|
+
...selection})` activates the prepared role afterward. The default remains
|
|
57
|
+
`true` for existing hosts. This option changes service startup, not the selected
|
|
58
|
+
engine's missing-model errors or inference readiness requirement.
|
|
59
|
+
|
|
43
60
|
`current()` and `speech.status` return the same snapshot:
|
|
44
61
|
|
|
45
62
|
| Field | Meaning |
|
|
@@ -56,9 +73,16 @@ An engine's state includes `providerId`, `modelId`, `state`, `loaded`, and
|
|
|
56
73
|
`busy`, with its complete error or progress information when present. An
|
|
57
74
|
omitted role has `state: 'unavailable'`, `loaded: false`, and `busy: false`.
|
|
58
75
|
File presence, a spawned process, and a callable bridge do not establish loaded
|
|
59
|
-
model readiness.
|
|
60
|
-
|
|
61
|
-
|
|
76
|
+
model readiness. The shared AI adapter follows the selected engine identity
|
|
77
|
+
and model in these per-role snapshots, including Nemotron and Magpie. Only the
|
|
78
|
+
older aggregate-only fallback retains fixed `whisper.cpp`/`whisper-small` and
|
|
79
|
+
`kokoro-onnx`/`kokoro` expectations.
|
|
80
|
+
|
|
81
|
+
Each role also reports `supportsStreaming`: STT requires the configured
|
|
82
|
+
engine's `openStream()` method, and TTS requires its `synthesizeStream()` method.
|
|
83
|
+
This capability is independent of `loaded` and `available`; a facade method
|
|
84
|
+
alone does not establish support in the selected engine. Complete engine state,
|
|
85
|
+
including `defaultVoice` and `voices` when supplied, remains present.
|
|
62
86
|
|
|
63
87
|
## Explicit role activation
|
|
64
88
|
|
|
@@ -100,6 +124,127 @@ cancelled requests publish no later progress.
|
|
|
100
124
|
Role load/unload acknowledgements additionally carry `operation: 'load'` or
|
|
101
125
|
`'unload'` in that progress record.
|
|
102
126
|
|
|
127
|
+
## Continuous transcription and streaming synthesis
|
|
128
|
+
|
|
129
|
+
`Arcane.speech.openStream(request, {signal, onUpdate, onProgress, onRequest})`
|
|
130
|
+
returns an owned handle immediately:
|
|
131
|
+
|
|
132
|
+
```js
|
|
133
|
+
const recognition = Arcane.speech.openStream({sampleRate: 16000}, {
|
|
134
|
+
signal,
|
|
135
|
+
onUpdate: function retainNativeSnapshot(result) {
|
|
136
|
+
// Keep the complete snapshot. The application decides when to take a turn.
|
|
137
|
+
showRecognition(result);
|
|
138
|
+
}
|
|
139
|
+
});
|
|
140
|
+
await recognition.ready;
|
|
141
|
+
await recognition.push(pcm); // Float32Array, supplied in capture order.
|
|
142
|
+
await recognition.forceEndpoint(); // Finalizes recognition, keeps the stream.
|
|
143
|
+
const complete = await recognition.finish(); // Drains and closes this stream.
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
The handle exposes `streamId`, `ready`, `completion`, `current()`,
|
|
147
|
+
`push(Float32Array)`, `forceEndpoint()`, `finish()`, and `cancel(reason)`.
|
|
148
|
+
`ready` resolves with the native-open acknowledgement; `completion` resolves
|
|
149
|
+
with the engine's complete terminal result or rejects with its actual failure.
|
|
150
|
+
`current()` returns the latest delivered complete snapshot. Pushes and endpoint
|
|
151
|
+
controls are ordered, and each returned promise lets the producer apply
|
|
152
|
+
backpressure without dropping captured samples. `forceEndpoint()` waits for
|
|
153
|
+
ordered update callbacks preceding its control response, so an external Reply
|
|
154
|
+
control can consume those final segments. `finish()` follows every accepted
|
|
155
|
+
push and waits for final callback delivery. An update callback must not await
|
|
156
|
+
its own `forceEndpoint()`, `finish()`, or `completion` delivery join; invoke these
|
|
157
|
+
controls outside the callback. `cancel()` targets this
|
|
158
|
+
stream, suppresses later callbacks immediately, and awaits its explicit native
|
|
159
|
+
cleanup control; it leaves the other speech role alone. Aborting the caller's
|
|
160
|
+
signal uses ordinary Core request cancellation and can settle the browser's
|
|
161
|
+
request before native cleanup finishes.
|
|
162
|
+
Creation opens no microphone: the selected capture owner supplies PCM.
|
|
163
|
+
|
|
164
|
+
Updates are the engine's unchanged snapshots, including provisional/final
|
|
165
|
+
status, words, timestamps, and speaker scope when supplied. A provisional
|
|
166
|
+
snapshot replaces the preceding provisional snapshot; it is not appended text.
|
|
167
|
+
An ASR final or silence boundary is recognition output, not a completed human
|
|
168
|
+
turn, an instruction, or approval. `forceEndpoint()` preserves the native
|
|
169
|
+
stream's speaker scope according to its engine contract. The application owns
|
|
170
|
+
conversational meaning, turn completion, tool selection, and whether to issue
|
|
171
|
+
an LLM request. The SDK starts no chat request from an update.
|
|
172
|
+
|
|
173
|
+
The underlying `speech.open` request remains pending for the entire stream,
|
|
174
|
+
with no fixed request timeout. `speech.opened` and `speech.transcription`
|
|
175
|
+
events use that request's scoped emitter, carrying `{streamId, ...metadata}`
|
|
176
|
+
and `{streamId, result}` respectively. They reach only the owning connection.
|
|
177
|
+
`speech.push`, `speech.endpoint`, `speech.finish`, and `speech.cancel` address
|
|
178
|
+
the same stream. Client close, document retirement, transport disconnect, or
|
|
179
|
+
the request signal cancels the native stream through normal Core request
|
|
180
|
+
ownership. This stream is not accepted service-lifetime background work.
|
|
181
|
+
|
|
182
|
+
`Arcane.speech.synthesizeStream(request, {signal, onAudio, onProgress,
|
|
183
|
+
onRequest})` returns a promise for the complete native terminal metadata.
|
|
184
|
+
The complete synthesis payload reaches the selected engine unchanged.
|
|
185
|
+
The required `onAudio(audio)` consumer receives every ordered native PCM contribution, preserving
|
|
186
|
+
its metadata; `audio.pcm` is a `Uint8Array`, interpreted using the supplied
|
|
187
|
+
`format`, `channels`, and `sampleRate`. Magpie supplies signed little-endian
|
|
188
|
+
16-bit mono PCM. The facade does not play audio, infer audible completion, or
|
|
189
|
+
add a voice, language, speed, or output limit. The selected playback owner
|
|
190
|
+
consumes this stream and owns its actual playback completion and interruption.
|
|
191
|
+
Give each utterance its own signal to stop precisely that utterance without
|
|
192
|
+
stopping microphone capture or unrelated synthesis.
|
|
193
|
+
|
|
194
|
+
The synthesis request has no fixed timeout. Its request-scoped `speech.audio`
|
|
195
|
+
events contain `{streamId, sequence, audio}`; `speech.progress` also carries the
|
|
196
|
+
stream correlation. The facade serializes asynchronous callbacks and sends
|
|
197
|
+
`speech.audioConsumed` for that exact stream and sequence only after `onAudio`
|
|
198
|
+
completes. The native producer awaits this acknowledgement before contributing
|
|
199
|
+
the next PCM chunk; the event parser remains free to handle control messages.
|
|
200
|
+
Sequence and acknowledgement metadata stay outside the audio payload.
|
|
201
|
+
|
|
202
|
+
Aborting the utterance signal suppresses later callback delivery and sends
|
|
203
|
+
`speech.cancelSynthesis`. That scoped control releases pending audio
|
|
204
|
+
acknowledgements and joins the native synthesis task. The facade keeps the
|
|
205
|
+
original response pending and joins this cleanup control before settling;
|
|
206
|
+
ordinary cancellation rejects with the caller's reason. Callback failure
|
|
207
|
+
cancels the same owned operation and retains its complete failure together with
|
|
208
|
+
any native cleanup failure. Successful completion waits for all callback
|
|
209
|
+
delivery. A callback may abort its caller-owned signal, while its own completion
|
|
210
|
+
must not await the same synthesis promise. A transport failure remains an error,
|
|
211
|
+
not evidence of completed native cleanup. Playback completion remains with the
|
|
212
|
+
selected playback owner, independently of synthesis completion.
|
|
213
|
+
|
|
214
|
+
The Core transport uses base64 only for PCM framing: STT transfers exact
|
|
215
|
+
Float32 bit patterns as little-endian samples, and TTS transfers the engine's
|
|
216
|
+
complete PCM before decoding it back to the original sample representation.
|
|
217
|
+
Routing and callbacks stay outside transcript and synthesis payloads.
|
|
218
|
+
Existing file transcription, complete-audio synthesis, model loading, and
|
|
219
|
+
unloading methods keep their original contracts. A selected engine without the
|
|
220
|
+
requested streaming method reports `SPEECH_OPERATION_UNAVAILABLE`; the SDK
|
|
221
|
+
selects no replacement engine or capture path.
|
|
222
|
+
|
|
223
|
+
### Shared AI capture and playback
|
|
224
|
+
|
|
225
|
+
The built-in native AI adapter advertises continuous capture only when the
|
|
226
|
+
selected STT engine reports streaming support. `ai.createTranscriptionCapture`
|
|
227
|
+
uses the shared microphone owner, forwards the caller's media constraints and
|
|
228
|
+
actual sample rate, and preserves every ordered PCM sample. Its `onUpdate`,
|
|
229
|
+
`onInterim`, and `onSegment` callbacks receive the complete recognition record
|
|
230
|
+
and a second `{requestStop}` argument. A callback can call `requestStop()` and
|
|
231
|
+
return; the external owner joins `capture.stop()` or `capture.done`. Do not
|
|
232
|
+
await those joins or `forceEndpoint()` from their own recognition callback.
|
|
233
|
+
Stop drains accepted audio and final recognition; cancel joins microphone and
|
|
234
|
+
native cleanup without inventing a completed transcript. Neither action sends
|
|
235
|
+
a chat turn automatically.
|
|
236
|
+
|
|
237
|
+
For a selected streaming TTS engine, `ai.prepareTTSPlayback()` returns an inert
|
|
238
|
+
playback descriptor. Its `play()` consumes native PCM through the existing AI
|
|
239
|
+
provider lane and shared audio output. Playback stop affects only that utterance;
|
|
240
|
+
the control's `released` promise joins native cleanup. Existing full-audio
|
|
241
|
+
`fetchTTS()` remains available. `streamTTS(text, end, {languageCode})`,
|
|
242
|
+
`prepareTTSPlayback({input, languageCode, ...})`, and
|
|
243
|
+
`fetchTTS({input, languageCode, ...})` carry the caller's per-utterance language
|
|
244
|
+
to the native engine separately from the unchanged text. The SDK does not infer
|
|
245
|
+
language or create conversation replies. See the
|
|
246
|
+
[Nemotron](native-nemotron.md) and [Magpie](native-magpie.md) engine contracts.
|
|
247
|
+
|
|
103
248
|
## Engine interface
|
|
104
249
|
|
|
105
250
|
Each supplied engine implements:
|
|
@@ -115,6 +260,13 @@ Each supplied engine implements:
|
|
|
115
260
|
- `transcribe(request, {signal, onProgress, requestId})` for STT, or
|
|
116
261
|
`synthesize(request, {signal, onProgress, requestId})` for TTS. `requestId` is
|
|
117
262
|
the actual Core request's control metadata, separate from the unchanged payload.
|
|
263
|
+
- Optional `openStream({...options, signal, onUpdate})` for continuous STT,
|
|
264
|
+
returning `{push, forceEndpoint, finish, cancel, completion, current}`.
|
|
265
|
+
Its cancellation must settle `completion` without requiring service shutdown.
|
|
266
|
+
- Optional `synthesizeStream(request, {signal, onProgress, onAudio, requestId})`
|
|
267
|
+
for streaming TTS. Each `onAudio` value contains the complete `pcm` Buffer
|
|
268
|
+
and its format metadata; the engine awaits each callback before producing the
|
|
269
|
+
next contribution and joins native cleanup before settling cancellation.
|
|
118
270
|
- `close()`, cancelling and joining its native work and releasing its resources.
|
|
119
271
|
|
|
120
272
|
The service forwards each complete transcription/synthesis request and result
|
|
@@ -144,6 +296,99 @@ share one shutdown promise. Actual engine cleanup failures reject with an
|
|
|
144
296
|
releases the service's event subscriptions. New work after shutdown begins
|
|
145
297
|
rejects with `CORE_CLOSING`.
|
|
146
298
|
|
|
299
|
+
## Selected NeMo model acquisition
|
|
300
|
+
|
|
301
|
+
`NEMO_SPEECH_MODELS`, `prepareNemotronModels`, and `prepareMagpieModels` are
|
|
302
|
+
named exports from `arcane-os/local-ai/nemotron`. Importing them performs no
|
|
303
|
+
network access, installation, model loading, or inference. The catalog records
|
|
304
|
+
NVIDIA's selected [NeMo-Speech.cpp v0.2.0 model index](https://github.com/NVIDIA/NeMo-Speech.cpp/blob/v0.2.0/models/index.json):
|
|
305
|
+
|
|
306
|
+
| Role | Complete default selection |
|
|
307
|
+
|---|---|
|
|
308
|
+
| `stt` | Nemotron 3.5 streaming ASR Q8_0 and Nemotron 3 Diarization Q8_0. |
|
|
309
|
+
| `tts` | Magpie multilingual v2607 F16, NanoCodec 22 kHz F16 decoder, and all ten matching tokenizer/configuration/speaker-map members from the same Magpie revision. |
|
|
310
|
+
|
|
311
|
+
Each role exposes `{modelId, providerId, name, members, resourcePaths}`.
|
|
312
|
+
`members` contains ordered `{path, url}` file sources. TTS also exposes
|
|
313
|
+
`archives`, with the upstream uncompressed `.nemo` TAR source, its selected
|
|
314
|
+
transport ranges, and exact `{path, member}` tokenizer entries. The shared
|
|
315
|
+
[model-assets service](model-assets.md) owns discovery, acquisition, progress,
|
|
316
|
+
cancellation, persistence, and projection lifetime; these helpers do not create
|
|
317
|
+
another downloader or store. Tokenizer files retain their complete original
|
|
318
|
+
contents. The selected tokenizer configuration and its referenced JSON speaker
|
|
319
|
+
map continue to own voice IDs.
|
|
320
|
+
|
|
321
|
+
```js
|
|
322
|
+
import {createNemotronRuntime, prepareNemotronModels} from 'arcane-os/local-ai/nemotron';
|
|
323
|
+
import {createSpeechService} from 'arcane-os/core/speech';
|
|
324
|
+
|
|
325
|
+
const stt = createNemotronRuntime({runtime, modelAssets, gpu: -1});
|
|
326
|
+
const speech = createSpeechService({stt, autoLoad: false});
|
|
327
|
+
// Run this only for the user's explicit model preparation action.
|
|
328
|
+
const prepared = await prepareNemotronModels({
|
|
329
|
+
modelAssets,
|
|
330
|
+
workingDirectory: modelsDirectory,
|
|
331
|
+
directories: [modelsDirectory, appModelsDirectory],
|
|
332
|
+
signal,
|
|
333
|
+
onProgress: showModelProgress
|
|
334
|
+
});
|
|
335
|
+
try {
|
|
336
|
+
await stt.load({
|
|
337
|
+
assetProjectionId: prepared.assetProjectionId,
|
|
338
|
+
resourcePaths: prepared.resourcePaths,
|
|
339
|
+
signal
|
|
340
|
+
});
|
|
341
|
+
} finally {
|
|
342
|
+
await prepared.release();
|
|
343
|
+
}
|
|
344
|
+
```
|
|
345
|
+
|
|
346
|
+
Both preparation helpers accept `{modelAssets, workingDirectory, directories,
|
|
347
|
+
id, signal, onProgress, refresh = false}`. With `directories`, they ask the
|
|
348
|
+
shared owner for the first complete matching catalog selection in those roots
|
|
349
|
+
before downloading. `openExisting()` borrows its original files and never
|
|
350
|
+
deletes them. Incomplete selections stay visible through the owner's discovery
|
|
351
|
+
results and are not mistaken for a complete model. When no complete match is
|
|
352
|
+
available, the existing acquisition path prepares the complete selected set;
|
|
353
|
+
its completed downloads remain reusable. Explicit `refresh: true` requests a
|
|
354
|
+
fresh acquisition. The host chooses the roots: its portable native launch
|
|
355
|
+
context supplies `modelsDirectory` under the resolved per-user `stateRoot` by
|
|
356
|
+
default, while explicit directory overrides remain supported. An application
|
|
357
|
+
may include its own `models` directory among the discovery roots.
|
|
358
|
+
|
|
359
|
+
The result is `{assetProjectionId, directory, members, resourcePaths, release}`.
|
|
360
|
+
`release()` is idempotent and releases preparation ownership. Each engine
|
|
361
|
+
retains the projection synchronously when `load()` accepts it, so the caller
|
|
362
|
+
may release its preparation after starting that load. Nemotron accepts
|
|
363
|
+
`modelAssets`, `assetProjectionId`, and `resourcePaths` in its constructor and
|
|
364
|
+
`assetProjectionId`/`resourcePaths` in `load()`, with roles `{model, diarization}`.
|
|
365
|
+
Magpie uses `{model, codec, tokenizerConfig}`. Native file paths still work
|
|
366
|
+
through the engines' existing options. Nemotron requires unloading before
|
|
367
|
+
changing an active selection; its retained files are released only after
|
|
368
|
+
native work and helper exit have joined. Load failures also join asset cleanup,
|
|
369
|
+
and cleanup failures remain observable.
|
|
370
|
+
|
|
371
|
+
`onProgress` receives ordered, awaited shared-owner phase and file-completion
|
|
372
|
+
events, including reuse and ready for installed selections. Signal cancellation
|
|
373
|
+
stops that preparation's work; an abort after projection creation releases it
|
|
374
|
+
before rejecting. Engine load cancellation remains separate: cancelling the
|
|
375
|
+
caller's wait does not cancel a shared activation; explicit engine unload/close
|
|
376
|
+
owns that cancellation.
|
|
377
|
+
|
|
378
|
+
The default GGUF selections are individual complete upstream files. The shared
|
|
379
|
+
owner also discovers ordered GGUF shard sets and reports their missing members;
|
|
380
|
+
HTTP range transfers and tokenizer TAR ranges are transport mechanisms, not
|
|
381
|
+
additional model shards. A discovered shard set alone does not establish that a
|
|
382
|
+
selected native engine can load that model. The engine remains the authority
|
|
383
|
+
for model compatibility and actual readiness.
|
|
384
|
+
|
|
385
|
+
The [upstream native runtime](https://github.com/NVIDIA/NeMo-Speech.cpp/blob/v0.2.0/docs/install.md)
|
|
386
|
+
provides Windows, Linux, and macOS distributions. Model files and that runtime
|
|
387
|
+
remain separate upstream assets, not vendored SDK payloads. This acquisition
|
|
388
|
+
integration received source review only; no download, inference, or new platform
|
|
389
|
+
execution is claimed here. See the [Nemotron](native-nemotron.md) and
|
|
390
|
+
[Magpie](native-magpie.md) contracts for native preparation and execution evidence.
|
|
391
|
+
|
|
147
392
|
## Native Kokoro engine
|
|
148
393
|
|
|
149
394
|
`createNativeKokoroRuntime` is the named and default export from
|