arcane-os 0.94.0 → 0.96.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +36 -0
- package/README.md +1 -1
- package/browser-runtime/ai/core-model-assets.mjs +74 -0
- package/browser-runtime/core/client.mjs +195 -0
- package/docs/reference/README.md +9 -0
- package/docs/reference/ai/browser-speech.md +61 -10
- package/docs/reference/core-client.md +35 -0
- package/docs/reference/core-native-packaging.md +38 -6
- package/docs/reference/inventory/package-api.json +133 -5
- package/docs/reference/inventory/runtime-modules.json +39 -13
- package/docs/reference/model-assets.md +164 -3
- package/docs/reference/native-magpie.md +98 -0
- package/docs/reference/native-nemotron.md +204 -0
- package/docs/reference/native-speech.md +250 -5
- package/docs/reference/runtime-modules.md +157 -11
- package/docs/reference/sdk-api.md +247 -4
- package/package.json +7 -3
- package/runtime/arcane/modules/AI.js +114 -14
- package/runtime/arcane/modules/AIProviderRuntime.js +32 -15
- package/runtime/arcane/modules/ContinuousVoiceCapture.js +118 -21
- package/runtime/arcane/modules/NativeSpeechCapture.js +198 -0
- package/runtime/arcane/modules/NativeSpeechPlayback.js +304 -0
- package/runtime/arcane/modules/VoiceCaptureWorklet.js +62 -2
- package/src/core/host.mjs +1 -0
- package/src/core/services/local-ai.mjs +46 -12
- package/src/core/services/model-assets.mjs +491 -59
- package/src/core/services/speech.mjs +237 -4
- package/src/local-ai/magpie/build.mjs +36 -0
- package/src/local-ai/magpie/helper.mjs +254 -0
- package/src/local-ai/magpie/index.mjs +488 -0
- package/src/local-ai/magpie/native/CMakeLists.txt +10 -0
- package/src/local-ai/magpie/native/main.cpp +270 -0
- package/src/local-ai/nemo-models.mjs +160 -0
- package/src/local-ai/nemotron/build.mjs +53 -0
- package/src/local-ai/nemotron/helper.mjs +308 -0
- package/src/local-ai/nemotron/index.mjs +693 -0
- package/src/local-ai/nemotron/native/CMakeLists.txt +17 -0
- package/src/local-ai/nemotron/native/main.cpp +282 -0
- package/src/native/macos-provider.mjs +14 -7
- package/src/native/portable-assembly.mjs +180 -0
- package/src/native/portable-provider.mjs +17 -161
- package/src/native/publish-output.mjs +101 -0
- package/src/native/windows-provider.mjs +12 -6
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,41 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.96.0
|
|
4
|
+
|
|
5
|
+
- Publish successful portable, Windows and macOS native builds to one stable
|
|
6
|
+
app-specific output directory. Assemble separately and retain the previous
|
|
7
|
+
output through promotion, callback acceptance and recoverable rollback;
|
|
8
|
+
keep model caches and user profiles outside replacement.
|
|
9
|
+
- Add persistent native model acquisition and installed-model discovery through
|
|
10
|
+
the shared Core model-assets owner. Preserve complete split GGUF sets and
|
|
11
|
+
companion files, stream selected TAR members, report file-based progress,
|
|
12
|
+
and join cancellation and cleanup while retaining completed originals.
|
|
13
|
+
- Expose upstream Nemotron/diarization and Magpie/NanoCodec model selections
|
|
14
|
+
through the existing native speech API. Nemotron retains selected resource
|
|
15
|
+
projections until its owned helper closes; no model or runtime redistribution
|
|
16
|
+
is added.
|
|
17
|
+
- Add explicit native language-model startup and an app-scoped Models directory.
|
|
18
|
+
Preserve existing automatic startup defaults and ordinary speech APIs.
|
|
19
|
+
Package delivery remains distinct from native inference or app acceptance.
|
|
20
|
+
|
|
21
|
+
## 0.95.0
|
|
22
|
+
|
|
23
|
+
- Add retained native Nemotron streaming recognition and optional diarization,
|
|
24
|
+
with ordered complete snapshots, explicit utterance endpoints, speaker scope,
|
|
25
|
+
and joined cancellation. Preserve existing Whisper and complete transcription.
|
|
26
|
+
- Add native Magpie/NanoCodec synthesis with model-owned voice selection,
|
|
27
|
+
per-request language, ordered PCM streaming, complete WAV/PCM export, and
|
|
28
|
+
exact-utterance cancellation. Preserve authored input and natural model rate;
|
|
29
|
+
the caller supplies upstream runtime and model assets without SDK vendoring.
|
|
30
|
+
- Extend shared Core speech and browser clients with correlated streaming
|
|
31
|
+
capture/output, consumed-audio acknowledgements and native cleanup joins.
|
|
32
|
+
Connect AI playback through its existing queues and shared AudioContext,
|
|
33
|
+
retaining per-utterance pause/resume/stop and complete-audio APIs.
|
|
34
|
+
- Keep capture ownership, recognition segments and model-owned conversational
|
|
35
|
+
decisions distinct. Windows x64 native helpers compiled and installed against
|
|
36
|
+
the approved NeMo 0.2.0 runtime; inference, microphone/audio acceptance and
|
|
37
|
+
other-platform execution remain separate evidence boundaries.
|
|
38
|
+
|
|
3
39
|
## 0.94.0
|
|
4
40
|
|
|
5
41
|
- Recover all TWiN HTTP 429 quota responses through the shared chat/System One
|
package/README.md
CHANGED
|
@@ -25,7 +25,7 @@ selected SDK projection or an explicit live SDK source mount. Both profiles
|
|
|
25
25
|
share the theme, packaging, event, cancellation,
|
|
26
26
|
and browser run contracts while retaining their selected application layout.
|
|
27
27
|
|
|
28
|
-
This checkout defines the `0.
|
|
28
|
+
This checkout defines the `0.96.0` SDK contract. Development applications declare
|
|
29
29
|
`arcane-os` as `latest` and retain their lockfile. Exact SDK pins begin when Roshi
|
|
30
30
|
designates a production build. Follow [Consumer updates](docs/consumer-updates.md)
|
|
31
31
|
to refresh and compare the installed SDK and lockfile with the published channel;
|
|
@@ -3,6 +3,80 @@ import {getInstalledCoreClient} from '../core/client.mjs';
|
|
|
3
3
|
|
|
4
4
|
const is = new Is(false);
|
|
5
5
|
|
|
6
|
+
/** Inspect selected native directories without downloading or loading a model. */
|
|
7
|
+
export function discoverCoreModelAssets({client = getInstalledCoreClient(), directories, catalog, recursive, signal} = {}) {
|
|
8
|
+
requireCore(client);
|
|
9
|
+
return client.invoke('modelAssets.discover', {directories, catalog, recursive}, {signal, timeoutMs: 0});
|
|
10
|
+
}
|
|
11
|
+
|
|
12
|
+
/** Retain existing complete files without copying them into an application build. */
|
|
13
|
+
export function openExistingCoreModelAssets({client = getInstalledCoreClient(), directory, members, signal} = {}) {
|
|
14
|
+
return prepareNativeProjection(client, 'modelAssets.openExisting', {directory, members}, {signal});
|
|
15
|
+
}
|
|
16
|
+
|
|
17
|
+
/** Explicitly download a selected complete native model and its companions. */
|
|
18
|
+
export function downloadCoreModelAssets({
|
|
19
|
+
client = getInstalledCoreClient(), workingDirectory, members, archives, refresh = false, signal, onProgress
|
|
20
|
+
} = {}) {
|
|
21
|
+
return prepareNativeProjection(client, 'modelAssets.prepare', {workingDirectory, members, archives, refresh}, {signal, onProgress});
|
|
22
|
+
}
|
|
23
|
+
|
|
24
|
+
function requireCore(client) {
|
|
25
|
+
if (!is.function(client?.invoke)) {
|
|
26
|
+
const error = new Error('Native model assets require an available Core connection.');
|
|
27
|
+
error.code = 'ARCANE_MODEL_ASSETS_CORE_UNAVAILABLE';
|
|
28
|
+
throw error;
|
|
29
|
+
}
|
|
30
|
+
}
|
|
31
|
+
|
|
32
|
+
async function prepareNativeProjection(client, method, params, {signal, onProgress}) {
|
|
33
|
+
requireCore(client);
|
|
34
|
+
if (!is.function(client.uuid)) throw new TypeError('Model asset preparation requires Core operation ids.');
|
|
35
|
+
if (onProgress !== undefined && !is.function(onProgress)) throw new TypeError('Model asset onProgress must be a function.');
|
|
36
|
+
if (onProgress && !is.function(client.events?.on)) throw new TypeError('Model asset progress requires Core events.');
|
|
37
|
+
const id = client.uuid();
|
|
38
|
+
const controller = new AbortController();
|
|
39
|
+
const operationSignal = signal ? AbortSignal.any([signal, controller.signal]) : controller.signal;
|
|
40
|
+
operationSignal.throwIfAborted();
|
|
41
|
+
let delivery = Promise.resolve();
|
|
42
|
+
let callbackFailure;
|
|
43
|
+
let releaseTask;
|
|
44
|
+
const unsubscribe = onProgress ? client.events.on('modelAssets.progress', function receiveProgress(event) {
|
|
45
|
+
if (event?.id !== id || operationSignal.aborted) return;
|
|
46
|
+
delivery = delivery.then(function orderedProgress() {
|
|
47
|
+
if (!operationSignal.aborted) return onProgress(event.progress);
|
|
48
|
+
});
|
|
49
|
+
delivery.catch(function failedProgress(error) {
|
|
50
|
+
callbackFailure ??= error;
|
|
51
|
+
controller.abort(error);
|
|
52
|
+
});
|
|
53
|
+
}) : undefined;
|
|
54
|
+
|
|
55
|
+
function release() {
|
|
56
|
+
releaseTask ??= client.invoke('modelAssets.release', {id}, {timeoutMs: 0});
|
|
57
|
+
return releaseTask;
|
|
58
|
+
}
|
|
59
|
+
|
|
60
|
+
try {
|
|
61
|
+
const projection = await client.invoke(method, {id, ...params}, {signal: operationSignal, timeoutMs: 0});
|
|
62
|
+
await delivery;
|
|
63
|
+
operationSignal.throwIfAborted();
|
|
64
|
+
return {id, directory: projection.directory, members: projection.members, release};
|
|
65
|
+
} catch (error) {
|
|
66
|
+
const reason = callbackFailure ?? (error?.code === 'ARCANE_REQUEST_ABORTED' && operationSignal.aborted ? operationSignal.reason : error);
|
|
67
|
+
controller.abort(reason);
|
|
68
|
+
const errors = [reason];
|
|
69
|
+
try {
|
|
70
|
+
if (method === 'modelAssets.prepare') await client.invoke('modelAssets.cancelPrepare', {id}, {timeoutMs: 0});
|
|
71
|
+
else await release();
|
|
72
|
+
} catch (cleanupError) { if (!errors.includes(cleanupError)) errors.push(cleanupError); }
|
|
73
|
+
try { await delivery; }
|
|
74
|
+
catch (deliveryError) { if (!errors.includes(deliveryError)) errors.push(deliveryError); }
|
|
75
|
+
if (errors.length > 1) throw new AggregateError(errors, 'Native model preparation and cleanup failed.', {cause: reason});
|
|
76
|
+
throw reason;
|
|
77
|
+
} finally { unsubscribe?.(); }
|
|
78
|
+
}
|
|
79
|
+
|
|
6
80
|
/** Projects complete caller-owned assets through Core without changing their store. */
|
|
7
81
|
export async function prepareCoreModelAssets({
|
|
8
82
|
client = getInstalledCoreClient(), workingDirectory, members, signal, onProgress
|
|
@@ -569,6 +569,199 @@ export function createCoreClient({
|
|
|
569
569
|
export function createCoreFacade(client){
|
|
570
570
|
const {invoke,events,uuid}=client;
|
|
571
571
|
const long={timeoutMs:LONG_OPERATION_TIMEOUT};
|
|
572
|
+
function speechStreamEvents(streamId,options){
|
|
573
|
+
const controller=new AbortController();
|
|
574
|
+
const signal=options.signal?AbortSignal.any([options.signal,controller.signal]):controller.signal;
|
|
575
|
+
const subscriptions=[];
|
|
576
|
+
let delivery=Promise.resolve();
|
|
577
|
+
let callbackFailure;
|
|
578
|
+
function listen(name,callback){
|
|
579
|
+
if(callback===undefined)return;
|
|
580
|
+
if(typeof callback!=='function')throw new TypeError(`${name} observer must be a function.`);
|
|
581
|
+
subscriptions.push(events.on(name,function receiveSpeechStreamEvent(event){
|
|
582
|
+
if(event?.streamId!==streamId||signal.aborted)return;
|
|
583
|
+
delivery=delivery.then(function deliverOrderedSpeechEvent(){
|
|
584
|
+
if(!signal.aborted)return callback(event);
|
|
585
|
+
});
|
|
586
|
+
delivery.catch(function cancelFailedSpeechObserver(error){
|
|
587
|
+
callbackFailure??=error;
|
|
588
|
+
controller.abort(error);
|
|
589
|
+
});
|
|
590
|
+
}));
|
|
591
|
+
}
|
|
592
|
+
function dispose(){
|
|
593
|
+
for(const unsubscribe of subscriptions.splice(0))unsubscribe();
|
|
594
|
+
}
|
|
595
|
+
async function delivered(){
|
|
596
|
+
await delivery;
|
|
597
|
+
signal.throwIfAborted();
|
|
598
|
+
}
|
|
599
|
+
async function settle(task,{preserveTaskFailure=false}={}){
|
|
600
|
+
let result;
|
|
601
|
+
let failure;
|
|
602
|
+
let failed=false;
|
|
603
|
+
let deliveryFailure;
|
|
604
|
+
let deliveryFailed=false;
|
|
605
|
+
try{
|
|
606
|
+
try{result=await task;}catch(error){failed=true;failure=error;}
|
|
607
|
+
try{await delivery;}catch(error){deliveryFailed=true;deliveryFailure=callbackFailure??error;}
|
|
608
|
+
if(preserveTaskFailure&&failed&&deliveryFailed&&failure!==deliveryFailure){
|
|
609
|
+
throw new AggregateError([deliveryFailure,failure],'Speech callback and native cleanup failed.');
|
|
610
|
+
}
|
|
611
|
+
if(failed||deliveryFailed)throw callbackFailure??deliveryFailure??failure;
|
|
612
|
+
signal.throwIfAborted();
|
|
613
|
+
return result;
|
|
614
|
+
}finally{dispose();}
|
|
615
|
+
}
|
|
616
|
+
return {controller,signal,listen,settle,dispose,delivered};
|
|
617
|
+
}
|
|
618
|
+
function encodeSpeechSamples(samples){
|
|
619
|
+
if(!(samples instanceof Float32Array))throw new TypeError('Speech stream push requires Float32Array PCM.');
|
|
620
|
+
const encoded=new Uint8Array(samples.length*4);
|
|
621
|
+
const view=new DataView(encoded.buffer);
|
|
622
|
+
const bits=new Uint32Array(samples.buffer,samples.byteOffset,samples.length);
|
|
623
|
+
for(let index=0;index<bits.length;index+=1)view.setUint32(index*4,bits[index],true);
|
|
624
|
+
let binary='';
|
|
625
|
+
for(const value of encoded)binary+=String.fromCharCode(value);
|
|
626
|
+
return {encoding:'f32le',data:globalThis.btoa(binary)};
|
|
627
|
+
}
|
|
628
|
+
function decodeSpeechAudio(audio){
|
|
629
|
+
if(audio?.pcm?.encoding!=='base64'||typeof audio.pcm.data!=='string'){
|
|
630
|
+
throw new TypeError('Streaming speech audio requires base64 PCM transport.');
|
|
631
|
+
}
|
|
632
|
+
const binary=globalThis.atob(audio.pcm.data);
|
|
633
|
+
const pcm=new Uint8Array(binary.length);
|
|
634
|
+
for(let index=0;index<binary.length;index+=1)pcm[index]=binary.charCodeAt(index);
|
|
635
|
+
return {...audio,pcm};
|
|
636
|
+
}
|
|
637
|
+
function openSpeechStream(request={},options={}){
|
|
638
|
+
const streamId=uuid();
|
|
639
|
+
const owned=speechStreamEvents(streamId,options);
|
|
640
|
+
let resolveReady;
|
|
641
|
+
let rejectReady;
|
|
642
|
+
let opened=false;
|
|
643
|
+
let ending=false;
|
|
644
|
+
let snapshot=null;
|
|
645
|
+
let controls=Promise.resolve();
|
|
646
|
+
let finishing=null;
|
|
647
|
+
let cancelling=null;
|
|
648
|
+
const ready=new Promise(function waitForSpeechStream(resolve,reject){
|
|
649
|
+
resolveReady=resolve;
|
|
650
|
+
rejectReady=reject;
|
|
651
|
+
});
|
|
652
|
+
ready.catch(function observeSpeechStreamReadinessFailure(){});
|
|
653
|
+
try{
|
|
654
|
+
owned.listen('speech.opened',function speechStreamOpened(event){
|
|
655
|
+
opened=true;
|
|
656
|
+
resolveReady(event);
|
|
657
|
+
});
|
|
658
|
+
owned.listen('speech.transcription',function transcriptionUpdated(event){
|
|
659
|
+
snapshot=event.result;
|
|
660
|
+
return options.onUpdate?.(event.result);
|
|
661
|
+
});
|
|
662
|
+
if(options.onProgress!==undefined){
|
|
663
|
+
owned.listen('speech.progress',function speechStreamProgress(event){return options.onProgress(event);});
|
|
664
|
+
}
|
|
665
|
+
}catch(error){owned.dispose();throw error;}
|
|
666
|
+
const completion=owned.settle(invoke('speech.open',{streamId,options:request},{
|
|
667
|
+
timeoutMs:0,signal:owned.signal,onRequest:options.onRequest
|
|
668
|
+
})).then(function speechStreamCompleted(result){
|
|
669
|
+
if(!opened)throw new CoreError({code:'SPEECH_STREAM_NOT_OPENED',message:'The speech stream ended before readiness.'});
|
|
670
|
+
return result;
|
|
671
|
+
}).catch(function speechStreamFailed(error){
|
|
672
|
+
rejectReady(error);
|
|
673
|
+
throw error;
|
|
674
|
+
}).finally(function releaseSpeechStream(){ending=true;});
|
|
675
|
+
completion.catch(function observeOwnedSpeechStreamFailure(){});
|
|
676
|
+
function control(method,parameters={}){
|
|
677
|
+
const task=controls.then(async function sendOrderedSpeechControl(){
|
|
678
|
+
await ready;
|
|
679
|
+
owned.signal.throwIfAborted();
|
|
680
|
+
return invoke(method,{streamId,...parameters},{timeoutMs:0,signal:owned.signal});
|
|
681
|
+
});
|
|
682
|
+
controls=task;
|
|
683
|
+
task.catch(function cancelFailedSpeechControl(error){owned.controller.abort(error);});
|
|
684
|
+
return task;
|
|
685
|
+
}
|
|
686
|
+
return {
|
|
687
|
+
streamId,ready,completion,
|
|
688
|
+
current:function currentSpeechTranscript(){return snapshot;},
|
|
689
|
+
push:function pushSpeechSamples(samples){
|
|
690
|
+
if(ending)return Promise.reject(new CoreError({code:'SPEECH_STREAM_ENDING',message:'The speech stream is finishing.'}));
|
|
691
|
+
return control('speech.push',{audio:encodeSpeechSamples(samples)});
|
|
692
|
+
},
|
|
693
|
+
forceEndpoint:function endpointSpeechStream(){
|
|
694
|
+
if(ending)return Promise.reject(new CoreError({code:'SPEECH_STREAM_ENDING',message:'The speech stream is finishing.'}));
|
|
695
|
+
return control('speech.endpoint').then(async function deliverEndpointSnapshots(result){
|
|
696
|
+
await owned.delivered();
|
|
697
|
+
return result;
|
|
698
|
+
});
|
|
699
|
+
},
|
|
700
|
+
finish:function finishSpeechStream(){
|
|
701
|
+
if(!ending){
|
|
702
|
+
ending=true;
|
|
703
|
+
finishing=control('speech.finish').then(function finishSpeechDelivery(){return completion;});
|
|
704
|
+
finishing.catch(function observeSpeechFinishFailure(){});
|
|
705
|
+
}
|
|
706
|
+
return finishing??completion;
|
|
707
|
+
},
|
|
708
|
+
cancel:function cancelSpeechStream(reason){
|
|
709
|
+
if(cancelling)return cancelling;
|
|
710
|
+
ending=true;
|
|
711
|
+
const stopping=invoke('speech.cancel',{streamId},{timeoutMs:0});
|
|
712
|
+
owned.controller.abort(reason);
|
|
713
|
+
// A transcript callback may await cancellation. Native cleanup
|
|
714
|
+
// must not wait for that same callback through completion.
|
|
715
|
+
cancelling=stopping;
|
|
716
|
+
return cancelling;
|
|
717
|
+
}
|
|
718
|
+
};
|
|
719
|
+
}
|
|
720
|
+
function synthesizeSpeechStream(request={},options={}){
|
|
721
|
+
if(typeof options.onAudio!=='function')throw new TypeError('Streaming speech synthesis requires an onAudio consumer.');
|
|
722
|
+
const streamId=uuid();
|
|
723
|
+
const owned=speechStreamEvents(streamId,options);
|
|
724
|
+
try{
|
|
725
|
+
owned.listen('speech.audio',async function synthesisAudio(event){
|
|
726
|
+
await options.onAudio(decodeSpeechAudio(event.audio));
|
|
727
|
+
owned.signal.throwIfAborted();
|
|
728
|
+
await invoke('speech.audioConsumed',{streamId,sequence:event.sequence},{timeoutMs:0,signal:owned.signal});
|
|
729
|
+
});
|
|
730
|
+
if(options.onProgress!==undefined){
|
|
731
|
+
owned.listen('speech.progress',function synthesisProgress(event){return options.onProgress(event);});
|
|
732
|
+
}
|
|
733
|
+
}catch(error){owned.dispose();throw error;}
|
|
734
|
+
if(owned.signal.aborted)return owned.settle(Promise.reject(owned.signal.reason));
|
|
735
|
+
// Keep the response pending until the native operation settles. Generic
|
|
736
|
+
// invoke cancellation settles locally before native cleanup can finish.
|
|
737
|
+
const operation=invoke('speech.synthesizeStream',{streamId,request},{
|
|
738
|
+
timeoutMs:0,onRequest:options.onRequest
|
|
739
|
+
});
|
|
740
|
+
let cancellation=null;
|
|
741
|
+
function cancelSynthesis(){
|
|
742
|
+
if(cancellation)return;
|
|
743
|
+
cancellation=invoke('speech.cancelSynthesis',{streamId},{timeoutMs:0});
|
|
744
|
+
cancellation.catch(function observeSynthesisCleanupFailure(){});
|
|
745
|
+
}
|
|
746
|
+
owned.signal.addEventListener('abort',cancelSynthesis,{once:true});
|
|
747
|
+
if(owned.signal.aborted)cancelSynthesis();
|
|
748
|
+
const joined=(async function joinSynthesisCleanup(){
|
|
749
|
+
let result;
|
|
750
|
+
const failures=[];
|
|
751
|
+
try{result=await operation;}catch(error){
|
|
752
|
+
if(!owned.signal.aborted||error?.name!=='AbortError')failures.push(error);
|
|
753
|
+
}
|
|
754
|
+
if(cancellation){
|
|
755
|
+
try{await cancellation;}catch(error){failures.push(error);}
|
|
756
|
+
}
|
|
757
|
+
if(failures.length===1)throw failures[0];
|
|
758
|
+
if(failures.length)throw new AggregateError(failures,'Speech synthesis and cleanup failed.');
|
|
759
|
+
return result;
|
|
760
|
+
})();
|
|
761
|
+
return owned.settle(joined,{preserveTaskFailure:true}).finally(function releaseSynthesisCancellation(){
|
|
762
|
+
owned.signal.removeEventListener('abort',cancelSynthesis);
|
|
763
|
+
});
|
|
764
|
+
}
|
|
572
765
|
function ollamaInvoke(operation,request={},options={}){
|
|
573
766
|
const onChunk=typeof options==='function'?options:options?.onChunk;
|
|
574
767
|
const timeoutMs=options?.timeoutMs??(['pull','push','create'].includes(operation)?LONG_OPERATION_TIMEOUT:10*60*1000);
|
|
@@ -598,6 +791,8 @@ export function createCoreFacade(client){
|
|
|
598
791
|
remove:name=>invoke('environment.delete',{name})},
|
|
599
792
|
mail:{send:request=>invoke('mail.send',request??{},{timeoutMs:450000})},
|
|
600
793
|
speech:{status:()=>invoke('speech.status',{},{timeoutMs:10000}),
|
|
794
|
+
openStream:openSpeechStream,
|
|
795
|
+
synthesizeStream:synthesizeSpeechStream,
|
|
601
796
|
load:function loadSpeech(request,options){return invoke('speech.load',request??{},{timeoutMs:0,signal:options?.signal,onRequest:options?.onRequest});},
|
|
602
797
|
unload:function unloadSpeech(request,options){return invoke('speech.unload',request??{},{timeoutMs:0,signal:options?.signal,onRequest:options?.onRequest});},
|
|
603
798
|
synthesize:(request,options)=>invoke('speech.synthesize',request??{},{timeoutMs:180000,signal:options?.signal}),
|
package/docs/reference/README.md
CHANGED
|
@@ -76,6 +76,9 @@ alone does not make bare module names resolve in a browser.
|
|
|
76
76
|
| Select FLUX.2 Klein 4B and its three complementary model resources | [FLUX.2 Klein 4B through Core](local-image-flux.md) |
|
|
77
77
|
| Select the complete SDXL Base 1.0 checkpoint for generation and whole-image editing | [SDXL Base 1.0 through Core](local-image-sdxl.md) |
|
|
78
78
|
| Compose independently loaded transcription and synthesis engines | [Native speech service](native-speech.md) |
|
|
79
|
+
| Stream complete native recognition with retained speaker scope | [Native Nemotron](native-nemotron.md) and [Core speech streaming](native-speech.md#continuous-transcription-and-streaming-synthesis) |
|
|
80
|
+
| Produce native PCM speech with selected voice and language | [Native Magpie](native-magpie.md) |
|
|
81
|
+
| Capture and play native speech through shared AI controls | [Shared AI capture and playback](native-speech.md#shared-ai-capture-and-playback) |
|
|
79
82
|
| Keep a native Whisper model loaded across complete recordings | [Native Whisper transcription](local-whisper.md) |
|
|
80
83
|
| Select the optional Intel NPU encoder with independent CUDA or CPU decoding | [Intel NPU Whisper](../guides/native-whisper-openvino.md) |
|
|
81
84
|
| Edit an original PNG with a complete prompt and explicit strength | [Local image editing](local-image-editing.md) |
|
|
@@ -243,6 +246,12 @@ what those artifacts let a developer do.
|
|
|
243
246
|
|
|
244
247
|
## Normalized AI
|
|
245
248
|
|
|
249
|
+
A speech-only interface explicitly passes `null` for both the first and fourth
|
|
250
|
+
`AI` constructor arguments:
|
|
251
|
+
`new AI(null, 'LOCAL_SPEACH', 'LOCAL_SPEACH', null, ttsModel, sttModel)`.
|
|
252
|
+
This leaves the LLM unselected from construction; omitted or empty arguments
|
|
253
|
+
retain their historical defaults. See [native speech startup](native-speech.md#startup-and-status).
|
|
254
|
+
|
|
246
255
|
Portable applications start with the provider-neutral runtime rather than an
|
|
247
256
|
Ollama, Wllama, Whisper, Kokoro, native, or cloud transport:
|
|
248
257
|
|
|
@@ -233,6 +233,53 @@ Choose preparation according to the result the application needs:
|
|
|
233
233
|
| `ai.fetchTTS(payload,signal)` | Promise resolving a real audio Blob | Audio-file providers only; does not play |
|
|
234
234
|
| `ai.prepareTTS(options)` | Preparation handle with `ready` and `getAudio(index)` | Audio-file providers only; optional caller-owned durable storage |
|
|
235
235
|
|
|
236
|
+
## Native Core streaming speech
|
|
237
|
+
|
|
238
|
+
The shared AI controls also consume an explicitly selected native Core speech
|
|
239
|
+
service. [Nemotron](../native-nemotron.md) supplies retained streaming recognition
|
|
240
|
+
and optional diarization; [Magpie](../native-magpie.md) supplies streamed PCM and
|
|
241
|
+
complete WAV/PCM audio. These are separate from browser Web Speech voices:
|
|
242
|
+
Magpie can export audio, whereas the browser-native synthesis provider exposes
|
|
243
|
+
playback only. No selected route silently falls back to another engine.
|
|
244
|
+
|
|
245
|
+
For speech-only construction, pass `null` for both LLM arguments:
|
|
246
|
+
`new AI(null, 'LOCAL_SPEACH', 'LOCAL_SPEACH', null, ttsModel, sttModel)`.
|
|
247
|
+
This leaves the LLM unselected; omitted or empty arguments preserve defaults.
|
|
248
|
+
The existing per-realm AI singleton is reused if already present. Import and
|
|
249
|
+
construction do not prove native readiness.
|
|
250
|
+
|
|
251
|
+
After the selected STT role is ready, `ai.supportsTranscriptionCapture()`
|
|
252
|
+
reports its capture capability. `ai.createTranscriptionCapture(options)`
|
|
253
|
+
returns `null` without a provider capture implementation, otherwise an owned
|
|
254
|
+
`{start, stop, cancel, destroy, done, forceEndpoint?}` handle. Start from the
|
|
255
|
+
user's capture action with `{signal, mediaConstraints}`. Native options include
|
|
256
|
+
`sampleRate` (default 16000), `pcmChunkMs` (default 100),
|
|
257
|
+
`activityThreshold` (default 0.02), `streamOptions`, `onUpdate`,
|
|
258
|
+
`onInterim`, `onSegment`, `onActivity`, `onState`, `onError`,
|
|
259
|
+
`onProgress`, and `onRequest`. The actual capture rate must match the native
|
|
260
|
+
stream; every PCM contribution is delivered in order.
|
|
261
|
+
|
|
262
|
+
Native recognition callbacks receive `(result, {requestStop})`. Keep full
|
|
263
|
+
snapshots: interim results replace, and final segments retain their native
|
|
264
|
+
boundaries, words, timestamps and speaker scope. A callback can call
|
|
265
|
+
`requestStop()` without awaiting it and return; the external owner awaits
|
|
266
|
+
`stop()` or `done`. Never await those joins or `forceEndpoint()` from their
|
|
267
|
+
own callback. External `forceEndpoint()` flushes pending capture PCM and
|
|
268
|
+
recognition callbacks while leaving the microphone and native stream open.
|
|
269
|
+
`stop()` closes capture and drains recognition; `cancel()` and `destroy()`
|
|
270
|
+
join cleanup without claiming a complete transcript. No control sends chat.
|
|
271
|
+
|
|
272
|
+
For a streaming TTS catalog, `ai.prepareTTSPlayback({input, voice, languageCode, speed}, signal)`
|
|
273
|
+
returns an inert descriptor. Its explicit `play()` uses the shared provider
|
|
274
|
+
lane and Web Audio output. The returned control exposes `finished`,
|
|
275
|
+
`released`, `state`, `error`, `pause()`, `resume()`, and `stop()`.
|
|
276
|
+
`finished` describes playback completion, while `released` joins native
|
|
277
|
+
cleanup. Stop targets that utterance, independently of capture.
|
|
278
|
+
`fetchTTS({input, languageCode, ...}, signal)` still returns complete audio
|
|
279
|
+
when the selected engine supports it. Language is caller-selected control data,
|
|
280
|
+
separate from the text; Magpie supports natural speed `1` only. See
|
|
281
|
+
[shared native capture/playback](../native-speech.md#shared-ai-capture-and-playback).
|
|
282
|
+
|
|
236
283
|
<a id="digitalocean-fal-text-to-speech"></a>
|
|
237
284
|
|
|
238
285
|
## TWiN Cloud text-to-speech
|
|
@@ -665,13 +712,13 @@ selected execution device or assess the generated audio.
|
|
|
665
712
|
## Queue complete passages and wait for playback
|
|
666
713
|
|
|
667
714
|
For a complete page or passage, call
|
|
668
|
-
`ai.streamTTS(text, true, {voice, speed, pauseAfterMs, waitForPlayback:true})`.
|
|
715
|
+
`ai.streamTTS(text, true, {voice, speed, languageCode, pauseAfterMs, waitForPlayback:true})`.
|
|
669
716
|
The existing AI queue owns segmentation, concurrent synthesis, ordered
|
|
670
717
|
playback, and cancellation. Supply the exact text; there is no need for an
|
|
671
718
|
application sentence queue, audio cache, or playback scheduler.
|
|
672
719
|
|
|
673
720
|
This function uses the configured `ai` above. Its application-supplied
|
|
674
|
-
`passages` argument is an ordered array of `{text, voice?, speed?, pauseAfterMs?}`
|
|
721
|
+
`passages` argument is an ordered array of `{text, voice?, speed?, languageCode?, pauseAfterMs?}`
|
|
675
722
|
records. An omitted voice uses the selected model's default voice, an omitted
|
|
676
723
|
speed uses `ai.voiceSpeed`, and an omitted pause is zero. Each supplied voice
|
|
677
724
|
must be supported by the selected model; speed must be positive. A pause is
|
|
@@ -689,6 +736,7 @@ async function speakPassages(passages) {
|
|
|
689
736
|
{
|
|
690
737
|
voice: passage.voice,
|
|
691
738
|
speed: passage.speed,
|
|
739
|
+
languageCode: passage.languageCode,
|
|
692
740
|
pauseAfterMs: passage.pauseAfterMs,
|
|
693
741
|
waitForPlayback: true
|
|
694
742
|
}
|
|
@@ -709,14 +757,17 @@ streamed speech and prepared playback owned by that AI instance and settles
|
|
|
709
757
|
pending playback results `false`. Detached preparation described below keeps
|
|
710
758
|
its independent lifetime.
|
|
711
759
|
|
|
712
|
-
|
|
713
|
-
|
|
714
|
-
|
|
715
|
-
|
|
716
|
-
|
|
717
|
-
|
|
718
|
-
the
|
|
719
|
-
|
|
760
|
+
Voice, speed and languageCode remain associated with an unfinished `end:false`
|
|
761
|
+
utterance and its later `finishTTS()` flush; a supplied option replaces its retained
|
|
762
|
+
selection. An explicitly changed language first queues the preceding remainder
|
|
763
|
+
with its old selection, then accepts the new text. Language is separate control
|
|
764
|
+
information and does not rewrite the text. End, mute and stop clear the retained
|
|
765
|
+
selection. Pause and waitForPlayback remain per-call options. Finish a producer
|
|
766
|
+
before starting an independent passage; the automatic speech-input cleanup
|
|
767
|
+
still retains a pending single formatting mark until final flush.
|
|
768
|
+
|
|
769
|
+
A call extracting no segments returns `true` without waiting for earlier jobs,
|
|
770
|
+
except that a language-change call also joins its preceding remainder preparation.
|
|
720
771
|
An already muted call returns `false`.
|
|
721
772
|
|
|
722
773
|
Autoplay permission waiting and recoverable audio-resume attempts leave the
|
|
@@ -161,6 +161,41 @@ pre-aborted signal sends no transcription request; an in-flight abort sends
|
|
|
161
161
|
service owns interruption and release of its actual engine operation.
|
|
162
162
|
Its optional request observer uses the same pre-dispatch correlation contract.
|
|
163
163
|
|
|
164
|
+
## Streaming native speech
|
|
165
|
+
|
|
166
|
+
`Arcane.speech.openStream(request, {signal, onUpdate, onProgress, onRequest})`
|
|
167
|
+
returns `{streamId, ready, completion, current, push, forceEndpoint, finish, cancel}`
|
|
168
|
+
immediately. Await `ready` before feeding ordered mono `Float32Array` input at
|
|
169
|
+
the request's actual `sampleRate`. This API opens no microphone.
|
|
170
|
+
`forceEndpoint()` drains the current utterance and its ordered update callbacks
|
|
171
|
+
without closing the retained stream; `finish()` drains accepted input and final
|
|
172
|
+
delivery terminally. Partial snapshots replace prior partials; finals retain
|
|
173
|
+
their engine-defined segment and speaker scope. Recognition never decides a
|
|
174
|
+
human turn, instruction or approval.
|
|
175
|
+
|
|
176
|
+
An `onUpdate` callback must return before its own `forceEndpoint()`,
|
|
177
|
+
`finish()`, or `completion` join. Explicit `cancel(reason)` immediately
|
|
178
|
+
suppresses later callbacks and awaits the matching native cleanup control.
|
|
179
|
+
Aborting the signal instead follows ordinary request cancellation, whose browser
|
|
180
|
+
promise may settle before native cleanup.
|
|
181
|
+
|
|
182
|
+
`Arcane.speech.synthesizeStream(request, {signal, onAudio, onProgress, onRequest})`
|
|
183
|
+
requires an `onAudio` consumer and resolves to the engine's complete terminal
|
|
184
|
+
metadata. Every callback receives `{pcm: Uint8Array, format, sampleRate, channels, ...}`
|
|
185
|
+
in order. The facade awaits each callback before acknowledging that contribution
|
|
186
|
+
to the native producer. It does not play audio. An utterance abort joins its
|
|
187
|
+
explicit native cancellation and callback cleanup; actual failures remain
|
|
188
|
+
observable. A callback may abort its caller-owned signal and return, then the
|
|
189
|
+
external owner joins synthesis completion.
|
|
190
|
+
|
|
191
|
+
Both streaming operations have no fixed request timeout. Their scoped events
|
|
192
|
+
and controls retain exact request ownership through client close, document
|
|
193
|
+
retirement and transport loss. Engine support and loaded readiness are separate;
|
|
194
|
+
a facade method alone supplies neither. Complete-recording transcription and
|
|
195
|
+
complete-audio synthesis retain their existing contracts. See
|
|
196
|
+
[Core streaming speech](native-speech.md#continuous-transcription-and-streaming-synthesis)
|
|
197
|
+
for the complete protocol, cancellation and engine interfaces.
|
|
198
|
+
|
|
164
199
|
## Current-window state
|
|
165
200
|
|
|
166
201
|
`Arcane.window.state({signal}?)` reads the native window hosting the calling
|
|
@@ -93,11 +93,43 @@ See [the complete selection contract](protocols.md#native-only-application-resou
|
|
|
93
93
|
|
|
94
94
|
## Output and host contract
|
|
95
95
|
|
|
96
|
-
Each build
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
96
|
+
Each build assembles in a fresh temporary directory beneath `outputRoot`, then
|
|
97
|
+
publishes the complete artifact to one stable app-specific child. Portable and
|
|
98
|
+
Windows outputs use `<outputRoot>/<app-id>/`; macOS uses
|
|
99
|
+
`<outputRoot>/<app-id>.app/`. The result remains `{app, target, manifest}`, with
|
|
100
|
+
its actual payload directory at `target.rootDir` and the macOS bundle directory
|
|
101
|
+
also returned as `bundleRoot`. A later successful build replaces that app's
|
|
102
|
+
previous generated output instead of accumulating completed build directories.
|
|
103
|
+
Windows device-name spellings such as `con` receive an underscore in the output
|
|
104
|
+
folder name (`_con` or `_con.app`) on every build host; the app ID stays unchanged.
|
|
105
|
+
|
|
106
|
+
Assembly leaves the current output available. Publication uses same-filesystem
|
|
107
|
+
renames and retains the previous output in a temporary backup until the
|
|
108
|
+
`native.payload.completed` callback accepts the new final paths. A callback
|
|
109
|
+
rejection or cancellation restores the previous output; a successful build
|
|
110
|
+
removes its backup and temporary assembly. The completion callback runs before
|
|
111
|
+
build acceptance, including the following cancellation check; receiving that
|
|
112
|
+
event alone does not establish an accepted build. `native.payload.started` identifies
|
|
113
|
+
the temporary assembly, while `native.payload.completed` identifies the final
|
|
114
|
+
payload and, on macOS, final bundle. Manifest paths stay relative to that payload;
|
|
115
|
+
application content and explicitly supplied configuration paths stay unchanged.
|
|
116
|
+
|
|
117
|
+
Only final replacement of the same app is reserved; independent assemblies and
|
|
118
|
+
other apps remain concurrent. A competing publication reports the exact
|
|
119
|
+
`.arcane-publish-<app-output-name>` reservation so its owner can finish before
|
|
120
|
+
retrying. If recovery or backup cleanup fails, the error identifies the actual
|
|
121
|
+
output, assembly and reservation paths, and whether publication committed.
|
|
122
|
+
The reservation's `owner.json` records the process, acquisition time, advisory
|
|
123
|
+
five-minute TTL/expiry and recovery locations. Expiry never permits automatic
|
|
124
|
+
takeover or deletion. After an interrupted build, inspect those paths before restoring the previous
|
|
125
|
+
output or removing the reservation. A running or locked app may prevent Windows
|
|
126
|
+
replacement; close that app and retry the build after the reported failure.
|
|
127
|
+
|
|
128
|
+
Replacement applies only to the matching app's generated artifact directory.
|
|
129
|
+
Source/release roots, unrelated outputs, host/runtime download caches and the
|
|
130
|
+
host's separate saved profile/model directories remain outside replacement.
|
|
131
|
+
Existing randomly named outputs from earlier SDK versions are left untouched;
|
|
132
|
+
the new stable-output lifecycle does not sweep old directories or user data.
|
|
101
133
|
|
|
102
134
|
The selected app is copied unchanged under `app/`. Dependencies are copied
|
|
103
135
|
unchanged under `dependencies/<index>/`; the manifest relates each directory to
|
|
@@ -751,7 +783,7 @@ produce their actual download error. SDK collaborators may select an existing
|
|
|
751
783
|
version, another architecture or a browser-only artifact.
|
|
752
784
|
|
|
753
785
|
The complete unchanged portable payload lives at
|
|
754
|
-
`<app-id>.app/Contents/Resources/`. That directory remains `target.rootDir` and
|
|
786
|
+
`<outputRoot>/<app-id>.app/Contents/Resources/`. That directory remains `target.rootDir` and
|
|
755
787
|
contains `arcane-native.json`, `app/`, dependency releases, installed SDK files,
|
|
756
788
|
Core entry/client and `runtime/node`. The additive `bundleRoot` identifies the
|
|
757
789
|
`.app` directory. `Contents/MacOS/Arcane` and `Contents/Info.plist` belong to the
|