arcane-os 0.93.0 → 0.95.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +33 -0
- package/README.md +1 -1
- package/browser-runtime/ai/twin-cloud.mjs +109 -46
- package/browser-runtime/core/client.mjs +197 -0
- package/docs/reference/ai/twin-cloud.md +35 -14
- package/docs/reference/availability-and-normalization.md +1 -1
- package/docs/reference/inventory/package-api.json +96 -12
- package/docs/reference/local-ai.md +81 -5
- package/docs/reference/native-magpie.md +98 -0
- package/docs/reference/native-nemotron.md +204 -0
- package/docs/reference/native-speech.md +364 -11
- package/docs/reference/runtime-modules.md +21 -18
- package/docs/reference/sdk-api.md +152 -11
- package/package.json +6 -1
- package/runtime/arcane/modules/AI.js +114 -14
- package/runtime/arcane/modules/AIProviderRuntime.js +32 -15
- package/runtime/arcane/modules/ContinuousVoiceCapture.js +118 -21
- package/runtime/arcane/modules/NativeSpeechCapture.js +198 -0
- package/runtime/arcane/modules/NativeSpeechPlayback.js +304 -0
- package/runtime/arcane/modules/VoiceCaptureWorklet.js +62 -2
- package/schemas/arcane-app.schema.json +2 -2
- package/schemas/arcane-package.schema.json +2 -2
- package/src/core/services/speech.mjs +320 -25
- package/src/local-ai/config.mjs +12 -1
- package/src/local-ai/dev.mjs +71 -4
- package/src/local-ai/install.mjs +113 -5
- package/src/local-ai/kokoro/frontend.mjs +219 -0
- package/src/local-ai/kokoro/helper.mjs +168 -0
- package/src/local-ai/kokoro/index.mjs +645 -0
- package/src/local-ai/kokoro/native/CMakeLists.txt +45 -0
- package/src/local-ai/kokoro/native/main.cpp +249 -0
- package/src/local-ai/magpie/build.mjs +36 -0
- package/src/local-ai/magpie/helper.mjs +254 -0
- package/src/local-ai/magpie/index.mjs +488 -0
- package/src/local-ai/magpie/native/CMakeLists.txt +10 -0
- package/src/local-ai/magpie/native/main.cpp +270 -0
- package/src/local-ai/nemotron/build.mjs +53 -0
- package/src/local-ai/nemotron/helper.mjs +308 -0
- package/src/local-ai/nemotron/index.mjs +637 -0
- package/src/local-ai/nemotron/native/CMakeLists.txt +17 -0
- package/src/local-ai/nemotron/native/main.cpp +282 -0
- package/src/native/portable-layout.mjs +67 -9
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,38 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.95.0
|
|
4
|
+
|
|
5
|
+
- Add retained native Nemotron streaming recognition and optional diarization,
|
|
6
|
+
with ordered complete snapshots, explicit utterance endpoints, speaker scope,
|
|
7
|
+
and joined cancellation. Preserve existing Whisper and complete transcription.
|
|
8
|
+
- Add native Magpie/NanoCodec synthesis with model-owned voice selection,
|
|
9
|
+
per-request language, ordered PCM streaming, complete WAV/PCM export, and
|
|
10
|
+
exact-utterance cancellation. Preserve authored input and natural model rate;
|
|
11
|
+
the caller supplies upstream runtime and model assets without SDK vendoring.
|
|
12
|
+
- Extend shared Core speech and browser clients with correlated streaming
|
|
13
|
+
capture/output, consumed-audio acknowledgements and native cleanup joins.
|
|
14
|
+
Connect AI playback through its existing queues and shared AudioContext,
|
|
15
|
+
retaining per-utterance pause/resume/stop and complete-audio APIs.
|
|
16
|
+
- Keep capture ownership, recognition segments and model-owned conversational
|
|
17
|
+
decisions distinct. Windows x64 native helpers compiled and installed against
|
|
18
|
+
the approved NeMo 0.2.0 runtime; inference, microphone/audio acceptance and
|
|
19
|
+
other-platform execution remain separate evidence boundaries.
|
|
20
|
+
|
|
21
|
+
## 0.94.0
|
|
22
|
+
|
|
23
|
+
- Recover all TWiN HTTP 429 quota responses through the shared chat/System One
|
|
24
|
+
transport. Honor Retry-After or exponential backoff, share a transient
|
|
25
|
+
same-endpoint/credential cooldown, and pace concurrent recovery dispatches.
|
|
26
|
+
Fetch rejection, 429 and 529 share three retries; cancellation stops waits.
|
|
27
|
+
Preserve exact requests and complete terminal errors, including provider
|
|
28
|
+
status_code and nested error records. Do not replay 400/413 errors, consumed
|
|
29
|
+
streams, callbacks, or image submissions.
|
|
30
|
+
- Include the completed native Kokoro synthesis capability: retained voices,
|
|
31
|
+
explicit model preparation, owned cancellation and audio output through the
|
|
32
|
+
shared native speech contract. Reuse its retained compatible runtime and
|
|
33
|
+
source archives alongside the existing native host, Whisper and ONNX assets.
|
|
34
|
+
Package inclusion is distinct from application or native-runtime acceptance.
|
|
35
|
+
|
|
3
36
|
## 0.93.0
|
|
4
37
|
|
|
5
38
|
- Add native desktop notification status, submission, state and removal through
|
package/README.md
CHANGED
|
@@ -25,7 +25,7 @@ selected SDK projection or an explicit live SDK source mount. Both profiles
|
|
|
25
25
|
share the theme, packaging, event, cancellation,
|
|
26
26
|
and browser run contracts while retaining their selected application layout.
|
|
27
27
|
|
|
28
|
-
This checkout defines the `0.
|
|
28
|
+
This checkout defines the `0.95.0` SDK contract. Development applications declare
|
|
29
29
|
`arcane-os` as `latest` and retain their lockfile. Exact SDK pins begin when Roshi
|
|
30
30
|
designates a production build. Follow [Consumer updates](docs/consumer-updates.md)
|
|
31
31
|
to refresh and compare the installed SDK and lockfile with the published channel;
|
|
@@ -4,6 +4,8 @@ import {arcaneLogging} from '../logging.mjs';
|
|
|
4
4
|
const is = new Is(false);
|
|
5
5
|
const twinChatURL = 'https://inference.do-ai.run/v1/chat/completions';
|
|
6
6
|
const twinSystemOneURL = 'https://inference.do-ai.run/v1/systemone';
|
|
7
|
+
const activeHTTPRequests = new Map();
|
|
8
|
+
const httpRetryDelayMs = 3000;
|
|
7
9
|
|
|
8
10
|
/** A stateless TWiN request; browser AI shares the HTTP and format owners below. */
|
|
9
11
|
export async function fetchRequest({
|
|
@@ -450,10 +452,24 @@ export function openAIResponseFormat(format){
|
|
|
450
452
|
/** Shared with browser streaming; consume no successful body at this boundary. */
|
|
451
453
|
export async function fetchHTTPResponse(url, options, {onRetry = null} = {}) {
|
|
452
454
|
const {signal} = options;
|
|
453
|
-
const retryDelayMs = 3000;
|
|
454
455
|
const maxRecoveryRetries = 3;
|
|
455
|
-
let recoveryRetries = 0;
|
|
456
456
|
let attempt = 0;
|
|
457
|
+
let retry = null;
|
|
458
|
+
const destination = url?.url ?? String(url);
|
|
459
|
+
const headers = new Headers(options.headers ?? url?.headers);
|
|
460
|
+
const requestOptions = {...options, headers};
|
|
461
|
+
const authorization = headers.get('authorization') ?? '';
|
|
462
|
+
let accounts = activeHTTPRequests.get(destination);
|
|
463
|
+
if (!accounts) {
|
|
464
|
+
accounts = new Map();
|
|
465
|
+
activeHTTPRequests.set(destination, accounts);
|
|
466
|
+
}
|
|
467
|
+
let pacing = accounts.get(authorization);
|
|
468
|
+
if (!pacing) {
|
|
469
|
+
pacing = {active: 0, rateLimited: false, resumeAt: 0, nextDispatchAt: 0, error: null};
|
|
470
|
+
accounts.set(authorization, pacing);
|
|
471
|
+
}
|
|
472
|
+
pacing.active += 1;
|
|
457
473
|
try {
|
|
458
474
|
while (true) {
|
|
459
475
|
if (signal?.aborted) {
|
|
@@ -462,7 +478,7 @@ export async function fetchHTTPResponse(url, options, {onRetry = null} = {}) {
|
|
|
462
478
|
let response;
|
|
463
479
|
let error;
|
|
464
480
|
try {
|
|
465
|
-
response = await
|
|
481
|
+
response = await dispatchHTTPRequest(url, requestOptions, pacing, onRetry, retry, attempt);
|
|
466
482
|
} catch (fetchError) {
|
|
467
483
|
if (isAIRequestAbort(fetchError, signal)) {
|
|
468
484
|
throw normalizeAIRequestAbort(fetchError);
|
|
@@ -483,7 +499,7 @@ export async function fetchHTTPResponse(url, options, {onRetry = null} = {}) {
|
|
|
483
499
|
? await response.json()
|
|
484
500
|
: await response.text();
|
|
485
501
|
} catch (bodyError) {
|
|
486
|
-
if (status !== 529 || isAIRequestAbort(bodyError, signal)) {
|
|
502
|
+
if ((status !== 429 && status !== 529) || isAIRequestAbort(bodyError, signal)) {
|
|
487
503
|
throw bodyError;
|
|
488
504
|
}
|
|
489
505
|
error = bodyError;
|
|
@@ -495,61 +511,108 @@ export async function fetchHTTPResponse(url, options, {onRetry = null} = {}) {
|
|
|
495
511
|
const message = is.string(error)
|
|
496
512
|
? error
|
|
497
513
|
: error?.error?.message ?? error?.message;
|
|
498
|
-
const
|
|
499
|
-
|
|
500
|
-
|
|
501
|
-
if (
|
|
502
|
-
|
|
503
|
-
|
|
504
|
-
|
|
505
|
-
|
|
506
|
-
|
|
514
|
+
const retryDelayMs = status === 429
|
|
515
|
+
? httpRateLimitDelay(response, httpRetryDelayMs * (2 ** Math.min(attempt, maxRecoveryRetries - 1)))
|
|
516
|
+
: httpRetryDelayMs;
|
|
517
|
+
if (status === 429) {
|
|
518
|
+
pacing.rateLimited = true;
|
|
519
|
+
pacing.resumeAt = Math.max(pacing.resumeAt, Date.now() + retryDelayMs);
|
|
520
|
+
pacing.error = error;
|
|
521
|
+
}
|
|
522
|
+
if ((status !== null && status !== 429 && status !== 529)
|
|
523
|
+
|| attempt >= maxRecoveryRetries) {
|
|
524
|
+
throw error;
|
|
507
525
|
}
|
|
508
526
|
attempt += 1;
|
|
509
527
|
arcaneLogging.warn(
|
|
510
528
|
`${message ?? 'AI request transport failed.'}\nRetrying in ${retryDelayMs / 1000} seconds`,
|
|
511
529
|
error
|
|
512
530
|
);
|
|
513
|
-
|
|
514
|
-
|
|
515
|
-
|
|
516
|
-
);
|
|
517
|
-
await new Promise(
|
|
518
|
-
function waitForRequestRetry(resolve, reject) {
|
|
519
|
-
function finishRetryDelay() {
|
|
520
|
-
signal?.removeEventListener('abort', cancelRetryDelay);
|
|
521
|
-
resolve();
|
|
522
|
-
}
|
|
523
|
-
function cancelRetryDelay() {
|
|
524
|
-
clearTimeout(timer);
|
|
525
|
-
signal.removeEventListener('abort', cancelRetryDelay);
|
|
526
|
-
reject(normalizeAIRequestAbort(signal.reason));
|
|
527
|
-
}
|
|
528
|
-
const timer = setTimeout(finishRetryDelay, retryDelayMs);
|
|
529
|
-
signal?.addEventListener(
|
|
530
|
-
'abort',
|
|
531
|
-
cancelRetryDelay,
|
|
532
|
-
{once: true}
|
|
533
|
-
);
|
|
534
|
-
if (signal?.aborted) {
|
|
535
|
-
cancelRetryDelay();
|
|
536
|
-
}
|
|
537
|
-
}
|
|
538
|
-
);
|
|
539
|
-
if (signal?.aborted) {
|
|
540
|
-
throw normalizeAIRequestAbort(signal.reason);
|
|
541
|
-
}
|
|
542
|
-
observeRequestRetry(
|
|
543
|
-
onRetry,
|
|
544
|
-
{phase: 'requesting', attempt, delayMs: retryDelayMs, status, error}
|
|
545
|
-
);
|
|
531
|
+
retry = {phase: 'waiting', attempt, delayMs: retryDelayMs, status, error};
|
|
532
|
+
observeRequestRetry(onRetry, retry);
|
|
533
|
+
await waitForHTTPDelay(retryDelayMs, signal);
|
|
546
534
|
}
|
|
547
535
|
} catch (error) {
|
|
548
536
|
if (isAIRequestAbort(error, signal)) {
|
|
549
537
|
throw normalizeAIRequestAbort(error);
|
|
550
538
|
}
|
|
551
539
|
throw error;
|
|
540
|
+
} finally {
|
|
541
|
+
pacing.active -= 1;
|
|
542
|
+
if (pacing.active === 0) {
|
|
543
|
+
// Retain neither credentials nor quota state beyond the active request group.
|
|
544
|
+
accounts.delete(authorization);
|
|
545
|
+
if (accounts.size === 0) {
|
|
546
|
+
activeHTTPRequests.delete(destination);
|
|
547
|
+
}
|
|
548
|
+
}
|
|
549
|
+
}
|
|
550
|
+
}
|
|
551
|
+
|
|
552
|
+
function httpRateLimitDelay(response, fallback) {
|
|
553
|
+
const value = response.headers.get('retry-after')?.trim();
|
|
554
|
+
if (!value) return fallback;
|
|
555
|
+
const seconds = Number(value);
|
|
556
|
+
if (is.finite(seconds)) {
|
|
557
|
+
return seconds >= 0 && is.finite(seconds * 1000) ? seconds * 1000 : fallback;
|
|
552
558
|
}
|
|
559
|
+
// Accept HTTP weekday date forms, not an arbitrary numeric string interpreted as a date.
|
|
560
|
+
const date = /^(Mon|Tue|Wed|Thu|Fri|Sat|Sun)/i.test(value) ? Date.parse(value) : NaN;
|
|
561
|
+
return is.finite(date) ? Math.max(0, date - Date.now()) : fallback;
|
|
562
|
+
}
|
|
563
|
+
|
|
564
|
+
async function dispatchHTTPRequest(url, options, pacing, onRetry, retry, attempt) {
|
|
565
|
+
const {signal} = options;
|
|
566
|
+
let observation = null;
|
|
567
|
+
// Only an observed quota response coordinates otherwise independent requests.
|
|
568
|
+
while (pacing.rateLimited) {
|
|
569
|
+
if (signal?.aborted) throw normalizeAIRequestAbort(signal.reason);
|
|
570
|
+
const delayMs = Math.max(pacing.resumeAt, pacing.nextDispatchAt) - Date.now();
|
|
571
|
+
if (delayMs <= 0) {
|
|
572
|
+
// Claim and initiate Fetch in this continuation, with no intervening asynchronous wait.
|
|
573
|
+
pacing.nextDispatchAt = Date.now() + httpRetryDelayMs;
|
|
574
|
+
break;
|
|
575
|
+
}
|
|
576
|
+
if (!retry && !observation) {
|
|
577
|
+
observation = {phase: 'waiting', attempt, delayMs, status: 429, error: pacing.error};
|
|
578
|
+
observeRequestRetry(onRetry, observation);
|
|
579
|
+
}
|
|
580
|
+
await waitForHTTPDelay(delayMs, signal);
|
|
581
|
+
}
|
|
582
|
+
if (observation && !signal?.aborted) {
|
|
583
|
+
observeRequestRetry(onRetry, {...observation, phase: 'requesting'});
|
|
584
|
+
}
|
|
585
|
+
if (retry) {
|
|
586
|
+
observeRequestRetry(onRetry, {...retry, phase: 'requesting'});
|
|
587
|
+
}
|
|
588
|
+
if (signal?.aborted) throw normalizeAIRequestAbort(signal.reason);
|
|
589
|
+
return fetch(url, options);
|
|
590
|
+
}
|
|
591
|
+
|
|
592
|
+
async function waitForHTTPDelay(delayMs, signal) {
|
|
593
|
+
let remaining = delayMs;
|
|
594
|
+
do {
|
|
595
|
+
if (signal?.aborted) throw normalizeAIRequestAbort(signal.reason);
|
|
596
|
+
// Only split at the host timer range; never shorten the provider's requested wait.
|
|
597
|
+
const milliseconds = Math.min(remaining, 2147483647);
|
|
598
|
+
await new Promise(
|
|
599
|
+
function waitForRequestRetry(resolve, reject) {
|
|
600
|
+
function finishRetryDelay() {
|
|
601
|
+
signal?.removeEventListener('abort', cancelRetryDelay);
|
|
602
|
+
resolve();
|
|
603
|
+
}
|
|
604
|
+
function cancelRetryDelay() {
|
|
605
|
+
clearTimeout(timer);
|
|
606
|
+
signal.removeEventListener('abort', cancelRetryDelay);
|
|
607
|
+
reject(normalizeAIRequestAbort(signal.reason));
|
|
608
|
+
}
|
|
609
|
+
const timer = setTimeout(finishRetryDelay, milliseconds);
|
|
610
|
+
signal?.addEventListener('abort', cancelRetryDelay, {once: true});
|
|
611
|
+
if (signal?.aborted) cancelRetryDelay();
|
|
612
|
+
}
|
|
613
|
+
);
|
|
614
|
+
remaining -= milliseconds;
|
|
615
|
+
} while (remaining > 0);
|
|
553
616
|
}
|
|
554
617
|
|
|
555
618
|
function observeRequestRetry(onRetry, state) {
|
|
@@ -569,6 +569,199 @@ export function createCoreClient({
|
|
|
569
569
|
export function createCoreFacade(client){
|
|
570
570
|
const {invoke,events,uuid}=client;
|
|
571
571
|
const long={timeoutMs:LONG_OPERATION_TIMEOUT};
|
|
572
|
+
function speechStreamEvents(streamId,options){
|
|
573
|
+
const controller=new AbortController();
|
|
574
|
+
const signal=options.signal?AbortSignal.any([options.signal,controller.signal]):controller.signal;
|
|
575
|
+
const subscriptions=[];
|
|
576
|
+
let delivery=Promise.resolve();
|
|
577
|
+
let callbackFailure;
|
|
578
|
+
function listen(name,callback){
|
|
579
|
+
if(callback===undefined)return;
|
|
580
|
+
if(typeof callback!=='function')throw new TypeError(`${name} observer must be a function.`);
|
|
581
|
+
subscriptions.push(events.on(name,function receiveSpeechStreamEvent(event){
|
|
582
|
+
if(event?.streamId!==streamId||signal.aborted)return;
|
|
583
|
+
delivery=delivery.then(function deliverOrderedSpeechEvent(){
|
|
584
|
+
if(!signal.aborted)return callback(event);
|
|
585
|
+
});
|
|
586
|
+
delivery.catch(function cancelFailedSpeechObserver(error){
|
|
587
|
+
callbackFailure??=error;
|
|
588
|
+
controller.abort(error);
|
|
589
|
+
});
|
|
590
|
+
}));
|
|
591
|
+
}
|
|
592
|
+
function dispose(){
|
|
593
|
+
for(const unsubscribe of subscriptions.splice(0))unsubscribe();
|
|
594
|
+
}
|
|
595
|
+
async function delivered(){
|
|
596
|
+
await delivery;
|
|
597
|
+
signal.throwIfAborted();
|
|
598
|
+
}
|
|
599
|
+
async function settle(task,{preserveTaskFailure=false}={}){
|
|
600
|
+
let result;
|
|
601
|
+
let failure;
|
|
602
|
+
let failed=false;
|
|
603
|
+
let deliveryFailure;
|
|
604
|
+
let deliveryFailed=false;
|
|
605
|
+
try{
|
|
606
|
+
try{result=await task;}catch(error){failed=true;failure=error;}
|
|
607
|
+
try{await delivery;}catch(error){deliveryFailed=true;deliveryFailure=callbackFailure??error;}
|
|
608
|
+
if(preserveTaskFailure&&failed&&deliveryFailed&&failure!==deliveryFailure){
|
|
609
|
+
throw new AggregateError([deliveryFailure,failure],'Speech callback and native cleanup failed.');
|
|
610
|
+
}
|
|
611
|
+
if(failed||deliveryFailed)throw callbackFailure??deliveryFailure??failure;
|
|
612
|
+
signal.throwIfAborted();
|
|
613
|
+
return result;
|
|
614
|
+
}finally{dispose();}
|
|
615
|
+
}
|
|
616
|
+
return {controller,signal,listen,settle,dispose,delivered};
|
|
617
|
+
}
|
|
618
|
+
function encodeSpeechSamples(samples){
|
|
619
|
+
if(!(samples instanceof Float32Array))throw new TypeError('Speech stream push requires Float32Array PCM.');
|
|
620
|
+
const encoded=new Uint8Array(samples.length*4);
|
|
621
|
+
const view=new DataView(encoded.buffer);
|
|
622
|
+
const bits=new Uint32Array(samples.buffer,samples.byteOffset,samples.length);
|
|
623
|
+
for(let index=0;index<bits.length;index+=1)view.setUint32(index*4,bits[index],true);
|
|
624
|
+
let binary='';
|
|
625
|
+
for(const value of encoded)binary+=String.fromCharCode(value);
|
|
626
|
+
return {encoding:'f32le',data:globalThis.btoa(binary)};
|
|
627
|
+
}
|
|
628
|
+
function decodeSpeechAudio(audio){
|
|
629
|
+
if(audio?.pcm?.encoding!=='base64'||typeof audio.pcm.data!=='string'){
|
|
630
|
+
throw new TypeError('Streaming speech audio requires base64 PCM transport.');
|
|
631
|
+
}
|
|
632
|
+
const binary=globalThis.atob(audio.pcm.data);
|
|
633
|
+
const pcm=new Uint8Array(binary.length);
|
|
634
|
+
for(let index=0;index<binary.length;index+=1)pcm[index]=binary.charCodeAt(index);
|
|
635
|
+
return {...audio,pcm};
|
|
636
|
+
}
|
|
637
|
+
function openSpeechStream(request={},options={}){
|
|
638
|
+
const streamId=uuid();
|
|
639
|
+
const owned=speechStreamEvents(streamId,options);
|
|
640
|
+
let resolveReady;
|
|
641
|
+
let rejectReady;
|
|
642
|
+
let opened=false;
|
|
643
|
+
let ending=false;
|
|
644
|
+
let snapshot=null;
|
|
645
|
+
let controls=Promise.resolve();
|
|
646
|
+
let finishing=null;
|
|
647
|
+
let cancelling=null;
|
|
648
|
+
const ready=new Promise(function waitForSpeechStream(resolve,reject){
|
|
649
|
+
resolveReady=resolve;
|
|
650
|
+
rejectReady=reject;
|
|
651
|
+
});
|
|
652
|
+
ready.catch(function observeSpeechStreamReadinessFailure(){});
|
|
653
|
+
try{
|
|
654
|
+
owned.listen('speech.opened',function speechStreamOpened(event){
|
|
655
|
+
opened=true;
|
|
656
|
+
resolveReady(event);
|
|
657
|
+
});
|
|
658
|
+
owned.listen('speech.transcription',function transcriptionUpdated(event){
|
|
659
|
+
snapshot=event.result;
|
|
660
|
+
return options.onUpdate?.(event.result);
|
|
661
|
+
});
|
|
662
|
+
if(options.onProgress!==undefined){
|
|
663
|
+
owned.listen('speech.progress',function speechStreamProgress(event){return options.onProgress(event);});
|
|
664
|
+
}
|
|
665
|
+
}catch(error){owned.dispose();throw error;}
|
|
666
|
+
const completion=owned.settle(invoke('speech.open',{streamId,options:request},{
|
|
667
|
+
timeoutMs:0,signal:owned.signal,onRequest:options.onRequest
|
|
668
|
+
})).then(function speechStreamCompleted(result){
|
|
669
|
+
if(!opened)throw new CoreError({code:'SPEECH_STREAM_NOT_OPENED',message:'The speech stream ended before readiness.'});
|
|
670
|
+
return result;
|
|
671
|
+
}).catch(function speechStreamFailed(error){
|
|
672
|
+
rejectReady(error);
|
|
673
|
+
throw error;
|
|
674
|
+
}).finally(function releaseSpeechStream(){ending=true;});
|
|
675
|
+
completion.catch(function observeOwnedSpeechStreamFailure(){});
|
|
676
|
+
function control(method,parameters={}){
|
|
677
|
+
const task=controls.then(async function sendOrderedSpeechControl(){
|
|
678
|
+
await ready;
|
|
679
|
+
owned.signal.throwIfAborted();
|
|
680
|
+
return invoke(method,{streamId,...parameters},{timeoutMs:0,signal:owned.signal});
|
|
681
|
+
});
|
|
682
|
+
controls=task;
|
|
683
|
+
task.catch(function cancelFailedSpeechControl(error){owned.controller.abort(error);});
|
|
684
|
+
return task;
|
|
685
|
+
}
|
|
686
|
+
return {
|
|
687
|
+
streamId,ready,completion,
|
|
688
|
+
current:function currentSpeechTranscript(){return snapshot;},
|
|
689
|
+
push:function pushSpeechSamples(samples){
|
|
690
|
+
if(ending)return Promise.reject(new CoreError({code:'SPEECH_STREAM_ENDING',message:'The speech stream is finishing.'}));
|
|
691
|
+
return control('speech.push',{audio:encodeSpeechSamples(samples)});
|
|
692
|
+
},
|
|
693
|
+
forceEndpoint:function endpointSpeechStream(){
|
|
694
|
+
if(ending)return Promise.reject(new CoreError({code:'SPEECH_STREAM_ENDING',message:'The speech stream is finishing.'}));
|
|
695
|
+
return control('speech.endpoint').then(async function deliverEndpointSnapshots(result){
|
|
696
|
+
await owned.delivered();
|
|
697
|
+
return result;
|
|
698
|
+
});
|
|
699
|
+
},
|
|
700
|
+
finish:function finishSpeechStream(){
|
|
701
|
+
if(!ending){
|
|
702
|
+
ending=true;
|
|
703
|
+
finishing=control('speech.finish').then(function finishSpeechDelivery(){return completion;});
|
|
704
|
+
finishing.catch(function observeSpeechFinishFailure(){});
|
|
705
|
+
}
|
|
706
|
+
return finishing??completion;
|
|
707
|
+
},
|
|
708
|
+
cancel:function cancelSpeechStream(reason){
|
|
709
|
+
if(cancelling)return cancelling;
|
|
710
|
+
ending=true;
|
|
711
|
+
const stopping=invoke('speech.cancel',{streamId},{timeoutMs:0});
|
|
712
|
+
owned.controller.abort(reason);
|
|
713
|
+
// A transcript callback may await cancellation. Native cleanup
|
|
714
|
+
// must not wait for that same callback through completion.
|
|
715
|
+
cancelling=stopping;
|
|
716
|
+
return cancelling;
|
|
717
|
+
}
|
|
718
|
+
};
|
|
719
|
+
}
|
|
720
|
+
function synthesizeSpeechStream(request={},options={}){
|
|
721
|
+
if(typeof options.onAudio!=='function')throw new TypeError('Streaming speech synthesis requires an onAudio consumer.');
|
|
722
|
+
const streamId=uuid();
|
|
723
|
+
const owned=speechStreamEvents(streamId,options);
|
|
724
|
+
try{
|
|
725
|
+
owned.listen('speech.audio',async function synthesisAudio(event){
|
|
726
|
+
await options.onAudio(decodeSpeechAudio(event.audio));
|
|
727
|
+
owned.signal.throwIfAborted();
|
|
728
|
+
await invoke('speech.audioConsumed',{streamId,sequence:event.sequence},{timeoutMs:0,signal:owned.signal});
|
|
729
|
+
});
|
|
730
|
+
if(options.onProgress!==undefined){
|
|
731
|
+
owned.listen('speech.progress',function synthesisProgress(event){return options.onProgress(event);});
|
|
732
|
+
}
|
|
733
|
+
}catch(error){owned.dispose();throw error;}
|
|
734
|
+
if(owned.signal.aborted)return owned.settle(Promise.reject(owned.signal.reason));
|
|
735
|
+
// Keep the response pending until the native operation settles. Generic
|
|
736
|
+
// invoke cancellation settles locally before native cleanup can finish.
|
|
737
|
+
const operation=invoke('speech.synthesizeStream',{streamId,request},{
|
|
738
|
+
timeoutMs:0,onRequest:options.onRequest
|
|
739
|
+
});
|
|
740
|
+
let cancellation=null;
|
|
741
|
+
function cancelSynthesis(){
|
|
742
|
+
if(cancellation)return;
|
|
743
|
+
cancellation=invoke('speech.cancelSynthesis',{streamId},{timeoutMs:0});
|
|
744
|
+
cancellation.catch(function observeSynthesisCleanupFailure(){});
|
|
745
|
+
}
|
|
746
|
+
owned.signal.addEventListener('abort',cancelSynthesis,{once:true});
|
|
747
|
+
if(owned.signal.aborted)cancelSynthesis();
|
|
748
|
+
const joined=(async function joinSynthesisCleanup(){
|
|
749
|
+
let result;
|
|
750
|
+
const failures=[];
|
|
751
|
+
try{result=await operation;}catch(error){
|
|
752
|
+
if(!owned.signal.aborted||error?.name!=='AbortError')failures.push(error);
|
|
753
|
+
}
|
|
754
|
+
if(cancellation){
|
|
755
|
+
try{await cancellation;}catch(error){failures.push(error);}
|
|
756
|
+
}
|
|
757
|
+
if(failures.length===1)throw failures[0];
|
|
758
|
+
if(failures.length)throw new AggregateError(failures,'Speech synthesis and cleanup failed.');
|
|
759
|
+
return result;
|
|
760
|
+
})();
|
|
761
|
+
return owned.settle(joined,{preserveTaskFailure:true}).finally(function releaseSynthesisCancellation(){
|
|
762
|
+
owned.signal.removeEventListener('abort',cancelSynthesis);
|
|
763
|
+
});
|
|
764
|
+
}
|
|
572
765
|
function ollamaInvoke(operation,request={},options={}){
|
|
573
766
|
const onChunk=typeof options==='function'?options:options?.onChunk;
|
|
574
767
|
const timeoutMs=options?.timeoutMs??(['pull','push','create'].includes(operation)?LONG_OPERATION_TIMEOUT:10*60*1000);
|
|
@@ -598,6 +791,10 @@ export function createCoreFacade(client){
|
|
|
598
791
|
remove:name=>invoke('environment.delete',{name})},
|
|
599
792
|
mail:{send:request=>invoke('mail.send',request??{},{timeoutMs:450000})},
|
|
600
793
|
speech:{status:()=>invoke('speech.status',{},{timeoutMs:10000}),
|
|
794
|
+
openStream:openSpeechStream,
|
|
795
|
+
synthesizeStream:synthesizeSpeechStream,
|
|
796
|
+
load:function loadSpeech(request,options){return invoke('speech.load',request??{},{timeoutMs:0,signal:options?.signal,onRequest:options?.onRequest});},
|
|
797
|
+
unload:function unloadSpeech(request,options){return invoke('speech.unload',request??{},{timeoutMs:0,signal:options?.signal,onRequest:options?.onRequest});},
|
|
601
798
|
synthesize:(request,options)=>invoke('speech.synthesize',request??{},{timeoutMs:180000,signal:options?.signal}),
|
|
602
799
|
transcribe:(request,options)=>invoke('speech.transcribe',request??{},{timeoutMs:180000,signal:options?.signal,onRequest:options?.onRequest})},
|
|
603
800
|
localAI:{status:()=>invoke('localai.status',{},{timeoutMs:15000}),
|
|
@@ -94,17 +94,34 @@ The key is
|
|
|
94
94
|
transport authentication, not part of either callback's message payload. Keep
|
|
95
95
|
credentials out of application logging as well.
|
|
96
96
|
|
|
97
|
-
Before a successful response is consumed,
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
97
|
+
Before a successful response is consumed, rejected Fetch calls and HTTP `429`
|
|
98
|
+
or `529` share a budget of three retries (four total attempts). Every `429`
|
|
99
|
+
qualifies, including request-per-minute and token-per-minute quota responses;
|
|
100
|
+
the error message does not select recovery. A readable `Retry-After` header
|
|
101
|
+
supplies the `429` delay as seconds or an HTTP date. Otherwise the upcoming
|
|
102
|
+
retry number selects a `3000`, `6000`, or `12000` millisecond delay. Rejected
|
|
103
|
+
Fetch and `529` retain their `3000` millisecond delay. A provider's longer wait
|
|
104
|
+
is preserved, split only at the host timer's range when necessary.
|
|
105
|
+
|
|
106
|
+
After an observed `429`, overlapping requests through this module with the
|
|
107
|
+
same exact endpoint and Authorization value share the latest cooldown and
|
|
108
|
+
dispatch at least three seconds apart. A later quota response extends that
|
|
109
|
+
cooldown, including a terminal quota response while sibling calls remain active;
|
|
110
|
+
its fallback remains at most `12000` milliseconds. Different endpoints and credentials remain independent, and requests
|
|
111
|
+
before any quota response remain concurrent. This is active-operation pacing,
|
|
112
|
+
not a persistent global rate limiter: the module releases the group, including
|
|
113
|
+
its credential key and error, when its last HTTP operation settles. It does not
|
|
114
|
+
coordinate separate tabs, processes, or independently loaded module instances.
|
|
115
|
+
|
|
116
|
+
Each attempt reuses the exact destination, Fetch option values, serialized request,
|
|
117
|
+
and signal. Headers are normalized once through standard Fetch `Headers` and
|
|
118
|
+
retained for retries, including when the caller supplied a one-shot iterable.
|
|
119
|
+
Other HTTP errors (including `400` and `413`), successful-response
|
|
120
|
+
decoding failures, and application callback failures do not retry. A known
|
|
121
|
+
`429` or `529` also retries when reading its diagnostic body fails;
|
|
122
|
+
`onRetry.error` retains that exact failure. The last complete error is thrown
|
|
123
|
+
unchanged on exhaustion, preserving provider fields such as `status_code` and
|
|
124
|
+
nested `error.type`/`error.message`. No error envelope is rebuilt or sanitized.
|
|
108
125
|
Pass a fresh
|
|
109
126
|
`AbortController`'s `signal` and call `abort()` to cancel. Cancellation during
|
|
110
127
|
the request, response-body read, retry wait, or callback settlement prevents
|
|
@@ -114,12 +131,16 @@ A missing key uses `AI_PROVIDER_NOT_CONFIGURED`, a missing explicit model
|
|
|
114
131
|
throws `TypeError`, and an unsupported `structuredOutput` input uses
|
|
115
132
|
`AI_STRUCTURED_OUTPUT_INVALID`.
|
|
116
133
|
|
|
117
|
-
Optional `onRetry({phase,attempt,delayMs,status,error})` observes
|
|
134
|
+
Optional `onRetry({phase,attempt,delayMs,status,error})` observes recovery.
|
|
118
135
|
`phase:'waiting'` arrives before the abortable delay; `phase:'requesting'`
|
|
119
136
|
arrives immediately before the next Fetch. `attempt` is the one-based upcoming
|
|
120
|
-
retry number across
|
|
137
|
+
retry number across the shared budget, `delayMs` is the selected wait, `status` is the
|
|
121
138
|
HTTP status or `null` for a rejected Fetch, and `error` is the complete original
|
|
122
|
-
failure value.
|
|
139
|
+
failure value. An initial request waiting behind an already observed quota
|
|
140
|
+
reports `attempt:0`, `status:429`, and that complete quota error. Shared pacing
|
|
141
|
+
can extend the actual wait beyond its initial observation; `requesting` is
|
|
142
|
+
emitted only after that wait, immediately before dispatch. This observer runs
|
|
143
|
+
synchronously without awaiting its returned
|
|
123
144
|
promise. Synchronous throws and rejected promises are reported through the
|
|
124
145
|
shared console logger and cannot change the request outcome or trigger another
|
|
125
146
|
retry. `onRequest` still runs once per logical request. Retry observations stay
|
|
@@ -99,7 +99,7 @@ where listed. See [native output and host requirements](core-native-packaging.md
|
|
|
99
99
|
| Label speakers in caller-fed mono audio | [`arcane-os/diarization`, `arcane-os/core/diarization`](diarization.md) | **Node** with built helper, selected official NeMo libraries and Nemotron model; official CPU archives cover Windows x64 and Linux/macOS x64/arm64 | Independent retained streams, complete timed segments and optional probability frames. Generic labels are not personal identities or speech transcripts; cancellation and close observe native cleanup. Android needs host adaptation. |
|
|
100
100
|
| Use local AI without coupling app code to Ollama HTTP | `Arcane.localAI`, `Arcane.ai`, or `/arcane/modules/Ollama.js` | Primarily **Native**; Android exposes a narrower admitted inference projection | Admission, errors, and managed-operation events are normalized. Direct Ollama response envelopes remain **Provider-native**. |
|
|
101
101
|
| Use TWiN Cloud from the renderer profile | `/arcane/modules/AI.js` | **Cloud** from an allowed browser/native renderer | High-level chat behavior is normalized by the module. The TWiN access key authenticates remote LLM chat; raw provider diagnostics remain provider-specific. No automatic cloud fallback is inferred from local failure. |
|
|
102
|
-
| Send one TWiN Cloud request with an explicit key and model | `fetchRequest` from `arcane-os/ai/twin-cloud` | **Node** and **Browser**, using standard Fetch and a remote HTTPS provider | Keeps complete messages and returns the full parsed provider JSON. Rejected Fetch and HTTP 529 share three retries
|
|
102
|
+
| Send one TWiN Cloud request with an explicit key and model | `fetchRequest` from `arcane-os/ai/twin-cloud` | **Node** and **Browser**, using standard Fetch and a remote HTTPS provider | Keeps complete messages and returns the full parsed provider JSON. Rejected Fetch and all HTTP 429/529 share three retries. Quota recovery honors Retry-After or exponential delay; active same-endpoint/credential requests share cooldown and paced dispatch. Optional `onRetry` reports waiting/requesting without blocking recovery. Complete errors and cancellation match browser TWiN transport. No browser profile, AI/user singleton, DBOPFS, or retained request history is created. |
|
|
103
103
|
| Evaluate caller-owned state and questions with TWiN System One | `fetchSystemOneRequest` from `arcane-os/ai/twin-cloud` | **Node** and **Browser**, using standard Fetch and a remote HTTPS provider | Posts the explicit model and complete state/questions through the existing JSON transport. Returns the entire parsed provider result and shares its retry, diagnostic, and cancellation behavior. Model choice, question definitions, answer interpretation, and scoring policy stay with the caller; no conversation state is retained. |
|
|
104
104
|
| Generate images from a complete caller-owned prompt | `generateImages` from `arcane-os/ai/twin-cloud` | **Node** and **Browser**, using standard Fetch, Blob, and a remote HTTPS provider | Caller selects `fal-ai/flux/schnell` or `stable-diffusion-3.5-large`, credentials, and provider parameters. Returns every image as a Blob with media metadata in provider order; owns progress, job polling, and cancellation. Submission is not automatically replayed after an ambiguous failure. Display and storage stay application-owned. |
|
|
105
105
|
| Use speech through one application helper | `/arcane/modules/AI.js` and `Arcane.speech` | **Browser** or **Native** | Whisper and Kokoro keep their on-device paths; explicitly selected Web Speech STT/TTS and TWiN Cloud TTS declare their non-local behavior. `prepareTTSPlayback` supports real audio and silent native descriptors; `fetchTTS` and durable `prepareTTS` require real audio. Provider choice, model, voice, and credentials stay application-owned. |
|